最近在学习automl这个方向,整理总结了一些automl相关的资料。
什么是automl?
自动化机器学习(AutoML)提供了一些方法和过程,使非机器学习专家可以使用机器学习,以提高机器学习的效率并加速机器学习的研究。
近年来,机器学习(ML)取得了相当大的成就,并且越来越多的学科依赖它。但是,这一成功关键取决于人类机器学习专家来执行以下任务:
1数据预处理
2特征筛选
3模型选择
4超参数调优
由于这些任务的复杂性通常超出了非ML专家的范围,因此机器学习应用程序的快速增长产生了对现成的机器学习方法的需求,这些方法可以轻松使用而无需专家知识。我们将最终的研究领域称为针对机器学习AutoML的渐进自动化。作为机器学习的一个新领域,AutoML不仅在机器学习方面得到了更多的关注,而且在计算机视觉,自然语言处理和图形计算方面也得到了越来越多的关注。
没有AutoML的正式定义,从大多数论文的描述中,AutoML的基本过程可以显示如下。
AutoML中需要的基本技术的最新概述:
Company | AutoFE | HPO | NAS |
---|---|---|---|
4paradigm | √ | √ | × |
Alibaba | × | √ | × |
Baidu | × | × | √ |
√ | √ | √ | |
H2O.ai | √ | √ | × |
Microsoft | × | √ | √ |
RapidMiner | √ | √ | × |
Tencent | × | √ | × |
Transwarp | √ | √ | √ |
自动数据清理(Auto clean)
自动化特征工程(Auto FE)
超参数优化(HPO)
元学习(meta learning)
神经建筑搜索(NAS)
Table of Contents
- Papers
- Tutorials
- Articles
- Slides
- Books
- Projects
- Prominent Researchers
Papers
Surveys
- 2019 | AutoML: A Survey of the State-of-the-Art | Xin He, et al. | arXiv |
PDF
- 2019 | Survey on Automated Machine Learning | Marc Zoeller, Marco F. Huber | arXiv |
PDF
- 2019 | Automated Machine Learning: State-of-The-Art and Open Challenges | Radwa Elshawi, et al. | arXiv |
PDF
- 2018 | Taking Human out of Learning Applications: A Survey on Automated Machine Learning | Quanming Yao, et al. | arXiv |
PDF
Automated Feature Engineering
Expand Reduce
- 2017 | AutoLearn — Automated Feature Generation and Selection | Ambika Kaul, et al. | ICDM |
PDF
- 2017 | One button machine for automating feature engineering in relational databases | Hoang Thanh Lam, et al. | arXiv |
PDF
- 2016 | Automating Feature Engineering | Udayan Khurana, et al. | NIPS |
PDF
- 2016 | ExploreKit: Automatic Feature Generation and Selection | Gilad Katz, et al. | ICDM |
PDF
- 2015 | Deep Feature Synthesis: Towards Automating Data Science Endeavors | James Max Kanter, Kalyan Veeramachaneni | DSAA |
PDF
- 2017 | AutoLearn — Automated Feature Generation and Selection | Ambika Kaul, et al. | ICDM |
Hierarchical Organization of Transformations
- 2016 | Cognito: Automated Feature Engineering for Supervised Learning | Udayan Khurana, et al. | ICDMW |
PDF
- 2016 | Cognito: Automated Feature Engineering for Supervised Learning | Udayan Khurana, et al. | ICDMW |
Meta Learning
- 2017 | Learning Feature Engineering for Classification | Fatemeh Nargesian, et al. | IJCAI |
PDF
- 2017 | Learning Feature Engineering for Classification | Fatemeh Nargesian, et al. | IJCAI |
Reinforcement Learning
Evolutionary Algorithms
- 2019 | Evolutionary Neural AutoML for Deep Learning | Jason Liang, et al. | GECCO |
PDF
- 2017 | Large-Scale Evolution of Image Classifiers | Esteban Real, et al. | PMLR |
PDF
- 2002 | Evolving Neural Networks through Augmenting Topologies | Kenneth O.Stanley, Risto Miikkulainen | Evolutionary Computation |
PDF
- 2019 | Evolutionary Neural AutoML for Deep Learning | Jason Liang, et al. | GECCO |
Local Search
- 2017 | Simple and Efficient Architecture Search for Convolutional Neural Networks | Thomoas Elsken, et al. | ICLR |
PDF
- 2017 | Simple and Efficient Architecture Search for Convolutional Neural Networks | Thomoas Elsken, et al. | ICLR |
Meta Learning
- 2016 | Learning to Optimize | Ke Li, Jitendra Malik | arXiv |
PDF
- 2016 | Learning to Optimize | Ke Li, Jitendra Malik | arXiv |
Reinforcement Learning
- 2018 | AMC: AutoML for Model Compression and Acceleration on Mobile Devices | Yihui He, et al. | ECCV |
PDF
- 2018 | Efficient Neural Architecture Search via Parameter Sharing | Hieu Pham, et al. | arXiv |
PDF
- 2017 | Neural Architecture Search with Reinforcement Learning | Barret Zoph, Quoc V. Le | ICLR |
PDF
- 2018 | AMC: AutoML for Model Compression and Acceleration on Mobile Devices | Yihui He, et al. | ECCV |
Transfer Learning
- 2017 | Learning Transferable Architectures for Scalable Image Recognition | Barret Zoph, et al. | arXiv |
PDF
- 2017 | Learning Transferable Architectures for Scalable Image Recognition | Barret Zoph, et al. | arXiv |
Network Morphism
- 2018 | Efficient Neural Architecture Search with Network Morphism | Haifeng Jin, et al. | arXiv |
PDF
- 2018 | Efficient Neural Architecture Search with Network Morphism | Haifeng Jin, et al. | arXiv |
Continuous Optimization
Frameworks
- 2019 | Auptimizer – an Extensible, Open-Source Framework for Hyperparameter Tuning | Jiayi Liu, et al. | IEEE Big Data |
PDF
- 2019 | Towards modular and programmable architecture search | Renato Negrinho, et al. | NeurIPS |
PDF
- 2019 | Evolutionary Neural AutoML for Deep Learning | Jason Liang, et al. | arXiv |
PDF
- 2017 | ATM: A Distributed, Collaborative, Scalable System for Automated Machine Learning | T. Swearingen, et al. | IEEE |
PDF
- 2017 | Google Vizier: A Service for Black-Box Optimization | Daniel Golovin, et al. | KDD |
PDF
- 2015 | AutoCompete: A Framework for Machine Learning Competitions | Abhishek Thakur, et al. | ICML |
PDF
Hyperparameter Optimization
Bayesian Optimization
- 2019 | Bayesian Optimization with Unknown Search Space | NeurIPS |
PDF
- 2019 | Constrained Bayesian optimization with noisy experiments |
PDF
- 2019 | Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning | NeurIPS |
PDF
- 2019 | Practical Two-Step Lookahead Bayesian Optimization | NeurIPS |
PDF
- 2019 | Predictive entropy search for multi-objective bayesian optimization with constraints |
PDF
- 2018 | BOCK: Bayesian optimization with cylindrical kernels | ICML |
PDF
- 2018 | Efficient High Dimensional Bayesian Optimization with Additivity and Quadrature Fourier Features | Mojmír Mutný, et al. | NeurIPS |
PDF
- 2018 | High-Dimensional Bayesian Optimization via Additive Models with Overlapping Groups. | PMLR |
PDF
- 2018 | Maximizing acquisition functions for Bayesian optimization | NeurIPS |
PDF
- 2018 | Scalable hyperparameter transfer learning | NeurIPS |
PDF
- 2016 | Bayesian Optimization with Robust Bayesian Neural Networks | Jost Tobias Springenberg, et al. | NIPS |
PDF
- 2016 | Scalable Hyperparameter Optimization with Products of Gaussian Process Experts | Nicolas Schilling, et al. | PKDD |
PDF
- 2016 | Taking the Human Out of the Loop: A Review of Bayesian Optimization | Bobak Shahriari, et al. | IEEE |
PDF
- 2016 | Towards Automatically-Tuned Neural Networks | Hector Mendoza, et al. | JMLR |
PDF
- 2016 | Two-Stage Transfer Surrogate Model for Automatic Hyperparameter Optimization | Martin Wistuba, et al. | PKDD |
PDF
- 2015 | Efficient and Robust Automated Machine Learning |
PDF
- 2015 | Hyperparameter Optimization with Factorized Multilayer Perceptrons | Nicolas Schilling, et al. | PKDD |
PDF
- 2015 | Hyperparameter Search Space Pruning - A New Component for Sequential Model-Based Hyperparameter Optimization | Martin Wistua, et al. |
PDF
- 2015 | Joint Model Choice and Hyperparameter Optimization with Factorized Multilayer Perceptrons | Nicolas Schilling, et al. | ICTAI |
PDF
- 2015 | Learning Hyperparameter Optimization Initializations | Martin Wistuba, et al. | DSAA |
PDF
- 2015 | Scalable Bayesian optimization using deep neural networks | Jasper Snoek, et al. | ACM |
PDF
- 2015 | Sequential Model-free Hyperparameter Tuning | Martin Wistuba, et al. | ICDM |
PDF
- 2013 | Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms |
PDF
- 2013 | Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures | J. Bergstra | JMLR |
PDF
- 2012 | Practical Bayesian Optimization of Machine Learning Algorithms |
PDF
- 2011 | Sequential Model-Based Optimization for General Algorithm Configuration(extended version) |
PDF
- 2019 | Bayesian Optimization with Unknown Search Space | NeurIPS |
Evolutionary Algorithms
- 2018 | Autostacker: A Compositional Evolutionary Learning System | Boyuan Chen, et al. | arXiv |
PDF
- 2017 | Large-Scale Evolution of Image Classifiers | Esteban Real, et al. | PMLR |
PDF
- 2016 | Automating biomedical data science through tree-based pipeline optimization | Randal S. Olson, et al. | ECAL |
PDF
- 2016 | Evaluation of a tree-based pipeline optimization tool for automating data science | Randal S. Olson, et al. | GECCO |
PDF
- 2018 | Autostacker: A Compositional Evolutionary Learning System | Boyuan Chen, et al. | arXiv |
Lipschitz Functions
- 2017 | Global Optimization of Lipschitz functions | C´edric Malherbe, Nicolas Vayatis | arXiv |
PDF
- 2017 | Global Optimization of Lipschitz functions | C´edric Malherbe, Nicolas Vayatis | arXiv |
Local Search
- 2009 | ParamILS: An Automatic Algorithm Configuration Framework | Frank Hutter, et al. | JAIR |
PDF
- 2009 | ParamILS: An Automatic Algorithm Configuration Framework | Frank Hutter, et al. | JAIR |
Meta Learning
Particle Swarm Optimization
- 2017 | Particle Swarm Optimization for Hyper-parameter Selection in Deep Neural Networks | Pablo Ribalta Lorenzo, et al. | GECCO |
PDF
- 2008 | Particle Swarm Optimization for Parameter Determination and Feature Selection of Support Vector Machines | Shih-Wei Lin, et al. | Expert Systems with Applications |
PDF
- 2017 | Particle Swarm Optimization for Hyper-parameter Selection in Deep Neural Networks | Pablo Ribalta Lorenzo, et al. | GECCO |
Random Search
Transfer Learning
- 2016 | Efficient Transfer Learning Method for Automatic Hyperparameter Tuning | Dani Yogatama, Gideon Mann | JMLR |
PDF
- 2016 | Flexible Transfer Learning Framework for Bayesian Optimisation | Tinu Theckel Joy, et al. | PAKDD |
PDF
- 2016 | Hyperparameter Optimization Machines | Martin Wistuba, et al. | DSAA |
PDF
- 2013 | Collaborative Hyperparameter Tuning | R´emi Bardenet, et al. | ICML |
PDF
- 2016 | Efficient Transfer Learning Method for Automatic Hyperparameter Tuning | Dani Yogatama, Gideon Mann | JMLR |
Miscellaneous
- 2020 | Automated Machine Learning Techniques for Data Streams | Alexandru-Ionut Imbrea |
PDF
- 2018 | Accelerating Neural Architecture Search using Performance Prediction | Bowen Baker, et al. | ICLR |
PDF
- 2017 | Automatic Frankensteining: Creating Complex Ensembles Autonomously | Martin Wistuba, et al. | SIAM |
PDF
Tutorials
Bayesian Optimization
- 2018 | A Tutorial on Bayesian Optimization. |
PDF
- 2010 | A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning |
PDF
Meta Learning
- 2008 | Metalearning - A Tutorial |
PDF
Blog
Type | Blog Title | Link |
---|---|---|
HPO | Bayesian Optimization for Hyperparameter Tuning | Link |
Meta-Learning | Learning to learn | Link |
Meta-Learning | Why Meta-learning is Crucial for Further Advances of Artificial Intelligence? | Link |
Books
Year of Publication | Type | Book Title | Authors | Publisher | Link |
---|---|---|---|---|---|
2009 | Meta-Learning | Metalearning - Applications to Data Mining | Brazdil, P., Giraud Carrier, C., Soares, C., Vilalta, R. | Springer | Download |
2019 | HPO, Meta-Learning, NAS | AutoML: Methods, Systems, Challenges | Frank Hutter, Lars Kotthoff, Joaquin Vanschoren | Download |
Projects
Project | Type | Language | License | Link |
---|---|---|---|---|
AdaNet | NAS | Python | Apache-2.0 | Github |
Advisor | HPO | Python | Apache-2.0 | Github |
AMLA | HPO, NAS | Python | Apache-2.0 | Github |
ATM | HPO | Python | MIT | Github |
Auger | HPO | Python | Commercial | Homepage |
auptimizer | HPO, NAS | Python (support R script) | GPL-3.0 | Github |
Auto-Keras | NAS | Python | License |
Github |
AutoML Vision | NAS | Python | Commercial | Homepage |
AutoML Video Intelligence | NAS | Python | Commercial | Homepage |
AutoML Natural Language | NAS | Python | Commercial | Homepage |
AutoML Translation | NAS | Python | Commercial | Homepage |
AutoML Tables | AutoFE, HPO | Python | Commercial | Homepage |
auto-sklearn | HPO | Python | License |
Github |
auto_ml | HPO | Python | MIT | Github |
BayesianOptimization | HPO | Python | MIT | Github |
BayesOpt | HPO | C++ | AGPL-3.0 | Github |
comet | HPO | Python | Commercial | Homepage |
DataRobot | HPO | Python | Commercial | Homepage |
DEvol | NAS | Python | MIT | Github |
DeepArchitect | NAS | Python | MIT | Github |
Driverless AI | AutoFE | Python | Commercial | Homepage |
FAR-HO | HPO | Python | MIT | Github |
H2O AutoML | HPO | Python, R, Java, Scala | Apache-2.0 | Github |
HpBandSter | HPO | Python | BSD-3-Clause | Github |
HyperBand | HPO | Python | License |
Github |
Hyperopt | HPO | Python | License |
Github |
Hyperopt-sklearn | HPO | Python | License |
Github |
Hyperparameter Hunter | HPO | Python | MIT | Github |
Katib | HPO | Python | Apache-2.0 | Github |
MateLabs | HPO | Python | Commercial | Github |
Milano | HPO | Python | Apache-2.0 | Github |
MLJAR | HPO | Python | Commercial | Homepage |
nasbot | NAS | Python | MIT | Github |
neptune | HPO | Python | Commercial | Homepage |
NNI | HPO, NAS | Python | MIT | Github |
Oboe | HPO | Python | BSD-3-Clause | Github |
Optunity | HPO | Python | License |
Github |
R2.ai | HPO | Commercial | Homepage |
|
RBFOpt | HPO | Python | License |
Github |
RoBO | HPO | Python | BSD-3-Clause | Github |
Scikit-Optimize | HPO | Python | License |
Github |
SigOpt | HPO | Python | Commercial | Homepage |
SMAC3 | HPO | Python | License |
Github |
TPOT | AutoFE, HPO | Python | LGPL-3.0 | Github |
TransmogrifAI | HPO | Scala | BSD-3-Clause | Github |
Tune | HPO | Python | Apache-2.0 | Github |
Xcessiv | HPO | Python | Apache-2.0 | Github |
SmartML | HPO | R | GPL-3.0 | Github |
MLBox | AutoFE, HPO | Python | BSD-3 License | Github |
AutoAI Watson | AutoFE, HPO | Commercial | Homepage |
Slides
Type | Slide Title | Authors | Link |
---|---|---|---|
AutoFE | Automated Feature Engineering for Predictive Modeling | Udyan Khurana, etc al. | Download |
HPO | A Tutorial on Bayesian Optimization for Machine Learning | Ryan P. Adams | Download |
HPO | Bayesian Optimisation | Gilles Louppe | Download |
引用:
【1】https://github.com/hibayesian/awesome-automl-papers#bayesian-optimization
【2】https://github.com/keras-team/autokeras
【3】https://github.com/h2oai/h2o-3/
【4】https://github.com/Microsoft/nni