automl survey

auto ml
auto ml
Publish Date:   2020-04-13

最近在学习automl这个方向，整理总结了一些automl相关的资料。

什么是automl?

自动化机器学习（AutoML）提供了一些方法和过程，使非机器学习专家可以使用机器学习，以提高机器学习的效率并加速机器学习的研究。
近年来，机器学习（ML）取得了相当大的成就，并且越来越多的学科依赖它。但是，这一成功关键取决于人类机器学习专家来执行以下任务：
1数据预处理
2特征筛选
3模型选择
4超参数调优
由于这些任务的复杂性通常超出了非ML专家的范围，因此机器学习应用程序的快速增长产生了对现成的机器学习方法的需求，这些方法可以轻松使用而无需专家知识。我们将最终的研究领域称为针对机器学习AutoML的渐进自动化。作为机器学习的一个新领域，AutoML不仅在机器学习方面得到了更多的关注，而且在计算机视觉，自然语言处理和图形计算方面也得到了越来越多的关注。

没有AutoML的正式定义，从大多数论文的描述中，AutoML的基本过程可以显示如下。

AutoML中需要的基本技术的最新概述：

Company AutoFE HPO NAS
4paradigm √ √ ×
Alibaba × √ ×
Baidu × × √
Google √ √ √
H2O.ai √ √ ×
Microsoft × √ √
RapidMiner √ √ ×
Tencent × √ ×
Transwarp √ √ √

自动数据清理（Auto clean）

自动化特征工程（Auto FE）

超参数优化（HPO）

元学习(meta learning)

神经建筑搜索（NAS）

Table of Contents

Papers

Surveys

  • 2019 | AutoML: A Survey of the State-of-the-Art | Xin He, et al. | arXiv | PDF
  • 2019 | Survey on Automated Machine Learning | Marc Zoeller, Marco F. Huber | arXiv | PDF
  • 2019 | Automated Machine Learning: State-of-The-Art and Open Challenges | Radwa Elshawi, et al. | arXiv | PDF
  • 2018 | Taking Human out of Learning Applications: A Survey on Automated Machine Learning | Quanming Yao, et al. | arXiv | PDF

    Automated Feature Engineering

  • Expand Reduce

    • 2017 | AutoLearn — Automated Feature Generation and Selection | Ambika Kaul, et al. | ICDM | PDF
    • 2017 | One button machine for automating feature engineering in relational databases | Hoang Thanh Lam, et al. | arXiv | PDF
    • 2016 | Automating Feature Engineering | Udayan Khurana, et al. | NIPS | PDF
    • 2016 | ExploreKit: Automatic Feature Generation and Selection | Gilad Katz, et al. | ICDM | PDF
    • 2015 | Deep Feature Synthesis: Towards Automating Data Science Endeavors | James Max Kanter, Kalyan Veeramachaneni | DSAA | PDF

  • Hierarchical Organization of Transformations

    • 2016 | Cognito: Automated Feature Engineering for Supervised Learning | Udayan Khurana, et al. | ICDMW | PDF

  • Meta Learning

    • 2017 | Learning Feature Engineering for Classification | Fatemeh Nargesian, et al. | IJCAI | PDF

  • Reinforcement Learning

    • 2017 | Feature Engineering for Predictive Modeling using Reinforcement Learning | Udayan Khurana, et al. | arXiv | PDF
    • 2010 | Feature Selection as a One-Player Game | Romaric Gaudel, Michele Sebag | ICML | PDF

  • Evolutionary Algorithms

    • 2019 | Evolutionary Neural AutoML for Deep Learning | Jason Liang, et al. | GECCO | PDF
    • 2017 | Large-Scale Evolution of Image Classifiers | Esteban Real, et al. | PMLR | PDF
    • 2002 | Evolving Neural Networks through Augmenting Topologies | Kenneth O.Stanley, Risto Miikkulainen | Evolutionary Computation | PDF
    • 2017 | Simple and Efficient Architecture Search for Convolutional Neural Networks | Thomoas Elsken, et al. | ICLR | PDF

  • Meta Learning

    • 2016 | Learning to Optimize | Ke Li, Jitendra Malik | arXiv | PDF

  • Reinforcement Learning

    • 2018 | AMC: AutoML for Model Compression and Acceleration on Mobile Devices | Yihui He, et al. | ECCV | PDF
    • 2018 | Efficient Neural Architecture Search via Parameter Sharing | Hieu Pham, et al. | arXiv | PDF
    • 2017 | Neural Architecture Search with Reinforcement Learning | Barret Zoph, Quoc V. Le | ICLR | PDF

  • Transfer Learning

    • 2017 | Learning Transferable Architectures for Scalable Image Recognition | Barret Zoph, et al. | arXiv | PDF

  • Network Morphism

    • 2018 | Efficient Neural Architecture Search with Network Morphism | Haifeng Jin, et al. | arXiv | PDF

  • Continuous Optimization

    • 2018 | Neural Architecture Optimization | Renqian Luo, et al. | arXiv | PDF
    • 2019 | DARTS: Differentiable Architecture Search | Hanxiao Liu, et al. | ICLR | PDF

Frameworks

  • 2019 | Auptimizer – an Extensible, Open-Source Framework for Hyperparameter Tuning | Jiayi Liu, et al. | IEEE Big Data | PDF
  • 2019 | Towards modular and programmable architecture search | Renato Negrinho, et al. | NeurIPS | PDF
  • 2019 | Evolutionary Neural AutoML for Deep Learning | Jason Liang, et al. | arXiv | PDF
  • 2017 | ATM: A Distributed, Collaborative, Scalable System for Automated Machine Learning | T. Swearingen, et al. | IEEE | PDF
  • 2017 | Google Vizier: A Service for Black-Box Optimization | Daniel Golovin, et al. | KDD |PDF
  • 2015 | AutoCompete: A Framework for Machine Learning Competitions | Abhishek Thakur, et al. | ICML | PDF

Hyperparameter Optimization

  • Bayesian Optimization

    • 2019 | Bayesian Optimization with Unknown Search Space | NeurIPS | PDF
    • 2019 | Constrained Bayesian optimization with noisy experiments | PDF
    • 2019 | Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning | NeurIPS | PDF
    • 2019 | Practical Two-Step Lookahead Bayesian Optimization | NeurIPS | PDF
    • 2019 | Predictive entropy search for multi-objective bayesian optimization with constraints | PDF
    • 2018 | BOCK: Bayesian optimization with cylindrical kernels | ICML | PDF
    • 2018 | Efficient High Dimensional Bayesian Optimization with Additivity and Quadrature Fourier Features | Mojmír Mutný, et al. | NeurIPS | PDF
    • 2018 | High-Dimensional Bayesian Optimization via Additive Models with Overlapping Groups. | PMLR | PDF
    • 2018 | Maximizing acquisition functions for Bayesian optimization | NeurIPS | PDF
    • 2018 | Scalable hyperparameter transfer learning | NeurIPS | PDF
    • 2016 | Bayesian Optimization with Robust Bayesian Neural Networks | Jost Tobias Springenberg， et al. | NIPS | PDF
    • 2016 | Scalable Hyperparameter Optimization with Products of Gaussian Process Experts | Nicolas Schilling, et al. | PKDD | PDF
    • 2016 | Taking the Human Out of the Loop: A Review of Bayesian Optimization | Bobak Shahriari, et al. | IEEE | PDF
    • 2016 | Towards Automatically-Tuned Neural Networks | Hector Mendoza, et al. | JMLR | PDF
    • 2016 | Two-Stage Transfer Surrogate Model for Automatic Hyperparameter Optimization | Martin Wistuba, et al. | PKDD | PDF
    • 2015 | Efficient and Robust Automated Machine Learning | PDF
    • 2015 | Hyperparameter Optimization with Factorized Multilayer Perceptrons | Nicolas Schilling, et al. | PKDD | PDF
    • 2015 | Hyperparameter Search Space Pruning - A New Component for Sequential Model-Based Hyperparameter Optimization | Martin Wistua, et al. | PDF
    • 2015 | Joint Model Choice and Hyperparameter Optimization with Factorized Multilayer Perceptrons | Nicolas Schilling, et al. | ICTAI | PDF
    • 2015 | Learning Hyperparameter Optimization Initializations | Martin Wistuba, et al. | DSAA | PDF
    • 2015 | Scalable Bayesian optimization using deep neural networks | Jasper Snoek, et al. | ACM | PDF
    • 2015 | Sequential Model-free Hyperparameter Tuning | Martin Wistuba, et al. | ICDM | PDF
    • 2013 | Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms | PDF
    • 2013 | Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures | J. Bergstra | JMLR | PDF
    • 2012 | Practical Bayesian Optimization of Machine Learning Algorithms | PDF
    • 2011 | Sequential Model-Based Optimization for General Algorithm Configuration(extended version) | PDF

  • Evolutionary Algorithms

    • 2018 | Autostacker: A Compositional Evolutionary Learning System | Boyuan Chen, et al. | arXiv | PDF
    • 2017 | Large-Scale Evolution of Image Classifiers | Esteban Real, et al. | PMLR | PDF
    • 2016 | Automating biomedical data science through tree-based pipeline optimization | Randal S. Olson, et al. | ECAL | PDF
    • 2016 | Evaluation of a tree-based pipeline optimization tool for automating data science | Randal S. Olson, et al. | GECCO | PDF

  • Lipschitz Functions

    • 2017 | Global Optimization of Lipschitz functions | C´edric Malherbe, Nicolas Vayatis | arXiv | PDF

  • Local Search

    • 2009 | ParamILS: An Automatic Algorithm Configuration Framework | Frank Hutter, et al. | JAIR | PDF

  • Meta Learning

    • 2008 | Cross-Disciplinary Perspectives on Meta-Learning for Algorithm Selection | PDF
    • 2019 | SMARTML: A Meta Learning-Based Framework for Automated Selection and Hyperparameter Tuning for Machine Learning Algorithms | PDF

  • Particle Swarm Optimization

    • 2017 | Particle Swarm Optimization for Hyper-parameter Selection in Deep Neural Networks | Pablo Ribalta Lorenzo, et al. | GECCO | PDF
    • 2008 | Particle Swarm Optimization for Parameter Determination and Feature Selection of Support Vector Machines | Shih-Wei Lin, et al. | Expert Systems with Applications | PDF
    • 2016 | Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization | Lisha Li, et al. | arXiv | PDF
    • 2012 | Random Search for Hyper-Parameter Optimization | James Bergstra, Yoshua Bengio | JMLR | PDF
    • 2011 | Algorithms for Hyper-parameter Optimization | James Bergstra, et al. | NIPS | PDF

  • Transfer Learning

    • 2016 | Efficient Transfer Learning Method for Automatic Hyperparameter Tuning | Dani Yogatama, Gideon Mann | JMLR | PDF
    • 2016 | Flexible Transfer Learning Framework for Bayesian Optimisation | Tinu Theckel Joy, et al. | PAKDD | PDF
    • 2016 | Hyperparameter Optimization Machines | Martin Wistuba, et al. | DSAA | PDF
    • 2013 | Collaborative Hyperparameter Tuning | R´emi Bardenet, et al. | ICML | PDF

Miscellaneous

  • 2020 | Automated Machine Learning Techniques for Data Streams | Alexandru-Ionut Imbrea | PDF
  • 2018 | Accelerating Neural Architecture Search using Performance Prediction | Bowen Baker, et al. | ICLR | PDF
  • 2017 | Automatic Frankensteining: Creating Complex Ensembles Autonomously | Martin Wistuba, et al. | SIAM | PDF

Tutorials

Bayesian Optimization

  • 2018 | A Tutorial on Bayesian Optimization. | PDF
  • 2010 | A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning | PDF

    Meta Learning

  • 2008 | Metalearning - A Tutorial | PDF

Blog

Type Blog Title Link
HPO Bayesian Optimization for Hyperparameter Tuning Link
Meta-Learning Learning to learn Link
Meta-Learning Why Meta-learning is Crucial for Further Advances of Artificial Intelligence? Link

Books

Year of Publication Type Book Title Authors Publisher Link
2009 Meta-Learning Metalearning - Applications to Data Mining Brazdil, P., Giraud Carrier, C., Soares, C., Vilalta, R. Springer Download
2019 HPO, Meta-Learning, NAS AutoML: Methods, Systems, Challenges Frank Hutter, Lars Kotthoff, Joaquin Vanschoren Download

Projects

Project Type Language License Link
AdaNet NAS Python Apache-2.0 Github
Advisor HPO Python Apache-2.0 Github
AMLA HPO, NAS Python Apache-2.0 Github
ATM HPO Python MIT Github
Auger HPO Python Commercial Homepage
auptimizer HPO, NAS Python (support R script) GPL-3.0 Github
Auto-Keras NAS Python License Github
AutoML Vision NAS Python Commercial Homepage
AutoML Video Intelligence NAS Python Commercial Homepage
AutoML Natural Language NAS Python Commercial Homepage
AutoML Translation NAS Python Commercial Homepage
AutoML Tables AutoFE, HPO Python Commercial Homepage
auto-sklearn HPO Python License Github
auto_ml HPO Python MIT Github
BayesianOptimization HPO Python MIT Github
BayesOpt HPO C++ AGPL-3.0 Github
comet HPO Python Commercial Homepage
DataRobot HPO Python Commercial Homepage
DEvol NAS Python MIT Github
DeepArchitect NAS Python MIT Github
Driverless AI AutoFE Python Commercial Homepage
FAR-HO HPO Python MIT Github
H2O AutoML HPO Python, R, Java, Scala Apache-2.0 Github
HpBandSter HPO Python BSD-3-Clause Github
HyperBand HPO Python License Github
Hyperopt HPO Python License Github
Hyperopt-sklearn HPO Python License Github
Hyperparameter Hunter HPO Python MIT Github
Katib HPO Python Apache-2.0 Github
MateLabs HPO Python Commercial Github
Milano HPO Python Apache-2.0 Github
MLJAR HPO Python Commercial Homepage
nasbot NAS Python MIT Github
neptune HPO Python Commercial Homepage
NNI HPO, NAS Python MIT Github
Oboe HPO Python BSD-3-Clause Github
Optunity HPO Python License Github
R2.ai HPO Commercial Homepage
RBFOpt HPO Python License Github
RoBO HPO Python BSD-3-Clause Github
Scikit-Optimize HPO Python License Github
SigOpt HPO Python Commercial Homepage
SMAC3 HPO Python License Github
TPOT AutoFE, HPO Python LGPL-3.0 Github
TransmogrifAI HPO Scala BSD-3-Clause Github
Tune HPO Python Apache-2.0 Github
Xcessiv HPO Python Apache-2.0 Github
SmartML HPO R GPL-3.0 Github
MLBox AutoFE, HPO Python BSD-3 License Github
AutoAI Watson AutoFE, HPO Commercial Homepage

Slides

Type Slide Title Authors Link
AutoFE Automated Feature Engineering for Predictive Modeling Udyan Khurana, etc al. Download
HPO A Tutorial on Bayesian Optimization for Machine Learning Ryan P. Adams Download
HPO Bayesian Optimisation Gilles Louppe Download

引用：
【1】https://github.com/hibayesian/awesome-automl-papers#bayesian-optimization
【2】https://github.com/keras-team/autokeras
【3】https://github.com/h2oai/h2o-3/
【4】https://github.com/Microsoft/nni

Author: inner peace
Link: https://dshub.cn/2020/04/13/automl-survey/
Reprint policy: All articles in this blog are used except for special statements CC BY 4.0 reprint polocy. If reproduced, please indicate source inner peace !
auto ml
赏
评论
  TOC