Hyperband has become a popular method to tune the hyperparameters (HPs) of expensive machine learning models, whose performance depends on the amount of resources allocated for training. While Hyperband is conceptually simple, combining random search to a successive halving technique to reallocate resources to the most promising HPs, it often outperforms standard Bayesian optimization when solutions with moderate precision are sufficient. In this paper, we propose a model-based extension of Hyperband, replacing the uniform random sampling of HP candidates by an adaptive non-uniform sampling procedure. We show that our extension not only improves the precision resolution of Hyperband but also supports transfer learning, both, within a Hyperband run and across previous HP tuning tasks. We apply the method to the problem of tuning the learning rate when solving linear regression problems and to the optimization of the HPs of XGBoost binary classifiers across different datasets, showing that we favorably compare with recently proposed extensions of Hyperband.
Research areas
