Machine learning (ML) is a key AI technology these days and has a major impact on all kinds of fields in research, industry, and even people's daily lives. However, developing new ML applications still comes with major challenges. One of these challenges is that ML practitioners need many years of training and experience to efficiently develop new applications. Even after acquiring sufficient expertise, many design decisions are still made by trial and error, incl. hyperparameters and the architecture of deep neural networks. This implies that the development of ML applications is typically time-consuming, tedious, and error-prone. Nevertheless, the performance of ML applications heavily depends on these design decisions; it can make the difference between state-of-the-art performance and failing to learn anything reasonable.

Automated machine learning (AutoML) supports ML practitioners in making these design decisions efficiently. The main idea is to leverage the strength of efficient optimization in computer science. In particular, AutoML approaches build e.g., on Bayesian Optimization, Evolutionary Algorithms or Reinforcement Learning to determine well-performing hyperparameter configurations, neural architectures of DNNs, or even entire ML pipelines, including feature engineering, feature preprocessing, model selection, and postprocessing. Thus, AutoML improves the efficiency of developing new AI applications, the overall performance of ML models, and even reproducibility.

Research on AutoML in the last decade has led to tremendous speedups in how to discover ML designs for all the aspects mentioned above. For example, multi-fidelity optimization improved hyperparameter optimization by at least an order of magnitude, differentiable architecture search reduced the amount of required GPU time from thousands of GPU hours to less than a single GPU day, and meta-learning allowed the transfer of knowledge between AutoML tasks, e.g., to warmstart a surrogate-based AutoML optimizers.

Although AutoML is very successful, the community is still very scattered in different subcommunities, e.g., evolutionary computation (e.g., GECCO and PPSN), machine learning (e.g., ICML and NeurIPS) or computer vision (e.g., CVPR and ICCV), and around the globe, in particular east Asia, Europe and North America. Therefore, coordinated efforts are needed to bring the community together to create synergies between different directions in AutoML.

With a Shonan seminar, we strive to develop novel solutions for the biggest challenges in AutoML, including

1. How can we make AutoML applicable to the largest models (in particular foundational models such as LLMs and multi-modal models)? Although AutoML has become a lot more efficient in recent years, there are still no feasible solutions for how to apply AutoML to model trainings that run for weeks on hundreds of GPUs and can only be afforded very few times – in the worst case, only a single time.
2. How can AutoML contribute to the threat of the climate crisis? On the one hand, AutoML has the potential to contribute to applications for sustainability by making AI usable for non-domain experts. On the other hand, AutoML itself requires a lot of computing resources and thus more compute-efficient approaches need to be developed.
3. How can we make AutoML more interactive, rather than simply providing a black-box system? Often the solution to an AutoML approach is not exactly what the data scientist has in mind, but with current AutoML systems, there is no obvious way to address this. The data scientist has to resort to running the system again from scratch, tweaking the problem formulation in a way that they think will give a better solution, rather than being able to interact with the system directly. Similarly, there is little opportunity for the user to monitor runs of AutoML systems and nudge them in a different direction while running.
4. How can we help users of AutoML systems formulate their ML problems to be solved? The first hurdle to using ML and AutoML is specifying the problem, which often is more difficult than it appears. Users may not be familiar with what ML can and cannot do, how to set up data to be suitable for ML, and what kinds and quantities of data may be required. While AutoML has made great strides in making state-of-the-art ML more accessible, very little research focuses on helping the user specify their problem.

The above challenges will guide the discussions at the proposed seminar. We expect the following technical topics to feature prominently in the discussions:

• Multi-fidelity methods as a means of addressing the large resource requirements of current AutoML approaches. While recent work has popularized multi-fidelity approaches, e.g. through the Hyperband system, this is still a relatively under-explored area with the promise to help address most of the biggest challenges mentioned above. In particular multi-fidelity approaches that consider different qualities of data instead of simply different approximations of e.g. generalization accuracy are currently not considered by the community. For example, we can estimate the performance of a system that allows to make predictions from multi-modal data by looking at each modality in turn, but are estimates for one modality meaningful for another modality? This question is of interest in many application areas of AutoML, for example in Materials Science.

• Visualization and human-computer interaction methods can help to open the black box of an AutoML system and to ensure that what AutoML does is relevant to the user and they are in control of the process. This is where most existing AutoML systems fall short, which inhibits their wider adoption, in particular in the sciences, where explainability and trust are especially important.

By bringing together AutoML experts from different communities and with different backgrounds, an NII Shonan seminar will substantially contribute to finding solutions for the challenges mentioned above.