Publications proposing novel machine learning methods are often primarily rated by exhibited predictive performance on selected problems. In this position paper we argue that predictive performance alone is not a good indicator for the worth of a publication. Using it as such even fosters problems like inefficiencies of the machine learning research community as a whole and setting wrong incentives for researchers. We therefore put out a call for the publication of “negative” results, which can help alleviate some of these problems and improve the scientific output of the machine learning research community. To substantiate our position, we present the advantages of publishing negative results and provide concrete measures for the community to move towards a paradigm where their publication is normalized.

Automated machine learning (AutoML) was formed around the fundamental objectives of automatically and efficiently configuring machine learning (ML) workflows, aiding the research of new ML algorithms, and contributing to the democratization of ML by making it accessible to a broader audience. Over the past decade, commendable achievements in AutoML have primarily focused on optimizing predictive performance. This focused progress, while substantial, raises questions about how well AutoML has met its broader, original goals. In this position paper, we argue that a key to unlocking AutoML’s full potential lies in addressing the currently underexplored aspect of user interaction with AutoML systems, including their diverse roles, expectations, and expertise. We envision a more human-centered approach in future AutoML research, promoting the collaborative design of ML systems that tightly integrates the complementary strengths of human expertise and AutoML methodologies.

Hyperparameter optimization constitutes a large part of typical modern machine learning (ML) workflows. This arises from the fact that ML methods and corresponding preprocessing steps often only yield optimal performance when hyperparameters are properly tuned. But in many applications, we are not only interested in optimizing ML pipelines solely for predictive accuracy; additional metrics or constraints must be considered when determining an optimal configuration, resulting in a multi-objective optimization problem. This is often neglected in practice, due to a lack of knowledge and readily available software implementations for multi-objective hyperparameter optimization. In this work, we introduce the reader to the basics of multi-objective hyperparameter optimization and motivate its usefulness in applied ML. Furthermore, we provide an extensive survey of existing optimization strategies from the domains of evolutionary algorithms and Bayesian optimization. We illustrate the utility of multi-objective optimization in several specific ML applications, considering objectives such as operating conditions, prediction time, sparseness, fairness, interpretability, and robustness.

Innovation processes of technical products underlie dynamic company internal and external influencing factors, which require a high degree of flexibility. Part of this flexibility is an adaptive behavior towards technical changes, in general within development as well as within production. These cycles of adaptations and changes necessitate an active coordination of information flows between the relevant domains. Accordingly, information about changes, e.g. the reasons as well as the consequences of changes, is transmitted at various interfaces within the innovation process. This provides the basis for deciding about the implementation of changes and adaptations across the different domains. Research has addressed the support of information exchange within innovation processes and specifically between the domains of product development and production. Nonetheless, the dynamics of internal and external influencing factors as well as of the related information exchange are not reflected comprehensively in approaches to support information exchange between these domains. Therefore, this paper addresses shortcomings of existing approaches regarding dynamics of information exchange through a literature based elaboration of interfaces and information flows between strategic product planning, product development and production planning. In this context, the production planning is represented by manufacturing technology, production resource and production structure planning. Accordingly, this paper aims at supporting sound decision making on the implementation of cyclic changes and adaptations in development and production by providing a classification of communication interfaces and information flows across domains.