Beyond accuracy, adversarial robustness and interpretability are crucial elements of trustworthy machine learning systems.Prototypebased models such as generalized learning vector quantization (GLVQ) have favourable robustness and interpretability properties but do not achieve state-of-the-art accuracy in many practical domains, such as image classification.Applying prototype-based models in the embedding space of a deep neural network boosts their accuracy but removes their interpretability and robustness.We partially resolve this dilemma: We prove that robustness guarantees of shallow classifiers translate to robustness guarantees of deep classifiers when imposing Lipschitz continuity, we provide a training scheme to achieve Lipschitz continuity, and we empirically validate our approach on three image classification data sets against fast gradient sign attacks.

Evaluation is an integral part of developing machine learning and AI-based systems for real-world applications.Given the transformative changes induced by ML/AI systems like large language models, evaluation needs to go beyond performance and include robustness, fairness, user perception, and legal compliance to ensure responsible usage.Further, evaluation practices need to also consider the full range of applications beyond the classic batch setting of machine learning, i.e., data streams, recommender systems, reinforcement learning, and foundation models.This paper provides an analysis of current evaluation practices and gaps across settings and dimensions, and argues for holistic, reproducible evaluation beyond benchmark performance.Recently, machine learning (ML) and artificial intelligence (AI) research has been grappling with an evaluation paradox: while ML systems, especially large language models (LLMs), appear to perform ever better in benchmarks, outperforming humans in many cases, this performance does not translate to success of deployed ML/AI systems in the real world, with 95% of AI projects in industry failing to provide meaningful return on investment [1,2,3].This highlights the need to rethink evaluation to keep pace with current developments.First, the appearance of foundation models, including LLMs, poses new challenges as foundation models are intended for, and hence need to be evaluated on, many different tasks at the same time [1,3].Being trained on huge amounts of data from the internet, data leakage between test benchmarks and training data becomes a considerable risk [4].Second, we observe an increase in real-world applications beyond batch machine learning, inducing additional challenges, like noisy data and dynamic environments, which need to be reflected in the evaluation process [5].Third, when applying ML/AI systems in settings that affect human users, there is a need for evaluation beyond performance: Considering robustness and safety, fairness, explainability, privacy, and legal aspects in addition to performance is paramount to ensure performant and accountable application of ML/AI systems [6].In this tutorial paper, we will analyze the state of evaluation along two axes (see Table 1), namely evaluation dimensions and settings.In terms of * VV, UK and BP gratefully acknowledge funding for the project KI-Akademie OWL, financed by the Federal Ministry of Research, Technology and Space (BMFTR) and supported by the VDI

The risk posed by a tumor can be described on two axes: how far the tumor has already grown and spread in the body (staging) and how dangerous/(de-)differentiated the tumor is irrespective of staging due to its pathological features (grading). When we apply survival analysis to cancer data, we implicitly mix staging and grading. Our goal is to develop novel survival analysis approaches to support medical experts in deriving a grading scheme from data. Hence, we need models that are small and sparse enough for medical experts to analyze, take bio-medical constraints into account, are still predictive of survival, and, most importantly, are decorrelated from staging indicators. Learning such models from small data is a novel machine learning problem that we dub decorrelated sparse survival regression (DSSR). In this paper, we provide three algorithms for DSSR, develop further mechanisms for model interpretability, and evaluate all algorithms on both simulated and real-world cancer data. • We define the machine learning problem of decorrelated sparse survival regression (DSSR) with applications in tumor grading. • We provide three novel algorithms to address the problem. • We evaluate the algorithms on both simulated and real-world data and find substantial improvements beyond prior algorithms, especially in terms of interpretability.

Machine learning approaches have shown impressive achievements in bionic prostheses control.However, translating the machine learning models from labs to patient's everyday lives remains a challenge due to various disturbances, such as electrodes shifts.To mitigate the influence of electrode shifts, we investigate two linear domain adaptation methods and a robust training approach.In experiments, we compare all methods on both simulated electrode shifts on the Ninapro DB2 data set as well as real electrode shifts on Ninapro DB8.We find that linear domain adaptation could estimate the shift and reduce the impact of electrodes shift best, but robust training approaches similar performance without the need for new data.

Federated learning (FL) facilitates shared training of machine learning models while maintaining data privacy. Unfortunately, it suffers from data imbalance among participating clients, causing the performance of the shared model to drop. To diminish the negative effects of unfavorable data-specific properties, both algorithm- and data-based approaches seek to make FL more resilient against them. In this regard, data-based approaches prove to be more versatile and require less domain knowledge to be applied efficiently. Hence, they seem particularly suitable for widespread application in various FL environments. Although data-based approaches such as local data sampling have been applied to FL in the past, previous research did not provide a systematic analysis of the potential and limitations of individual data sampling strategies to improve FL. To this end, we (1) identify relevant local data sampling strategies for FL, (2) identify data-specific properties that negatively affect FL performance, and (3) provide a benchmark of local data sampling strategies regarding their effect on model performance, convergence, and training time in synthetic, real-world, and large-scale FL environments. Moreover, we propose and rigorously test a novel method for data sampling in FL that locally optimizes the choice of sampling strategy prior to FL participation. Our results show that FL can benefit from applying local data sampling in terms of performance and convergence rate, especially when data imbalance is high or the number of clients and samples is low. Furthermore, our proposed sampling strategy offers the best trade-off between model performance and training time.

Solidarity is a crucial concept to understand social relations in societies.In this paper, we explore fine-grained solidarity frames to study solidarity towards women and migrants in German parliamentary debates between 1867 and 2022.Using 2,864 manually annotated text snippets (with a cost exceeding 18k Euro), we evaluate large language models (LLMs) like Llama 3, GPT-3.5, and GPT-4.We find that GPT-4 outperforms other LLMs, approaching human annotation quality.Using GPT-4, we automatically annotate more than 18k further instances (with a cost of around 500 Euro) across 155 years and find that solidarity with migrants outweighs anti-solidarity but that frequencies and solidarity types shift over time.Most importantly, group-based notions of (anti-)solidarity fade in favor of compassionate solidarity, focusing on the vulnerability of migrant groups, and exchange-based anti-solidarity, focusing on the lack of (economic) contribution.Our study highlights the interplay of historical events, socio-economic needs, and political ideologies in shaping migration discourse and social cohesion.We also show that powerful LLMs, if carefully prompted, can be costeffective alternatives to human annotation for hard social scientific tasks.

Accurate motion classification from surface electromyography signals is crucial for controlling bionic prostheses.Unfortunately, most state-of-the-art classifiers need to be re-trained with lots of data to recognize any new motion.Therefore, we propose a few-shot similarity learning approach that can be applied to new classes without any re-training, just using one to five reference points per new class.In experiments on two real-world data sets, we find that our proposed approach outperforms two state-of-the-art approaches for few-shot learning on sEMG signals, namely a transfer learning and a contrastive learning approach.Our experiments also reveal that the choice of loss function is crucial for performance whereas the choice of similarity function has less effect.

In medical pathology, tumor grading is concerned with estimating the risk posed by a tumor, based on its pathological features.One way to infer risk scores is survival regression, i.e. using machine learning to infer a score that predicts the remaining survival time of a patient.Unfortunately, if applied naively, such a score is a mix of the intrinsic risk posed by the tumor and other risk factors, like the progression of the tumor or patient gender and age.We provide the first survival regression model that disentangles tumor grading from undesired correlations, while retaining a high degree of model interpretability, thanks to convex optimization, nonnegativity constraints, sparsity, and linearity.We evaluate the proposed approach both on simulated and real-world data from N = 114 patients at the University Clinic Aachen.

Abstract Machine learning on trees has been mostly focused on trees as input. Much less research has investigated trees as output, which has many applications, such as molecule optimization for drug discovery, or hint generation for intelligent tutoring systems. In this work, we propose a novel autoencoder approach, called recursive tree grammar autoencoder (RTG-AE), which encodes trees via a bottom-up parser and decodes trees via a tree grammar, both learned via recursive neural networks that minimize the variational autoencoder loss. The resulting encoder and decoder can then be utilized in subsequent tasks, such as optimization and time series prediction. RTG-AEs are the first model to combine three features: recursive processing, grammatical knowledge, and deep learning. Our key message is that this unique combination of all three features outperforms models which combine any two of the three. Experimentally, we show that RTG-AE improves the autoencoding error, training time, and optimization score on synthetic as well as real datasets compared to four baselines. We further prove that RTG-AEs parse and generate trees in linear time and are expressive enough to handle all regular tree grammars.

Differentiable neural computers (DNCs) extend artificial neural networks with an explicit memory without interference, thus enabling the model to perform classic computation tasks, such as graph traversal. However, such models are difficult to train, requiring long training times and large datasets. In this work, we achieve some of the computational capabilities of DNCs with a model that can be trained very efficiently, namely, an echo state network with an explicit memory without interference. This extension enables echo state networks to recognize all regular languages, including those that contractive echo state networks provably cannot recognize. Furthermore, we demonstrate experimentally that our model performs comparably to its fully trained deep version on several typical benchmark tasks for DNCs.

Deep learning for graphs encompasses all those neural models endowed with multiple layers of computation operating on data represented as graphs.The most common building blocks of these models are graph encoding layers, which compute a vector embedding for each node in a graph using message-passing operators.In this paper, we provide an overview of the key concepts in the field, point towards open questions, and frame the contributions of the ESANN 2021 special session into the broader context of deep learning for graphs.

Tree data occurs in many forms, such as computer programs, chemical molecules, or natural language. Unfortunately, the non-vectorial and discrete nature of trees makes it challenging to construct functions with tree-formed output, complicating tasks such as optimization or time series prediction. Autoencoders address this challenge by mapping trees to a vectorial latent space, where tasks are easier to solve, and then mapping the solution back to a tree structure. However, existing autoencoding approaches for tree data fail to take the specific grammatical structure of tree domains into account and rely on deep learning, thus requiring large training datasets and long training times. In this paper, we propose tree echo state autoencoders (TES-AE), which are guided by a tree grammar and can be trained within seconds by virtue of reservoir computing. In our evaluation on three datasets, we demonstrate that our proposed approach is not only much faster than a state-of-the-art deep learning autoencoding approach (D-VAE) but also has less autoencoding error if little data and time is given.

Metric learning has the aim to improve classification accuracy by learning a distance measure which brings data points from the same class closer together and pushes data points from different classes further apart. Recent research has demonstrated that metric learning approaches can also be applied to trees, such as molecular structures, abstract syntax trees of computer programs, or syntax trees of natural language, by learning the cost function of an edit distance, i.e. the costs of replacing, deleting, or inserting nodes in a tree. However, learning such costs directly may yield an edit distance which violates metric axioms, is challenging to interpret, and may not generalize well. In this contribution, we propose a novel metric learning approach for trees which we call embedding edit distance learning (BEDL) and which learns an edit distance indirectly by embedding the tree nodes as vectors, such that the Euclidean distance between those vectors supports class discrimination. We learn such embeddings by reducing the distance to prototypical trees from the same class and increasing the distance to prototypical trees from different classes. In our experiments, we show that BEDL improves upon the state-of-the-art in metric learning for trees on six benchmark data sets, ranging from computer science over biomedical data to a natural-language processing data set containing over 300,000 nodes.