Concept drift -the change of the distribution over timeposes significant challenges for learning systems and is of central interest for monitoring.Understanding drift is thus paramount, and drift localization -determining which samples are affected by the drift -is essential.While several approaches exist, most rely on local testing schemes, which tend to fail in high-dimensional, low-signal settings.In this work, we consider a fundamentally different approach based on conformal predictions.We discuss and show the shortcomings of common approaches and demonstrate the performance of our approach on state-of-the-art image datasets.

Algorithmic fairness, a key concern in algorithmic decision making, is a well-studied topic in the batch setup.Recently, several extensions for improving fairness in classification tasks have been proposed for the important scenario of non-stationary data streams.Yet, the question of how to reliably evaluate fairness for non-stationary data streams is still open, as popular batch measures can lead to misleading results.Specifically, typically cumulative fairness measures can be problematic if concept drift results in significant changes in model fairness across the data stream.In this contribution, we propose novel fairness scores that are suitable for the streaming scenario, and we demonstrate their suitability on streaming data benchmarks.

Recent knowledge graph embedding (KGE) models increasingly exploit algebraic structures to encode relational semantics.Cliffordbased models, in particular, offer strong expressiveness and geometric interpretability.In this work, we analyze the representations and decision boundaries of such models using an embedding-based reasoner as a classification function.To interpret Clifford-based geometric effects, we adapt DeepView, a visualization framework that approximates decision functions of deep classification models.This study provides one of the first systematic visual analyses of Clifford-based KGE models, helping bridge algebraic representation learning and interpretability.

Assessing the similarity between two images remains a core challenge in computer vision.Traditional full-reference image quality assessment (FR-IQA) metrics measure pixel-wise or low-level structural distortions and falter when human perception effortlessly recognizes equivalence.Perceptual metrics such as LPIPS and DISTS improve correlation with human judgments but remain opaque black boxes.We propose Conceptual Similarity (CSIM), a transparent and steerable image similarity metric that operates directly on human-interpretable semantic concepts.The resulting metric is simultaneously (1) a FR-IQA metric that is robust to non-semantic distortions while remaining sensitive to meaningful semantic changes, and (2) a general image-to-image similarity measure.Most importantly, CSIM offers transparency-users can inspect which concepts drive the score-and gives rise to a novel capability, which we call Human Similarity Steering that permits user-determined per-concept weighting to influence the similarity score according to their preferences.

Identifying how sensitive attributes like ethnicity are encoded in language models can yield valuable insights in terms of fairness.This knowledge could enhance explanations of model decisions, aid in mitigating social biases, or indicate under-represented minorities.Based on the literature on fairness and explainable AI, it should be possible to learn sensitive attributes such as gender or ethnicity with linear methods.Unfortunately, there are not many papers on the intersection of concept learning and fairness.On the other hand, too many fairness papers restrict their evaluation to binary gender and do not consider more complex test cases.So, it is not entirely clear whether all sensitive attributes and identity groups are encoded linearly in language models.Hence, we evaluate this question on a broad selection of identity groups, datasets, and language models.

Data-driven surrogate models provide fast and fully differentiable approximations of complex dynamical systems. In this work, we develop such surrogates for the Rayleigh–Bénard convection (RBC), which governs thermally driven flows in natural and industrial environments. Specifically, the proposed models approximate the discrete-time flow map of the RBC system, advancing the full system state by a fixed time step. We train Fourier Neural Operator (FNO)–based models to learn the dynamics of RBC in two and three dimensions and compare them to a convolutional U-Net baseline and a Koopman-based Linear Recurrent Autoencoder Network (LRAN). The two-dimensional system serves as a baseline for the more challenging three-dimensional case, which exhibits increased spatial complexity and turbulent dynamics. Across all settings, FNO-based models consistently outperform the LRAN, while achieving performance comparable to the U-Net in several regimes. Incorporating spatio-temporal inputs via FNOs leads to improved long-term prediction accuracy, particularly for turbulent flows. The physical fidelity of the predictions is assessed using convective heat flux statistics, profiles, and fluctuations, showing that FNOs most closely reproduce the ground-truth flow statistics. In addition, we demonstrate that FNOs enable zero-shot super-resolution across unseen spatial discretizations, a capability not shared by the convolutional baselines. These results highlight the potential of neural operator–based models as accurate, physically consistent, and resolution-independent surrogates for downstream tasks such as flow control.

Automatic label correction methods have been developed to address issues that come with a growing volume of annotated data in Machine Learning pipelines. The majority of these methods have been primarily evaluated on artificial noise in Euclidean data. Yet, particularly the subject of Geometric Deep Learning could benefit from well-performing label correction algorithms since non-Euclidean data, such as curved surfaces in 3D, is especially cumbersome and error prone to annotate. To assess the current applicability of existing label correction methods to 3D data, we evaluate automatic noise detection methods using realistic and artificial label noise in the 3D domain. We not only evaluate their effectiveness in 3D, but also examine general correction behavior regarding properties like carefulness and aggressiveness and compare those between both noise types. Our investigation shows that finding realistic noise in a 3D segmentation problem is of different nature compared to finding artificial noise.

Federated learning enables privacy-preserving machine learning across distributed data sources, but real-world deployments face challenges that extend beyond standard protocols. This tutorial provides a structured overview of the field, addressing issues such as non-stationary data, client heterogeneity, resource constraints, and security threats. Beyond existing surveys, it incorporates hands-on insights and deployment experiences, including concrete war stories illustrating how federated learning methods perform under real-world conditions. The tutorial also outlines emerging directions, including federated graph learning, game-theoretic approaches, and sustainable AI concepts. It aims to provide both a conceptual framework and practical guidance for researchers and practitioners advancing federated learning in dynamic and evolving environments.

The practical deployment of Federated Learning (FL) on resource-constrained devices is fundamentally limited by the high cost of training large models and the instability caused by heterogeneous (non-IID) client data. Conventional pruning methods often treat data heterogeneity as a problem to be mitigated. In this work, we introduce a paradigm shift: we reframe client diversity as a feature to be harnessed. We propose AutoFLIP, a framework that begins not with training, but with a one-time federated loss exploration. During this phase, clients collaboratively build a map of the collective loss landscape, using their diverse data to reveal the problem's essential structure. This shared intelligence then guides an adaptive pruning strategy that is dynamically refined by client agreement throughout training. This approach allows AutoFLIP to identify robust and efficient sub-networks from the outset. Our extensive experiments show that AutoFLIP reduces computational overhead by an average of 52% and communication costs by over 65% while simultaneously achieving state-of-the-art accuracy in challenging non-IID settings.

Feature-based explanations, using perturbations or gradients, are a prevalent tool to understand decisions of black box machine learning models. Yet, differences between these methods still remain mostly unknown, which limits their applicability for practitioners. In this work, we introduce a unified framework for local and global feature-based explanations using two well-established concepts: functional ANOVA (fANOVA) from statistics, and the notion of value and interaction from cooperative game theory. We introduce three fANOVA decompositions that determine the influence of feature distributions, and use game-theoretic measures, such as the Shapley value and interactions, to specify the influence of higher-order interactions. Our framework combines these two dimensions to uncover similarities and differences between a wide range of explanation techniques for features and groups of features. We then empirically showcase the usefulness of our framework on synthetic and real-world datasets.

Non-stationary environments subject to concept drift require the design of adaptive models that can continuously learn and update.Two primary research communities have emerged to address this challenge: Continual Learning (CL) and Streaming Machine Learning (SML).CL manages virtual drifts by learning new concepts without forgetting past knowledge, while SML focuses on real drifts, rapidly adapting to evolving data distributions.However, a unified approach is needed to balance adaptation and knowledge retention.Streaming Continual Learning (SCL) bridges the gap between CL and SML, ensuring models retain useful past information while efficiently adapting to new data.We explore key challenges in SCL, including handling temporal dependencies in data streams and adapting latent representations for personalization and knowledge editing.Additionally, we identify promising SCL benchmarks which can foster and promote a unified research effort between CL and SML. 35

Concept drift refers to the change of data distributions over time.While drift poses a challenge for learning models, requiring their continual adaption, it is also relevant in system monitoring to detect malfunctions, system failures, and unexpected behavior.In the latter case, the robust and reliable detection of drifts is imperative.This work studies the shortcomings of commonly used drift detection schemes.We show that they are prone to adversarial attacks, i.e., streams with undetected drift.In particular, we give necessary and sufficient conditions for their existence, provide methods for their construction, and demonstrate this behavior in experiments.

Joint-Embedding Predictive Architectures (JEPA) have recently become popular as promising architectures for self-supervised learning.Vision transformers have been trained using JEPA to produce embeddings from images and videos, which have been shown to be highly suitable for downstream tasks like classification and segmentation.In this paper, we show how to adapt the JEPA architecture to reinforcement learning from images.We discuss model collapse, show how to prevent it, and provide exemplary data on the classical Cart Pole task.* Acknowledgments.The authors were supported

Concept drift refers to the phenomenon that the underlying data distribution changes over time.While detection methods or model adjustment methods exist, a proper explanation of drift in high-dimensional settings is still widely unsolved.This problem is crucial since it enables an understanding of the most prominent drift characteristics.In this work, we propose to explain concept drift of high-dimensional data objects by means of concept activation vectors which give rise to local, phase, and a novel, global explanation called the Concept 2 Drift Distribution.

This paper examines how to improve interpretability of language models in the context of fairness.While traditional concept learning focuses on identifying the most important concepts for a task, this study explores how to locate the representation of sensitive attributes in pretrained language models.We address challenges such as the potential low importance and sparsity of sensitive attributes in training data, and the limited amount of labeled data for this purpose.Our experiments evaluate potential methods to obtain such identity concepts, considering factors like label sparsity, generalizability, and the influence of different language models on the representation of sensitive attributes.

We train Fourier Neural Operator (FNO) surrogate models for Rayleigh-Bénard Convection (RBC), a model for convection processes that occur in nature and industrial settings.We compare the prediction accuracy and model properties of FNO surrogates to two popular surrogates used in fluid dynamics: Dynamic Mode Decomposition (DMD) and the Linearly-Recurrent Autoencoder Network (LRAN).We regard Direct Numerical Simulations (DNS) of the RBC equations as the ground truth on which the models are trained and evaluated in different settings.The FNO performs favorably when compared to the DMD and LRAN and its predictions are fast and highly accurate for this task.Additionally, we show its zero-shot super-resolution ability for the convection dynamics.The FNO model has a high potential to be used in downstream tasks such as flow control in RBC.

Catastrophic forgetting is a key challenge in continual learning.In the adjoining field of stream machine learning, few methods target the related problem of re-occurring drift by avoiding forgetting old data.In this work, we investigate whether we can transfer such strategies from the stream machine learning to the continual learning setup.Based on our consideration, we propose a simple yet efficient compression-based kNN scheme and evaluate it experimentally.

In unsupervised machine learning, Isolation Forest (IsoForest) is a widely used algorithm for the efficient detection of outliers.Identifying the features responsible for observed anomalies is crucial for practitioners, yet the ensemble nature of IsoForest complicates interpretation and comparison.As a remedy, SHAP is a prevalent method to interpret outlier scoring models by assigning contributions to individual features based on the Shapley Value (SV).However, complex anomalies typically involve interaction of features, and it is paramount for practitioners to distinguish such complex anomalies from simple cases.In this work, we propose Shapley Interactions (SIs) to enrich explanations of outliers with feature interactions.SIs, as an extension of the SV, decompose the outlier score into contributions of individual features and interactions of features up to a specified explanation order.We modify IsoForest to compute SI using TreeSHAP-IQ, an extension of TreeSHAP for tree-based models, using the shapiq package.Using a qualitative and quantitative analysis on synthetic and real-world datasets, we demonstrate the benefit of SI and feature interactions for outlier explanations over feature contributions alone.

Albeit the ubiquitous use of Graph Neural Networks (GNNs) in machine learning (ML) prediction tasks involving graph-structured data, their interpretability remains challenging. In explainable artificial intelligence (XAI), the Shapley Value (SV) is the predominant method to quantify contributions of individual features to a ML model’s output. Addressing the limitations of SVs in complex prediction models, Shapley Interactions (SIs) extend the SV to groups of features. In this work, we explain single graph predictions of GNNs with SIs that quantify node contributions and interactions among multiple nodes. By exploiting the GNN architecture, we show that the structure of interactions in node embeddings are preserved for graph prediction. As a result, the exponential complexity of SIs depends only on the receptive fields, i.e. the message-passing ranges determined by the connectivity of the graph and the number of convolutional layers. Based on our theoretical results, we introduce GraphSHAP-IQ, an efficient approach to compute any-order SIs exactly. GraphSHAP-IQ is applicable to popular message passing techniques in conjunction with a linear global pooling and output layer. We showcase that GraphSHAP-IQ substantially reduces the exponential complexity of computing exact SIs on multiple benchmark datasets. Beyond exact computation, we evaluate GraphSHAP-IQ’s approximation of SIs on popular GNN architectures and compare with existing baselines. Lastly, we visualize SIs of real-world water distribution networks and molecule structures using a SI-Graph.

Water distribution systems (WDSs) are an important part of critical infrastructure becoming increasingly significant in the face of climate change and urban population growth. We propose a robust and scalable surrogate deep learning (DL) model to enable efficient planning, expansion, and rehabilitation of WDSs. Our approach incorporates an improved graph neural network architecture, an adapted physics-informed algorithm, an innovative training scheme, and a physics-preserving data normalization method. Evaluation results on a number of WDSs demonstrate that our model outperforms the current state-of-the-art DL model. Moreover, our method allows us to scale the model to bigger and more realistic WDSs. Furthermore, our approach makes the model more robust to out-of-distribution input features (demands, pipe diameters). Hence, our proposed method constitutes a significant step towards bridging the simulation-to-real gap in the use of artificial intelligence for WDSs.

With the increasing availability of data collected by edge devices over time, efficient algorithms running remotely on low-energy devices such as FPGAs are required. This includes Machine Learning algorithms, which constitute a valuable tool when analyzing and processing vast amounts of data. To keep accurate models under distributional changes, commonly referred to as concept drift, adaptive online learning models are required. While first works proposed FPGA implementations of several machine learning algorithms, in this work, we will focus on online learning using the neighbor-based SAM-kNN model, which showed good performance under heterogenous drifts. We propose an efficient FPGA implementation that yields considerable speed and energy efficiency advantages while keeping a competitive accuracy over a range of artificial and real-world benchmarks.The implementation code is available on GitHub at https://github.com/jvaquet/SAMkNN-on-FPGA.

Wearable sensors that measure signals from the human body are increasingly used for rehabilitation, healthcare, sports, and fitness. Since it reflects human’s individual constitution and geometry, a personalized design often leads to a significantly better accuracy. Personalization can take place on the hardware level, but usually at the cost of scalability of the overarching product. Therefore, in this contribution, we focus on a personalization on the software side. This leads to the question how to optimize the arrangement of the basic sensors to simultaneously achieve best results for all users, allowing adaptability on the software side. In this contribution, we focus on the challenge of solving this task robustly, given the fact that sensors are usually subject to small variations when positioned on the human body. We propose a novel formalization, which phrases this task as a feature selection task for an adversarial loss, which takes into account both physically plausible disturbances and individualized sensor processing. We compare different approaches to compute such attacks efficiently, and we propose a closed form objective based on which robust feature selection becomes possible. We demonstrate the efficiency of the approach for designing robust wearable sensor devices based on the example of pressure sensors in a shoe insole.

Maximilian Spliethöver, Tim Knebler, Fabian Fumagalli, Maximilian Muschalik, Barbara Hammer, Eyke Hüllermeier, Henning Wachsmuth. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Language-image pre-training (LIP) enables the development of vision-language models capable of zero-shot classification, localization, multimodal retrieval, and semantic understanding. Various explanation methods have been proposed to visualize the importance of input image-text pairs on the model's similarity outputs. However, popular saliency maps are limited by capturing only first-order attributions, overlooking the complex cross-modal interactions intrinsic to such encoders. We introduce faithful interaction explanations of LIP models (FIxLIP) as a unified approach to decomposing the similarity in vision-language encoders. FIxLIP is rooted in game theory, where we analyze how using the weighted Banzhaf interaction index offers greater flexibility and improves computational efficiency over the Shapley interaction quantification framework. From a practical perspective, we propose how to naturally extend explanation evaluation metrics, such as the pointing game and area between the insertion/deletion curves, to second-order interaction explanations. Experiments on the MS COCO and ImageNet-1k benchmarks validate that second-order methods, such as FIxLIP, outperform first-order attribution methods. Beyond delivering high-quality explanations, we demonstrate the utility of FIxLIP in comparing different models, e.g. CLIP vs. SigLIP-2.

The rapid advancement of generative AI enables highly realistic synthetic video, posing significant challenges for content authentication and raising urgent concerns about misuse. Existing detection methods often struggle with generalization and capturing subtle temporal inconsistencies. We propose $ReStraV$ ($Re$presentation $Stra$ightening for $V$ideo), a novel approach to distinguish natural from AI-generated videos. Inspired by the ``perceptual straightening'' hypothesis—which suggests real-world video trajectories become more straight in neural representation domain—we analyze deviations from this expected geometric property. Using a pre-trained self-supervised vision transformer (DINOv2), we quantify the temporal curvature and stepwise distance in the model's representation domain. We aggregate statistical and signals descriptors of these measures for each video and train a classifier. Our analysis shows that AI-generated videos exhibit significantly different curvature and distance patterns compared to real videos. A lightweight classifier achieves state-of-the-art detection performance (e.g., $97.17$ % accuracy and $98.63$ % AUROC on the VidProM benchmark, substantially outperforming existing image- and video-based methods. ReStraV is computationally efficient, it is offering a low-cost and effective detection solution. This work provides new insights into using neural representation geometry for AI-generated video detection.

SUMMARY: In this paper, we introduce the first diffusion model designed to generate complete synthetic human genotypes, which, by standard protocols, one can straightforwardly expand into full-length, DNA-level genomes. The synthetic genotypes mimic real human genotypes without just reproducing known genotypes, in terms of approved metrics. When training biomedically relevant classifiers with synthetic genotypes, accuracy is near-identical to the accuracy achieved when training classifiers with real data. We further demonstrate that augmenting small amounts of real with synthetically generated genotypes drastically improves performance rates. This addresses a significant challenge in translational human genetics: real human genotypes, although emerging in large volumes from genome wide association studies, are sensitive private data, which limits their public availability. Therefore, the integration of additional, insensitive data when striving for rapid sharing of biomedical knowledge of public interest appears imperative. AVAILABILITY AND IMPLEMENTATION: All non proprietary data and the code to replicate the experiments is available on Github.

Water distribution systems (WDS) are an integral part of critical infrastructure which is pivotal to urban development. As 70% of the world's population will likely live in urban environments in 2050, efficient simulation and planning tools for WDS play a crucial role in reaching UN's sustainable developmental goal (SDG) 6 - "Clean water and sanitation for all". In this realm, we propose a novel and efficient machine learning emulator, more precisely, a physics-informed deep learning (DL) model, for hydraulic state estimation in WDS. Using a recursive approach, our model only needs a few graph convolutional neural network (GCN) layers and employs an innovative algorithm based on message passing. Unlike conventional machine learning tasks, the model uses hydraulic principles to infer two additional hydraulic state features in the process of reconstructing the available ground truth feature in an unsupervised manner. To the best of our knowledge, this is the first DL approach to emulate the popular hydraulic simulator EPANET, utilizing no additional information. Like most DL models and unlike the hydraulic simulator, our model demonstrates vastly faster emulation times that do not increase drastically with the size of the WDS. Moreover, we achieve high accuracy on the ground truth and very similar results compared to the hydraulic simulator as demonstrated through experiments on five real-world WDS datasets.

While shallow decision trees may be interpretable, larger ensemble models like gradient-boosted trees, which often set the state of the art in machine learning problems involving tabular data, still remain black box models. As a remedy, the Shapley value (SV) is a well-known concept in explainable artificial intelligence (XAI) research for quantifying additive feature attributions of predictions. The model-specific TreeSHAP methodology solves the exponential complexity for retrieving exact SVs from tree-based models. Expanding beyond individual feature attribution, Shapley interactions reveal the impact of intricate feature interactions of any order. In this work, we present TreeSHAP-IQ, an efficient method to compute any-order additive Shapley interactions for predictions of tree-based models. TreeSHAP-IQ is supported by a mathematical framework that exploits polynomial arithmetic to compute the interaction scores in a single recursive traversal of the tree, akin to Linear TreeSHAP. We apply TreeSHAP-IQ on state-of-the-art tree ensembles and explore interactions on well-established benchmark datasets.

Addressing the limitations of individual attribution scores via the Shapley value (SV), the field of explainable AI (XAI) has recently explored intricate interactions of features or data points. In particular, extensions of the SV, such as the Shapley Interaction Index (SII), have been proposed as a measure to still benefit from the axiomatic basis of the SV. However, similar to the SV, their exact computation remains computationally prohibitive. Hence, we propose with SVARM-IQ a sampling-based approach to efficiently approximate Shapley-based interaction indices of any order. SVARM-IQ can be applied to a broad class of interaction indices, including the SII, by leveraging a novel stratified representation. We provide non-asymptotic theoretical guarantees on its approximation quality and empirically demonstrate that SVARM-IQ achieves state-of-the-art estimation results in practical XAI scenarios on different model classes and application domains.

As artificial intelligence (AI) systems become increasingly integrated into everyday life, the need for trustworthiness in these systems has emerged as a critical challenge.This tutorial paper addresses the complexity of building trust in AI systems by exploring recent advances in explainable AI (XAI) and related areas that go beyond mere interpretability.After reviewing recent trends in XAI, we discuss how to control AI systems, align them with societal concerns, and address the robustness, reproducibility, and evaluation concerns inherent in these systems.This review highlights the multifaceted nature of the mechanisms for building trust in AI, and we hope it will pave the way for further research in this area.1 https://digital-strategy.

Prototype-based methods constitute a robust and transparent family of machine-learning models.To increase robustness in real-world applications, they are frequently coupled with reject options.While the state-of-the-art method, relative similarity, couples the rejection of samples with high aleatoric and epistemic uncertainty, the technique lacks transparency, i.e., an explanation of why a sample has been rejected.In this work, we analyze the relative similarity analytically and derive an explanation scheme for reject options in prototype-based classification.

Feature selection is one of the most relevant preprocessing and analysis techniques in machine learning, allowing for increases in model performance and knowledge discovery.In online setups, both can be affected by concept drift, i.e., changes of the underlying distribution.Recently, an adaption of classical feature relevance approaches to drift detection was introduced.While the method increases detection performance significantly, there is only little discussion on the explanatory aspects.In this work, we focus on understanding the structure of the ongoing drift by transferring the concept of strongly and weakly relevant features to it.We empirically evaluate our methodology using graphical models.

In the domain of network intrusion detection, robustness against contaminated and noisy data inputs remains a critical challenge.This study introduces a probabilistic version of the Temporal Graph Network Support Vector Data Description (TGN-SVDD) model, designed to enhance detection accuracy in the presence of input noise.By predicting parameters of a Gaussian distribution for each network event, our model is able to naturally address noisy adversarials and improve robustness compared to a baseline model.Our experiments on a modified CIC-IDS2017 data set with synthetic noise demonstrate significant improvements in detection performance compared to the baseline TGN-SVDD model, especially as noise levels increase.

While 3D data is rich in information, it often comes with the drawback of being tedious to handle.Recent work in the Geometric Deep Learning community focused on developing high quality 3D datasets for tasks like mesh segmentation.However, the label quality can never be assured to be perfect.To improve label quality in 3D datasets, we propose an interactive algorithm combining DeepView, a method to visualize the classification function of neural networks, with Intrinsic Mesh CNNs, which generalize the convolution to Riemannian manifolds, to smartly select adequate sets of vertices from triangle mesh data for label correction.

This tutorial provides an overview of machine learning methodologies applied in distributed, federated, and non-stationary environments.We focus on recent advancements and novel research contributions of the field.Key topics include data analysis and pattern recognition for non-stationary environments, model compression, federated learning algorithms, and privacy preservation.This tutorial aims to equip researchers and practitioners with insights into current challenges and innovative solutions in this dynamic field. 47

Supervised online learning relies on the assumption that ground truth information is available for model updates at each time step.As this is not realistic in every setting, alternatives such as active online learning, or online learning with verification latency have been proposed.In this work, we assume that no label information is available after intitial training.We argue that provided we can characterize the expected concept drift as incremental drift, we can rely on a self-labeling strategy to keep updated models.We derive a k-NN-based self-labeling online learner implementing the presented self-supervised scheme and experimentally show that this is an option for learning from incrementally drifting data streams in the absence of label information.

Biosensors measure signals from the human body, and usually process them with a small ML model on simple hardware.When a new person starts using such a device, a domain adaptation problem arises.We consider the case where no labels are known for the new person, but data (including labels) from several other people are available (unsupervised, multi-source).As an application scenario, we look at a shoe insole with 3-8 pressure sensors that estimates how much weight/force is put on the foot (regression problem).We propose a distance measure between a source and target domain, and a combination of all source models.Experiments on real world data from 13 persons show that our method outperforms all other tested methods by a good margin.

The Shapley value (SV) is a prevalent approach of allocating credit to machine learning (ML) entities to understand black box ML models. Enriching such interpretations with higher-order interactions is inevitable for complex systems, where the Shapley Interaction Index (SII) is a direct axiomatic extension of the SV. While it is well-known that the SV yields an optimal approximation of any game via a weighted least square (WLS) objective, an extension of this result to SII has been a long-standing open problem, which even led to the proposal of an alternative index. In this work, we characterize higher-order SII as a solution to a WLS problem, which constructs an optimal approximation via SII and k-Shapley values (k-SII). We prove this representation for the SV and pairwise SII and give empirically validated conjectures for higher orders. As a result, we propose KernelSHAP-IQ, a direct extension of KernelSHAP for SII, and demonstrate state-of-the-art performance for feature interactions.

Plenty of works have brought social biases in language models to attention and proposed methods to detect such biases. As a result, the literature contains a great deal of different bias tests and scores, each introduced with the premise to uncover yet more biases that other scores fail to detect. What severely lacks in the literature, however, are comparative studies that analyse such bias scores and help researchers to understand the benefits or limitations of the existing methods. In this work, we aim to close this gap for cosine based bias scores. By building on a geometric definition of bias, we propose requirements for bias scores to be considered meaningful for quantifying biases. Furthermore, we formally analyze cosine based scores from the literature with regard to these requirements. We underline these findings with experiments to show that the bias scores' limitations have an impact in the application case.

Federated learning (FL) has emerged as a ground-breaking paradigm in machine learning (ML), offering privacy-preserving collaborative model training across diverse datasets. Despite its promise, FL faces significant hurdles in non-identically and independently distributed (non-IID) data scenarios, where most existing methods often struggle with data heterogeneity and lack robustness in performance. This paper introduces Federated Loss Exploration (FedLEx), an innovative approach specifically designed to tackle these challenges. FedLEx distinctively addresses the shortcomings of existing FL methods in non-IID settings by optimizing its learning behavior for scenarios in which assumptions about data heterogeneity are impractical or unknown. It employs a federated loss exploration technique, where clients contribute to a global guidance matrix by calculating gradient deviations for model parameters. This matrix serves as a strategic compass to guide clients’ gradient updates in subsequent FL rounds, thereby fostering optimal parameter updates for the global model. FedLEx effectively navigates the complex loss surfaces inherent in non-IID data, enhancing knowledge transfer in an efficient manner, since only a small number of epochs and small amount of data are required to build a strong global guidance matrix that can achieve model convergence without the need for additional data sharing or data distribution statics in a large client scenario. Our extensive experiments with state-of-the art FL algorithms demonstrate significant improvements in performance, particularly under realistic non-IID conditions, thus highlighting FedLEx’s potential to overcome critical barriers in diverse FL applications.

In recent studies, line search methods have been demonstrated to significantly enhance the performance of conventional stochastic gradient descent techniques across various datasets and architectures, while making an otherwise critical choice of learning rate schedule superfluous. In this paper, we identify problems of current state-of-the-art of line search methods, propose enhancements, and rigorously assess their effectiveness. Furthermore, we evaluate these methods on orders of magnitude larger datasets and more complex data domains than previously done.More specifically, we enhance the Armijo line search method by speeding up its computation and incorporating a momentum term into the Armijo criterion, making it better suited for stochastic mini-batching. Our optimization approach outperforms both the previous Armijo implementation and a tuned learning rate schedule for the Adam and SGD optimizers. Our evaluation covers a diverse range of architectures, such as Transformers, CNNs, and MLPs, as well as data domains, including NLP and image data.Our work is publicly available as a Python package, which provides a simple Pytorch optimizer.

Several related works have introduced Koopman-based Machine Learning architectures as a surrogate model for dynamical systems. These architectures aim to learn non-linear measurements (also known as observables) of the system’s state that evolve by a linear operator and are, therefore, amenable to model-based linear control techniques. So far, mainly simple systems have been targeted, and Koopman architectures as reduced-order models for more complex dynamics have not been fully explored. Hence, we use a Koopman-inspired architecture called the Linear Recurrent Autoencoder Network (LRAN) for learning reduced-order dynamics in convection flows of a Rayleigh Bénard Convection (RBC) system at different amounts of turbulence. The data is obtained from direct numerical simulations of the RBC system. A traditional fluid dynamics method, the Kernel Dynamic Mode Decomposition (KDMD), is used to compare the LRAN. For both methods, we performed hyperparameter sweeps to identify optimal settings. We used a Normalized Sum of Square Error measure for the quantitative evaluation of the models, and we also studied the model predictions qualitatively. We obtained more accurate predictions with the LRAN than with KDMD in the most turbulent setting. We conjecture that this is due to the LRAN’s flexibility in learning complicated observables from data, thereby serving as a viable surrogate model for the main structure of fluid dynamics in turbulent convection settings. In contrast, KDMD was more effective in lower turbulence settings due to the repetitiveness of the convection flow. The feasibility of Koopman-based surrogate models for turbulent fluid flows opens possibilities for efficient model-based control techniques useful in a variety of industrial settings.

With the enourmous popularity of large language models, many researchers have raised ethical concerns regarding social biases incorporated in such models. Several methods to measure social bias have been introduced, but apparently these methods do not necessarily agree regarding the presence or severity of bias. Furthermore, some works have shown theoretical issues or severe limitations with certain bias measures.For that reason, we introduce SAME, a novel bias score for semantic bias in embeddings. We conduct a thorough theoretical analysis as well as experiments to show its benefits compared to similar bias scores from the literature. We further highlight a substantial relation of semantic bias measured by SAME with downstream bias, a connection that has recently been argued to be negligible. Instead, we show that SAME is capable of measuring semantic bias and identify potential causes for social bias in downstream tasks.

Facing climate change, the already limited availability of drinking water will decrease in the future, rendering drinking water an increasingly scarce resource. Considerable amounts of it are lost through leakages in water transportation and distribution networks. Thus, anomaly detection and localization, in particular for leakages, are crucial but challenging tasks due to the complex interactions and changing demands in water distribution networks. In this work, we conceptually analyze the effects of anomalies on the dynamics of critical infrastructure systems by modeling them with Bayesian networks. We then discuss how the problem is connected to and can be considered through the lens of concept drift. This analysis yields our proposal to leverage model-based drift explanations as a tool for localizing anomalies given limited information about the network. The methodology is experimentally evaluated using realistic benchmark scenarios. To showcase that our methodology applies to critical infrastructure more generally, in addition to considering leakages and sensor faults in water systems, we investigate the suitability of the derived technique to localize sensor faults in power systems.

In the realm of fashion object detection and segmentation for online shopping images, existing state-of-the-art fashion parsing models encounter limitations, particularly when exposed to non-model-worn apparel and close-up shots. To address these failures, we introduce FashionFail; a new fashion dataset with e-commerce images for object detection and segmentation. The dataset is efficiently curated using our novel annotation tool that leverages recent foundation models. The primary objective of FashionFail is to serve as a test bed for evaluating the robustness of models. Our analysis reveals the shortcomings of leading models, such as Attribute-Mask R-CNN and Fashionformer. Additionally, we propose a baseline approach using naive data augmentation to mitigate common failure cases and improve model robustness. Through this work, we aim to inspire and support further research in fashion item detection and segmentation for industrial applications. The dataset, annotation tool, code, and models are available at https://rizavelioglu.github.io/fashionfail/.

Originally rooted in game theory, the Shapley Value (SV) has recently become an important tool in machine learning research. Perhaps most notably, it is used for feature attribution and data valuation in explainable artificial intelligence. Shapley Interactions (SIs) naturally extend the SV and address its limitations by assigning joint contributions to groups of entities, which enhance understanding of black box machine learning models. Due to the exponential complexity of computing SVs and SIs, various methods have been proposed that exploit structural assumptions or yield probabilistic estimates given limited resources. In this work, we introduce shapiq, an open-source Python package that unifies state-of-the-art algorithms to efficiently compute SVs and any-order SIs in an application-agnostic framework. Moreover, it includes a benchmarking suite containing 11 machine learning applications of SIs with pre-computed games and ground-truth values to systematically assess computational performance across domains. For practitioners, shapiq is able to explain and visualize any-order feature interactions in predictions of models, including vision transformers, language models, as well as XGBoost and LightGBM with TreeSHAP-IQ. With shapiq, we extend shap beyond feature attributions and consolidate the application of SVs and SIs in machine learning that facilitates future research. The source code and documentation are available at https://github.com/mmschlk/shapiq.

Feature selection is one of the most relevant preprocessing and analysis techniques in machine learning. It can dramatically increase the performance of learning algorithms and at the same time provide relevant information on the data. In the scenario of online and stream learning, concept drift, i.e., changes of the underlying distribution over time, can cause significant problems for learning models and data analysis. While there do exist feature selection methods for online learning, none of the methods targets feature selection for drift detection, i.e., the challenge to increase the performance of drift detectors by analyzing the drift rather than increasing model accuracy. However, this challenge is particularly relevant for common unsupervised scenarios. In this work, we study feature selection for drift detection and drift monitoring. We develop a formal definition for a feature-wise notion of drift that allows semantic interpretation. Besides, we derive an efficient algorithm by reducing the problem to classical feature selection and analyze the applicability of our approach to feature selection for drift detection on a theoretical level. Finally, we empirically show the relevance of our considerations on several benchmarks.

Removal-based explanations are a general framework to provide feature importance scores, where feature removal, i.e. restricting a model on a subset of features, is a central component.While many machine learning applications require dynamic modeling environments, where distributions and models change over time, removal-based explanations and feature removal have mainly been considered in a static batch learning environment.Recently, an interventional and observational perturbation method was presented that allows to remove features efficiently in dynamic learning environments with concept drift.In this paper, we compare these two algorithms on two synthetic data streams.We showcase how both yield substantially different explanations when features are correlated and provide guidance on the choice based on the application.

Feature selection is one of the most relevant preprocessing and analysis techniques in machine learning.It can dramatically increase the performance of learning algorithms and also provide relevant information on the data.In online and stream learning concept drift, i.e., the change of the underlying distribution over time, can cause tremendous problems for learning models and data analysis.While there do exist feature selection methods for online learning, to the best of our knowledge there do not exist methods to perform feature selection for drift detection, i.e., to increase the performance of drift detectors and to analyze the drift itself.In this work, we study feature selection for concept drift detection and provide a formal derivation and semantic interpretation thereof.We empirically show the relevance of our considerations on several benchmarks. 53

Hyperspectral imaging is a suitable measurement tool across domains.However, when combined with machine learning techniques, frequently intensity and transversal shifts hinder the transfer between different sensors and settings.Established approaches focus on eliminating sensor shifts in the data or recalibrating sensors.In this contribution, we target the training procedure, propose robust training, and derive a robust feature selection strategy that can cope with multiple shift dynamics at the same time.We evaluate our approaches experimentally on artificial and real-world datasets.

Hinder F, Vaquet V, Brinkrolf J, Hammer B. On the Hardness and Necessity of Supervised Concept Drift Detection. In: De Marsico M, Sanniti di Baja G, Fred A, eds. Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods ICPRAM. Vol. 1. Setúbal: SCITEPRESS - Science and Technology Publications; 2023: 164-175.

Kenneweg P, Schroeder S, Schulz A, Hammer B. Debiasing Sentence Embedders Through Contrastive Word Pairs. In: Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods. Setúbal, Portugal: SCITEPRESS - Science and Technology Publications; 2023: 205-212.

Recent works have shown that line search methods greatly increase performance of traditional stochastic gradient descent methods on a variety of datasets and architectures [1], [2]. In this work we succeed in extending line search methods to the novel and highly popular Transformer architecture and dataset domains in natural language processing. More specifically, we combine the Armijo line search with the Adam optimizer and extend it by subdividing the networks architecture into sensible units and perform the line search separately on these local units. Our optimization method outperforms the traditional Adam optimizer and achieves significant performance improvements for small data sets or small training budgets, while performing equal or better for other tested cases. Our work is publicly available as a python package, which provides a hyperparameter-free pytorch optimizer that is compatible with arbitrary network architectures.

Predominately in explainable artificial intelligence (XAI) research, the Shapley value (SV) is applied to determine feature attributions for any black box model. Shapley interaction indices extend the SV to define any-order feature interactions. Defining a unique Shapley interaction index is an open research question and, so far, three definitions have been proposed, which differ by their choice of axioms. Moreover, each definition requires a specific approximation technique. Here, we propose SHAPley Interaction Quantification (SHAP-IQ), an efficient sampling-based approximator to compute Shapley interactions for arbitrary cardinal interaction indices (CII), i.e. interaction indices that satisfy the linearity, symmetry and dummy axiom. SHAP-IQ is based on a novel representation and, in contrast to existing methods, we provide theoretical guarantees for its approximation quality, as well as estimates for the variance of the point estimates. For the special case of SV, our approach reveals a novel representation of the SV and corresponds to Unbiased KernelSHAP with a greatly simplified calculation. We illustrate the computational efficiency and effectiveness by explaining language, image classification and high-dimensional synthetic models.

Machine learning based decision making systems in safety critical areas place high demands on the accuracy and generalization ability of the underlying model. A common strategy to deal with uncertainties and possible mistakes is offered by learning with reject option, i.e. a model can refrain from prediction in ambiguous cases and leave the decision to a human expert. Yet, as for the models themselves, human decision-making is hampered by the fact that reject options are often implemented as black-box rules: Experts cannot readily understand the reasons for rejection. In this work, we propose a model-agnostic framework that enriches classification with reject option by explanation mechanisms. More specifically, we combine conformal prediction as a popular mathematically based technology of certainty estimation with local surrogates derived for the region of interest. This allows us to provide local explanations in terms of example-based explanation methods, including counterfactual, semi-factual, and factual methods. We demonstrate the performance of this technology through a series of benchmarks using 6 different data sets; the associated code is open source.1

Concept drift refers to the phenomenon that the distribution generating the observed data changes over time. If drift is present, machine learning models can become inaccurate and need adjustment. While there do exist methods to detect concept drift or to adjust models in the presence of observed drift, the question of explaining drift, i.e., describing the potentially complex and high dimensional change of distribution in a human-understandable fashion, has hardly been considered so far. This problem is of importance since it enables an inspection of the most prominent characteristics of how and where drift manifests itself. Hence, it enables human understanding of the change and it increases acceptance of life-long learning models. In this paper, we present a novel technology characterizing concept drift in terms of the characteristic change of spatial features based on various explanation techniques. To do so, we propose a methodology to reduce the explanation of concept drift to an explanation of models that are trained in a suitable way to extract relevant information regarding the drift. This way, a large variety of explanation schemes is available. Thus, a suitable method can be selected for the problem of drift explanation at hand. We outline the potential of this approach and demonstrate its usefulness in several examples.

Abstract Explainable artificial intelligence has mainly focused on static learning scenarios so far. We are interested in dynamic scenarios where data is sampled progressively, and learning is done in an incremental rather than a batch mode. We seek efficient incremental algorithms for computing feature importance (FI). Permutation feature importance (PFI) is a well-established model-agnostic measure to obtain global FI based on feature marginalization of absent features. We propose an efficient, model-agnostic algorithm called iPFI to estimate this measure incrementally and under dynamic modeling conditions including concept drift. We prove theoretical guarantees on the approximation quality in terms of expectation and variance. To validate our theoretical findings and the efficacy of our approaches in incremental scenarios dealing with streaming data rather than traditional batch settings, we conduct multiple experimental studies on benchmark data with and without concept drift.

Transfer learning schemes based on deep networks which have been trained on huge image corpora offer state-of-the-art technologies in computer vision. Here, supervised and semi-supervised approaches constitute efficient technologies which work well with comparably small data sets. Yet, such applications are currently restricted to application domains where suitable deep network models are readily available. In this contribution, we address an important application area in the domain of biotechnology, the automatic analysis of CHO-K1 suspension growth in microfluidic single-cell cultivation, where data characteristics are very dissimilar to existing domains and trained deep networks cannot easily be adapted by classical transfer learning. We propose a novel transfer learning scheme which expands a recently introduced Twin-VAE architecture, which is trained on realistic and synthetic data, and we modify its specialized training procedure to the transfer learning domain. In the specific domain, often only few to no labels exist and annotations are costly. We investigate a novel transfer learning strategy, which incorporates a simultaneous retraining on natural and synthetic data using an invariant shared representation as well as suitable target variables, while it learns to handle unseen data from a different microscopy technology. We show the superiority of the variation of our Twin-VAE architecture over the state-of-the-art transfer learning methodology in image processing as well as classical image processing technologies, which persists, even with strongly shortened training times and leads to satisfactory results in this domain. The source code is available at https://github.com/dstallmann/transfer_learning_twinvae, works cross-platform, is open-source and free (MIT licensed) software. We make the data sets available at https://pub.uni-bielefeld.de/record/2960030.

Abstract Many of today’s decision making systems deployed in the real world are not static—they are changing and adapting over time, a phenomenon known as model adaptation takes place. Because of their wide reaching influence and potentially serious consequences, the need for transparency and interpretability of AI-based decision making systems is widely accepted and thus have been worked on extensively—e.g. a very prominent class of explanations are contrasting explanations which try to mimic human explanations. However, usually, explanation methods assume a static system that has to be explained. Explaining non-static systems is still an open research question, which poses the challenge how to explain model differences, adaptations and changes. In this contribution, we propose and (empirically) evaluate a general framework for explaining model adaptations and differences by contrasting explanations. We also propose a method for automatically finding regions in data space that are affected by a given model adaptation—i.e. regions where the internal reasoning of the other (e.g. adapted) model changed—and thus should be explained. Finally, we also propose a regularization for model adaptations to ensure that the internal reasoning of the adapted model does not change in an unwanted way.

The application of machine learning based decision making systems in safety critical areas requires reliable high certainty predictions.Reject options are a common way of ensuring a sufficiently high certainty of predictions.While being able to reject uncertain samples is important, it is also of importance to be able to explain why a particular sample was rejected.However, explaining reject options is still an open problem.We propose a model-agnostic method for locally explaining reject options by means of interpretable models and counterfactual explanations.

Federated learning is an efficient methodology to reduce the data transmissions to the server when working with large amounts of (sen- sor) data from diverse physical locations. When using data from different sensor devices concept drift between the single sensors poses an additional challenge. In this contribution we define a formal framework for federated learning with concept drift and propose a version of federated LVQ dealing with concept drift induced by different hyperspectral cameras. We evalu- ate this approach experimentally and demonstrate its robustness to class imbalance and missing classes.

Pre training of language models on large text corpora is common practice in Natural Language Processing.Following, fine tuning of these models is performed to achieve the best results on a variety of tasks.In this paper we question the common practice of only adding a single output layer as a classification head on top of the network.We perform an AutoML search to find architectures that outperform the current single layer at only a small compute cost.We validate our classification architecture on a variety of NLP benchmarks from the GLUE dataset.The source code is open-source and free (MIT licensed) software, available at https://github.com/TheMody/NASforSentenceEmbeddingHeads. * We gratefully acknowledge

Explanation methods are considered the most prominent way of achieving the ubiquitous requirement of transparency.Ideally, in order to be useful, explanations should be "easy to understand" -i.e.being of low complexity.In this work, we empirically study explanations generated by Zorro, an explanation method for Graph Neural Networks.In the context of a standard reinforcement learning scenario, we propose a methodology to improve the quality of generated explanations in case of sparse observations.

High-resolution hyperspectral sensors provide precise but expensive information on an object's chemical composition in various industries.We present a method for transferring this capability to customized low-cost multispectral solutions.Taking a relevance analysis of spectra for a given problem as our starting point, we simulated and designed a multispectral sensor based on inverse spectroscopy.The corresponding calibration model, which was derived from the simulation of such a multispectral sensor and connected with its hardware, may not drop in precision significantly.Different methods of calibration model transfer capable of handling a limited subset of the data were tested for this purpose.The latent space transformation with Chebyshev polynomials outperformed all other methods by yielding the fewest labeled data.

Even though deep neural networks succeed on many different tasks including semantic segmentation, they lack on robustness against adversarial examples. To counteract this exploit, often adversarial training is used. However, it is known that adversarial training with weak adversarial attacks (e.g. using the Fast Gradient Method) does not improve the robustness against stronger attacks. Recent research shows that it is possible to increase the robustness of such single-step methods by choosing an appropriate step size during the training. Finding such a step size, without increasing the computational effort of single-step adversarial training, is still an open challenge. In this work we address the computationally particularly demanding task of semantic segmentation and propose a new step size control algorithm that increases the robustness of single-step adversarial training. The proposed algorithm does not increase the computational effort of single-step adversarial training considerably and also simplifies training, because it is free of meta-parameter. We show that the robustness of our approach can compete with multi-step adversarial training on two popular benchmarks for semantic segmentation.

Accurate traffic prediction is a key ingredient to enable traffic management like rerouting cars to reduce road congestion or regulating traffic via dynamic speed limits to maintain a steady flow. One way to represent traffic data is as temporally changing heatmaps visualizing attributes of traffic, such as speed and volume. In recent approaches, U-Net models have shown state of the art performance on traffic forecasting from such heatmaps. We propose to combine the U-Net architecture with graph layers which improves spatial generalization to unseen road networks compared to a Vanilla U-Net. In particular, we specialize existing graph operations to be sensitive to geographical topology and generalize pooling and upsampling operations to be applicable to graphs.

The notion of concept drift refers to the phenomenon that the distribution which is underlying the observed data changes over time. As a consequence machine learning models may become inaccurate and need adjustment. While there do exist methods to detect concept drift, to find change points in data streams, or to adjust models in the presence of observed drift, the problem of localizing drift, i.e. identifying it in data space, is yet widely unsolved - in particular from a formal perspective. This problem however is of importance, since it enables an inspection of the most prominent characteristics, e.g. features, where drift manifests itself and can therefore be used to make informed decisions, e.g. efficient updates of the training set of online learning algorithms, and perform precise adjustments of the learning model. In this paper we present a general theoretical framework that reduces drift localization to a supervised machine learning problem. We construct a new method for drift localization thereon and demonstrate the usefulness of our theory and the performance of our algorithm by comparing it to other methods from the literature.

The denoising of super-resolution structured illumination microscopy (SR-SIM) images using deep convolutional neural networks (CNNs) has re-ceived much attention in recent years. The contrast and fidelity of biological cell structures in denoised SR-SIM images critically depend on the CNN architecture and the chosen loss function. In this work, we propose two new combinations of loss functions for denoising SR-SIM images with low signal-to-noise ratio. The first combination consists of a loss component computed in the frequency domain by applying Fast Fourier Transform (FFT) to the image data and of pixel-wise loss. The second combination combines the loss in the frequency domain with a feature-based loss. In a series of experiments, these new loss functions are compared with various traditional loss functions for image de-noising. The results show that CNNs trained with the novel losses are superior in terms of PSNR and SSIM values. Moreover, critical structures of biological cells are better preserved in the denoised network output. Further results on the BSD dataset are also promising.

Abstract We present a modelling framework for the investigation of supervised learning in non-stationary environments. Specifically, we model two example types of learning systems: prototype-based learning vector quantization (LVQ) for classification and shallow, layered neural networks for regression tasks. We investigate so-called student–teacher scenarios in which the systems are trained from a stream of high-dimensional, labeled data. Properties of the target task are considered to be non-stationary due to drift processes while the training is performed. Different types of concept drift are studied, which affect the density of example inputs only, the target rule itself, or both. By applying methods from statistical physics, we develop a modelling framework for the mathematical analysis of the training dynamics in non-stationary environments. Our results show that standard LVQ algorithms are already suitable for the training in non-stationary environments to a certain extent. However, the application of weight decay as an explicit mechanism of forgetting does not improve the performance under the considered drift processes. Furthermore, we investigate gradient-based training of layered neural networks with sigmoidal activation functions and compare with the use of rectified linear units. Our findings show that the sensitivity to concept drift and the effectiveness of weight decay differs significantly between the two types of activation function.

While machine learning (ML) gives rise to astonishing results in automated systems, it is usually at the cost of large data requirements. This makes many successful algorithms from ML unsuitable for human-machine interaction, where the machine must learn from a small number of training samples that can be provided by a user within a reasonable time frame. Fortunately, the user can tailor the training data they create to be as useful as possible, severely limiting its necessary size—as long as they know about the machine’s requirements and limitations. Of course, acquiring this knowledge can in turn be cumbersome and costly. This raises the question of how easy ML algorithms are to interact with. In this work, we address this issue by analyzing the intuitiveness of certain algorithms when they are actively taught by users. After developing a theoretical framework of intuitiveness as a property of algorithms, we introduce an active teaching paradigm involving a prototypical two-dimensional spatial learning task as a method to judge the efficacy of human-machine interactions. Finally, we present and discuss the results of a large-scale user study into the performance and teaching strategies of 800 users interacting with two prominent ML algorithms in our system, providing first evidence for the role of intuition as an important factor impacting human-machine interaction.

Differentiable neural computers (DNCs) extend artificial neural networks with an explicit memory without interference, thus enabling the model to perform classic computation tasks, such as graph traversal. However, such models are difficult to train, requiring long training times and large datasets. In this work, we achieve some of the computational capabilities of DNCs with a model that can be trained very efficiently, namely, an echo state network with an explicit memory without interference. This extension enables echo state networks to recognize all regular languages, including those that contractive echo state networks provably cannot recognize. Furthermore, we demonstrate experimentally that our model performs comparably to its fully trained deep version on several typical benchmark tasks for DNCs.

Brinkrolf J, Hammer B. Federated Learning Vector Quantization. In: Verleysen M, ed. Proceedings of the ESANN, 29th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. Accepted.

In this paper, we propose a framework for application of a novel machine learning-based system for analyzing online social communications.As a example, we are targeting anti-Semitic graphical memes posted to social media.We presented very promising preliminary results on a Facebook dataset that consists of a total of 10000 labeled memes.We can conclude that machine learning will soon be able to successfully analyze and monitor complex social communications.

Prediction of movements is essential for successful cooperation with intelligent systems.We propose a model that integrates organized spatial information as given through the moving body's skeletal structure.This inherent structure is exploited in our model through application of Graph Convolutions and we demonstrate how this allows leveraging the structured spatial information into competitive predictions that are based on a lightweight model that requires a comparatively small number of parameters.* This research was supported by the research training group "Dataninja" (Trustworthy AI for Seamless Problem Solving: Next Generation Intelligence Joins Robust Data Analysis) funded by the German federal state of North Rhine-Westphalia.1 https://github.com/LucaHermes

The notion of concept drift refers to the phenomenon that the data distribution changes over time.If drift occurs, machine learning models need adjustment.Since drift can be inhomogeneous, suitable actions depending on the location in data space.In this paper we address the challenge to partition the data space into segments with homogeneous drift characteristics.We formalize this objective as an independence criterion, and derive a robust and efficient training algorithm based thereon.We evaluate the efficiency of the method in comparison to existing technologies: the identification of drifting clusters, and the estimation of a conditional density distribution.

When measuring data with hyperspectral cameras drift in the data distribution occurs over time and when the sensing device is changed.Frequently, this drift is a combination of intensity and wavelength shifts.In this contribution, we demonstrate that transfer component analysis together with subsampling constitutes a particular efficient and simple technology for spectral offset elimination which is applied to avoid the negative impact of drift on the classification performance.We demonstrate that this approach performs on par or better in comparison to established methods, and we also provide a theoretical motivation why this technology can deal with both, intensity as well as wavelength shift provided bounds on the smoothness of the functional data are given.

With the increasing deployment of machine learning systems in practice, transparency and explainability have become serious issues. Contrastive explanations are considered to be useful and intuitive, in particular when it comes to explaining decisions to lay people, since they mimic the way in which humans explain. Yet, so far, comparably little research has addressed computationally feasible technologies, which allow guarantees on uniqueness and optimality of the explanation and which enable an easy incorporation of additional constraints. Here, we will focus on specific types of models rather than black-box technologies. We study the relation of contrastive and counterfactual explanations and propose mathematical formalizations as well as a 2-phase algorithm for efficiently computing (plausible) pertinent positives of many standard machine learning models.

MOTIVATION: Innovative microfluidic systems carry the promise to greatly facilitate spatio-temporal analysis of single cells under well-defined environmental conditions, allowing novel insights into population heterogeneity and opening new opportunities for fundamental and applied biotechnology. Microfluidics experiments, however, are accompanied by vast amounts of data, such as time series of microscopic images, for which manual evaluation is infeasible due to the sheer number of samples. While classical image processing technologies do not lead to satisfactory results in this domain, modern deep-learning technologies, such as convolutional networks can be sufficiently versatile for diverse tasks, including automatic cell counting as well as the extraction of critical parameters, such as growth rate. However, for successful training, current supervised deep learning requires label information, such as the number or positions of cells for each image in a series; obtaining these annotations is very costly in this setting. RESULTS: We propose a novel machine-learning architecture together with a specialized training procedure, which allows us to infuse a deep neural network with human-powered abstraction on the level of data, leading to a high-performing regression model that requires only a very small amount of labeled data. Specifically, we train a generative model simultaneously on natural and synthetic data, so that it learns a shared representation, from which a target variable, such as the cell count, can be reliably estimated. AVAILABILITY AND IMPLEMENTATION: The project is cross-platform, open-source and free (MIT licensed) software. We make the source code available at https://github.com/dstallmann/cell_cultivation_analysis; the dataset is available at https://pub.uni-bielefeld.de/record/2945513.

Decentralization is a central characteristic of biological motor control that allows for fast responses relying on local sensory information. In contrast, the current trend of Deep Reinforcement Learning (DRL) based approaches to motor control follows a centralized paradigm using a single, holistic controller that has to untangle the whole input information space. This motivates to ask whether decentralization as seen in biological control architectures might also be beneficial for embodied sensori-motor control systems when using DRL. To answer this question, we provide an analysis and comparison of eight control architectures for adaptive locomotion that were derived for a four-legged agent, but with their degree of decentralization varying systematically between the extremes of fully centralized and fully decentralized. Our comparison shows that learning speed is significantly enhanced in distributed architectures-while still reaching the same high performance level of centralized architectures-due to smaller search spaces and local costs providing more focused information for learning. Second, we find an increased robustness of the learning process in the decentralized cases-it is less demanding to hyperparameter selection and less prone to becoming trapped in poor local minima. Finally, when examining generalization to uneven terrains-not used during training-we find best performance for an intermediate architecture that is decentralized, but integrates only local information from both neighboring legs. Together, these findings demonstrate beneficial effects of distributing control into decentralized units and relying on local information. This appears as a promising approach towards more robust DRL and better generalization towards adaptive behavior.

The notion of concept drift refers to the phenomenon that the distribution, which is underlying the observed data, changes over time; as a consequence machine learning models may become inaccurate and need adjustment. Many online learning schemes include drift detection to actively detect and react to observed changes. Yet, reliable drift detection constitutes a challenging problem in particular in the context of high dimensional data, varying drift characteristics, and the absence of a parametric model such as a classification scheme which reflects the drift. In this paper we present a novel concept drift detection method, Dynamic Adapting Window Independence Drift Detection (DAWIDD), which aims for non-parametric drift detection of diverse drift characteristics. For this purpose, we establish a mathematical equivalence of the presence of drift to the dependency of specific random variables in an according drift process. This allows us to rely on independence tests rather than parametric models or the classification loss, resulting in a fairly robust scheme to universally detect different types of drift, as it is also confirmed in experiments.

Convolutional neural networks have been used to achieve a string of successes during recent years, but their lack of interpretability remains a serious issue. Adversarial examples are designed to deliberately fool neural networks into making any desired incorrect classification, potentially with very high certainty. Several defensive approaches increase robustness against adversarial attacks, demanding attacks of greater magnitude, which lead to visible artifacts. By considering human visual perception, we compose a technique that allows to hide such adversarial attacks in regions of high complexity, such that they are imperceptible even to an astute observer. We carry out a user study on classifying adversarially modified images to validate the perceptual quality of our approach and find significant evidence for its concealment with regards to human visual perception.

Machine learning algorithms using deep architectures have been able to implement increasingly powerful and successful models. However, they also become increasingly more complex, more difficult to comprehend and easier to fool. So far, most methods in the literature investigate the decision of the model for a single given input datum. In this paper, we propose to visualize a part of the decision function of a deep neural network together with a part of the data set in two dimensions with discriminative dimensionality reduction. This enables us to inspect how different properties of the data are treated by the model, such as outliers, adversaries or poisoned data. Further, the presented approach is complementary to the mentioned interpretation methods from the literature and hence might be even more useful in combination with those. Code is available at https://github.com/LucaHermes/DeepView

Real-time learning from data streams in non-stationary environments gains ever more relevance due to the exponentially increasing amounts of generated data. Recently, the Self-Adjusting Memory (SAM) was proposed, an algorithm able to robustly handle heterogeneous types on the basis of two dedicated memories for the current and former concepts that continuously preserve consistency with explicit filtering. Yet, since the algorithm is restricted to one memory architecture, the variety of possible alternatives is limited by design in favor of an overall model consistency. Moreover, it does not actively detect drift, thus adapting with a relatively high delay in case of abrupt changes. We propose a dynamic ensemble on the basis of the SAM algorithm, which is triggered by both, the inherent passive adaptation of SAM and active drift detection. Further, since SAM is based on the stable k-Nearest-Neighbor algorithm, we investigate multiple approaches to obtain a high diversity in the ensemble, resulting in an effective overall strategy. The increased computational demand is countered on the basis of a parallel implementation. We extensively evaluate the method on numerous benchmarks, where it consistently achieves superior results in comparison to state-of-the-art methods.

Abstract Convolutional neural networks (CNNs) are deep learning frameworks which are well known for their notable performance in classification tasks. Hence, many skeleton-based action recognition and segmentation (SBARS) algorithms benefit from them in their designs. However, a shortcoming of such applications is the general lack of spatial relationships between the input features in such data types. Besides, non-uniform temporal scalings are a common issue in skeleton-based data streams which leads to having different input sizes even within one specific action category. In this work, we propose a novel deep-aligned convolutional neural network (DACNN) to tackle the above challenges for the particular problem of SBARS. Our network is designed by introducing a new type of filters in the context of CNNs which are trained based on their alignments to the local subsequences in the inputs. These filters result in efficient predictions as well as learning interpretable patterns in the data. Also, our DACNN framework can incrementally expand its deep structure based on the learning progress, which makes it flexible regarding different SBARS datasets. We empirically evaluate our framework on real-world benchmarks showing that the proposed DACNN algorithm obtains a competitive performance compared to the state of the art while benefiting from a less complicated yet more interpretable model.

Most existing feature selection methods are insufficient for analytic purposes as soon as high dimensional data or redundant sensor signals are dealt with since features can be selected due to spurious effects or correlations rather than causal effects. To support the finding of causal features in biomedical experiments, we hereby present FRI, an open source Python library that can be used to identify all-relevant variables in linear classification and (ordinal) regression problems. Using the recently proposed feature relevance interval method, FRI is able to provide the base for further general experimentation or in specific can facilitate the search for alternative biomarkers. It can be used in an interactive context, by providing model manipulation and visualization methods, or in a batch process as a filter method.

Prototype-based methods are of the particular interest for domain specialists and practitioners as they summarize a dataset by a small set of representatives. Therefore, in a classification setting, interpretability of the prototypes is as significant as the prediction accuracy of the algorithm. Nevertheless, the state-of-the-art methods make inefficient trade-offs between these concerns by sacrificing one in favor of the other, especially if the given data has a kernel-based (or multiple-kernel) representation. In this paper, we propose a novel interpretable multiple-kernel prototype learning (IMKPL) to construct highly interpretable prototypes in the feature space, which are also efficient for the discriminative representation of the data. Our method focuses on the local discrimination of the classes in the feature space and shaping the prototypes based on condensed class-homogeneous neighborhoods of data. Besides, IMKPL learns a combined embedding in the feature space in which the above objectives are better fulfilled. When the base kernels coincide with the data dimensions, this embedding results in a discriminative features selection. We evaluate IMKPL on several benchmarks from different domains which demonstrate its superiority to the related state-of-the-art methods regarding both interpretability and discriminative representation.

Online action classification is an important field of research, enabling the particularly interesting application scenario of controlling wearable devices which actively support the user's motions. The majority of machine learning applications of real-world systems are based on pre-trained average-user models without any personalization. Our long-term goal is to provide a system that adapts to its user's personal behavior patterns on the fly and in real-time. Ideally, we want to initiate a continuous collaboration between the system and the user where both alternatively adjust to each other to maximize the system's utility. Such tasks are not feasible with static models. In this paper, we investigate the potential and benefits of personalized online learning in the task of online action classification. We record motion sequences of different subjects wearing the Xsens bodysuit, which incorporates multiple inertial measuring units, enabling a fine-grained discrimination of motions. On this basis, we first perform a feature selection, showing that only a few sensors are necessary to achieve a high classification performance. Subsequently, we compare the recognition capabilities of offline average user models against personalized models trained in an online way. Our experiments conclude that personalized models require only few data to outperform average user systems and are particularly valuable for applications with limited computational hardware which rely on the raw sensor inputs only.

Multiple kernel learning (MKL) algorithms combine different base kernels to obtain a more efficient representation in the feature space. Focusing on discriminative tasks, MKL has been used successfully for feature selection and finding the significant modalities of the data. In such applications, each base kernel represents one dimension of the data or is derived from one specific descriptor. Therefore, MKL finds an optimal weighting scheme for the given kernels to increase the classification accuracy. Nevertheless, the majority of the works in this area focus on only binary classification problems or aim for linear separation of the classes in the kernel space, which are not realistic assumptions for many real-world problems. In this paper, we propose a novel multi-class MKL framework which improves the state-of-the-art by enhancing the local separation of the classes in the feature space. Besides, by using a sparsity term, our large-margin multiple kernel algorithm (LMMK) performs discriminative feature selection by aiming to employ a small subset of the base kernels. Based on our empirical evaluations on different real-world datasets, LMMK provides a competitive classification accuracy compared with the state-of-the-art algorithms in MKL. Additionally, it learns a sparse set of non-zero kernel weights which leads to a more interpretable feature selection and representation learning.

In recent years, kernel-based sparse coding (K-SRC) has received a special attention due to its efficient representation of nonlinear data structures in the feature space. Nevertheless, the existing K-SRC methods suffer from the lack of consistency between their training and test optimization frameworks. In this work, we propose a novel confident K-SRC and dictionary learning algorithm (CKSC) which focuses on the discriminative reconstruction of the data based on its representation in the kernel space. CKSC focuses on reconstructing each data sample via weighted contributions which are confident in its corresponding class of data. We employ novel discriminative terms to apply this scheme to both training and test frameworks in our algorithm. This increases the consistency of these optimization frameworks and improves the discriminative performance in the recall phase. In addition, CKSC directly employs the supervised information in its dictionary learning framework to enhance the discriminative structure of the dictionary. For empirical evaluations, we implement our CKSC algorithm on multivariate time-series benchmarks such as DynTex++ and UTKinect. Our claims regarding the superior performance of the proposed algorithm are justified throughout comparing its classification results to the state-of-the-art K-SRC algorithms.

An increasing number of industrial areas recognize the opportunities of Big Data, requiring highly efficient algorithms which enable real-time processing to reduce the burden of data storage and maintenance. Decision trees are extremely fast, highly accurate and easy to use in practice. Merging multiple decision trees to an ensemble leads to one of the most powerful machine learning methods. The Very Fast Decision Tree is the state-of-the-art incremental decision tree induction algorithm, capable of learning from massive data streams. It is successful due to its theoretical guarantees based on the Hoeffding bound as well as its competitive performance in terms of classification accuracy and time / space efficiency. In this paper, we increase the efficiency even further by replacing its global splitting scheme, which periodically tries to split every nminexamples. Instead, we utilize local statistics to predict the split-time, thus, avoiding unnecessary split-attempts, usually dominating the computational cost. Concretely, we use the class distributions of previous split-attempts to approximate the minimum number of examples until the Hoeffding bound is met. This cautious approach yields by design a low delay and reduces the number of split-attempts at the same time. We extensively evaluate our method using common stream-learning benchmarks also considering non-stationary environments. The experiments confirm a substantially reduced run-time without a loss in classification performance.

Metric learning has the aim to improve classification accuracy by learning a distance measure which brings data points from the same class closer together and pushes data points from different classes further apart. Recent research has demonstrated that metric learning approaches can also be applied to trees, such as molecular structures, abstract syntax trees of computer programs, or syntax trees of natural language, by learning the cost function of an edit distance, i.e. the costs of replacing, deleting, or inserting nodes in a tree. However, learning such costs directly may yield an edit distance which violates metric axioms, is challenging to interpret, and may not generalize well. In this contribution, we propose a novel metric learning approach for trees which we call embedding edit distance learning (BEDL) and which learns an edit distance indirectly by embedding the tree nodes as vectors, such that the Euclidean distance between those vectors supports class discrimination. We learn such embeddings by reducing the distance to prototypical trees from the same class and increasing the distance to prototypical trees from different classes. In our experiments, we show that BEDL improves upon the state-of-the-art in metric learning for trees on six benchmark data sets, ranging from computer science over biomedical data to a natural-language processing data set containing over 300,000 nodes.

Recent clustering based anomaly detection technologies classify new observations in different ways, e.g. using probability distributions, cluster centers or whole data points. Some of which suffer from high false classification rate, while others require high computational resources. In this paper, we propose a geometric approach to clustering based anomaly detection, in which the boundaries of clusters are utilized to classify new observations instead. To identify the cluster boundaries, a new algorithm for generating n-dimensional non-convex hulls has been developed. The proposed approach can improve the accuracy of clustering based anomaly detection, meanwhile, doesn't need high computational resources. Furthermore, it is universally applicable for any kind of cluster algorithms. The effectiveness of this approach is evaluated with real world data collected from different industrial automation systems.

Deep neural network based condition monitoring systems are used to detect system failures of cyber-physical production systems. However, a vulnerability of deep neural networks are adversarial examples. They are manipulated inputs, e.g. process data, with the ability to mislead a deep neural network into misclassification. Adversarial example attacks can manipulate the physical production process of a cyber-physical production system without being recognized by the condition monitoring system. Manipulation of the physical process poses a serious threat for production systems and employees. This paper introduces CyberProtect, a novel approach to prevent misclassification caused by adversarial example attacks. CyberProtect generates adversarial examples and uses them to retrain deep neural networks. This results in a hardened deep neural network with a significant reduced misclassification rate. The proposed countermeasure increases the classification rate from 20% to 82%, as proved by empirical results.

Motivation: Identification of cell populations in flow cytometry is a critical part of the analysis and lays the groundwork for many applications and research discovery. The current paradigm of manual analysis is time consuming and subjective. A common goal of users is to replace manual analysis with automated methods that replicate their results. Supervised tools provide the best performance in such a use case, however they require fine parameterization to obtain the best results. Hence, there is a strong need for methods that are fast to setup, accurate and interpretable. Results: flowLearn is a semi-supervised approach for the quality-checked identification of cell populations. Using a very small number of manually gated samples, through density alignments it is able to predict gates on other samples with high accuracy and speed. On two state-of-the-art datasets, our tool achieves median(F1)-measures exceeding 0.99 for 31%, and 0.90 for 80% of all analyzed populations. Furthermore, users can directly interpret and adjust automated gates on new sample files to iteratively improve the initial training. Availability and implementation: FlowLearn is available as an R package on https://github.com/mlux86/flowLearn. Evaluation data is publicly available online. Details can be found in the Supplementary Material. Supplementary information: Supplementary data are available at Bioinformatics online.

Data Mining in non-stationary data streams is particularly relevant in the context of the Internet of Things and Big Data. Its challenges arise from fundamentally different drift types violating assumptions of data independence or stationarity. Available methods often struggle with certain forms of drift or require unavailable a priori task knowledge. We propose the Self-Adjusting Memory (SAM) model for the k Nearest Neighbor (kNN) algorithm. SAM-kNN can deal with heterogeneous concept drift, i.e. different drift types and rates. Its basic idea are dedicated models for current and former concepts used according to the demands of the given situation. It can be robustly applied in practice without meta parameter optimization. We conduct an extensive evaluation on various benchmarks, consisting of artificial streams with known drift characteristics and real-world datasets. Highly competitive results throughout all experiments underline the robustness of SAM-kNN as well as its capability to handle heterogeneous concept drift.

Label noise-tolerant machine learning techniques address datasets which are affected by mislabelling of the instances. Since labelling quality is a severe issue in particular for large or streaming data sets, this setting becomes more and more relevant in the context of life-long learning, big data and crowd sourcing. In this contribution, we extend a powerful online learning method, soft robust learning vector quantisation, by a probabilistic model for noise tolerance, which is applicable for streaming data, including label-noise drift. The superiority of the technique is demonstrated in several benchmark problems.

This issue marks the first anniversary issue since I was honored to serve as the Editor-in-Chief (EiC) of the IEEE Transactions on Neural Networks and Learning Systems (TNNLS). I am happy to report that we had a very successful year and here are a few highlights that I would like to share with the community.•The latest impact factor of TNNLS is 4.854 according to the Journal Citation Reports. This marks a record high impact factor for our journal and places TNNLS as the number one scholarly publication in Computer Science (Hardware & Architecture), number three in Computer Science (Theory & Methods), and number ten in Electrical and Electronic Engineering journals.

Data Mining in non-stationary data streams is gaining more attentionrecently, especially in the context of Internet of Things and Big Data. It is a highly challenging task, since the fundamentally different typesof possibly occurring drift undermine classical assumptions such asi.i.d. data or stationary distributions. Available algorithms are either struggling with certain forms of drift or require a priori knowledge in terms of a task specific setting. We propose the Self Adjusting Memory (SAM) model for the k Nearest Neighbor (kNN) algorithm since kNN constitutes a proven classifier within the streaming setting. SAM-kNN can deal with heterogeneous concept drift, i.e different drift types and rates, using biologically inspiredmemory models and their coordination. It can be easilyapplied in practice since an optimization of the meta parameters is not necessary. The basic idea is to construct dedicated models for thecurrent and former concepts and apply them according tothe demands of the given situation. An extensive evaluation on various benchmarks, consisting of artificial streamswith known drift characteristics as well as real world datasets is conducted. Thereby, we explicitly add new benchmarks enabling a precise performance evaluation on multiple types of drift. The highly competitive results throughout all experiments underline the robustness of SAM-kNN as well as its capabilityto handle heterogeneous concept drift.

One of the main aims of lifelong learning architectures is to efficiently and reliably cope with the stability-plasticity dilemma. A viable solution of this dilemma combines a static offline classifier, which preserves ground knowledge that should be respected during training, with an incremental online learning of new or specific information encountered during use. A feasible realisation has been published lately based on intuitive distance-based classifiers using the concept of metric learning (Fischer et al.: Combining offline and online classifiers for life-long learning (OOL), IJCNN'15). One crucial aspect of such a system is how to combine the offline and online model. A generic approach, taken in OOL, uses a dynamic classifier selection strategy based on confidences of both classifiers. This can cause problems in the case of confidence drift, especially when the validity of the confidence estimation of the static offline classifier changes. This pitfall occurs in the context of metric learning whenever the metric tensor of the online system becomes orthogonal to the metric of the offline system, hence the respective internal data description mismatch. We propose an efficient metric learning strategy which allows an online adaptation of an invalid confidence estimation of the OOL architecture in case of confidence drift.

BACKGROUND: A major obstacle in single-cell sequencing is sample contamination with foreign DNA. To guarantee clean genome assemblies and to prevent the introduction of contamination into public databases, considerable quality control efforts are put into post-sequencing analysis. Contamination screening generally relies on reference-based methods such as database alignment or marker gene search, which limits the set of detectable contaminants to organisms with closely related reference species. As genomic coverage in the tree of life is highly fragmented, there is an urgent need for a reference-free methodology for contaminant identification in sequence data. RESULTS: We present acdc, a tool specifically developed to aid the quality control process of genomic sequence data. By combining supervised and unsupervised methods, it reliably detects both known and de novo contaminants. First, 16S rRNA gene prediction and the inclusion of ultrafast exact alignment techniques allow sequence classification using existing knowledge from databases. Second, reference-free inspection is enabled by the use of state-of-the-art machine learning techniques that include fast, non-linear dimensionality reduction of oligonucleotide signatures and subsequent clustering algorithms that automatically estimate the number of clusters. The latter also enables the removal of any contaminant, yielding a clean sample. Furthermore, given the data complexity and the ill-posedness of clustering, acdc employs bootstrapping techniques to provide statistically profound confidence values. Tested on a large number of samples from diverse sequencing projects, our software is able to quickly and accurately identify contamination. Results are displayed in an interactive user interface. Acdc can be run from the web as well as a dedicated command line application, which allows easy integration into large sequencing project analysis workflows. CONCLUSIONS: Acdc can reliably detect contamination in single-cell genome data. In addition to database-driven detection, it complements existing tools by its unsupervised techniques, which allow for the detection of de novo contaminants. Our contribution has the potential to drastically reduce the amount of resources put into these processes, particularly in the context of limited availability of reference species. As single-cell genome data continues to grow rapidly, acdc adds to the toolkit of crucial quality assurance tools.

We investigate metric learning in the context of dynamic time warping (DTW), the by far most popular dissimilarity measure used for the comparison and analysis of motion capture data. While metric learning enables a problem-adapted representation of data, the majority of methods has been proposed for vectorial data only. In this contribution, we extend the popular principle offered by the large margin nearest neighbours learner (LMNN) to DTW by treating the resulting component-wise dissimilarity values as features. We demonstrate, that this principle greatly enhances the classification accuracy in several benchmarks. Further, we show that recent auxiliary concepts such as metric regularisation can be transferred from the vectorial case to component-wise DTW in a similar way. We illustrate, that metric regularisation constitutes a crucial prerequisite for the interpretation of the resulting relevance profiles.

We present a theoretical analysis of Learning Vector Quantization (LVQ) with adaptive distance measures. Specifically, we consider generalized Euclidean distances which are parameterized in terms of a quadratic matrix of adaptive relevance parameters. Winner-takes-all prescriptions based on the heuristic LVQ1 are in the center of our interest. We derive and study stationarity conditions and show, among other results, that stationary prototypes can be written as linear combinations of the training data apart from irrelevant contributions in the null-space of the relevance matrix. The investigation of the metrics updates reveals that relevance matrices become singular with only one or very few non-zero eigenvalues. Implications of this property are discussed and, furthermore, the effect of preventing singularity by introducing an appropriate penalty term is studied. Theoretical findings are confirmed in terms of illustrative example data sets.

One of the greatest challenges of life-long learning architectures is how to efficiently and reliably cope with the stability-plasticity dilemma. We propose an extension of a flexible system combining a static offline classifier and an incremental online classifier that is well suited for life-long learning scenarios. The pre-trained offline classifier preserves ground knowledge that should be respected during training, while the online classifier enables learning of new or specific information encountered during use. The combination is realised by a dynamic classifier selection strategy based on confidences of both ingredients. We report exemplary results of this architecture for the case of learning vector quantization (LVQ) for several data sets, thereby including an extensive comparison to alternative state of the art algorithms for incremental learning such as incremental generalised LVQ and the support vector machine.

We present an architecture for incremental online learning in high-dimensional feature spaces and apply it on a mobile robot. The model is based on learning vector quantization, approaching the stability-plasticity problem of incremental learning by adaptive insertions of representative vectors. We employ a cost-function-based learning vector quantization approach and introduce a new insertion strategy optimizing a cost-function based on a subset of samples. We demonstrate this model within a real-time application for a mobile robot scenario, where we perform interactive real-time learning of visual categories.

Binning constitutes a crucial step of de novo metagenomics data analysis, and several promising attempts to partially automate this process have been proposed; quite a few recent approaches rely on machine learning techniques, in particular clustering. However, so far, there does not exist a fully automated process, nor a thorough evaluation of its accuracy and robustness with respect to parameterisation. This contribution addresses the following issues: (i) an integration of modern dimensionality reduction and clustering techniques suitable for high dimensional data, and an automated selection of the number of clusters, (ii) a formal quantitative evaluation of the pipeline in benchmarks, (iii) and an evaluation of an optimum parameter choice, resulting in a complete automation of the process.

Discriminative dimensionality reduction refers to the goal of visualizing given high-dimensional data in the plane such that the structure relevant for a specified aspect is displayed. While this framework has been successfully applied to visualize data with auxiliary label information, its extension to real-valued information is lacking. In this contribution, we propose a general way to shape data distances based on auxiliary real-valued information with the Fisher metric which is derived from a Gaussian process model of the data. This can directly be integrated into high quality non-linear dimensionality reduction methods such as t-SNE, as we will demonstrate in artificial as well as real life benchmarks.

Linear data transformations constitute essential operations in various machine learning algorithms, ranging from linear regression up to adaptive metric transformation. Often, linear scalings are not only used to improve the model accuracy, rather feature coefficients as provided by the mapping are interpreted as an indicator for the relevance of the feature for the task at hand. This principle, however, can be misleading in particular for high-dimensional or correlated features, since it easily marks irrelevant features as relevant or vice versa. In this contribution, we propose a mathematical formalisation of the minimum and maximum feature relevance for a given linear transformation which can efficiently be solved by means of linear programming. We evaluate the method in several benchmarks, where it becomes apparent that the minimum and maximum relevance closely resembles what is often referred to as weak and strong relevance of the features; hence unlike the mere scaling provided by the linear mapping, it ensures valid interpretability.

Linear data transformations are essential operations in many machine learning algorithms, helping to make such models more flexible or to emphasize certain data directions. In particular for high dimensional data sets linear transformations are not necessarily uniquely determined, though, and alternative parameterizations exist which do not change the mapping of the training data. Thus, regularization is required to make the model robust to noise and more interpretable for the user. In this contribution, we characterize the group of transformations which leave a linear mapping invariant for a given finite data set, and we discuss the consequences on the interpretability of the models. We propose an intuitive regularization mechanism to avoid problems in under-determined configurations, and we test the approach in two machine learning models.

Building an Intelligent Tutoring System (ITS) from scratch usually requires technological skills, expertise about the domain and tasks, and pedagogical knowledge. We propose a middleware which facilitates the construction of intelligently supported learning systems independently from the underlying (formalized) domain knowledge using typical re-usable components of ITSs, exchangeable plug-ins, and machine learning techniques running in Matlab. We proved our concept using a web-based programming environment as an example of an user interface component interacting with our ITS middleware.

We investigate the potential of modern nonlinear dimensionality reduction techniques for an interactive cluster detection in bioinformatics applications. We demonstrate that recent non-parametric techniques such as t-distributed stochastic neighbor embedding (t-SNE) allow a cluster identification which is superior to direct clustering of the original data or cluster detection based on classical parametric dimensionality reduction approaches. Non-parametric approaches, however, display quadratic complexity which makes them unsuitable in interactive devices. As speedup, we propose kernel-t-SNE as a fast parametric counterpart based on t-SNE.

t-distributed stochastic neighbor embedding (t-SNE) constitutes a nonlinear dimensionality reduction technique which is particularly suited to visualize high dimensional data sets with intrinsic nonlinear structures. A major drawback, however, consists in its squared complexity which makes the technique infeasible for large data sets or online application in an interactive framework. In addition, since the technique is non parametric, it possesses no direct method to extend the technique to novel data points. In this contribution, we propose an extension of t-SNE to an explicit mapping. In the limit, it reduces to standard non-parametric t-SNE, while offering a feasible nonlinear embedding function for other parameter choices. We evaluate the performance of the technique when trained on a small subpart of the given data only. It turns out that its generalization ability is good when evaluated with the standard quality curve. Further, in many cases, it obtains a quality which approximates the quality of t-SNE when trained on the full data set, albeit only 10% of the data are used for training. This opens the way towards efficient nonlinear dimensionality reduction techniques as required in interactive settings.

Digital data characterizing physiological processes over time are becoming increasingly important such as spectrometric data or gene expression profiles. Typical characteristics of such data are high dimensionality due to a fine grained measurement, but usually only few time points of the series. Due to the short length, classical time series models cannot be used. At the same time, due to the high dimensionality, data cannot be treated by means of time windows using simple vectorial techniques. Here, we consider the generative topographic mapping through time (GTM-TT) as a highly regularized model for time series inspection in the unsupervised setting, based on hidden Markov models enhanced with topographic mapping facilities. We extend the model such that supervised classification can be built on top of GTM-TT, resulting in supervised GTM-TT, and we extend the technique by supervised relevance learning. The latter adapts the metric according to given auxiliary information resulting in an interpretable form which can deal with high dimensional inputs. We demonstrate the technique in simulated data as well as an example from the biomedical domain, reaching state of the art classification accuracy in both cases.

Prototype based learning offers an intuitive interface to inspect large quantities of electronic data in supervised or unsupervised settings. Recently, many techniques have been extended to data described by general dissimilarities rather than Euclidean vectors, so-called relational data settings. Unlike the Euclidean counterparts, the techniques have quadratic time complexity due to the underlying quadratic dissimilarity matrix. Thus, they are infeasible already for medium sized data sets. The contribution of this article is twofold: On the one hand we propose a novel supervised prototype based classification technique for dissimilarity data based on popular learning vector quantization (LVQ), on the other hand we transfer a linear time approximation technique, the Nyström approximation, to this algorithm and an unsupervised counterpart, the relational generative topographic mapping (GTM). This way, linear time and space methods result. We evaluate the techniques on three examples from the biomedical domain.

In recent years, a wealth of dimension-reduction techniques for data visualization and preprocessing has been established. Nonparametric methods require additional effort for out-of-sample extensions, because they provide only a mapping of a given finite set of points. In this letter, we propose a general view on nonparametric dimension reduction based on the concept of cost functions and properties of the data. Based on this general principle, we transfer nonparametric dimension reduction to explicit mappings of the data manifold such that direct out-of-sample extensions become possible. Furthermore, this concept offers the possibility of investigating the generalization ability of data visualization to new data points. We demonstrate the approach based on a simple global linear mapping, as well as prototype-based local linear mappings. In addition, we can bias the functional form according to given auxiliary information. This leads to explicit supervised visualization mappings with discriminative properties comparable to state-of-the-art approaches.

The increasing size and complexity of modern data sets turns modern data mining techniques to indispensable tools when inspecting biomedical data sets. Thereby, dedicated data formats and detailed information often cause the need for problem specific similarities or dissimilarities instead of the standard Euclidean norm. Therefore, a number of clustering techniques which rely on similarities or dissimilarities only have recently been proposed. In this contribution, we review some of the most popular dissimilarity based clustering techniques and we discuss possibilities how to get around the usually squared complexity of the models due to their dependency on the full dissimilarity matrix. We evaluate the techniques on two benchmarks from the biomedical domain.

A wealth of powerful dimensionality reduction methods has been established which can be used for data visualization and preprocessing. These are accompanied by formal evaluation schemes, which allow a quantitative evaluation along general principles and which even lead to further visualization schemes based on these objectives. Most methods, however, provide a mapping of a priorly given finite set of points only, requiring additional steps for out-of-sample extensions. We propose a general view on dimensionality reduction based on the concept of cost functions, and, based on this general principle, extend dimensionality reduction to explicit mappings of the data manifold. This offers simple out-of-sample extensions. Further, it opens a way towards a theory of data visualization taking the perspective of its generalization ability to new data points. We demonstrate the approach based on a simple global linear mapping as well as prototype-based local linear mappings.

Prototype based classifiers are effective algorithms in modeling classification problems and have been applied in multiple domains. While many supervised learning algorithms have been successfully extended to kernels to improve the discrimination power by means of the kernel concept, prototype based classifiers are typically still used with Euclidean distance measures. Kernelized variants of prototype based classifiers are currently too complex to be applied for larger data sets. Here we propose an extension of Kernelized Generalized Learning Vector Quantization (KGLVQ) employing a sparsity and approximation technique to reduce the learning complexity. We provide generalization error bounds and experimental results on real world data, showing that the extended approach is comparable to SVM on different public data.

We present two approaches to extend Robust Soft Learning Vector Quantization (RSLVQ). This algorithm for nearest prototype classification is derived from an explicit cost function and follows the dynamics of a stochastic gradient ascent. The RSLVQ cost function is defined in terms of a likelihood ratio and involves a hyperparameter which is kept constant during training. We propose to adapt the hyperparameter in the training phase based on the gradient information . Besides, we propose to base the classifier's decision on the value of the likelihood ratio instead of using the distance based classification approach. Experiments on artificial and real life data show that the hyperparameter crucially influences the performance of RSLVQ. However, it is not possible to estimate the best value from the data prior to learning. We show that the proposed variant of RSLVQ is very robust with respect to the initial value of the hyperparameter. The classification approach based on the likelihood ratio turns out to be superior to distance based classification, if local hyperparameters are adapted for each prototype.

Topographic maps such as the self-organizing map (SOM) or neural gas (NG) constitute powerful data mining techniques that allow simultaneously clustering data and inferring their topological structure, such that additional features, for example, browsing, become available. Both methods have been introduced for vectorial data sets; they require a classical feature encoding of information. Often data are available in the form of pairwise distances only, such as arise from a kernel matrix, a graph, or some general dissimilarity measure. In such cases, NG and SOM cannot be applied directly. In this article, we introduce relational topographic maps as an extension of relational clustering algorithms, which offer prototype-based representations of dissimilarity data, to incorporate neighborhood structure. These methods are equivalent to the standard (vectorial) techniques if a Euclidean embedding exists, while preventing the need to explicitly compute such an embedding. Extending these techniques for the general case of non-Euclidean dissimilarities makes possible an interpretation of relational clustering as clustering in pseudo-Euclidean space. We compare the methods to well-known clustering methods for proximity data based on deterministic annealing and discuss how far convergence can be guaranteed in the general case. Relational clustering is quadratic in the number of data points, which makes the algorithms infeasible for huge data sets. We propose an approximate patch version of relational clustering that runs in linear time. The effectiveness of the methods is demonstrated in a number of examples.

A variety of modifications have been employed to learning vector quantization (LVQ) algorithms using either crisp or soft windows for selection of data. Although these schemes have been shown in practice to improve performance, a theoretical study on the influence of windows has so far been limited. Here we rigorously analyze the influence of windows in a controlled environment of gaussian mixtures in high dimensions. Concepts from statistical physics and the theory of online learning allow an exact description of the training dynamics, yielding typical learning curves, convergence properties, and achievable generalization abilities. We compare the performance and demonstrate the advantages of various algorithms, including LVQ 2.1, generalized LVQ (GLVQ), Learning from Mistakes (LFM) and Robust Soft LVQ (RSLVQ). We find that the selection of the window parameter highly influences the learning curves but not, surprisingly, the asymptotic performances of LVQ 2.1 and RSLVQ. Although the prototypes of LVQ 2.1 exhibit divergent behavior, the resulting decision boundary coincides with the optimal decision boundary, thus yielding optimal generalization ability.

In this paper, we present a regularization technique to extend recently proposed matrix learning schemes in learning vector quantization (LVQ). These learning algorithms extend the concept of adaptive distance measures in LVQ to the use of relevance matrices. In general, metric learning can display a tendency towards oversimplification in the course of training. An overly pronounced elimination of dimensions in feature space can have negative effects on the performance and may lead to instabilities in the training. We focus on matrix learning in generalized LVQ (GLVQ). Extending the cost function by an appropriate regularization term prevents the unfavorable behavior and can help to improve the generalization ability. The approach is first tested and illustrated in terms of artificial model data. Furthermore, we apply the scheme to benchmark classification data sets from the UCI Repository of Machine Learning. We demonstrate the usefulness of regularization also in the case of rank limited relevance matrices, i.e., matrix learning with an implicit, low-dimensional representation of the data.

Discriminative vector quantization schemes such as learning vector quantization (LVQ) and extensions thereof offer efficient and intuitive classifiers based on the representation of classes by prototypes. The original methods, however, rely on the Euclidean distance corresponding to the assumption that the data can be represented by isotropic clusters. For this reason, extensions of the methods to more general metric structures have been proposed, such as relevance adaptation in generalized LVQ (GLVQ) and matrix learning in GLVQ. In these approaches, metric parameters are learned based on the given classification task such that a data-driven distance measure is found. In this letter, we consider full matrix adaptation in advanced LVQ schemes. In particular, we introduce matrix learning to a recent statistical formalization of LVQ, robust soft LVQ, and we compare the results on several artificial and real-life data sets to matrix learning in GLVQ, a derivation of LVQ-like learning based on a (heuristic) cost function. In all cases, matrix adaptation allows a significant improvement of the classification accuracy. Interestingly, however, the principled behavior of the models with respect to prototype locations and extracted matrix dimensions shows several characteristic differences depending on the data sets.

We propose a new matrix learning scheme to extend relevance learning vector quantization (RLVQ), an efficient prototype-based classification algorithm, toward a general adaptive metric. By introducing a full matrix of relevance factors in the distance measure, correlations between different features and their importance for the classification scheme can be taken into account and automated, and general metric adaptation takes place during training. In comparison to the weighted Euclidean metric used in RLVQ and its variations, a full matrix is more powerful to represent the internal structure of the data appropriately. Large margin generalization bounds can be transferred to this case, leading to bounds that are independent of the input dimensionality. This also holds for local metrics attached to each prototype, which corresponds to piecewise quadratic decision boundaries. The algorithm is tested in comparison to alternative learning vector quantization schemes using an artificial data set, a benchmark multiclass problem from the UCI repository, and a problem from bioinformatics, the recognition of splice sites for C. elegans.

In the present contribution we propose two recently developed classification algorithms for the analysis of mass-spectrometric data-the supervised neural gas and the fuzzy-labeled self-organizing map. The algorithms are inherently regularizing, which is recommended, for these spectral data because of its high dimensionality and the sparseness for specific problems. The algorithms are both prototype-based such that the principle of characteristic representants is realized. This leads to an easy interpretation of the generated classifcation model. Further, the fuzzy-labeled self-organizing map is able to process uncertainty in data, and classification results can be obtained as fuzzy decisions. Moreover, this fuzzy classification together with the property of topographic mapping offers the possibility of class similarity detection, which can be used for class visualization. We demonstrate the power of both methods for two exemplary examples: the classification of bacteria (listeria types) and neoplastic and non-neoplastic cell populations in breast cancer tissue sections.

K-means clustering combines a variety of striking properties because of which it is widely used in applications: training is intuitive and simple, the final classifier represents classes by geometrically meaningful prototypes, and the algorithm is quite powerful compared to more complex alternative clustering algorithms. In this contribution, we focus on extensions which incorporate additional information into the clustering algorithm to achieve a better accuracy: neighborhood cooperation from neural gas, (possibly fuzzy) label information of input data, and general problem-adapted distances instead of the standard Euclidean metric. These extensions can be formulated in a simple general framework by means of a cost function. We demonstrate the ability of these variants on several representative clustering problems from computational biology.

We extend the self-organizing map in the variant as proposed by Heskes to a supervised fuzzy classification method. This leads to a robust classifier where efficient learning with fuzzy labeled or partially contradictory data is possible. Further, the integration of labeling into the location of prototypes in a self-organizing map leads to a visualization of those parts of the data relevant for the classification. The method is incorporated in a clinical proteomics toolkit dedicated for biomarker search which allows the necessary preprocessing and further data analysis with additional visualizations.

Learning vector quantization (LVQ) constitutes a powerful and intuitive method for adaptive nearest prototype classification. However, original LVQ has been introduced based on heuristics and numerous modifications exist to achieve better convergence and stability. Recently, a mathematical foundation by means of a cost function has been proposed which, as a limiting case, yields a learning rule similar to classical LVQ2.1. It also motivates a modification which shows better stability. However, the exact dynamics as well as the generalization ability of many LVQ algorithms have not been thoroughly investigated so far. Using concepts from statistical physics and the theory of on-line learning, we present a mathematical framework to analyse the performance of different LVQ algorithms in a typical scenario in terms of their dynamics, sensitivity to initial conditions, and generalization ability. Significant differences in the algorithmic stability and generalization ability can be found already for slightly different variants of LVQ. We study five LVQ algorithms in detail: Kohonen's original LVQ1, unsupervised vector quantization (VQ), a mixture of VQ and LVQ, LVQ2.1, and a variant of LVQ which is based on a cost function. Surprisingly, basic LVQ1 shows very good performance in terms of stability, asymptotic generalization ability, and robustness to initializations and model parameters which, in many cases, is superior to recent alternative proposals.

We extend soft nearest neighbor classification to fuzzy classification with adaptive class labels. The adaptation follows a gradient descent on a cost function. Further, it is applicable for general distance measures, in particular task specific choices and relevance learning for metric adaptation can be done. The performance of the algorithm is shown on synthetic as well as on real life data taken from proteomic research.

Cascade correlation (CC) constitutes a training method for neural networks that determines the weights as well as the neural architecture during training. Various extensions of CC to structured data have been proposed: recurrent cascade correlation (RCC) for sequences, recursive cascade correlation (RecCC) for tree structures with limited fan-out, and contextual recursive cascade correlation (CRecCC) for rooted directed positional acyclic graphs (DPAGs) with limited fan-in and fan-out. We show that these models possess the universal approximation property in the following sense: given a probability measure P on the input set, every measurable function from sequences into a real vector space can be approximated by a sigmoidal RCC up to any desired degree of accuracy up to inputs of arbitrary small probability. Every measurable function from tree structures with limited fan-out into a real vector space can be approximated by a sigmoidal RecCC with multiplicative neurons up to any desired degree of accuracy up to inputs of arbitrary small probability. For sigmoidal CRecCC networks with multiplicative neurons, we show the universal approximation capability for functions on an important subset of all DPAGs with limited fan-in and fan-out for which a specific linear representation yields unique codes. We give one sufficient structural condition for the latter property, which can easily be tested: the enumeration of ingoing and outgoing edges should becom patible. This property can be fulfilled for every DPAG with fan-in and fan-out two via reenumeration of children and parents, and for larger fan-in and fan-out via an expansion of the fan-in and fan-out and reenumeration of children and parents. In addition, the result can be generalized to the case of input-output isomorphic transductions of structures. Thus, CRecCC networks consti-tute the first neural models for which the universal approximation ca-pability of functions involving fairly general acyclic graph structures is proved.

The paper deals with the application of the novel generalized relevance learning vector quantization based classification algorithm in comparison to the unified maximum separability analysis as a special variant of support vector machine algorithms. The algorithms are compared and their performance on real life data, taken from clinical studies, is demonstrated. It is shown that the vector quantization classifier gives competitive results in comparison to the considered support vector machine algorithm and shows the recently theoretical proven equivalence in classification capability of both paradigms.

A regret-based biased random sampling scheme (RBRS) is a simple but powerful priority-rule based method to solve the resource constrained project scheduling problem (RCPSP), a well-known NP-hard benchmark problem. We present a generic machine learning method to improve results of RBRS. The rout-algorithm of reinforcement learning is combined with the support vector machine (SVM) to learn an appropriate value function which guides the search strategy given by RBRS. The specific properties of the SVM allow to reduce the size of the training set and show improved results even after a short period of training as demonstrated for benchmark instances of the RCPSP.

Recent experimental studies indicate that recurrent neural networks initialized with “small” weights are inherently biased toward definite memory machines (Tiňno, Čerňanský, & Beňušková, 2002a, 2002b). This article establishes a theoretical counterpart: transition function of recurrent network with small weights and squashing activation function is a contraction. We prove that recurrent networks with contractive transition function can be approximated arbitrarily well on input sequences of unbounded length by a definite memory machine. Conversely, every definite memory machine can be simulated by a recurrent network with contractive transition function. Hence, initialization with small weights induces an architectural bias into learning with recurrent neural networks. This bias might have benefits from the point of view of statistical learning theory: it emphasizes one possible region of the weight space where generalization ability can be formally proved. It is well known that standard recurrent neural networks are not distribution independent learnable in the probably approximately correct (PAC) sense if arbitrary precision and inputs are considered. We prove that recurrent networks with contractive transition function with a fixed contraction parameter fulfill the so-called distribution independent uniform convergence of empirical distances property and hence, unlike general recurrent networks, are distribution independent PAC learnable.

We have recently shown that when initialized with “small” weights, recurrent neural networks (RNNs) with standard sigmoid-type activation functions are inherently biased toward Markov models; even prior to any training, RNN dynamics can be readily used to extract finite memory machines (Hammer & Tiňo, 2002; Tiňo, Čerňanský, &Beňušková, 2002a, 2002b). Following Christiansen and Chater (1999), we refer to this phenomenon as the architectural bias of RNNs. In this article, we extend our work on the architectural bias in RNNs by performing a rigorous fractal analysis of recurrent activation patterns. We assume the network is driven by sequences obtained by traversing an underlying finite-state transition diagram&a scenario that has been frequently considered in the past, for example, when studying RNN-based learning and implementation of regular grammars and finite-state transducers. We obtain lower and upper bounds on various types of fractal dimensions, such as box counting and Hausdorff dimensions. It turns out that not only can the recurrent activations inside RNNs with small initial weights be explored to build Markovian predictive models, but also the activations form fractal clusters, the dimension of which can be bounded by the scaled entropy of the underlying driving source. The scaling factors are fixed and are given by the RNN parameters.

The information theoretical learnability of folding networks, a very successful approach capable of dealing with tree structured inputs, is examined. We find bounds on the VC, pseudo-, and fat shattering dimension of folding networks with various activation functions. As a consequence, valid generalization of folding networks can be guaranteed. However, distribution independent bounds on the generalization error cannot exist in principle. We propose two approaches which take the specific distribution into account and allow us to derive explicit bounds on the deviation of the empirical error from the real error of a learning algorithm. The first approach requires the probability of large trees to be limited a priori and the second approach deals with situations where the maximum input height in a concrete learning example is restricted.