Positron Emission Tomography (PET) neuroimaging is a valuable tool for studying neurodegenerative disorders.Using N = 236 FDG PET scans from healthy individuals and three patient classes, we compare linear and non-linear feature extraction using Principal Component Analysis (PCA), a convolutional autoencoder (CAE) and a variational autoencoder (VAE).We investigate whether non-linear dimensionality reduction improves disease classification performance when used with Generalised Matrix Learning Vector Quantisation (GMLVQ) classifiers trained in the latent space.Although PCA had a smaller reconstruction error between the original and reconstructed images, the features from both AEs achieved higher classification performance, with the VAE showing a slight advantage.The interpretability of the AE-GMLVQ combination was retained by visualising the GMLVQ classification space and the decoded prototypes in voxel space.Even with limited training data, using AE for feature extraction improved classification performance by a significant margin while maintaining interpretability.* R.J. Veen and S.S. Lövdal share first authorship.We thank the Center for IT of the UG for their support and

Developing fair classification models is a crucial aspect of machine learning research. However, unintended distortion in training data - biased data - can lead to discriminatory decisions. In this paper, we developed a workflow for detecting and mitigating bias in data using a shallow, interpretable machine learning models: the Generalized Matrix Learning Vector Quantization. We extent the approach by a relevance-based analysis to identify and reduce bias in the data. Combining similarity metric adaptation and relevance-based analysis, we can develop fair classification models that minimize the influence of bias in the data. Our results demonstrate that this method is effective in reducing bias in classification models and therefore supports fair decision-making.

Generalized Matrix Learning Vector Quantization (GMLVQ) is an inherently interpretable classifier, but the learned distance measure may converge into different competing solutions. Therefore, the resulting trained relevance matrix and its interpretation may differ significantly per training process, diminishing the interpretability benefit, in particular in the presence of small training sets and multiple correlated features. Iterated Relevance Matrix Analysis (IRMA) recursively applies GMLVQ while projecting out all previously found, orthogonal subspaces. Here, we propose a significant extension of the method which combines individual relevance matrices from the application of IRMA into a single, interpretable distance measure. We evaluated the robustness of the combined compared to standard GMLVQ. Moreover, we demonstrate that the associated distance measure can be exploited in the construction of a novel, improved prototype-based classifier by combining the IRMA relevance space with GLVQ (IRMA-GLVQ). Using three open source data sets, we repeatedly drew subsets for model training and evaluated the similarity in between subsets as well as model performance on a holdout test set. Performance metrics were evaluated for various subset sizes. The pairwise differences between the obtained relevance matrices were significantly smaller for IRMA than for GMLVQ for all settings and data sets, and IRMA-GLVQ performed better on holdout test sets for small and medium training set sizes. These findings indicate that IRMA-GLVQ provides a more stable solution with better classification performance for limited training set sizes compared to GMLVQ. • IRMA combines information from multiple competing solutions by recursive application of GMLVQ. • We propose a combined relevance matrix from the individual IRMA iterations. • is more robust to random sampling effects in the training set compared to standard GMVLQ. • The resulting IRMA distance metric gives higher performance on holdout test sets for small and medium sized training sets. • IRMA provides benefits for performance and stable interpretability in the presence of limited training data and correlated features.

Abstract This paper introduces Ordinal Generalized Matrix Learning Vector Quantization (ORGMLVQ), an enhanced version of the GMLVQ algorithm designed for classifying data with an inherent order among classes. ORGMLVQ incorporates ordinal constraints directly into the metric learning process, allowing the model to better capture the progression between categories-an important aspect in applications such as medical diagnostics or risk grading. Through experiments on multiple ordinal regression datasets, as well as standard UCI benchmarks and real-world problems, the proposed method demonstrates significant improvement of MAUC while maintaining the interpretability and prototype-based nature of the original GMLVQ. These results suggest that our method is a strong, interpretable alternative for learning from structured, ordered data.

We present a machine learning approach to the challenging differentiation of hyperkinetic movement disorders, based on accelerometric sensor data.We address the diagnosis of essential tremor and cortical myoclonus as a specific example.Generalized Matrix Relevance Learning Vector Quantization (GMLVQ) systems are applied directly to power spectra obtained from eight sensors recording upper body movements.We find excellent validation performance of the classifiers.Moreover, GMLVQ provides insight into the characteristic patterns of the phenotypes and the importance of particular frequency ranges in the spectra.We demonstrate that the explanatory power of the classifier is further enhanced when integrating information from several tasks per subject.

Using the statistical physics framework, we study the online learning dynamics in a particular case of shallow feed-forward neural networks with ReLU activation.By expanding the activation function in terms of Hermite polynomials we derive analytical results for the evolution of order parameters for any learning rate.Moreover, we compare our results with online gradient descent simulations and show how our method describes the typical learning curves.We also present results on how the learning rate affects the overall behavior of the network and its equilibria, showing different learning regimes and critical values of the learning rate.

We propose a bias detection and mitigating scheme for data in the context of classification tasks based on learning vector quantizers (LVQ) as classifier.For this purpose generalized LVQ endowed with an advanced matrix adaptation scheme is used for bias detection.The bias removal from data is realized applying a nullspace data projection using the adjusted matrix.The usefulness of the approach is demonstrated and illustrated in terms of two real world datasets.* M.K. is

Following the standard statistical mechanics methods we analyze the training by online stochastic gradient descent of two-layer neural networks in a student–teacher scenario. We focus on understanding the role that different activations play, in particular mismatches between the student and the teacher, in these learning scenarios. By expanding the activation functions in the Hermite polynomial basis, we are able to effectively approximate the relevant integrals with much less computational effort than naive numerical integration. Moreover, we also extend the framework to study scenarios of concept drift and weight decay also with arbitrary activation functions. All these extensions comprise relevant advances in the field, allowing us to obtain analytical results for more realistic scenarios.

Applications of interpretable machine learning (ML) techniques on medical datasets facilitate early and fast diagnoses, along with getting deeper insight into the data. Furthermore, the transparency of these models increase trust among application domain experts. Medical datasets face common issues such as heterogeneous measurements, imbalanced classes with limited sample size, and missing data, which hinder the straightforward application of ML techniques. In this paper we present a family of prototype-based (PB) interpretable models which are capable of handling these issues. Moreover we propose a strategy of harnessing the power of ensembles while maintaining the intrinsic interpretability of the PB models, by averaging over the model parameter manifolds. All the models were evaluated on a synthetic (publicly available dataset) in addition to detailed analyses of two real-world medical datasets (one publicly available). The models and strategies we introduce address the challenges of real-world medical data, while remaining computationally inexpensive and transparent. Moreover, they exhibit similar or superior in performance compared to alternative techniques.

We revisit and extend the statistical physics based analysis of layered neural networks trained by online gradient descent.We focus on the influence of the hidden unit activation functions on the typical learning behavior in model scenarios.Expanding activation functions in terms of Hermite polynomials enables us to extend the formalism to the analysis of soft committee machines with arbitrary activation in student-teacher scenarios.This approach requires much lower computational effort than naive numerical integration, which is practically infeasible.Moreover, it now becomes possible to treat mismatched scenarios in which the student activation function differs from the one used in the target rule definition.This makes it possible to study realistic models of machine learning.

Explainable AI models and methods have seen a rise in interest in recent years as a reaction to the widespread use of neural networks and similar black-box models in machine learning.In this project, we combine explainable, prototype-based systems and neural networks in an effort to benefit from both approaches.Specifically, we employ Generalized Matrix Relevance Learning Vector Quantization in combination with autoencoder networks.This allows us to perform automated non-linear feature extraction from high-dimensional inputs before feeding them into LVQ for classification.Moreover, the approach enables the mapping of the low-dimensional representatives and relevances back to the original feature space for visual inspection and interpretation.

In machine learning, data often comes from different sources, but combining them can introduce extraneous variation that affects both generalization and interpretability. For example, we investigate the classification of neurodegenerative diseases using FDG-PET data collected from multiple neuroimaging centers. However, data collected at different centers introduces unwanted variation due to differences in scanners, scanning protocols, and processing methods. To address this issue, we propose a two-step approach to limit the influence of center-dependent variation on the classification of healthy controls and early vs. late-stage Parkinson's disease patients. First, we train a Generalized Matrix Learning Vector Quantization (GMLVQ) model on healthy control data to identify a "relevance space" that distinguishes between centers. Second, we use this space to construct a correction matrix that restricts a second GMLVQ system's training on the diagnostic problem. We evaluate the effectiveness of this approach on the real-world multi-center datasets and simulated artificial dataset. Our results demonstrate that the approach produces machine learning systems with reduced bias - being more specific due to eliminating information related to center differences during the training process - and more informative relevance profiles that can be interpreted by medical experts. This method can be adapted to similar problems outside the neuroimaging domain, as long as an appropriate "relevance space" can be identified to construct the correction matrix.

We introduce and investigate the iterated application of Generalized Matrix Learning Vector Quantizaton for the analysis of feature relevances in classification problems, as well as for the construction of class-discriminative subspaces. The suggested Iterated Relevance Matrix Analysis (IRMA) identifies a linear subspace representing the classification specific information of the considered data sets using Generalized Matrix Learning Vector Quantization (GMLVQ). By iteratively determining a new discriminative subspace while projecting out all previously identified ones, a combined subspace carrying all class-specific information can be found. This facilitates a detailed analysis of feature relevances, and enables improved low-dimensional representations and visualizations of labeled data sets. Additionally, the IRMA-based class-discriminative subspace can be used for dimensionality reduction and the training of robust classifiers with potentially improved performance.

We introduce and investigate the iterated application of Generalized Matrix Relevance Learning for the analysis of feature relevances in classification problems.The suggested Iterated Relevance Matrix Analysis (IRMA), identifies a linear subspace representing the classification specific information of the considered data sets in feature space using Generalized Matrix Learning Vector Quantization.By iteratively determining a new discriminative direction while projecting out all previously identified ones, all features carrying relevant information about the classification can be found, facilitating a detailed analysis of feature relevances.Moreover, IRMA can be used to generate improved low-dimensional representations and visualizations of labeled data sets.

Understanding the influence of activation functions on the learning behaviour of neural networks is of great practical interest.The GELU, being similar to swish and ReLU, is analysed for soft committee machines in the statistical physics framework of off-line learning.We find phase transitions with respect to the relative training set size, which are always continuous.This result rules out the hypothesis that convexity is necessary for continuous phase transitions.Moreover, we show that even a small contribution of a sigmoidal function like erf in combination with GELU leads to a discontinuous transition.

Abstract We present a modelling framework for the investigation of supervised learning in non-stationary environments. Specifically, we model two example types of learning systems: prototype-based learning vector quantization (LVQ) for classification and shallow, layered neural networks for regression tasks. We investigate so-called student–teacher scenarios in which the systems are trained from a stream of high-dimensional, labeled data. Properties of the target task are considered to be non-stationary due to drift processes while the training is performed. Different types of concept drift are studied, which affect the density of example inputs only, the target rule itself, or both. By applying methods from statistical physics, we develop a modelling framework for the mathematical analysis of the training dynamics in non-stationary environments. Our results show that standard LVQ algorithms are already suitable for the training in non-stationary environments to a certain extent. However, the application of weight decay as an explicit mechanism of forgetting does not improve the performance under the considered drift processes. Furthermore, we investigate gradient-based training of layered neural networks with sigmoidal activation functions and compare with the use of rectified linear units. Our findings show that the sensitivity to concept drift and the effectiveness of weight decay differs significantly between the two types of activation function.

The sklvq package is an open-source Python implementation of a set of learning vector quantization (LVQ) algorithms. In addition to providing the core functionality for the GLVQ, GMLVQ, and LGMLVQ algorithms, sklvq is distinctive by putting emphasis on its modular and customizable design. Not only resulting in a feature-rich implementation for users but enabling easy extensions of the algorithms for researchers. The theory behind this design is described in this paper. To facilitate adoptions and inspire future contributions, sklvq is publicly available on Github (under the BSD license) and can be installed through the Python package index (PyPI). Next to being well-covered by automated testing to ensure code quality, it is accompanied by detailed online documentation. The documentation covers usage examples and provides an in-depth API including theory and scientific references.

We present our initial proof of concept study towards the development of a low-cost 3-D printed smartphone add-on spectrometer. The study aimed at developing a cheap technology (less than 5 USD) to be used for detection of crop diseases in the field using spectrometry. Previously, we experimented with the problem of disease diagnosis using an off-the-shelf and expensive spectrometer (approximately 1000 USD). However, in real world practice, this off-the-shelf device can not be used by typical users (smallholder farmers). Therefore, the study presents a tool that is cheap and user friendly. We present preliminary results and identify requirements for a future version aiming at an accurate diagnostic technology to be used in the field before disease symptoms are visibly seen by the naked eye. Evaluation shows performance of the tool is better than random however below performance of an industry grade spectrometer.

Domain-specific proximity measures, like divergence measures in signal processing or alignment scores in bioinformatics, often lead to non-metric, indefinite similarities or dissimilarities. However, many classical learning algorithms like kernel machines assume metric properties and struggle with such metric violations. For example, the classical support vector machine is no longer able to converge to an optimum. One possible direction to solve the indefiniteness problem is to transform the non-metric (dis-)similarity data into positive (semi-)definite matrices. For this purpose, many approaches have been proposed that adapt the eigenspectrum of the given data such that positive definiteness is ensured. Unfortunately, most of these approaches modify the eigenspectrum in such a strong manner that valuable information is removed or noise is added to the data. In particular, the shift operation has attracted a lot of interest in the past few years despite its frequently reoccurring disadvantages. In this work, we propose a modified advanced shift correction method that enables the preservation of the eigenspectrum structure of the data by means of a low-rank approximated nullspace correction. We compare our advanced shift to classical eigenvalue corrections like eigenvalue clipping, flipping, squaring, and shifting on several benchmark data. The impact of a low-rank approximation on the data’s eigenspectrum is analyzed.

Abstract We present a framework for distance-based classification of functional data. We consider the analysis of labeled spectral data and time series by means of generalized matrix relevance learning vector quantization (GMLVQ) as an example. To take advantage of the functional nature, a functional expansion of the input data is considered. Instead of using a predefined set of basis functions for the expansion, a more flexible scheme of an adaptive functional basis is employed. GMLVQ is applied on the resulting functional parameters to solve the classification task. For comparison of the classification, a GMLVQ system is also applied to the raw input data, as well as on data expanded by a different predefined functional basis. Computer experiments show that the methods offer potential to improve classification performance significantly. Furthermore, the analysis of the adapted set of basis functions give further insights into the data structure and yields an option for a drastic reduction of dimensionality.

Abstract In this contribution, we consider the classification of time series and similar functional data which can be represented in complex Fourier and wavelet coefficient space. We apply versions of learning vector quantization (LVQ) which are suitable for complex-valued data, based on the so-called Wirtinger calculus. It allows for the formulation of gradient-based update rules in the framework of cost-function-based generalized matrix relevance LVQ (GMLVQ). Alternatively, we consider the concatenation of real and imaginary parts of Fourier coefficients in a real-valued feature vector and the classification of time-domain representations by means of conventional GMLVQ. In addition, we consider the application of the method in combination with wavelet-space features to heartbeat classification.

We present a machine learning analysis of five labelled galaxy catalogues from the Galaxy And Mass Assembly (GAMA): The SersicCatVIKING and SersicCatUKIDSS catalogues containing morphological features, the GaussFitSimple catalogue containing spectroscopic features, the MagPhys catalogue including physical parameters for galaxies, and the Lambdar catalogue, which contains photometric measurements. Extending work previously presented at the ESANN 2018 conference – in an analysis based on Generalized Relevance Matrix Learning Vector Quantization and Random Forests – we find that neither the data from the individual catalogues nor a combined dataset based on all 5 catalogues fully supports the visual-inspection-based galaxy classification scheme employed to categorise the galaxies. In particular, only one class, the Little Blue Spheroids , is consistently separable from the other classes. To aid further insight into the nature of the employed visual-based classification scheme with respect to physical and morphological features, we present the galaxy parameters that are discriminative for the achieved class distinctions.

In this work we explore the relevance of markers that are used for the early detection of fetal chromosomal abnormalities. For medical applications, it is important to optimize the number of used markers with respect to the number of necessary clinical examinations. We use the Generalized Matrix Learning Vector Quantization (GMLVQ) method to identify the most relevant markers from a set of 18 clinical examinations. We cross-validated our results using ten different training and test sets and we repeated our experiments using different parameters of GMLVQ. We identified the seven most relevant markers and we found that with these seven markers we obtain results that are comparable with the results that can be achieved with the full set of 18 markers. The results are in line with previous work that is found in the literature.

Using mRNA-Seq and clinical data for 469 clear cell Renal Cell Carcinoma (ccRCC) samples from The Cancer Genome Atlas (TCGA), we develop a protocol to identify patients likely to have early recurrence of their disease. We first split the data into two sets, with 380 samples in the training set and 89 samples in the test set. Using the training set, we identify genes whose outlier status (high or low mRNA expression) is predictive of recurrence, based on Kaplan-Meier recurrence free survival log-rank p-value. We find a significant overlap among genes identified as predictive biomarkers in Reads per Kilobase Million (RPKM) normalized data and Raw Reads mRNA-Seq data. Using 80 consensus genes predictive in both RPKM and Raw Reads data, we define an outlier-based risk score R to stratify patients into two groups, a high-risk (early recurrence) group (R2). The KM recurrence curve using this stratification shows excellent separation in training and test sets. Restricting the analysis to patients who had recurrence within two years (109 cases) and those who had no recurrence in five years (107 cases) we find that the risk predictor achieves ca. 80 percent sensitivity and specificity. The 80 genes identified by the outlier analysis were used to develop a more intuitive classifier based on Generalized Matrix Learning Vector Quantization (GMLVQ). This method stratifies samples into risk classes based on defining prototypes in feature space and an appropriate distance metric. GMLVQ identified a subset of 12 genes that have high accuracy in predicting recurrence, which suggests that an assay with a small number of genes might be able to predict recurrence in ccRCC.

We present a theoretical analysis of Learning Vector Quantization (LVQ) with adaptive distance measures. Specifically, we consider generalized Euclidean distances which are parameterized in terms of a quadratic matrix of adaptive relevance parameters. Winner-takes-all prescriptions based on the heuristic LVQ1 are in the center of our interest. We derive and study stationarity conditions and show, among other results, that stationary prototypes can be written as linear combinations of the training data apart from irrelevant contributions in the null-space of the relevance matrix. The investigation of the metrics updates reveals that relevance matrices become singular with only one or very few non-zero eigenvalues. Implications of this property are discussed and, furthermore, the effect of preventing singularity by introducing an appropriate penalty term is studied. Theoretical findings are confirmed in terms of illustrative example data sets.

MOTIVATION: Animal models are widely used in biomedical research for reasons ranging from practical to ethical. An important issue is whether rodent models are predictive of human biology. This has been addressed recently in the framework of a series of challenges designed by the systems biology verification for Industrial Methodology for Process Verification in Research (sbv IMPROVER) initiative. In particular, one of the sub-challenges was devoted to the prediction of protein phosphorylation responses in human bronchial epithelial cells, exposed to a number of different chemical stimuli, given the responses in rat bronchial epithelial cells. Participating teams were asked to make inter-species predictions on the basis of available training examples, comprising transcriptomics and phosphoproteomics data. RESULTS: Here, the two best performing teams present their data-driven approaches and computational methods. In addition, post hoc analyses of the datasets and challenge results were performed by the participants and challenge organizers. The challenge outcome indicates that successful prediction of protein phosphorylation status in human based on rat phosphorylation levels is feasible. However, within the limitations of the computational tools used, the inclusion of gene expression data does not improve the prediction quality. The post hoc analysis of time-specific measurements sheds light on the signaling pathways in both species. AVAILABILITY AND IMPLEMENTATION: A detailed description of the dataset, challenge design and outcome is available at www.sbvimprover.com. The code used by team IGB is provided under http://github.com/uci-igb/improver2013. Implementations of the algorithms applied by team AMG are available at http://bhanot.biomaps.rutgers.edu/wiki/AMG-sc2-code.zip. CONTACT: [email protected].

MOTIVATION: Using gene expression to infer changes in protein phosphorylation levels induced in cells by various stimuli is an outstanding problem. The intra-species protein phosphorylation challenge organized by the IMPROVER consortium provided the framework to identify the best approaches to address this issue. RESULTS: Rat lung epithelial cells were treated with 52 stimuli, and gene expression and phosphorylation levels were measured. Competing teams used gene expression data from 26 stimuli to develop protein phosphorylation prediction models and were ranked based on prediction performance for the remaining 26 stimuli. Three teams were tied in first place in this challenge achieving a balanced accuracy of about 70%, indicating that gene expression is only moderately predictive of protein phosphorylation. In spite of the similar performance, the approaches used by these three teams, described in detail in this article, were different, with the average number of predictor genes per phosphoprotein used by the teams ranging from 3 to 124. However, a significant overlap of gene signatures between teams was observed for the majority of the proteins considered, while Kyoto Encyclopedia of Genes and Genomes (KEGG) pathways were enriched in the union of the predictor genes of the three teams for multiple proteins. AVAILABILITY AND IMPLEMENTATION: Gene expression and protein phosphorylation data are available from ArrayExpress (E-MTAB-2091). Software implementation of the approach of Teams 49 and 75 are available at http://bioinformaticsprb.med.wayne.edu and http://people.cs.clemson.edu/∼luofeng/sbv.rar, respectively. CONTACT: [email protected] or [email protected] or [email protected] SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

MOTIVATION: Translating findings in rodent models to human models has been a cornerstone of modern biology and drug development. However, in many cases, a naive 'extrapolation' between the two species has not succeeded. As a result, clinical trials of new drugs sometimes fail even after considerable success in the mouse or rat stage of development. In addition to in vitro studies, inter-species translation requires analytical tools that can predict the enriched gene sets in human cells under various stimuli from corresponding measurements in animals. Such tools can improve our understanding of the underlying biology and optimize the allocation of resources for drug development. RESULTS: We developed an algorithm to predict differential gene set enrichment as part of the sbv IMPROVER (systems biology verification in Industrial Methodology for Process Verification in Research) Species Translation Challenge, which focused on phosphoproteomic and transcriptomic measurements of normal human bronchial epithelial (NHBE) primary cells under various stimuli and corresponding measurements in rat (NRBE) primary cells. We find that gene sets exhibit a higher inter-species correlation compared with individual genes, and are potentially more suited for direct prediction. Furthermore, in contrast to a similar cross-species response in protein phosphorylation states 5 and 25 min after exposure to stimuli, gene set enrichment 6 h after exposure is significantly different in NHBE cells compared with NRBE cells. In spite of this difference, we were able to develop a robust algorithm to predict gene set activation in NHBE with high accuracy using simple analytical methods. AVAILABILITY AND IMPLEMENTATION: Implementation of all algorithms is available as source code (in Matlab) at http://bhanot.biomaps.rutgers.edu/wiki/codes_SC3_Predicting_GeneSets.zip, along with the relevant data used in the analysis. Gene sets, gene expression and protein phosphorylation data are available on request. CONTACT: [email protected].

Transverse Flux Permanent Magnet Motor (TFPMM) has received extensive attention in the field of electric vehicles. The magnetic circuit of TFPMM is three-dimensional and non-linear, which leads to the high nonlinearity of electromagnetic torque. Although the three-dimensional finite element method (3DFEM) could be used to estimate torque, it is very time-consuming. Instead of it, this study adopts a kind of machine learning method—Gaussian process regression (GPR). For the hyper-parameters optimization of GPR, most previous studies used single-objective algorithms for improving the regression accuracy. However, GPR is an algorithm of probability prediction and it could not guarantee to have the satisfactory confidence interval characteristic simultaneously while the regression precision achieves optimal. Therefore this paper proposes a variable parameters fuzzy dominance genetic algorithm (VPFDGA) which is suitable for the multi-objective optimization, including the optimization of regression precision, confidence interval reliability, confidence interval width and skill score. By combining GPR with VPFDGA, the electromagnetic torque of a building-block transverse flux permanent magnet motor (B-TFPMM) is estimated by VPFDGA-GPR (GPR based on variable parameters fuzzy dominance genetic algorithm). Besides, two other GPRs based multi-objective optimization, three GPRs based on single-objective optimization and a GPR based on weighted sum method that is the classic multi-objective optimization algorithm are all implemented to compare with VPFDGA-GPR. The results of comparison show that VPFDGA-GPR has the better performances including the higher regression precision, more powerful ability of probability prediction, higher stability, less convergence time and so on.

Linear data transformations constitute essential operations in various machine learning algorithms, ranging from linear regression up to adaptive metric transformation. Often, linear scalings are not only used to improve the model accuracy, rather feature coefficients as provided by the mapping are interpreted as an indicator for the relevance of the feature for the task at hand. This principle, however, can be misleading in particular for high-dimensional or correlated features, since it easily marks irrelevant features as relevant or vice versa. In this contribution, we propose a mathematical formalisation of the minimum and maximum feature relevance for a given linear transformation which can efficiently be solved by means of linear programming. We evaluate the method in several benchmarks, where it becomes apparent that the minimum and maximum relevance closely resembles what is often referred to as weak and strong relevance of the features; hence unlike the mere scaling provided by the linear mapping, it ensures valid interpretability.

Linear data transformations are essential operations in many machine learning algorithms, helping to make such models more flexible or to emphasize certain data directions. In particular for high dimensional data sets linear transformations are not necessarily uniquely determined, though, and alternative parameterizations exist which do not change the mapping of the training data. Thus, regularization is required to make the model robust to noise and more interpretable for the user. In this contribution, we characterize the group of transformations which leave a linear mapping invariant for a given finite data set, and we discuss the consequences on the interpretability of the models. We propose an intuitive regularization mechanism to avoid problems in under-determined configurations, and we test the approach in two machine learning models.

In the present paper we investigate the application of differentiable kernel for generalized matrix learning vector quantization as an alternative kernel-based classifier, which additionally provides classification dependent data visualization. We show that the concept of differentiable kernels allows a prototype description in the data space but equipped with the kernel metric. Moreover, using the visualization properties of the original matrix learning vector quantization we are able to optimize the class visualization by inherent visualization mapping learning also in this new kernel-metric data space.

We suggest and investigate the use of Generalized Matrix Relevance Learning (GMLVQ) in the context of discriminative visualization. This prototype-based, supervised learning scheme parameterizes an adaptive distance measure in terms of a matrix of relevance factors. By means of a few benchmark problems, we demonstrate that the training process yields low rank matrices which can be used efficiently for the discriminative visualization of labeled data. Comparison with well known standard methods illustrate the flexibility and discriminative power of the novel approach. The mathematical analysis of GMLVQ shows that the corresponding stationarity condition can be formulated as an eigenvalue problem with one or several strongly dominating eigenvectors. We also study the inclusion of a penalty term which enforces non-singularity of the relevance matrix and can be used to control the role of higher order eigenvalues, efficiently.

One aspect of self-organizing systems is their desired ability to be self-learning, i.e., to be able to adapt dynamically to conditions in their environment. This quality is awkward especially if it comes to applications in security or safety-sensitive areas. Here a step towards more trustful systems could be taken by providing transparency of the processes of a system. An important means of giving feedback to an operator is the visualization of the internal processes of a system. In this position paper we address the problem of visualizing dynamic processes especially in self-learning systems. We take an existing self-learning system from the field of computer vision as an example from which we derive questions of general interest such as possible options to visualize the flow of information in a dynamic learning system or the visualization of symbolic data. As a side effect the visualization of learning processes may provide a better understanding of underlying principles of learning in general, i.e, also in biological systems. That may also facilitate improved designs of future self-learning systems.

In recent years, a wealth of dimension-reduction techniques for data visualization and preprocessing has been established. Nonparametric methods require additional effort for out-of-sample extensions, because they provide only a mapping of a given finite set of points. In this letter, we propose a general view on nonparametric dimension reduction based on the concept of cost functions and properties of the data. Based on this general principle, we transfer nonparametric dimension reduction to explicit mappings of the data manifold such that direct out-of-sample extensions become possible. Furthermore, this concept offers the possibility of investigating the generalization ability of data visualization to new data points. We demonstrate the approach based on a simple global linear mapping, as well as prototype-based local linear mappings. In addition, we can bias the functional form according to given auxiliary information. This leads to explicit supervised visualization mappings with discriminative properties comparable to state-of-the-art approaches.

A wealth of powerful dimensionality reduction methods has been established which can be used for data visualization and preprocessing. These are accompanied by formal evaluation schemes, which allow a quantitative evaluation along general principles and which even lead to further visualization schemes based on these objectives. Most methods, however, provide a mapping of a priorly given finite set of points only, requiring additional steps for out-of-sample extensions. We propose a general view on dimensionality reduction based on the concept of cost functions, and, based on this general principle, extend dimensionality reduction to explicit mappings of the data manifold. This offers simple out-of-sample extensions. Further, it opens a way towards a theory of data visualization taking the perspective of its generalization ability to new data points. We demonstrate the approach based on a simple global linear mapping as well as prototype-based local linear mappings.

We discuss the use of divergences in dissimilarity-based classification. Divergences can be employed whenever vectorial data consists of non-negative, potentially normalized features. This is, for instance, the case in spectral data or histograms. In particular, we introduce and study divergence based learning vector quantization (DLVQ). We derive cost function based DLVQ schemes for the family of γ‐divergences which includes the well-known Kullback–Leibler divergence and the so-called Cauchy–Schwarz divergence as special cases. The corresponding training schemes are applied to two different real world data sets. The first one, a benchmark data set (Wisconsin Breast Cancer) is available in the public domain. In the second problem, color histograms of leaf images are used to detect the presence of cassava mosaic disease in cassava plants. We compare the use of standard Euclidean distances with DLVQ for different parameter settings. We show that DLVQ can yield superior classification accuracies and Receiver Operating Characteristics.

We present two approaches to extend Robust Soft Learning Vector Quantization (RSLVQ). This algorithm for nearest prototype classification is derived from an explicit cost function and follows the dynamics of a stochastic gradient ascent. The RSLVQ cost function is defined in terms of a likelihood ratio and involves a hyperparameter which is kept constant during training. We propose to adapt the hyperparameter in the training phase based on the gradient information . Besides, we propose to base the classifier's decision on the value of the likelihood ratio instead of using the distance based classification approach. Experiments on artificial and real life data show that the hyperparameter crucially influences the performance of RSLVQ. However, it is not possible to estimate the best value from the data prior to learning. We show that the proposed variant of RSLVQ is very robust with respect to the initial value of the hyperparameter. The classification approach based on the likelihood ratio turns out to be superior to distance based classification, if local hyperparameters are adapted for each prototype.

A variety of modifications have been employed to learning vector quantization (LVQ) algorithms using either crisp or soft windows for selection of data. Although these schemes have been shown in practice to improve performance, a theoretical study on the influence of windows has so far been limited. Here we rigorously analyze the influence of windows in a controlled environment of gaussian mixtures in high dimensions. Concepts from statistical physics and the theory of online learning allow an exact description of the training dynamics, yielding typical learning curves, convergence properties, and achievable generalization abilities. We compare the performance and demonstrate the advantages of various algorithms, including LVQ 2.1, generalized LVQ (GLVQ), Learning from Mistakes (LFM) and Robust Soft LVQ (RSLVQ). We find that the selection of the window parameter highly influences the learning curves but not, surprisingly, the asymptotic performances of LVQ 2.1 and RSLVQ. Although the prototypes of LVQ 2.1 exhibit divergent behavior, the resulting decision boundary coincides with the optimal decision boundary, thus yielding optimal generalization ability.

In this paper, we present a regularization technique to extend recently proposed matrix learning schemes in learning vector quantization (LVQ). These learning algorithms extend the concept of adaptive distance measures in LVQ to the use of relevance matrices. In general, metric learning can display a tendency towards oversimplification in the course of training. An overly pronounced elimination of dimensions in feature space can have negative effects on the performance and may lead to instabilities in the training. We focus on matrix learning in generalized LVQ (GLVQ). Extending the cost function by an appropriate regularization term prevents the unfavorable behavior and can help to improve the generalization ability. The approach is first tested and illustrated in terms of artificial model data. Furthermore, we apply the scheme to benchmark classification data sets from the UCI Repository of Machine Learning. We demonstrate the usefulness of regularization also in the case of rank limited relevance matrices, i.e., matrix learning with an implicit, low-dimensional representation of the data.

Discriminative vector quantization schemes such as learning vector quantization (LVQ) and extensions thereof offer efficient and intuitive classifiers based on the representation of classes by prototypes. The original methods, however, rely on the Euclidean distance corresponding to the assumption that the data can be represented by isotropic clusters. For this reason, extensions of the methods to more general metric structures have been proposed, such as relevance adaptation in generalized LVQ (GLVQ) and matrix learning in GLVQ. In these approaches, metric parameters are learned based on the given classification task such that a data-driven distance measure is found. In this letter, we consider full matrix adaptation in advanced LVQ schemes. In particular, we introduce matrix learning to a recent statistical formalization of LVQ, robust soft LVQ, and we compare the results on several artificial and real-life data sets to matrix learning in GLVQ, a derivation of LVQ-like learning based on a (heuristic) cost function. In all cases, matrix adaptation allows a significant improvement of the classification accuracy. Interestingly, however, the principled behavior of the models with respect to prototype locations and extracted matrix dimensions shows several characteristic differences depending on the data sets.

We propose a new matrix learning scheme to extend relevance learning vector quantization (RLVQ), an efficient prototype-based classification algorithm, toward a general adaptive metric. By introducing a full matrix of relevance factors in the distance measure, correlations between different features and their importance for the classification scheme can be taken into account and automated, and general metric adaptation takes place during training. In comparison to the weighted Euclidean metric used in RLVQ and its variations, a full matrix is more powerful to represent the internal structure of the data appropriately. Large margin generalization bounds can be transferred to this case, leading to bounds that are independent of the input dimensionality. This also holds for local metrics attached to each prototype, which corresponds to piecewise quadratic decision boundaries. The algorithm is tested in comparison to alternative learning vector quantization schemes using an artificial data set, a benchmark multiclass problem from the UCI repository, and a problem from bioinformatics, the recognition of splice sites for C. elegans.

Learning vector quantization (LVQ) constitutes a powerful and intuitive method for adaptive nearest prototype classification. However, original LVQ has been introduced based on heuristics and numerous modifications exist to achieve better convergence and stability. Recently, a mathematical foundation by means of a cost function has been proposed which, as a limiting case, yields a learning rule similar to classical LVQ2.1. It also motivates a modification which shows better stability. However, the exact dynamics as well as the generalization ability of many LVQ algorithms have not been thoroughly investigated so far. Using concepts from statistical physics and the theory of on-line learning, we present a mathematical framework to analyse the performance of different LVQ algorithms in a typical scenario in terms of their dynamics, sensitivity to initial conditions, and generalization ability. Significant differences in the algorithmic stability and generalization ability can be found already for slightly different variants of LVQ. We study five LVQ algorithms in detail: Kohonen's original LVQ1, unsupervised vector quantization (VQ), a mixture of VQ and LVQ, LVQ2.1, and a variant of LVQ which is based on a cost function. Surprisingly, basic LVQ1 shows very good performance in terms of stability, asymptotic generalization ability, and robustness to initializations and model parameters which, in many cases, is superior to recent alternative proposals.