We propose a new approach for propagating stable probability distributions through neural networks. Our method is based on local linearization, which we show to be an optimal approximation in terms of total variation distance for the ReLU non-linearity. This allows propagating Gaussian and Cauchy input uncertainties through neural networks to quantify their output uncertainties. To demonstrate the utility of propagating distributions, we apply the proposed method to predicting calibrated confidence intervals and selective prediction on out-of-distribution data. The results demonstrate a broad applicability of propagating distributions and show the advantages of our method over other approaches such as moment matching.

We consider the training of the first layer of vision models and notice the clear relationship between pixel values and gradient update magnitudes: the gradients arriving at the weights of a first layer are by definition directly proportional to (normalized) input pixel values. Thus, an image with low contrast has a smaller impact on learning than an image with higher contrast, and a very bright or very dark image has a stronger impact on the weights than an image with moderate brightness. In this work, we propose performing gradient descent on the embeddings produced by the first layer of the model. However, switching to discrete inputs with an embedding layer is not a reasonable option for vision models. Thus, we propose the conceptual procedure of (i) a gradient descent step on first layer activations to construct an activation proposal, and (ii) finding the optimal weights of the first layer, i.e., those weights which minimize the squared distance to the activation proposal. We provide a closed form solution of the procedure and adjust it for robust stochastic training while computing everything efficiently. Empirically, we find that TrAct (Training Activations) speeds up training by factors between 1.25x and 4x while requiring only a small computational overhead. We demonstrate the utility of TrAct with different optimizers for a range of different vision models including convolutional and transformer architectures.

When training neural networks with custom objectives, such as ranking losses and shortest-path losses, a common problem is that they are, per se, non-differentiable. A popular approach is to continuously relax the objectives to provide gradients, enabling learning. However, such differentiable relaxations are often non-convex and can exhibit vanishing and exploding gradients, making them (already in isolation) hard to optimize. Here, the loss function poses the bottleneck when training a deep neural network. We present Newton Losses, a method for improving the performance of existing hard to optimize losses by exploiting their second-order information via their empirical Fisher and Hessian matrices. Instead of training the neural network with second-order techniques, we only utilize the loss function's second-order information to replace it by a Newton Loss, while training the network with gradient descent. This makes our method computationally efficient. We apply Newton Losses to eight differentiable algorithms for sorting and shortest-paths, achieving significant improvements for less-optimized differentiable algorithms, and consistent improvements, even for well-optimized differentiable algorithms.

With the increasing inference cost of machine learning models, there is a growing interest in models with fast and efficient inference. Recently, an approach for learning logic gate networks directly via a differentiable relaxation was proposed. Logic gate networks are faster than conventional neural network approaches because their inference only requires logic gate operators such as NAND, OR, and XOR, which are the underlying building blocks of current hardware and can be efficiently executed. We build on this idea, extending it by deep logic gate tree convolutions, logical OR pooling, and residual initializations. This allows scaling logic gate networks up by over one order of magnitude and utilizing the paradigm of convolution. On CIFAR-10, we achieve an accuracy of 86.29% using only 61 million logic gates, which improves over the SOTA while being 29x smaller.

Despite recent advancements in AI and Data Science, the vast Big Data sources available to medical and health care providers are far from living up to their potential. Addressing the underlying transparency and data protection concerns helps to unleash these advancements in the medical domain and thus benefits research and patient care. Subsequently, we propose a system for trustworthy medical operational AI in this paper. We present our vision to align medical operational AI with regulatory demands of the medical domain. We propose guiding principles to marry data-driven diagnostic recommendations with legal frameworks, clinical protocols, and expert-driven reasoning. Through this research, we aim for AI systems in medicine that not only provide accurate predictions but also empower users to comprehend and trust the underlying decision-making processes.

In this work, we present and study a generalized family of differentiable renderers. We discuss from scratch which components are necessary for differentiable rendering and formalize the requirements for each component. We instantiate our general differentiable renderer, which generalizes existing differentiable renderers like SoftRas and DIB-R, with an array of different smoothing distributions to cover a large spectrum of reasonable settings. We evaluate an array of differentiable renderer instantiations on the popular ShapeNet 3D reconstruction benchmark and analyze the implications of our results. Surprisingly, the simple uniform distribution yields the best overall results when averaged over 13 classes; in general, however, the optimal choice of distribution heavily depends on the task.

The top-k classification accuracy is one of the core metrics in machine learning. Here, k is conventionally a positive integer, such as 1 or 5, leading to top-1 or top-5 training objectives. In this work, we relax this assumption and optimize the model for multiple k simultaneously instead of using a single k. Leveraging recent advances in differentiable sorting and ranking, we propose a family of differentiable top-k cross-entropy classification losses. This allows training while not only considering the top-1 prediction, but also, e.g., the top-2 and top-5 predictions. We evaluate the proposed losses for fine-tuning on state-of-the-art architectures, as well as for training from scratch. We find that relaxing k not only produces better top-5 accuracies, but also leads to top-1 accuracy improvements. When fine-tuning publicly available ImageNet models, we achieve a new state-of-the-art for these models.

Recently, research has increasingly focused on developing efficient neural network architectures. In this work, we explore logic gate networks for machine learning tasks by learning combinations of logic gates. These networks comprise logic gates such as "AND" and "XOR", which allow for very fast execution. The difficulty in learning logic gate networks is that they are conventionally non-differentiable and therefore do not allow training with gradient descent. Thus, to allow for effective training, we propose differentiable logic gate networks, an architecture that combines real-valued logics and a continuously parameterized relaxation of the network. The resulting discretized logic gate networks achieve fast inference speeds, e.g., beyond a million images of MNIST per second on a single CPU core.

Sorting and ranking supervision is a method for training neural networks end-to-end based on ordering constraints. That is, the ground truth order of sets of samples is known, while their absolute values remain unsupervised. For that, we propose differentiable sorting networks by relaxing their pairwise conditional swap operations. To address the problems of vanishing gradients and extensive blurring that arise with larger numbers of layers, we propose mapping activations to regions with moderate gradients. We consider odd-even as well as bitonic sorting networks, which outperform existing relaxations of the sorting operation. We show that bitonic sorting networks can achieve stable training on large input sets of up to 1024 elements.

The integration of algorithmic components into neural architectures has gained increased attention recently, as it allows training neural networks with new forms of supervision such as ordering constraints or silhouettes instead of using ground truth labels. Many approaches in the field focus on the continuous relaxation of a specific task and show promising results in this context. But the focus on single tasks also limits the applicability of the proposed concepts to a narrow range of applications. In this work, we build on those ideas to propose an approach that allows to integrate algorithms into end-to-end trainable neural network architectures based on a general approximation of discrete conditions. To this end, we relax these conditions in control structures such as conditional statements, loops, and indexing, so that resulting algorithms are smoothly differentiable. To obtain meaningful gradients, each relevant variable is perturbed via logistic distributions and the expectation value under this perturbation is approximated. We evaluate the proposed continuous relaxation model on four challenging tasks and show that it can keep up with relaxations specifically designed for each individual task.

A naïve implementation of k -means clustering requires computing for each of the n data points the distance to each of the k cluster centers, which can result in fairly slow execution. However, by storing distance information obtained by earlier computations as well as information about distances between cluster centers, the triangle inequality can be exploited in different ways to reduce the number of needed distance computations, e.g. [ 3 , 4 , 5 , 7 , 11 ]. In this paper I present an improvement of the Exponion method [ 11 ] that generally accelerates the computations. Furthermore, by evaluating several methods on a fairly wide range of artificial data sets, I derive a kind of map, for which data set parameters which method (often) yields the lowest execution times.

For the last decade Deep Reinforcement Learning has undergone exponential development; however, less has been done to integrate linear methods into it. Our Wide and Deep Reinforcement Learning framework provides a tool that combines linear and non-linear methods into one. For practical implementations, our framework can help integrate expert knowledge while improving the performance of existing Deep Reinforcement Learning algorithms. Our research aims to generate a simple practical framework to extend such algorithms. To test this framework we develop an extension of the popular Deep Q-Networks algorithm, which we name Wide Deep Q-Networks. We analyze its performance compared to Deep Q-Networks and Linear Agents, as well as human players. We apply our new algorithm to Berkley’s Pac-Man environment. Our algorithm considerably outperforms Deep Q-Networks’ both in terms of learning speed and ultimate performance showing its potential for boosting existing algorithms.

Index investing has an advantage over active investment strategies, because less frequent trading results in lower expenses, yielding higher long-term returns. Index tracking is a popular investment strategy that attempts to find a portfolio replicating the performance of a collection of investment vehicles. This paper considers index tracking from the perspective of solution space exploration. Three search space heuristics in combination with three portfolio tracking error methods are compared in order to select a tracking portfolio with returns that mimic a benchmark index. Experimental results conducted on real-world datasets show that Widening, a metaheuristic using diverse parallel search paths, finds superior solutions than those found by the reference heuristics. Presented here are the first results using Widening on time-series data.

Our contribution in this paper is threefold. First, we present a framework for characterizing spike synchrony in neuronal spike-train recordings that is based on the identification of spikes (also called electrical impulses or action potentials) with what we call influence maps: real-valued functio ns that describe an influence region around the corresponding spike times within which a continuous and possibly graded notion of synchrony among spikes is defined. Second, we provide a model of synchrony within our framework that is based on a continuous, two-valued (i.e., bivalent) measure of synchrony, aimed at overcoming the drawbacks of time discretization in the bin-based one (the almost exclusively applied model in the field), which we also describe within our framework. Third, in connection with the assessment of synchrony in our continuous model, we provide methodology and algorithms for the identification of frequent parallel episodes in sequences of events (i.e., sets of items, normally required to occur within a certain time span in the sequence). Special attention is given to the notion of frequency of parallel episodes and its computation.

Most known frequent item set mining algorithms work by enumerating candidate item sets and pruning infrequent candidates. An alternative method, which works by intersecting transactions, is much less researched. To the best of our knowledge, there are only two basic algorithms: a cumulative scheme, which is based on a repository with which new transactions are intersected, and the Carpenter algorithm, which enumerates and intersects candidate transaction sets. These approaches yield the set of so-called closed frequent item sets, since any such item set can be represented as the intersection of some subset of the given transactions. In this paper we describe a considerably improved implementation scheme of the cumulative approach, which relies on a prefix tree representation of the already found intersections. In addition, we present an improved way of implementing the Carpenter algorithm. We demonstrate that on specific data sets, which occur particularly often in the area of gene expression analysis, our implementations significantly outperform enumeration approaches to frequent item set mining.

Emerging trends in urban mobility have accelerated the need for effective traffic prediction and management systems. The present paper proposes a novel approach to using continuously streaming moving object trajectories for traffic prediction and management. The approach continuously performs three functions for streams of moving object positions in road networks: 1) management of current evolving trajectories, 2) incremental mining of closed frequent routes, and 3) prediction of near-future locations and densities based on 1) and 2). The approach is empirically evaluated on a large real-world data set of moving object trajectories, originating from a fleet of taxis, illustrating that detailed closed frequent routes can be efficiently discovered and used for prediction.

Chances to detect assembly activity are expected to increase if the spiking activities of large numbers of neurons are recorded simultaneously. Although such massively parallel recordings are now becoming available, methods able to analyze such data for spike correlation are still rare, because it is often infeasible to extend methods developed for smaller data sets due to a combinatorial explosion. By evaluating pattern complexity distributions the existence of correlated groups can be detected, but their member neurons cannot be identified. In this contribution, we present approaches to actually identify the individual neurons involved in assemblies. Our results may complement other methods and also provide the opportunity for a reduction of data sets to the ldquorelevantrdquo neurons, thus allowing us to carry out a refined analysis of the detailed correlation structure due to reduced computation time.

This paper studies the problem of weighting and selecting attributes and principal axes in fuzzy clustering. Its main contribution is a selection method that is not based on simply applying a threshold to computed feature weights, but directly assigns zero weights to features that are not informative enough. This has the important advantage that the clustering result that can be obtained on the selected subspace coincides with the projection (to the selected subspace) of the clustering result that is obtained on the full data space.

In contrast to standard fuzzy clustering, which optimizes a set of prototypes, one for each cluster, this paper studies fuzzy clustering without prototypes. Starting from an objective function that only involves the distances between data points and the membership degrees of the data points to the different clusters, an iterative update rule is derived. The properties of the resulting algorithm are then examined, especially w.r.t. to schemes that focus on a constrained neighborhood for each data point. Corresponding experimental results are reported that demonstrate the merits of this approach.

Approaches based on conditional independence tests are among the most popular methods for learning graphical models from data. Due to the predominance of Bayesian networks in the field, they are usually developed for directed graphs. For possibilistic networks of a certain kind, however, undirected graphs are a more natural basis and thus algorithms for learning undirected graphs are desirable in this area. In this paper I present an algorithm for learning undirected graphical models, which is derived from the well-known Cheng-Bell-Liu algorithm. Its main advantage is the lower number of conditional independence tests that are needed, while it achieves results of comparable quality.

This paper describes a Java-based graphical user interface to a large number of data analysis programs the first author has written in C over the years. In addition, this toolbox is equipped with basic visualization capabilities, like scatter plots and bar charts, but also with specialized visualization modules for decision and regression trees as well as prototype-based classifiers. The architecture is like a toolbox: individual tools refer to the different data analysis methods. All parts of this toolbox (Java as well as C based) are free and open software under the Gnu Lesser (Library) Public License.

Resampling methods are among the best approaches to determine the number of clusters in prototype-based clustering. The core idea is that with the right choice for the number of clusters basically the same cluster structures should be obtained from subsamples of the given data set, while a wrong choice should produce considerably varying cluster structures. In this paper I give an overview how such resampling approaches can be transferred to fuzzy and probabilistic clustering. I study several cluster comparison measures, which can be parameterized with t-norms, and report experiments that provide some guidance which of them may be the best choice.

Recently several papers studied resampling approaches to determine the number of clusters in prototype-based clustering. The core idea underlying these approaches is that with the right choice for the number of clusters basically the same cluster structures should be obtained from subsamples of the given data set, while a wrong choice should produce considerably varying cluster structures. In this paper we investigate whether these approaches can be transferred to fuzzy clustering. It turns out that they are applicable to fuzzy clustering as well, but that not all relative cluster evaluation measures that work for crisp clustering can also be used for fuzzy clustering.

We study an extension of fuzzy learning vector quantization that draws on ideas from the more sophisticated approaches to fuzzy clustering, enabling us to find fuzzy clusters of ellipsoidal shape and differing size with a competitive learning scheme. This approach may be seen as a kind of online fuzzy clustering, which can have advantages w.r.t. the execution time of the clustering algorithm. We demonstrate the usefulness of our approach by applying it to document collections, which are, in general, difficult to cluster due to the high number of dimensions and the special distribution characteristics of the data

In fuzzy clustering soft cluster partitions are formed based on the similarity of data points to the respective cluster prototypes. Similarity is defined in terms of simultaneous closeness regarding all attributes. In some applications the values of many attributes have been measured, but a natural clustering, if it exists, occurs within a (small) subset of attributes. The remaining dimensions can be considered irrelevant. They can obscure an existing grouping and make it harder to discover the cluster structure. In probabilistic fuzzy clustering irrelevant attributes can lead to coincidental cluster centers in the worst case. We study this effect in detail as well as the robustness of different similarity functions and their possible parameterizations against irrelevant input dimensions. Empirical evidence is given for the different properties of the membership functions

Real life transaction data often miss some occurrences of items that are actually present. As a consequence some potentially interesting frequent patterns cannot be discovered, since with exact matching the number of supporting transactions may be smaller than the user-specified minimum. In order to allow approximate matching during the mining process, we propose an approach based on transaction editing. Our recursive algorithm relies on a step by step elimination of items from the transaction database together with a recursive processing of transaction subsets. This algorithm works without complicated data structures and allows us to find fuzzy frequent patterns easily.

Decision trees are a popular form of classification models. It is well known that classical trees lack the ability of modelling vagueness. By connecting fuzzy systems and classical decision trees, we try to achieve classifiers that can model vagueness and are comprehensible. We discuss the core problem of how to compute the information measure used in the induction of fuzzy trees and propose some improvements. In addition, we consider fuzzy rule bases derived from fuzzy decision trees and present some heuristic strategies to prune them. We report the results of experiments in which we compare our approach to other well-known classification methods.

We explore how techniques that were developed to improve the training process of artificial neural networks can be used to speed up fuzzy clustering. The basic idea of our approach is to regard the difference between two consecutive steps of the alternating optimization scheme of fuzzy clustering as providing a gradient, which may be modified in the same way as the gradient of neural network back-propagation is modified in order to improve training. Our experimental results show that some methods actually lead to a considerable acceleration of the clustering process.

Graphical models - especially probabilistic networks like Bayes networks and Markov networks - are very popular to make reasoning in high-dimensional domains feasible. Since constructing them manually can be tedious and time consuming, a large part of recent research has been devoted to learning them from data. However, if the dataset to learn from contains imprecise information in the form of sets of alternatives instead of precise values, this learning task can pose unpleasant problems. In this paper, we survey an approach to cope with these problems, which is not based on probability theory as the more common approaches like, e.g., expectation maximization, but uses the possibility theory as the underlying calculus of a graphical model. We provide semantic foundations of possibilistic graphical models, explain the rationale of possibilistic decomposition as well as the graphical representation of decompositions of possibility distributions and finally discuss the main approaches to learn possibilistic graphical models from data.

We present an algorithm to find fragments in a set of molecules that help to discriminate between different classes of for instance, activity in a drug discovery context. Instead of carrying out a brute-force search, our method generates fragments by embedding them in all appropriate molecules in parallel and prunes the search tree based on a local order of the atoms and bonds, which results in substantially faster search by eliminating the need for frequent, computationally expensive reembeddings and by suppressing redundant search. We prove the usefulness of our algorithm by demonstrating the discovery of activity-related groups of chemical compounds in the well-known National Cancer Institute's HIV-screening dataset.

A major topic of recent research in graphical models has been to develop algorithms to learn them from a dataset of sample cases. However, most of these algorithms do not take into account that learned graphical models may be used for time-critical reasoning tasks and that in this case the time complexity of evidence propagation may have to be restricted, even if this can be achieved only by accepting approximations. In this paper we suggest a simulated annealing approach to learn graphical models with hypertree structure, with which the complexity of the popular join tree evidence propagation method can be controlled at learning time by restricting the size of the cliques of the learned network.

Although probabilistic networks and fuzzy clustering may seem to be disparate areas of research, they can both be seen as generalizations of naive Bayes classifiers. If all descriptive attributes are numeric, naive Bayes classifiers often assume an axis-parallel multidimensional normal distribution for each class. Probabilistic networks remove the requirement that the distributions must be axis-parallel by taking covariances into account where this is necessary. Fuzzy clustering tries to find general or axis-parallel distributions to cluster the data. Although it neglects the classification information, it can be used to improve the result of the above mentioned methods by removing the restriction to only one distribution per classification.