We leverage probabilistic models of neural representations to investigate how residual networks fit classes. To this end, we estimate class-conditional density models for representations learned by deep ResNets. We then use these models to characterize distributions of representations across learned classes. Surprisingly, we find that classes in the investigated models are not fitted in a uniform way. On the contrary: we uncover two groups of classes that are fitted with markedly different distributions of representations. These distinct modes of class-fitting are evident only in the deeper layers of the investigated models, indicating that they are not related to low-level image features. We show that the uncovered structure in neural representations correlate with memorization of training examples and adversarial robustness. Finally, we compare class-conditional distributions of neural representations between memorized and typical examples. This allows us to uncover where in the network structure class labels arise for memorized and standard inputs.

Internet of things (IoT) has opened new horizons in connecting all sorts of devices to the internet.However, continuous demand for connectivity increases the cybersecurity risks, rendering IoT devices more prone to cyberattacks.At the same time, rapid advances in Deep Learning (DL)-based algorithms provide state-of-the-art results in many classification tasks, including classification of network traffic or system logs.That said, deep learning algorithms are considered computationally expensive as they require substantial processing and storage capacity.Sadly, IoT devices have limited resources, making renowned DL models hard to implement in this environment.In this paper we present a Residual Neural Network inspired DLbased Intrusion Detection System (IDS) that incorporates weight pruning to make the model more compact in size and resource consumption.Additionally, the proposed system leverages feature selection algorithms to reduce the feature-space size.The model was trained on the NSL-KDD dataset benchmark.Experimental results show that the proposed system is effective, being able to classify network traffic with an F1 score of up to 98.9% before the pruning and an F1 score of up to 97.5% after pruning 90% of network weights.

We investigate neural network representations from a probabilistic perspective. Specifically, we leverage Bayesian nonparametrics to construct models of neural activations in Convolutional Neural Networks (CNNs) and latent representations in Variational Autoencoders (VAEs). This allows us to formulate a tractable complexity measure for distributions of neural activations and to explore global structure of latent spaces learned by VAEs. We use this machinery to uncover how memorization and two common forms of regularization, i.e. dropout and input augmentation, influence representational complexity in CNNs. We demonstrate that networks that can exploit patterns in data learn vastly less complex representations than networks forced to memorize. We also show marked differences between effects of input augmentation and dropout, with the latter strongly depending on network width. Next, we investigate latent representations learned by standard $\beta$-VAEs and Maximum Mean Discrepancy (MMD) $\beta$-VAEs. We show that aggregated posterior in standard VAEs quickly collapses to the diagonal prior when regularization strength increases. MMD-VAEs, on the other hand, learn more complex posterior distributions, even with strong regularization. While this gives a richer sample space, MMD-VAEs do not exhibit independence of latent dimensions. Finally, we leverage our probabilistic models as an effective sampling strategy for latent codes, improving quality of samples in VAEs with rich posteriors.

Recently there has been an increasing interest in telematics solutions for vehicles. These systems provide real-time danger detection, driving style evaluation or crash detection services. The provided information can significantly increase driving safety, help to improve driving style, support rescue operations and enable better-informed insurance pricing. While recently introduced vehicles often provide inbuilt telematics systems, older cars usually lack such functionalities. One option to remedy this issue is to develop telematics solutions that are based on sensors other than the original car equipment. In this paper we present our telematics solution that employs smartphone sensors. This research stems from our previous work on crash detection which we substantially expanded towards a system for automatic profiling of driving style, detection of driving anomalies, and community-based identification of dangerous places on public roads. We describe our approach to the processing of sensor data, driving profile construction, anomaly detection, and interaction with the driver, illustrating these aspects with selected tangible results.

Training deep neural networks (DNNs) on high-dimensional data with no spatial structure poses a major computational problem. It implies a network architecture with a huge input layer, which greatly increases the number of weights, often making the training infeasible. One solution to this problem is to reduce the dimensionality of the input space to a manageable size, and then train a deep network on a representation with fewer dimensions. Here, we focus on performing the dimensionality reduction step by randomly projecting the input data into a lower-dimensional space. Conceptually, this is equivalent to adding a random projection (RP) layer in front of the network. We study two variants of RP layers: one where the weights are fixed, and one where they are fine-tuned during network training. We evaluate the performance of DNNs with input layers constructed using several recently proposed RP schemes. These include: Gaussian, Achlioptas’, Li’s, subsampled randomized Hadamard transform (SRHT) and Count Sketch-based constructions. Our results demonstrate that DNNs with RP layer achieve competitive performance on high-dimensional real-world datasets. In particular, we show that SRHT and Count Sketch-based projections provide the best balance between the projection time and the network performance.

In this work we propose rank densities, a novel density measure for non-metric data. Unlike typical measures employed in density clustering algorithms, rank densities are defined via ranks of patterns in k-nearest neighbor lists. Therefore they depend on the structure of the k-nearest neighbor graph, rather than exact values of the similarity measure. This property makes them robust to noise and easier to use with complicated similarity measures. We propose a simple iterative algorithm to estimate rank densities and demonstrate that it converges to a solution that does not depend on the initial configuration or the optimization step. Finally, we demonstrate an example application of rank densities to a real-world non-metric dataset.

We present disambiguated skip-gram: a neural-probabilistic model for learning multisense distributed representations of words.Disambiguated skip-gram jointly estimates a skip-gram-like context word prediction model and a word sense disambiguation model.Unlike previous probabilistic models for learning multi-sense word embeddings, disambiguated skip-gram is end-to-end differentiable and can be interpreted as a simple feed-forward neural network.We also introduce an effective pruning strategy for the embeddings learned by disambiguated skip-gram.This allows us to control the granularity of representations learned by our model.In experimental evaluation disambiguated skip-gram improves stateof-the are results in several word sense induction benchmarks.

Nowadays, the most common way to model the driver behavior is to create, under some assumptions, a model of common patterns in driver maneuvers. These patterns are often modeled with averaged driver model. While this idea is very simple and intuitive in the context of driver classification by his/her patterns of maneuvers, our previous works demonstrated that assumptions underlying such models are often inaccurate and not applicable in general settings. In fact, it is very hard to express driving patterns with simple models. In this article we present a new way of modeling drivers: we employ Long-Short Term Memory networks to learn driver models from telematics data. In particular, our neural network models learn to predict driving-related signals, such as speed or acceleration, given the evolution of these signals up to the point of prediction. Solving this prediction task allows us to capture the behavioral model of the driver. We tested our models on several drivers, by predicting their future decisions. By learning our models on one driver and then evaluating them on another driver, we demonstrate that LSTM models are a powerful tool for driver profiling and detection of abnormal situations. We also evaluate the influence of data preprocessing on the quality of predictions. In this context we use Kalman filering, which can remove noise from uncertain dynamic measurements, in effect giving the best linearly estimated data.

SUMMARY This work presents an efficient implementation of affinity propagation (AP) on clusters of graphical processing units (GPUs). AP is a state‐of‐the‐art method for finding exemplars in data sets described by similarity matrices. It is typically employed in crisp clustering applications. However, when finding exemplars in ann‐pattern data set with dense, non‐metric similarities, AP performs iterative processing of threen × nfloating point matrices. One of them stores the similarities, and the other two store the values that will ultimately pinpoint the exemplars. For large similarity matrices, AP is therefore computationally expensive. Although matrix operations of AP are well suited for GPUs, its memory footprint limits the size of tasks that can be solved on one unit. We present, however, a decomposition scheme for AP that distributes the calculations over multiple GPUs, with low communication‐to‐computation ratio. Because of this favorable communication pattern, our implementation finds exemplars in large, dense similarity data efficiently, even when GPUs are connected by a slow network. Furthermore, by combining global device memory of multiple GPUs, it can solve problems that would not fit in a single unit. Copyright © 2012 John Wiley & Sons, Ltd.