We take a deep look into the behaviour of selfattention heads in the transformer architecture.In light of recent work discouraging the use of attention distributions for explaining a model's behaviour, we show that attention distributions can nevertheless provide insights into the local behaviour of attention heads.This way, we propose a distinction between local patterns revealed by attention and global patterns that refer back to the input, and analyze BERT from both angles.We use gradient attribution to analyze how the output of an attention head depends on the input tokens, effectively extending the local attention-based analysis to account for the mixing of information throughout the transformer layers.We find that there is a significant mismatch between attention and attribution distributions, caused by the mixing of context inside the model.We quantify this discrepancy and observe that interestingly, there are some patterns that persist across all layers despite the mixing.

In this work we provide new insights into the transformer architecture, and in particular, its best-known variant, BERT. First, we propose a method to measure the degree of non-linearity of different elements of transformers. Next, we focus our investigation on the feed-forward networks (FFN) inside transformers, which contain two thirds of the model parameters and have so far not received much attention. We find that FFNs are an inefficient yet important architectural element and that they cannot simply be replaced by attention blocks without a degradation in performance. Moreover, we study the interactions between layers in BERT and show that, while the layers exhibit some hierarchical structure, they extract features in a fuzzy manner. Our results suggest that BERT has an inductive bias towards layer commutativity, which we find is mainly due to the skip connections. This provides a justification for the strong performance of recurrent and weight-shared transformer models.

Human activity recognition from raw sensor data has enabled modern wearable devices to track and analyze everyday activities. However, when used in real world conditions, the performance of off-the-shelf devices is often insufficient. This paper tackles the problem of swimming style recognition and lap counting using sensor data from a single smartwatch. In total 17 hours of this data was collected from 40 swimmers of diverse backgrounds. The data was then used to train a convolutional neural network to recognize the four main swimming styles, transition periods and lap turns. Our method achieves an F1 score of 97.4% for style recognition and 99.2% for counting laps. To the best of our knowledge, these results are the first to enable accurate automatic swimming recognition in a realistic and completely uncontrolled environment.

In this paper we study the problem of monaural music source separation, where a piece of music is to be separated into its main constituent sources. We propose a simple yet effective deep neural network architecture based on a ResNet autoencoder. We investigate several data augmentation and post-processing methods to improve the separation results and outperform various state of the art monaural source separation methods on the DSD100 and MUSDB18 datasets. Our results suggest that in order to further push the state of the art in monaural music source separation we need more data, better data augmentation methods, as well as more effective post-processing methods; and not necessarily ever more complex neural network architectures.

The ability to use a 2D map to navigate a complex 3D environment is quite remarkable, and even difficult for many humans. Localization and navigation is also an important problem in domains such as robotics, and has recently become a focus of the deep reinforcement learning community. In this paper we teach a reinforcement learning agent to read a map in order to find the shortest way out of a random maze it has never seen before. Our system combines several state-of-the-art methods such as A3C and incorporates novel elements such as a recurrent localization cell. Our agent learns to localize itself based on 3D first person images and an approximate orientation angle. The agent generalizes well to bigger mazes, showing that it learned useful localization and navigation capabilities.

In this paper we investigate how to measure real world performance of Android devices using app start durations. To this end we collect ground truth app start times using an automated mechanical setup. The ground truth start times are highly correlated with the outputs from Android's ActivityManager, which we then use to obtain app start times during normal use on a range of rooted devices. We then predict app start times with supervised learning to detect if device performance has changed over time. We show that training data can be gathered on a small set of rooted devices and then applied to other, non rooted devices. We also present an unsupervised method that can track the evolution of the system performance without requiring root access at all.

Learning in sparse reward settings remains a challenge in Reinforcement Learning, which is often addressed by using intrinsic rewards. One promising strategy is inspired by human curiosity, requiring the agent to learn to predict the future. In this paper a curiosity-driven agent is extended to use these predictions directly for training. To achieve this, the agent predicts the value function of the next state at any point in time. Subsequently, the consistency of this prediction with the current value function is measured, which is then used as a regularization term in the loss function of the algorithm. Experiments were made on grid-world environments as well as on a 3D navigation task, both with sparse rewards. In the first case the extended agent is able to learn significantly faster than the baselines.

Deep generative models such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) have recently been applied to style and domain transfer for images, and in the case of VAEs, music. GAN-based models employing several generators and some form of cycle consistency loss have been among the most successful for image domain transfer. In this paper we apply such a model to symbolic music and show the feasibility of our approach for music genre transfer. Evaluations using separate genre classifiers show that the style transfer works well. In order to improve the fidelity of the transformed music, we add additional discriminators that cause the generators to keep the structure of the original music mostly intact, while still achieving strong genre transfer. Visual and audible results further show the potential of our approach. To the best of our knowledge, this paper represents the first application of GANs to symbolic music domain transfer.

We propose a novel approach for the generation of polyphonic music based on LSTMs. We generate music in two steps. First, a chord LSTM predicts a chord progression based on a chord embedding. A second LSTM then generates polyphonic music from the predicted chord progression. The generated music sounds pleasing and harmonic, with only few dissonant notes. It has clear long-term structure that is similar to what a musician would play during a jam session. We show that our approach is sensible from a music theory perspective by evaluating the learned chord embeddings. Surprisingly, our simple model managed to extract the circle of fifths, an important tool in music theory, from the dataset.