Abstract Large Language Models (LLMs) exhibit In-Context Learning (ICL), which enables the model to perform new tasks conditioning only on the examples provided in the context without updating the model’s weights. While ICL offers fast adaptation across natural language tasks and domains, its emergence is less straightforward for modalities beyond text. In this work, we systematically uncover properties present in LLMs that support the emergence of ICL for autoregressive models and various modalities by promoting the learning of the mechanisms needed for ICL. We identify exact token repetitions in the training data sequences as an important factor for ICL. Such repetitions further improve stability and reduce transiency in ICL performance. We analyse in detail the training dynamics of such data sequences and explain how token repetitions enhance the ICL learning mechanisms. Moreover, we emphasise the importance of the training task difficulty for the emergence of ICL. Finally, by applying our novel insights on ICL emergence, we unlock ICL capabilities across various visual datasets used for few-shot classification, and confirm the generalisability of our insights to much harder real-world examples of large-scale object classification, and a more challenging EEG classification task. Code is available at https://github.com/jelenab98/unlocking_icl

Deep generative networks trained via maximum likelihood on a natural image dataset like CIFAR10 often assign high likelihoods to images from datasets with different objects (e.g., SVHN). We refine previous investigations of this failure at anomaly detection for invertible generative networks and provide a clear explanation of it as a combination of model bias and domain prior: Convolutional networks learn similar low-level feature distributions when trained on any natural image dataset and these low-level features dominate the likelihood. Hence, when the discriminative features between inliers and outliers are on a high-level, e.g., object shapes, anomaly detection becomes particularly challenging. To remove the negative impact of model bias and domain prior on detecting high-level differences, we propose two methods, first, using the log likelihood ratios of two identical models, one trained on the in-distribution data (e.g., CIFAR10) and the other one on a more general distribution of images (e.g., 80 Million Tiny Images). We also derive a novel outlier loss for the in-distribution network on samples from the more general distribution to further improve the performance. Secondly, using a multi-scale model like Glow, we show that low-level features are mainly captured at early scales. Therefore, using only the likelihood contribution of the final scale performs remarkably well for detecting high-level feature differences of the out-of-distribution and the in-distribution. This method is especially useful if one does not have access to a suitable general distribution. Overall, our methods achieve strong anomaly detection performance in the unsupervised setting, and only slightly underperform state-of-the-art classifier-based methods in the supervised setting. Code can be found at https://github.com/boschresearch/hierarchicalanomalydetection.

Fully immersive virtual reality (VR) has the potential to improve neurosurgical planning. However, there is a lack of research tools for this area. We present a research framework for VR neurosurgery based on open-source tools. We showcase the potential of such a framework using clinical data of two patients and research data of one subject. As first step toward practical evaluations, certified neurosurgeons positively assessed the VR visualizations and interactions using head-mounted displays. Methods and findings described in our study thus provide a foundation for research and development of versatile and user-friendly VR tools for improving neurosurgical planning and training.

Biometric measures such as the electroencephalogram (EEG) promise to become viable alternatives to subjective questionnaire ratings for the evaluation of psychophysical effects associated with Virtual Reality (VR) systems, as they provide objective and continuous measurements without breaking the exposure. The extent to which the EEG signal can be disturbed by the presence of VR systems, however, has been barely investigated. This study outlines how to evaluate the compatibility of a given EEG-VR setup on the example of two commercial head-mounted displays (HMDs), the Oculus Rift and the HTC Vive Pro. We use a novel experimental protocol to compare the spectral composition between conditions with and without an HMD present during an eyes-open vs. eyes-closed task. We found general artifacts at the line hum of 50 Hz, and additional HMD refresh rate artifacts (90 Hz) for the Oculus rift exclusively. Frequency components typically most interesting to non-invasive EEG research and applications , however, remained largely unaffected. We observed similar topographies of visually-induced modulation of alpha band power for both HMD conditions in all subjects. Hence, the study introduces a necessary validation test for HMDs in combination with EEG and further promotes EEG as a potential biometric measurement method for psychophysical effects in VR systems.

Predicting future brain signal is highly sought-after, yet difficult to achieve. To predict the future phase of cortical activity at localized ECoG and MEG recording sites, we exploit its predominant, large-scale, spatiotemporal dynamics. The dynamics are extracted from the brain signal through Fourier analysis and principal components analysis (PCA) only, and cast in a data model that predicts future signal at each site and frequency of interest. The dominant eigenvectors of the PCA that map the large-scale patterns of past cortical phase to future ones take the form of smoothly propagating waves over the entire measurement array. In ECoG data from 3 subjects and MEG data from 20 subjects collected during a self-initiated motor task, mean phase prediction errors were as low as 0.5 radians at local sites, surpassing state-of-the-art methods of within-time-series or event-related models. Prediction accuracy was highest in delta to beta bands, depending on the subject, was more accurate during episodes of high global power, but was not strongly dependent on the time-course of the task. Prediction results did not require past data from the to-be-predicted site. Rather, best accuracy depended on the availability in the model of long wavelength information. The utility of large-scale, low spatial frequency traveling waves in predicting future phase activity at local sites allows estimation of the error introduced by failing to account for irreducible trajectories in the activity dynamics.

When it comes to the classification of brain signals in real-life applications, the training and the prediction data are often described by different distributions. Furthermore, diverse data sets, e.g., recorded from various subjects or tasks, can even exhibit distinct feature spaces. The fact that data that have to be classified are often only available in small amounts reinforces the need for techniques to generalize learned information, as performances of brain-computer interfaces (BCIs) are enhanced by increasing quantity of available data. In this paper, we apply transfer learning to a framework based on deep convolutional neural networks (deep ConvNets) to prove the transferability of learned patterns in error-related brain signals across different tasks. The experiments described in this paper demonstrate the usefulness of transfer learning, especially improving performances when only little data can be used to distinguish between erroneous and correct realization of a task. This effect could be delimited from a transfer of merely general brain signal characteristics, underlining the transfer of error-specific information. Furthermore, we could extract similar patterns in time-frequency analyses in identical channels, leading to selective high signal correlations between the two different paradigms. Classification on the intracranial data yields in median accuracies up to (81.50 ± 9.49)%. Decoding on only 10% of the data without pre-training reaches performances of (54.76 ± 3.56)%, compared to (64.95 ± 0.79)% with pre-training.

EEG is the most common signal source for noninvasive BCI applications. For such applications, the EEG signal needs to be decoded and translated into appropriate actions. A recently emerging EEG decoding approach is deep learning with Convolutional or Recurrent Neural Networks (CNNs, RNNs) with many different architectures already published. Here we present a novel framework for the large-scale evaluation of different deep-learning architectures on different EEG datasets. This framework comprises (i) a collection of EEG datasets currently including 100 examples (recording sessions) from six different classification problems, (ii) a collection of different EEG decoding algorithms, and (iii) a wrapper linking the decoders to the data as well as handling structured documentation of all settings and (hyper-) parameters and statistics, designed to ensure transparency and reproducibility. As an applications example we used our framework by comparing three publicly available CNN architectures: the Braindecode Deep4 ConvNet, Braindecode Shallow ConvNet, and two versions of EEGNet. We also show how our framework can be used to study similarities and differences in the performance of different decoding methods across tasks. We argue that the deep learning EEG framework as described here could help to tap the full potential of deep learning for BCI applications.

Deep learning techniques have revolutionized the field of machine learning and were recently successfully applied to various classification problems in noninvasive electroencephalography (EEG). However, these methods were so far only rarely evaluated for use in intracranial EEG. We employed convolutional neural networks (CNNs) to classify and characterize the error-related brain response as measured in 24 intracranial EEG recordings. Decoding accuracies of CNNs were significantly higher than those of a regularized linear discriminant analysis. Using time-resolved deep decoding, it was possible to classify errors in various regions in the human brain, and further to decode errors over 200 ms before the actual erroneous button press, e.g., in the precentral gyrus. Moreover, deeper networks performed better than shallower networks in distinguishing correct from error trials in all-channel decoding. In single recordings, up to 100 % decoding accuracy was achieved. Visualization of the networks' learned features indicated that multivariate decoding on an ensemble of channels yields related, albeit non-redundant information compared to single-channel decoding. In summary, here we show the usefulness of deep learning for both intracranial error decoding and mapping of the spatio-temporal structure of the human error processing network.

Motion analysis is important in a broad range of contexts, including animation, bio-mechanics, robotics and experiments investigating animal behavior. For applications, in which tracking accuracy is one of the main requirements, passive optical motion capture systems are widely used. Many skeleton tracking methods based on such systems use a predefined skeleton model, which is scaled once in the initialization step to the individual size of the character to be tracked. However, there are remarkable differences in the bone length relations across gender and even more across mammal races. In practice, the optimal skeleton model has to be determined in a manual and time-consuming process. In this paper, we reformulate this task as an optimization problem aiming to rescale a rough hierarchical skeleton structure to optimize probabilistic skeleton tracking performance. We solve this optimization problem by means of state-of-the-art blackbox optimization methods based on sequential model-based Bayesian optimization (SMBO). We compare different SMBO methods on three real-world datasets with an animal and humans, demonstrating that we can automatically find skeleton structures for previously unseen mammals. The same methods also allow an automated choice of a suitable starting frame for initializing tracking.

Stroke and neurodegenerative diseases, among a range of other neurologic disorders, can cause chronic paralysis. Patients suffering from paralysis may remain unable to achieve even basic everyday tasks such as liquid intake. Currently, there is a great interest in developing robotic assistants controlled via brain-machine interfaces (BMIs) to restore the ability to perform such tasks. This paper describes an autonomous robotic assistant for liquid intake. The components of the system include autonomous online detection both of the cup to be grasped and of the mouth of the user. It plans motions of the robot arm under the constraints that the cup stays upright while moving towards the mouth and that the cup stays in direct contact with the user's mouth while the robot tilts it during the drinking phase. To achieve this, our system also includes a technique for online estimation of the location of the user's mouth even under partial occlusions by the cup or robot arm. We tested our system in a real environment and in a shared-control setting using frequency-specific modulations recorded by electroencephalography (EEG) from the brain of the user. Our experiments demonstrate that our BMI-controlled robotic system enables a reliable liquid intake. We believe that our approach can easily be extended to other useful tasks including food intake and object manipulation.

The ability to track skeletal movements is important in a variety of applications including animation, biological studies and animal experiments. To detect even small movements, such a method should provide highly accurate estimates. Besides that it should not impede the mammal in its motion. This motivates the usage of a passive optical motion capture system. Thereby the main challenges are the initialization, the association of the unlabeled markers to their corresponding segment also across the frames, and the estimation of the skeleton configuration. While many existing approaches can deal with the latter two problems, they typically need a specific pose for initialization. This is rather unpractical in the context of animal tracking and often requires a manual initialization process. In this paper, we present an approach to reliably track animals and humans in marker-based optical motion capture systems with freely attached markers. Our method is also able to perform an automatic initialization without any pre-or post-processing of the data. To achieve this, our approach utilizes a large database of previously observed poses. We present our algorithm and its evaluation on real-world data sets with an animal and humans. The results demonstrate that our initialization method performs accurately for the most kind of initial poses and our tracking approach outperforms a popular fully automatic skeleton tracking method especially with respect to the smoothness of the motion.

We present an Internet-based brain-computer interface (BCI) for controlling an intelligent robotic device with autonomous reinforcement-learning. BCI control was achieved through dry-electrode electroencephalography (EEG) obtained during imaginary movements. Rather than using low-level direct motor control, we employed a high-level control scheme of the robot, acquired via reinforcement learning, to keep the users cognitive load low while allowing control a reaching-grasping task with multiple degrees of freedom. High-level commands were obtained by classification of EEG responses using an artificial neural network approach utilizing time-frequency features and conveyed through an intuitive user interface. The novel ombination of a rapidly operational dry electrode setup, autonomous control and Internet connectivity made it possible to conveniently interface subjects in an EEG laboratory with remote robotic devices in a closed-loop setup with online visual feedback of the robots actions to the subject. The same approach is also suitable to provide home-bound patients with the possibility to control state-of-the-art robotic devices currently confined to a research environment. Thereby, our BCI approach could help severely paralyzed patients by facilitating patient-centered research of new means of communication, mobility and independence.

A multilinear subspace regression model based on so called latent variable decomposition is introduced. Unlike standard regression methods which typically employ matrix (2D) data representations followed by vector subspace transformations, the proposed approach uses tensor subspace transformations to model common latent variables across both the independent and dependent data. The proposed approach aims to maximize the correlation between the so derived latent variables and is shown to be suitable for the prediction of multidimensional dependent data from multidimensional independent data, where for the estimation of the latent variables we introduce an algorithm based on Multilinear Singular Value Decomposition (MSVD) on a specially defined cross-covariance tensor. It is next shown that in this way we are also able to unify the existing Partial Least Squares (PLS) and N-way PLS regression algorithms within the same framework. Simulations on benchmark synthetic data confirm the advantages of the proposed approach, in terms of its predictive ability and robustness, especially for small sample sizes. The potential of the proposed technique is further illustrated on a real world task of the decoding of human intracranial electrocorticogram (ECoG) from a simultaneously recorded scalp electroencephalograph (EEG).