People can easily recognize human facial expressions on unnatural head shapes such as those of cartoon characters or animals. Current machine learning algorithms, however, struggle with out-of-domain transfer in facial expression recognition if not trained with large amounts of data. Here, we show that insights from neuroscience can be integrated into computer vision models to facilitate the transfer of learned expressions to novel head shapes. Specifically, we propose a biologically inspired mechanism based on norm-referenced encoding, which represents inputs as deviations from a domain-specific reference vector. By assuming that deviations from an appropriately chosen reference are preserved across domains, the model is able to generalize to a new domain using only a single additional training image. We conduct experiments on two datasets consisting of facial expressions on highly varying head shapes,demonstrating the model's generalization abilities and data efficiency. In doing so, we show that norm-referenced encod-ing is scalable and can be leveraged effectively by computer vision models, paving the way towards further applications related to faces, and potentially other appropriate classes of patterns.

Recent work has shown that the attention maps of the widely popular DINOv2 model exhibit artifacts, which hurt both model interpretability and performance on dense image tasks. These artifacts emerge due to the model repurposing patch tokens with redundant local information for the storage of global image information. To address this problem, additional register tokens have been incorporated in which the model can store such information instead. We carefully examine the influence of these register tokens on the relationship between global and local image features, showing that while register tokens yield cleaner attention maps, these maps do not accurately reflect the integration of local image information in large models. Instead, global information is dominated by information extracted from register tokens, leading to a disconnect between local and global features. Inspired by these findings, we show that the [CLS] token itself leads to a very similar phenomenon in models without explicit register tokens. Our work shows that care must be taken when interpreting attention maps of large ViTs. Further, by clearly attributing the faulty behavior to register and [CLS] tokens, we show a path towards more interpretable vision models.

Understanding how the human brain processes body movements is essential for clarifying the mechanisms underlying social cognition and interaction. This study investigates the encoding of biomechanically possible and impossible body movements in occipitotemporal cortex using ultra-high field 7T fMRI. By predicting the response of single voxels to impossible/possible movements using a computational modelling approach, our findings demonstrate that a combination of low-level, postural, biomechanical, and categorical features significantly predicts neural responses in the ventral visual cortex, particularly within the extrastriate body area (EBA), underscoring the brain's sensitivity to biomechanical plausibility.

High-level visual brain regions contain subareas in which neurons appear to respond more strongly to examples of a particular semantic category, like faces or bodies, rather than objects. However, recent work has shown that while this finding holds on average, some out-of-category stimuli also activate neurons in these regions. This may be due to visual features common among the preferred class also being present in other images. Here, we propose a deep-learning-based approach for visualizing these features. For each neuron, we identify relevant visual features driving its selectivity by modelling responses to images based on latent activations of a deep neural network. Given an out-of-category image which strongly activates the neuron, our method first identifies a reference image from the preferred category yielding a similar feature activation pattern. We then backpropagate latent activations of both images to the pixel level, while enhancing the identified shared dimensions and attenuating non-shared features. The procedure highlights image regions containing shared features driving responses of the model neuron. We apply the algorithm to novel recordings from body-selective regions in macaque IT cortex in order to understand why some images of objects excite these neurons. Visualizations reveal object parts which resemble parts of a macaque body, shedding light on neural preference of these objects.

State-of-the-art motorized hand prostheses are endowed with actuators able to provide independent and proportional control of as many as six degrees of freedom (DOFs). The control signals are derived from residual electromyographic (EMG) activity, recorded concurrently from relevant forearm muscles. Nevertheless, the functional mapping between forearm EMG activity and hand kinematics is only known with limited accuracy. Therefore, no robust method exists for the reliable computation of control signals for the independent and proportional actuation of more than two DOFs. A common approach to deal with this limitation is to preprogram the prostheses for the execution of a restricted number of behaviors (e.g., pinching, grasping, and wrist rotation) that are activated by the detection of specific EMG activation patterns. However, this approach severely limits the range of activities users can perform with the prostheses during their daily living. In this work, we introduce a novel method, based on a long short-term memory (LSTM) network, to map forearm EMG activity onto hand kinematics online. Critically, unlike previous research efforts that tend to focus on simple and highly controlled motor tasks, we tested our method on a dataset of daily living activities (ADLs): the KIN-MUS UJI dataset. To the best of our knowledge, ours is the first reported work on the prediction of hand kinematics that uses this challenging dataset. Remarkably, we show that our network is able to generalize to novel untrained ADLs. Our results suggest that the presented method is suitable for the generation of control signals for the independent and proportional actuation of the multiple DOFs of state-of-the-art hand prostheses.

The locomotion and balance capabilities of bipedal robots have greatly improved in recent years. However, maintaining balance on difficult terrain still poses a significant challenge. In this paper, we examined how humans maintain mediolateral balance when standing on a narrow beam with bare feet and wearing rigid soles. Our results show that foot-beam interaction dynamics critically influence balancing behavior. Importantly, this suggests that differences in human balancing behavior across different support surfaces may not solely result from changes in their neural control strategy. They may also result from changes in foot-ground interaction. Thus, the altered foot-ground interaction dynamics must be considered to accurately capture changes in the human controller across different support surfaces. A simplified model of foot-beam interaction was added to a double inverted pendulum model for human balancing. This extended model could replicate the change in human behavior across different foot contact conditions (bare feet vs. rigid feet). A better understanding of how humans coordinate whole-body behavior across a range of conditions may inform the development of balance controllers for bipedal robots.

Motor rehabilitation is in increasingly high demand to deal with minor functional motor impairments resulting from stroke, cerebellar ataxia, or Parkinson's disease. Juggling physiotherapy has shown to induce brain plasticity and to improve coordination and balance in this context. The physiotherapy, however, relies on large number of repetitions to be effective which prompts to deploy robots to release the burden on therapists both in terms of time as well as physical strain. This paper provides a framework to enable juggling games for patients in interacting with robots through Virtual Reality (VR). A set of throwing motions is recorded from the therapist and is retargeted to the humanoid robot COMAN's wrist. The respective whole-body motion is then solved in a stack of Quadratic Programs (QP) in a real-time architecture that integrates OROCOS and Gazebo. The resulting motion is finally streamed to VR for animation of the robot and the thrown ball, which the user can catch in VR using a controller device. We regard the VR setting as an essential step towards physiotherapeutic robotic juggling, because it ensures safety of the patients and effective testing of the methods and already has potential for actual therapeutic intervention. The control framework, however, is already validated in this paper for switching to full real-time operation on the physical robot.

Building on our previous work [Taubert et al. 2012], we present an approach for real-time interaction between a real human and an avatar. We generate reactive motions by a dynamical extension of a hierarchical Gaussian process latent variable model, including latent dimensions for emotional style variation and target positions. This allows the avatar to produce accurate reactive motions to the human. To validate our approach, we developed a real-time application where an avatar and a human actor engage in emotional 'high fives'. Furthermore, we show preliminary results indicating that humans do perceive emotions more accurately when engaging in interaction as opposed to passive observation.

The online synthesis of stylized interactive movements with high levels of realism is a difficult problem in computer graphics. We present a new approach for the learning of structured dynamical models for the synthesis of interactive body movements that is based on hierarchical Gaussian process latent variable models. The latent spaces of this model encode postural manifolds and the dependency between the postures of the interacting characters. In addition, our model includes dimensions representing emotional style variations (for neutral, happy, angry, sad) and individually-specific motion style. The dynamics of the state in the latent space is modeled by a Gaussian Process Dynamical Model, a probabilistic dynamical model that can learn to generate arbitrary smooth trajectories in real-time. The proposed framework offers a large degree of flexibility, in terms of the definition of the model structure as well as the complexity of the learned motion trajectories. In order to assess the suitability of the proposed framework for the generation of highly realistic motion, we performed a 'Turing test': a psychophysical study where human observers classified the emotions and rated the naturalness of the generated and natural emotional handshakes. Classification results for both stimulus groups were not significantly different, and for all emotional styles, except for neutral, participants rated the synthesized handshakes equally natural as animations with the original trajectories. This shows that the proposed method generates highly-realistic interactive movements that are almost indistinguishable from natural ones. As a further extension, we demonstrate the capability of the method to interpolate between different emotional styles.

Natural body movements arise in the form of temporal sequences of individual actions. During visual action analysis, the human visual system must accomplish a temporal segmentation of the action stream into individual actions. Such temporal segmentation is also essential to build hierarchical models for action synthesis in computer animation. Ideally, such segmentations should be computed automatically in an unsupervised manner. We present an unsupervised segmentation algorithm that is based on Bayesian Binning (BB) and compare it to human segmentations derived from psychophysical data. BB has the advantage that the observation model can be easily exchanged. Moreover, being an exact Bayesian method, BB allows for the automatic determination of the number and positions of segmentation points. We applied this method to motion capture sequences from martial arts and compared the results to segmentations provided by humans from movies that showed characters that were animated with the motion capture data. Human segmentation was then assessed by an interactive adjustment paradigm, where participants had to indicate segmentation points by selection of the relevant frames. Results show a good agreement between automatically generated segmentations and human performance when the trajectory segments between the transition points were modeled by polynomials of at least third order. This result is consistent with theories about differential invariants of human movements.

Blind source separation, i.e. the extraction of unknown sources from a set of given signals, is relevant for many applications. A special case of this problem is dimension reduction, where the goal is to approximate a given set of signals by superpositions of a minimal number of sources. Since in this case the signals outnumber the sources the problem is over-determined. Most popular approaches for addressing this problem are based on purely linear mixing models. However, many applications like the modeling of acoustic signals, EMG signals, or movement trajectories, require temporal shift-invariance of the extracted components. This case has only rarely been treated in the computational literature, and specifically for the case of dimension reduction almost no algorithms have been proposed. We present a new algorithm for the solution of this problem, which is based on a timefrequency transformation (Wigner-Ville distribution) of the generative model. We show that this algorithm outperforms classical source separation algorithms for linear mixtures, and also a related method for mixtures with delays. In addition, applying the new algorithm to trajectories of human gaits, we demonstrate that it is suitable for the extraction of spatio-temporal components that are easier to interpret than components extracted with other classical algorithms.

Visual processing in the cortex can be characterized by a predominantly hierarchical architecture, in which specialized brain regions along the processing pathways extract visual features of increasing complexity, accompanied by greater invariance in stimulus properties such as size and position. Various studies have postulated that a nonlinear pooling function such as the maximum (MAX) operation could be fundamental in achieving such selectivity and invariance. In this article, we are concerned with neurally plausible mechanisms that may be involved in realizing the MAX operation. Different canonical models are proposed, each based on neural mechanisms that have been previously discussed in the context of cortical processing. Through simulations and mathematical analysis, we compare the performance and robustness of these mechanisms. We derive experimentally verifiable predictions for each model and discuss the relevant physiological considerations.

Asymmetric lateral connections are one possible mechanism that can ac- count for the direction selectivity of cortical neurons. We present a math- ematical analysis for a class of these models. Contrasting with earlier theoretical work that has relied on methods from linear systems theory, we study the network’s nonlinear dynamic properties that arise when the threshold nonlinearity of the neurons is taken into account. We show that such networks have stimulus-locked traveling pulse solutions that are appropriate for modeling the responses of direction selective cortical neurons. In addition, our analysis shows that outside a certain regime of stimulus speeds the stability of this solutions breaks down giving rise to another class of solutions that are characterized by specific spatio- temporal periodicity. This predicts that if direction selectivity in the cor- tex is mainly achieved by asymmetric lateral connections lurching activ- ity waves might be observable in ensembles of direction selective cortical neurons within appropriate regimes of the stimulus speed.