People can easily recognize human facial expressions on unnatural head shapes such as those of cartoon characters or animals. Current machine learning algorithms, however, struggle with out-of-domain transfer in facial expression recognition if not trained with large amounts of data. Here, we show that insights from neuroscience can be integrated into computer vision models to facilitate the transfer of learned expressions to novel head shapes. Specifically, we propose a biologically inspired mechanism based on norm-referenced encoding, which represents inputs as deviations from a domain-specific reference vector. By assuming that deviations from an appropriately chosen reference are preserved across domains, the model is able to generalize to a new domain using only a single additional training image. We conduct experiments on two datasets consisting of facial expressions on highly varying head shapes,demonstrating the model's generalization abilities and data efficiency. In doing so, we show that norm-referenced encod-ing is scalable and can be leveraged effectively by computer vision models, paving the way towards further applications related to faces, and potentially other appropriate classes of patterns.

Recent work has shown that the attention maps of the widely popular DINOv2 model exhibit artifacts, which hurt both model interpretability and performance on dense image tasks. These artifacts emerge due to the model repurposing patch tokens with redundant local information for the storage of global image information. To address this problem, additional register tokens have been incorporated in which the model can store such information instead. We carefully examine the influence of these register tokens on the relationship between global and local image features, showing that while register tokens yield cleaner attention maps, these maps do not accurately reflect the integration of local image information in large models. Instead, global information is dominated by information extracted from register tokens, leading to a disconnect between local and global features. Inspired by these findings, we show that the [CLS] token itself leads to a very similar phenomenon in models without explicit register tokens. Our work shows that care must be taken when interpreting attention maps of large ViTs. Further, by clearly attributing the faulty behavior to register and [CLS] tokens, we show a path towards more interpretable vision models.

High-level visual brain regions contain subareas in which neurons appear to respond more strongly to examples of a particular semantic category, like faces or bodies, rather than objects. However, recent work has shown that while this finding holds on average, some out-of-category stimuli also activate neurons in these regions. This may be due to visual features common among the preferred class also being present in other images. Here, we propose a deep-learning-based approach for visualizing these features. For each neuron, we identify relevant visual features driving its selectivity by modelling responses to images based on latent activations of a deep neural network. Given an out-of-category image which strongly activates the neuron, our method first identifies a reference image from the preferred category yielding a similar feature activation pattern. We then backpropagate latent activations of both images to the pixel level, while enhancing the identified shared dimensions and attenuating non-shared features. The procedure highlights image regions containing shared features driving responses of the model neuron. We apply the algorithm to novel recordings from body-selective regions in macaque IT cortex in order to understand why some images of objects excite these neurons. Visualizations reveal object parts which resemble parts of a macaque body, shedding light on neural preference of these objects.