Inspired by the robustness of human vision, various attempts have been made to incorporate brain-inspired mechanisms into artificial neural networks.A popular candidate has been predictive coding, a prominent theory in neuroscience, that theorizes that feedback connections communicate top-down predictions to earlier regions.While recurrence in models has been demonstrated to be useful when processing noisy and di cult stimuli, a direct evidence of its utility for explaining brain data under such situation was yet to be shown.Here, we investigated whether such brain-inspired mechanism actually helps to capture neural dynamics.Specifically, we measured the brain alignment between representations of a predictive version of a popular feedforward CNN often used as a computational model of the visual cortex-VGG16-and human EEG collected when viewing images that were relatively easy (Control) or di cult to classify (Challenge).We demonstrate that the recurrent dynamics significantly enhanced the model's alignment with EEG responses, underscoring the importance of recurrent connectivity in computational models of human vision, an e↵ect distinctly visible for challenging stimuli.

Multiple theories exist for the role of feedback connections in the brain and in the artificial neural networks, but remain untested using modern tools.In this work, we undertake this task by exploring the utility of explainability methods like GradCAMs [1] in investigating bio-inspired recurrent networks-provided with the predify[2] package-that perform hierarchical updates inspired by the predictive coding theory in neuroscience.We report an extensive search with different levels of feedforward and feedback information.Our preliminary results show that the dynamics are able to recover the GradCAMs on noisy images, providing promising avenues for future work aiming to understand the role of recurrence.

Predictive coding-a popular theory in neuroscience-has garnered significant attention in the machine learning community aiming to incorporate brain-inspired components in neural networks.While various proposals have demonstrated the ability of predictive dynamics to render robustness and entail human-like perception of illusions, it remains unclear if they improve the alignment between brain and artificial representations.Here, we systematically investigate the conditions under which brain-inspired modifications in predictive processing improve alignment between model and neural representations in the brain.Our results reveal that the feedback component significantly increases similarity between model representations and those found in higher-level visual brain areas, especially when processing complex visual scenes.

Reconstructing perceived natural images from fMRI signals is one of the most engaging topics of neural decoding research. Prior studies had success in reconstructing either the low-level image features or the semantic/high-level aspects, but rarely both. In this study, we utilized an Instance-Conditioned GAN (IC-GAN) model to reconstruct images from fMRI patterns with both accurate semantic attributes and preserved low-level details. The IC-GAN model takes as input a 119-dim noise vector and a 2048-dim instance feature vector extracted from a target image via a self-supervised learning model (SwAV ResNet-50); these instance features act as a conditioning for IC-GAN image generation, while the noise vector introduces variability between samples. We trained ridge regression models to predict instance features, noise vectors, and dense vectors (the output of the first dense layer of the IC-GAN generator) of stimuli from corresponding fMRI patterns. Then, we used the IC-GAN generator to reconstruct novel test images based on these fMRI-predicted variables. The generated images presented state-of-the-art results in terms of capturing the semantic attributes of the original test images while remaining relatively faithful to low-level image details. Finally, we use the learned regression model and the IC-GAN generator to systematically explore and visualize the semantic features that maximally drive each of several regions-of-interest in the human brain.

The human hippocampus possesses "concept cells", neurons that fire when presented with stimuli belonging to a specific concept, regardless of the modality. Recently, similar concept cells were discovered in a multimodal network called CLIP (Radford et al., 2021). Here, we ask whether CLIP can explain the fMRI activity of the human hippocampus better than a purely visual (or linguistic) model. We extend our analysis to a range of publicly available uni- and multi-modal models. We demonstrate that "multimodality" stands out as a key component when assessing the ability of a network to explain the multivoxel activity in the hippocampus.

Vision models trained on multimodal datasets can benefit from the wide availability of large image-caption datasets.A recent model (CLIP) was found to generalize well in zeroshot and transfer learning settings.This could imply that linguistic or "semantic grounding" confers additional generalization abilities to the visual feature space.Here, we systematically evaluate various multimodal architectures and vision-only models in terms of unsupervised clustering, few-shot learning, transfer learning and adversarial robustness.In each setting, multimodal training produced no additional generalization capability compared to standard supervised visual training.We conclude that work is still required for semantic grounding to help improve vision models.

Deep neural networks excel at image classification, but their performance is far less robust to input perturbations than human perception. In this work we explore whether this shortcoming may be partly addressed by incorporating brain-inspired recurrent dynamics in deep convolutional networks. We take inspiration from a popular framework in neuroscience: "predictive coding". At each layer of the hierarchical model, generative feedback "predicts" (i.e., reconstructs) the pattern of activity in the previous layer. The reconstruction errors are used to iteratively update the network’s representations across timesteps, and to optimize the network's feedback weights over the natural image dataset--a form of unsupervised training. We show that implementing this strategy into two popular networks, VGG16 and EfficientNetB0, improves their robustness against various corruptions and adversarial attacks. We hypothesize that other feedforward networks could similarly benefit from the proposed framework. To promote research in this direction, we provide an open-sourced PyTorch-based package called \textit{Predify}, which can be used to implement and investigate the impacts of the predictive coding dynamics in any convolutional neural network.