Yuko Nakagi, Takuya Matsuyama, Naoko Koide-Majima, Hiroto Q. Yamaguchi, Rieko Kubo, Shinji Nishimoto, Yu Takagi. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Visual images observed by humans can be reconstructed from their brain activity. However, the visualization (externalization) of mental imagery is challenging. Only a few studies have reported successful visualization of mental imagery, and their visualizable images have been limited to specific domains such as human faces or alphabetical letters. Therefore, visualizing mental imagery for arbitrary natural images stands as a significant milestone. In this study, we achieved this by enhancing a previous method. Specifically, we demonstrated that the visual image reconstruction method proposed in the seminal study by Shen et al. (2019) heavily relied on low-level visual information decoded from the brain and could not efficiently utilize the semantic information that would be recruited during mental imagery. To address this limitation, we extended the previous method to a Bayesian estimation framework and introduced the assistance of semantic information into it. Our proposed framework successfully reconstructed both seen images (i.e., those observed by the human eye) and imagined images from brain activity. Quantitative evaluation showed that our framework could identify seen and imagined images highly accurately compared to the chance accuracy (seen: 90.7%, imagery: 75.6%, chance accuracy: 50.0%). In contrast, the previous method could only identify seen images (seen: 64.3%, imagery: 50.4%). These results suggest that our framework would provide a unique tool for directly investigating the subjective contents of the brain such as illusions, hallucinations, and dreams.

Reconstructing visual experiences from human brain activity offers a unique way to understand how the brain represents the world, and to interpret the connection between computer vision models and our visual system. While deep generative models have recently been employed for this task, reconstructing realistic images with high semantic fidelity is still a challenging problem. Here, we propose a new method based on a diffusion model (DM) to reconstruct images from human brain activity obtained via functional magnetic resonance imaging (fMRI). More specifically, we rely on a latent diffusion model (LDM) termed Stable Diffusion. This model reduces the computational cost of DMs, while preserving their high generative performance. We also characterize the inner mechanisms of the LDM by studying how its different components (such as the latent vector of image Z, conditioning inputs C, and different elements of the denoising U-Net) relate to distinct brain functions. We show that our proposed method can reconstruct high-resolution images with high fidelity in straight-forward fashion, without the need for any additional training and fine-tuning of complex deep-learning models. We also provide a quantitative interpretation of different LDM components from a neuroscientific perspective. Overall, our study proposes a promising method for reconstructing images from human brain activity, and provides a new framework for understanding DMs. Please check out our webpage at https://sites.google.com/view/stablediffusion-withbrain/.

A content recommendation system based on human brain activity has become a reality. However, the cost of collecting the information from people is problematic. This article proposes a scheme that resolves the tradeoff between the inference performance from a profile model to a brain model and the cost of collecting profile information. In the proposed scheme, a machine learning model infers the brain model from the profile model and a feature selection method is applied to reduce the cost, i.e., the number of questionnaire items, of collecting profile information. Since only the top questionnaire items with the highest importance scores are used, we can maintain the inference performance as high as possible while limiting the number of questionnaire items. We demonstrate the effectiveness of the proposed scheme with a performance evaluation using an experimentally obtained brain model and a profile model created from real profile information. The results over different experimental parameters, video lengths, and feature selection methods demonstrate that the proposed scheme successfully identifies the top questionnaire items that contribute most significantly to the inference of brain models.

The quantitative modeling of semantic representations in the brain plays a key role in understanding the neural basis of semantic processing. Previous studies have demonstrated that word vectors, which were originally developed for use in the field of natural language processing, provide a powerful tool for such quantitative modeling. However, whether semantic representations in the brain revealed by the word vector-based models actually capture our perception of semantic information remains unclear, as there has been no study explicitly examining the behavioral correlates of the modeled brain semantic representations. To address this issue, we compared the semantic structure of nouns and adjectives in the brain estimated from word vector-based brain models with that evaluated from human behavior. The brain models were constructed using voxelwise modeling to predict the functional magnetic resonance imaging (fMRI) response to natural movies from semantic contents in each movie scene through a word vector space. The semantic dissimilarity of brain word representations was then evaluated using the brain models. Meanwhile, data on human behavior reflecting the perception of semantic dissimilarity between words were collected in psychological experiments. We found a significant correlation between brain model- and behavior-derived semantic dissimilarities of words. This finding suggests that semantic representations in the brain modeled via word vectors appropriately capture our perception of word meanings.

The human brain can effectively learn a new task from a small number of samples, which indicates that the brain can transfer its prior knowledge to solve tasks in different domains. This function is analogous to transfer learning (TL) in the field of machine learning. TL uses a well-trained feature space in a specific task domain to improve performance in new tasks with insufficient training data. TL with rich feature representations, such as features of convolutional neural networks (CNNs), shows high generalization ability across different task domains. However, such TL is still insufficient in making machine learning attain generalization ability comparable to that of the human brain. To examine if the internal representation of the brain could be used to achieve more efficient TL, we introduce a method for TL mediated by human brains. Our method transforms feature representations of audiovisual inputs in CNNs into those in activation patterns of individual brains via their association learned ahead using measured brain responses. Then, to estimate labels reflecting human cognition and behavior induced by the audiovisual inputs, the transformed representations are used for TL. We demonstrate that our brain-mediated TL (BTL) shows higher performance in the label estimation than the standard TL. In addition, we illustrate that the estimations mediated by different brains vary from brain to brain, and the variability reflects the individual variability in perception. Thus, our BTL provides a framework to improve the generalization ability of machine-learning feature representations and enable machine learning to estimate human-like cognition and behavior, including individual variability.

Recent studies using functional magnetic resonance imaging (fMRI) have enabled quantitative evaluation of the semantic space during processing of visual stimuli. In the semantic space of the natural language processing model, called a skip-gram, decoders were shown to generalize to natural scenes of a movie that was not included in the training data of the decoders. Combined with electrocorticography (ECoG), which has a higher sampling rate than fMRI, this approach is expected to aid the development of a practical brain-machine interface. Here, we decoded vector representations of scenes within the semantic space of a skip-gram model to assess whether a decoder trained using ECoG features still generalizes to scenes new to the decoder.

Quantitative modeling of human brain activity based on language representations has been actively studied in systems neuroscience. However, previous studies examined word-level representation, and little is known about whether we could recover structured sentences from brain activity. This study attempts to generate natural language descriptions of semantic contents from human brain activity evoked by visual stimuli. To effectively use a small amount of available brain activity data, our proposed method employs a pre-trained image-captioning network model using a deep learning framework. To apply brain activity to the image-captioning network, we train regression models that learn the relationship between brain activity and deep-layer image features. The results demonstrate that the proposed model can decode brain activity and generate descriptions using natural language sentences. We also conducted several experiments with data from different subsets of brain regions known to process visual stimuli. The results suggest that semantic information for sentence generations is widespread across the entire cortex.

Music-genre recognition (MGR) has been a central issue in understanding human preferences of music. Previous studies have used various acoustic features to achieve MGR, though it has been largely unknown how music genres and related features are represented in the brain. Here, we measured brain activity while subjects passively listened to naturalistic music of various genres. A voxel-wise encoding model showed different activation patterns for each music genre in the bilateral superior temporal gyrus. We further performed music-genre classification using both a feature-based approach and a brain activity-based approach. Both approaches provided above-chance classification accuracy. Among four feature models, a biologically plausible spectro-temporal modulation transfer function (MTF) model showed the highest performance. These results provide a new insight into biologically plausible models of music genre.

In this study, we investigate whether sparse coding helps explain the semantic representation in human cerebral cortex. We show this by using sparse coding to model semantic representation in the cerebral cortex. We propose three methods for estimating semantic representation from brain activity data. For estimating a new semantic representation, in the first method, we use only a semantic representation dictionary obtained via sparse coding. The semantic representation estimated using this method is more similar to the actual semantic representation of the cerebral cortex than that estimated without sparse coding. In the second method, we use only a brain activity dictionary obtained via sparse coding. The semantic representation estimated using this method is also better than that estimated without sparse coding. In addition, in the third method, we estimate semantic representation by applying sparse coding to both semantic representation and brain activity data. The semantic representation estimated by using this third method is better than that estimated by the first or second methods. Through the above three experiments, we have confirmed that sparse coding helps explain the semantic representation in human cerebral cortex.