Advances in brain decoding, particularly in visual image reconstruction, have sparked discussions about the societal implications and ethical considerations of neurotechnology. As reconstruction methods aim to recover visual experiences from brain activity and achieve prediction beyond training samples (zero-shot prediction), it is crucial to assess their capabilities and limitations to inform public expectations and regulations. Our case study of recent text-guided reconstruction methods, which leverage a large-scale dataset (Natural Scenes Dataset, NSD) and text-to-image diffusion models, reveals critical limitations in their generalizability, demonstrated by poor reconstructions on a different dataset. UMAP visualization of the text features from NSD images shows limited diversity with overlapping semantic and visual clusters between training and test sets. We identify that clustered training samples can lead to "output dimension collapse," restricting predictable output feature dimensions. While diverse training data improves generalization over the entire feature space without requiring exponential scaling, text features alone prove insufficient for mapping to the visual space. Our findings suggest that the apparent realism in current text-guided reconstructions stems from a combination of classification into trained categories and inauthentic image generation (hallucination) through diffusion models, rather than genuine visual reconstruction. We argue that careful selection of datasets and target features, coupled with rigorous evaluation methods, is essential for achieving authentic visual image reconstruction. These insights underscore the importance of grounding interdisciplinary discussions in a thorough understanding of the technology's current capabilities and limitations to ensure responsible development.

Neuroscience and artificial intelligence (AI) share a long history of collaboration. Advances in neuroscience, alongside huge leaps in computer processing power over the last few decades, have given rise to a new generation of in silico neural networks inspired by the architecture of the brain. These AI systems are now capable of many of the advanced perceptual and cognitive abilities of biological systems, including object recognition and decision making. Moreover, AI is now increasingly being employed as a tool for neuroscience research and is transforming our understanding of brain functions. In particular, deep learning has been used to model how convolutional layers and recurrent connections in the brain's cerebral cortex control important functions, including visual processing, memory, and motor control. Excitingly, the use of neuroscience-inspired AI also holds great promise for understanding how changes in brain networks result in psychopathologies, and could even be utilized in treatment regimes. Here we discuss recent advancements in four areas in which the relationship between neuroscience and AI has led to major advancements in the field; (1) AI models of working memory, (2) AI visual processing, (3) AI analysis of big neuroscience datasets, and (4) computational psychiatry.

The mental contents of perception and imagery are thought to be encoded in hierarchical representations in the brain, but previous attempts to visualize perceptual contents have failed to capitalize on multiple levels of the hierarchy, leaving it challenging to reconstruct internal imagery. Recent work showed that visual cortical activity measured by functional magnetic resonance imaging (fMRI) can be decoded (translated) into the hierarchical features of a pre-trained deep neural network (DNN) for the same input image, providing a way to make use of the information from hierarchical visual features. Here, we present a novel image reconstruction method, in which the pixel values of an image are optimized to make its DNN features similar to those decoded from human brain activity at multiple layers. We found that our method was able to reliably produce reconstructions that resembled the viewed natural images. A natural image prior introduced by a deep generator neural network effectively rendered semantically meaningful details to the reconstructions. Human judgment of the reconstructions supported the effectiveness of combining multiple DNN layers to enhance the visual quality of generated images. While our model was solely trained with natural images, it successfully generalized to artificial shapes, indicating that our model was not simply matching to exemplars. The same analysis applied to mental imagery demonstrated rudimentary reconstructions of the subjective content. Our results suggest that our method can effectively combine hierarchical neural representations to reconstruct perceptual and subjective images, providing a new window into the internal contents of the brain.

Recent studies using functional magnetic resonance imaging (fMRI) have enabled quantitative evaluation of the semantic space during processing of visual stimuli. In the semantic space of the natural language processing model, called a skip-gram, decoders were shown to generalize to natural scenes of a movie that was not included in the training data of the decoders. Combined with electrocorticography (ECoG), which has a higher sampling rate than fMRI, this approach is expected to aid the development of a practical brain-machine interface. Here, we decoded vector representations of scenes within the semantic space of a skip-gram model to assess whether a decoder trained using ECoG features still generalizes to scenes new to the decoder.

Neural encoding and decoding provide perspectives for understanding neural representations of sensory inputs. Recent functional magnetic resonance imaging (fMRI) studies have succeeded in building prediction models for encoding and decoding numerous stimuli by representing a complex stimulus as a combination of simple elements. While arbitrary visual images were reconstructed using a modular model that combined the outputs of decoder modules for multiscale local image bases (elements), the shapes of the image bases were heuristically determined. In this work, we propose a method to establish mappings between the stimulus and the brain by automatically extracting modules from measured data. We develop a model based on Bayesian canonical correlation analysis, in which each module is modeled by a latent variable that relates a set of pixels in a visual image to a set of voxels in an fMRI activity pattern. The estimated mapping from a latent variable to pixels can be regarded as an image basis. We show that the model estimates a modular representation with spatially localized multiscale image bases. Further, using the estimated mappings, we derive encoding and decoding models that produce accurate predictions for brain activity and stimulus images. Our approach thus provides a novel means of revealing neural representations of stimuli by automatically extracting modules, which can be used to generate effective prediction models for encoding and decoding.

Brain activity can be seen as "codes" that encode mental states. Recent advances in human neuroimaging such as functional magnetic resonance imaging (fMRI) have revealed brain regions that encode specific behavior and cognition. Despite the wide-spread use of human neuroimaging, its potential to read out, or "decode", mental contents from brain activity has not been fully explored. In this talk, I present methods for decoding visual representations from fMRI activity patterns based on machine learning techniques. I show how early visual features represented in "subvoxel" neural structures could be decoded from ensemble fMRI responses. Decoding of stimulus features is extended to the method for neural mind-reading, which attempts to predict a person's subjective state using a decoder trained with unambiguous stimulus presentation. We then discuss a modular decoding approach, in which a wide variety of percepts can be decoded by combining the outputs of multiple decoder modules. On the basis of this approach, we were able to reconstruct arbitrary visual images using the decoder trained on fMRI responses to only several hundred random images. Finally, I discuss potential applications of neural decoding to brain-based communications.

In this paper, a method for extracting and classifying movement-related brain signals is proposed. A single-trial MEG observation is first processed with a pre-whitening filter so that strong stationary interference is eliminated. Next, a brain signal effective for classification is extracted using an adaptive spatial filter. The extracted signal is then classified with a support vector machine. From the experimental results, it is shown that the classification rate of 62.6 % is obtained for the brain signals related to the three types of hand movements (ldquoscissors-paper-rockrdquo).

Image representation based on image bases provides a framework for understanding neural representation of visual perception. A recent fMRI study has shown that arbitrary contrast-defined visual images can be reconstructed from fMRI activity patterns using a combination of multi-scale local image bases. In the reconstruction model, the mapping from an fMRI activity pattern to the contrasts of the image bases was learned from measured fMRI responses to visual images. But the shapes of the images bases were fixed, and thus may not be optimal for reconstruction. Here, we propose a method to build a reconstruction model in which image bases are automatically extracted from the measured data. We constructed a probabilistic model that relates the fMRI activity space to the visual image space via a set of latent variables. The mapping from the latent variables to the visual image space can be regarded as a set of image bases. We found that spatially localized, multi-scale image bases were estimated near the fovea, and that the model using the estimated image bases was able to accurately reconstruct novel visual images. The proposed method provides a means to discover a novel functional mapping between stimuli and brain activity patterns.

In this paper, a generalized eigenvalue decomposition (GEVD)-based approach is proposed to remove continuous interferences included in MEG observations. The GEVD-based approach can be interpreted as a general framework of pre-whitening that can be combined with various types of signal separation techniques such as principal component analysis (PCA), independent component analysis (ICA) or adaptive beamforming (ABF). An example of extracting movement-related brain activities from single trial data is presented.