Many recent studies have shown that the perception of speech can be decoded from brain signals and subsequently reconstructed as continuous language.However, there is a lack of neurological basis for how the semantic information embedded within brain signals can be used more effectively to guide language reconstruction.Predictive coding theory suggests the human brain naturally engages in continuously predicting future words that span multiple timescales.This implies that the decoding of brain signals could potentially be associated with a predictable future.To explore the predictive coding theory within the context of language reconstruction, this paper proposes PREDFT (FMRI-to-Text decoding with Predictive coding).PREDFT consists of a main network and a side network.The side network obtains brain predictive representation from related regions of interest (ROIs) with a self-attention module.The representation is then fused into the main network for continuous language decoding.Experiments on two naturalistic language comprehension fMRI datasets show that PREDFT outperforms current decoding models on several evaluation metrics.

The clinical diagnosis of most mental disorders primarily relies on the conversations between psychiatrist and patient. The creation of such diagnostic conversation datasets is promising to boost the AI mental healthcare community. However, directly collecting the conversations in real diagnosis scenarios is near impossible due to stringent privacy and ethical considerations. To address this issue, we seek to synthesize diagnostic conversation by exploiting anonymized patient cases that are easier to access. Specifically, we design a neuro-symbolic multi-agent framework for synthesizing the diagnostic conversation of mental disorders with large language models. It takes patient case as input and is capable of generating multiple diverse conversations with one single patient case. The framework basically involves the interaction between a doctor agent and a patient agent, and generates conversations under symbolic control via a dynamic diagnosis tree. By applying the proposed framework, we develop the largest Chinese mental disorders diagnosis dataset MDD-5k. This dataset is built upon 1000 real, anonymized patient cases by cooperating with Shanghai Mental Health Center and comprises 5000 high-quality long conversations with diagnosis results and treatment opinions as labels. To the best of our knowledge, it's also the first labeled dataset for Chinese mental disorders diagnosis. Human evaluation demonstrates the proposed MDD-5k dataset successfully simulates human-like diagnostic process of mental disorders.

The neural basis of semantics has long been a central topic across disciplines such as linguistics, cognitive neuroscience, and artificial intelligence. Traditional approaches in neuroscience often struggle to isolate semantic processing from syntactic influences without compromising the naturalness of sentence comprehension, due to constraints in experimental paradigms. Meanwhile, large language models (LLMs) have shown remarkable alignment with human brain activity, offering new tools to investigate language representation in the brain through brain encoding techniques. However, extracting purely semantic representations from LLMs remains a significant challenge, as their internal embeddings inherently entangle syntax and semantics. To address this challenge, we introduce a semantic representation isolation method based on large language models. Specifically, we propose Augmented Semantic Extraction (ASE), which conditionally perturbs word order to disrupt syntactic structures, thereby isolating semantic representations from the mixed encodings produced by LLMs. We evaluate ASE using multiple probing techniques and demonstrate that it effectively purifies semantic content from language model representations under a linear decoding framework. Leveraging ASE, we perform brain encoding experiments on a functional magnetic resonance imaging (fMRI) dataset collected during natural spoken narrative comprehension. Our findings reveal the cortical organization of semantic processing. The results provide neural evidence that semantic processing constitutes the dominant component of language comprehension in the brain.

Recent major milestones have successfully reconstructed natural language from non-invasive brain signals (e.g.functional Magnetic Resonance Imaging (fMRI) and Electroencephalogram (EEG)) across subjects.However, we find current dataset splitting strategies for cross-subject brain-to-text decoding are wrong.Specifically, we first demonstrate that all current splitting methods suffer from data leakage problem, which refers to the leakage of validation and test data into training set, resulting in significant overfitting and overestimation of decoding models.In this study, we develop a right cross-subject data splitting criterion without data leakage for decoding fMRI and EEG signal to text.Some SOTA brain-to-text decoding models are re-evaluated correctly with the proposed criterion for further research.

Dialogue structure discovery is essential in dialogue generation. Well-structured topic flow can leverage background information and predict future topics to help generate controllable and explainable responses. However, most previous work focused on dialogue structure learning in task-oriented dialogue other than open-domain dialogue which is more complicated and challenging. In this paper, we present a new framework CTRLStruct for dialogue structure learning to effectively explore topic-level dialogue clusters as well as their transitions with unlabelled information. Precisely, dialogue utterances encoded by bi-directional Transformer are further trained through a special designed contrastive learning task to improve representation. Then we perform clustering to utterance-level representations and form topic-level clusters that can be considered as vertices in dialogue structure graph. The edges in the graph indicating transition probability between vertices are calculated by mimicking expert behavior in datasets. Finally, dialogue structure graph is integrated into dialogue model to perform controlled response generation. Experiments on two popular open-domain dialogue datasets show our model can generate more coherent responses compared to some excellent dialogue models, as well as outperform some typical sentence embedding methods in dialogue utterance representation. Code is available in GitHub1.