Through preliminary studies, we observed that some figurative language detection tasks show consistent biases: a conservative preference for precision over recall and higher agreement within a model’s own judgments than between different models. To address these biases, we first introduce the Inter-Intra Agreement Ratio (IIAR), a novel metric that systematically quantifies LLM generalizability by comparing intra-and inter-model agreement patterns. Then, we propose Consensus-Based LLM Prediction Aggregation (CLPA), which leverages the collective strengths of multiple LLMs to mitigate individual model biases and achieve balanced precision and recall—without fine-tuning or compromising generality. Finally, we present a human–AI co-annotation framework that uses minimal expert input to efficiently determine optimal consensus thresholds, significantly reducing annotation effort. Experiments on the SemEval-2017 pun detection datasets demonstrate that our approach achieves a new state-of-the-art F1 score of 0.963 on heterographic pun detection through multi-LLM consensus aggregation; furthermore, our co-annotation framework requires only 10% of golden labels to accurately determine optimal consensus thresholds for capturing nuanced human annotation boundaries.

Metaphors are a fundamental cognitive tool for articulating abstract and subjective experiences and implicit semantics, making them potent indicators of psychological state, particularly in individuals with depression. The proliferation of social media has created a vast repository of such metaphorical expressions, offering an unprecedented opportunity to understand mental health struggles. These metaphors can provide crucial insights for clinical assessment and therapeutic intervention. However, their potential remains largely untapped in automated depression detection, primarily due to the lack of large-scale, annotated datasets. To bridge this gap, we introduce the Depression-Related Metaphor Dataset (DRMD), a novel resource of social media posts related to depression, incorporating depression levels (severe, moderate, minimum, and null), the presence or absence of metaphors, and their conceptual source domain mappings. We leverage this dataset to fine-tune Large Language Models (LLMs), integrating metaphorical features to enhance detection capabilities. Our results demonstrate that models incorporating metaphorical information achieve superior accuracy in depression detection and, importantly, generate high-quality explanations for their decisions by referencing specific metaphorical expressions. This work underscores the critical role of metaphorical analysis in computational mental health and provides a foundation for future research in explainable AI for depression detection. The dataset is publicly available.

Graph transformers are a recent advancement in machine learning, offering a new class of neural network models for graph-structured data. The synergy between transformers and graph learning demonstrates strong performance and versatility across various graph-related tasks. This survey provides an in-depth review of recent progress and challenges in graph transformer research. We begin with foundational concepts of graphs and transformers. We then explore design perspectives of graph transformers, focusing on how they integrate graph inductive biases and graph attention mechanisms into the transformer architecture. Furthermore, we propose a taxonomy classifying graph transformers based on depth, scalability, and pretraining strategies, summarizing key principles for effective development of graph transformer models. Beyond technical analysis, we discuss the applications of graph transformer models for node-level, edge-level, and graph-level tasks, exploring their potential in other application scenarios as well. Finally, we identify remaining challenges in the field, such as scalability and efficiency, generalization and robustness, interpretability and explainability, dynamic and complex graphs, as well as data quality and diversity, charting future directions for graph transformer research.

Senqi Yang, Dongyu Zhang, Jing Ren, Ziqi Xu, Xiuzhen Zhang, Yiliao Song, Hongfei Lin, Feng Xia. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Early detection of Alzheimer's disease (AD) through spontaneous speech analysis represents a promising, non-invasive diagnostic approach. Existing methods predominantly rely on fusion-based multimodal deep learning, effectively integrating linguistic and acoustic features. However, these methods inadequately model higher-order interactions between modalities, reducing diagnostic accuracy. To address this, we introduce SpeechHGT, a multimodal hypergraph transformer designed to capture and learn higher-order interactions in spontaneous speech features. SpeechHGT encodes multimodal features as hypergraphs, where nodes represent individual features and hyperedges represent grouped interactions. A novel hypergraph attention mechanism enables robust modeling of both pairwise and higher-order interactions. Experimental evaluations on the DementiaBank datasets reveal that SpeechHGT achieves state-of-the-art performance, surpassing baseline models in accuracy and F1 score. These results highlight the potential of hypergraph-based models to improve AI-driven diagnostic tools for early AD detection.

Metaphors play a crucial role in human communication, yet their comprehension remains a significant challenge for natural language processing (NLP) due to the cognitive complexity involved. According to Conceptual Metaphor Theory (CMT), metaphors map a target domain onto a source domain, and understanding this mapping is essential for grasping the nature of metaphors. Existing NLP research has focused on tasks like metaphor detection and sentiment analysis. However, there has been limited attention to identifying mappings between source and target domains. Moreover, non-English multimodal metaphor resources remain largely neglected in the literature, hindering a deeper understanding of the key elements involved in metaphor interpretation. To address this gap, we developed a Chinese multimodal metaphor advertisement dataset (namely CM3D) that includes annotations of specific target and source domains. This dataset aims at fostering further research into metaphor comprehension, particularly in non-English languages. Furthermore, we propose a Chain-of-Thought (CoT) Prompting-based Metaphor Mapping Identification Model (CPMMIM), which simulates the human cognitive process for identifying these mappings. Drawing inspiration from CoT reasoning and Bi-Level Optimization (BLO), we treat the task as a hierarchical identification problem, enabling more accurate and interpretable metaphor mapping. Our experimental results demonstrate the effectiveness of CPMMIM, highlighting its potential for advancing metaphor comprehension in NLP. Our dataset and code are both publicly available to encourage further advancements in this field.

Dynamic brain networks play a pivotal role in diagnosing brain disorders by capturing temporal changes in brain activity and connectivity. Previous methods often rely on sliding-window approaches for constructing these networks using fMRI data. However, these methods face two key limitations: a fixed temporal length that inadequately captures brain activity dynamics and a global spatial scope that introduces noise and reduces sensitivity to localized dysfunctions. These challenges can lead to inaccurate brain network representations and potential misdiagnoses.To address these challenges, we propose BrainDGT, a dynamic Graph Transformer model designed to enhance the construction and analysis of dynamic brain networks for more accurate diagnosis of brain disorders. BrainDGT leverages adaptive brain states by deconvolving the Hemodynamic Response Function (HRF) within individual functional brain modules to generate dynamic graphs, addressing the limitations of fixed temporal length and global spatial scope. The model learns spatio-temporal local features through attention mechanisms within these graphs and captures global interactions across modules using adaptive fusion. This dual-level integration enhances the model's ability to analyze complex brain connectivity patterns. We validate BrainDGT's effectiveness through classification experiments on three fMRI datasets (ADNI, PPMI, and ABIDE), where it outperforms state-of-the-art methods. By enabling adaptive, localized analysis of dynamic brain networks, BrainDGT advances neuroimaging and supports the development of more precise diagnostic and treatment strategies in biomedical research.

Researchers have attempted to mitigate lexical bias in toxic language detection (TLD). However, existing methods fail to disentangle the “useful” and “misleading” impact of lexical bias on model decisions. Therefore, they do not effectively exploit the positive effects of the bias and lead to a degradation in the detection performance of the debiased model. In this paper, we propose a Counterfactual Causal Debiasing Framework (CCDF) to mitigate lexical bias in TLD. It preserves the “useful impact” of lexical bias and eliminates the “misleading impact”. Specifically, we first represent the total effect of the original sentence and biased tokens on decisions from a causal view. We then conduct counterfactual inference to exclude the direct causal effect of lexical bias from the total effect. Empirical evaluations demonstrate that the debiased TLD model incorporating CCDF achieves state-of-the-art performance in both accuracy and fairness compared to competitive baselines applied on several vanilla models. The generalization capability of our model outperforms current debiased models for out-of-distribution data.

Harmful memes have proliferated on the Chinese Internet, while research on detecting Chinese harmful memes significantly lags behind due to the absence of reliable datasets and effective detectors.To this end, we present the comprehensive detection of Chinese harmful memes.We introduce ToxiCN MM, the first Chinese harmful meme dataset, which consists of 12,000 samples with fine-grained annotations for meme types. Additionally, we propose a baseline detector, Multimodal Knowledge Enhancement (MKE), designed to incorporate contextual information from meme content, thereby enhancing the model's understanding of Chinese memes.In the evaluation phase, we conduct extensive quantitative experiments and qualitative analyses on multiple baselines, including LLMs and our MKE. Experimental results indicate that detecting Chinese harmful memes is challenging for existing models, while demonstrating the effectiveness of MKE.

Facial perception is of significant influence on the positions of people in social networks. Particularly, students’ facial traits can affect their social centrality in educational settings (e.g., students looking intelligent can attract more friends). However, in educational environments, the social biases associated with appearances have alarming consequences, and little research has been done to investigate the effect of facial perception on social networks. Therefore, it is necessary to comprehensively analyze the influence of perceived facial traits on students’ status in social interaction. In this article, we explore the effect of facial perception on the social centrality of students in social networks. Because students’ social centrality is based on both their study ability and facial traits, this study does a comparative analysis of how facial perception and academic performance influence the social centrality of students. Subsequently, the experimental results demonstrate that facial perception, as well as academic performance, closely correlates with the social centrality of students. Finally, this study contributes to a comprehensive and deep understanding of social networks by analyzing facial trait-based social biases.

Dongyu Zhang, Minghao Zhang, Heting Zhang, Liang Yang, Hongfei Lin. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Mental health is an integral part of human health and well-being. Unhealthy mentality leads to serious consequences such as self-mutilation and suicide, especially for college students. While the literature focused on analysing the relationship between mental health and a single factor such as personality or behavior, accurate prediction is yet to be achieved due to the lack of cross-dimensional analysis and multi-dimensional joint prediction. To this end, this work proposes leveraging multiple factors from three crucial dimensions of mental health: behaviors, personality, and social networks. We recruited 490 college students, and collected their behavioral records from smart cards. In addition, we extracted their psychological traits from questionnaires, and social networks by conducting the survey on the nominating community members. We created a neural network-based model to integrate behavioral, psychological, and social network factors to predict mental health problems. The experimental results verify the efficacy of the proposed model, and demonstrate that the classification model of various factors effectively predicts the students’ mental issues.

Metaphor plays an important role in human communication, which often conveys and evokes sentiments. Numerous approaches to sentiment analysis of metaphors have thus gained attention in natural language processing (NLP). The primary focus of these approaches is on linguistic features and text rather than other modal information and data. However, visual features such as facial expressions also play an important role in expressing sentiments. In this paper, we present a novel neural network approach to sentiment analysis of metaphorical expressions that combines both linguistic and visual features and refer to it as the multimodal model approach. For this, we create a Chinese dataset, containing textual data from metaphorical sentences along with visual data on synchronized facial images. The experimental results indicate that our multimodal model outperforms several other linguistic and visual models, and also outperforms the state-of-the-art methods. The contribution is realized in terms of novelty of the approach and creation of a new, sizeable, and scarce dataset with linguistic and synchronized facial expressive image data. The dataset is particularly useful in languages other than English and the approach addresses one of the most challenging NLP issue: sentiment analysis in metaphor.

The failure of landing a job for college students could cause serious social consequences such as drunkenness and suicide. In addition to academic performance, unconscious biases can become one key obstacle for hunting jobs for graduating students. Thus, it is necessary to understand these unconscious biases so that we can help these students at an early stage with more personalized intervention. In this paper, we develop a framework, i.e., MAYA (Multi-mAjor emploYment stAtus) to predict students' employment status while considering biases. The framework consists of four major components. Firstly, we solve the heterogeneity of student courses by embedding academic performance into a unified space. Then, we apply a generative adversarial network (GAN) to overcome the class imbalance problem. Thirdly, we adopt Long Short-Term Memory (LSTM) with a novel dropout mechanism to comprehensively capture sequential information among semesters. Finally, we design a bias-based regularization to capture the job market biases. We conduct extensive experiments on a large-scale educational dataset and the results demonstrate the effectiveness of our prediction framework.

Dongyu Zhang, Heting Zhang, Xikai Liu, Hongfei Lin, Feng Xia. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Heterographic pun plays a critical role in human writing and literature, which usually has a similar sounding or spelling structure. It is important and difficult research to recognize the heterographic pun because of the ambiguity. However, most existing methods for this task only focus on designing features with rule-based or machine learning methods. In this paper, we propose an end-to-end computational approach - Pronunciation Spelling Understanding Gated Attention (PSUGA) network. For pronunciation, we exploit the hierarchical attention model with phoneme embedding. While for spelling, we consider the character-level, word-level, tag-level, position-level and contextual-level embedding with attention model. To deal with the two parts, we present a gated attention mechanism to control the information integration. We have conducted extensive experiments on SemEval2017 task7 and Pun of the Day datasets. Experimental results show that our approach significantly outperforms state-of-the-art methods.

Facial appearance matters in social networks. Individuals frequently make trait judgments from facial clues. Although these face-based impressions lack the evidence to determine validity, they are of vital importance, because they may relate to human network-based social behavior, such as seeking certain individuals for help, advice, dating, and cooperation, and thus they may relate to centrality in social networks. However, little to no work has investigated the apparent facial traits that influence network centrality, despite the large amount of research on attributions of the central position including personality and behavior. In this paper, we examine whether perceived traits based on facial appearance affect network centrality by exploring the initial stage of social network formation in a first-year college residential area. We took face photos of participants who are freshmen living in the same residential area, and we asked them to nominate community members linking to different networks. We then collected facial perception data by requiring other participants to rate facial images for three main attributions: dominance, trustworthiness, and attractiveness. Meanwhile, we proposed a framework to discover how facial appearance affects social networks. Our results revealed that perceived facial traits were correlated with the network centrality and that they were indicative to predict the centrality of people in different networks. Our findings provide psychological evidence regarding the interaction between faces and network centrality. Our findings also offer insights in to a combination of psychological and social network techniques, and they highlight the function of facial bias in cuing and signaling social traits. To the best of our knowledge, we are the first to explore the influence of facial perception on centrality in social networks.

BACKGROUND: The number of biomedical research articles have increased exponentially with the advancement of biomedicine in recent years. These articles have thus brought a great difficulty in obtaining the needed information of researchers. Information retrieval technologies seek to tackle the problem. However, information needs cannot be completely satisfied by directly introducing the existing information retrieval techniques. Therefore, biomedical information retrieval not only focuses on the relevance of search results, but also aims to promote the completeness of the results, which is referred as the diversity-oriented retrieval. RESULTS: We address the diversity-oriented biomedical retrieval task using a supervised term ranking model. The model is learned through a supervised query expansion process for term refinement. Based on the model, the most relevant and diversified terms are selected to enrich the original query. The expanded query is then fed into a second retrieval to improve the relevance and diversity of search results. To this end, we propose three diversity-oriented optimization strategies in our model, including the diversified term labeling strategy, the biomedical resource-based term features and a diversity-oriented group sampling learning method. Experimental results on TREC Genomics collections demonstrate the effectiveness of the proposed model in improving the relevance and the diversity of search results. CONCLUSIONS: The proposed three strategies jointly contribute to the improvement of biomedical retrieval performance. Our model yields more relevant and diversified results than the state-of-the-art baseline models. Moreover, our method provides a general framework for improving biomedical retrieval performance, and can be used as the basis for future work.

Yufeng Diao, Hongfei Lin, Di Wu, Liang Yang, Kan Xu, Zhihao Yang, Jian Wang, Shaowu Zhang, Bo Xu, Dongyu Zhang. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.