Recently, Few-shot Learning (FSL) has become a popular task that aims to recognize new classes from only a few labeled examples and has been widely applied in fields such as natural science, remote sensing, and medical images. However, most existing methods focus only on the visual modality and compute prototypes directly from raw support images, which lack comprehensive and rich multimodal information. To address these limitations, we propose a novel Multimodal Prototype Augmentation FSL framework called MPA, including LLM-based Multi-Variant Semantic Enhancement (LMSE), Hierarchical Multi-View Augmentation (HMA), and an Adaptive Uncertain Class Absorber (AUCA). LMSE leverages large language models to generate diverse paraphrased category descriptions, enriching the support set with additional semantic cues. HMA exploits both natural and multi-view augmentations to enhance feature diversity (e.g., changes in viewing distance, camera angles, and lighting conditions). AUCA models uncertainty by introducing uncertain classes via interpolation and Gaussian sampling, effectively absorbing uncertain samples. Extensive experiments on four single-domain and six cross-domain FSL benchmarks demonstrate that MPA achieves superior performance compared to existing state-of-the-art methods across most settings. Notably, MPA surpasses the second-best method by 12.29% and 24.56% in the single-domain and cross-domain setting, respectively, in the 5-way 1-shot setting.

Collaborative reasoning with multiple agents offers the potential for more robust and diverse problem-solving. However, existing approaches often suffer from homogeneous agent behaviors and lack of reflective and rethinking capabilities. We propose Multi-Agent Personality Shaping ((MAPS), a novel framework that enhances reasoning through agent diversity and internal critique. Inspired by the Big Five personality theory, MAPS assigns distinct personality traits to individual agents, shaping their reasoning styles and promoting heterogeneous collaboration. To enable deeper and more adaptive reasoning, MAPS introduces a Critic agent that reflects on intermediate outputs, revisits flawed steps, and guides iterative refinement. This integration of personality-driven agent design and structured collaboration improves both reasoning depth and flexibility. Empirical evaluations across three benchmarks demonstrate the strong performance of MAPS, with further analysis confirming its generalizability across different large language models and validating the benefits of multi-agent collaboration.

Large language models (LLMs) typically operate in a question-answering paradigm, where the quality of the input prompt critically affects the response. Automated Prompt Optimization (APO) aims to overcome the cognitive biases of manually crafted prompts and explore a broader prompt design space. However, existing APO methods often suffer from rigid template structures and inefficient exploration in the prompt space. To this end, we propose a Multi-Agent Adaptive Reasoning with Socratic guidance framework (MARS) for APO. MARS consists of five complementary agents and formulates the optimization process as a Partially Observable Markov Decision Process (POMDP), enabling adaptive prompt refinement through explicit state modeling and interactive feedback. Specifically, a Planner agent generates flexible optimization trajectories, a Teacher-Critic-Student triad engages in Socratic-style dialogue to iteratively optimize the prompt based on pseudo-gradient signals in the text space, and a Target agent executes the prompt in downstream tasks to provide performance feedback. MARS integrates reasoning, feedback, and state transition into a unified hidden-state evolution process, improving both the effectiveness and interpretability of optimization. Extensive experiments on multiple datasets demonstrate that MARS outperforms existing APO methods in terms of optimization performance, search efficiency, and interpretability.

Hang Yan, Fangzhi Xu, Rongman Xu, Yifei Li, Jian Zhang, Haoran Luo, Xiaobao Wu, Anh Tuan Luu, Haiteng Zhao, Qika Lin, Jun Liu. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Large language models (LLMs) have achieved remarkable success across a wide range of applications especially when augmented by external knowledge through retrieval-augmented generation (RAG). Despite their widespread adoption, recent studies have shown that LLMs often struggle to perform faithful reasoning when conflicting knowledge is retrieved. However, existing work primarily focuses on conflicts between external knowledge and the parametric knowledge of LLMs, leaving conflicts across external knowledge largely unexplored. Meanwhile, modern RAG systems increasingly emphasize the integration of unstructured text and (semi-)structured data like knowledge graphs (KGs) to improve knowledge completeness and reasoning faithfulness. To address this gap, we introduce ConflictQA, a novel benchmark that systematically instantiates conflicts between textual evidence and KG evidence. Extensive evaluations across representative LLMs reveal that, facing such cross-source conflicts, LLMs often fail to identify reliable evidence for correct reasoning. Instead, LLMs become more sensitive to prompting choices and tend to rely exclusively on either KG or textual evidence, resulting in incorrect responses. Based on these findings, we further propose XoT, a two-stage explanation-based thinking framework tailored for reasoning over heterogeneous conflicting evidence, and verify its effectiveness with extensive experiments.

Large language models (LLMs) have emerged as a promising paradigm for recommender systems, due to their powerful capabilities in global knowledge integration and reasoning. However, LLMs are inherently prone to confirmation bias -- the tendency to favor information that reinforces users' existing views -- which leads to an overemphasis on previously shown viewpoints and ignores diverse user beliefs for recommendations. To address this issue, in this paper, we propose SCoTRec, a social chain-of-thought reasoning framework for recommendation. SCoTRec first constructs sentiment-aware user profiles by extracting sentiment terms from user reviews. It then incorporates users' social sentiment information into the social chain-of-thought reasoning units to improve recommendations. In particular, we categorize the social chain-of-thought into sentiment-based pathways and apply human evaluation operations -- backtracking, discarding, retaining, and aggregating -- to simulate nuanced sentiment cognition and interpersonal influence, effectively alleviating confirmation bias. Extensive experiments on four benchmark datasets demonstrate the effectiveness of SCoTRec in alleviating confirmation bias and improving recommendations.

Deep learning has made remarkable progress in medical image analysis, yet its black-box nature still limits interpretability and clinician trust. Concept-based modeling offers a promising direction for explainable AI by integrating human-understandable concepts. However, existing approaches typically rely on global concept annotations and infer diagnosis based solely on the presence or absence of individual concepts. This oversimplified paradigm ignores the rich relationships among concepts and their causal influence on disease outcomes. To overcome these limitations, we propose the Concept Relationship Embedding Model (CREM) for interpretable medical diagnosis. CREM mirrors coarse-to-fine clinical reasoning by first extracting fine-grained subregional concepts, then explicitly encoding their relationships as a concept interaction graph, and finally performing causal inference between concepts and diagnoses to enable reliable and transparent diagnostic predictions. We evaluate CREM on four public medical imaging benchmarks, where it achieves state-of-the-art performance on both concept recognition and disease classification tasks, while exhibiting improved robustness, label efficiency, and interpretability. Furthermore, we deploy CREM as an interactive web-based demo that allows clinicians to visualize concept activations, trace diagnostic reasoning paths, and iteratively refine concept cues, facilitating effective human-in-the-loop decision-making.

Large language models (LLMs), which serve as a bridge between pre-training and task-specific adaptation, achieve state-of-the-art performance across several downstream tasks through full-parameter fine-tuning (FPFT). However, with the continuous growth of model parameter scales, FPFT requires substantial computational resources, which limits its practicality. Consequently, there is a growing shift toward parameter-efficient fine-tuning (PEFT) methods that update only a limited subset of model parameters, markedly reducing resource consumption. Although this paradigm fosters accelerated research advancements, it also introduces security risks. Empirical studies demonstrate that if LLM weights are backdoored, the backdoors can still be activated even after fine-tuning leveraging PEFT algorithms. To address the aforementioned issue, in this paper, we introduce a novel Uniform Fusion of multiple LoRA Experts algorithm, named UniFLE, designed to defend against backdoor attacks. Specifically, the UniFLE algorithm pioneers the insertion of multiple LoRA experts into the MLP blocks to expand the updatable feature subspace during fine-tuning, and fuses all experts to decouple backdoor features. Additionally, to enhance the diversity of LoRA experts, we introduce a diversity regularization loss that constrains correlations between different experts and encourages them to learn more novel features. The theoretical analysis demonstrates that the UniFLE algorithm effectively decreases the mutual information between the model's intermediate representations and the backdoor representations. To validate the effectiveness of the UniFLE algorithm, we conduct experiments across four tasks, four state-of-the-art LLMs, and three backdoor attack methods. The results consistently demonstrate that our UniFLE algorithm effectively defends against backdoor attacks while preserving model performance. We aspire for our method to enhance model security and contribute to the advancement of the LLM community.

Early disease diagnosis is critical for timely clinical intervention and treatment. Intelligent models have shown significant potential in addressing the challenges of misdiagnosis, especially given the shortage of experienced experts. However, there exists complexity of ultrasound image information, and subtle differences between positive and negative samples, combined with limited disease data, pose challenges in feature extraction, class imbalance, and the lack of representative positive class prototypes. To this end, we propose a large vision model distillation framework with fuzzy perception to improve rare disease diagnosis. Specifically, we first fine-tune the pre-trained Medical Segment Anything Model (MedSAM) to adapt it to the target domain. Through image augmentation and latent feature relationship distillation, we enhance feature extraction robustness and reduce inter-class ambiguity, which helps mitigate the impact of class imbalance on the lightweight student model. Second, as imaging style differences in ultrasound images are often more pronounced than subtle variations between positive and negative samples, we construct image style subsets and introduce a fuzzy style matching strategy to perceive these differences. Finally, we combine features from both the differential perception and semantic enhancement branches to strengthen disease classification. Extensive experiments on an internal fetal spina bifida dataset and three widely used imbalanced benign-malignant medical datasets demonstrate the effectiveness of the proposed method.

Retrieval-Augmented Generation (RAG) offers a promising solution to the limitations of static knowledge and hallucinations in Large Language Models (LLMs). While prior research has introduced numerous enhancements to RAG systems, a significant challenge remains under-explored: the potential conflict between external retrievals and LLMs' internal priors, which can undermine the quality of generated outputs. To tackle this issue, we present theEpitome-AugmentedGeneration (EAG) framework, which strategically aligns queries, external retrievals, and internal priors to produce high-quality LLM generations by selecting fuzzy inputs. EAG employs two novel lightweight modules, Criticism and Distillation, allowing traditional RAGs to be upgraded to EAGs without the need for specialized training data. Extensive experiments on five datasets across general and medical domains, including both open-ended and closed-ended tasks, validate the effectiveness of EAG. Our framework achieves substantial F1 score improvements: 7.03%, 23.35%, and 21.58% over baseline RAGs in medical QA tasks, 11.80% in law domain, 7.95% in finance domain, and 4.13% and 5.16% in general domain. Beyond performance gains, our study delves into the interplay between LLMs' internal priors and external retrievals, uncovering key principles that govern generation quality and providing valuable insights for future retrieval-augmented frameworks.

Multimodal survival analysis aims to combine heterogeneous data sources to improve the prediction quality of survival outcomes. However, this task is particularly challenging due to high heterogeneity and noise across data sources. Additionally, the exact survival time is often censored (partially known) due to incomplete event observation. To address the above challenges, we propose a novel interpretable evidential multimodal survival fusion model, EsurvFusion. This model is designed to combine multimodal data at the decision level using Epistemic Random Fuzzy Sets that jointly handle both data and model uncertainty while incorporating modality-level reliability. Specifically, EsurvFusion first models unimodal data with newly introduced Gaussian random fuzzy numbers, producing possible unimodal survival predictions along with corresponding aleatory and epistemic uncertainty. It then estimates modality-level reliability through a reliability discounting layer to correct the misleading impact of noisy data modalities. Finally, a multimodal evidence fusion layer is introduced to combine the discounted predictions, revealing modality-level influence based on the learned reliability coefficients. Extensive experiments on four multimodal cancer survival datasets demonstrate the effectiveness of our model in handling highly heterogeneous data, establishing a new state-of-the-art performance on several benchmarks.

Modelling patient trajectories from longitudinal electronic health records (EHRs) is crucial for early chronic disease prediction. Foundation models (FMs), benefiting from the computational power and generalization abilities, offer a promising direction towards understanding patient health progression. However, key challenges of adopting FMs in clinical decisions remain in (1) incorporating multi-modal EHR data into an FM effectively for unified patient representations and (2) ensuring model generalizability across various clinical domains with distribution shifts. To address these challenges, we propose MsHeCare, an FM-based, two-stage learning framework integrating multi-modal representation learning and multi-source domain adaptation (MSDA). In the pre-training stage, MsHeCare performs self-supervised contrastive learning and masked language modelling tasks to mitigate semantic biases across text-based diagnostic and treatment sequences. A cross-attention mechanism further fuses these temporal features with structured static demographic information, enhancing personalized patient representations. In the fine-tuning stage, MsHeCare incorporates an MSDA framework with a novel source importance estimation method, facilitating adaptive knowledge transfer across domains and improving model generalizability. Experiments on two real-world EHR datasets demonstrate that MsHeCare significantly outperforms single-domain baselines and state-of-the-art MSDA methods. Furthermore, we validate the robustness of MsHeCare across varying target-domain data sizes and demonstrate its alignment with clinical practices through case studies, underscoring its potential for multi-modal, domain-adaptive predictive healthcare systems.

Accurate identification of neurological disorders such as Alzheimer's disease (AD), Parkinson's disease (PD), and Autism Spectrum Disorder (ASD) is challenging due to subtle early-stage symptoms and heterogeneous brain dynamics. Resting-state functional MRI (rs-fMRI) enables the construction of functional brain networks, where Graph Neural Networks (GNNs) have shown promise for disease classification. However, existing GNN-based methods face three key limitations: correlation-based graph construction introduces noise and negative edges; domain knowledge about brain regions is ignored; and demographic or clinical metadata are fused through simplistic encodings. To overcome these limitations, we propose BrainPrompt+, a knowledge-guided framework that integrates Large Language Models (LLMs) with multi-level natural language prompts. Five types of prompts are introduced: spectral (frequency-domain BOLD features), spatial (inter-ROI connectivity), ROI (anatomical and functional knowledge), disease (progression stages), and subject (demographic context). These prompts are encoded by a frozen LLM and incorporated into a GNN pipeline, unifying imaging, clinical, and external knowledge in a semantically enriched and interpretable manner. Experiments on three rs-fMRI datasets show that BrainPrompt+ consistently outperforms state-of-the-art baselines, achieving accuracy gains of up to 8.93%. Biomarker analysis further demonstrates that the highlighted ROIs align with established neuroscience findings, confirming the interpretability of the model. BrainPrompt+ thus establishes a flexible and generalizable paradigm for knowledge-guided brain network analysis. The source code is available at https://github.com/AngusMonroe/BrainPromptPlus.

Kai He, Yucheng Huang, Wenqing Wang, Delong Ran, Dongming Sheng, Junxuan Huang, Qika Lin, Jiaxing Xu, Wenqiang Liu, Mengling Feng. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Qika Lin, Tianzhe Zhao, Kai He, Zhen Peng, Fangzhi Xu, Ling Huang, Jingying Ma, Mengling Feng. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Cardiovascular disease diagnosis via ultrasound screening relies on manually measured metrics and the experience level of human experts, which is time-consuming and may overlook subtle cross-anatomical pathological patterns. Recent vision-language models offer end-to-end diagnostic potential but lack mechanisms to handle heterogeneous multi-modal, multiview ultrasound data while preserving modality-specific semantics. To fill this gap, we propose an end-to-end framework called MMVL that directly fuses raw ultrasound sequences from diverse anatomical regions, bypassing intermediate measurements, and enabling direct diagnosis. We design lightweight adapters for domain-specific multi-modal feature fusion and refinement, a gating mechanism that dynamically reweights modality importance based on global context, and disease-aware prompt-guided classification. MMVL ensures robust performance across both common and rare conditions. The proposed multi-view, multimodal vision-language framework enables end-to-end cardiovascular disease diagnosis with a 10.9% accuracy gain, and opens a new avenue for automated and generalizable diagnostic solutions.

Medical AI has achieved strong predictive performance, yet most systems remain limited by shallow reasoning, poor transparency, and weak generalisation in safety-critical settings. Neurosymbolic AI offers a path beyond these constraints by combining neural models' ability to learn from complex clinical data with the explicit structure, logic, and domain knowledge of symbolic methods. This article examines how neurosymbolic approaches can address core challenges in healthcare AI through five key areas: hybrid reasoning that unifies learning and logic; symbol grounding that links internal representations to clinically meaningful concepts; clinical interpretability that exposes reasoning steps; human-integrated decision-making that keeps clinicians in control; and knowledge-driven diagnosis that incorporates guidelines, ontologies, and causal understanding. Together, these elements outline how neurosymbolic AI can support systems that are not only accurate but also transparent, clinically aligned, and robust in complex or data-sparse scenarios. Advancing this paradigm will require collaboration across AI research, clinical practice, and knowledge engineering, as well as governance mechanisms that ensure fairness and accountability. Neurosymbolic AI thus represents a promising direction for building trustworthy, knowledge-rich intelligence in healthcare.

Large-scale multi-modal models have demonstrated remarkable performance across various visual recognition tasks by leveraging extensive paired multi-modal training data. However, in real-world applications, the presence of missing or incomplete modality inputs often leads to significant performance degradation. Recent research has focused on prompt-based strategies to tackle this issue; however, existing methods are hindered by two major limitations: (1) static prompts lack the flexibility to adapt to varying missing-data conditions, and (2) basic prompt-tuning methods struggle to ensure reliable performance when critical modalities are missing.To address these challenges, we propose a novel Synergistic Prompting (SyP) framework for robust visual recognition with missing modalities. The proposed SyP introduces two key innovations: (I) a Dynamic Adapter, which computes adaptive scaling factors to dynamically generate prompts, replacing static parameters for flexible multi-modal adaptation, and (II) a Synergistic Prompting Strategy, which combines static and dynamic prompts to balance information across modalities, ensuring robust reasoning even when key modalities are missing. The proposed SyP achieves significant performance improvements over existing approaches across three widely-used visual recognition datasets, demonstrating robustness under diverse missing rates and conditions. Extensive experiments and ablation studies validate its effectiveness in handling missing modalities, highlighting its superior adaptability and reliability.

Knowledge Base Question Answering (KBQA) aims to answer natural language questions with a large-scale structured knowledge base (KB). Despite advancements with large language models (LLMs), KBQA still faces challenges in weak KB awareness, imbalance between effectiveness and efficiency, and high reliance on annotated data. To address these challenges, we propose KBQA-o1, a novel agentic KBQA method with Monte Carlo Tree Search (MCTS). It introduces a ReAct-based agent process for stepwise logical form generation with KB environment exploration. Moreover, it employs MCTS, a heuristic search method driven by policy and reward models, to balance agentic exploration’s performance and search space. With heuristic exploration, KBQA-o1 generates high-quality annotations for further improvement by incremental fine-tuning. Experimental results show that KBQA-o1 outperforms previous low-resource KBQA methods with limited annotated data, boosting Llama-3.1-8B model’s GrailQA F1 performance to 78.5% compared to 48.5% of the previous sota method with GPT-3.5-turbo. Our code is publicly available.

Emotion recognition from EEG signals is crucial for understanding complex brain dynamics. Existing methods typically rely on static frequency bands and graph convolutional networks (GCNs) to model brain connectivity. However, EEG signals are inherently non-stationary and exhibit substantial individual variability, making static-band approaches inadequate for capturing their dynamic properties. Moreover, spatial-temporal dependencies in EEG often lead to feature degradation during node aggregation, ultimately limiting recognition performance. To address these challenges, we propose the Spatial-Temporal Electroencephalograph Collaboration framework (Stella). Our approach introduces an Adaptive Bands Selection module (ABS) that dynamically extracts low- and high-frequency components, generating dual-path features comprising phase brain networks for connectivity modeling and time-series representations for local dynamics. To further mitigate feature degradation, the Fourier Graph Operator (FGO) operates in the spectral domain, while the Spatial-Temporal Encoder (STE) enhances representation stability and density. Extensive experiments on benchmark EEG datasets demonstrate that Stella achieves state-of-the-art performance in emotion recognition, offering valuable insights for graph-based modeling of non-stationary neural signals. The code is available at https://github.com/sun2017bupt/EEGBrainNetwork.

Sleep staging is critical to assess sleep quality and diagnose disorders. Despite advancements in artificial intelligence enabling automated sleep staging, significant challenges remain: (1) Simultaneously extracting prominent temporal and spatial sleep features from multi-channel raw signals, including characteristic sleep waveforms and salient spatial brain networks. (2) Capturing the spatial-temporal coupling patterns essential for accurate sleep staging. To address these challenges, we propose a novel framework named ST-USleepNet, comprising a spatial-temporal graph construction module (ST) and a U-shaped sleep network (USleepNet). The ST module converts raw signals into a spatial-temporal graph based on signal similarity, temporal, and spatial relationships to model spatial-temporal coupling patterns. The USleepNet employs a U-shaped structure for both the temporal and spatial streams, mirroring its original use in image segmentation to isolate significant targets. Applied to raw sleep signals and graph data from the ST module, USleepNet effectively segments these inputs, simultaneously extracting prominent temporal and spatial sleep features. Testing on three datasets demonstrates that ST-USleepNet outperforms existing baselines, and model visualizations confirm its efficacy in extracting prominent sleep features and temporal-spatial coupling patterns across various sleep stages. The code is available at https://github.com/Majy-Yuji/ST-USleepNet.

Most self-supervised graph learning studies typically follow an offline training paradigm, assuming that all data are readily available.This assumption, however, is not always tenable in real-world scenarios as many graph data are generated continuously.Although several continual graph learning models have emerged and achieved empirical success, they almost all rely on external supervision, making it difficult to adapt to applications with a large amount of unlabeled data from the wild.To be honest, research on self-supervised continual graph learning is still surprisingly in its infancy.Therefore, we select several well-known self-supervised graph embedding models as representatives and explore whether they are resistant to catastrophic forgetting in a continual learning setting.Empirical studies find that self-supervised representation models may be potentially better continual learners than supervised counterparts.Driven by this advantage, we propose a self-supervised continual graph representation learning framework based on adaptive spaced replay on node proxies, named Trace.Inspired by the Complementary Learning System theory, Trace employs a dual-system architecture to simulate the functionality and cooperation of the hippocampus and neocortex in the brain.Among them, the fastlearning system efficiently encodes the current input graph to acquire new knowledge and adaptively extracts node proxies from it as important knowledge cached into the memory through progressive clustering.Drawing inspiration from the Ebbinghaus forgetting curve, the slow-learning system implements adaptive spaced replay based on the memory retention rate of each preceding task instead of the widely used consecutive replay scheme for promising flexibility and efficiency.Experiments under task-incremental and class-incremental learning settings on multiple datasets corroborate

The natural combination of intricate topological structures and rich textual information in text-attributed graphs (TAGs) opens up a novel perspective for graph anomaly detection (GAD). However, existing GAD methods primarily focus on designing complex optimization objectives within the graph domain, overlooking the complementary value of the textual modality, whose features are often encoded by shallow embedding techniques, such as bag-of-words or skip-gram, so that semantic context related to anomalies may be missed. To unleash the enormous potential of textual modality, large language models (LLMs) have emerged as promising alternatives due to their strong semantic understanding and reasoning capabilities. Nevertheless, their application to TAG anomaly detection remains nascent, and they struggle to encode high-order structural information inherent in graphs due to input length constraints. For high-quality anomaly detection in TAGs, we propose CoLL, a novel framework that combines LLMs and graph neural networks (GNNs) to leverage their complementary strengths. CoLL employs multi-LLM collaboration for evidence-augmented generation to capture anomaly-relevant contexts while delivering human-readable rationales for detected anomalies. Moreover, CoLL integrates a GNN equipped with a gating mechanism to adaptively fuse textual features with evidence while preserving high-order topological information. Extensive experiments demonstrate the superiority of CoLL, achieving an average improvement of 13.37% in AP. This study opens a new avenue for incorporating LLMs in advancing GAD.

Standard Retrieval-Augmented Generation (RAG) relies on chunk-based retrieval, whereas GraphRAG advances this approach by graph-based knowledge representation. However, existing graph-based RAG approaches are constrained by binary relations, as each edge in an ordinary graph connects only two entities, limiting their ability to represent the n-ary relations (n >= 2) in real-world knowledge. In this work, we propose HyperGraphRAG, the first hypergraph-based RAG method that represents n-ary relational facts via hyperedges. HyperGraphRAG consists of a comprehensive pipeline, including knowledge hypergraph construction, retrieval, and generation. Experiments across medicine, agriculture, computer science, and law demonstrate that HyperGraphRAG outperforms both standard RAG and previous graph-based RAG methods in answer accuracy, retrieval efficiency, and generation quality.

Diagram question answering (DQA) is a challenging task that requires models to combine with domain-specific knowledge and reason over the diagrams to answer questions. Multimodal Large Language Models (MLLMs) have recently made notable strides in combining textual and visual information, emerging as a promising solution for addressing the DQA task. However, they still encounter challenges in deliberate multimodal reasoning over the fine-grained visual details of content-rich and knowledge-grounded diagrams. The tight interweaving of visual and textual reasoning for MLLMs is also susceptible to hallucinations. To overcome these limitations, we propose a Glance-then-Focus Multimodal Reasoning framework named GlFoMR for DQA, which features a flexible architecture for comprehensive visual and text interaction. Firstly, the diagram is parsed into a hierarchical structure spanning different granularities including isolated single-object, object-group, and whole-diagram. Subsequently, the Glance-Plan and Focus-Reason stages collaborate to decouple the complex reasoning process. Glance-Plan first generates a preliminary plan by glancing at the multimodal context, specifying sub-goals related to knowledge extraction, visual perception, and visual reasoning. Based on these sub-goals, Focus-Reason further integrates domain-specific knowledge and visual details to enable more deliberate reasoning. The parsed multi-granularity diagram information is seamlessly incorporated into the corresponding sub-goal achievement process, enhancing the perception and reasoning capabilities of MLLMs for better DQA performance. Extensive experimental results on four DQA datasets demonstrate that GlFoMR achieves substantial improvements, showcasing its potential to advance the development of multimodal reasoning.

Continual knowledge graph embedding (CKGE) has gained wide attention for managing dynamic knowledge graphs (KGs), which are continuously updated with new facts. Unlike traditional methods designed for static KGs, CKGE enables incremental updates to KG embeddings to accommodate new facts while retaining previously learned knowledge. Despite these advancements, current CKGE studies and benchmarks primarily focus on handling the increasing scale of data while overlooking changes in graph patterns. These changes, altering the graph structure of KGs, are referred to as pattern shifts in this paper. Pattern shifts frequently arise as new facts are added, introducing significant challenges to the stability and adaptability of CKGE methods. To address this gap, we introduce a suite of novel and challenging benchmarks, called PS-CKGE, specifically designed to evaluate CKGE methods under pattern shifts, where logic rules are utilized to capture and manage structural changes in dynamic KGs. Through these benchmarks, we comprehensively evaluate current CKGE methods in terms of their overall performance, resistance to catastrophic forgetting, and adaptability to new knowledge. The results show that pattern shifts not only exacerbate their risk of catastrophic forgetting but also impair their adaptability, usually with greater performance degradation over triples associated with more significant changes.

Multimodal aspect-based sentiment classification (MASC) is an emerging task due to an increase in user-generated multimodal content on social platforms, aimed at predicting sentiment polarity toward specific aspect targets (i.e., entities or attributes explicitly mentioned in text-image pairs). Despite extensive efforts and significant achievements in existing MASC, substantial gaps remain in understanding fine-grained visual content and the cognitive rationales derived from semantic content and impressions (cognitive interpretations of emotions evoked by image content). In this study, we present Chimera: acognitive and aesthetic sentiment causality understanding framework to derive fine-grained holistic features of aspects and infer the fundamental drivers of sentiment expression from both semantic perspectives and affective-cognitive resonance (the synergistic effect between emotional responses and cognitive interpretations). The framework aligns visual patches with words, extracts coarse and fine-grained visual features, translates them into textual descriptions, and uses LLM-generated sentimental causes and impressions to boost sensitivity to affective cues. Experiments on MASC datasets show the model's effectiveness and greater flexibility compared to LLMs like GPT-4o. We have publicly released the complete implementation and dataset athttps://github.com/Xillv/Chimera

Multimodal medical applications have garnered considerable attention due to their potential to offer comprehensive and robust support for medical assistance. Specifically, within this domain, difference-aware medical Visual Question Answering (VQA) has emerged as a topic of increasing interest that enables the recognition of changes in physical conditions over time when compared to previous states and provides customized suggestions accordingly. However, it is challenging because samples usually exhibit characteristics of complexity, diversity, and inherent noise. Besides, there is a need for multimodal knowledge understanding of the medical domain. The difference-aware setting requiring image comparison further intensifies these situations. To this end, we propose a cross-Modal knowlEdge diffusioN-baseD gEneration netwoRk (MENDER), where the diffusion mechanism with multi-step denoising and knowledge injection from global to local level are employed to tackle the aforementioned challenges, respectively. The diffusion process is to gradually generate answers with the sequence input of questions, random noises for the answer masks and virtual vision prompts of images. The strategy of answer nosing and knowledge cascading is specifically tailored for this task and is implemented during forward and reverse diffusion processes. Moreover, the visual and structure knowledge injection are proposed to learn virtual vision prompts to guide the diffusion process, where the former is realized using a pre-trained medical image-text network and the latter is modeled with spatial and semantic graph structures processed by the heterogeneous graph Transformer models. Experiment results demonstrate the effectiveness of MENDER for difference-aware medical VQA. Furthermore, it also exhibits notable performance in the low-resource setting and conventional medical VQA tasks.

Logical reasoning consistently plays a fundamental and significant role in the domains of knowledge engineering and artificial intelligence. Recently, Large Language Models (LLMs) have emerged as a noteworthy innovation in natural language processing (NLP). However, the question of whether LLMs can effectively address the task of logical reasoning, which requires gradual cognitive inference similar to human intelligence, remains unanswered. To this end, we aim to bridge this gap and provide comprehensive evaluations in this paper. First, to offer systematic evaluations, we select fifteen typical logical reasoning datasets and organize them into deductive, inductive, abductive and mixed-form reasoning settings. Considering the comprehensiveness of evaluations, we include 3 early-era representative LLMs and 4 trending LLMs. Second, different from previous evaluations relying only on simple metrics (e.g.,accuracy), we propose fine-level evaluations in objective and subjective manners, covering both answers and explanations, includinganswer correctness,explain correctness,explain completenessandexplain redundancy. Additionally, to uncover the logical flaws of LLMs, problematic cases will be attributed to five error types from two dimensions, i.e.,evidence selection processandreasoning process. Third, to avoid the influences of knowledge bias and concentrate purely on benchmarking the logical reasoning capability of LLMs, we propose a new dataset with neutral content. Based on the in-depth evaluations, this paper finally forms a general evaluation scheme of logical reasoning capability from six dimensions (i.e.,Correct,Rigorous,Self-aware,Active,OrientedandNo hallucination). It reflects the pros and cons of LLMs and gives guiding directions for future works.

Abnormal subgraph (AS) detection plays a significant role in ensuring the security of many high-impact domains. Unlike node anomaly detection, identifying subgraph anomalies is extremely challenging due to the exponentially large subgraph space caused by various combinations of nodes and edges. Moreover, in the absence of supervisory signals, how to quantify the abnormality of subgraphs poses another pressing challenge. Traditional methods typically rely on handcrafted subgraph anomaly measures, making it hard to handle potential unknown anomalies with limited prior knowledge. Recent deep learning-based techniques are predominantly designed to discover individual node anomalies, which could be suboptimal for AS detection due to the inconsideration of collaborative behaviors between nodes in the subgraph. In fact, existing studies have put very little effort into this task, and even dedicated performance evaluation metrics are not yet available. To address the above challenges and promote related research, in this article, we propose a end-to-end unsupervised subgraph anomaly detection framework (EndSubG), which jointly models subgraph partition and AS detection as a whole instead of treating them as two separate stages. Specifically, EndSubG uncovers potential AS boundaries that violate the Homophily assumption by modeling the edge existence probability, then achieves anomaly-aware graph embedding and subgraph partition based on the refined topology. By forming a coarsened subgraph network, EndSubG picks out subgraph anomalies by learning the "subgraph-vicinity" matching patterns. Additionally, we design an evaluation metric weighted normalized mutual information centered on AS (AS-WNMI) specifically for subgraph anomaly detection, which is a variant of vanilla NMI and quantifies detection performance from both subgraph partition and anomaly recognition. The experimental results on synthetic and real-world datasets corroborate the superiority of end-to-end unsupervised subgraph anomaly detection framework (EndSubG) in terms of area under the curve (AUC), average precision (AP), and AS-WNMI. We also provide an intuitive analysis of the detected subgraphs through visualization for better understanding.

Logical reasoning task has attracted great interest since it was proposed.Faced with such a task, current competitive models, even large language models (e.g., ChatGPT and PaLM 2), still perform badly.Previous promising LMs struggle in logical consistency modeling and logical structure perception.To this end, we model the logical reasoning task by transforming each logical sample into reasoning paths and propose an architecture PathReasoner.It addresses the task from the views of both data and model.To expand the diversity of the logical samples, we propose an atom extension strategy supported by equivalent logical formulas, to form new reasoning paths.From the model perspective, we design a stack of transformer-style blocks.In particular, we propose a path-attention module to joint model in-atom and cross-atom relations with the highorder diffusion strategy.Experiments show that PathReasoner achieves competitive performances on two logical reasoning benchmarks and great generalization abilities.

Fangzhi Xu, Zhiyong Wu, Qiushi Sun, Siyu Ren, Fei Yuan, Shuai Yuan, Qika Lin, Yu Qiao, Jun Liu. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Document-level Event Argument Extraction (DEAE) aims to identify arguments and their specific roles from an unstructured document. The advanced approaches on DEAE utilize prompt-based methods to guide pre-trained language models (PLMs) in extracting arguments from input documents. They mainly concentrate on establishing relations between triggers and entity mentions within documents, leaving two unresolved problems: a) independent modeling of entity mentions; b) document-prompt isolation. To this end, we propose a semantic mention Graph Augmented Model (GAM) to address these two problems in this paper. Firstly, GAM constructs a semantic mention graph that captures relations within and between documents and prompts, encompassing co-existence, co-reference and co-type relations. Furthermore, we introduce an ensemble graph transformer module to address mentions and their three semantic relations effectively. Later, the graph-augmented encoder-decoder module incorporates the relation-specific graph into the input embedding of PLMs and optimizes the encoder section with topology information, enhancing the relations comprehensively. Extensive experiments on the RAMS and WikiEvents datasets demonstrate the effectiveness of our approach, surpassing baseline methods and achieving a new state-of-the-art performance.

Embedding symbolic logical formulas into a low-dimensional continuous space provides an effective way for the Neural-Symbolic system. However, current studies are all constrained by the syntactic structure modeling and fail to preserve intrinsic semantics. To this end, we propose a novel model of Contrastive Graph Representations (ConGR) for logical formulas embedding. Firstly, it introduces a densely connected graph convolutional network (GCN) with an attention mechanism to process syntax parsing graphs of formulas. Secondly, the contrastive instances for each anchor formula are generated by the transformation under the guidance of logical properties. Two types of contrast, global-local and global-global, are carried out to refine formula embeddings with semantic information. Extensive experiments demonstrate that ConGR obtains superior performance against state-of-the-art baselines.

Currently, the increasing information overload on Massive Open Online Courses(MOOCs) inhibits the appropriate choice of learning objects by learners, leading to low efficiency and high dropout rates. However, in MOOC platforms, recommendation network structures that can selectively extract implicit features such as heterogeneous learning preference and knowledge organization of learning objects are still not comprehensively studied. To this end, we propose a learning object recommendation model namely ACGCN based on heterogeneous learning behavior and knowledge graph. By introducing an attention mechanism, information is amplified when updating the representation of the heterogeneous graph, which eliminates the impact of noise and improves the robustness of ACGCN. Experimental results using a real-world dataset revealed that our proposed model has the best performance compared to those of several existing baselines.

Knowledge graph embedding (KGE) methods have achieved great success in handling various knowledge graph (KG) downstream tasks. However, KGE methods may learn biased representations on low-quality KGs that are prevalent in the real world. Some recent studies propose adversarial attacks to investigate the vulnerabilities of KGE methods, but their attackers are target-oriented with the KGE method and the target triples to predict are given in advance, which lacks practicability. In this work, we explore untargeted attacks with the aim of reducing the global performances of KGE methods over a set of unknown test triples and conducting systematic analyses on KGE robustness. Considering logic rules can effectively summarize the global structure of a KG, we develop rule-based attack strategies to enhance the attack efficiency. In particular, we consider adversarial deletion which learns rules, applying the rules to score triple importance and delete important triples, and adversarial addition which corrupts the learned rules and applies them for negative triples as perturbations. Extensive experiments on two datasets over three representative classes of KGE methods demonstrate the effectiveness of our proposed untargeted attacks in diminishing the link prediction results. And we also find that different KGE methods exhibit different robustness to untargeted attacks. For example, the robustness of methods engaged with graph neural networks and logic rules depends on the density of the graph. But rule-based methods like NCRL are easily affected by adversarial addition attacks to capture negative rules.

Logical reasoning task involves diverse types of complex reasoning over text, based on the form of multiple-choice question answering (MCQA). Given the context, question and a set of options as the input, previous methods achieve superior performances on the full-data setting. However, the current benchmark dataset has the ideal assumption that the reasoning type distribution on the train split is close to the test split, which is inconsistent with many real application scenarios. To address it, there remain two problems to be studied: 1) how is the zero-shot capability of the models (train on seen types and test on unseen types)? and 2) how to enhance the perception of reasoning types for the models? For problem 1, we propose a new benchmark for generalized zero-shot logical reasoning, named ZsLR. It includes six splits based on the three type sampling strategies. For problem 2, a type-aware model TaCo is proposed. It utilizes the heuristic input reconstruction and builds a text graph with a global node. Incorporating graph reasoning and contrastive learning, TaCo can improve the type perception in the global representation. Extensive experiments on both the zero-shot and full-data settings prove the superiority of TaCo over the state-of-the-art (SOTA) methods. Also, we experiment and verify the generalization capability of TaCo on other logical reasoning dataset.

Extrapolation reasoning on temporal knowledge graphs (TKGs) aims to forecast future facts based on past counterparts.There are two main challenges: (1) incorporating the complex information, including structural dependencies, temporal dynamics, and hidden logical rules;(2) implementing differentiable logical rule learning and reasoning for explainability.To this end, we propose an explainable extrapolation reasoning framework TEemporal logiCal grapH networkS (TECHS), which mainly contains a temporal graph encoder and a logical decoder.The former employs a graph convolutional network with temporal encoding and heterogeneous attention to embed topological structures and temporal dynamics.The latter integrates propositional reasoning and first-order reasoning by introducing a reasoning graph that iteratively expands to find the answer.A forward message-passing mechanism is also proposed to update node representations, and their propositional and first-order attention scores.Experimental results demonstrate that it outperforms state-of-the-art baselines.

Modeling the dynamics into graph neural networks (GNNs) contributes to the understanding of evolution in dynamic graphs, which helps optimize temporal-spatial representations for real-world dynamic network problems. Empirically, dynamic GNN embedding requires additional temporal encoders, which inevitably introduces additional learning parameters to make dynamic GNNs oversized and inefficient. Furthermore, previous dynamic GNN models are under the same fixed temporal term, which causes the short-temporal optimum. To address these issues, we propose the WinGNN framework to model dynamic graphs, which is realized by a simple GNN model with the meta-learning strategy and a novel mechanism of random gradient aggregation. WinGNN calculates the frame-wise loss of the current snapshot and passes the loss gradient to the next to model graph dynamics without temporal encoders. Then it introduces the randomized sliding-window to acquire the window-aware gradienton consecutive snapshots, and the calculated two types of gradient are aggregated to update the GNN, thereby reducing the parameter size and improving the robustness. Experiments on six public datasets show the advantage of our WinGNN compared with existing baselines, where it has reached the optimum in twenty-two out of twenty-four performance metrics.

Textbook Question Answering (TQA) task requires answering questions by reasoning based on both the given diagrams and text context. There are mainly two challenges for the task. First, the diagrams are different from the natural images. Similar shapes or color blocks may express different semantics and there is also a large intra-topic variation for diagrams. Hence, the characteristics of visual semantic ambiguity and variable visual appearance make the diagram understanding more challenging. Second, for the text, the specific education domain with terminologies exists a great gap with the general domain. Therefore, it is difficult to represent the text semantics effectively using a text encoder pretrained in the general domain. In this paper, we propose a Spatial-Semantic Collaborative Graph Network (SSCGN) for TQA task, which can help enhance the diagram and text understanding and facilitate multimodal reasoning. Specifically, the Spatial-guided Semantic Enhancing (SSE) module fully exploits the spatial and semantic relationships between visual objects and OCR tokens to collaboratively enhance the diagram semantic understanding. Moreover, based on the semantically enhanced region representations of the SSE module, the Fine-grained Spatial-Aware Graph Network (FSA-GN) can help obtain richer relation-aware region representations for joint reasoning by capturing more fine-grained spatial relationships. We further propose multiple self-supervised auxiliary tasks to enhance the initial diagram and text semantic representations by pretraining the diagram encoder and text encoder. Extensive experiments and ablation studies are conducted to validate the effectiveness of SSCGN.

Currently, the non-transparent computing process of deep learning has become a significant reason hindering its further development. The Neural-Symbolic (NS) system formed by integrating logic rules into neural networks has attracted increasing attention owing to its direct interpretability. Embedding symbolic logical formulas into a low-dimensional continuous space provides an effective way for the NS system. However, current studies are all constrained by the modeling ability for its syntactic structure and fail to preserve the intrinsic semantics in embeddings, which causes poor performance on downstream reasoning tasks. To this end, this paper proposes a novel method ofContrastiveGraphRepresentations (ConGR) for logical formulas embedding. First, to improve the modeling ability for the syntactic structure, ConGR introduces a densely connected graph convolutional network (GCN) with an attention mechanism to process syntax parsing graphs of formulas. In this way, discriminative local and global embeddings of formulas are obtained at the syntax level. Second, the contrastive instances (positive or negative) for each anchor formula are generated by the transformation under the guidance of logical properties. To preserve semantic information, two types of contrast, global-local and global-global, are carried out to refine formula embeddings. Extensive experiments demonstrate that ConGR obtains superior performance against state-of-the-art baselines on entailment checking and premise selection datasets.

Few-shot learning (FSL) that acquires new knowledge with little supervision, attracts much attention due to expensive cost of data annotation. Various meta-learning methods have made a great progress for few-shot problem in image and text data. In reality, data samples are not independent but rich in link relations. Large amounts of data exists in the form of graph structure such as citation, social, and biological networks. However, FSL study on graph data is still in its infancy because of the obstacle on extracting meta-knowledge from a meta node classification task. Current research just simply combines the FSL methods experienced in computer vision with node representation models together, but ignores the effect of rich links among support and query nodes in few-shot meta-task. For this issue, we propose a novel Multi-Level Graph Relation Network (MuL-GRN) for the challenging few-shot node classification. MuL-GRN extracts node embeddings through the popular graph neural networks (GNNs). And it includes a relation learning module to mine the deep node relations from three views, namely node-level, global subgraph-level, and local subgraph-level relations. For any two nodes, the node-level relation is computed on their node embeddings, global subgraph-level relation is measured on their subgraph embeddings, and the local subgraph-level relation is mined according to the pairwise node comparison information in their subgraphs. The three-view relation vectors are fused together with an interesting relation fusion module, which measures the importance of relation vector for the current few-shot classification task automatically. Extensive experiments on five real datasets show that MuL-GRN significantly outperforms existing state-of-the-art methods by a large margin.

Massive Open Online Courses (MOOCs) have received unprecedented attention, in which learners can obtain a large number of learning objects anytime and anywhere. However, the increasing information overload on MOOCs inhibits the appropriate choice of learning objects by learners, leading to a low efficiency and high dropout rates in the learning process of this human-computer interaction scenario. E-learning recommendation systems have been studied to present learning objects directly to learners, thereby relieving such problem. However, in MOOC platforms, recommendation network structures which can selectively extract implicit feature such as heterogeneous learning preference and knowledge organization of learning objects are still not comprehensively studied. To this end, we propose a learning object recommendation model based on heterogeneous learning behavior and knowledge graph. To generate a unified representation of each entity and relation, we first propose an Attentive Composition based Graph Convolutional Network (ACGCN). By introducing an attention mechanism, information is amplified when updating the representation of the heterogeneous graph, which eliminates the impact of noise and improves the robustness of the model. Then, a Dense Feature based Operation-Aware Network (DFOAN) is utilized to capture implicit and complex learners’ interactive behaviors, and to further provide a recommendation. Experimental results using two real-world datasets revealed that our proposed model has the best precision, recall, F1, and accuracy scores compared to those of several existing models.

Visual question answering (VQA) is a task that machines should provide an accurate natural language answer given an image and a question about the image. Many studies have found that the current VQA methods are heavily driven by the surface correlation or statistical bias in the training data, and lack sufficient image grounding. To address this issue, we devise a novel end-to-end architecture that uses multitask learning to promote more sufficient image grounding and learn effective multimodality representations. The tasks consist of VQA and our proposed image cloze (IC) task requires machines to fill in the blanks accurately given an image and a textual description of the image. To ensure our model performs sufficient image grounding as much as possible, we propose a novel word-masking algorithm to develop the multimodal IC task based on the part-of-speech of words. Our model predicts the VQA answer and fills in the blanks after the multimodality representation learning that is shared by the two tasks. Experimental results show that our model achieves almost the equivalent, state-of-the-art, second-best performance on the VQA v2.0, VQA-changing priors (CP) v2, and grounded question answering (GQA) datasets, respectively, with fewer parameters and without additional data compared with baselines.

Relation prediction in knowledge graphs (KGs) aims at predicting missing relations in incomplete triples, whereas the dominant embedding paradigm has a restriction on handling unseen entities during testing.In the realworld scenario, the inductive setting is more common because entities in the training process are finite.Previous methods capture an inductive ability by implicit logic in KGs.However, it would be challenging to preciously acquire entity-independent relational semantics of compositional logic rules and to deal with the deficient supervision of logic caused by the scarcity of relational semantics.To this end, we propose a novel graph convolutional network (GCN)-based model LogCo with logical reasoning by contrastive representations.LogCo firstly extracts enclosing subgraphs and relational paths between two entities to supply the entity-independence.Then a contrastive strategy for relational path instances and the subgraph is proposed for the issue of deficient supervision.The contrastive representations are learned for a joint training regime.Finally, prediction results and logic rules for reasoning are attained.Comprehensive experiments on twelve inductive datasets show that LogCo achieves outstanding performance comparing with SOTA inductive baselines.

Relation prediction on knowledge graphs (KGs) aims to infer missing valid triples from observed ones. Although this task has been deeply studied, most previous studies are limited to the transductive setting and cannot handle emerging entities. Actually, the inductive setting is closer to real-life scenarios because it allows entities in the testing phase to be unseen during training. However, it is challenging to precisely conduct inductive relation prediction as there exists requirements of entity-independent relation modeling and discrete logical reasoning for interoperability. To this end, we propose a novel model ConGLR to incorporate context graph with logical reasoning. Firstly, the enclosing subgraph w.r.t. target head and tail entities are extracted and initialized by the double radius labeling. And then the context graph involving relational paths, relations and entities is introduced. Secondly, two graph convolutional networks (GCNs) with the information interaction of entities and relations are carried out to process the subgraph and context graph respectively. Considering the influence of different edges and target relations, we introduce edge-aware and relation-aware attention mechanisms for the subgraph GCN. Finally, by treating the relational path as rule body and target relation as rule head, we integrate neural calculating and logical reasoning to obtain inductive scores. And to focus on the specific modeling goals of each module, the stop-gradient is utilized in the information interaction between context graph and subgraph GCNs in the training process. In this way, ConGLR satisfies two inductive requirements at the same time. Extensive experiments demonstrate that ConGLR obtains outstanding performance against state-of-the-art baselines on twelve inductive dataset versions of three common KGs.

Machine reading comprehension has aroused wide concerns, since it explores the potential of model for text understanding. To further equip the machine with the reasoning capability, the challenging task of logical reasoning is proposed. Previous works on logical reasoning have proposed some strategies to extract the logical units from different aspects. However, there still remains a challenge to model the long distance dependency among the logical units. Also, it is demanding to uncover the logical structures of the text and further fuse the discrete logic to the continuous text embedding. To tackle the above issues, we propose an end-to-end model Logiformer which utilizes a two-branch graph transformer network for logical reasoning of text. Firstly, we introduce different extraction strategies to split the text into two sets of logical units, and construct the logical graph and the syntax graph respectively. The logical graph models the causal relations for the logical branch while the syntax graph captures the co-occurrence relations for the syntax branch. Secondly, to model the long distance dependency, the node sequence from each graph is fed into the fully connected graph transformer structures. The two adjacent matrices are viewed as the attention biases for the graph transformer layers, which map the discrete logical structures to the continuous text embedding space. Thirdly, a dynamic gate mechanism and a question-aware self-attention module are introduced before the answer prediction to update the features. The reasoning process provides the interpretability by employing the logical units, which are consistent with human cognition. The experimental results show the superiority of our model, which outperforms the state-of-the-art single model on two logical reasoning benchmarks.

The widespread communication of academic ideas and research achievements through literature and media has generated massive academic big data. Analyzing such academic big data and discovering knowledge can discover comprehensive and predictive academic intelligence and provide corresponding services, which are valuable for scholars, journals, institutions and governments for career assessment, topic selection, funding management and resource allocation. This paper proposes a heterogeneous knowledge-learning method for understanding and predicting academic impact in the transportation field. The proposed method is illustrated on the academic big data collected from the papers published in 34 transportation journals from 2008 to 2018. We extract four types of features including bibliometric, altmetric, network and semantic features, and build hybrid feature embedding via TransR and Doc2vec that involving domain knowledge. Further, an academic impact prediction model for articles named as Hy-LSTM-Att is proposed, which weighs the hybrid features by the attention mechanism and predicts academic impact with the bi-LSTM recurrent neural networks. Experimental results demonstrate that the proposed Hy-LSTM-Att model outperforms competing shallow and deep learning models. Additionally, the feature ablation experiments illustrate that the four types of features positively contribute to the performance of impact prediction.

Coreference resolution aims at recognizing different forms in a document which refer to the same entity in the real world. Although many models have been proposed and achieved success, there still exist some challenges. Recent models that use recurrent neural networks to obtain mention representations ignore dependencies between spans and their proceeding distant spans, which will lead to predicted clusters that are locally consistent but globally inconsistent. In addition, these models are trained only by maximizing the marginal likelihood of gold antecedent spans from coreference clusters, which will make some gold mentions undetectable and cause unsatisfactory coreference results. To address these challenges, we propose a neural coreference resolution model. It employs mutual attention to take into account the dependencies between spans and their proceeding spans directly (use attention mechanism to capture global information between spans and their proceeding spans). And our model is trained by jointly optimizing mention clustering and imbalanced mention detection, which enables it to detect more gold mentions in a document to make more accurate coreference decisions. Experimental results on the CoNLL-2012 English dataset show that our model can detect the most gold mentions and achieve the state-of-the-art coreference performance compared with baselines.