Retrieval Question Answering (ReQA) is a pivotal task in biomedical natural language processing, where the bi-encoders is a commonly employed solution due to its efficiency in retrieving answers from large candidate pools. However, bi-encoders falls short in capturing fine-grained interactions between questions and answers, a limitation that is even more pronounced in the biomedical field due to the inadequate model training caused by data scarcity. In recent developments, researchers have introduced hard in-batch negative sampling to enhance performance by enriching the training process with informative instances. However, the utilization of hard negative samples introduces new challenges: the emergence of false negatives that can mislead the training process, and the suboptimal quality of sentence embeddings further hampers the bi-encoderss ability to discriminate hard negative samples effectively. To address these challenges, we propose the Rectifying and Discriminating Hard negatives (RigHt) framework. RigHt rectifies negative sample labels through cross-encoder interaction, mitigating the impact of false negatives. Simultaneously, it enhances the bi-encoders to discriminate hard negative samples by refining disentangled sentence embeddings. Extensive experimental results on five datasets substantiate the efficacy of our proposed approach in enhancing the training of the bi-encoders model with hard negative samples.

The recent progress in Large Language Models (LLM) has spurred various advancements in image-language con-versation agents, while how to build a proficient video-based dialogue system is still under exploration. Consid-ering the extensive scale of LLM and visual backbone, min-imal GPU memory is left for facilitating effective temporal modeling, which is crucial for comprehending and providing feedback on videos. To this end, we propose Branching Temporal Adapter (BT-Adapter), a novel method for ex-tending image-language pretrained models into the video domain. Specifically, BT-Adapter serves as a plug-and-use temporal modeling branch alongside the pretrained vi-sual encoder, which is tuned while keeping the backbone frozen. Just pretrained once, BT-Adapter can be seamlessly integrated into all image conversation models using this version of CLIP, enabling video conversations without the need for video instructions. Besides, we develop a unique asymmetric token masking strategy inside the branch with tailor-made training tasks for BT-Adapter, facilitating faster convergence and better results. Thanks to BT-Adapter, we are able to empower existing multimodal dialogue models with strong video understanding capabilities without incur-ring excessive GPU costs. Without bells and whistles, BT-Adapter achieves (1) state-of-the-art zero-shot results on various video tasks using thousands of fewer GPU hours. (2) better performance than current video chatbots without any video instruction tuning. (3) state-of-the-art results of video chatting using video instruction tuning, outperforming previous SOTAs by a large margin. The code has been available at https://github.com/farewellthreeIBT-Adapter.

Text ranking has witnessed significant advancements, attributed to the utilization of dualencoder enhanced by Pre-trained Language Models (PLMs).Given the proliferation of available PLMs, selecting the most effective one for a given dataset has become a non-trivial challenge.As a promising alternative to human intuition and brute-force fine-tuning, Transferability Estimation (TE) has emerged as an effective approach to model selection.However, current TE methods are primarily designed for classification tasks, and their estimated transferability may not align well with the objectives of text ranking.To address this challenge, we propose to compute the expected rank as transferability, explicitly reflecting the model's ranking capability.Furthermore, to mitigate anisotropy and incorporate training dynamics, we adaptively scale isotropic sentence embeddings to yield an accurate expected rank score.Our resulting method, Adaptive Ranking Transferability (AiRTran), can effectively capture subtle differences between models.On challenging model selection scenarios across various text ranking datasets, it demonstrates significant improvements over previous classificationoriented TE methods, human intuition, and ChatGPT with minor time consumption.

The great success of Large Language Models (LLMs) has expanded the potential of multimodality, contributing to the gradual evolution of General Artificial Intelligence (AGI). A true AGI agent should not only possess the capability to perform predefined multi-tasks but also exhibit emergent abilities in an open-world context. However, despite the considerable advancements made by recent multimodal LLMs, they still fall short in effectively unifying comprehension and generation tasks, let alone open-world emergent abilities. We contend that the key to overcoming the present impasse lies in enabling text and images to be represented and processed interchangeably within a unified autoregressive Transformer. To this end, we introduce $\textbf{SEED}$, an elaborate image tokenizer that empowers LLMs with the ability to $\textbf{SEE}$ and $\textbf{D}$raw at the same time. We identify two crucial design principles: (1) Image tokens should be independent of 2D physical patch positions and instead be produced with a $\textit{1D causal dependency}$, exhibiting intrinsic interdependence that aligns with the left-to-right autoregressive prediction mechanism in LLMs. (2) Image tokens should capture $\textit{high-level semantics}$ consistent with the degree of semantic abstraction in words, and be optimized for both discriminativeness and reconstruction during the tokenizer training phase. With SEED tokens, LLM is able to perform scalable multimodal autoregression under its original training recipe, i.e., next-word prediction. SEED-LLaMA is therefore produced by large-scale pretraining and instruction tuning on the interleaved textual and visual data, demonstrating impressive performance on a broad range of multimodal comprehension and generation tasks. More importantly, SEED-LLaMA has exhibited compositional emergent abilities such as multi-turn in-context multimodal generation, acting like your AI assistant. The code (training and inference) and models are released in https://github.com/AILab-CVC/SEED.

Graph neural networks (GNNs) become popular in recommender systems treating the interaction data of user and item as a bipartite graph. Recently, graph contrastive learning achieves superior results for collaborative filtering by reinforcing the learned representations by generating contrastive views through data augmentation. Despite their successful application in recommendation scenarios, there is still some room for improvement: most of these methods perform data augmentation from the data perspective, and the model potential is not exploited enough because more contrastive perspectives are not considered; negative sample bias caused by the different degrees of nodes exists in the contrastive loss. In this paper, we propose a Siamese Adaptive Contrastive learning framework (SACL) to mitigate these issues. Our model utilizes Siamese network as a small perturbation to the model and combines it with data augmentation to learn more robust representations and realizes adaptive contrastive learning introducing the common neighbors’ information of users and items to weight negative samples. Experiments on several public datasets show better performance of our model compared to existing representative methods.

In the absence of text, training generative models directly on speech data through next token prediction task, similar to text-based language models, has demonstrated its feasibility. However, speech data encompasses more intricate feature information compared to text. To capitalize on these additional features, we propose a feature-enhanced generative spoken language modeling (fGSLM). We calculate the difference between the original speech and its normalized version, and extract quantized features with a VQVAE-structured model. These features are subsequently integrated into the generative spoken language modeling (GSLM) by fine-tuning the unit language model (uLM) through a multi-stream transformer. To evaluate the effectiveness of our model, we conduct experiments on the ProsAudit evaluation task in the Zero Resource Speech Challenge. Experimental results show that our model significantly improves prosody comprehension both at the sentence and lexical levels, and achieves superior performance against baseline models.

Text retrieval, a pivotal application in Natural Language Processing (NLP), involves retrieving pertinent documents from a candidate pool for a certain query. Due to the outstanding performance and improved convergence, Pre-trained Language Models (PLMs) are extensively employed in text retrieval. However, selecting the prime model from a multitude of PLMs remains a challenge, termed Model Selection (MS). In this paper, we address the MS by proposing a novel two-stage approach, Teaching and Exploiting Large Language Models (TELLMe). In the first stage, we efficiently retrieve models through Large Language Models (LLMs) prompting, leveraging the unparalleled world knowledge of LLM, where In-Context Learning (ICL) and Chain-of-Thought (CoT) are incorporated to overcome the inherent challenges in LLMs’ understanding of MS tasks. After retrieving potential high-performing models, the second stage employs Transferability Estimation (TE) for further model ranking. To tackle the issue of untrue hard labels faced by TE approaches, we propose a novel method, Evidence maximized from Soft labels (EaSe), which utilizes soft labels from LLM-encoded query embeddings and document embeddings to evaluate transferability scores. Our proposed approach combines the exceptional capabilities of LLMs and provides an efficient and generalized solution for MS regarding text retrieval. We further systematically evaluate the TELLMe framework on three diverse datasets, demonstrating its effectiveness and superiority.

Most graph neural networks (GNNs) are used to learn graph representation by the message passing paradigm. Recent works revealed that under this paradigm, due to the problem of rapid expansion of neighbors, GNNs can not efficiently extract or acquire the information of distant nodes, referred to as over-squashing. For message passing paradigm, over-squashing is an inherent problem, and several graph rewiring methods have been proposed to address this problem. In this work, we propose a more efficient method based on graph rewiring with node-to-node distance relationships (NNDR) and ordered neurons for graph neural networks (O-GNN). Our method strengthens the interactions with distant nodes and uniquely differentiates between neighbor and long-distance node information by ordering their representations hierarchically. Extensive experiments confirm that our proposed method outperforms existing graph rewiring methods across a diverse range of graph classification tasks.

Summary Social networks collect enormous amounts of user personal and behavioral data, which could threaten users' privacy if published or shared directly. Privacy‐preserving graph publishing (PPGP) can make user data available while protecting private information. For this purpose, in PPGP, anonymization methods like perturbation and generalization are commonly used. However, traditional anonymization methods are challenging in balancing high‐level privacy and utility, ineffective at defending against both various link and hybrid inference attacks, as well as vulnerable to graph neural network (GNN)‐based attacks. To solve those problems, we present a novel privacy‐disentangled approach that disentangles private and non‐private information for a better privacy‐utility trade‐off. Moreover, we propose a unified graph deep learning framework for PPGP, denoted privacy‐disentangled variational information bottleneck (PDVIB). Using low‐dimensional perturbations, the model generates an anonymized graph to defend against various inference attacks, including GNN‐based attacks. Particularly, the model fits various privacy settings by employing adjustable perturbations at the node level. With three real‐world datasets, PDVIB is demonstrated to generate robust anonymous graphs that defend against various privacy inference attacks while maintaining the utility of non‐private information.

Bipartite graph representation learning aims to obtain node embeddings by compressing sparse vectorized representations of interactions between two types of nodes, e.g., users and items. Incorporating structural attributes among homogeneous nodes, such as user communities, improves the identification of similar interaction preferences, namely, user/item embeddings, for downstream tasks. However, existing methods often fail to proactively discover and fully utilize these latent structural attributes. Moreover, the manual collection and labeling of structural attributes is always costly. In this article, we propose a novel approach called Dirichlet Max-margin Matrix Factorization (DM3F), which adopts a self-supervised strategy to discover latent structural attributes and model discriminative node representations. Specifically, in self-supervised learning, our approach generates pseudo group labels (i.e., structural attributes) as a supervised signal using the Dirichlet process without relying on manual collection and labeling, and employs them in a max-margin classification. Additionally, we introduce a Variational Markov Chain Monte Carlo algorithm (Variational MCMC) to effectively update the parameters. The experimental results on six real datasets demonstrate that, in the majority of cases, the proposed method outperforms existing approaches based on matrix factorization and neural networks. Furthermore, the modularity analysis confirms the effectiveness of our model in capturing structural attributes to produce high-quality user embeddings.

Document-level event factuality identification (DocEFI) is an important task in event knowledge acquisition, which aims to detect whether an event actually occurs or not from the perspective of the document. Unlike the sentence-level task, a document can have multiple sentences with different event factualities, leading to event factuality conflicts in DocEFI. Existing studies attempt to aggregate local event factuality by exploiting document structures, but they mostly consider textual components in the document separately, degrading complicated correlations therein. To address the above issues, this paper proposes a novel approach, namely UR-HAT, to improve DocEFI with uncertain relational hypergraph attention networks. Particularly, we reframe a document graph as a hypergraph, and establish beneficial n-ary correlations among textual nodes with relational hyperedges, which helps to globally consider local factuality features to resolve event factuality conflicts. To better discern the importance of event factuality features, we further represent textual nodes with uncertain Gaussian distributions, and propose novel uncertain relational hypergraph attention networks to refine textual nodes with the document hypergraph. In addition, we select factuality-related keywords as nodes to enrich event factuality features. Experimental results demonstrate the effectiveness of our proposed method, and outperforms previous methods on two widely used benchmark datasets.

Identifying sentences sharing similar meanings is crucial to speech and text understandings. Although currently popular cross-encoder solutions with pre-trained language models as backbone have achieved remarkable performance, they suffer from the lack of the permutation invariance or symmetry that is one of the most important inductive biases to such task. To alleviate this issue, in this research we propose a permutation invariant training framework, in which a symmetry regularization is introduced during training that forces the model to produce the same predictions for input sentence pairs in both forward and backward directions. Empirical studies exhibit improved performance over competitive baselines.

Question answering (QA) plays a vital role in biomedical natural language processing. Among question answering tasks, the retrieval question answering (ReQA) aims to directly retrieve the correct answer from candidates and has attracted much attention in the community for its efficiency. Recently, researchers have introduced ReQA into the biomedical domain as BioReQA. Typically BioReQA models rely on the dual-encoder to gain semantic representation and are trained following the settings of dense retrieval. However, they normally utilize easy in-batch negative samples in training process to avoid the extra forwarding cost and GPU memory required by encoding additional negative samples. However, hard negative samples have been proved more important with regard to the overall performance of BioReQA tasks. Therefore in this research, we focus on effectively constructing hard in-batch negative samples. Inspired by the classic linear assignment problem, we propose an Iterative Linear Assignment Grouping (ILAG) algorithm to construct hard in-batch negative samples. To further enhance performance for given hard batches in a low-resource scenario, we also employ adversarial training to augment the difficulty of batches. Extensive experiments have shown our proposed method's promising potential in the area of biomedical retrieval question answering.

Event Extraction bridges the gap between text and event signals. Based on the assumption of trigger-argument dependency, existing approaches have achieved state-of-the-art performance with expert-designed templates or complicated decoding constraints. In this paper, for the first time we introduce the prompt-based learning strategy to the domain of Event Extraction, which empowers the automatic exploitation of label semantics on both input and output sides. To validate the effectiveness of the proposed generative method, we conduct extensive experiments with 11 diverse baselines. Empirical results show that, in terms of F1 score on Argument Extraction, our simple architecture is stronger than any other generative counterpart and even competitive with algorithms that require template engineering. Regarding the measure of recall, it sets new overall records for both Argument and Trigger Extractions. We hereby recommend this framework to the community, with the code publicly available at https://github.com/RingBDStack/GDAP.

The Knowledge Base Question Generation (KBQG) is an essential natural language processing task. Taking knowledge graph and answer entities as input, KBQG aims to generate corresponding natural language question. Recently Graph2Seq has been proposed to encode the knowledge graph and achieved remarkable results, while one important challenge still remains, i.e., the graph encoding lacks the interaction with the target question. To deal with the above challenge, we propose a graph and question interaction enhanced Graph2Seq model, in which we design an encoder-decoder parallel enhancement mechanism and apply the knowledge distillation for both inter-mediate representation and prediction distribution to employ the knowledge of the target question into the graph representation. Experiments have been conducted on KBQG benchmark dataset and experimental results have shown the promising potential of proposed method.

Recently, large volumes of false or unverified information (e.g., fake news and rumors) appear frequently in emerging social media, which are often discussed on a large scale and widely disseminated, causing bad consequences. Many studies on rumor detection indicate that the stance distribution of posts is closely related to the rumor veracity. However, these two tasks are generally considered separately or just using a shared encoder/layer via multitask learning, without exploring the more profound correlation between them. In particular, the performance of existing methods relies heavily on the quality of hand-crafted features and the quantity of labeled data, which is not conducive to early rumor detection and few-shot detection. In this article, we construct a hierarchical heterogeneous graph by associating posts containing the same high-frequency words to facilitate the feature cross-topic propagation and jointly formulate stance and rumor detection as multistage classification tasks. To realize the updating of node embeddings jointly driven by stance and rumor detection, we propose a multigraph neural network framework, which can more flexibly capture the attribute and structure information of the context. Experiments on real datasets collected from Twitter and Reddit show that our method outperforms state-of-the-art by a large margin on both stance and rumor detection. And the experimental results also show that our method has better interpretability and requires less labeled data.

The cross-device user matching task is to identify the behavior-logs (i.e., behavior sequences) on multiple devices that belong to one real person. Due to its anonymous and long-term properties, most previous methods of learning behavior embeddings cannot effectively capture two important features in the sequences, namely high-order connections and long-range dependencies. To this end, we propose a novel framework called Two-tier Graph Contextual Embedding (TGCE) to solve the above problems simultaneously. In the first tier, we construct behavior evolutionary graphs (BEGs) for behavior sequences and design an order-preserving neighbor aggregation network to collectively model transitions of behaviors with their neighbors. As repeated behaviors can be grouped into single nodes, our model joints neighboring environments around behaviors in a collective way, and behavior embeddings can be enriched. In the second tier, we further build scaled shortcut graphs (SSGs) by refining BEGs with random walk-based edge addition, then a position-aware graph attention network is further imposed on SSGs to facilitate fast information propagation. As distant graph nodes can be directly connected by shortcut edges, we can further capture long-range dependencies. By stacking two graph tiers, our approach can obtain graph contextual embeddings for behaviors to further improve user matching. Experimental results on the benchmark dataset show that our model outperforms various baselines in the user matching task. Our code is released on https://github.com/13061051/TGCE_2021.

Graph-based algorithms have drawn much attention thanks to their impressive success in semi-supervised setups. For better model performance, previous studies have learned to transform the topology of the input graph. However, these works only focus on optimizing the original nodes and edges, leaving the direction of augmenting existing data insufficiently explored. In this paper, we propose a novel heuristic pre-processing technique, namelyLocal Label Consistency Strengthening (ŁLCS), which automatically expands new nodes and edges to refine the label consistency within a dense subgraph. Our framework can effectively benefit downstream models by substantially enlarging the original training set with high-quality generated labeled data and refining the original graph topology. To justify the generality and practicality of ŁLCS, we couple it with the popular graph convolution network and graph attention network to perform extensive evaluations on three standard datasets. In all setups tested, our method boosts the average accuracy by a large margin of 4.7% and consistently outperforms the state-of-the-art.

Compared with traditional sequential learning models, graph-based neural networks exhibit excellent properties when encoding text, such as the capacity of capturing global and local information simultaneously. Especially in the semi-supervised scenario, propagating information along the edge can effectively alleviate the sparsity of labeled data. In this paper, beyond the existing architecture of heterogeneous word-document graphs, for the first time, we investigate how to construct lightweight non-heterogeneous graphs based on different linguistic information to better serve free text representation learning. Then, a novel semi-supervised framework for text classification that refines graph topology under theoretical guidance and shares information across different text graphs, namely Text-oriented Graph-based Transductive Learning (TextGTL), is proposed. TextGTL also performs attribute space interpolation based on dense substructure in graphs to predict low-entropy labels with high-quality feature nodes for data augmentation. To verify the effectiveness of TextGTL, we conduct extensive experiments on various benchmark datasets, observing significant performance gains over conventional heterogeneous graphs. In addition, we also design ablation studies to dive deep into the validity of components in TextTGL.

Script event prediction (SEP) aims to choose a correct subsequent event from a candidate list, given a chain of ordered context events. Event representation learning has been proposed and successfully applied to this task. Most previous methods learning representations mainly focus on coarse-grained connections at event or chain level, while ignoring more fine-grained connections between events. Here we propose a novel framework which can enhance the representation learning of events by mining their connections at multiple granularity levels, including argument level, event level and chain level. In our method, we first employ a masked self-attention mechanism to model the relations between the components of events (i.e. arguments). Then, a directed graph convolutional network is further utilized to model the temporal or causal relations between events in the chain. Finally, we introduce an attention module to the context event chain, so as to dynamically aggregate context events with respect to the current candidate event. By fusing threefold connections in a unified framework, our approach can learn more accurate argument/event/chain representations, and thus leads to better prediction performance. Comprehensive experiment results on public New York Times corpus demonstrate that our model outperforms other state-of-the-art baselines. Our code is available in https://github.com/YueAWu/MCer.

Word embedding has been widely used in many natural language processing tasks. In this paper, we focus on learning word embeddings through selective higher-order relationships in sentences to improve the embeddings to be less sensitive to local context and more accurate in capturing semantic compositionality. We present a novel multi-order dependency-based strategy to composite and represent the context under several essential constraints. In order to realize selective learning from the word contexts, we automatically assign the strengths of different dependencies between co-occurred words in the stochastic gradient descent process. We evaluate and analyze our proposed approach using several direct and indirect tasks for word embeddings. Experimental results demonstrate that our embeddings are competitive to or better than state-of-the-art methods and significantly outperform other methods in terms of context stability. The output weights and representations of dependencies obtained in our embedding model conform to most of the linguistic characteristics and are valuable for many downstream tasks.