With the rapid growth of Web-based academic publications, more and more papers are being published annually, making it increasingly difficult to find relevant prior work. Citation prediction aims to automatically suggest appropriate references, helping scholars navigate the expanding scientific literature. Here we present CiteRAG, the first comprehensive retrieval-augmented generation (RAG)-integrated benchmark for evaluating large language models on academic citation prediction, featuring a multi-level retrieval strategy, specialized retrievers, and generators. Our benchmark makes four core contributions: (1) We establish two instances of the citation prediction task with different granularity. Task 1 focuses on coarse-grained list-specific citation prediction, while Task 2 targets fine-grained position-specific citation prediction. To enhance these two tasks, we build a dataset containing 7,267 instances for Task 1 and 8,541 instances for Task 2, enabling comprehensive evaluation of both retrieval and generation. (2) We construct a three-level large-scale corpus with 554k papers spanning many major subfields, using an incremental pipeline. (3) We propose a multi-level hybrid RAG approach to citation prediction, fine-tuning embedding models with contrastive learning to capture complex citation relationships, paired with specialized generation models. (4) We conduct extensive experiments across state-of-the-art language models, including closed-source APIs, open-source models, and our fine-tuned generators, demonstrating the effectiveness of our framework. Our open-source toolkit enables reproducible evaluation and focuses on academic literature, providing the first comprehensive evaluation framework for citation prediction and serving as a methodological template for other scientific domains. Our source code and data are released at https://github.com/LQgdwind/CiteRAG.

With the widespread adoption of collaborative filtering techniques for personalized recommendations, exposure bias has become a significant challenge.Exposure biasrefers to the tendency of recommendation models to disproportionately favor items with high exposure over those with low exposure. In graph collaborative filtering that uses graph neural networks (GNNs) for recommendations, exposure bias can be exacerbated due to 1) the reliance on positive feedback during graph construction and 2) the effects of the neighbor aggregation step in GNNs. To tackle this challenge, we propose a novel and efficient framework called FUGCF (training-Free andUnbiasedGraphCollaborativeFiltering) to improve both the accuracy and bias mitigation of graph-based personalized recommendations. FUGCF employs a two-stage calculation strategy: it estimates exposure probabilities in the first stage and then leverages them to help derive debiased node embeddings in the second stage. Furthermore, we design a training-free estimation method for FUGCF based on closed-form solutions to enhance its computational efficiency. The extensive experiments on a synthetic dataset and three real-world datasets demonstrate the effectiveness of FUGCF in reducing exposure bias, improving recommendation accuracy, and optimizing computational efficiency.

Graph embedding aims to extract low-dimensional representation vectors, commonly referred to as embeddings, from graph data. The generated embeddings simplify subsequent data analysis and machine learning tasks. Recently, researchers have proposed the use of contrastive learning on graphs to extract node embeddings in an unsupervised manner. Although existing graph contrastive learning methods have significantly advanced this field, there is still potential for further exploration, particularly in optimizing training efficiency and enhancing embedding quality. In this paper, we propose an efficient unsupervised graph embedding method named GEARED. First, the method involves an attributed graph reduction module that converts the raw graph into a reduced graph, greatly improving training efficiency. Second, GEARED employs a dual-level loss with adaptive scaling factors to obtain high-quality embeddings. Experimental results demonstrate that GEARED achieves high classification accuracy, good scalability, and enhanced embedding quality.

The heterogeneous graph, as an effective representation of real-world data, encapsulates rich structural and semantic information. In recent years, numerous Heterogeneous Graph Neural Networks (HGNNs) have been proposed to learn node representations on heterogeneous graphs. Although existing methods have introduced various unique information aggregation and semantic fusion mechanisms, they still exhibit limitations in effectiveness and scalability. In this study, we introduce the gatekeeping theory in heterogeneous graph learning and investigate the primary challenges limiting current HGNNs. To address these challenges, we propose a novel, effective, and scalable heterogeneous graph neural network framework, the Heterogeneous Convolution-oriented Attention Network (HCAN). HCAN enhances the heterogeneous attention mechanism to learn far-sighted weights by encoding long-range relation information into node representation with a convolutional subgraph encoder. To further improve heterogeneous graph representation learning, we propose effective and scalable models based on the HCAN framework. We evaluate HCAN on various commonly used heterogeneous datasets and show that it outperforms the state-of-the-art methods, especially on challenging datasets.

Recent advances in graph contrastive learning (GCL) have significantly enhanced recommendation systems. However, most existing approaches predominantly focus on optimizing training data fit while overlooking exposure bias, a critical issue that can substantially impact recommendation effectiveness. Drawing inspiration from sociological theories of human interaction patterns-specifically how individuals balance self-presentation and self-hiding behaviors in social contexts-this paper proposes BPH4Rec, a novel Balancing self-Presentation and self-Hiding approach for exposure-aware Recommendation based on GCL. Within the GCL framework, BPH4Rec introduces two complementary mechanisms: (1) a self-hiding mechanism that modifies the adjacency matrix of contrastive views through custom inverse propensity scoring (IPS), effectively addressing exposure bias, and (2) a self-presentation mechanism that incorporates densification factors during matrix reconstruction to mitigate sparsity-induced biases. Through extensive evaluation on six public benchmark datasets, BPH4Rec demonstrates substantial improvements over state-of-the-art baselines, particularly in promoting long-tail item discovery while maintaining recommendation accuracy.

Procedural Language extensions of SQL (abbr. PL/SQL) enhance database programming by integrating procedural constructs with SQL's declarative syntax, thereby improving the reusability, modularity, and maintainability of SQL. Besides, PL/SQL in database systems presents significant challenges in real-world development, primarily due to the inherent complexity of programming. To reduce the development difficulty of PL/SQL, this paper studies the novel task of translating natural language (NL) to PL/SQL (i.e., NL-to-PL/SQL), aimed at simplifying PL/SQL development. Recent advancements in language models have shown promise in translating natural language questions into SQL queries (i.e., Text-to-SQL). However, the state-of-the-art Text-to-SQL methods focus only on single SQL queries, neglecting the procedural extensions of SQL, which limits their effectiveness for the NL-to-PL/SQL task. In this paper, we propose PLForge, a suite of pre-trained language models with parameter configurations of 3B, 7B, and 15B, tailored for NL-to-PL/SQL tasks. To enhance the PL/SQL generation capabilities of PLForge, we leverage a curated PL/SQL-centric data corpus and employ an incremental pre-training approach. Furthermore, to fully exploit the potential of PLForge, we propose a comprehensive prompt construction strategy tailored specifically for PL/SQL. Given the scarcity of NL-to-PL/SQL datasets, we develop a template-based method for generating NL-to-PL/SQL data. We conduct a series of experiments on PLForge and several baseline models. Based on execution match and exact match metrics that are designed specifically for the NL-to-PL/SQL task, the experimental results demonstrate that PLForge outperforms existing models in both in-context learning and supervised fine-tuning settings.

Recommender systems mitigate information overload by offering personalized suggestions to users. As the interactions between users and items can inherently be depicted as a bipartite graph, recent years have seen a surge in the interest in harnessing graph neural networks (GNNs) for enhancing recommender systems. However, a notable limitation of current GNN-based recommendation models is their exclusive emphasis on positive feedback, while disregarding the potentially insightful negative feedback. In this article, we introduce Pone-GNN, a novel recommendation model that synergistically integrates both Po sitive and ne gative feedback in G raph N eural N etworks. Pone-GNN utilizes dual embeddings–interest embeddings for capturing a user’s likes and disinterest embeddings for a user’s dislikes. Also, Pone-GNN performs message passing for both positive and negative feedback, and integrates two sets of embeddings through contrastive learning, which is crucial for extracting robust and discriminative embeddings. Our comprehensive experimental analysis demonstrates that Pone-GNN outperforms state-of-the-art models on diverse real-world recommendation datasets. For example, Pone-GNN achieves a 6.15% increase in relative nDCG@10 compared to the runner-up on the KuaiRec dataset.

To analyze and process graph data, researchers have proposed Graph Neural Network (GNN) models. In this paper, we focus on methods for boosting the performance of existing GNN models and propose GraphHI, a GNN framework that integrates Hidden Insights to enhance the performance of a given GNN model. We propose to utilize both inter-model and intra-model hidden insights. The inter-model hidden insights encompass the embedding vectors and logit vectors derived from other pretrained models using the same graph data. The intra-model hidden insights incorporate the embedding vectors of other nodes from the same GNN model. To optimize the suitability of hidden insights for GNN model training, we conduct a theoretical analysis of the influence of various forms of the transformed logits and the parameter$T$in the data transformation function. Based on this analysis, a method for setting dynamic personalized parameters in the data transformation is proposed, which is tailored to the current state of each node in the GNN model. To integrate multiple sources of hidden insights, we propose ALC, an algorithm that dynamically sets appropriate combination coefficients for various loss terms. The experimental results show that GraphHI can boost the performance of GNN models using different pretrained models in four different tasks.

Contrastive learning, a powerful self-supervised learning paradigm, has shown its efficacy in learning embed dings from independent and identically distributed (IID) as well as non-IID data without relying on label information. Since high-quality discriminative embeddings form a rich embedding space, which benefits model performance on downstream tasks, it is necessary to study how to improve the quality of contrastive node embeddings in graph contrastive learning. However, there has been limited research on this area. In this paper, we investigate how to generate high-quality contrastive node embeddings based on an in-depth analysis of graph contrastive losses. Firstly, we propose a novel and effective method, GLATE, for estimating the temperatures in three mainstream graph contrastive losses during the training phase. Secondly, we conduct the derivation of GLATE, and the derivation results reveal the specific relationship between the quality of contrastive node embeddings and tem-peratures. Finally, the extensive experiments on 16 benchmark datasets demonstrate that GLATE consistently outperforms the state-of-the-art graph contrastive learning models in terms of both model performance and training efficiency.

Recommender systems have emerged as an indispensable mean to meet personalized interests of users and alleviate information overload. Despite the great success, accuracy-oriented recommendation models are creating information cocoons, i.e., it is becoming increasingly difficult for users to see other items they might be interested in. Although recent studies start paying attention to enhancing recommendation diversity, models based on point embedding fail to describe the range of user preferences and item features well, which is essential for diversified matching. To this end, we propose LCD-UC , a novel List-Check-Decide framework with UnCertainty masking based on box embedding to improve recommendation diversity with recommendation accuracy maintained. Specifically, LCD-UC creates hypercubes to represent users and items using box embedding for high model flexibility and expressiveness. Then, a hypercube similarity scoring function is designed to measure the similarity between hypercubes representing users and items. To make a balance between the accuracy and diversity of recommendations and achieve personalized diversity needs, we further develop a user-item pairwise attention mechanism as well as a user uncertainty masking mechanism in LCD-UC. Besides, we present two new metrics for better evaluation on recommendation diversity, which address the issue that existing metrics only consider the coverage of categories while ignore the frequency of categories. The extensive experiments on three real-world datasets show that LCD-UC can improve both recommendation accuracy and diversity over three base models, and is superior to six state-of-the-art recommendation models. An online 10-day AB test also demonstrates that LCD-UC can improve the performance of a real-world advertising system.

Graph embedding aims to extract low-dimensional representation vectors, commonly referred to as embeddings, from graph data. The generated embeddings simplify subsequent data analysis and machine learning tasks. Recently, researchers have proposed the use of contrastive learning on graphs to extract node embeddings in an unsupervised manner. Although existing graph contrastive learning methods have significantly advanced this field, there is still potential for further exploration, particularly in optimizingtraining efficiencyand enhancingembedding quality. In this paper, we propose an efficient unsupervised graph embedding method named GEARED. First, the method involves an attributed graph reduction module that converts the raw graph into a reduced graph, greatly improving model training efficiency. Second, GEARED employs a dual-level loss with adaptive scaling factors to ensure the acquisition of high-quality embeddings. Finally, we conduct a partial derivative analysis to elucidate the specific mechanisms through which GEARED is capable of generating high-quality embeddings. Extensive experimental evaluations on 14 benchmark datasets show that GEARED consistently outperforms state-of-the-art methods in terms of training efficiency and classification accuracy. For instance, GEARED achieves a training speedup of over 40 times on both the CS and Physics datasets while maintaining superior classification accuracy.

Unsupervised graph representation learning, i.e., learning node or graph embeddings from graph data in an unsupervised manner, has become an important problem when we study graph data. With the development of self-supervised learning, researchers have designed graph-level self-supervised learning paradigms and learn embeddings under these paradigms. The learned embeddings can serve as a fine initial solution to downstream tasks such as node classification or graph classification. In this paper, we propose a fast unsupervised graph embedding method, which follows the way of self-supervised learning. This method performs representation learning on the graph under a novel concept called Graph Zoom Learning (abbr. GZL), which is orthogonal to the existing concepts of unsupervised graph embedding, such as random walk and contrastive learning. Two crucial components, graph zoom-out and point-to-point contrast, help GZL reduce the overall training time cost. Specifically, on the one hand, a lightweight miniature graph is generated from the raw graph by graph zoom-out and the learning on the miniature graph is more efficient than the learning on the raw graph; on the other hand, we design the miniature-scale learning on the miniature graph and introduce community structure into this learning pattern, which contributes to the final point-to-point contrast. Since point-to-point contrast is independent of negatives, it makes the whole training more efficient. We conduct extensive experiments to verify the advantage of GZL on representation learning. On two downstream tasks of node classification and graph classification, GZL outperforms the state-of-the-art unsupervised graph embedding methods. Particularly, on the largest experimental graph dataset (ogbn-arxiv) with 169k nodes and 1.1m edges, GZL outperforms the runner-up by 3.3% relative accuracy and achieves up to 22.6x speedup over it.

Graph Neural Networks (GNNs) have demonstrated promising results on exploiting node representations for many downstream tasks through supervised end-to-end training. To deal with the widespread label scarcity issue in real-world applications, Graph Contrastive Learning (GCL) is leveraged to train GNNs with limited or even no labels by maximizing the mutual information between nodes in its augmented views generated from the original graph. However, the distribution of graphs remains unconsidered in view generation, resulting in the ignorance of unseen edges in most existing literature, which is empirically shown to be able to improve GCL's performance in our experiments. To this end, we propose to incorporate graph generative adversarial networks (GANs) to learn the distribution of views for GCL, in order to i) automatically capture the characteristic of graphs for augmentations, and ii) jointly train the graph GAN model and the GCL model. Specifically, we present GACN, a novel Generative Adversarial Contrastive learning Network for graph representation learning. GACN develops a view generator and a view discriminator to generate augmented views automatically in an adversarial style. Then, GACN leverages these views to train a GNN encoder with two carefully designed self-supervised learning losses, including the graph contrastive loss and the Bayesian personalized ranking Loss. Furthermore, we design an optimization framework to train all GACN modules jointly. Extensive experiments on seven real-world datasets show that GACN is able to generate high-quality augmented views for GCL and is superior to twelve state-of-the-art baseline methods. Noticeably, our proposed GACN surprisingly discovers that the generated views in data augmentation finally conform to the well-known preferential attachment rule in online networks.

Graph Neural Networks (GNNs) have demonstrated great power in many network analytical tasks. However, graphs (i.e., networks) in the real world are usually text-rich, implying that valuable semantic information needs to be carefully considered. Existing GNNs for text-rich networks typically treat text as attribute words alone, which inevitably leads to the loss of important semantic structures, limiting the representation capability of GNNs. In this paper, we propose an end-to-end adaptive semantic architecture of graph convolutional networks, namely AS-GCN, which unifies neural topic model and graph convolutional networks, for text-rich network representation. Specifically, we utilize a neural topic model to extract the global topic semantics, and accordingly augment the original text-rich network into a tri-typed heterogeneous network, capturing both the local word-sequence semantic structure and the global topic semantic structure from text. We then design an effective semantic-aware propagation of information by introducing a discriminative convolution mechanism. We further propose two strategies, that is, distribution sharing and joint training, to adaptively generate a proper network structure based on the learning objective to improve network representation. Extensive experiments on text-rich networks illustrate that our new architecture outperforms the state-of-the-art methods by a significant improvement. Meanwhile, this architecture can also be applied to e-commerce search scenes, and experiments on a real e-commerce problem from JD further demonstrate the superiority of the proposed architecture over the baselines.

Graph convolutional networks (GCNs), aiming to obtain node embeddings by integrating high-order neighborhood information through stacked graph convolution layers, have demonstrated great power in many network analysis tasks such as node classification and link prediction. However, a fundamental weakness of GCNs, that is, topological limitations, including over-smoothing and local homophily of topology, limits their ability to represent networks. Existing studies for solving these topological limitations typically focus only on the convolution of features on network topology, which inevitably relies heavily on network structure. Moreover, most networks are text-rich, so it is important to integrate not only document-level information, but also the local text information which is particularly significant while often ignored by the existing methods. To solve these limitations, we propose BiTe-GCN, a novel GCN architecture modeling via bidirectional convolution of topology and features on text-rich networks. Specifically, we first transform the original text-rich network into an augmented bi-typed heterogeneous network, capturing both the global document-level information and the local text-sequence information from texts. We then introduce discriminative convolution mechanisms, which performs convolution on this augmented bi-typed network, realizing the convolutions of topology and features altogether in the same system, and learning different contributions of these two parts (i.e., network part and text part), automatically for the given learning objectives. Extensive experiments on text-rich networks demonstrate that our new architecture outperforms the state-of-the-arts by a breakout improvement. Moreover, this architecture can also be applied to several e-commerce search scenes such as JD searching, and experiments on JD dataset show the superiority of the proposed architecture over the related methods.

Relevance has significant impact on user experience and business profit for e-commerce search platform. In this work, we propose a data-driven framework for search relevance prediction, by distilling knowledge from BERT and related multi-layer Transformer teacher models into simple feed-forward networks with large amount of unlabeled data. The distillation process produces a student model that recovers more than 97% test accuracy of teacher models on new queries, at a serving cost that's several magnitude lower (latency 150x lower than BERT-Base and 15x lower than the most efficient BERT variant, TinyBERT). The applications of temperature rescaling and teacher model stacking further boost model accuracy, without increasing the student model complexity. We present experimental results on both in-house e-commerce search relevance data as well as a public data set on sentiment analysis from the GLUE benchmark. The latter takes advantage of another related public data set of much larger scale, while disregarding its potentially noisy labels. Embedding analysis and case study on the in-house data further highlight the strength of the resulting model. By making the data processing and model training source code public, we hope the techniques presented here can help reduce energy consumption of the state of the art Transformer models and also level the playing field for small organizations lacking access to cutting edge machine learning hardwares.

Network embedding (NE), aiming to embed a network into a low dimensional latent representation while preserving the inherent structural properties of the network, has attracted considerable attention recently. Variational Autoencoder (VAE) has been widely studied for NE. Existing VAE based methods let the network follow a unimodal distribution, that is, they typically use some fixed distribution as the prior, e.g. Gaussian distribution. However, in reality networks often contain many complicated structural properties [5], [6] (such as the first/second order proximity, the motif or community structures, power-law, etc). The latent representation from unimodal and fixed distribution is not capable of describing such multi-modal characteristic of networks. To address this issue, we develop a new VAE method for NE, named Normalizing Flow Variational Graph Autoencoder (NF-VGA). We design a prior-generative module based on normalizing flows to generate flexible, multi-modal distribution as the prior of the latent representation. To make the generated prior better describe the coupling relationship between nodes, we further utilize network local structures to guide the prior generation. Extensive experiments on some real-world networks show a superior performance of the new approach over some state-of-the-art methods on some popular network embedding tasks.