Despite their success, large language models (LLMs) suffer from notorious hallucination issue.By introducing external knowledge stored in knowledge graphs (KGs), existing methods use paths as the medium to represent the graph information sent into LLMs.However, paths only contain limited graph structure information and are unorganized with redundant sequentially appearing keywords, which are difficult for LLMs to digest.We aim to find a suitable medium that captures the essence of structural knowledge in KGs.Inspired by Neural Message Passing in Graph Neural Networks, we propose Language Message Passing (LMP), which first learns a concise facts graph by iteratively aggregating neighbor entities and transforming them into semantic facts, and then performs Topological Readout that encodes the graph structure information into multi-level lists of texts to augment LLMs.Our method serves as a brand-new innovative framework that brings a new perspective into KG-enhanced LLMs, and also offers humanlevel semantic explainability with significant performance improvements over existing methods on all five knowledge graph question answering datasets.

Although spatio-temporal graph neural networks (STGNNs) become widely used methods in traffic forecasting, they still encounter an issue named short-sightedness. Specifically, due to high model complexity and GPU memory usage, STGNNs are restricted to processing only very short input time series. This limited context often causes STGNNs to focus on local variations and overlook long-term patterns, leading to misinterpretation of time series trends. To tackle this issue, recent studies propose to perform mask reconstruction pre-training on traffic series to enhance STGNNs. However, we argue that mask reconstruction is a suboptimal pre-training paradigm for traffic forecasting, because there exists a great gap between pre-training and downstream forecasting, caused by their inconsistent training targets. To eliminate this gap, we propose a new pre-training paradigm named next patch prediction and prove its advantages from both empirical and theoretical perspectives. Based on this paradigm, we introduce a new framework called Decoder-only Pre-training Enhancement (DoP) to unleash the potential of traffic pre-training model. Specifically, DoP uses Transformer decoders as infrastructure, and leverages next patch prediction as target to conduct pre-training. In addition, we propose a new dual-view temporal embedding to fully capture temporal information and spatial spectral enhancement to model spatial information. After pre-training, DoP enhances existing STGNNs seamlessly with periodic enhancement mechanism. On four real-world traffic benchmarks, we demonstrate its start-of-the-art performance.

Traditional physics-informed deep learning combines the data-driven methods with the model-based methods by incorporating physics loss as a constraint in total loss function, which aims to enforce the neural network to behave according to the physics laws. However, the potential of physical knowledge is severely underestimated by this approach. Firstly, the physical knowledge fails to demonstrate its intended effects since the physics loss could have extremely small magnitude, more fluctuating convergence rates, and conflicting directions of the gradients compared to the data loss. Secondly, existing methods implicitly employ physics laws as auxiliary terms, which ignores that explicitly utilizing certain properties of physics laws can compensate for the shortcomings of data-driven models, particularly with regard to the data noise and relationships between variables. To alleviate these issues, we propose a Twin-Propeller Network (TPN) to realize fully message exchange among physical knowledge and data information, that releases the potential of the physics laws. Practically, we independently train data-driven model and physics-based model as two student models to get the information separated. Considering the measurement noise present in the data-driven model and the relatively robust physics-based model, we quantify the data uncertainty and utilize it as a weight to balance the two students in a integrated robust teacher model. The stronger teacher in turn transfers the respective knowledge to another student, where we innovatively propose traffic state relation distillation and physical knowledge distillation to guide the training of the data student and the physics student respectively. Through extensive experiments on both synthetic and real-world datasets, our model demonstrates better performance than the existing state-of-the-art methods.

Unsupervised anomaly detection in multivariate time series (MTS) has always been a challenging problem, and the modeling based on reconstruction has garnered significant attention. The insensitivity of these methods towards normal patterns poses challenges in distinguishing between normal and abnormal points. Firstly, the general reconstruction strategies may exhibit limited sensitivity to spatio-temporal dependencies, and their performance remains largely unaffected by such dependencies. Secondly, most methods fail to model the heteroscedastic uncertainty in MTS, hindering their abilities to derive a distinguishable criterion. For instance, normal data with high noise levels may lead to detection failure due to excessively high reconstruction errors. In this work, we emphasize the necessity of sensitivity to normal patterns, which could improve the discrimination between normal and abnormal points remarkably. To this end, we propose SensitiveHUE, a probabilistic network by implementing both reconstruction and heteroscedastic uncertainty estimation. Its core includes a statistical feature removal strategy to ensure the dependency sensitive property, and a novel MTS-NLL loss for modeling the normal patterns in important regions. Experimental results demonstrate that SensitiveHUE exhibits nontrivial sensitivity to normal patterns and outperforms the existing state-of-the-art alternatives by a large margin. Code is publicly available at this URL\footnotehttp://github.com/yuesuoqingqiu/SensitiveHUE.

The recent developments in technologies bring not only increasing amount of information but also multiple information sources for Graph Representation Learning. With the success of Graph Neural Networks (GNN), there have been increasing attempts to learn representation of multi-source information leveraging its graph structures. However, existing graph methods basically combine multi-source information with different contribution scores and over-simplify the graph structures based on prior knowledge, which fail to unify complex and conflicting multi-source information. Multisensory Processing theory in cognitive neuroscience reveals human mechanism of learning multi-source information by identifying the redundancy and complementarity. Inspired by that, we propose Graph Redundancy Network (GRN) that: 1). learns a suitable representation space that maximizes multi-source interactions; 2). encodes the redundant and complementary information according to Graph Intersection and Difference of their graph structures; 3). further reinforces and explores the redundant and complementary information through low-pass and high-pass graph filters. The empirical study shows that GRN outperforms existing methods on various tasks.

Graph Convolutional Networks (GCNs) have made outstanding achievements in many tasks on graphs in recent years, but their success relies on sufficient training data. In practice, sparsely labeled graphs widely exist in the real world so self-training methods have become popular approaches by adding pseudo labeled nodes to enhance the performance of GCNs. However, we observe that most selected high-confidence pseudo labeled nodes by the existing methods would surround the true labeled nodes. It is what we called pseudo label over-dependence, which could lead to the non-uniform pseudo label distribution. Furthermore, a thorough experiment shows that the classification accuracy changes significantly under different label densities and the label-sparse regions show great potential improvement in the model performance. Based on the above findings, we theoretically analyze the constraint factors in the label-sparse regions and further propose reducing the feature distribution difference between the label-dense regions and label-sparse regions can effectively decrease the classification error. Thus, in this paper, we propose a novel Graph Label Information Migration framework (GLIM) to liberate pseudo labels from over-dependence. Specifically, we first propose a training dynamics module (TDM) that uses abundant training process information to find more reliable node labels and improve the model robustness against label noise. Then we propose a label migration module (LMM) that migrates label information from label-dense regions to label-sparse regions by a spectral based graph matching algorithm. These migrated labels are like the glimmers in the darkness, providing the supervision signals for the unlabeled nodes in label-sparse regions. Finally, we conduct extensive experiments to demonstrate the effectiveness of the proposed GLIM.

Graph Neural Networks (GNNs) have achieved great success in recent years for their remarkable ability to extract effective representations from both node features and graph structures. Most of GNNs only focus on graphs with homogeneous features that correspond to one single feature field. For tabular features that are heterogeneous with multiple feature fields, GNNs often perform less favorably compared to machine learning methods such as boosted trees. In this work, we propose a new perspective to uncover the problem of GNNs on graphs with tabular features through both empirical study and theoretical analysis. The assumption of GNNs that connected nodes exhibit similar patterns can barely hold true for tabular features since multiple feature fields already exhibit different patterns. And propagation on such mismatched graph causes propagated features overcorrelated on graphs, which leads to the reduction of feature diversity and the increase of information redundancy. Therefore, we propose Field-aware Decorrelation Neural Network for graphs with tabular features (GraphFADE), a novel framework that directly optimizes the overcorrelation problem for graphs with tabular features. We first hierarchically partition the dataset into subsets with minimal correlation and then according to the decorrelation clustering results assemble the optimal matched graphs for each feature dimension to propagate on. The empirical study shows that our method achieves superior performance on multiple graphs with tabular features, demonstrating the effectiveness of our model.

Graph Neural Networks (GNNs) have been a powerful tool to acquire high-quality node representations dealing with graphs, which strongly depends on a promising graph structure. In the real world scenarios, it is inevitable to introduce noises in graph topology. To prevent GNNs from the disturbance of irrelevant edges or missing edges, graph structure learning is proposed and has attracted considerable attentions in recent years. In this paper, we argue that current graph structure learning methods still pay no regard to the status of nodes and just judge all of their connections simultaneously using a monotonous standard, which will lead to indeterminacy and instability in the optimization process. We designate these methods as status-unaware models. To demonstrate the rationality of our point of view, we conduct exploratory experiments on publicly available datasets, and discover some exciting observations. Afterwards, we propose a new model named Graph Structure Learning via Progressive Strategy (PROSE) according to the observations, which uses a progressive strategy to acquire ideal graph structure in a status-aware way. Concretely, PROSE consists of progressive structure splitting module (PSS) and progressive structure refining module (PSR) to modify node connections according to their global potency, and we also introduce horizontal position encoding and vertical position encoding in order to capture fruitful graph topology information ignored by previous methods. On several widely-used graph datasets, we conduct extensive experiments to demonstrate the effectiveness of our model, and the source code 1 https://github.com/tigerbunny2023/PROSE is provided.

Network data in real-world is error-prone, which results in inaccurate results when performing network analysis or modeling such as node classification and link prediction on these flawed networks. In this paper, we target at reconstructing a reliable network from a flawed network, named as network enhancement. Specifically, network enhancement aims to both detect the noisy links which are observed in the network but should not exist in the real world, and predict the missing links that indeed exist in the real world yet being unobserved in the network. Different from existing works that calculate a unified score to measure the above two kinds of links, we propose E-Net, an end-to-end graph neural network model, to leverage the mutual influence of the two tasks to achieve both the goals more effectively. Because on one hand, detecting noisy links can benefit the performance of predicting missing links; and on the other hand, predicting missing links can provide indirect supervision for detecting noisy links when the labels of the noisy links are unavailable. The experimental results on several datasets show that the proposed model obtains significant improvement for predicting missing links and detecting noisy links.

Multi-turn dialogue system has attracted increasing attention in both academic and industry community. Multi-turn dialogue generation task is a challenging work as the relations among words, utterances and external knowledge are extremely complex. However, the existing methods only focus on constructing the relations between current utterance and historical utterances, and they even oversimplify the relation mining process. Moreover, with the accumulation of dialogue information, the deep semantic information is difficult to understand so that it needs a mechanism with the ability of reasoning and digesting information repeatedly, which is ignored by previous methods. In order to solve the above problems, we propose a Memory Graph with Message Rehearsal (MGMR) for dialogue generation based on the cognitive process of human memory. MGMR contains three main modules: sensory memory, short-term memory and long-term memory. Sensory memory converts the current utterance into embeddings from both word-level and sentence-level. We design a message rehearsal module in short-term memory to extract valuable information of current utterance deeply and repeatedly combined with the relative historical dialogue information and external knowledge stored in long-term memory. Furthermore, we innovatively design a memory graph in long-term memory to construct the relations among words, utterances and knowledge. The memory graph achieves three goals: extracting accurate relations between current utterance and historical utterances, updating the historical dialogue information, and achieving knowledge precipitation by expanding memory graph with the key words and relevant external knowledge of current utterance. We evaluate our model on real-world datasets and achieve better performance compared with the existing state-of-the-art methods.

While Graph Neural Networks (GNNs) have become de facto criterion in graph representation learning, they still suffer from label scarcity and poor generalization. To alleviate these issues, graph pre-training has been proposed to learn universal patterns from unlabeled data via applying self-supervised tasks. Most existing graph pre-training methods only use a single self-supervised task, which will lead to insufficient knowledge mining. Recently, there are also some works that try to use multiple self-supervised tasks, however, we argue that these methods still suffer from a serious problem, which we call it graph structure impairment. That is, there actually exists structural gaps among several tasks due to the divergence of optimization objectives, which means customized graph structures should be provided for different self-supervised tasks. Graph structure impairment not only significantly hurts the generalizability of pre-trained GNNs, but also leads to suboptimal solution, and there is no study so far to address it well. Motivated by Meta-Cognitive theory, we propose a novel model named Core Graph Cognizing and Differentiating (CORE) to deal with the problem in an effective approach. Specifically, CORE consists of cognizing network and differentiating process, the former cognizes a core graph which stands for the essential structure of the graph, and the latter allows it to differentiate into several task-specific graphs for different tasks. Besides, this is also the first study to combine graph pre-training with cognitive theory to build a cognition-aware model. Several experiments have been conducted to demonstrate the effectiveness of CORE.

Reasoning over knowledge graph explores valuable information for amounts of tasks. However, most methods adopt the coarse-grained and single representation of each entity for reasoning, ignoring simultaneously processing various semantics contained in internal information and external information. On the one hand, the surrounding nodes and relations existing in the graph structure express the internal information of the entity, which contains abundant graph context information, but the extracted internal features are still limited. On the other hand, different scenarios as the external information focus on different aspects of the certain entity, meanwhile the external information should have message interaction with the internal information to learn the adaptive embedding, both of which are seldom considered by the existing methods. In this paper, we propose a Graph Intention Neural Network (GINN) for knowledge graph reasoning to explore fine-grained entity representations, which use external-intention and internal-intention simultaneously. For external-intention, a novel constructed matrix is used to calculate the triple-attention that determines the aggregated information to learn different embeddings adapting to the different scenarios. Furthermore, a communication bridge is leveraged to have message interaction between the external information and the internal information. For the internal-intention, the surrounding nodes and relations are integrated to update the entity embedding with the consideration of the interaction features between the external and internal information. The triple-attention can capture relevancy among the reasoning hops, which contributes to figuring out reasonable paths. We evaluate our approach on real-world datasets, achieving better performance compared to the state-of-the-art methods and showing plausible interpretability for the results.

In recent years, Graph Convolutional Networks (GCNs) have achieved great success in graph representations learning by incorporating features and topology information in each layer. Yet most GCNs are limited to the shallow network architecture due to over-smoothing which leads to indistinguishable nodes representations. In this paper, we provide a unique perspective on the key factor of over-smoothing which is the topological expressiveness loss when the stacked graph diffusion operators are over-densifying in deep layers. To tackle the over-smoothing issue, we propose the Graph Hollow Convolution Network (GHCN) with two key innovations. First, we design a hollow filter applied to the stacked graph diffusion operators to retain the topological expressiveness. Second, in order to further exploit the topology information, we integrate information from different layers based on graph local structures using topological layer fusion without propagating through nodes repeatedly. Extensive experiments on benchmark datasets show that our proposed method outperforms the state-of-the-art methods and effectively relieves the over-smoothing problem.

Knowledge graph is considered as a significant information structure for recommendation, therefore the graph based recommendation has attracted increasing attention in recent years. However, the existing methods face two major challenges. First, the users' preferences should be well considered in the algorithmic model and explicitly shown after model training. Second, there need a better solution to simultaneously learn and combine the information on multiple graphs from different aspects rather than the methods designed for single homogeneous graph. In this paper, we propose the preferences embeddings, which are able to learn the explicit representations for the preferences that influence the users' choices. Further, we innovatively design three channels in a new graph neural network that contains different graph convolutions specifically for the recommendation scenario. This framework can effectively excavate and combine heterogeneous information among user graph, item graph and interaction graph. Extensive experiments on real-world datasets demonstrate the effectiveness and good interpretability of the proposed framework.