Despite their success, large language models (LLMs) suffer from notorious hallucination issue.By introducing external knowledge stored in knowledge graphs (KGs), existing methods use paths as the medium to represent the graph information sent into LLMs.However, paths only contain limited graph structure information and are unorganized with redundant sequentially appearing keywords, which are difficult for LLMs to digest.We aim to find a suitable medium that captures the essence of structural knowledge in KGs.Inspired by Neural Message Passing in Graph Neural Networks, we propose Language Message Passing (LMP), which first learns a concise facts graph by iteratively aggregating neighbor entities and transforming them into semantic facts, and then performs Topological Readout that encodes the graph structure information into multi-level lists of texts to augment LLMs.Our method serves as a brand-new innovative framework that brings a new perspective into KG-enhanced LLMs, and also offers humanlevel semantic explainability with significant performance improvements over existing methods on all five knowledge graph question answering datasets.

Although spatio-temporal graph neural networks (STGNNs) become widely used methods in traffic forecasting, they still encounter an issue named short-sightedness. Specifically, due to high model complexity and GPU memory usage, STGNNs are restricted to processing only very short input time series. This limited context often causes STGNNs to focus on local variations and overlook long-term patterns, leading to misinterpretation of time series trends. To tackle this issue, recent studies propose to perform mask reconstruction pre-training on traffic series to enhance STGNNs. However, we argue that mask reconstruction is a suboptimal pre-training paradigm for traffic forecasting, because there exists a great gap between pre-training and downstream forecasting, caused by their inconsistent training targets. To eliminate this gap, we propose a new pre-training paradigm named next patch prediction and prove its advantages from both empirical and theoretical perspectives. Based on this paradigm, we introduce a new framework called Decoder-only Pre-training Enhancement (DoP) to unleash the potential of traffic pre-training model. Specifically, DoP uses Transformer decoders as infrastructure, and leverages next patch prediction as target to conduct pre-training. In addition, we propose a new dual-view temporal embedding to fully capture temporal information and spatial spectral enhancement to model spatial information. After pre-training, DoP enhances existing STGNNs seamlessly with periodic enhancement mechanism. On four real-world traffic benchmarks, we demonstrate its start-of-the-art performance.

The recent developments in technologies bring not only increasing amount of information but also multiple information sources for Graph Representation Learning. With the success of Graph Neural Networks (GNN), there have been increasing attempts to learn representation of multi-source information leveraging its graph structures. However, existing graph methods basically combine multi-source information with different contribution scores and over-simplify the graph structures based on prior knowledge, which fail to unify complex and conflicting multi-source information. Multisensory Processing theory in cognitive neuroscience reveals human mechanism of learning multi-source information by identifying the redundancy and complementarity. Inspired by that, we propose Graph Redundancy Network (GRN) that: 1). learns a suitable representation space that maximizes multi-source interactions; 2). encodes the redundant and complementary information according to Graph Intersection and Difference of their graph structures; 3). further reinforces and explores the redundant and complementary information through low-pass and high-pass graph filters. The empirical study shows that GRN outperforms existing methods on various tasks.

Graph Neural Networks (GNNs) have achieved great success in recent years for their remarkable ability to extract effective representations from both node features and graph structures. Most of GNNs only focus on graphs with homogeneous features that correspond to one single feature field. For tabular features that are heterogeneous with multiple feature fields, GNNs often perform less favorably compared to machine learning methods such as boosted trees. In this work, we propose a new perspective to uncover the problem of GNNs on graphs with tabular features through both empirical study and theoretical analysis. The assumption of GNNs that connected nodes exhibit similar patterns can barely hold true for tabular features since multiple feature fields already exhibit different patterns. And propagation on such mismatched graph causes propagated features overcorrelated on graphs, which leads to the reduction of feature diversity and the increase of information redundancy. Therefore, we propose Field-aware Decorrelation Neural Network for graphs with tabular features (GraphFADE), a novel framework that directly optimizes the overcorrelation problem for graphs with tabular features. We first hierarchically partition the dataset into subsets with minimal correlation and then according to the decorrelation clustering results assemble the optimal matched graphs for each feature dimension to propagate on. The empirical study shows that our method achieves superior performance on multiple graphs with tabular features, demonstrating the effectiveness of our model.

Reasoning over knowledge graph explores valuable information for amounts of tasks. However, most methods adopt the coarse-grained and single representation of each entity for reasoning, ignoring simultaneously processing various semantics contained in internal information and external information. On the one hand, the surrounding nodes and relations existing in the graph structure express the internal information of the entity, which contains abundant graph context information, but the extracted internal features are still limited. On the other hand, different scenarios as the external information focus on different aspects of the certain entity, meanwhile the external information should have message interaction with the internal information to learn the adaptive embedding, both of which are seldom considered by the existing methods. In this paper, we propose a Graph Intention Neural Network (GINN) for knowledge graph reasoning to explore fine-grained entity representations, which use external-intention and internal-intention simultaneously. For external-intention, a novel constructed matrix is used to calculate the triple-attention that determines the aggregated information to learn different embeddings adapting to the different scenarios. Furthermore, a communication bridge is leveraged to have message interaction between the external information and the internal information. For the internal-intention, the surrounding nodes and relations are integrated to update the entity embedding with the consideration of the interaction features between the external and internal information. The triple-attention can capture relevancy among the reasoning hops, which contributes to figuring out reasonable paths. We evaluate our approach on real-world datasets, achieving better performance compared to the state-of-the-art methods and showing plausible interpretability for the results.

In recent years, Graph Convolutional Networks (GCNs) have achieved great success in graph representations learning by incorporating features and topology information in each layer. Yet most GCNs are limited to the shallow network architecture due to over-smoothing which leads to indistinguishable nodes representations. In this paper, we provide a unique perspective on the key factor of over-smoothing which is the topological expressiveness loss when the stacked graph diffusion operators are over-densifying in deep layers. To tackle the over-smoothing issue, we propose the Graph Hollow Convolution Network (GHCN) with two key innovations. First, we design a hollow filter applied to the stacked graph diffusion operators to retain the topological expressiveness. Second, in order to further exploit the topology information, we integrate information from different layers based on graph local structures using topological layer fusion without propagating through nodes repeatedly. Extensive experiments on benchmark datasets show that our proposed method outperforms the state-of-the-art methods and effectively relieves the over-smoothing problem.

Knowledge graph is considered as a significant information structure for recommendation, therefore the graph based recommendation has attracted increasing attention in recent years. However, the existing methods face two major challenges. First, the users' preferences should be well considered in the algorithmic model and explicitly shown after model training. Second, there need a better solution to simultaneously learn and combine the information on multiple graphs from different aspects rather than the methods designed for single homogeneous graph. In this paper, we propose the preferences embeddings, which are able to learn the explicit representations for the preferences that influence the users' choices. Further, we innovatively design three channels in a new graph neural network that contains different graph convolutions specifically for the recommendation scenario. This framework can effectively excavate and combine heterogeneous information among user graph, item graph and interaction graph. Extensive experiments on real-world datasets demonstrate the effectiveness and good interpretability of the proposed framework.