Knowledge distillation technology shows significant potential in transferring the knowledge of graph neural network (GNN) teacher models to multi-layer perceptron (MLP) student models. However, the existing methods are primarily designed for node classification task and are difficult to directly apply to graph classification. Graph classification task requires knowledge transfer via the generation of graph-level soft labels. Such global supervisory signals are inherently sparse, making it difficult for the student model to capture fine-grained structural information. Additionally, the student MLP model lacks the explicit ability to handle graph structures and has a significantly different representation space from the teacher GNN which further complicates the knowledge transfer process. We address these problems by proposing multigranularity distillation with a dynamic prior knowledge framework named MuDPK. The proposed framework integrates multigranularity structural information with the dynamic prior knowledge mechanism to effectively address the challenges of sparse supervision signals and the representation bottleneck in graph classification. We first design an adaptive graph propagation-based node feature enhancement module to enhance the representation capability of the student model in a limited feature space. We then design a triple distillation loss function that includes graph-level, subgraph-level, and node-level components. We alleviate the problem of sparse supervision signals by hierarchically aligning the structural representations of the teacher-student model at different levels. Finally, we use the intermediate layer features of the teacher as both the target representation and input knowledge, dynamically adjusted by our prior knowledge mechanism. Knowledge transfer is achieved through feature alignment while the student model is regulated by adjusting the proportion of injected knowledge in a controlled manner. Extensive experiments on eight benchmark graph datasets demonstrate the effectiveness and superiority of MuDPK, which achieves an average accuracy gain of +1.37% over the baselines.

Graph prompt learning represents a novel paradigm aimed at enhancing the performance of graph learning models on a variety of downstream tasks by providing specific graph prompts. Despite its promise, current graph prompt learning methods are limited by the following limitations. On the one hand, existing methods often rely on manually selected graph information or simple learnable vectors, which can introduce human biases and lack expressiveness. These methods also fall short in guiding models to induce historical prior knowledge and improve generalization. Furthermore, the direct end-to-end tuning strategy of prompts lacks a necessary gentle transition, which impacts model stability and generalization. To overcome these limitations, we introduce the generalizable graph prompt learning framework (GGPL), which incorporates model-level prompt injection and a two-stage prompt tuning strategy. GGPL focuses on encoding subgraph structures and attributes during pre-training and uses SimGRACE to predict subgraph similarities, enhancing the base model's generalization. The model-level prompt injection module, with its prompt embedding backbone and self-prompt generation, seamlessly integrates invariant knowledge. Our two-stage tuning strategy, including transition and task-specific tuning, ensures better guidance and stability. By designing learnable prompt tokens and fine-tuning them with task-specific information, GGPL enables the model to generalize more robustly to downstream tasks. We conduct extensive experiments on six benchmark datasets to verify the model's effectiveness.

Deep learning models are often trained on datasets that are limited in size and distribution, which may not fully represent the entire range of data encountered in practice. Thus, making deep learning models generalize to out-of-distribution data has received a significant amount of attention in recent studies due to the critical importance of this ability in real-world applications. Meta learning as an effective knowledge transfer paradigm, which learns a base model with high generalization ability to adapt to new data distributions by minimizing domain shifts across tasks during meta-training. However, most existing meta learning methods assume that the base model can access the labels of different domains, and this assumption is demanding in many real application scenarios. In addition, these methods focus on narrowing data-level domain shifts, while ignoring task-level domain shifts, which may lead to inadequate or even negative transfer. Inspired by human learners who use induction to learn and master new tasks, we propose a novel domain-aware meta learning framework for out-of-distribution generalization, termed SMLG. This framework enables the base model to generalize effectively to unseen domains without relying on domain-specific labels. Specifically, we develop a domain-aware transformation module to obtain meta representation and pseudo domain labels. As a result, the base model can be trained robustly without the need for direct domain label input. Furthermore, to investigate the impact of domain shifts at different levels, we introduce a joint loss function that combines cross-entropy with a domain alignment constraint. Extensive experiments on benchmark datasets demonstrate the efficacy of our framework.

Despite the promising representation learning of graph neural networks (GNNs), the supervised training of GNNs notoriously requires large amounts of labeled data from each application. An effective solution is to apply the transfer learning in graph: using easily accessible information to pre-train GNNs, and fine-tuning them to optimize the downstream task with only a few labels. Recently, many efforts have been paid to design the self-supervised pretext tasks, and encode the universal graph knowledge among the various applications. However, they rarely notice the inherent training objective gap between the pretext and downstream tasks. This significant gap often requires costly fine-tuning for adapting the pre-trained model to downstream problem, which prevents the efficient elicitation of pre-trained knowledge and then results in poor results. Even worse, the naive pre-training strategy usually deteriorates the downstream task, and damages the reliability of transfer learning in graph data. To bridge the task gap, we propose a novel transfer learning paradigm to generalize GNNs, namely graph pre-training and prompt tuning (GPPT). Specifically, we first adopt the masked edge prediction, the most simplest and popular pretext task, to pre-train GNNs. Based on the pre-trained model, we propose the graph prompting function to modify the standalone node into a token pair, and reformulate the downstream node classification looking the same as edge prediction. The token pair is consisted of candidate label class and node entity. Therefore, the pre-trained GNNs could be applied without tedious fine-tuning to evaluate the linking probability of token pair, and produce the node classification decision. The extensive experiments on eight benchmark datasets demonstrate the superiority of GPPT, delivering an average improvement of 4.29% in few-shot graph analysis and accelerating the model convergence up to 4.32X. The code is available in: https://github.com/MingChen-Sun/GPPT.

Graph neural networks (GNNs) have emerged as a successful method on graph structured data. Limited by expensive labeled data, contrastive learning has been adopted to the graph domain. In most existing node-level graph contrastive learning methods, when applying contrastive learning to a certain unlabeled node (the center node), its corresponding “similar” node (positive sample) is usually generated by data augmentation. Other nodes in the graph are served as the “dissimilar” nodes (negative samples), which leads to two major problems. First, the computational cost can be prohibitively expensive, especially when the graph is large. Second, utilizing some nodes which share the same label with the center node as the negative samples will damage the learning process. Hence, to address these issues, we explore the feasibility of only sampling a part of nodes for graph contrastive learning process. And unlike the previous self-supervised contrastive methods, we use joint training to exploit supervised signals as much as possible in contrastive learning. Hence, we propose a Negative Samples Selecting Strategy to utilize the classification prediction to guide the selection of the negative samples for sampled nodes. Then, we further incorporate this strategy for performing contrastive learning on graphs and propose a framework named Graph Contrastive Learning with Negative Samples Selecting Strategy (GCNSS). We demonstrate that GCNSS can be trained much faster with much less computation memory than graph contrastive learning baselines, and GCNSS can effectively boost the performance of existing GNN models on semi-supervised node classification tasks across many different datasets. The code is in: https://github.com/MR9812/GCNSS.

Human-curated knowledge graphs provide critical supportive information to various natural language processing tasks, but these graphs are usually incomplete, urging auto-completion of them (a.k.a. knowledge graph completion). Prevalent graph embedding approaches, e.g., TransE, learn structured knowledge via representing graph elements (i.e., entities/relations) into dense embeddings and capturing their triple-level relationship with spatial distance. However, they are hardly generalizable to the elements never visited in training and are intrinsically vulnerable to graph incompleteness. In contrast, textual encoding approaches, e.g., KG-BERT, resort to graph triple’s text and triple-level contextualized representations. They are generalizable enough and robust to the incompleteness, especially when coupled with pre-trained encoders. But two major drawbacks limit the performance: (1) high overheads due to the costly scoring of all possible triples in inference, and (2) a lack of structured knowledge in the textual encoder. In this paper, we follow the textual encoding paradigm and aim to alleviate its drawbacks by augmenting it with graph embedding techniques – a complementary hybrid of both paradigms. Specifically, we partition each triple into two asymmetric parts as in translation-based graph embedding approach, and encode both parts into contextualized representations by a Siamese-style textual encoder. Built upon the representations, our model employs both deterministic classifier and spatial measurement for representation and structure learning respectively. It thus reduces the overheads by reusing graph elements’ embeddings to avoid combinatorial explosion, and enhances structured knowledge by exploring the spatial characteristics. Moreover, we develop a self-adaptive ensemble scheme to further improve the performance by incorporating triple scores from an existing graph embedding model. In experiments, we achieve state-of-the-art performance on three benchmarks and a zero-shot dataset for link prediction, with highlights of inference costs reduced by 1-2 orders of magnitude compared to a sophisticated textual encoding method.