Graph anomaly detection (GAD) refers to identifying abnormal graph nodes or edges that heavily deviate from normal observations. Existing approaches inevitably suffer from the influence of imbalanced data and privacy protection. This shortcoming poses challenges in optimizing node embeddings and detecting multitype anomalies simultaneously, resulting in decreased accuracy of existing GAD models. To address this shortcoming, we introduce a new federated learning model for graph anomaly detection (FedGAD). FedGAD enables collaborative unsupervised learning among decentralized data centers without requiring direct access to the distributed subgraphs. Specifically, FedGAD masks and reconstructs the neighborhood features to enhance the knowledge of node representations. Considering the data diversity across distributed clients, we also design a cross-clients' node representation module that enables nodes to reconstruct neighbors by leveraging information from other clients. Furthermore, we use a multiscale contrastive learning function, which includes both structure-level and contextual-level learning functions, to detect graph anomalies in the condition that subgraphs located at different clients show imbalanced data distributions. Experimental results on seven benchmark datasets demonstrate the superior performance of FedGAD compared with baseline methods, verifying its capability of improving GAD performance.

Personalized federated learning (PFL) on graphs is an emerging field focusing on the collaborative development of architectures across multiple clients, each with distinct graph data distributions while adhering to strict privacy standards. This area often requires extensive expert intervention in model design, which is a significant limitation. Recent advancements have aimed to automate the search for graph neural network architectures, incorporating large language models (LLMs) for their advanced reasoning and self-reflection capabilities. However, two technical challenges persist. First, although LLMs are effective in natural language processing, their ability to meet the complex demands of graph neural architecture search (GNAS) is still being explored. Second, while LLMs can guide the architecture search process, they do not directly solve the issue of client drift due to heterogeneous data distributions. To address these challenges, we introduce a novel method, Personalized Federated Graph Neural Architecture Search (PFGNAS). This approach employs a task-specific prompt to identify and integrate optimal GNN architectures continuously. To counteract client drift, PFGNAS utilizes a weight-sharing strategy of supernet, which optimizes the local architectures while ensuring client-specific personalization. Extensive evaluations show that PFGNAS significantly outperforms traditional PFL methods, highlighting the advantages of integrating LLMs into personalized federated learning environments.

Heterogeneous graphs are increasingly used to represent complex real-world scenarios with diverse entities and interactions by meta structures. Recently, the search of meta structures is combined with graph neural architecture search to automatically extract the semantic knowledge for various tasks in heterogeneous graphs. However, prior research primarily focuses on identifying meta structures that are universally applicable across all nodes in a graph, neglecting the variations in meta structure selection that arise from the unique features and topology of individual nodes. To address this challenge, we introduce a Node-Centric approach to search Meta Structures in heterogeneous graphs (NC-MS for short). NC-MS implements a node level method that discover meaningful meta structures tailored to each node, capturing subtle differences in meta structure choices between nodes and providing nuanced identification. Additionally, NC-MS utilizes an efficient and differentiable network to enhance operational efficiency. Empirical studies across three real-world datasets validate the superiority of NC-MS, demonstrating its ability to outperform existing models in heterogeneous graph neural networks.

Graph Transformers (GTs) have emerged as powerful tools for handling graph-structured data through global attention mechanisms. While GTs can effectively capture long-range dependencies, they introduce difficulties in optimization due to their complex, non-differentiable operators, which cannot be directly handled by standard gradient-based optimizers (such as Adam or AdamW). To investigate the above issues, this work adopts the line of Zeroth-Order Optimization (ZOO) technique. However, direct integration of ZOO incurs considerable challenges due to the sharp loss landscape and steep gradients within the GT parameter space. Under the above observations, we propose a Sharpness-aware Zeroth-order Optimizer (SZO) that combines Sharpness-Aware Minimization (SAM) technique facilitating convergence within a flatter neighborhood, and leverages parallel computing for efficient gradient estimation. Theoretically, we provide a comprehensive analysis of the optimizer from both convergence and generalization perspectives. Empirically, we conduct extensive experiments on various classical GTs across a wide range of benchmark datasets, which underscore the superior performance of SZO over the state-of-the-art optimizers.

Recently link weight prediction has attracted an increasing research interest due to its merits in quantifying the strength between nodes within a graph. Nonetheless, current link weight prediction methods focus solely on graph topology, disregarding node feature information embedded in graphs. In real-world applications, we often collect heterogeneous graph data where multiple types of nodes linked by multiple types of edges are available for analysis, and it is essential and challenging to quantify the proximity of different types of nodes. To solve this challenge, we present a new model for Heterogeneous Graph Link Weight Prediction (HLWP for short). In HLWP, message passing in heterogeneous graph neural networks is described as a meta structure, which can be effectively designed by Differentiable Neural Architecture Search (DARTS) algorithms. Thus, HLWP can enhance the message passing in heterogeneous graphs by DARTS. In addition, HLWP employs a perturbation-based algorithm to enhance stability and precision. Through empirical experiments conducted on three real-world datasets, we demonstrate that HLWP achieves accurate predictions of link weights. Our results highlight the superiority of HLWP over existing methods for link weight prediction and baseline GNN models in terms of accurately predicting link weights within heterogeneous graphs.

Recent neural information retrieval models using dense text representations generated by pre-trained models commonly face two issues. First, a pre-trained model (e.g., BERT) usually truncates a long document before giving its representation, which may cause the loss of some important semantic information. Second, although pre-training models like BERT have been widely used in generating sentence embeddings, a substantial body of literature has shown that the pre-training models often represent sentence embeddings in a homogeneous and narrow space, known as the problem of representation anisotropy, which hurts the quality of dense vector retrieval. In this paper, we split the query and the document in information retrieval into two sets of natural sentences and generate their sentence embeddings with BERT, the most popular pre-trained model. Before aggregating the sentence embeddings to get the entire embedding representations of the input query and document, to alleviate the usual representation degeneration problem of sentence embeddings from BERT, we sample the variational auto-encoder's latent space distribution to obtain isotropic sentence embeddings and utilize supervised contrastive learning to uniform the distribution of these sentence embeddings in the representation space. Our proposed model undergoes training optimization for both the query and the document in the abovementioned aspects. Our model performs well in evaluating three extensively researched neural information retrieval datasets.

Graph neural networks (GNNs) are popularly used to analyze non-Euclidean graph data. Despite their successes, the design of graph neural networks requires heavy manual work and rich domain knowledge. Recently, neural architecture search algorithms are widely used to automatically design neural architectures for CNNs and RNNs. Inspired by the success of neural architecture search algorithms, we present a graph neural architecture search algorithm GraphNAS that enables automatic design of the best graph neural architecture based on reinforcement learning. Specifically, GraphNAS uses a recurrent network as the controller to generate variable-length strings that describe the architectures of graph neural networks, and trains the recurrent network with policy gradient to maximize the expected accuracy of the generated architectures on a validation data set. Moreover, based on GraphNAS, we design a new GraphNAS++ model using distributed neural architecture search. Compared with GraphNAS that generates and evaluates only one candidate architecture at each iteration, GraphNAS++ generates a mini-batch of candidate architectures and evaluates them in a distributed computing environment until convergence. Experiments on real-world datasets demonstrate that GraphNAS can design a novel network architecture that rivals the best human-invented architecture. Moreover, GraphNAS++ can speed up the design process at least five times by using the distributed training framework with GPUs.

Heterogeneous graphs are commonly used to describe networked data with multiple types of nodes and edges. Heterogeneous Graph Neural Networks (HGNNs) are powerful tools for analyzing heterogeneous graphs. However, designing neural architectures of HGNNs requires extensive domain knowledge and time-consuming manual work. Recently, neural architecture search algorithms have become popular in automatically designing neural architectures for homogeneous graph neural networks. In this paper, we present a Heterogeneous Graph Neural Architecture Search algorithm (HGNAS for short) which allows the automatic design of heterogeneous graph neural architectures. Specifically, HGNAS first designs a new search space based on existing popular HGNNs. Then, HGNAS uses a policy network as the controller to sample and find the best neural architecture from the designed search space by maximizing the expected accuracy of the selected architectures on a given validation dataset. Moreover, we design a new method HGNAS++ to improve the efficiency of HGNAS by training the RNN controller within a generative adversarial learning framework. The basic idea of HGNAS++ is to embed a pairwise ranker into the reinforcement learning based architecture search algorithm. The pairwise ranker can be taken as a discriminator which selects more accurate architectures between pairs of candidate architectures. Then, the RNN controller can be updated more efficiently by only using a relatively small number of candidate architectures selected by the pairwise ranker. Experiments on real-world heterogeneous graph datasets show that HGNAS is capable of designing novel HGNNs that beat the best human-invented HGNNs. On the benchmark datasets, HGNAS++ improves HGNAS in terms of evaluation cost, with a reduction of 50% of the evaluated candidate architectures and a decrease of 24% in search time on average. As a byproduct, HGNAS++ can find sparse yet powerful neural architectures for HGNNs.

Heterogeneous Graph Neural networks (HGNNs) have been popularly used in processing complicated networks such as academic networks, social networks, and knowledge graphs. Despite their success, the design of the neural architectures of HGNNs still requires rich domain knowledge and heavy manual work. In this paper, we propose a Heterogeneous Graph Neural Architecture Search algorithm (HGNAS for short) which enables automatic design of the best neural architectures with minimal human effort. Specifically, HGNAS first defines a general HGNN framework on top of existing popular HGNNs. A search space of HGNAS is designed based on the general framework that includes multiple groups of message encoding and aggregation functions. Then, HGNAS uses a policy network as the controller to sample and find the best neural architecture from the designed search space by maximizing the expected accuracy of the selected architectures on a validation dataset. Moreover, we introduce effective methods to improve HGNAS from three aspects, i.e., the optimization of hyper-parameters, the improvement of search space, and the selection of message receptive fields. Experiments on public datasets show that HGNAS is capable of designing novel HGNNs that rival the best human-invented HGNNs. More interestingly, HGNAS finds some sparse yet powerful neural architectures for HGNNs on the benchmark datasets.

Graph neural networks (GNNs) emerged recently as a powerful tool for analyzing non-Euclidean data such as social network data. Despite their success, the design of graph neural networks requires heavy manual work and domain knowledge. In this paper, we present a graph neural architecture search method (GraphNAS) that enables automatic design of the best graph neural architecture based on reinforcement learning. Specifically, GraphNAS uses a recurrent network to generate variable-length strings that describe the architectures of graph neural networks, and trains the recurrent network with policy gradient to maximize the expected accuracy of the generated architectures on a validation data set. Furthermore, to improve the search efficiency of GraphNAS on big networks, GraphNAS restricts the search space from an entire architecture space to a sequential concatenation of the best search results built on each single architecture layer. Experiments on real-world datasets demonstrate that GraphNAS can design a novel network architecture that rivals the best human-invented architecture in terms of validation set accuracy. Moreover, in a transfer learning task we observe that graph neural architectures designed by GraphNAS, when transferred to new datasets, still gain improvement in terms of prediction accuracy.