Large Language Models (LLMs) are constrained by their large parameter size and high computational resource consumption, making deployment on terminal devices with limited computing resources difficult. This severely hinders the deployment and application of LLMs in power grid systems. To address this, we propose a hybrid structured pruning method based on layer and channel pruning to compress LLMs, enabling their efficient deployment in power grid systems. Our method first removes noncritical layers based on inter-layer cosine similarity, thereby achieving a significant reduction in model parameters. Then channel-pruning based on PCA is employed, aiming to improve model inference speed while preserving performance. Experimental results on Wikitext2 and PTB demonstrate that at a 30% pruning rate, our method exhibits a superior performance retention compared to single-layer pruning and channel-wise pruning methods. For generation tasks, the model inference speed is increased by 5.91%. Our work provides a novel insight for future research on structured pruning, and inspires more research on composite pruning from multi-dimensional perspectives.

Neural architecture performance predictor is an efficient approach for architecture estimation in Neural Architecture Search (NAS). However, existing predictors based on Graph Neural Networks (GNNs) are deficient in modeling long-range interactions between operation nodes and prone to the problem of over-smoothing, which limits their ability to learn neural architecture representation. Furthermore, some Transformer-based predictors use simple position encodings to improve performance via self-attention mechanism, but they fail to fully exploit the subgraph structure information of the graph. To solve this problem, we propose a novel method to enhance the graph representation of neural architectures by combining GNNs and Transformer blocks. We evaluate the effectiveness of our predictor on NAS-Bench-101 and NAS-bench-201 benchmarks, the discovered architecture on DARTS search space achieves an accuracy of 97.61% on CIFAR-10 dataset, which outperforms traditional position encoding methods such as adjacency and Laplacian matrices. The code of our work is available at \url{https://github.com/GNET}.

Performance estimation of neural architecture is a crucial component of neural architecture search (NAS). Meanwhile, neural predictor is a current mainstream performance estimation method. However, it is a challenging task to train the predictor with few architecture evaluations for efficient NAS. In this paper, we propose a graph masked autoencoder (GMAE) enhanced predictor, which can reduce the dependence on supervision data by self-supervised pre-training with untrained architectures. We compare our GMAE-enhanced predictor with existing predictors in different search spaces, and experimental results show that our predictor has high query utilization. Moreover, GMAE-enhanced predictor with different search strategies can discover competitive architectures in different search spaces. Code and supplementary materials are available at https://github.com/kunjing96/GMAENAS.git.

Graph data is ubiquitous in the real world and graph neural networks (GNNs) are effective for modeling the complex relationships and dependencies between the entities. However, it's difficult to design data-specific GNNs. Recently, researchers have started to apply neural architecture search (NAS) to design GNNs. In this work, we propose a graph architecture search method to decrease the instability with a large number of candidate operations. Following SANE(Search to Aggregate NEighborhood), we focus on searching to aggregate neighbourhoods but we divide the candidate operations into groups. We use a continuous relaxation of our search space and optimize the hyper-networks with a gradient-based algorithm. Extensive experiments on several node-level and graph-level tasks demonstrate that our method achieves a promising performance.

Recently, neural architecture search (NAS) has emerged as a technique of growing concern in automatic machine learning (AutoML). Meanwhile, attention-based models, such as attention-based recurrent neural network, transformer-based model, etc., have been widely used in deep learning applications. However, there is no efficient NAS method that can search the architecture of attention-based model so far. To solve this problem, we propose a framework named neural architecture search for attention-based networks (NASABN) by abstracting attention-based models and extracting undefined parts of the model, including the attention layers and cells. NASABN is flexible and general enough to fit different NAS methods, which can also be transferred across different datasets. We conduct extensive experiments with NASABN using gradient descent-based methods like DARTS on Penn Treebank (PTB) and WikiText-2 (WT2) datasets respectively, and achieve competitive performance compared with the state-of-the-art methods.