Spiking Large Language Models (LLMs) have emerged as an energy-efficient alternative to conventional LLMs through their event-driven computation. To effectively obtain spiking LLMs, researchers develop different ANN-to-SNN conversion methods by leveraging pre-trained ANN parameters while inheriting the energy efficiency of SNN. However, existing conversion methods struggle with extreme activation outliers and incompatible nonlinear operations of ANN-based LLMs. To address this, we propose a loss-less ANN-SNN conversion for fully spike-driven LLMs, termed LAS. Specifically, LAS introduces two novel neurons to convert the activation outlier and nonlinear operation of ANN-based LLMs. Moreover, LAS tailors the spike-equivalent Transformer components for spiking LLMs, which can ensure full spiking conversion without any loss of performance. Experimental results on six language models and two vision-language models demonstrate that LAS achieves loss-less conversion. Notably, on OPT-66B, LAS even improves the accuracy of 2% on the WSC task. In addition, the parameter and ablation studies further verify the effectiveness of LAS.

Neural architecture search (NAS) aims to automatically design high-performance architectures of deep neural networks, which have shown great potential in various fields. However, the search process of NAS is computationally expensive since plenty of deep neural networks are trained to get the performance on GPUs. Performance predictors can directly estimate the performance of architectures without GPU-based training, thus can overcome this barrier. However, the construction of performance predictors requires labeling plenty of architectures sampled from the corresponding NAS search space, which is still prohibitively costly. In this paper, we propose a Domain Adaptive performance Predictor (DAP), which can construct a performance predictor based on the labeled architectures provided by existing benchmarks and then enable it to other search spaces via domain adaptive techniques. To achieve this, we first propose a domain-agnostic feature extraction method to refine the domain-invariant features of neural architectures. Then, we propose a novel embedding method to learn the shared representations of architecture operations. Experimental results demonstrate that DAP outperforms eight baselines upon six popular search spaces. Notably, we only require the search cost of 0:0002 GPU Days to find the architecture with 77:10% top-1 accuracy on ImageNet and 97:86% on CIFAR-10. In addition, we show the theoretical upper bound of the generalization error in the target search space, further illustrating the generalizability of DAP. The source code is available athttps://anonymous.4open.science/r/DAP-2F1F/.

Spiking neural networks (SNNs) have the merit of energy efficiency, and have been widely used for various real-world applications. Similar to other types of neural networks, the performance of SNN is also significantly decided by its architecture. In this paper, we propose an Evolutionary Multi-objective Spiking Neural Architecture Search (EMO-SNAS) method that completely enables the automatic design of SNN architectures with both high performance and low power consumption. To achieve this, we first design a variable-length encoding strategy for SNNs, addressing the issue that traditional encoding strategies need to manually set the depth in advance. Furthermore, we propose an exploitation operator focusing on the local search for the variable-length encoding, as well as an exploration operator focusing on the global search based on the temporal expansion. Based on NSGA-II, EMO-SNAS can greatly balance the performance and power consumption during the architecture design. Experiments on three widely used image classification datasets show that EMO-SNAS can achieve the best among the state-of-the-art methods. Specifically, EMO-SNAS gains 0.45%, 0.26%, and 7.52% in terms of classification accuracy, yet significantly contributes to 33%, 29%, and 12% fewer spike numbers on CIFAR10, CIFAR100, and TinyImageNet datasets. Ablation studies show that the temporal expansion can improve the performance of EMO-SNAS. Moreover, the measurement of power consumption and theoretical convergence of EMO-SNAS are also discussed to justify its component design. In addition, with EMO-SNAS, the impact of initial channels for SNNs is also systemically investigated, based on which a conclusion against existing consensus is achieved. The source code is available at https://github.com/songxt3/EMO-SNAS.

NAS can automatically design well-performing architectures of deep neural networks (DNNs), which have been widely investigated for real-world applications. However, neural architecture search (NAS) is computationally expensive because vast DNNs are trained to obtain the performance during the search process. Performance predictors can directly obtain the performance of DNNs without training, thus having great potential to overcome this barrier. However, existing performance predictors are typically offline, and are trained with rare architectures labeled. As a result, their predictive ability is limited due to ignoring the promising DNN architectures generated in the NAS process. In this article, we propose a Gaussian process-based online performance predictor (GPOPP) tailored to evolutionary NAS. To achieve this, GPOPP selects the proper architectures to update itself during the search process upon EI as an acquisition function. Further, as architectures cannot be directly used as training data, GPOPP includes a binary encoding schema that can convert the architectures into a suitable format. Although the original intention of designing GPOPP is to provide an alternative way to build promising performance predictors that may compromise performance, the experiments show that GPOPP can outperform most state-of-the-arts. For example, GPOPP-assisted NAS gains 76.1% in accuracy on ImageNet with only 1.1 graphic processing unit (GPU) Days. In addition, the ablation studies also demonstrate the effectiveness of the components designed in GPOPP. The source code is available at https://github.com/songxt3/GPOPP.