Siheng Li, Kejiao Li, Zenan Xu, Guanhua Huang, Kun Li, Haoyuan Wu, Wujiajia, Zihao Zheng, Chenchen Zhang, Kun Shi, Xue Gong, Qi Yi, Ruibin Xiong, Tingqiang Xu, Yuhao Jiang, Jianfeng Yan, Yuyuan Zeng, Guanghui Xu, Jinbao Xue, Zhijiang xu, Zheng Fang, Shuai LI, Qibin Liu, Xiaoxue Li, Zhuoyu Li, Yangyu Tao, Fei Gao, Cheng Jiang, Bochao Wang, Kai Liu, Jianchen Zhu, Wai Lam, Bo Zhou, Di Wang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Lookup-table (LUT) based neural networks can deliver ultra-low latency and excellent hardware efficiency on FPGAs by mapping arithmetic operations directly onto the logic primitives. However, state-of-the-art LUT-aware training (LAT) approaches remain difficult to use in practice: they are often orders of magnitude slower to train than conventional networks, require non-trivial manual tuning for hardware efficiency, and lack an end-to-end work flow. This work presents HGQ-LUT1, a new LAT approach that achieves state-of-the-art hardware efficiency while accelerating training by over 100 times on modern GPUs. HGQ-LUT introduces LUT-Dense and LUT-Conv layers that are implemented with regular, accelerator-Efficient tensor operations during training, which are then compiled into logic LUTs for hardware. By combining these layers with ne-grained, element-wise heterogeneous quantization (including zero-bit pruning) and a LUT-aware resource surrogate, HGQ-LUT enables the automatic exploration of accuracy-resource trade-offs without manual bit-width tuning. We further integrate HGQ-LUT into open-source toolchains, enabling unified design, compilation, and bit-exact verification of hybrid architectures that mix LUT-based with conventional arithmetic blocks. These features make LAT-based DNNs practical for real-world deployment, such as at the CERN Large Hadron Collider’s experiments.

Nowadays, Large Language Models (LLMs) have been trained using extended context lengths to foster more creative applications. However, long context training poses great challenges considering the constraint of GPU memory. It not only leads to substantial activation memory consumption during training, but also incurs considerable memory fragmentation. To facilitate long context training, existing frameworks have adopted strategies such as recomputation and various forms of parallelisms. Nevertheless, these techniques rely on redundant computation or extensive communication, resulting in low Model FLOPS Utilization (MFU). In this paper, we propose MEMO, a novel LLM training framework designed for fine-grained activation memory management. Given the quadratic scaling of computation and linear scaling of memory with sequence lengths when using FlashAttention, we offload memory-consuming activations to CPU memory after each layer's forward pass and fetch them during the backward pass. To maximize the swapping of activations without hindering computation, and to avoid exhausting limited CPU memory, we implement a token-wise activation recomputation and swapping mechanism. Furthermore, we tackle the memory fragmentation issue by employing a bi-level Mixed Integer Programming (MIP) approach, optimizing memory reuse across transformer layers. Empirical results demonstrate that MEMO achieves an average of 1.97x and 1.80x MFU compared to Megatron-LM and DeepSpeed, respectively. This improvement is attributed to MEMO's ability to minimize memory fragmentation, reduce recomputation and intensive communication, and circumvent the delays associated with the memory reorganization process due to fragmentation. By leveraging fine-grained activation memory management, MEMO facilitates efficient training of 7B LLM with 1 million sequence length on just 8 A800 GPUs, achieving an MFU of 52.30%.

Larger transformer models perform better on various downstream tasks but require more cost to scale up the model size. To efficiently enlarge models, the Mixture-of-Expert (MoE) architecture is widely adopted, which consists of a gate network and a series of experts and keep the training cost constant by routing the input data to a fixed number of experts instead of all. In existing large-scale MoE training systems, experts would be distributed among different GPUs for parallelization, and thus input data requires additional all-to-all communication to access the target expert and conduct corresponding computation. However, upon evaluating the training process of three mainstream MoE models on commonly used GPU clusters, we found that the all-to-all communication ratio averaged around 45\%, which significantly hinders the training efficiency and scalability of MoE models. In this paper, we propose LSH-MoE, a communication-efficient MoE training framework using locality-sensitive hashing (LSH). We first present the problems of scaling MoE training in existing systems and highlight the potential of exploiting token similarity to facilitate data compression. Then, we introduce an efficient LSH-based compression technique, which utilizes the cross-polytope hashing for rapid clustering and implements a residual-based error compensation scheme to alleviate the adverse impact of compression. To verify the effectiveness of our methods, we conduct experiments on both language models (e.g., RoBERTa, GPT, and T5) and vision models (e.g., Swin) for both pre-training and fine-tuning tasks. The results demonstrate that our method substantially outperforms its counterparts across different tasks by 1.28-2.2$\times$ of speedup.

The electrical capacitance tomography technology has potential benefits for the process industry by providing visualization of material distributions. One of the main technical gaps and impediments that must be overcome is the low-quality tomogram. To address this problem, this study introduces the data-guided prior and combines it with the electrical measurement mechanism and the sparsity prior to produce a new difference of convex functions programming problem that turns the image reconstruction problem into an optimization problem. The data-guided prior is learned from a provided dataset and captures the details of imaging targets since it is a specific image. A new numerical scheme that allows a complex optimization problem to be split into a few less difficult subproblems is developed to solve the challenging difference of convex functions programming problem. A new dimensionality reduction method is developed and combined with the relevance vector machine to generate a new learning engine for the forecast of the data-guided prior. The new imaging method fuses multisource information and unifies data-guided and measurement physics modeling paradigms. Performance evaluation results have validated that the new method successfully works on a series of test tasks with higher reconstruction quality and lower noise sensitivity than the popular imaging methods.

The temperature distribution in real‐world industrial environments is often in a three‐dimensional (3D) space, and developing an efficient algorithm to reconstruct such volume information may be beneficial for increasing the system efficiency, improving the energy saving and reducing the pollutant emission. A new algorithm is put forward for 3D temperature distribution reconstruction (3DTDR) tasks according to the given finite temperature observation data. Owing to the distinct advantages, including fast learning speed, good generalisation ability etc. a new robust regularised extreme learning machine (RRELM) algorithm is developed for the 3DTDR task. Unlike existing tomography‐based measurement methods and local point measurement technologies, the proposed algorithm can reconstruct 3D temperature distributions on the basis of the given local temperature observation information. Furthermore, different from available inverse heat transfer problems, the proposed reconstruction method does not solve complicated partial differential equations. Numerical simulation results verify the effectiveness of the RRELM algorithm for 3DTDR problems. Furthermore, the proposed RRELM method has a satisfactory robustness for the noises in the measurement data and can achieve the high‐quality reconstruction when the sampling ratio is low. As a result, an effective algorithm is introduced for the 3DTDR problem.