Handling communication overhead in large-scale tensor-parallel training remains a critical challenge due to the dense, near-zero distributions of intermediate tensors, which exacerbate errors under frequent communication and introduce significant computational overhead during compression. To this end, we propose TACO (Tensor-parallel Adaptive COmmunication compression), a robust FP8-based framework for compressing TP intermediate tensors. First, we employ a data-driven reshaping strategy combined with an Adaptive Scale–Hadamard Transform to enable high-fidelity FP8 quantization, while its Dual-Scale Quantization mechanism ensures numerical stability throughout training. Second, we design a highly fused compression operator to reduce memory traffic and kernel launch overhead, allowing efficient overlap with communication. Finally, we integrate TACO with existing state-of-the-art methods for Data and Pipeline Parallelism to develop a compression-enabled 3D-parallel training framework. Detailed experiments on GPT models and Qwen model demonstrate up to 1.87 × end-to-end throughput improvement while maintaining near-lossless accuracy, validating the effectiveness and efficiency of TACO in large-scale training.

Key-value (KV) storage is widely used in domains such as big data analytics, AI training, databases, and distributed file systems. Traditional KV systems built on DRAM or disk-based architectures struggle to meet the dual requirements of high throughput and strong data persistence demanded by modern applications. Non-Volatile Memory (NVM), with its combination of high bandwidth and persistence, offers a promising foundation for building high-performance, persistent, and largescale KV stores. Consequently, NVM-based KV storage has gained substantial research and industrial interest in recent years. However, directly porting DRAM-or disk-oriented KV designs to NVM devices often yields suboptimal performance. In Non-Uniform Memory Access (NUMA) architectures, frequent NVM persistence operations and high cross-NUMA access latency significantly limit I/O efficiency. To address these challenges, we propose SaneKV, an NVM-optimized KV store. SaneKV introduces a metadata asynchronous persistence mechanism that reduces I/O latency by aggregating metadata write-back operations, and an adaptive NVM data allocation policy to improve throughput. Experimental results show that for small-sized KV workloads, SaneKV achieves up to 75 % higher write throughput and 40 % higher read throughput than state-of-the-art NVM-based KV stores. For large-sized KV workloads, SaneKV achieves comparable peak throughput while requiring up to 40 % fewer threads to reach saturation and delivers over$2 \times$higher overall performance than prior NVM-based designs, demonstrating superior scalability and resource utilization.

Lossless compression is a classic technique for reducing data storage and transmission requirements. Asymmetric Numeral Systems (ANS) is a high-throughput, high-ratio lossless compression algorithm, but it lacks effective support for multi-byte data and cross-platform compatibility. To address this issue, we propose an Adaptive Data Mapping (ADM) scheme, which maps multi-byte integer data into single-byte space based on the data’s characteristics, improving the compression ratio of ANS while maintaining low encoding redundancy. We also optimize the ADM algorithm and the ANS encoder for GPU and CPU architectures, respectively, and combine them to create an efficient and portable ANS encoding method for multi-byte integer data, called MANS. Experimental results show that MANS improves compression ratios by an average of 1.24 ×, achieves 870.27MB/s throughput on CPUs, and delivers up to 288.45 × and 135.86 × speedups on an NVIDIA A100 and an AMD MI210 GPU compared to the CPU version—demonstrating its efficiency and portability across platforms.