Vector quantization (VQ), which treats a vector as a compression unit, gains increasing research interests for its potential to accelerate large language models (LLMs). Compared to conventional element-wise quantization methods, VQ algorithms can compress weight and KV cache tensors in LLMs with a greater ratio while maintaining the high model accuracy. However, translating a VQ algorithm’s memory reduction into the actual latency improvement is challenging. We profile and analyze the current approach of integrating VQ into computation kernels and show that its major inefficiency lies in the poor access efficiency of codebooks in VQ algorithms and uncoordinated computation dataflow. Meanwhile, the diversity of VQ algorithms (e.g., different vector sizes and entry counts) and LLMs, computation kernels (e.g matrix-matrix/vector multiplication and attention computation) makes it impractical to manually craft efficient kernel implementations for each specific case. In this work, we design and implement VQ-LLM, an efficient fused VQ kernel generation framework. We first introduce a software abstraction called codebook cache to optimize codebook access efficiency and support the integration of VQ with various computations. The codebook cache adaptively stores different entries across the GPU’s memory hierarchy, including off-chip global memory, on-chip shared memory, and registers. Centered around the codebook cache, we design an efficient computation engine that optimizes memory traffic during computations involving codebooks. This compute engine adopts the codebook-centric dataflow and fusion optimizations. Additionally, we provide adaptive heuristics to tailor parameter selection in our optimizations to diverse VQ configurations. Our optimizations achieve the latency reduction of $\mathbf{6 4. 3 6 \%}$ to $\mathbf{9 9. 1 \%}$ compared to existing open-source implementations. A final comparison with state-of-the-art element-wise quantization methods like AWQ and QoQ shows that our VQ-LLM is practically viable, achieving latencies close or even better latencies to those at equivalent bit-widths, potentially offering greater accuracy.

Deep learning has significantly enhanced protein structure prediction, and AlphaFold2 marked a particular milestone among these methods for predicting protein monomer and complex structures. The AlphaFold3 represents a pivotal further advancement in biomolecular structure prediction, extending beyond proteins to model diverse assemblies. Despite attracting a huge number of users, there is still an absence of third-party benchmarks to fairly demonstrate the performance of the AlphaFold3. In this work, we benchmark AlphaFold3's performance across nine datasets, protein monomers, orphan proteins, alternative conformations, protein multimers, peptide-protein complexes, antigen-antibody complexes, RNA, RNA multimers, and protein-nucleic acid complexes, compared to AlphaFold2, AlphaFold-Multimer, and RoseTTAFoldNA, RhoFold+, NuFold and trRosettaRNA. For protein monomers, AlphaFold3 demonstrates improved local structural accuracy over AlphaFold2, though global accuracy gains are limited. In modeling general protein complexes, AlphaFold3 surpasses AlphaFold-Multimer in local structural prediction. For peptide-protein complexes, their performances are nearly indistinguishable, whereas on antigen-antibody complexes, AlphaFold3 is significantly superior. AlphaFold3 shows substantial superiority over RoseTTAFoldNA in protein-nucleic acid predictions, with significant gains in TM-score, local distance difference test scores, and interaction network fidelity scores, whereas for RNA multimers its advantage is limited to significant gains in local distance difference test scores. For RNA monomers, trRosettaRNA achieves higher global prediction accuracy. These results highlight AlphaFold3's ability to predict both structural detail and interactions, positioning it as a versatile tool for diverse biomolecular systems and suggesting promising applications in structural biology and molecular interaction research, while at the same time highlighting areas ripe for continuing improvements in performance.

Approximate nearest neighbor (ANN) search is a widely applied technique in modern intelligent applications, such as recommendation systems and vector databases. Therefore, efficient and high-throughput execution of ANN search has become increasingly important. In this paper, we first characterize the state-of-the-art product quantization-based method of ANN search and identify a significant source of inefficiency in the form of unnecessary pairwise distance calculations and accumulations. To improve efficiency, we propose Juno, an end-to-end ANN search system that adopts a carefully designed sparsity- and locality-aware search algorithm. We also present an efficient hardware mapping that utilizes ray tracing cores in modern GPUs with pipelined execution on tensor cores to execute our sparsity-aware ANN search algorithm. Our evaluations on four datasets from 1 to 100 million search points demonstrate 2.2×-8.5× improvements in search throughput. Moreover, our algorithmic enhancements alone achieve a maximal 2.6× improvement on the hardware without the acceleration of the RT core.