As vector databases scale in Retrieval-Augmented Generation (RAG), the retrieval phase increasingly bottlenecks end-to-end latency. While Compute Express Link (CXL) offers scalable memory expansion, naïve CXL deployments suffer from intra-device bandwidth saturation and inter-device load imbalance, which collectively hinder system responsiveness.

Fully homomorphic encryption (FHE) is a promising solution for privacy-preserving cloud computing as it enables computations on sensitive data without any risk of data leakage. Despite the significant attention FHE has received, substantial computation and memory demands make it hardly practical for real-world applications. We propose a readily available and practical hardware solution to tackle this problem by using GPUs. GPUs have adequate computational and memory resources to handle complex operations in FHE, including number-theoretic transform (NTT), on which most prior work has deeply focused. However, through detailed analyses, we discover that the performance bottleneck on GPUs is primarily due to simpler element-wise operations, which are limited by off-chip memory (DRAM) bandwidth. Motivated by these observations, we develop Anaheim, a processing-in-memory (PIM) architecture for FHE. We develop optimized FHE execution flows and an end-toend software framework for using PIM with GPUs. Also, we design a versatile PIM unit that handles various modular integer arithmetic PIM instructions, along with an efficient data mapping and associated PIM execution algorithms that minimize data access overhead by leveraging the internal structure of DRAM. Our concerted efforts substantially enhance the performance and energy efficiency of various FHE workloads on GPUs.

An embedding layer is one of the most critical building blocks of deep neural networks, especially for recommender systems and graph neural networks. The embedding layer dominates a large portion of the total execution time due to its large memory requirements and little data reuse in operations. To accelerate the embedding layers, dual in-line memory module (DIMM) based near-data processing architectures have been proposed. They amplify bandwidth by adding a processing unit to the DIMM’s buffer. However, prior architectures have less capacity scalability due to the limited number of memory channels. Crucially, they are limited in performance improvement due to the load imbalance problem and the limitations of DIMM-based memory systems with a multi-drop bus structure between the processing units and the host.

Large language models (LLMs) have emerged due to their capability to generate high-quality content across diverse contexts. To reduce their explosively increasing demands for computing resources, a mixture of experts (MoE) has emerged. The MoE layer enables exploiting a huge number of parameters with less computation. Applying state-of-the-art continuous batching increases throughput; however, it leads to frequent DRAM access in the MoE and attention layers. We observe that conventional computing devices have limitations when processing the MoE and attention layers, which dominate the total execution time and exhibit low arithmetic intensity (Op/B). Processing MoE layers only with devices targeting low-Op/B such as processing-in-memory (PIM) architectures is challenging due to the fluctuating Op/B in the MoE layer caused by continuous batching, To address these challenges, we propose Duplex, which comprises xPU tailored for high-Op/B and Logic-PIM to effectively perform low-Op/B operation within a single device. Duplex selects the most suitable processor based on the Op/B of each layer within LLMs. As the Op/B of the MoE layer is at least 1 and that of the attention layer has a value of 4–8 for grouped query attention, prior PIM architectures are not efficient, which place processing units inside DRAM dies and only target extremely low-Op/B (under one) operations. Based on recent trends, Logic-Pimadds more through-silicon vias (TSVs) to enable high-bandwidth communication between the DRAM die and the logic die and place powerful processing units on the logic die, which is best suited for handling low-Op/B operations ranging from few to a few dozens. To maximally utilize the xPU and Logic-Pim,we propose expert and attention co-processing. By exploiting proper processing units for MoE and attention layers, Duplex shows up to 2.67 × higher throughput and consumes 42.0% less energy compared to GPU systems for LLM inference.

Graph Neural Network (GNN) models have attracted attention, given their high accuracy in interpreting graph data. One of the primary building blocks of a GNN model is aggregation, which gathers and averages the feature vectors corresponding to the nodes adjacent to each node. Aggregation works by multiplying the adjacency and feature matrices. The size of both matrices exceeds the on-chip cache capacity for many realistic datasets, and the adjacency matrix is highly sparse. These characteristics lead to little data reuse, causing intensive main-memory accesses during the aggregation process. Thus, aggregation exhibits memory-intensive characteristics and dominates most of the total execution time. In this paper, we propose GraNDe, an NDP architecture that accelerates memory-intensive aggregation operations by locating NDP modules near DRAM datapath to exploit rank-level parallelism. GraNDe maximizes bandwidth utilization by separating the memory channel path with the buffer chip in between so that pre-/post-processing in the host processor and reduction in NDP modules operate simultaneously. By exploring the preferred data mappings of the operand matrices to DRAM ranks, we architect GraNDe to support adaptive matrix mapping that applies the optimal mapping for each layer depending on the dimension of the layer and the configuration of a memory system. We also propose adj-bundle broadcasting and re-tiling optimizations to reduce the transfer time for adjacency matrix data and to improve feature vector data reusability by exploiting tiling with consideration of adjacency between nodes. GraNDe achieves 3.01× and 1.69× on average, and up to$4.00\times$and$1.98\times$speedups of GCN aggregation over the baseline system and the state-of-the-art NDP architecture for GCN, respectively.

Personalized recommendation systems are gaining significant traction due to their industrial importance. An important building block of recommendation systems consists of the embedding layers, which exhibit a highly memory-intensive characteristic. A fundamental primitive of embedding layers is the embedding vector gathers followed by vector reductions, exhibiting low arithmetic intensity and becoming bottlenecked by the memory throughput. To tackle such a challenge, recent proposals employ a near-data processing (NDP) solution at the DRAM rank-level, achieving impressive performance speedups. We observe that prior rank-level-parallelism-based NDP solutions leave significant performance potential on the table as they do not fully reap the abundant transfer throughput inherent in DRAM datapaths.