Transformer-based Large Language Models (LLMs) rely on both General Matrix-Matrix Multiplication (GEMM) and General Matrix-Vector Multiplication (GEMV) for inference. While existing Processing-in-Memory (PIM) architectures, like HBM-PIM and Newton, efficiently handle GEMV operations, they suffer from low computing unit utilization during GEMM due to frequent input preparation and activation cycles. To address this, we propose Dual Half-mode PIM (DH-PIM), an extension of the Half Division Mode from previous PipePIM. DH-PIM splits sense amplifiers, and computing units into halves, allowing them to execute different operations independently. Dual half mode, which allows both halves of the computing unit to process data broadcast from either half of the sense amplifier, significantly enhances utilization via overlapped and pipelined execution. Our simulations show DH-PIM improves GEMM performance by up to 2.06x in GEMM kernels, with about 1.46x speedup in LLM tasks, compared to HBM-PIM. These performance improvements are achieved with minimal architectural modifications, resulting in low area and energy overhead.

For future exascale computing systems, ultra-high-density memories would be required that consume low power to process massive data. Of the various memory devices, 3D-stacked DRAMs using TSVs are a perfect solution for this purposes. In addition to providing high capacity, these provide functional flexibility to the computing system by attaching a logic die in each 3D-stacked DRAM chip. However, the high capacity 3D-stacked DRAMs suffer from a significant loss of refresh power, which is solely required to maintain data. Although various schemes have been proposed to mitigate this issue, they cannot be adopted by commercial products due to compatibility issues. To tackle this issue, we propose EXTREME, which effectively reduces the refresh power of 3D-Stacked DRAMs. In order to retain the compatibility with OS, a simple page table manager is implemented at the logic die of 3D-stacked DRAM devices, which pins the page table to a specific memory space. The experiment results demonstrate that this reduces the refresh power at idle time by up to 98 percent with 16 KB of SRAM (static RAM) and 64 KB of DRAM register overhead for a 2 GB 3D-stacked DRAM device.