We observe two major trends in LLM-based generative AI: (1) inference is becoming the dominant factor in terms of cost and power consumption, surpassing training, and (2) retrieval augmented generation (RAG) is becoming prevalent. When processing long inputs in RAG, the prefill phase of computing the key-value vectors of input text is energy-intensive and time-consuming even with high-end GPUs. Thus, it is crucial to make the prefill phase in RAG inference efficient. To address this issue, we propose MatKV, a scheme that precomputes the key-value vectors (KVs) of RAG objects (e.g., documents), materializes them in inexpensive but fast and power-efficient flash storage, and reuses them at inference time instead of recomputing the KVs using costly and power-inefficient GPU. Experimental results using Hugging Face's Transformers library across state-of-the-art GPUs and flash memory SSDs confirm that, compared to full KV computation on GPUs, MatKV reduces both inference time and power consumption by half for RAG workloads, without severely impacting accuracy in the question-answering task. Furthermore, we demonstrate that MatKV enables additional optimizations in two ways. First, a GPU can decode text while simultaneously loading the materialized KVs for the next instance, reducing load latency. Second, since decoding speed is less sensitive to GPU performance than KV computation, low-end GPUs can be leveraged for decoding without significantly compromising speed once the materialized KVs are loaded into GPU memory. These findings underscore MatKV's potential to make large-scale generative AI applications more cost-effective, power-efficient, and accessible across a wider range of tasks and hardware environments.

The advent of persistent memory (PM) into our everyday computing environment is now imminent. New programming models and algorithms based on these models are being developed for such systems. However, current models require programs to be rewritten with persistence related primitives such as clflush and clwb or at least recompiled so that persistent mechanisms can be automatically inserted. This is a burden to program developers. Furthermore, executing legacy programs as-is can lead to application and system inconsistencies as unexpected faults occur. In this article, we propose µSnap, an operating system support that is based on checkpointing that allows legacy applications to be executed as-is without compromising consistency in systems that deploy PM. We implement a prototype of µSnap in the Linux kernel version 4.3.3, and measure and quantify the effect of µSnap for a wide range of applications. We find that µSnap incurs overhead for application execution compared to applications run without any notion of persistency, but that the overhead can be controlled to be minimal by appropriately setting the checkpointing interval. We argue that the benefit for paying this small cost can be tremendous in the sense that one can transparently guarantee the consistency of all legacy software written under the traditional programming model.