Modern deep neural networks (DNNs) are widely utilized across a broad range of domains, scaling rapidly and often comprising hundreds of diverse layers with varying types and configurations. To accelerate DNN execution, specialized hardware solutions, known as neural processing units (NPUs), have been developed. However, this heterogeneity of layers in a DNN model may cause performance degradation on NPUs. For example, while a layer’s execution or dataflow is generally associated with a specific data access order, the data layout in on-chip memory may not be well aligned with it, introducing bubble cycles for layout reordering. Given the hundreds of diverse layers in DNNs, this layout reordering overhead presents a new challenge for achieving efficient end-to-end DNN inference on NPUs. To address this problem, this article introduces HopScotch, a holistic approach to data layout-aware mapping of DNNs on NPUs. First, HopScotch adopts a routing interconnect between the on-chip memory and the systolic array utilizing three-input multiplexers, paired with an on-chip programmable vector processor to manage arbitrary data layout reordering at runtime. Additionally, it introduces a tailored data layout to accommodate a variety of convolutional configurations within the proposed microarchitecture. Second, HopScotch presents a novel layout mapping solver that employs a top-k selection strategy based on a beam search algorithm, facilitating the efficient exploration of the vast layout mapping space at compile time. Third, the proposed layout mapping solver is integrated into the HopScotch mapping framework (HMF) to explore the layout mapping space and evaluate the resulting performance. Experiments with popular DNN models show that HopScotch reduces layout reordering costs by up to 98.2% and 90.3%, resulting in speedups of 2.62× and 1.64× in end-to-end latency, compared to XLA and GCD 2 , respectively.

Accelerating end-to-end inference of transformer-based large language models (LLMs) is a critical component of AI services in datacenters. However, the diverse compute characteristics of LLMs' end-to-end inference present challenges as previously proposed accelerators only address certain operations or stages (e.g., self-attention, generation stage, etc.). To address the unique challenges of accelerating end-to-end inference, we propose IANUS - Integrated Accelerator based on NPU-PIM Unified Memory System. IANUS is a domain-specific system architecture that combines a Neural Processing Unit (NPU) with a Processing-in-Memory (PIM) to leverage both the NPU's high computation throughput and the PIM's high effective memory bandwidth. In particular, IANUS employs a unified main memory system where the PIM memory is used both for PIM operations and for NPU's main memory. The unified main memory system ensures that memory capacity is efficiently utilized and the movement of shared data between NPU and PIM is minimized. However, it introduces new challenges since normal memory accesses and PIM computations cannot be performed simultaneously. Thus, we propose novel PIM Access Scheduling that manages not only the scheduling of normal memory accesses and PIM computations but also workload mapping across the PIM and the NPU. Our detailed simulation evaluations show that IANUS improves the performance of GPT-2 by 6.2× and 3.2×, on average, compared to the NVIDIA A100 GPU and the state-of-the-art accelerator. As a proof-of-concept, we develop a prototype of IANUS with a commercial PIM, NPU, and an FPGA-based PIM controller to demonstrate the feasibility of IANUS.

A new in-loop filter, sample adaptive offset (SAO) allows the latest video compression standard, HEVC or H.265, to achieve higher coding efficiency both in objective and subjective measures. However, it requires additional operations to estimate the best SAO parameters per coding tree unit (CTU) during the video encoding process, which often leads to many practical issues such as high computational complexity or architectural inefficiency that might not be affordable for low power or real-time video encoders. In this paper, we explore various SAO encoding policies including complexity reduction in the SAO parameter estimation and efforts to remove inefficiency caused by its implementation in a pipelined manner through numerous experimental tests.

In this paper, we present the fast parameter estimation algorithm for sample adaptive offset (SAO) in High Efficiency Video Coding (HEVC). The main idea of a proposed method is to simplify decision of the best SAO edge offset (EO) class based on the dominant edge direction information instead of searching all EO classes exhaustively. Prior to SAO encoding process, an edge map is created by using edge operator to detect edge detections on the image and a local edge direction histogram is then established for each coding tree block (CTB). Based on the distribution of the edge direction histogram, only one EO class is chosen for rate-distortion optimization (RDO) calculation. Experimental results show that the proposed method saves around 42.1% of the SAO encoding time only with 0.25% BD-rate degradation compared to HEVC reference software version 12.0.

New fusion memory devices consisting of multiple heterogeneous memory components in a single die or package offer efficient ways to optimize embedded systems in terms of energy, performance, and cost. Samsung Electronics recently announced the OneNAND fusion memory, in which a NAND flash array is integrated with dual SRAM buffers to provide a nor-type I/O interface. OneNAND has the low cost and large capacity of a NAND flash but also permits eXecution-in-Place (XIP) like a nor flash. The deployment of such devices requires careful system-level resource management because of their impact on energy consumption and performance, and existing memory optimization techniques, such as the demand paging used with NAND flash, may no longer be appropriate for systems with a fusion memory. We introduce a new online demand paging scheme that fully exploits the XIP capability of OneNAND flash by classifying pages as load preferred (residing in the on-chip SRAM) and XIP preferred (accessed directly from the OneNAND flash and discarded after use). This achieves, on average, a 26% reduction in energy consumption and a 19% increase in performance, compared with conventional NAND flash demand paging.

Most digital systems are equipped with dc-dc converters to supply various levels of voltages from batteries to logic devices. DC-DC converters maintain legal voltage ranges regardless of the load current variation as well as battery voltage drop. Although the efficiency of dc-dc converters is changed by the output voltage level and the load current, most existing power management techniques simply ignore the efficiency variation of dc-dc converters. However, without a careful consideration of the efficiency variation of dc-dc converters, finding a true optimal power management will be impossible. In this paper, we solve the problem of energy minimization with the consideration of the characteristics of power consumption of dc-dc converters. Specifically, the contributions of our work are as follows: 1) We analyze the effects of the efficiency variation of dc-dc converters on a single-task execution in dynamic voltage scaling (DVS) scheme and propose the technique for dc-dc converter-aware energy-minimal DVS. 2) is then extended to embed an awareness of the characteristics of dc-dc converters in general DVS techniques for multiple tasks. 3) We go on to propose a technique called for generating a dc-dc converter that is most energy efficient for a particular application. 4) We also present an integrated framework, i.e., , based on and , which addresses dc-dc converter configuration and DVS simultaneously. Experimental results show that is able to save up to 24.8% of energy compared with previous power management schemes, which do not consider the efficiency variation of dc-dc converters.

Most digital systems are equipped with DC-DC converters to supply various levels of voltages from batteries to logic devices. DCDC converters maintain legal voltage ranges regardless of the load current variation as well as battery voltage drop. Although the efficiency of DC-DC converters is changed by the output voltage level and the load current, most existing power management techniques simply ignore the efficiency variation of DC-DC converters. However, without a careful consideration of the efficiency variation of DC-DC converters, finding a true optimal power management will be impossible. In this work, we solve the problem of energy minimization with the consideration of the characteristics of power consumption of DC-DC converter. Specifically, the contributions of our work are: (1) We analyze the effects of the efficiency variation of DC-DC converters on a single task execution in DVS (dynamic voltage scaling) scheme, and propose a technique, called DC_DVS, of DC-DC converter-aware energy-minimal DVS; (2) DC_DVS is then extended to combine the effects of DC-DC converters with the procedures of general DVS techniques with multiple tasks; (3) Conversely, we propose a technique, called DC_CONF, of generating a DC-DC converter that is best suited, in terms of total energy efficiency, to the intended application, and (4) finally, we complete our integrated framework DC-lp, which is based on DC_DVS and DC_CONF, that attempts to solve the DC-DC converter configuration selection problem and the DVS problem simultaneously. To show the effectiveness of the proposed techniques, a set of experimental results is provided. In summary, it is shown that DC-lp is able to save 16.0%~22.1% of energy on the average, which otherwise was dissipated in the previous power management schemes with no consideration of DC-DC converter efficiency variation.

Memory systems are dominant energy consumers, and thus many energy reduction techniques for memory buses and devices have been proposed. For practical energy reduction practices, we have to take into account the interaction between a processor and cache memories together with application programs. Furthermore, energy characterization of memory systems must be accurate enough to justify various techniques. In this article, we build an in-house energy simulator for memory systems that is accelerated by special hardware support while maintaining accuracy. We explore energy behavior of memory systems for various values of the processor and memory clock frequencies and cache configuration. Each experiment is performed with 24M instruction steps of real application programs to guarantee accuracy.The simulator is based on precise energy characterization of memory systems including buses, bus drivers, and memory devices by a cycle-accurate energy measurement technique. We characterize energy consumption of each component by an energy state machine whose states and transitions are associated with the dynamic and static energy costs, respectively. Our approach easily characterizes the energy consumption of complex SDRAMs. We divide and quantify energy components of main memory systems for high-level reduction. The energy simulator enables us to devise practical energy reduction schemes by providing the actual amount of reduction out of the total energy consumption in main memory systems. We introduce several practical energy reduction techniques for SDRAM memory systems and demonstrate energy reduction ratio over the SDRAM memory systems with commercial SDRAM controller chipsets. We classify the SDRAM memory systems into high-performance and mid-performance classes and achieve suitable system configurations for each class. For instance, a typical high-performance 32-bit, 64 MB SDRAM memory system consumes 19.6 mJ, 33.8 mJ, 35.4 mJ, and 37.0 mJ for 24M instructions of an MP3 decoder, a JPEG compressor, a JPEG decompressor, and an MPEG4 decoder, respectively. Our reduction scheme saves 12.7 mJ, 15.1 mJ, 15.5 mJ, and 14.8 mJ, and the reduction ratios are 64.8%, 44.6%, 43.8%, and 40.1%, respectively, without compromising execution speed.

In this paper, we introduce a precise energy characterization of SDRAM main memory systems and explore the amount of energy associated with design parameters, leading to energy reduction techniques that we are able to recommend for practical use.