Modern DNNs are going deeper and wider to achieve higher accuracy. However, existing deep learning frameworks require the whole DNN model to fit into the GPU memory when training with GPUs, which puts an unwanted limitation on training large models. Utilizing NVIDIA Unified Memory (UM) could inherently support training DNN models beyond GPU memory capacity. However, naively adopting UM would suffer a significant performance penalty due to the delay of data transfer. In this paper, we propose PUMP, a Profiling-free Unified Memory Prefetcher. PUMP exploits GPU asynchronous execution for prefetch; that is, there exists a delay between the time that CPU launches a kernel and the time the kernel executes in GPU. PUMP extracts memory blocks accessed by the kernel when launching and swaps these blocks into GPU memory. Experimental results show PUMP achieves about 2x speedup on the average compared to the baseline that naively enables UM.

DRAMs are used as the main memory in most computing systems today. Studies show that DRAMs contribute to a significant part of overall system power consumption. One of the main challenges in low-power DRAM design is the inevitable refresh process. Due to process variation, memory cells exhibit retention time variations. Current DRAMs use a single refresh period determined by the cell with the largest leakage. Since prolonging refresh intervals introduces retention errors, a set of previous works adopt conventional error-correcting code (ECC) to correct retention errors. However, these approaches introduce significant area and energy overheads. In this article, we propose a novel error correction framework for retention errors in DRAMs, called SECRET (selective error correction for refresh energy reduction). The key observations we make are that retention errors are hard errors rather than soft errors, and only few DRAM cells have large leakage. Therefore, instead of equipping error correction capability for all memory cells as existing ECC schemes, we only allocate error correction information to leaky cells under a refresh interval. Our SECRET framework contains two parts: an offline phase to identify memory cells with retention errors given a target error rate and a low-overhead error correction mechanism. The experimental results show that among all test cases performed, the proposed SECRET framework can reduce refresh power by 87.2% and overall DRAM power up to 18.57% with negligible area and performance overheads.

DRAMs are used as the main memory in most computing systems today. Studies show that DRAMs contribute to a significant part of overall system power consumption. Therefore, one of the main challenges in low-power DRAM design is the inevitable refresh process. Due to process variation, memory cells exhibit retention time variations. Current DRAMs use a single worst-case refresh period. Prolonging refresh intervals introduces retention errors. Previous works adopt conventional ECC (Error Correcting Code) to correct retention errors. These approaches introduce significant area and energy overheads. In this paper, we propose a novel error correction framework for retention errors in DRAMs, called SECRET (Selective Error Correction for Refresh Energy reducTion). The key observation we make is that retention errors can be treated as hard errors rather than soft errors, and only few DRAM cells have large leakage. Therefore, instead of equipping error correction capability in all memory cells as existing ECC schemes, we only allocate error correction information to leaky cells under a refresh interval. Our SECRET framework contains two parts, an off-line phase to identify memory cells with retention errors given a target error rate, and a low-overhead error correction mechanism. The experimental results show that the proposed SECRET framework can reduce refresh power by 87.2%, and overall DRAM power by 18.57% with negligible area and performance overheads.

Memory Wall is a well-known obstacle to processor performance improvement. The popularity of multi-core architecture will further exaggerate the problem since the memory resource is shared by all cores. Interferences among requests from different cores may prolong the latency of memory accesses thereby degrading the system performance. To tackle the problem, this paper proposes to decouple application threads into compute and memory tasks, and restrict the number of concurrent memory tasks to avoid the interference among memory requests. Yet with this scheduling restriction, a CPU core may unnecessarily stay idle, which incurs adverse impact on the overall performance. Therefore, we develop a memory thread throttling mechanism that tunes the allowable memory threads dynamically under workload variation to improve system performance. The proposed run-time mechanism monitors memory and computation ratios of a program for phase detection. It then decides the memory thread constraint for the next program phase based on an analytical model that can estimate system performance under different constraint values. To prove the concept, we prototype the mechanism in some real-world applications as well as synthetic workloads. We evaluate their performance on real machines. The experimental results demonstrate up to 20% speedup with a pool of synthetic workloads on an Intel i7 (Nehalem) machine and match with the speedup estimated by the proposed analytical model. Furthermore, the intelligent run-time scheduling leads to a geometric mean of 12% performance improvement for real-world applications on the same hardware.

With the popularity of multi-core architecture, to sustain the memory demands from different cores, the memory system is expected to grow significantly in both speed and capacity. This will lead to increasing power consumption in the memory system. Therefore, it is critical to address the power issue in the memory subsystem. In designing a power-aware memory system, due to the interplay among power, thermal and performance, all the three factors need to be taken into account. In this paper, we propose the first joint performance, power and thermal management framework (PPT) through orchestrating task execution and page allocation. The PPT framework adapts to system loading to maximize power saving and avoid memory hotspot at the same time whiling sustaining the system bandwidth demand.

As technology continues to shrink, reducing leakage power of Field-Programmable Gate Arrays (FPGAs) becomes a critical issue for the practical use of FPGAs. In this article, we address the leakage issue of partially dynamically reconfigurable FPGA architectures with sleep transistors embedded into FPGA fabrics. In particular, we focus on eliminating leakage waste due to the delay between reconfiguration and execution time of a task. For partially dynamically reconfigurable FPGAs, the configuration prefetching technique is commonly used to hide runtime reconfiguration overhead. With prefetching, the configuration of a task is loaded into FPGAs as early as possible. Therefore, there is often a delay between reconfiguration and execution time of a task. In this period of time, the SRAM cells allocated to a task cannot be turned off even though they are not utilized. In this article, we propose a two-stage task scheduling methodology to reduce leakage waste due to the delay between reconfiguration and execution time of a task without sacrificing performance. In the first stage, a performance-driven task scheduler that targets at minimizing the schedule length is invoked to generate an initial placement. In the second stage, a postplacement leakage-aware task scheduling is applied to refine the initial placement such that leakage waste is minimized provided that the schedule length is not increased. To solve the postplacement leakage optimization problem, we propose two algorithms. The first one is an optimal algorithm based on Integer Linear Programming (ILP). The second algorithm is a heuristic approach that iteratively refines the placement to reduce leakage waste. Experimental results on real and synthetic designs show that the efficiency and effectiveness of the proposed postplacement leakage reduction techniques.

Off-chip data bus consumes a significant part of system power. Recent works use small caches (Value Cache) at each side of the off-chip data bus, and transmit cache indexes instead of data values to reduce bus switching activity. A larger VC has a higher VC hit rate, but it also incurs more switching activity on a VC hit. In this paper, we propose the hierarchical VC design concept that provides a good tradeoff between VC capacity and bus switching activity. Our experimental results show that the proposed hierarchical VC design reduces the off-chip data bus energy by 60.2%.