As advances in energy-efficiency become the primary limiter to increases in power-constrained supercomputing and machine learning performance, it is imperative developers, architects, and practitioners understand how modern GPUs consume energy when running HPC and ML applications. Rather than opaque coarse-grained metrics, in this paper, we develop an extensible, microbenchmark-parameterized energy model capable of attributing application energy not only by functional unit (FPU, tensor core, integer ALU) and memory level (L1, L2, HBM), but can also differentiate control energy from datapath energy. We examine trends in energy per operation among four generations of GPUs and validate our results using supercomputing and ML/AI procurement workloads. Our insights and extrapolations can be used to drive the future of CMOS and memory technologies, computer architecture research, algorithmic innovation, optimizations for power-constrained and mobile environments, and data center operations.

The end-to-end performance of deep learning model inference is often limited by excess data movement on GPUs. To reduce data movement, existing deep learning frameworks apply graph-level optimizations such as operator fusion to exploit data reuse across operators in deep learning graphs. Such optimizations are limited and cannot optimize arbitrary chains of compute- and time-intensive operations, including convolutions. To address these limitations, this paper presents BrickDL, a deep learning inference library that implements merged execution of a sequence of layers as an orthogonal approach to existing graph-level optimizations. BrickDL additionally employs fine-grain blocking using a brick data layout that further optimizes data locality on GPUs. We implement merged execution with the abstraction of bricks using two approaches – padded bricks and memoized bricks, and develop a performance model to choose between them using static analysis. Merged execution with bricks demonstrates performance gains on well-known deep learning models as compared to PyTorch JIT, TensorFlow XLA, and cuDNN baselines on NVIDIA A100 GPU. We also characterize the performance of the proposed optimizations with microbenchmarks and gain insights into their applicability and tradeoffs.