As AI inference becomes mainstream, research has begun to focus on improving the energy consumption of inference servers. Inference kernels commonly underutilize a GPU’s compute resources and waste power from idling components. To improve utilization and energy efficiency, multiple models can co-locate and share the GPU. However, typical GPU spatial partitioning techniques often experience significant overheads when reconfiguring spatial partitions, which can waste additional energy through repartitioning overheads or non-optimal partition configurations. In this paper, we present ECLIP, a framework to enable low-overhead energy-efficient kernel-wise resource partitioning between co-located inference kernels. ECLIP minimizes repartitioning overheads by pre-allocating pools of CU masked streams and assigns optimal CU assignments to groups of kernels through our resource allocation optimizer. Overall, ECLIP achieves an average of 13% improvement to throughput and 25% improvement to energy efficiency.

Scheduling real-time tasks that utilize GPUs with analyzable guarantees poses a significant challenge due to the intricate interaction between CPU and GPU resources, as well as the complex GPU hardware and software stack. While much research has been conducted in the real-time research community, several limitations persist, including the absence or limited availability of GPU-level preemption, extended blocking times, and/or the need for extensive modifications to program code. In this paper, we propose GCAPS, a GPU Context-Aware Preemptive Scheduling approach for real-time GPU tasks. Our approach exerts control over GPU context scheduling at the device driver level and enables preemption of GPU execution based on task priorities by simply adding one-line macros to GPU segment boundaries. In addition, we provide a comprehensive response time analysis of GPU-using tasks for both our proposed approach as well as the default Nvidia GPU driver scheduling that follows a work-conserving round-robin policy. Through empirical evaluations and case studies, we demonstrate the effectiveness of the proposed approaches in improving taskset schedulability and response time. The results highlight significant improvements over prior work as well as the default scheduling approach, with up to 40% higher schedulability, while also achieving predictable worst-case behavior on Nvidia Jetson embedded platforms.

The use of batteryless energy harvesting devices has been recognized as a promising solution for their low maintenance requirements and ability to work in harsh environments. However, these devices have to harvest energy from ambient energy sources and execute real-time sensing tasks periodically while satisfying data freshness constraints, which is especially challenging as the energy sources are often unreliable and intermittent. In this paper, we develop an energy-adaptive real-time sensing framework for batteryless devices. This framework includes a lightweight machine learning-based energy predictor that is capable of running on microcontroller devices and predicting the energy availability and intensity based on energy traces. Using this, the framework adapts the schedule of real-time tasks by effectively taking into account the predicted energy supply and the resulting age of information of each task, in order to achieve continuous sensing operations and satisfy given data freshness requirements. We discuss various design choices for adaptive scheduling and evaluate their performance in the context of batteryless devices. Experimental results show that the proposed adaptive real-time approach outperforms the recent methods based on static and reactive approaches, in both energy utilization and data freshness.

With the increasing demand for computational power, research on general-purpose graphics processing units (GPUs) has been active for various real-time systems spanning from autonomous vehicles to real-time clouds. While the use of GPUs can significantly benefit compute-intensive tasks with timing constraints, their high power consumption becomes an important problem given that it is not rare to see multiple GPUs in today's systems. In this paper, we present our study towards energy-efficient real-time scheduling in heterogeneous multi-GPU systems. We first make observations using a custom power monitoring setup that, in a multi-GPU system, conventional task allocation approaches for multiprocessors do not lead to energy efficiency and there is no clear winner. Then we propose a multi-GPU real-time scheduling framework, sBEET-mg, that builds upon prior work on single-GPU systems and makes offline and runtime scheduling decisions to execute a given job on the energy-optimal GPU while exploiting spatial multitasking on each GPU for better concurrency and real-time performance. We implemented the proposed framework on a real multi-GPU system and evaluated it with randomly-generated task sets of benchmark programs. We also experimentally simulated our method in a system containing more GPUs. Experimental results show that sBEET-mg reduces deadline misses by up to 23% and 18% compared to the conventional load distribution and load concentration methods, respectively, while simultaneously achieving lower energy consumption than them.

General-purpose graphics processing units (GPUs) made available on embedded platforms have gained much interest in real-time cyber-physical systems. Despite the fact that GPUs generally outperform CPUs on many compute-intensive tasks in a multitasking environment, high power consumption remains a challenging problem. In this paper, we first analyze the power and energy consumption of GPU kernels scheduled with spatial multitasking, which is found to be advantageous for schedulability in recent studies, and prove that its use, however, degrades energy efficiency even in the latest commercially available embedded GPUs like NVIDIA Jetson Xavier AGX. Then, based on our observations, we propose sBEET, a real-time energy-efficient GPU scheduling framework that makes scheduling decisions at runtime to optimize the energy consumption while utilizing spatial multitasking to improve real-time performance. We evaluate the performance of the proposed sBEET framework using well-known GPU benchmarks and randomly-generated timing parameters on real hardware. The results indicate that sBEET reduces deadline misses up to 13% when the system is overloaded, and also achieves 15% to 21% lower energy consumption when the tasksets are schedulable compared to the existing works.

With the rising demand for emerging DNN applications in safety-critical systems, much attention has been given to the reliability and trustworthiness of DNN inference output against malicious attacks. Although prior work has been conducted to improve the privacy of DNN inference by executing the entire DNN model inside Intel SGX enclaves, existing approaches pose severe performance challenges to achieve dependable and timely execution simultaneously. In this paper, we propose AegisDNN, a DNN inference framework to address this problem. AegisDNN leverages secure SGX enclaves for protecting only the critical part of real-time DNN tasks which are vulnerable to potential fault injection attacks. To choose the right set of layers for protection while ensuring the timeliness of task execution, AegisDNN includes a dynamic-programming based algorithm that finds a layer protection configuration for each task to meet the real-time and dependability requirements based on the layer-wise DNN time and SDC (Silent Data Corruption) profiling mechanism. AegisDNN also utilizes a machine-learning based SDC prediction method to significantly reduce the time for estimating SDC rates for all possible layer protection configurations. We implemented AegisDNN on Caffe, PyTorch, and Tensorflow with Eigen BLAS ported into SGX enclaves to comprehensively demonstrate the effectiveness of AegisDNN against state-of-the-art DNN fault-injection attacks. Experiment results indicate that AegisDNN could satisfy both dependability and real-time requirements simultaneously, when none of the other compared approaches could do so.

Intermittently powered devices (IPDs) have gained much interest in recent years. However, scheduling real-time tasks while supporting data consistency, timekeeping, and schedulability guarantees on these devices still remains a challenge. Many sensing tasks need long indivisible sensor reading operations, but most prior work has limited their focus to the forward progress of computation-only tasks. In this article, we propose a scheduling framework to execute real-time periodic tasks with atomic sensing operations. Our proposed method keeps track of time progress and ensures the periodic execution of sensing tasks while efficiently utilizing intermittent power sources. We provide schedulability analysis to determine if a taskset is schedulable under a given charging condition. As a proof-of-concept, we design a custom programmable RFID tag device, called R'tag, and demonstrate the effectiveness of our framework in a realistic sensing application. Evaluation results show that the proposed method satisfies the real-time task execution requirements on IPDs in terms of task scheduling, timekeeping, and periodic sensing while significantly outperforming prior work.

General-purpose graphics processing units (GPUs) made available on embedded platforms have gained much interest in real-time cyber-physical systems. Despite the fact that GPUs generally outperform CPUs on many compute-intensive tasks in a multitasking environment, higher power consumption remains a challenging problem. This paper presents our study on the energy consumption characteristics of an NVIDIA AGX Xavier GPU, the latest commercially available embedded hardware, under different concurrency levels and kernel scheduling orders. Our findings pave the way for designing an energy efficient scheduler for GPUs with real-time guarantees.