We introduce VisualCent, a unified human pose and instance segmentation framework to address generalizability and scalability limitations to multi-person visual human analysis. VisualCent leverages centroid-based bottomup keypoint detection paradigm and uses Keypoint Heatmap incorporating Disk Representation and KeyCentroid to identify the optimal keypoint coordinates. For the unified segmentation task, an explicit keypoint is defined as a dynamic centroid called MaskCentroid to swiftly cluster pixels to specific human instance during rapid changes in human body movement or significantly occluded environment. Experimental results on COCO and OCHuman datasets demonstrate VisualCent’s accuracy and real-time performance advantages, outperforming existing methods in mAP scores and execution frame rate per second. The implementation is available on the project page†.†https://sites.google.com/view/niazahmad/projects/visualcent

The dynamic movement of the human body presents a fundamental challenge for human pose estimation and body segmentation. State-of-the-art approaches primarily rely on combining keypoint heatmaps with segmentation masks, but often struggle in scenarios involving overlapping joints during pose estimation or rapidly changing poses for instance-level segmentation. To address these limitations, we leverage Keypoints as Dynamic Centroid (KDC), a new centroid-based representation for unified human pose estimation and instance-level segmentation. KDC adopts a bottom-up paradigm to generate keypoint heatmaps for easily distinguishable and complex keypoints, and improves keypoint detection and confidence scores by introducing KeyCentroids using a keypoint disk. It leverages high-confidence keypoints as dynamic centroids in the embedding space to generate MaskCentroids, allowing for the swift clustering of pixels to specific human instances during rapid changes in human body movements in a live environment. Our experimental evaluations focus on crowded and occluded cases using the CrowdPose, OCHuman, and COCO benchmarks, demonstrating KDC’s effectiveness and generalizability in challenging scenarios in terms of both accuracy and runtime performance. Our implementation is available at https://sites.google.com/view/niazahmad/projects/kdc.

Drones suffer unpredictable battery performance that may cause crashes or fatal accidents during aerial missions. Users have reported that their drones unexpectedly shutoff even when they show more than 10% remaining battery capacity. After examining both the drones and battery sides, we discovered that the causes of these unexpected shutoffs to be unbalanced cell thermal degradation caused by thermal coupling between the drones and their battery cells. Drone heat dissipation differently affects battery cells, causes a different cell voltage drop, and results in unpredictable supply voltage and unexpected shutoffs. This paper describes the design and implementation of a thermal and battery-aware power management framework designed specifically for drones. Our framework transparently profiles each battery cell's resistance to capture the cell-level thermal degradation using novel discharging pulses, and estimates an accurate cell capacity, state-of-charge, and state-of-power in real-time to prevent unexpected shutoffs. We have implemented and deployed our framework on commercial drones without additional hardware or system modifications. We have evaluated its effectiveness across different drones and three different batteries demonstrating our framework generates accurate cell state-of-charge and prevents unexpected shutoffs.

The increasing complexity and memory demands of Deep Neural Networks (DNNs) for real-time systems pose new significant challenges, one of which is the GPU memory capacity bottleneck, where the limited physical memory inside GPUs impedes the deployment of sophisticated DNN models. This paper presents, to the best of our knowledge, the first study of addressing the GPU memory bottleneck issues, while simultaneously ensuring the timely inference of multiple DNN tasks. We propose RT-Swap, a real-time memory management framework, that enables transparent and efficient swap scheduling of memory objects, employing the relatively larger CPU memory to extend the available GPU memory capacity, without compromising timing guarantees. We have implemented RT-Swap on top of representative machine-learning frameworks, demonstrating its effectiveness in making significantly more DNN task sets schedulable at least 72% over existing approaches even when the task sets demand up to 96.2% more memory than the GPU's physical capacity.

Despite the advances in multi-person pose estimation, state-of-the-art techniques only deliver the human pose structure.Yet, they do not leverage the keypoints of human pose to deliver whole-body shape information for human instance segmentation. This paper presents PosePlusSeg, a joint model designed for both human pose estimation and instance segmentation. For pose estimation, PosePlusSeg first takes a bottom-up approach to detect the soft and hard keypoints of individuals by producing a strong keypoint heat map, then improves the keypoint detection confidence score by producing a body heat map. For instance segmentation, PosePlusSeg generates a mask offset where keypoint is defined as a centroid for the pixels in the embedding space, enabling instance-level segmentation for the human class. Finally, we propose a new pose and instance segmentation algorithm that enables PosePlusSeg to determine the joint structure of the human pose and instance segmentation. Experiments using the COCO challenging dataset demonstrate that PosePlusSeg copes better with challenging scenarios, like occlusions, en-tangled limbs, and overlapped people. PosePlusSeg outperforms state-of-the-art detection-based approaches achieving a 0.728 mAP for human pose estimation and a 0.445 mAP for instance segmentation. Code has been made available at: https://github.com/RaiseLab/PosePlusSeg.

Users have reported that their drones unexpectedly shutoff even when they show more than 10% remaining battery capacity. We discovered that the causes of these unexpected shutoffs to be significant thermal degradation of a cell caused by thermal coupling between the drones and their battery cells. This causes a large voltage drop for the cell affected by the drone heat dissipation, which leads to low supply voltage and unexpected shutoffs. This paper describes the design and implementation of a thermal and battery-aware power management framework designed specifically for drones. Our framework provides an accurate state-of-charge and state-of-power estimation for individual battery cells by accounting for their different thermal degradation. We have implemented our framework on commodity drones without additional hardware or system modification. We have evaluated its effectiveness using three different batteries demonstrating our framework generates accurate state-of-charge and prevents unexpected shutoffs.

Multi-person pose estimation and instance segmentation suffer large performance loss when images are with an increasing number of people and their uncontrolled complex appearance. Yet, existing models cannot efficiently leverage unbalanced training images, i.e., few of them are with multi-person, and most are with single-person, making them ineffective for challenging multi-person scenarios. To tackle multi-person cases with a limited portion of them, we propose MultiPoseSeg, a data preparation and feedback knowledge transfer system designed for multi-person pose estimation and instance segmentation. First, MultiPoseSeg categorically performs random data reduction to reduce the single-person bias from the train dataset. Second, MultiPoseSeg employs the knowledge transfer from ancestor models to converge the model learning with a limited amount of data and time. This way, our model learns and train on human pose and instance segmentation to advance the training and testing accuracy. Finally, MultiPoseSeg proposes keypoint maps to identify the keypoint coordinates for soft and hard keypoints and segmentation maps to assign centroid to each human instance, which helps to cluster the pixels to a particular instance. We have evaluated MultiPoseSeg using COCO and OCHuman challenging datasets and demonstrated MultiPoseSeg outperforms state-of-the-art bottom-up models in terms of both accuracy and runtime performance, achieving 0.728 mAP for pose and 0.445 mAP for segmentation on COCO dataset. All the unbiased data and code has been made available at: https://github.com/RaiseLab/MultiPoseSeg

As real-time object detection systems, such as autonomous cars, need to process input images acquired from multiple cameras, they face significant challenges in delivering accurate and timely inferences often based on machine learning (ML). To meet these challenges, we want to provide different levels of object detection accuracy and timeliness to different portions within each input image with different criticality levels. Specifically, we develop DNN-SAM, a dynamic Split-And-Merge Deep Neural Network (DNN) execution and scheduling framework, that enables seamless split-and-merge DNN execution for unmodified DNN models. Instead of processing an entire input image once in a full DNN model, DNN-SAM first splits a DNN inference task into two smaller sub-tasks-a mandatory sub-task dedicated for a safety-critical (cropped) portion of each image and an optional sub-task for processing a down-scaled image–then executes them independently, and finally merges their results into a complete inference. To achieve DNN-SAM’s timely and accurate detection of objects in each image, we also develop two scheduling algorithms that prioritize sub-tasks according to their criticality levels and adaptively adjust the scale of the input image to meet the timing constraints while minimizing the response time of mandatory sub-tasks or maximizing the accuracy of optional sub-tasks. We have implemented and evaluated DNN-SAM on a representative ML framework. Our evaluation shows DNN-SAM to improve detection accuracy in the safety-critical region by $2.0-3.7\times$ and lower average inference latency by $4.8-9.7\times$ over existing approaches without violating any timing constraints.

Modern embedded systems face challenges in managing on-chip temperature as they are increasingly realized in powerful system-on-chips. This paper presents thermal-aware design and management of embedded systems by tightly coupling two mechanisms, thermal-aware utilization bound and real-time dynamic thermal management. The former provides the processor utilization upper-bound to meet the chip temperature constraint that depends not only on the system configurations and workloads but also chip cooling capacity and environment. The latter adaptively optimizes rates of individual task executions subject to the thermal-aware utilization bound. Our experiments on an automotive controller demonstrate the thermal-aware utilization bound and improved system utilization by 18.2 % compared with existing approaches.

Many users have reported that their smartphones shut off unexpectedly, even when they show >30% remaining battery capacity. After examining the problem from both the user and phone sides, we discovered the cause of these unexpected shutoffs to be a large and dynamic internal voltage drop of the phone battery, which is, in turn, caused by the dynamics of both battery's internal resistance and the phone's discharge current. To fix these unexpected shutoffs, we design a novel Battery-aware Power Management (BPM) middleware that accounts for these dual-dynamics in phone operation. Specifically, BPM profiles the battery's internal resistance --- which varies with battery state-of-charge (SoC), temperature, and aging --- using a novel duty-cycled charging method. BPM then regulates, at run-time, the phone's discharge current based on the constructed battery profile. We have implemented and evaluated BPM on 4 commodity smartphones from different OEMs with the latest battery firmware, demonstrating that BPM prevents unexpected phone shutoffs and extends their operation time by 1.16--2.03X. Our user study, which includes 121 mobile phone users, also corroborates BPM's usefulness/attractiveness.

Modern electric systems, such as electric vehicles, mobile robots, nano satellites, and drones, require to support various power-demand operations for user applications and system maintenance. This, in turn, calls for advanced power management that jointly considers power demand by the operations and power supply from various sources, such as batteries, solar panels, and supercapacitors. In this article, we develop a power scheduling framework for a reliable energy storage system with multiple power-supply sources and multiple power-demand operations. Specifically, we develop offline power-supply guarantee analysis and online power management. The former provides an offline power-supply guarantee such that every power-demand operation completes its execution in time while the sum of power required by individual operations does not exceed the total power supplied by the entire energy storage system at any time; to this end, we develop a plain power-supply analysis as well as its improved version using real-time scheduling techniques. On the other hand, the latter efficiently utilizes the surplus power available at runtime for improving system performance; we propose two approaches, depending on whether future scheduling information of power-demanding tasks is available or not. For evaluation, we perform simulations to evaluate both the plain and improved analyses for offline power guarantee under various synthetic power-demand operations. In addition, we have built a simulation model and demonstrated that the proposed framework with the offline analysis and online management not only guarantees the required power-supply, but also enhances system performance by up to 56.49 percent.

As modern embedded systems like cars need high-power integrated CPUs--GPU SoCs for various real-time applications such as lane or pedestrian detection, they face greater thermal problems than before, which may, in turn, incur higher failure rate and cooling cost. We demonstrate, via experimentation on a representative CPUs--GPU platform, the importance of accounting for two distinct thermal characteristics—the platform’s temperature imbalance and different power dissipations of different tasks —in real-time scheduling to avoid any burst of power dissipations while guaranteeing all timing constraints. To achieve this goal, we propose a new Real-Time Thermal-Aware Scheduling (RT-TAS) framework. We first capture different CPU cores’ temperatures caused by different GPU power dissipations (i.e., CPUs--GPU thermal coupling ) with core-specific thermal coupling coefficients. We then develop thermally-balanced task-to-core assignment and CPUs--GPU co-scheduling . The former addresses the platform’s temperature imbalance by efficiently distributing the thermal load across cores while preserving scheduling feasibility. Building on the thermally-balanced task assignment, the latter cooperatively schedules CPU and GPU computations to avoid simultaneous peak power dissipations on both CPUs and GPU, thus mitigating excessive temperature rises while meeting task deadlines. We have implemented and evaluated RT-TAS on an automotive embedded platform to demonstrate its effectiveness in reducing the maximum temperature by 6−12.2° C over existing approaches without violating any task deadline.

With an increasing demand for complex and powerful system-on-chips, modern real-time automotive systems face significant challenges in managing on-chip-temperature. We demonstrate, via real experiments, the importance of accounting for dynamic ambient temperature and task-level power dissipation in resource management so as to meet both thermal and timing constraints. To address this problem, we propose RT-TRM, a real-time thermal-aware resource management framework. We first introduce a task-level dynamic power model that can capture different power dissipations with a simple task-level parameter called the activity factor. We then develop two new mechanisms, adaptive parameter assignment and online idle-time scheduling. The former adjusts voltage/frequency levels and task periods according to the varying ambient temperature while preserving feasibility. The latter generates a schedule by allocating idle times efficiently without missing any task/job deadline. By tightly integrating the solutions of these two mechanisms, we can guarantee both thermal and timing constraints in the presence of dynamic ambient temperature variations. We have implemented RT-TRM on an automotive microcontroller to demonstrate its effectiveness, achieving better resource utilization by 18.2% over other runtime approaches while meeting both thermal and timing constraints.

Mobile apps suffer large performance degradation when the underlying processors are throttled to cool down the devices. Fans or heat sinks are not a viable option for mobile devices, thus calling for a new portable cooling solution. Thermoelectric coolers are scalable and controllable cooling devices that can be embedded into mobile devices on the chip surface. This paper presents a thermoelectric cooling solution that enables efficient processor thermal management in mobile devices. Our goal is to minimize performance loss from thermal throttling by efficiently using thermoelectric cooling. Since mobile devices experience large variations in workloads and ambient temperature, our solution adaptively controls cooling power at runtime. Our evaluation on a smartphone using mobile benchmarks demonstrated that the performance loss from the maximum speed is only 1.8% with the TEC compared to 19.2% without the TEC.

Since modern electric systems require to support various power-demand operations for user applications and system maintenance, they need advanced power management that jointly considers power demand by the operations and power supply from various sources, such as batteries, solar panels, and supercapacitors. In this paper, we develop a power scheduling framework for a reliable energy storage system with multiple power-supply sources and multiple power-demand operations. First, we provide an offline power-supply guarantee such that every power-demand operation completes its execution in time while the sum of power required by individual operations does not exceed the total power supplied by the entire energy storage system at any time. We find similarities between this and a real-time scheduling problem, and make a power-supply guarantee using real-time scheduling techniques. Second, we propose online power management that efficiently utilizes the surplus power (available at run-time) for system performance improvement. Our experimental results on a prototype demonstrate that the proposed framework not only guarantees the required power supply, but also enhances system performance by up to 33.1%.