Autonomous navigation systems (ANS) consist of several software modules, such as sensing, perception, and planning to achieve traffic perception and fast decision making. These modules are required to process large amounts of data, such as images, in real-time. GPUs are commonly exploited on ANS to speed up data processing. As GPUs provides tremendous computation resources, it is common that one software module uses only partial GPU resources, leading to low GPU utilization and energy inefficiency. Traditionally, GPU sharing is a method to address this problem. However, GPU sharing is not supported on current embedded GPU platforms, which are widely used by ANS. Furthermore, GPU sharing challenges task Quality of Service (QoS) that requires task execution in a fixed latency. To achieve QoS requirement, we propose a progress bar scheduling policy to provide GPU tasks with QoS guarantee in GPU sharing environments. Based on this policy, a GPU sharing framework named TCUDA, is proposed to endow existing GPU tasks with QoS guarantee and improve GPU utilization. Finally, results show TCUDA reduces GPU task latency by 16.8% with QoS guarantee on an embedded GPU platform Xavier.

Autonomous navigation systems consist of multiple software modules, such as sensing, object detection, and planning, to achieve traffic perception and fast decision making. Such a system generates a large amount of data and requires data processing and communication in real-time. Although accelerators, such as GPUs, have been exploited to speed up data processing, communicating GPU messages between modules is still lacking support, leading to high communication latency and resource contention. For such a latency-sensitive and resource-limited autonomous navigation system, high performance and lightweight message communication are crucial and demanding. To obtain both high performance and low resource usages, we first propose a novel pub-centric memory pool and an on-the-fly offset conversion algorithm to avoid unnecessary data movement. Secondly, we combine these two techniques and propose an efficient message communication on a single GPU. Finally, we extend this approach to multi-GPU and design a framework that natively supports GPU message communication for Inter-Process Communication. With comprehensive evaluation, results show our approach is able to reduce communication latency by 53.7% for PointCloud and Image messages compared to the state-of-the-art approach. Moreover, in the real autonomous navigation scenario, our approach reduces the end-to-end latency by 29.2% and decreases resource usage up to 58.9%.

GPUs have been widely adopted to speedup various throughput-originated applications running on HPC platforms, where typically there are a number of tasks sharing GPUs to maximize GPU utilization. To facilitate GPU sharing, GPU vendors provide tools, allowing multiple processes concurrently to use GPUs. For example, Nvidia provides MPS (Multi-Process Service) managing all GPU processes to achieve high throughput by fully exploiting hardware resources. However, such tool leads to undesired single point of failure for all GPU processes, namely, one process’s exception makes other processes abnormal. In this work, we investigate the seriousness of this GPU process interferences caused by MPS, and propose an approach to address one of these interferences, which takes place during process quitting. By using signal handling and thread synchronization techniques in this approach, GPU processes are able to quit safely without interfering other GPU processes.

There has been a remarkable increase in the speed of AI development over the past few years. Artificial intelligence and deep learning techniques are blooming and expanding in all forms to every sector possible. With the emerging intelligent autonomous navigation systems, both memory allocation and data movement are becoming the main bottlenecks in inter-process communication procedures, especially in supporting various types of messages between multiple programming languages. To reduce significant memory allocation and data movement cost, we propose a novel memory-centric mechanism, which includes a virtual layer based architecture and a pre-record memory allocation algorithm. Furthermore, we implement a memory-centric communication framework named Z-framework based on the proposed mechanism to achieve high efficient IPC procedures in autonomous navigation systems. Experimental results show that Z-framework is able to gain up to 41% and 35% performance improvement compared with the approach used in ROS2, which is an industry standard and the state-of-the-art approach used in CyberRT, respectively.

Benefiting from the major breakthrough of AI technology and increasing application of AIoT (AI + IoT), the intelligent robot system achieves high developing speed in recent years. As a core component of the intelligent system, service discovery plays a crucial role in system reliability. Because of the high CPU usages in conventional service discovery, a light weight service discovery mechanism is required in such a resourcelimited robot system. To improve the efficiency of CPU usages, we propose a weak centralized mechanism and a socket based notification mechanism, to reduce the amount of event in service discovery. The evaluation results show that our proposed light weight service discovery mechanism can reduce 95% CPU usages in average, compared with the conventional service discovery used in ROS2, which is an industry standard in robot systems.

With the significant advances of AI technology, intelligent robotic systems have achieved remarkable development and profound effects. To enable massive data transmissionin an efficient and reliable way, both high performance andhigh reliability should be taken into account in system design. However, the conventional communication middleware used in the majority of autonomous robotic systems, is based on socked-based methods, which always lead to high latency. Moreover, some sophisticated communication middleware utilizes shared memory upon ring buffers for high performance without consideration of the reliability. To obtain both high performance and high reliability, we employ shared memory for performance improvement and propose a novel socket-based communication control algorithm to improve reliability during data transmission. Furthermore, based on the proposed algorithm, we implement a novel robotic communication middleware, named Robust-Z, combining both high performance and high reliability. Experimental results show that (1) Robust-Z is able to gain up to 41% and 5% performance improvement compared to ROS2 and Apollo CyberRT, respectively; (2) Robust-Z is able to provide crash safety and reduce 5.2% data missing rate compared with CyberRT.

Recently, various applications including data analytics and machine learning have been developed for geo-distributed cloud data centers. For those applications, the ways of mapping parallel processes to physical nodes (i.e., “process mapping”) could significantly impact the performance of the applications because of non-uniform communication cost in geo-distributed environments. What's more, the different data privacy requirements in geo-distributed data centers pose additional constraints on process mapping solutions. While process mapping has been widely studied in grid/cluster environments, few of the existing studies have considered the problem in geo-distributed cloud environment, which is a challenging task due to the multi-level data privacy constraints, heterogeneous network performance and process failures. In this paper, we introduce the special privacy requirements in geo-distributed data centers and formulate the geo-distributed process mapping problem as an optimization problem with multiple constraints. We develop a new method to efficiently find good process mapping solutions to the problem. Experimental results on real clouds (including Amazon EC2 and Windows Azure) and simulations demonstrate that our proposed approach can achieve significant performance improvement compared to the state-of-the-art algorithms.

Deep learning has revolutionized computer vision and other fields since its big bang in 2012. However, it is challenging to deploy Deep Neural Networks (DNNs) into real-world applications due to their high computational complexity. Binary Neural Networks (BNNs) dramatically reduce computational complexity by replacing most arithmetic operations with bitwise operations. Existing implementations of BNNs have been focusing on GPU or FPGA, and using the conventional image-to-column method that doesn't perform well for binary convolution due to low arithmetic intensity and unfriendly pattern for bitwise operations. We propose BitFlow, a gemm-operator-network three-level optimization framework for fully exploiting the computing power of BNNs on CPU. BitFlow features a new class of algorithm named PressedConv for efficient binary convolution using locality-aware layout and vector parallelism. We evaluate BitFlow with the VGG network. On a single core of Intel Xeon Phi, BitFlow obtains 1.8x speedup over unoptimized BNN implementations, and 11.5x speedup over counterpart full-precision DNNs. Over 64 cores, BitFlow enables BNNs to run 1.1x faster than counterpart full-precision DNNs on GPU (GTX 1080).

Recently, various applications including data analytics and machine learning have been developed for geo-distributed cloud data centers. For those applications, the ways to map parallel processes to physical nodes (i.e., "process mapping") could significantly impact the performance of the applications because of non-uniform communication cost in such geo-distributed environments. While process mapping has been widely studied in grid/cluster environments, few of the existing studies have considered the problem in geo-distributed cloud environments. In this paper, we propose a novel model to formulate the geo-distributed process mapping problem and develop a new method to efficiently find the near optimal solution. Our algorithm considers both the network communication performance of geo-distributed data centers as well as the communication matrix of the target application. Evaluation results with real experiments on Amazon EC2 and simulations demonstrate that our proposal achieves significant performance improvement (50% on average) compared to the state-of-the-art algorithms.

Network performance aware optimizations have long been a hot research topic to optimize distributed applications on traditional network environments. However, those optimization techniques rely on a few measurements on pair-wise network performance, and such direct use of network measurements is no longer valid on Infrastructure-as-a-service (IaaS) clouds. First, the direct calibration is ineffective. Network performance measurements may not represent the long-term performance (informally the stable component inside network performance) because of virtualization and network performance interference in the cloud. Second, the direct calibration is inefficient because the measurement overhead of all pair-wise link performance in a cluster becomes prohibitively high as the number of instances increases. To effectively and efficiently utilize existing network performance aware optimizations on IaaS clouds, we propose to reduce the measurement overhead and decouple the constant component from the dynamic network performance while minimizing the difference between the network performance and the constant component. For effectiveness, we use the constant component to guide the network performance aware optimizations. For efficiency, we exploit a non-negative matrix factorization (NMF) method to reduce the calibration overhead. Furthermore, we observe a tradeoff between effectiveness and efficiency, and develop an adaptive approach to capture this tradeoff. We demonstrate effectiveness and efficiency of our approach by adopting network performance aware optimizations on two kinds of basic applications, collective communications of MPI and generic topology mapping, and two real-world applications, namely N-body and conjugate gradient (CG). Our experiments on Amazon EC2 and simulations demonstrate significant calibration overhead reduction and performance improvement on guiding network performance aware optimizations, when comparing our approach to other state-of-the-art approaches.

In this paper, we propose monetary cost optimizations for MPI-based applications with deadline constraints on Amazon EC2. Particularly, we consider to utilize two kinds of Amazon EC2 instances (on-demand and spot instances). As a spot instance can fail at any time due to out-of-bid events, fault tolerant executions are necessary. Through detailed studies, we have found that two common fault tolerant mechanisms, i.e., checkpoints and replicated executions, are complementary for cost-effective MPI executions on spot instances. We formulate the optimization problem and propose a novel cost model to minimize the expected monetary cost. The experimental results with NPB benchmarks on Amazon EC2 demonstrate that 1) it is feasible to run MPI applications with performance constraints on spot instances, 2) our proposal achieves significant monetary cost reduction compared to the state-of-the-art algorithm and 3) it is necessary to adaptively choose checkpoint and replication techniques for cost-effective and reliable MPI executions on Amazon EC2.

This paper examines the performance of collective communication operations in message passing interfaces (MPI) in the cloud computing environment. The awareness of network topology has been a key factor in performance optimizations for existing MPI implementations. However, virtualization in the cloud environment not only hides the network topology information from the users, but also causes traffic interference and dynamics to network performance. Existing topology-aware optimizations are no longer feasible in the cloud environment. Therefore, we develop novel network performance aware algorithms for a series of collective communication operations including broadcast, reduce, gather and scatter. We further implement two common applications, N-body and conjugate gradient (CG). We have conducted our experiments with two complementary methods (on Amazon EC2 and simulations). Our experimental results show that the network performance awareness results in 25.4 and 28.3 percent performance improvement over MPICH2 on Amazon EC2 and on simulations, respectively. Evaluations on N-body and CG show 41.6 and 14.3 percent respectively on application performance improvement.

Network performance aware optimizations have long been an effective approach to optimizing distributed applications on traditional network environments. However, the assumptions of network topology or direct use of several measurements of pair-wise network performance for optimizations are no longer valid on IaaS clouds. Virtualization hides network topology from users, and direct use of network performance measurements may not represent long-term performance. To enable existing network performance aware optimizations on IaaS clouds, we propose to decouple constant component from dynamic network performance while minimizing the difference by a mathematical method called RPCA (Robust Principal Component Analysis). We use the constant component to guide network performance aware optimizations and demonstrate the efficiency of our approach by adopting network aware optimizations for collective communications of MPI and generic topology mapping as well as two real-world applications, N-body and conjugate gradient (CG). Our experiments on Amazon EC2 and simulations demonstrate significant performance improvement on guiding the optimizations.

Cloud computing enables users to perform distributed computing tasks on many virtual machines, without owning a physical cluster. Recently, various distributed computing tasks such as scientific applications are being moved from supercomputers and private clusters to public clouds. Message passing interface (MPI) is a key and common component in distributed computing tasks. The virtualized computing environment of the public cloud hides the network topology information from the users, and existing topology-aware optimizations for MPI are no longer feasible in the cloud environment. We propose a network performance aware MPI library named CMPI. CMPI embraces a new model for capturing the network performance among different virtual machines in the cloud. Based on the network performance model, we develop novel network performance aware algorithms for communication operations. This poster gives an overview of CMPI design, and presents some preliminary results on collective operations such as broadcast.We demonstrate the effectiveness of our network performance aware optimizations on Amazon EC2.