Despite ceaseless efforts, extremely large and complex optimization space makes even the state-of-the-art compilers fail in delivering the most performant setting that can fully utilize the underlying hardware. Although this inefficiency suggests opportunity for tuning, it has been challenging for prior tuning methods to consider the complex interactions between optimizations and maximize the tuning quality while handling local optima efficiently. To tackle this problem, we suggest an intelligent auto-tuning strategy, called SRTuner, which searches for the best optimization setting by exposing important optimization interactions and directly using them to focus on promising subspaces. To reveal high-impact inter-optimization relations, SRTuner proposes a multistage structure and a distribution-based estimation method that approximates the impact of an optimization effectively. Besides, to efficiently handle local optima, our technique defines optimization decisions as a series of multi-armed bandit problems to formulate the exploration-exploitation dilemma. SRTuner is evaluated with three representative compilers from various domains on different target hardware: GCC (traditional C/ C++ compiler) on CPU, TVM (domain-specific machine learning compiler) on GPU, and OpenCL compilers (kernel compiler for heterogeneous computing) on both CPU/GPU. Results show that SRTuner accelerates target executions by $1. 24 \times$, $2. 03 \times$ and $34. 4 \times$ compared to the highest level of optimization provided by each compiler and outperforms state-of-the-art works by $1. 04 \times - 1. 14 \times$. As a byproduct of our unique tuning strategy, SRTuner can offer synergistic optimizations for each workload, which allows it to in part identify why it outperformed current compilers. With this information, we are able to find important optimizations that each compiler misused and demonstrate how this information can benefit future tuning strategies.

DNNs are deployed in mission critical applications such as object detection in autonomous vehicles (AVs). Ensuring the reliability of detections made by these models plays an essential role in robustness and practicality of these systems. By analyzing traditional object detectors, we observe a significant number of objects being missed by the detector, which is a problem for safe deployment of AVs. Traditionally, sensor fusion wherein detections from multiple sensors (camera, LiDAR, etc.) are combined to create a robust detection system is utilized, but suffers from high cost and computing overheads. To attack this seemingly necessary tradeoff, we propose SoftFusion in this paper. SoftFusion introduces the concept of intra-sensor fusion wherein a diversity of inputs is synthesized during runtime and can be efficiently evaluated by the original object detector. The predictions are then intelligently combined to realize robust detections over the baseline detector but at a fraction of the cost of traditional fusion. SoftFusion is evaluated across 7 different benchmarks over 2 different datasets, and is shown to achieve 3.45% gain in average precision with less than 23% latency overhead. As context for these results, prior research demonstrates that hard fusion of camera and LiDAR provides on average 4.02% gain in average precision in similar tasks with an overhead of approximately up to 2× computation.

This paper proposes SIEVE, Speculative Inference on the Edge with Versatile Exportation, which dynamically distributes CNN computation between the cloud and edge device based on the input data and environmental conditions to maximize efficiency and performance. A speculative CNN is created through aggressive precision reduction techniques to run most of the inferences on the edge device, while the original CNN is run on the cloud server. A runtime system directs each input to either the edge or cloud and decides whether to accept speculative inferences made on the edge or invoke recovery by replaying the inference on the cloud. Compared to the cloud-only approach, SIEVE reduces energy consumption by an average of 91%, 57% and 26% and increases performance by an average of 12.3×, 2.8× and 2.0× for 3G, LTE and WiFi connections without accuracy loss across a range of nine CNNs.

Deep neural networks (DNNs) are now starting to emerge in mission critical applications including autonomous vehicles and precision medicine. An important question is the dependability of DNNs and trustworthiness of their predictions. Considering the irreparable damage that can be caused by mispredictions, assessment of their potential misbehavior is necessary for safe deployment. In this paper, we first show the deficiency of current confidence-based methods as reliability measurement, and assess the effectiveness of traditional architecture reliability methods such as modular redundancy (MR). Then, we propose PolygraphMR and show that the combination of input preprocessing, smarter decision policies, and inclusion of prediction confidences can substantially improve the effectiveness of MR for DNNs. Next, we show how to prohibit explosive growth in the cost of MR by the help of reduced-precision designs and staged activations. Across six benchmarks, PolygraphMR detects an average of 33.5% of the baseline mispredictions with less than 2x overhead.

To facilitate the efficient execution of convolutional neural networks (CNNs) on cloud servers, this paper proposes Yin Yang (YY), an input-driven synergistic deep learning system, which dynamically distributes CNN computation between a complex (Yang) and a simple (Yin) CNN. YY runs most of the inferences on Yin, while Yang is invoked only when Yin has low confidence. On average, compared to the traditional CNN as a service approach, YY improves datacenter throughput by 1.8× and reduces inference latency by 31% on an NVIDIA TITAN X GPU without any accuracy loss across 21 CNNs.