With growing demands on the acceleration of machine learning (ML) computation, processors for edge systems have been integrating heterogeneous devices such as GPUs and NPUs (Neural Processing Units) for ML computing. However, as multiple ML models need to be processed simultaneously even in edge systems, ML inference schedulers for such heterogeneous devices must not only consider the efficiency of devices for different ML models, but also consider the interference among them carefully. Based on our analysis on the behaviors of inter-device interference, the study first builds an interference prediction scheme using a multilayer perceptron model. The interference prediction model is trained with the data from randomly generated ML models, and thus it can be prepared without any prior knowledge of actual target ML workloads. Using the highly accurate prediction model, this study proposes a goal-independent scheduling framework, which allows any scheduling objective set by users. The scheduling framework uses a sampled simulation method to support such flexible scheduling goals with a minimized latency. Our experimental results on a commercial edge system show that our framework backed by the interference prediction model can effectively improve performance for diverse goals.

Trojan attacks on deep neural networks (DNNs) exploit abackdoorembedded in a DNN model that can hijack any input with an attacker’s chosen signature trigger. Emerging defence mechanisms are mainly designed and validated on vision domain tasks (e.g., image classification) on 2D Convolutional Neural Network (CNN) model architectures; a defence mechanism that is general across vision, text, and audio domain tasks is demanded. This work designs and evaluates a run-time Trojan detection method exploitingSTRongIntentionalPerturbation of inputs that is a multi-domain input-agnostic Trojan detection defence acrossVision,Text andAudio domains—thus termed as STRIP-ViTA. Specifically, STRIP-ViTA is demonstratively independent of not only task domain but also model architectures. Most importantly, unlike other detection mechanisms, it requires neither machine learning expertise nor expensive computational resource, which are the reason behind DNN model outsourcing scenario—one main attack surface of Trojan attack. We have extensively evaluated the performance of STRIP-ViTA over: i) CIFAR10 and GTSRB datasets using 2D CNNs for vision tasks; ii) IMDB and consumer complaint datasets using both LSTM and 1D CNNs for text tasks; and iii) speech command dataset using both 1D CNNs and 2D CNNs for audio tasks. Experimental results based on more than 30 tested Trojaned models (including publicly Trojaned model) corroborate that STRIP-ViTA performs well across all nine architectures and five datasets. Overall, STRIP-ViTA can effectively detect trigger inputs with small false acceptance rate (FAR) with an acceptable preset false rejection rate (FRR). In particular, for vision tasks, we can always achieve a 0 percent FRR and FAR given strong attack success rate always preferred by the attacker. By setting FRR to be 3 percent, average FAR of 1.1 and 3.55 percent are achieved for text and audio tasks, respectively. Moreover, we have evaluated STRIP-ViTA against a number of advanced backdoor attacks and compare its effectiveness with other recent state-of-the-arts.

Hardware-based trusted execution has opened a promising new opportunity for enabling secure cloud computing. Nevertheless, the current trusted execution environments are limited to the traditional CPU-based computation, while more and more critical workloads such as machine learning require to use GPUs. For secure GPU computing, recent studies proposed to isolate critical GPU operations from the operating system by providing trusted execution environments for GPUs. However, a missing component in the prior GPU trusted execution studies is a hardware-based memory protection technique optimized for GPUs. Start-of-art memory protection techniques for CPUs use counter-mode encryption, where per-block counters are used as timestamps for generating one-time pads. The integrity of counters is protected by a counter tree. This study investigates the hardware memory encryption for GPUs and identifies counter cache misses as one of the key performance bottlenecks. To mitigate the overheads of counter cache misses, this paper proposes a novel technique, called common counters, for efficient counter-mode encryption and counter integrity protection. The proposed technique exploits unique characteristics of common GPU applications. In GPU applications, a large portion of application memory is written only once by the initial data transfer from the CPU memory, or the number of writes by the GPU applications to major data structures tends to be uniform. Exploiting the uniform write property, this study proposes to use a common counter representation to complement the existing per-block encryption counters. With the common counters, the proposed technique can almost eliminate the performance overheads caused by counter cache misses, reducing the degradation to 2.9%.

With the proliferation of applications with machine learning (ML), the importance of edge platforms has been growing to process streaming sensor, data locally without resorting to remote servers. Such edge platforms are commonly equipped with heterogeneous computing processors such as GPU, DSP, and other accelerators, but their computational and energy budget are severely constrained compared to the data center servers. However, as an edge platform must perform the processing of multiple machine learning models concurrently for multimodal sensor data, its scheduling problem poses a new challenge to map heterogeneous machine learning computation to heterogeneous computing processors. Furthermore, processing of each input must provide a certain level of bounded response latency, making the scheduling decision critical for the edge platform. This article proposes a set of new heterogeneity-aware ML inference scheduling policies for edge platforms. Based on the regularity of computation in common ML tasks, the scheduler uses the pre-profiled behavior of each ML model and routes requests to the most appropriate processors. It also aims to satisfy the service-level objective (SLO) requirement while reducing the energy consumption for each request. For such SLO supports, the challenge of ML computation on GPUs and DSP is its inflexible preemption capability. To avoid the delay caused by a long task, the proposed scheduler decomposes a large ML task to sub-tasks by its layer in the DNN model.

Federated learning (FL) and split neural networks (SplitNN) are state-of-art distributed machine learning techniques to enable machine learning without directly accessing raw data on clients or end devices. In theory, such distributed machine learning techniques have great potential in distributed applications, in which data are typically generated and collected at the client-side while the collected data should be processed by the application deployed at the server-side. However, there is still a significant gap in evaluating the performance of those techniques concerning their practicality in the Internet of Things (IoT)-enabled distributed systems constituted by resource-constrained devices. This work is the first attempt to provide empirical comparisons of FL and SplitNN in real-world IoT settings in terms of learning performance and device implementation overhead. We consider a variety of datasets, different model architectures, multiple clients, and various performance metrics. For the learning performance (i.e., model accuracy and convergence time), we empirically evaluate both FL and SplitNN under different types of data distributions such as imbalanced and non-independent and identically distributed (non-IID) data. We show that the learning performance of SplitNN is better than FL under an imbalanced data distribution but worse than FL under an extreme non-IID data distribution. For implementation overhead, we mount both FL and SplitNN on Raspberry Pi devices and comprehensively evaluate their overhead, including training time, communication overhead, power consumption, and memory usage. Our key observations are that under the IoT scenario where the communication traffic is the primary concern, FL appears to perform better over SplitNN because FL has a significantly lower communication overhead compared with SplitNN. However, our experimental results also demonstrate that neither FL or SplitNN can be applied to a heavy model, e.g., with several million parameters, on resource-constrained IoT devices because its training cost would be too expensive for such devices. Source code is released and available: https://github.com/Minki-Kim95/Federated-Learning-and-Split-Learning-with-raspberry-pi.