Tile-based streaming techniques have been widely used to save bandwidth in$360^{\circ }$video streaming. However, it is a challenge to determine the right tile size which directly affects the bandwidth usage. Moreover, downloading and processing many small tiles consume a large amount of energy on mobile devices. To solve this problem, we propose to encode the video by taking into account the viewing popularity, where the popularly viewed areas are encoded as macrotiles. We propose techniques for identifying and building macrotiles, and adjusting their sizes to take into account practical issues such as head movement randomness. In some cases, the user's viewing area may not be covered by the constructed macrotiles, and then the conventional tiling scheme is used. To support macrotile based$360^{\circ }$video streaming, the client selects the right tiles (a macrotile or a set of conventional tiles) with the right quality level to maximize the QoE under bandwidth constraint. We formulate this problem as an optimization problem which is NP-hard, and then propose a heuristic algorithm to solve it. Through extensive evaluations based on real head movement traces, we demonstrate that the proposed algorithm can significantly improve QoE, save bandwidth usage, and reduce energy consumption.

As Deep Neural Networks (DNNs) have been successfully applied to various fields, there is a tremendous demand for running DNNs on mobile devices. Although mobile GPU can be leveraged to improve performance, it consumes a large amount of energy. After a short period of time, the mobile device may become overheated and the processors are forced to reduce the clock speed, significantly reducing the processing speed. A different approach to support DNNs on mobile device is to leverage the Neural Processing Units (NPUs). Compared to GPU, NPU is much faster and more energy efficient, but with lower accuracy due to the use of low precision floating-point numbers. We propose to combine these two approaches to improve the performance of running DNNs on mobile devices by studying the thermal-aware scheduling problem, where the goal is to achieve a better tradeoff between processing time and accuracy while ensuring that the mobile device is not overheated. To solve the problem, we propose a heuristic-based scheduling algorithm to determine when to run DNNs on GPU and when to run DNNs on NPU based on the current states of the mobile device. The heuristic-based algorithm makes scheduling decisions greedily and ignores their future impacts. Thus, we propose a deep reinforcement learning based scheduling algorithm to further improve performance. Extensive evaluation results show that the proposed algorithms can significantly improve the performance of running DNNs on mobile devices while avoiding overheating.

Many mobile applications have been developed to apply deep learning for video analytics. Although these advanced deep learning models can provide us with better results, they also suffer from the high computational overhead which means longer delay and more energy consumption when running on mobile devices. To address this issue, we propose a framework called FastVA, which supports deep learning video analytics through edge processing and Neural Processing Unit (NPU) in mobile. The major challenge is to determine when to offload the computation and when to use NPU. Based on the processing time and accuracy requirement of the mobile application, we study three problems: \textit{Max-Accuracy} where the goal is to maximize the accuracy under some time constraints, \textit{Max-Utility} where the goal is to maximize the utility which is a weighted function of processing time and accuracy, and \textit{Min-Energy} where the goal is to minimize the energy under some time and accuracy constraints. We formulate them as integer programming problems and propose heuristics based solutions. We have implemented FastVA on smartphones and demonstrated its effectiveness through extensive evaluations.

The ubiquity of mobile devices allows mobile users to participate in crowdsourcing anywhere, anytime. One potential application is to crowdsource photos/videos on demand to search for interested targets. Crowdsourced photos/videos have much better coverage compared to surveillance cameras, and thus help improve the effectiveness of target search. However, broadcasting the crowdsourcing task to all mobile users can significantly increase the cost in terms of resource and incentive budget. To reduce cost, the crowdsourcing server selects a subset of participating workers, and there are many challenges on worker selection. For example, due to occlusions in the photo/video scene, each worker only covers part of the area with certain probability. Due to the non-deterministic nature of this problem, we study two kinds of optimization problems: max-coverage which maximizes the probability of finding the target given a cost, and min-selection which minimizes the number of workers given the required probability of finding the target. Considering that workers may report exact locations or coarse-grained locations, we formalize four probability-based optimization problems for worker selection, and develop optimal or efficient approximation algorithms to solve them. The effectiveness of the proposed algorithms is evaluated and validated via extensive trace-driven simulations and a real-world demo.

Deep Neural Network (DNN) is becoming adopted for video analytics on mobile devices. To reduce the delay of running DNNs, many mobile devices are equipped with Neural Processing Units (NPU). However, due to the resource limitations of NPU, these DNNs have to be compressed to increase the processing speed at the cost of accuracy. To address the low accuracy problem, we propose a Confidence Based Offloading (CBO) framework for deep learning video analytics. The major challenge is to determine when to return the NPU classification result based on the confidence level of running the DNN, and when to offload the video frames to the server for further processing to increase the accuracy. We first identify the problem of using existing confidence scores to make offloading decisions, and propose confidence score calibration techniques to improve the performance. Then, we formulate the CBO problem where the goal is to maximize accuracy under some time constraint, and propose an adaptive solution that determines which frames to offload at what resolution based on the confidence score and the network condition. Through real implementations and extensive evaluations, we demonstrate that the proposed solution can significantly outperform other approaches.

High quality video streaming for mobile devices implies high energy consumption due to the transmitted data and the variation of wireless signals. As an example, transmissions in mobile scenarios (e.g., inside a moving bus) consumes more energy for devices than when accessing from a static environment (e.g., at home). The QoE for the user does not substantially increase when watching high bitrate videos in a vibrating environment (i.e., a moving vehicle), as the context, in this case vehicle’s vibration, affects the perceived QoE. To address this problem, we propose to save energy by considering the context (environment) of video streaming. To model the impact of context, we exploit the embedded accelerometer in smartphones to record the vibration level during video streaming. Based on quality assessment experiments, we collect traces and model the impact of video bitrate and vibration level on QoE, and model the impact of video bitrate and signal strength on power consumption. Based on the QoE model and the power model, we formulate the context-aware and energy-aware video streaming problem as an optimization problem. We present an optimal algorithm which can maximize QoE and minimize energy. Since the optimal algorithm requires perfect knowledge of future tasks, we propose an online bitrate selection algorithm. To further improve the performance of the online algorithm, we propose a crowdsourcing based bitrate selection algorithm. Through real measurements and trace-driven simulations, we demonstrate that the proposed algorithms can significantly outperform existing approaches when considering both energy and QoE.

Tile-based streaming techniques have been widely used to save bandwidth in 360° video streaming. However, it is a challenge to determine the right tile size which directly affects the bandwidth usage. To address this problem, we propose to encode the video by considering the viewing popularity, where the popularly viewed areas are encoded as macrotiles to save bandwidth. We propose techniques to identify and build macrotiles, and adjust their sizes considering practical issues such as head movement randomness. In some cases, a user's viewing area may not be covered by the constructed macrotiles, and then the conventional tiling scheme is used. To support popularity-aware 360° video streaming, the client selects the right tiles (a macrotile or a set of conventional tiles) with the right quality level to maximize the QoE under bandwidth constraint. We formulate this problem as an optimization problem which is NP-hard, and then propose a heuristic algorithm to solve it. Through extensive evaluations based on real traces, we demonstrate that the proposed algorithm can significantly improve the QoE and save the bandwidth usage.

Many Deep Neural Network (DNN) based applications have been developed and run on mobile devices. Although these advanced DNN models can provide better results, they also suffer from high computational overhead which means long delay and more energy consumption when running on mobile devices. To address these problems, many companies have developed dedicated Neural Processing Units (NPUs) for mobile devices, which can process AI features. Compared to CPU, NPU can run DNN models much faster, but with lower accuracy. To address this issue, we leverage model partition techniques to improve the performance of DNN models on mobile devices with NPU. The challenge is to determine which part of the DNN model should be run on CPU and which part to be run on NPU. Based on the delay and the accuracy requirements of the applications, we study two problems: Max-Accuracy where the goal is to maximize the accuracy under some time constraint, and Min-Time where the goal is to minimize the processing time while ensuring the accuracy is above a certain threshold. To solve these problems, we propose heuristic based algorithms which are simple but only search a small number of layer combinations (i.e., where to run which DNN model layers). To further improve the performance, we propose a Machine Learning based Model Partition (MLMP) algorithm. MLMP searches more layer combinations and considers both accuracy loss and processing time simultaneously. We also address many implementation issues to support model partition techniques on mobile devices with NPU. Experimental results show that MLMP outperforms the heuristic based algorithms and it can significantly improve the accuracy or reduce the processing time based on the application requirements.

The rapid progress of deep learning-based techniques such as Convolutional Neural Network (CNN) has enabled many emerging applications related to video analytics and running them on mobile devices can help improve our daily lives in many ways. However, there are many challenges for video analytics on mobile devices using multiple CNN models. CNN models are resource hungry, and each model requires a large amount of computational power and occupies a large portion of memory space. Although video processing can be offloaded to reduce the computation time, transmitting large amount of video data is time consuming. Thus, offloading is not always the best option. Moreover, different CNN models have different memory usage and processing time, making the scheduling problem more complex. As a result, besides deciding which task to be offloaded, we must decide which CNN model should reside in the memory and for how long, and which CNN model should be switched out due to memory constraint. In this paper, we propose resource aware scheduling algorithms to address these challenges. We identify the task scheduling problem for running multiple CNN models on mobile devices under resource constraints and formulate it as an integer programming problem. We propose resource-aware scheduling algorithms which combine offloading and local processing methods to minimize the completion time of video processing. We implement the proposed scheduling algorithms on Android-based smartphones and demonstrate its effectiveness through extensive experiments.

Distributed collaborative learning has enabled building machine learning models from distributed mobile users' data. It allows the server and users to collaboratively train a learning model where users only share model parameters with the server. To protect privacy, the server can use secure multiparty computation to learn the global model without revealing users' parameter updates in the clear. However this privacy preserving distributed learning opens the door to poisoning attacks, where malicious users poison their training data to maliciously influence the behavior of the global model. In this paper, we propose MLGuard, a privacy preserving distributed collaborative learning system with poisoning attack mitigation. MLGuard employs lightweight secret sharing scheme and a novel poisoning attack mitigation technique. We address several challenges such as preserving users' privacy, mitigating poisoning attacks, respecting resource constraints of mobile devices, and scaling to large number of users. Evaluation results demonstrate the effectiveness of MLGuard on building high accurate learning models with the existence of malicious users, while imposing minimal communication cost on mobile devices.

Many mobile applications have been developed to apply deep learning for video analytics. Although these advanced deep learning models can provide us with better results, they also suffer from the high computational overhead which means longer delay and more energy consumption when running on mobile devices. To address this issue, we propose a framework called FastVA, which supports deep learning video analytics through edge processing and Neural Processing Unit (NPU) in mobile. The major challenge is to determine when to offload the computation and when to use NPU. Based on the processing time and accuracy requirement of the mobile application, we study two problems: Max-Accuracy where the goal is to maximize the accuracy under some time constraints, and Max-Utility where the goal is to maximize the utility which is a weighted function of processing time and accuracy. We formulate them as integer programming problems and propose heuristics based solutions. We have implemented FastVA on smartphones and demonstrated its effectiveness through extensive evaluations.

With the proliferation of IoT cameras, it is possible to use crowdsourced videos to help find interested targets (e.g., crime suspect, lost child, lost vehicle) on demand. Due to the ubiquity of IoT cameras such as dash mounted and phone cameras, the crowdsourced videos have much better spatial coverage compared to only using surveillance cameras, and, thus, can significantly improve the effectiveness of target search. However, this may raise privacy concerns when workers (owners of IoT cameras) are provided with photos of the target. Also, the videos captured by the workers may be misused to track bystanders. To address this problem, we design and implement TargetFinder, a privacy preserving system for target search through IoT cameras. By exploiting homomorphic encryption techniques, the server can search for the target on encrypted information. We also propose techniques to allow the requester (e.g., the police) to receive images that include the target, while all other captured images of the bystanders are not revealed. Moreover, the target’s face image is not revealed to the server and the participating workers. Due to the high computation overhead of the cryptographic primitives, we develop optimization techniques in order to run our privacy preserving protocol on mobile devices. We also formulate and solve a worker selection problem to maximize the probability of finding the target under some budget constraint. A real-world demo and extensive evaluations demonstrate the effectiveness of TargetFinder.

Although streaming video at a higher bitrate (resolution) can lead to better Quality of Experience (QoE), a larger amount of data will have to be downloaded and processed on smartphones and thus consuming more energy. On a moving bus where the wireless signal is weak, more energy will have to be spent on maintaining high bitrate video streaming than at a static environment such as at home or a cafe where the wireless signal is strong. On the other hand, the user perceived QoE does not increase too much by watching high bitrate videos in a vibrating environment (i.e., a moving vehicle), because the perception of video quality is affected by the environment such as the vibration or shaking on a moving bus. To address this problem, we propose to save energy by considering the context (environment) of video streaming. To model the impact of context, we exploit the embedded sensors (e.g., accelerometer) in smartphones to record the vibration level during video streaming. Based on quality assessment experiments, we collect traces and model the impacts of video bitrate and vibration level on QoE, and model the impacts of video bitrate and signal strength on power consumption. Based on the QoE model and the power model, we formulate the energy-aware and context-aware video streaming problem as an optimization problem. We present an optimal algorithm which can maximize QoE and minimize energy. Since the optimal algorithm requires perfect knowledge of future tasks, we further propose an online bitrate selection algorithm. Through real measurements and trace-driven simulations, we demonstrate that the proposed algorithm can significantly outperform existing approaches when considering both energy and QoE.