As the primary driver of intelligent mobile applications, deep neural networks (DNNs) have gradually deployed to millions of mobile devices, producing massive latency-sensitive and computation-intensive tasks daily. Mobile edge computing facilitates the deployment of computing resources at the edge, which enables fine-grained offloading of DNN inference tasks from mobile devices to edge nodes. However, most existing studies have not systematically considered three crucial performance aspects: scheduling multiple streams of DNN inference tasks, leveraging multi-exit models to hasten task processing, and partitioning inference models for partial offloading. To this end, this paper proposes an adaptive inference framework in mobile edge computing, which can dynamically select the exit point and partition point for multiple inference task streams. We design a dynamic programming algorithm to obtain an efficient solution under the ideal condition that task arrival information is known. Further, we design a learning-based algorithm for online scheduling, whose training efficiency is improved based on historical experience initialization and priority experience replay. Experimental results show that compared with the Greedy algorithm, the online algorithm improves the performance on two environmental parameters by an average of 5.9% and 32%, respectively.

With the rapid development of the Internet of Things (IoT), more and more smart devices are connected to the Internet, implementing Deep Neural Network (DNN) models on edges for collaborative inference via device-edge synergy has become a feasible method for improving application performance in many scenarios. However, when faced with the task stream scenario with latency guarantee such as video surveillance and industrial production line, we need adaptive edge intelligence to make adjustments in real-time according to the changes of the task stream. There are many adaptive edge intelligence technologies in the existing works, such as early-exit mechanism and model selection, but they don’t take the requirements of the task stream scenario into consideration. In this paper, we propose a device-edge collaborative inference system based on the early-exit mechanism to solve the problem of adaptive edge intelligence in the task stream scenario. Then, we design an offline dynamic programming (DP) algorithm and an online deep reinforcement learning (DRL) algorithm to dynamically select the exit point and partition point of the branchy model in the task stream, which aims to balance the number of tasks accomplished and task inference accuracy in the system. Experimental results show that the DRL algorithm can achieve performance close to that of the DP algorithm in the task stream scenario.