In recent years, deep neural networks have shown extraordinary power in various practical learning tasks, especially in object detection, classification, natural language processing. However, deploying such large models on resource-constrained devices or embedded systems is challenging due to their high computational cost. Efforts such as model partition, pruning, or quantization have been used at the expense of accuracy loss. Knowledge distillation is a technique that transfers model knowledge from a well-trained model (teacher) to a smaller and shallow model (student). Instead of using a learning model on the cloud, we can deploy distilled models on various edge devices, significantly reducing the computational cost, memory usage and prolonging the battery lifetime. In this work, we propose a novel neuron manifold distillation (NMD) method, where the student models imitate the teacher's output distribution and learn the feature geometry of the teacher model. In addition, to further improve the cloud-based learning system reliability, we propose a confident prediction mechanism to calibrate the model predictions. We conduct experiments with different distillation configurations over multiple datasets. Our proposed method demonstrates a consistent improvement in accuracy-speed trade-offs for the distilled model.

Federated learning, in contrast to traditional learning paradigms, has demonstrated its unique advantages in providing intelligence at the edge. However, existing federated learning approaches focus on the end-to-end classification tasks requiring a simple collaboration procedure where each participant can perform its local training independently. Unfortunately, there are still many tasks relying on learning the distinguishable feature metrics with respect to all the data, which is a different collaboration procedure across training participants. For example, the model for people identification has to ensure the feature representing a person is dissimilar to those representing others. To enable such federated learning for deep metrics (a.k.a federated deep metric learning) is challenging due to the data privacy and procedure robustness issues. With the consideration of these two challenges, this work proposes a novel computing framework for federated deep metric learning. This framework leverages the system-algorithm co-design to address privacy concerns via the Trusted Execution Environment (SGX enclave) and Differential Privacy mechanism. It also introduces a large-scale federated protocol which can robustly and efficiently deal with practical factors like the network fluctuation. We implement and evaluate our computing framework with two settings. One is a real-world implementation with a large number of mobile devices, while the other one is in our controllable environment for conducting experiments in various tasks. Our evaluation results show that our computing framework is able to train federated deep metric learning models with excellent scalability, data privacy preserving, and considerable accuracy even in exception conditions.

Training large-scale machine learning models usually demands a distributed approach to process the huge amount of training data efficiently. However, the high network communication cost introduced by parallel stochastic gradient descent (SGD) algorithms is a well-known bottleneck. To this end, we propose CE-SGD, a communication-efficient distributed machine learning algorithm that aggressively reduces the amount of gradient data exchanged among the training workers. CE-SGD belongs to the family of gradient sparsification schemes. CE-SGD adaptively adjusts the gradient sparsity according to the model's feedback. It also selectively transmits the gradients based on their degree of participation in the backpropagation. We mathematically prove the convergence of CE-SGD for both convex and non-convex cases and conduct a series of experiments on our CE-SGD implementation. Our experiments reveal that CE-SGD can achieve fast convergence, desirable gradient compression ratio, and high accuracy with low network bandwidth cost compared to state-of-the-art algorithms.

Although deep neural networks show their extraordinary power in various object detection tasks, it is very challenging for them to be deployed on resource constrained devices or embedded systems due to their high computational cost. Efforts such as model partition, pruning or quantization have been used at an expense of accuracy loss. Recently proposed knowledge distillation (KD) aims at transferring model knowledge from a well-trained model (teacher) to a smaller and faster model (student), which can significantly reduce the computational cost, memory usage, and prolong the battery lifetime. In this work, we propose a novel neuron manifold distillation (NMD), where the student models not only imitate teacher’s output activations, but also learn the feature geometry structure of the teacher. Our approach produces a high-quality, compact, and lightweight student model. We conduct comprehensive experiments with different distillation configurations over multiple datasets, and the proposed method demonstrates a consistent improvement in accuracy-speed trade-offs for the distilled model.

By combining the advantages of both quantum computing and deep learning, quantum neural networks have become popular in recent research. In order to collaborate multiple quantum machines with local training data to train a global model, quantum federated learning is proposed. However, similar to classic federated learning, when communicating with multiple machines, quantum federated learning also faces the threats of Byzantine attacks. The byzantine attack is a kind of attack in a distributed system when some machines upload malicious information instead of the honest computational results to the server. In this article, we compare the differences of Byzantine problems between classic distributed learning and quantum federated learning, and modify the previously proposed four kinds of Byzantine tolerant algorithms to the quantum version. We conduct simulated experiments to show a similar performance of the quantum version with the classic version.

Federated Learning is a machine learning scheme in which a shared prediction model can be collaboratively learned by a number of distributed nodes using their locally stored data. It can provide better data privacy because training data are not transmitted to a central server. Federated learning is well suited for edge computing applications and can leverage the the computation power of edge servers and the data collected on widely dispersed edge devices. To build such an edge federated learning system, we need to tackle a number of technical challenges. In this survey, we provide a new perspective on the applications, development tools, communication efficiency, security & privacy, migration and scheduling in edge federated learning.

Many times, training a large scale deep learning neural network on a single machine becomes more and more difficult for a complex network model. Distributed training provides an efficient solution, but Byzantine attacks may occur on participating workers. They may be compromised or suffer from hardware failures. If they upload poisonous gradients, the training will become unstable or even converge to a saddle point. In this paper, we propose FABA, a Fast Aggregation algorithm against Byzantine Attacks, which removes the outliers in the uploaded gradients and obtains gradients that are close to the true gradients. We show the convergence of our algorithm. The experiments demonstrate that our algorithm can achieve similar performance to non-Byzantine case and higher efficiency as compared to previous algorithms.

Many edge computing systems rely on virtual machines (VMs) to deliver their services. It is challenging, however, to deploy the virtualization mechanisms on edge computing hardware infrastructures. In this paper, we introduce the engineering and research trends of achieving efficient VM management in edge computing. We elaborate on: 1) the virtualization frameworks for edge computing developed in both the industry and the academia; 2) the virtualization techniques tailored for edge computing; 3) the placement and scheduling algorithms optimized for edge computing; and 4) the research problems in security related to virtualization of edge computing.