In recent years, deep neural networks have shown extraordinary power in various practical learning tasks, especially in object detection, classification, natural language processing. However, deploying such large models on resource-constrained devices or embedded systems is challenging due to their high computational cost. Efforts such as model partition, pruning, or quantization have been used at the expense of accuracy loss. Knowledge distillation is a technique that transfers model knowledge from a well-trained model (teacher) to a smaller and shallow model (student). Instead of using a learning model on the cloud, we can deploy distilled models on various edge devices, significantly reducing the computational cost, memory usage and prolonging the battery lifetime. In this work, we propose a novel neuron manifold distillation (NMD) method, where the student models imitate the teacher's output distribution and learn the feature geometry of the teacher model. In addition, to further improve the cloud-based learning system reliability, we propose a confident prediction mechanism to calibrate the model predictions. We conduct experiments with different distillation configurations over multiple datasets. Our proposed method demonstrates a consistent improvement in accuracy-speed trade-offs for the distilled model.

Training large-scale machine learning models usually demands a distributed approach to process the huge amount of training data efficiently. However, the high network communication cost introduced by parallel stochastic gradient descent (SGD) algorithms is a well-known bottleneck. To this end, we propose CE-SGD, a communication-efficient distributed machine learning algorithm that aggressively reduces the amount of gradient data exchanged among the training workers. CE-SGD belongs to the family of gradient sparsification schemes. CE-SGD adaptively adjusts the gradient sparsity according to the model's feedback. It also selectively transmits the gradients based on their degree of participation in the backpropagation. We mathematically prove the convergence of CE-SGD for both convex and non-convex cases and conduct a series of experiments on our CE-SGD implementation. Our experiments reveal that CE-SGD can achieve fast convergence, desirable gradient compression ratio, and high accuracy with low network bandwidth cost compared to state-of-the-art algorithms.

With the fast development of quantum computing and deep learning, quantum neural networks have attracted great attention recently. By leveraging the power of quantum computing, deep neural networks can potentially overcome computational power limitations in classic machine learning. However, when multiple quantum machines wish to train a global model using the local data on each machine, it may be very difficult to copy the data into one machine and train the model. Therefore, a collaborative quantum neural network framework is necessary. In this article, we borrow the core idea of federated learning to propose QuantumFed, a quantum federated learning framework to have multiple quantum nodes with local quantum data train a mode together. Our experiments show the feasibility and robustness of our framework.

Although deep neural networks show their extraordinary power in various object detection tasks, it is very challenging for them to be deployed on resource constrained devices or embedded systems due to their high computational cost. Efforts such as model partition, pruning or quantization have been used at an expense of accuracy loss. Recently proposed knowledge distillation (KD) aims at transferring model knowledge from a well-trained model (teacher) to a smaller and faster model (student), which can significantly reduce the computational cost, memory usage, and prolong the battery lifetime. In this work, we propose a novel neuron manifold distillation (NMD), where the student models not only imitate teacher’s output activations, but also learn the feature geometry structure of the teacher. Our approach produces a high-quality, compact, and lightweight student model. We conduct comprehensive experiments with different distillation configurations over multiple datasets, and the proposed method demonstrates a consistent improvement in accuracy-speed trade-offs for the distilled model.

A large volume of data is generated by ubiquitous Internet-of-Things (IoT) devices and utilized to train machine learning models by IoT manufacturers to provide users with better services. Many deep learning systems for IoT data are required to perform all computation locally on small devices, which is not suitable for these resource-constrained devices. The devices can also send all the collected data to a server for costly model training by ignoring privacy concerns. To design an efficient and secure deep learning model training system, in this paper, we propose a federated learning system on the edge using the differential privacy mechanism to protect sensitive information and offload computation work from edge devices to edge servers, with consideration of communication reduction. In our system, a large-scale deep learning model is partitioned onto edge devices and edge servers, and trained in a distributed manner, in which all untrusted components are prevented from retrieving protected information from the training and inference process. We evaluate the proposed approach with respect to computation, communication, and privacy protection. The experiment results show that the proposed approach can preserve users’ privacy while significantly reducing computation and communication costs.

By combining the advantages of both quantum computing and deep learning, quantum neural networks have become popular in recent research. In order to collaborate multiple quantum machines with local training data to train a global model, quantum federated learning is proposed. However, similar to classic federated learning, when communicating with multiple machines, quantum federated learning also faces the threats of Byzantine attacks. The byzantine attack is a kind of attack in a distributed system when some machines upload malicious information instead of the honest computational results to the server. In this article, we compare the differences of Byzantine problems between classic distributed learning and quantum federated learning, and modify the previously proposed four kinds of Byzantine tolerant algorithms to the quantum version. We conduct simulated experiments to show a similar performance of the quantum version with the classic version.

Federated Learning is a machine learning scheme in which a shared prediction model can be collaboratively learned by a number of distributed nodes using their locally stored data. It can provide better data privacy because training data are not transmitted to a central server. Federated learning is well suited for edge computing applications and can leverage the the computation power of edge servers and the data collected on widely dispersed edge devices. To build such an edge federated learning system, we need to tackle a number of technical challenges. In this survey, we provide a new perspective on the applications, development tools, communication efficiency, security & privacy, migration and scheduling in edge federated learning.

Many times, training a large scale deep learning neural network on a single machine becomes more and more difficult for a complex network model. Distributed training provides an efficient solution, but Byzantine attacks may occur on participating workers. They may be compromised or suffer from hardware failures. If they upload poisonous gradients, the training will become unstable or even converge to a saddle point. In this paper, we propose FABA, a Fast Aggregation algorithm against Byzantine Attacks, which removes the outliers in the uploaded gradients and obtains gradients that are close to the true gradients. We show the convergence of our algorithm. The experiments demonstrate that our algorithm can achieve similar performance to non-Byzantine case and higher efficiency as compared to previous algorithms.

Many edge computing systems rely on virtual machines (VMs) to deliver their services. It is challenging, however, to deploy the virtualization mechanisms on edge computing hardware infrastructures. In this paper, we introduce the engineering and research trends of achieving efficient VM management in edge computing. We elaborate on: 1) the virtualization frameworks for edge computing developed in both the industry and the academia; 2) the virtualization techniques tailored for edge computing; 3) the placement and scheduling algorithms optimized for edge computing; and 4) the research problems in security related to virtualization of edge computing.