Vast data and computing resources are commonly needed to train deep neural networks, causing an unaffordable price for individual users. Motivated by the increasing demands of deep learning applications, sharing well-trained models becomes popular. The owner of a pre-trained model can share it by publishing the model directly or providing a prediction interface. Either way, individual users can benefit from deep learning without much cost, and computing resources can be saved. However, recent studies of machine learning security have identified severe threats to these model publishing approaches. This paper will focus on the privacy leakage issue of publishing well-trained deep neural network models. To tackle this problem, we propose a series of secure model publishing solutions based on training task parallelism. Specifically, we show how to estimate private model parameters through parallel model training and generate new model parameters in a privacy-preserving manner to replace the original ones for publishing. Based on data parallelism and parameter generating techniques, we design another two solutions concentrating on model quality and parameter privacy, respectively. Through privacy leakage analysis and experimental attack evaluation, we conclude that deep neural network models published with our solutions can provide on-demand model quality guarantees and resist membership inference attacks.

Deep neural networks (DNNs) have brought significant performance improvements to various real-life applications. However, a DNN training task commonly requires intensive computing resources and a huge data collection, which makes it hard for personal devices to carry out the entire training, especially for mobile devices. The federated learning concept has eased this situation. However, it is still an open problem for individuals to train their own DNN models at an affordable price. In this article, we propose an alternative DNN training strategy for resource-limited users. With the help of an untrusted server, end users can offload their DNN training tasks to the server in a privacy-preserving manner. To this end, we study the possibility of the separation of a DNN. Then we design a differentially private activation algorithm for end users to ensure the privacy of the offloading after model separation. Furthermore, to meet the rising demand for federated learning, we extend the offloading solution to parallel DNN models training with a secure model weights aggregation scheme for the privacy concern. Experimental results prove the feasibility of computation offloading solutions for DNN models in both solo and parallel modes.

Machine learning as a service has emerged recently to relieve tensions between heavy deep learning tasks and increasing application demands. A deep learning service provider could help its clients to benefit from deep learning techniques at an affordable price instead of huge resource consumption. However, the service provider may have serious concerns about model privacy when a deep neural network model is published. Previous model publishing solutions mainly depend on additional artificial noise. By adding elaborated noises to parameters or gradients during the training phase, strong privacy guarantees like differential privacy could be achieved. However, this kind of approach cannot give guarantees on some other aspects, such as the quality of the disturbingly trained model and the convergence of the modified learning algorithm. In this paper, we propose an alternative private deep neural network model publishing solution, which caused no interference in the original training phase. We provide privacy, convergence and quality guarantees for the published model at the same time. Furthermore, our solution can achieve a smaller privacy budget when compared with artificial noise based training solutions proposed in previous works. Specifically, our solution gives an acceptable test accuracy with privacy budget ϵ = 1. Meanwhile, membership inference attack accuracy will be deceased from nearly 90% to around 60% across all classes.