In many real-world applications, data are distributed across different geographical regions, and may come from different distributions which result in multiple learning tasks. In such cases, distributed multi-task learning is usually used to learn the multiple related tasks to improve the generalization performance for each task. Among the distributed multi-task learning algorithms, distributed multi-task relationship learning (DMTRL) attracts much attention in the community as it learns task relationships from data, instead of imposing a prior task relatedness assumption. To perform DMTRL, task model or its gradient is transferred between task node and central server. There will be a potential privacy violation when the distributed data are possessed by different institutes and contain sensitive information (e.g., medical records). In this paper, we propose a distributed multi-task relationship learning approach under differential privacy called DRUPE, where privacy protection is achieved through perturbing the gradients at each task node. In particular, to reduce the high variance of the perturbed gradients and achieve fast convergence rate, we develop a variance reduction based gradient calibration method, which first estimates the gradient error from the previous perturbed gradients and then calibrates the current perturbed gradient by subtracting the gradient error term. Moreover, to alleviate the negative effect caused by the inaccurate task relationships that are inferred from the private task models, we present a task relationship calibration method which uses the least-squares approximation algorithm to calibrate the inaccurate pairwise relationships between the tasks. Experimental results on real-world datasets confirm the effectiveness of our approach.

Domain adaptation solves a learning problem in a target domain by utilizing the training data in a different but related source domain. As a simple and efficient method for domain adaptation, correlation alignment transforms the distribution of the source domain by utilizing the covariance matrix of the target domain, such that a model trained on the transformed source data can be applied to the target data. However, when source and target domains come from different institutes, exchanging information between the two domains might pose a potential privacy risk. In this paper, for the first time, we propose a differentially private correlation alignment approach for domain adaptation called PRIMA, which can provide privacy guarantees for both the source and target data. In PRIMA, to relieve the performance degradation caused by perturbing the covariance matrix in high dimensional setting, we present a random subspace ensemble based covariance estimation method which splits the feature spaces of source and target data into several low dimensional subspaces. Moreover, since perturbing the covariance matrix may destroy its positive semi-definiteness, we develop a shrinking based method for the recovery of positive semi-definiteness of the covariance matrix. Experimental results on standard benchmark datasets confirm the effectiveness of our approach.