Deep learning algorithms excel at extracting fine-grained patterns from data to enable accurate predictions. However, this capability can conflict with the goal of protecting the privacy of individuals. This paper addresses both the practical and theoretical challenges of developing privacy-preserving deep learning algorithms that maintain strong predictive performance. Specifically, we propose a differentially private GD algorithm for three-layer neural networks with gradient perturbation. Both privacy and utility guarantees of the proposed method are presented, attaining-up to constants-an optimal excess population risk of order$\mathcal{O}\left(\frac{1}{\sqrt{n}}+\frac{\sqrt{d \log (1 / \delta)}}{n \epsilon}\right)$, where$s$is the data dimension,$\epsilon$is the privacy budget, and$\delta$is the failure probability. To our knowledge, this is the first utility analysis achieving optimal rates, on par with their counterparts in the convex setting, for differentially private GD algorithms in multi-laver neural networks.

In this paper, we aim to develop stochastic hard thresholding algorithms for the important problem of AUC maximization in imbalanced classification. The main challenge is the pairwise loss involved in AUC maximization. We overcome this obstacle by reformulating the U-statistics objective function as an empirical risk minimization (ERM), from which a stochastic hard thresholding algorithm (SHT-AUC) is developed. To our best knowledge, this is the first attempt to provide stochastic hard thresholding algorithms for AUC maximization with a per-iteration cost O(bd) where d and b are the dimension of the data and the minibatch size, respectively. We show that the proposed algorithm enjoys the linear convergence rate up to a tolerance error. In particular, we show, if the data is generated from the Gaussian distribution, then its convergence becomes slower as the data gets more imbalanced. We conduct extensive experiments to show the efficiency and effectiveness of the proposed algorithms.