Differentially Private Stochastic Gradient Descent (DP-SGD) is a prime method for training machine learning models with rigorous privacy guarantees. Since its birth, DP-SGD has gained popularity and has been widely adopted in both academic and industrial research. One well-known challenge when using DP-SGD is how to improve utility while maintaining privacy. To this end, recently we have seen several proposals that clip the gradients with adaptive thresholds rather than a fixed one. Although each proposal comes with some theoretical justification, the theories often rely on strong assumptions and are not compatible with each other. It is hard to know whether they are good in practice and how good they are. In this paper, we investigate adaptive clipping in DP-SGD from an empirical perspective. With extensive experiments, we were able to gain some fresh insights and proposed two new adaptive clipping strategies based on them. We cross-compared the existing methods and our new strategies experimentally. Results showed that our strategies did provide a substantial improvement in model accuracy, and outperformed the state-of-the-art adaptive clipping methods consistently.

Over the past few years, a variety of membership inference attacks against deep learning models have emerged, raising significant privacy concerns. These attacks can easily infer whether a sample exists in the training set of the target model with little adversary knowledge, and the inference accuracy is often much higher than random guessing, which causes serious privacy leakage. To this end, defenses against membership inference attacks have attracted great interest. However, the current available defense methods such as regularization, differential privacy, and knowledge distillation are unable to balance the trade-off between privacy and utility well. In this paper, we combine knowledge distillation and generative adversarial networks to propose a novel training framework that can effectively defend against membership inference attacks, called KD-GAN. Extensive experiments show that our method implements an attack success rate of nearly 0.5 (random guesses) which can successfully defend against membership inference attacks without causing significant damage to model utility, and consistently outperforming other defense methods in the balance of privacy and utility.