With the prevalence of the Pretraining-Finetuning paradigm in transfer learning, the robustness of down-stream tasks has become a critical concern. In this work, we delve into adversarial robustness in transfer learning and reveal the critical role of initialization, including both the pretrained model and the linear head. First, we discover the necessity of an adversarially robust pre-trained model. Specifically, we reveal that with a standard pretrained model, Parameter-Efficient Finetuning (PEFT) methods either fail to be adversarially robust or continue to exhibit significantly degraded adversarial robustness on downstream tasks, even with adversarial training during finetuning. Leveraging a robust pretrained model, surprisingly, we observe that a simple linear probing can outper-form full finetuning and other PEFT methods with random initialization on certain datasets. We further identify that linear probing excels in preserving robustness from the robust pretraining. Based on this, we propose Robust Linear Initialization (RoLl) for adversarial finetuning, which initializes the linear head with the weights obtained by ad-versarial linear probing to maximally inherit the robust-ness from pretraining. Across five different image classification datasets, we demonstrate the effectiveness of RoLI and achieve new state-of-the-art results. Our code is available at https://github.com/DongXzz/RoLI.

Fine-tuning pretrained language models (PLMs) for downstream tasks is a large-scale optimization problem, in which the choice of the training algorithm critically determines how well the trained model can generalize to unseen test data, especially in the context of few-shot learning.To achieve good generalization performance and avoid overfitting, techniques such as data augmentation and pruning are often applied.However, adding these regularizations necessitates heavy tuning of the hyperparameters of optimization algorithms, such as the popular Adam optimizer.In this paper, we propose a two-stage fine-tuning method, PAC-tuning, to address this optimization challenge.First, based on PAC-Bayes training, PAC-tuning directly minimizes the PAC-Bayes generalization bound to learn proper parameter distribution.Second, PAC-tuning modifies the gradient by injecting noise with the variance learned in the first stage into the model parameters during training, resulting in a variant of perturbed gradient descent (PGD).In the past, the fewshot scenario posed difficulties for PAC-Bayes training because the PAC-Bayes bound, when applied to large models with limited training data, might not be stringent.Our experimental results across 5 GLUE benchmark tasks demonstrate that PAC-tuning 1 successfully handles the challenges of fine-tuning tasks and outperforms strong baseline methods by a visible margin, further confirming the potential to apply PAC training for any other settings where the Adam optimizer is currently used for training.

Boosting is a widely-used ensemble learning technique that combines a series of weak base learners sequentially. While it can achieve competitive predictive performance, boosting may induce unfairness within the ensemble process when applied to group-unbalanced data, which can pose a major obstacle for using boosting in human-centered predictions such as in health informatics and risk management. To address this issue, we propose a boosting-based framework that includes rectified fairness penalty, penalty intensity selection, and post- pruning. This framework can provide a certified fairness guarantee by stabilizing the ensemble process with a fairness constraint given by users, resulting in not only improved fairness but also higher averaged accuracy among different groups.

Instead of learning with pointwise loss functions, learning with pairwise loss functions (pairwise learning) has received much attention recently as it is more capable of modeling the relative relationship between pairs of samples. However, most of the existing algorithms for pairwise learning fail to take into consideration the privacy issue in their design. To address this issue, previous work studied pairwise learning in the Differential Privacy (DP) model. However, their utilities (population errors) are far from optimal. To address the sub-optimal utility issue, in this paper, we proposed new pure or approximate DP algorithms for pairwise learning. Specifically, under the assumption that the loss functions are Lipschitz, our algorithms could achieve the optimal expected population risk for both strongly convex and general convex cases. We also conduct extensive experiments on real-world datasets to evaluate the proposed algorithms, experimental results support our theoretical analysis and show the priority of our algorithms.