Machine unlearning is the primary way to fight for the “right to be forgotten” in machine learning field, which is promoted among multiple privacy legislations, such as GDPR and CCPA. However, the latest work has shown that machine unlearning in deep learning cannot be easily verified, making it challenging for the data owners to be convinced that their data has indeed been deleted as claimed. This is especially problematic for federated learning (FL), where a number of participants jointly train a global model while each participant should be free to join and leave the federation as they wish. However,the lack of a reliable approach to verify unlearning in FL will no doubt discourage certain users from joining the federation.In this work, we propose A2E, a black-box watermarking scheme from a leaving participant's perspective to realize verifiable federated unlearning which incurs minimum impact and no security threats to vanilla FL. The key idea is to leverage adversarial training to inject the anti-adversarial example (A2E) characteristic into the uploaded model updates of the last contribution round as the watermark of the leaving participant. Then, we verify whether the server has indeed executed the effective unlearning, with the newly developed probabilistic quantification of unlearning confidence, by checking the unlearned global model's resistance to the specially generated watermark-dependent adversarial examples of the leaver. We conducted large-scale experiments on various popular datasets (including natural images, medical images, and speech) and model structures (including LeNet, ResNet, VGG, and LSTM). The results confirm the effectiveness of A2E in verifying federated unlearning with a high confidence. We also show that A2E is robust against multiple adaptive strategies from the adversarial server and participants.

It is notoriously challenging to audit the potential unauthorized data usage in deep learning (DL) model development lifecycle, i.e., to judge whether certain private user data has been used to train or fine-tune a DL model without authorization. Yet, such data usage auditing is crucial to respond to the urgent requirements of trustworthy Artificial Intelligence (AI) such as data transparency, which are promoted and enforced in recent AI regulation rules or acts like General Data Protection Regulation (GDPR) and EU AI Act. In this work, we propose TeDA, a simple and flexible testing framework for auditing data usage in DL model development process. Given a set of user’s private data to protect (Dp), the intuition of TeDA is to apply membership inference (with good intention) for judging whether the model to audit (Ma) is likely to be trained with Dp. Notably, to significantly expose the usage under membership inference, TeDA applies imperceptible perturbation directed by boundary search to generate a carefully crafted test suite Dt (which we call ‘isotope’) based on Dp. With the test suite, TeDA then adopts membership inference combined with hypothesis testing to decide whether a user’s private data has been used to train Ma with statistical guarantee. We evaluated TeDA through extensive experiments on ranging data volumes across various model architectures for data-sensitive face recognition and medical diagnosis tasks. TeDA demonstrates high feasibility, effectiveness and robustness under various adaptive strategies (e.g., pruning and distillation).

Federated learning (FL) has emerged as a privacy-aware collaborative learning paradigm where participants jointly train a powerful model without sharing their private data. One desirable property for FL is the implementation of theright to be forgotten (RTBF), i.e., a leaving participant has the right to request the deletion of its private data from the global model. However,unlearning itself may not be enough to implement RTBF unless the unlearning effect can be independently verified, an important aspect that has been overlooked in the current literature. Unlearning verification is particularly challenging in FL as the unlearning effect on one participant's data could be canceled by the contribution of other participants. In this work, we prompt the concept ofverifiable federated unlearningand proposeVeriFi, a unified framework that allows systematic analysis of federated unlearning and quantification of its effect, with different combinations of various unlearning and verification methods. InVeriFi, the leaving participant is granted theright to verify (RTV)to actively verify the unlearning effect in the next few rounds immediately after notifying the server of its intention to leave, along with local verification done through two steps: 1)markingthat fingerprints the leaving participant by specially-designedmarkersand 2)checkingthat examines the global model's performance change on the markers. Based onVeriFi, we have conducted so far the most systematic study on verifiable federated unlearning, covering six unlearning methods and five verification methods. Our study sheds light on the existing drawbacks and potential alternatives for both unlearning and verification methods. During the study, we also propose a more efficient and FL-friendly unlearning method$^{u}$S2U, and two more effective and robust non-invasive (without training controllability, external data, white-box model access nor introducing new security risks) verification methods$^{v}$FM and$^{v}$EM. While the proposed methods may not be a panacea for all the challenges, they address several key drawbacks of existing methods and represent a promising step toward effective, efficient, robust, and more importantly, non-invasive federated unlearning and verification. We extensively evaluateVeriFion seven datasets, including natural/facial/medical images and audios, and four types of deep learning models, including both Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). We hope, such an extensive and holistic experimental evaluation, although admittedly complex and challenging, could help establish important empirical understandings, evidence, and insights for trustworthy federated unlearning.

Vertical federated learning (VFL) revolutionizes privacy-preserved collaboration for small businesses that have distinct but complementary feature sets. However, as the scope of VFL expands, the constant entering and leaving of participants and the subsequent exercise of the “right to be forgotten” pose a great challenge in practice. The question of how to efficiently erase one’s contribution from the shared model remains largely unexplored in the context of VFL. In this article, we introduce a vertical federated unlearning framework, which integrates model checkpointing techniques with a hybrid, first-order optimization technique. The core concept is to reduce backpropagation time and improve convergence/generalization by combining the advantages of the existing optimizers. We provide in-depth theoretical analysis and time complexity to illustrate the effectiveness of the proposed design. We conduct extensive experiments on six public datasets and demonstrate that our method could achieve up to 6.3× speedup compared to the baseline, with negligible influence on the original learning task.

Neural backdoors represent one primary threat to the security of deep learning systems. The intensive research has produced a plethora of backdoor attacks/defenses, resulting in a constant arms race. However, due to the lack of evaluation benchmarks, many critical questions remain under-explored: (i) what are the strengths and limitations of different attacks/defenses? (ii) what are the best practices to operate them? and (iii) how can the existing attacks/defenses be further improved? To bridge this gap, we design and implement TROJAN-ZOO, the first open-source platform for evaluating neural backdoor attacks/defenses in a unified, holistic, and practical manner. Thus far, focusing on the computer vision domain, it has incorporated 8 representative attacks, 14 state-of-the-art defenses, 6 attack performance metrics, 10 defense utility metrics, as well as rich tools for in-depth analysis of the attack-defense interactions. Leveraging TROJANZOO, we conduct a systematic study on the existing attacks/defenses, unveiling their complex design spectrum: both manifest intricate trade-offs among multiple desiderata (e.g., the effectiveness, evasiveness, and transferability of attacks). We further explore improving the existing attacks/defenses, leading to a number of interesting findings: (i) one-pixel triggers often suffice; (ii) training from scratch often outperforms perturbing benign models to craft trojan models; (iii) optimizing triggers and trojan models jointly greatly improves both attack effectiveness and evasiveness; (iv) individual defenses can often be evaded by adaptive attacks; and (v) exploiting model interpretability significantly improves defense robustness. We envision that TROJANZOO will serve as a valuable platform to facilitate future research on neural backdoors.

Deploying machine learning (ML)-based intrusion detection systems (IDS) is an effective way to improve the security of industrial control systems (ICS). However, ML models themselves are vulnerable to adversarial examples, generated by deliberately adding subtle perturbation to the input sample that some people are not aware of, causing the model to give a false output with high confidence. In this article, our goal is to investigate the possibility of stealthy cyber attacks towards IDS, including injection attack, function code attack and reconnaissance attack, and enhance its robustness to adversarial attack. However, adversarial algorithms are subject to communication protocol and legal range of data in ICS, unlike only limited by the distance between original samples and newly generated samples in image domain. We propose two strategies - optimal solution attack and GAN attack - oriented to flexibility and volume of data, formulating an optimization problem to find stealthy attacks, where the former is appropriate for not too large and more flexible samples while the latter provides a more efficient solution for larger and not too flexible samples. Finally, we conduct experiments on a semi-physical ICS testbed with a high detection performance ensemble ML-based detector to show the effectiveness of our attacks. The results indicate that new samples of reconnaissance and function code attack produced by both optimal solution and GAN algorithm possess 80 percent higher probability to evade the detector, still maintaining the same attack effect. In the meantime, we adopt adversarial training as a method to defend against adversarial attack. After training on the mixture of orginal dataset and newly generated samples, the detector becomes more robust to adversarial examples.