Voice data is crucial for modern artificial intelligence (AI) systems, powering various applications from speaker recognition (SR) to AI-generated content. The reliance on massive data raises serious concerns about privacy and property rights due to potential unauthorized misuse. Unfortunately, no effective method has been proposed to protect the privacy and copyright of voice data while meeting perceptual audio quality requirements. To bridge this gap, we introduce Volto, a unified dual-function framework for generating unlearnable yet traceable voice examples. Volto jointly integrates unlearnable perturbations and learnable watermarks, exploiting weaknesses in both human auditory perception and deep neural network (DNN) representations. This synergistic design enables Volto to corrupt critical model-sensitive features, hindering unauthorized model learning, while encoding watermark signals that remain recoverable for ownership verification. The unlearnable examples can confuse DNNs while preserving the perceptual quality, reducing the accuracy of unauthorized models by more than 21.44% in black-box scenarios. Volto also introduces a verifiable watermark, providing robust evidence of data ownership. Our work effectively balances protection effectiveness and usability, offering an effective defense for voice privacy.

Deep Neural Networks are vulnerable to data poisoning attacks, which inject a backdoor by poisoning the training data set with a predefined trigger pattern. However, most existing studies design trigger patterns as exogenous features introduced to clean samples (such as a checkerboard patch), whereas the endogenous features inherited from sample origins (such as deep generative models) have not been investigated yet. In this study, we investigate the efficacy of utilizing Generative Adversarial Network fingerprints to design trigger patterns by examining three attack patterns: the all-label attack, label-specific attack, and semantic-specific attack. Specifically, we select training data that satisfies the requirements of various attack patterns, train a GAN model, and employ samples embedded with GAN fingerprints to generate poisoned data sets. Our evaluations on three data sets (CIFAR-10, GTSRB, and LSUN) demonstrate that employing GAN fingerprints as trigger patterns 1) can achieve average attack success rate results of 39.40% in all-label attack, 89.84% in label-specific attack, and 91.06% in semantic-specific attack, 2) is stealthy by reducing the probability of being exposed, and 3) can resist six existing backdoor detection techniques, three backdoor erasing techniques, and two deepfake detection techniques. Furthermore, it exhibits practical applicability in federated learning scenario.

Transfer-based attacks craft adversarial examples on white-box surrogate models and directly deploy them against black-box target models, offering practical query-free threat scenarios. While flatness-enhanced methods have recently emerged to improve transferability by enhancing the loss surface flatness of adversarial examples, their divergent flatness definitions and heuristic attack designs suffer from unexamined optimization limitations and missing theoretical foundation, thus constraining their effectiveness and efficiency. This work exposes the severely imbalanced exploitation-exploration dynamics in flatness optimization, establishing the first theoretical foundation for flatness-based transferability and proposing a principled framework to overcome these optimization pitfalls. Specifically, we systematically unify fragmented flatness definitions across existing methods, revealing their imbalanced optimization limitations in over-exploration of sensitivity peaks or over-exploitation of local plateaus. To resolve these issues, we rigorously formalize average-case flatness and transferability gaps, proving that enhancing zeroth-order average-case flatness minimizes cross-model discrepancies. Building on this theory, we design a Maximin Expected Flatness (MEF) attack that enhances zeroth-order average-case flatness while balancing flatness exploration and exploitation. Extensive evaluations across 33 models and 43 current transfer-based attacks demonstrate MEF’s superiority: it surpasses the state-of-the-art PGN attack by 4% in attack success rate at half the computational cost and achieves 8% higher success rate under the same budget. When combined with input augmentation, MEF attains 15% additional gains against defense-equipped models, establishing new robustness benchmarks. Our code is available at https://github.com/SignedQiu/MEFAttack.

Recently, large language models (LLMs) have significantly improved the performance of text-to-SQL systems. Nevertheless, many state-of-the-art (SOTA) approaches have overlooked the critical aspect of system robustness. Our experiments reveal that while LLM-driven methods excel on standard datasets, their accuracy is notably compromised when faced with adversarial perturbations. To address this challenge, we propose a robust text-to-SQL solution, called Solid-SQL, designed to integrate with various LLMs. We focus on the pre-processing stage, training a robust schema-linking model enhanced by LLM-based data augmentation. Additionally, we design a two-round, structural similarity-based example retrieval strategy for in-context learning. Our method achieves SOTA SQL execution accuracy levels of 82.1% and 58.9% on the general Spider and Bird benchmarks, respectively. Furthermore, experimental results show that Solid-SQL delivers an average improvement of 11.6% compared to baselines on the perturbed Spider-Syn, Spider-Realistic, and Dr. Spider benchmarks.

Extensive research has shown that Automatic Speech Recognition (ASR) systems are vulnerable to audio adversarial attacks. Current attacks mainly focus on single-source scenarios, ignoring dual-source scenarios where two people are speaking simultaneously. To bridge the gap, we propose a Selective Masking Adversarial attack, namely SMA attack, which ensures that one audio source is selected for recognition while the other audio source is muted in dual-source scenarios. To better adapt to the dual-source scenario, our SMA attack constructs the normal dual-source audio from the muted audio and selected audio. SMA attack initializes the adversarial perturbation with a small Gaus-sian noise and iteratively optimizes it using a selective masking optimization algorithm. Extensive experiments demonstrate that the SMA attack can generate effective and imperceptible audio adversarial examples in the dual-source scenario, achieving an average success rate of attack of 100% and signal-to-noise ratio of 37.15dB on Conformer-CTC, outperforming the baselines.

Abstract Artificial intelligence (AI) is revolutionizing both industries and reshaping the global economy. However, the rapid advancement of AI technologies brings significant security and privacy challenges. Recent incidents highlight vulnerabilities in AI systems, such as data leakage and malicious code injection, leading to severe financial losses and privacy breaches. Although existing studies have discussed specific security threats, they often lack detailed granularity and cover a limited scope. In this survey, we fill this gap by systematically categorizing and analyzing the threats and countermeasures in AI systems, which span both the training and inference stages, encompass centralized and distributed settings, and address both conventional and foundation AI models. By reviewing existing literature, we aim to provide AI researchers and practitioners with a thorough understanding of system vulnerabilities and current countermeasures. We hope to inspire further research into robust solutions, ultimately contributing to the development of resilient AI technologies.

Deep learning-based automatic speech recognition (ASR) systems are capable of transcribing input audio of arbitrary duration into character sequences, which are widely used in daily life. However, recent research has found that deep learning models are vulnerable to backdoor attacks. A malicious adversary can embed a backdoor functionality into the model during the training phase and manipulate the output of the backdoored model by adding a specific trigger to the input during the inference phase. Unfortunately, TrojanModel, the existing state-of-the-art backdoor attack against ASR systems (Zong et al. S&P’23), relies on an overly strong assumption that requires the adversary to modify the model structure beyond data poisoning, which significantly limits its practicability. In this paper, we propose CuckooAttack, a more practical backdoor attack against ASR systems that only requires poisoning a small portion of the training data. We first construct a phoneme-level auxiliary dataset to generate effective, robust, and unnoticeable triggers, while substantially lowering computational expenses. Considering the real-world ASR application scenarios, we propose an adaptive trigger injection mechanism to ensure that the backdoor can be activated on variable-duration input audio under asynchronous temporal conditions. To further enhance the efficacy of CuckooAttack, we design a character-filling strategy tailored for ASR to construct poisoned samples, which facilitates the model in establishing backdoor connections. Extensive experiments show that CuckooAttack achieves comparable performance with TrojanModel under a weaker assumption. Specifically, CuckooAttack achieves an attack success rate of about 99% in the digital domain and over 90% in the physical domain, with a poison rate of only 1%.

SONICUMOS is an enhanced behavior-based face liveness detection system that combines ultrasonic and video signals to sense the 3D head gestures. As face authentication becomes increasingly prevalent, the need for a reliable liveness detection system is paramount. Traditional behavior-based liveness detection methods (e.g., eye-blinking, nodding, etc.), which are widely deployed in mission-critical scenarios like finance and banking applications today, are prone to advanced media-based facial forgery attacks. SONICUMOS aims to incorporate the traditional behaviorbased method for active liveness detection without introducing extra user burden. By employing ultrasonic signals, SONICUMOS capitalizes on the head gestures, significantly raising the security bar. Our approach utilizes the frequency-modulated continuouswave (FMCW) ultrasonic radar for robust 3D gesture recognition compatible with face authentication. We also propose a new dual-feature fusion network that integrates audio and video features at the feature level to increase detection accuracy and resilience against numerous attacks. Our prototype has been tested on seven off-the-shelf Android/iOS smartphones, achieving an overall detection accuracy of 95.83% at an equal error rate (EER) of 4.96% when dealing with 3D impersonation attacks

In recent years, extensive research has been conducted on the vulnerability of ASR systems, revealing that black-box adversarial example attacks pose significant threats to real-world ASR systems. However, most existing black-box attacks rely on queries to the target ASRs, which is impractical when queries are not permitted. In this paper, we propose ZQ-Attack, a transfer-based adversarial attack on ASR systems in the zero-query black-box setting. Through a comprehensive review and categorization of modern ASR technologies, we first meticulously select surrogate ASRs of diverse types to generate adversarial examples. Following this, ZQ-Attack initializes the adversarial perturbation with a scaled target command audio, rendering it relatively imperceptible while maintaining effectiveness. Subsequently, to achieve high transferability of adversarial perturbations, we propose a sequential ensemble optimization algorithm, which iteratively optimizes the adversarial perturbation on each surrogate model, leveraging collaborative information from other models. We conduct extensive experiments to evaluate ZQ-Attack. In the over-the-line setting, ZQ-Attack achieves a 100% success rate of attack (SRoA) with an average signal-to-noise ratio (SNR) of 21.91dB on 4 online speech recognition services, and attains an average SRoA of 100% and SNR of 19.67dB on 16 open-source ASRs. In the over-the-air setting, ZQ-Attack also achieves a 100% SRoA with an average SNR of 15.77dB on 2 commercial intelligent voice control devices.

Deep neural networks are vulnerable to adversarial attacks, leading to erroneous outputs. Adversarial training has been recognized as one of the most effective methods to counter such attacks. However, existing adversarial training techniques have predominantly been evaluated on balanced datasets, whereas real-world data often exhibit a long-tailed distribution, casting doubt on the efficacy of these methods in practical scenarios. In this paper, we delve into the performance of adversarial training under long-tailed distributions. Through an analysis of the prior method “RoBal” (Wu et al., CVPR'21), we discover that utilizing Balanced Softmax Loss (BSL) alone can obtain comparable performance to the complete RoBal approach while significantly reducing the training overhead. Then, we reveal that adversarial training under long-tailed distributions also suffers from robust overfitting similar to uniform distributions. We explore utilizing data augmentation to mitigate this issue and unexpectedly discover that, unlike results obtained with balanced data, data augmentation not only effectively alleviates robust overfitting but also significantly improves robustness. We further identify that the improvement is attributed to the increased diversity of training data. Extensive experiments further corroborate that data augmentation alone can significantly improve robustness. Finally, building on these findings, we demonstrate that compared to RoBal, the combination of BSL and data augmentation leads to a +6.66% improvement in model robustness under AutoAttack on CIFAR-10-LT. Our code is available at: https://github.com/NISPLab/AT-BSL.

DeepFakes technologies possess powerful capabilities to convincingly modify the expressions, appearances, and identities of targets in photos and videos. This capability has enabled various forms of misuse, e.g., blackmail, nonconsensual pornography, and political disinformation, that severely harm the reputation of people. To mitigate this issue, a leading defensive approach is to add adversarial perturbations to the original images or videos, causing the core components of image-translation-based DeepFake systems to fail. However, we found that existing perturbation techniques for image-translation-based DeepFake systems are mostly implemented in white-box settings, making them hard to apply in realistic scenarios. Moreover, these techniques indiscriminately alter the entire image, often failing to protect the most critical facial regions. In this paper, we propose a novel adversarial perturbation generation framework called ReDef in the black-box setting, which narrowly focuses on perturbing facial regions to fool image-translation-based DeepFake systems. By diversifying the output and using the prior knowledge to guide the direction of optimizing the adversarial perturbations, ReDef exhibits better query efficiency and attack success rates. Compared to the state-of-the-art works, ReDef can improve the ASR by 37.8%, and reduce the query count by 41.3%.

Recently, achieving query-efficient adversarial example attacks targeting black-box natural language models has attracted widespread attention from researchers. This task is considered difficult due to the discrete nature of texts, limited knowledge of the target model, and strict query access limitations in real-world systems. However, existing attacks often require a large number of queries or result in low attack success rates, having not met practical requirements. To address this, we propose FastTextDodger, a simple and compact decision-based black-box textual adversarial attack that generates grammatically correct adversarial texts with high attack success rates and few queries. Experimental results show that FastTextDodger achieves an impressive 97.4% attack success rate on benchmark datasets and models, and only needs about 200 queries. Compared to state-of-the-art attacks, FastTextDodger only requires one-tenth of the number of queries in text classification and entailment tasks while maintaining comparable attack success rates and perturbed word rates.

Recent advancements in adversarial attack research have seen a transition from white-box to black-box and even no-box threat models, greatly enhancing the practicality of these attacks. However, existing no-box attacks focus on instance-specific perturbations, leaving more powerful universal adversarial perturbations (UAPs) unexplored. This study addresses a crucial question: can UAPs be generated under a no-box threat model? Our findings provide an affirmative answer with a texture-based method. Artificially crafted textures can act as UAPs, termed Texture-Adv. With a modest density and a fixed budget for perturbations, it can achieve an attack success rate of 80% under the constraint of$l_{\infty }$= 10/255. In addition, Texture-Adv can also take effect under traditional black-box threat models. Building upon a phenomenon associated with dominant labels, we utilize Texture-Adv to develop a highly efficient decision-based attack strategy, named Adv-Pool. This approach creates and traverses a set of Texture-Adv instances with diverse classification distributions, significantly reducing the average query budget to less than 1.3, which is near the 1-query lower bound for decision-based attacks. Moreover, we empirically demonstrate that Texture-Adv, when used as a starting point, can enhance the success rates of existing transfer attacks and the efficiency of decision-based attacks. The discovery suggests its potential as an effective starting point for various adversarial attacks while preserving the original constraints of their threat models.

Adversarial examples (AEs) pose significant threats to deep neural networks (DNNs), as they can deceive models into making incorrect predictions through craftily-designed malicious perturbations. The emergence of decision-based attacks, which rely solely on the top-1 decision label, further increases risks for real-world black-box models. Currently, the prevailing practice for generating effective AEs in decision-based attacks involves penalizing adversarial perturbations using the ℓp-norm. However, this approach often fails to consider the human perception of adversarial perturbations in real-world scenarios. To tackle this issue, we propose a novel and efficient Imperceptible Decision-based Black-box Attack (IDBA). Our method prioritizes optimizing the perception-related distribution of perturbations, rather than solely focusing on the ℓp-norm. Specifically, IDBA analyzes the perceptual preferences of both models and the human vision system, selectively perturbing components that influence model decisions yet remain imperceptible to human eyes. Extensive experiments demonstrate the superior performance of IDBA in both invisibility and query efficiency, a widely used metric in prior works, in comparison to state-of-the-art methods. With only 4.8K queries, IDBA achieves a Feature SIMilarity (FSIM) score of 0.92 while reducing the Learned Perceptual Image Patch Similarity (LPIPS) to 0.12, indicating remarkable imperceptibility.

Optical Character Recognition (OCR) enables automatic text extraction from scanned or digitized text images, but it also makes it easy to pirate valuable or sensitive text from these images. Previous methods to prevent OCR piracy by distorting characters in text images are impractical in real-world scenarios, as pirates can capture arbitrary portions of the text images, rendering the defenses ineffective. In this work, we propose a novel and effective defense mechanism termed the Universal Defensive Underpainting Patch (UDUP) that modifies the underpainting of text images instead of the characters. UDUP is created through an iterative optimization process to craft a small, fixed-size defensive patch that can generate non-overlapping underpainting for text images of any size. Experimental results show that UDUP effectively defends against unauthorized OCR under the setting of any screenshot range or complex image background. It is agnostic to the content, size, colors, and languages of characters, and is robust to typical image operations such as scaling and compressing. In addition, the transferability of UDUP is demonstrated by evading several off-the-shelf OCRs. The code is available at https://github.com/QRICKDD/UDUP.

Adversarial Robustness Distillation (ARD) aims to transfer the robustness of large teacher models to small student models, facilitating the attainment of robust performance on resource-limited devices. However, existing research on ARD primarily focuses on the overall robustness of student models, overlooking the crucial aspect of $\textit{robust fairness}$. Specifically, these models may demonstrate strong robustness on some classes of data while exhibiting high vulnerability on other classes. Unfortunately, the "buckets effect" implies that the robustness of the deployed model depends on the classes with the lowest level of robustness. In this paper, we first investigate the inheritance of robust fairness during ARD and reveal that student models only partially inherit robust fairness from teacher models. We further validate this issue through fine-grained experiments with various model capacities and find that it may arise due to the gap in capacity between teacher and student models, as well as the existing methods treating each class equally during distillation. Based on these observations, we propose $\textbf{Fair}$ $\textbf{A}$dversarial $\textbf{R}$obustness $\textbf{D}$istillation (Fair-ARD), a novel framework for enhancing the robust fairness of student models by increasing the weights of difficult classes, and design a geometric perspective-based method to quantify the difficulty of different classes for determining the weights. Extensive experiments show that Fair-ARD surpasses both state-of-the-art ARD methods and existing robust fairness algorithms in terms of robust fairness (e.g., the worst-class robustness under AutoAttack is improved by at most 12.3\% and 5.3\% using ResNet18 on CIFAR10, respectively), while also slightly improving overall robustness. Our code is available at: [https://github.com/NISP-official/Fair-ARD](https://github.com/NISP-official/Fair-ARD).

Graphs nicely capture data from various domains, allowing the computations of many analytic tasks via graph queries. Graphs of real-world data are often large, albeit useful, and the involved computation can be too heavyweight for commodity computers. For secure outsourcing, we propose (SGX)$^{2}$, a forward-secure structured encryption scheme for graph data, which uses lightweight cryptographic techniques with a trusted execution environment such as SGX. To process million-scale graphs by the limited memory of SGX, we load data on-demand using Dijkstra's algorithm and Fibonacci heap. Compared with most prior graph encryption schemes, (SGX)$^{2}$supports exact shortest-distance queries instead of approximation and can be easily extended to other graph-based analytics.

Automatic speech recognition (ASR) has been widely and commercially employed in health care, autonomous vehicles, and finance. Yet, recent studies have shown that universal adversarial perturbations (UAPs) pose a serious threat to white-box ASR systems, when the adversary has access to the target model. Until now, the impacts of such a threat on commercial systems are still open since their models are not publicly available. To understand the security weakness in the practical black-box setting, this paper introduces the firstzero-queryUAP attacks, called AdvDDoS, with black-box access to ASR systems: we do not need to pay any query expense to estimate UAPs. Specifically, we craft targeted UAPs under a popular feature extractor and a local ASR model by reversing the robust target-category features, in which adversarial perturbations containing robust features are believed to have better transferability. Compared with vanilla UAPs, our UAPs incorporated with target-category features lead to better attacks against commercial ASR systems. We validate the efficacy of our AdvDDoS by launching attacks against a range of commercial ASR systems,i.e., three API services (Alibaba, Tencent, and Baidu), and three personal assistants (Apple Siri, iFlytek, and Google). Extensive experimental results demonstrate the superiority of AdvDDoS. For example, AdvDDoS achieves 83.26% word error rate (WER) and 53.25% success rates of attacks (SRoA) for the universal attack against Tencent ASR API, which outperforms the vanilla UAPs by up to 61.56% on WER and 11.6% on SRoA. The success of our attack sheds light on zero-query UAP attacks against Commercial ASR systems.

Federated learning facilitates the collaborative training of a global model among distributed clients without sharing their training data. Secure aggregation, a new security primitive for federated learning, aims to preserve the confidentiality of both local models and training data. Unfortunately, existing secure aggregation solutions fail to defend against Byzantine failures that are common in distributed computing systems. In this work, we propose a new secure and efficient aggregation framework, SEAR, for Byzantine-robust federated learning. Relying on the trusted execution environment, i.e., Intel SGX, SEAR protects clients’ private models while enabling Byzantine resilience. Considering the limitation of the current Intel SGX's architecture (i.e., the limited trusted memory), we propose two data storage modes to efficiently implement aggregation algorithms efficiently in SGX. Moreover, to balance the efficiency and performance of aggregation, we propose a sampling-based method to efficiently detect Byzantine failures without degrading the global model's performance. We implement and evaluate SEAR in a LAN environment, and the experiment results show that SEAR is computationally efficient and robust to Byzantine adversaries. Compared to the previous practical secure aggregation framework, SEAR improves aggregation efficiency by 4-6 times while supporting Byzantine resilience at the same time.

Collaborative learning allows multiple clients to train a joint model without sharing their data with each other. Each client performs training locally and then submits the model updates to a central server for aggregation. Since the server has no visibility into the process of generating the updates, collaborative learning is vulnerable to poisoning attacks where a malicious client can generate a poisoned update to introduce backdoor functionality to the joint model. The existing solutions for detecting poisoned updates, however, fail to defend against the recently proposed attacks, especially in the non-IID (independent and identically distributed) setting. In this article, we present a novel defense scheme to detect anomalous updates in both IID and non-IID settings. Our key idea is to realize client-side cross-validation, where each update is evaluated over other clients' local data. The server will adjust the weights of the updates based on the evaluation results when performing aggregation. To adapt to the unbalanced distribution of data in the non-IID setting, a dynamic client allocation mechanism is designed to assign detection tasks to the most suitable clients. During the detection process, we also protect the client-level privacy to prevent malicious clients from knowing the participations of other clients, by integrating differential privacy with our design without degrading the detection performance. Our experimental evaluations on three real-world datasets show that our scheme is significantly robust to two representative poisoning attacks.

Machine Learning as a Service (MLaaS) allows clients with limited resources to outsource their expensive ML tasks to powerful servers. Despite the huge benefits, current MLaaS solutions still lack strong assurances on: 1) service correctness (i.e., whether the MLaaS works as expected); 2) trustworthy accounting (i.e., whether the bill for the MLaaS resource consumption is correctly accounted); 3) fair payment (i.e., whether a client gets the entire MLaaS result before making the payment). Without these assurances, unfaithful service providers can return improperly-executed ML task results or partially-trained ML models while asking for over-claimed rewards. Moreover, it is hard to argue for wide adoption of MLaaS to both the client and the service provider, especially in the open market without a trusted third party. In this article, we present VeriML, a novel and efficient framework to bring integrity assurances and fair payments to MLaaS. With VeriML, clients can be assured that ML tasks are correctly executed on an untrusted server, and the resource consumption claimed by the service provider equals to the actual workload. We strategically use succinct non-interactive arguments of knowledge (SNARK) on randomly-selected iterations during the ML training phase for efficiency with tunable probabilistic assurance. We also develop multiple ML-specific optimizations to the arithmetic circuit required by SNARK. Our system implements six common algorithms: linear regression, logistic regression, neural network, support vector machine, K-means and decision tree. The experimental results have validated the practical performance of VeriML.

Nowadays, it usually requires a massive amount of labeled data to train a deep neural network. When no labeled data is available in some application scenarios, domain adaption can be employed to transfer a learner from one or more source domains with labeled data to a target domain with unlabeled data. However, due to the exposure of the trained model to the target domain, the user privacy may potentially be compromised. Nevertheless, the private information may be encoded into the representations in different stages of the deep neural networks, i.e., hierarchical convolutional feature maps, which poses a great challenge for a full-fledged privacy protection. In this paper, we propose a novel differentially private domain adaptation framework called DPDA to achieve domain adaptation with privacy assurance. Specifically, we perform domain adaptation in an adversarial-learning manner and embed the differentially private design into specific layers and learning processes. Although applying differential privacy techniques directly will undermine the performance of deep neural networks, DPDA can increase the classification accuracy for the unlabeled target data compared to the prior arts. We conduct extensive experiments on standard benchmark datasets, and the results show that our proposed DPDA can indeed achieve high accuracy in many domain adaptation tasks with only a modest privacy loss.

With powerful parallel computing GPUs and massive user data, neural-network-based deep learning can well exert its strong power in problem modeling and solving, and has archived great success in many applications such as image classification, speech recognition and machine translation etc. While deep learning has been increasingly popular, the problem of privacy leakage becomes more and more urgent. Given the fact that the training data may contain highly sensitive information, e.g., personal medical records, directly sharing them among the users (i.e., participants) or centrally storing them in one single location may pose a considerable threat to user privacy. In this paper, we present a practical privacy-preserving collaborative deep learning system that allows users to cooperatively build a collective deep learning model with data of all participants, without direct data sharing and central data storage. In our system, each participant trains a local model with their own data and only shares model parameters with the others. To further avoid potential privacy leakage from sharing model parameters, we use functional mechanism to perturb the objective function of the neural network in the training process to achieve ε-differential privacy. In particular, for the first time, we consider the existence of unreliable participants, i.e., the participants with low-quality data, and propose a solution to reduce the impact of these participants while protecting their privacy. We evaluate the performance of our system on two well-known real-world datasets for regression and classification tasks. The results demonstrate that the proposed system is robust against unreliable participants, and achieves high accuracy close to the model trained in a traditional centralized manner while ensuring rigorous privacy protection.

Data mining has heralded the major breakthrough in data analysis, serving as a “super cruncher” to discover hidden information and valuable knowledge in big data systems. For many applications, the collection of big data usually involves various parties who are interested in pooling their private data sets together to jointly train machine-learning models that yield more accurate prediction results. However, data owners may not be willing to disclose their own data due to privacy concerns, making it imperative to provide privacy guarantee in collaborative data mining over distributed data sets. In this paper, we focus on tree-based data mining. To begin with, we design novel privacy-preserving schemes for two most common tasks: regression and binary classification, where individual data owners can perform training locally in a differentially private manner. Then, for the first time, we design and implement a privacy-preserving system for gradient boosting decision tree (GBDT), where different regression trees trained by multiple data owners can be securely aggregated into an ensemble. We conduct extensive experiments to evaluate the performance of our system on multiple real-world data sets. The results demonstrate that our system can provide a strong privacy protection for individual data owners while maintaining the prediction accuracy of the original trained model.