EDBT 2026 Demo / reviewers in the wild / expert
Nathalie Baracaldo
dblp:87/10087
· DBLP profile ↗
31ranked-venue papers
7as first author
21since 2021 · last 2026
0000-0001-9469-045XORCID · verified
Domains — the database's venue-derived domains; a paper can count in several
Artificial intelligence and machine learning · 12 · 12 since 2021Security and privacy · 10 · 7 first-author · 4 since 2021Databases, data management, data science and information retrieval · 5 · 4 since 2021Applied, interdisciplinary, general and emerging computing · 5 · 4 since 2021Systems, architecture and hardware · 2 · 1 since 2021Human-computer interaction and ubiquitous computing · 2 · 1 since 2021Graphics, computer vision, multimedia, augmented reality and games · 1 · 1 since 2021
| Year | Publication | Venue | Position |
|---|---|---|---|
| 2026 | In-Context Probing for Membership Inference in Fine-Tuned Language Models
Zhexi Lu, Hongliang Chi, Nathalie Baracaldo, Swanand Kadhe, Yuseok Jeon, Lei Yu 0002 |
NDSS | 3 |
| 2025 | Membership Inference Attacks as Privacy Tools: Reliability, Disparity and EnsembleabstractMembership inference attacks (MIAs) pose a significant threat to the privacy of machine learning models and are widely used as tools for privacy assessment, auditing, and machine unlearning. While prior MIA research has primarily focused on performance metrics such as AUC, accuracy, and TPR@low FPR—either by developing new methods to enhance these metrics or using them to evaluate privacy solutions—we found that it overlooks the disparities among different attacks. These disparities, both between distinct attack methods and between multiple instantiations of the same method, have crucial implications for the reliability and completeness of MIAs as privacy evaluation tools. In this paper, we systematically investigate these disparities through a novel framework based on coverage and stability analysis. Extensive experiments reveal significant disparities in MIAs, their potential causes, and their broader implications for privacy evaluation. To address these challenges, we propose an ensemble framework with three distinct strategies to harness the strengths of state-of-the-art MIAs while accounting for their disparities. This framework not only enables the construction of more powerful attacks but also provides a more robust and comprehensive methodology for privacy evaluation. Yuetian Chen, Nathalie Baracaldo, Swanand Kadhe, Lei Yu 0002 |
CCS | 4 |
| 2025 | Reasoning Model Unlearning: Forgetting Traces, Not Just Answers, While Preserving Reasoning SkillsabstractChangsheng Wang, Chongyu Fan, Yihua Zhang, Jinghan Jia, Dennis Wei, Parikshit Ram, Nathalie Baracaldo, Sijia Liu. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. Changsheng Wang, Chongyu Fan, Jinghan Jia, Dennis Wei, Parikshit Ram, Nathalie Baracaldo, Sijia Liu 0001 |
EMNLP | 7 |
| 2025 | MAP: Multi-Human-Value Alignment PaletteabstractEnsuring that generative AI systems align with human values is essential but challenging, especially when considering multiple human values and their potential trade-offs. Since human values can be personalized and dynamically change over time, the desirable levels of value alignment vary across different ethnic groups, industry sectors, and user cohorts. Within existing frameworks, it is hard to define human values and align AI systems accordingly across different directions simultaneously, such as harmlessness, helpfulness, and positiveness. To address this, we develop a novel, first-principle approach called Multi-Human-Value Alignment Palette (MAP), which navigates the alignment across multiple human values in a structured and reliable way. MAP formulates the alignment problem as an optimization task with user-defined constraints, which define human value targets. It can be efficiently solved via a primal-dual approach, which determines whether a user-defined alignment target is achievable and how to achieve it. We conduct a detailed theoretical analysis of MAP by quantifying the trade-offs between values, the sensitivity to constraints, the fundamental connection between multi-value alignment and sequential alignment, and proving that linear weighted rewards are sufficient for multi-value alignment. Extensive experiments demonstrate MAP's ability to align multiple values in a principled manner while delivering strong empirical performance across various tasks. Qi Le, Enmao Diao, Yi Zhou 0015, Nathalie Baracaldo, Jie Ding 0002, Ali Anwar 0001 |
ICLR | 6 |
| 2025 | Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-TuningabstractMachine unlearning presents a promising approach to mitigating privacy and safety concerns in large language models (LLMs) by enabling the selective removal of targeted data or knowledge while preserving model utility. However, existing unlearning methods remain over-sensitive to downstream fine-tuning, which can rapidly recover what is supposed to be unlearned information even when the fine-tuning task is entirely unrelated to the unlearning objective. To enhance robustness, we introduce the concept of ‘invariance’ into unlearning for the first time from the perspective of invariant risk minimization (IRM), a principle for environment-agnostic training. By leveraging IRM, we develop a new invariance-regularized LLM unlearning framework, termed invariant LLM unlearning (ILU). We show that the proposed invariance regularization, even using only a single fine-tuning dataset during ILU training, can enable unlearning robustness to generalize effectively across diverse and new fine-tuning tasks at test time. A task vector analysis is also provided to further elucidate the rationale behind ILU’s effectiveness. Extensive experiments on the WMDP benchmark, which focuses on removing an LLM’s hazardous knowledge generation capabilities, reveal that ILU significantly outperforms state-of-the-art unlearning methods, including negative preference optimization (NPO) and representation misdirection for unlearning (RMU). Notably, ILU achieves superior unlearning robustness across diverse downstream fine-tuning scenarios (e.g., math, paraphrase detection, and sentiment analysis) while preserving the fine-tuning performance. Changsheng Wang, Jinghan Jia, Parikshit Ram, Dennis Wei, Yuguang Yao, Soumyadeep Pal, Nathalie Baracaldo, Sijia Liu 0001 |
ICML | 8 |
| 2025 | FedKDD 2025: The 2025 International Joint Workshop on Federated Learning for Data Mining and Graph AnalyticsabstractDeep Learning has facilitated various high-stakes applications such as crime detection, urban planning, drug discovery, and healthcare. Its continuous success hinges on learning from massive data in miscellaneous sources, ranging from data with independent distributions to graph-structured data capturing intricate inter-sample relationships. Scaling up the data access requires global collaboration from distributed data owners. Yet, centralizing all data sources to an untrustworthy centralized server will put users' data at risk of privacy leakage or regulation violation. Federated Learning (FL) is a de facto decentralized learning framework that enables knowledge aggregation from distributed users without exposing private data. Though promising advances are witnessed for FL, new challenges are emerging when integrating FL with the rising needs and opportunities in data mining, graph analytics, foundation models, generative AI, and new interdisciplinary applications in science. By hosting this workshop, we aim to attract a broad range of audiences, including researchers and practitioners from academia and industry interested in the emergent challenges in FL. As an effort to advance the fundamental development of FL, this workshop will encourage ideas exchange on the trustworthiness, scalability, and robustness of distributed data mining and graph analytics and their emergent challenges. Carl Yang 0001, Guancheng Wan, Zhuangdi Zhu, Zheng Xu 0002, Junyuan Hong, Nathalie Baracaldo, Neil Shah, Amir Salman Avestimehr |
KDD (2) | 6 |
| 2025 | Towards a Re-evaluation of Data Forging Attacks in Practice
Mohamed Suliman 0002, Anisa Halimi, Swanand Kadhe, Nathalie Baracaldo, Douglas J. Leith |
USENIX Security Symposium | 4 |
| 2024 | Enhancing In-context Learning via Linear Probe CalibrationabstractIn-context learning (ICL) is a new paradigm for natural language processing that utilizes Generative Pre-trained Transformer (GPT)-like models. This approach uses prompts that include in-context demonstrations to generate the corresponding output for a new query input. However, applying ICL in real cases does not scale with the number of samples, and lacks robustness to different prompt templates and demonstration permutations. In this paper, we first show that GPT-like models using ICL result in unreliable predictions based on a new metric based on Shannon entropy. Then, to solve this problem, we propose a new technique called the Linear Probe Calibration (LinC), a method that calibrates the model’s output probabilities, resulting in reliable predictions and improved performance, while requiring only minimal additional samples (as few as five labeled data samples). LinC significantly enhances the ICL test performance of GPT models on various benchmark datasets, with an average improvement of up to 21%, and up to a 50% improvement in some cases, and significantly boosts the performance of PEFT methods, especially in the low resource regime. Moreover, LinC achieves lower expected calibration error, and is highly robust to varying label proportions, prompt templates, and demonstration permutations. Momin Abbas, Yi Zhou 0015, Parikshit Ram, Nathalie Baracaldo, Horst Samulowitz, Theodoros Salonidis, Tianyi Chen 0002 |
AISTATS | 4 |
| 2024 | FedKDD: International Joint Workshop on Federated Learning for Data Mining and Graph AnalyticsabstractDeep Learning has facilitated various high-stakes applications such as crime detection, urban planning, drug discovery, and healthcare. Its continuous success hinges on learning from massive data in miscellaneous sources, ranging from data with independent distributions to graph-structured data capturing intricate inter-sample relationships. Scaling up the data access requires global collaboration from distributed data owners. Yet, centralizing all data sources to an untrustworthy centralized server will put users' data at risk of privacy leakage or regulation violation. Federated Learning (FL) is a de facto decentralized learning framework that enables knowledge aggregation from distributed users without exposing private data. Though promising advances are witnessed for FL, new challenges are emerging when integrating FL with the rising needs and opportunities in data mining, graph analytics, foundation models, generative AI, and new interdisciplinary applications in science. By hosting this workshop, we aim to attract a broad range of audiences, including researchers and practitioners from academia and industry interested in the emergent challenges in FL. As an effort to advance the fundamental development of FL, this workshop will encourage ideas exchange on the trustworthiness, scalability, and robustness of distributed data mining and graph analytics and their emergent challenges. Junyuan Hong, Carl Yang 0001, Zhuangdi Zhu, Zheng Xu 0002, Nathalie Baracaldo, Neil Shah, Amir Salman Avestimehr |
KDD | 5 |
| 2024 | WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language ModelsabstractThe need for effective unlearning mechanisms in large language models (LLMs) is increasingly urgent, driven by the necessity to adhere to data regulations and foster ethical generative AI practices. LLM unlearning is designed to reduce the impact of undesirable data influences and associated model capabilities without diminishing the utility of the model if unrelated to the information being forgotten. Despite growing interest, much of the existing research has focused on varied unlearning method designs to boost effectiveness and efficiency. However, the inherent relationship between model weights and LLM unlearning has not been extensively examined. In this paper, we systematically explore how model weights interact with unlearning processes in LLMs and we design the weight attribution-guided LLM unlearning method, WAGLE, which unveils the interconnections between 'influence' of weights and 'influence' of data to forget and retain in LLM generation. By strategically guiding the LLM unlearning across different types of unlearning methods and tasks, WAGLE can erase the undesired content, while maintaining the performance of the original tasks. We refer to the weight attribution-guided LLM unlearning method as WAGLE, which unveils the interconnections between 'influence' of weights and 'influence' of data to forget and retain in LLM generation. Our extensive experiments show that WAGLE boosts unlearning performance across a range of LLM unlearning methods such as gradient difference and (negative) preference optimization, applications such as fictitious unlearning (TOFU benchmark), malicious use prevention (WMDP benchmark), and copyrighted information removal, and models including Zephyr-7b-beta and Llama2-7b. To the best of our knowledge, our work offers the first principled method for attributing and pinpointing the influential weights in enhancing LLM unlearning. It stands in contrast to previous methods that lack weight attribution and simpler weight attribution techniques. Jinghan Jia, Jiancheng Liu, Parikshit Ram, Nathalie Baracaldo, Sijia Liu 0001 |
NeurIPS | 5 |
| 2023 | HDFL: A Heterogeneity and Client Dropout-Aware Federated Learning FrameworkabstractCross-device Federated Learning (FL) enables training machine learning (ML) models on private data that is heterogeneously distributed over many IoT end devices without violating privacy requirements. Clients typically vary significantly in data quality, hardware resources and stability, which results in challenges such as increased training times, higher resource costs, sub-par model performance and biased training. Existing works tend to address each of these challenges in isolation, but overlook how they might impact each other holistically. We perform a first of its kind characterization study that empirically demonstrates how these properties interact with each other to impact important performance metrics such as model error, fairness, resource cost and training time. We then propose a method called HDFL based on our observations, which is the first framework to our knowledge that comprehensively considers the multiple aforementioned important challenges of practical FL systems. We implement HDFL on a real distributed system and evaluate it on multiple benchmark datasets which show that HDFL achieves better Pareto frontier compared to both the state-of-the-practice and state-of-the-art systems with up to 4-10% better model accuracy, 33% improved good-intent fairness, 63% lower cost, and 17% faster training time. Syed Zawad, Ali Anwar 0001, Yi Zhou 0015, Nathalie Baracaldo, Feng Yan 0001 |
CCGrid | 4 |
| 2023 | Single-shot General Hyper-parameter Optimization for Federated Learning
Yi Zhou 0015, Parikshit Ram, Theodoros Salonidis, Nathalie Baracaldo, Horst Samulowitz, Heiko Ludwig |
ICLR | 4 |
| 2023 | LESS-VFL: Communication-Efficient Feature Selection for Vertical Federated LearningabstractWe propose LESS-VFL, a communication-efficient feature selection method for distributed systems with vertically partitioned data. We consider a system of a server and several parties with local datasets that share a sample ID space but have different feature sets. The parties wish to collaboratively train a model for a prediction task. As part of the training, the parties wish to remove unimportant features in the system to improve generalization, efficiency, and explainability. In LESS-VFL, after a short pre-training period, the server optimizes its part of the global model to determine the relevant outputs from party models. This information is shared with the parties to then allow local feature selection without communication. We analytically prove that LESS-VFL removes spurious features from model training. We provide extensive empirical evidence that LESS-VFL can achieve high accuracy and remove spurious features at a fraction of the communication cost of other feature selection approaches. Timothy Castiglia, Yi Zhou 0015, Shiqiang Wang 0001, Swanand Kadhe, Nathalie Baracaldo, Stacy Patterson |
ICML | 5 |
| 2022 | TIFF: Tokenized Incentive for Federated LearningabstractIn federated learning (FL), clients collectively train a global machine learning model with their own local data. Without sharing sensitive raw data, each client in FL only sends updated weights to consider privacy and security concerns. Most of existing FL works focus mainly on improving model accuracy and training time, but only a few works focus on FL incentive mechanisms. To build a high performance model after FL training, clients need to provide high quality and large amounts of data. However, in real FL scenarios, high-quality clients are reluctant to participate in FL process without reasonable compensation, because clients are self-interested and other clients can be business competitors. Even participation incurs some cost for contributing to the FL model with their local dataset. To address this problem, we propose TIFF, a novel tokenized incentive mechanism, where tokens are used as a means of paying for the services of providing participants and the training infrastructure. Without payment delays, participation can be monetized as both providers and consumers, which promotes continued long-term participation of high-quality data parties. Additionally, paid tokens are reimbursed to each client as consumers according to our newly proposed metrics (such as token reduction ratio and utility improvement ratio), which keeps clients engaged in FL process as consumers. To measure data quality, accuracy is calculated in training without additional overheads. We leverage historical accuracy records and random exploration to select high-utility participants and to prevent overfitting. Results show that TIFF provides more tokens to normal providers by up to 6.9% and less tokens to malicious providers by up to 18.1%, achieving improvement of the final model accuracy by up to 7.4%, compared to the default approach. Jingoo Han, Ahmad Khan 0001, Syed Zawad, Ali Anwar 0001, Nathalie Baracaldo, Yi Zhou 0015, Feng Yan 0001, Ali Raza Butt |
CLOUD | 5 |
| 2022 | DeTrust-FL: Privacy-Preserving Federated Learning in Decentralized Trust SettingabstractFederated learning has emerged as a privacy-preserving machine learning approach where multiple parties can train a single model without sharing their raw training data. Federated learning typically requires the utilization of multi-party computation techniques to provide strong privacy guarantees by ensuring that an untrusted or curious aggregator cannot obtain isolated replies from parties involved in the training process, thereby preventing potential inference attacks. Until recently, it was thought that some of these secure aggregation techniques were sufficient to fully protect against inference attacks coming from a curious aggregator. However, recent research has demonstrated that a curious aggregator can successfully launch a disaggregation attack to learn information about model updates of a target party. This paper presents DeTrust-FL, an efficient privacy-preserving federated learning framework for addressing the lack of transparency that enables isolation attacks, such as disaggregation attacks, during secure aggregation by assuring that parties’ model updates are included in the aggregated model in a private and secure manner. DeTrust-FL proposes a decentralized trust consensus mechanism and incorporates a recently proposed decentralized functional encryption scheme in which all parties agree on a participation matrix before collaboratively generating decryption key fragments, thereby gaining control and trust over the secure aggregation process in a decentralized setting. Our experimental evaluation demonstrates that DeTrust-FL outperforms state-of-the-art FE-based secure multi-party aggregation solutions in terms of training time and reduces the volume of data transferred. In contrast to existing approaches, this is achieved without creating any trust dependency on external trusted entities. Runhua Xu, Nathalie Baracaldo, Yi Zhou 0015, Ali Anwar 0001, Swanand Kadhe, Heiko Ludwig |
CLOUD | 2 |
| 2022 | Heterogeneity-Aware Adaptive Federated Learning SchedulingabstractFederated learning (FL) is becoming an important distributed machine learning approach that considers privacy and security concerns while training a shared model across various clients with localized data. One of the key challenges in FL is heterogeneity in both hardware resources and local datasets due to the nature of incorporating diverse clients. Given the resource heterogeneity, the availability of participating clients is not stable over time and their resource usage patterns become dynamic. This leads to resource wastage and straggler issues. Additional challenges are introduced due to data heterogeneity, causing model biasness and poor model performance. However, most existing FL systems are not well suited to heterogeneous environments because those approaches are not adaptive to various and dynamically changing resource usage patterns and accuracy trends during training process. To this end, we propose a heterogeneity-aware scheduling which is adaptive to the accuracy trends and various resource usage patterns. Our proposed scheduling provides different scheduling knobs for achieving different goals such as resource-efficient fast training, resource fairness, accuracy fairness, and high model performance. To the best of our knowledge, this is the first effort to mitigate effects of resource and data heterogeneity while providing adaptive scheduling based on dynamically changing resource usage patterns and accuracy trends. Jingoo Han, Ahmad Khan 0001, Syed Zawad, Ali Anwar 0001, Nathalie Baracaldo, Yi Zhou 0015, Feng Yan 0001, Ali Raza Butt |
IEEE Big Data | 5 |
| 2022 | The 1st International Workshop on Federated Learning with Graph Data (FedGraph)abstractThe field of graph data mining, one of the most important AI research areas, has been revolutionized by graph neural networks (GNNs), which benefit from training on real-world graph data with millions to billions of nodes and links. Unfortunately, the training data and process of GNNs involving graphs beyond millions of nodes are extremely costly on a centralized server, if not impossible. Moreover, due to the increasing concerns about data privacy, emerging data from realistic applications are naturally fragmented, forming distributed private graphs of multiple ''data silos", among which direct transferring of data is forbidden. The nascent field of federated learning (FL), which aims to enable individual clients to jointly train their models while keeping their local data decentralized and completely private, is a promising paradigm for large-scale distributed and private training of GNNs. øurs aims to bring together researchers from different backgrounds with a common interest in how to extend current FL algorithms to operate with graph data models such as GNNs. FL is an extremely hot topic of large commercial interest and has been intensively explored for machine learning with visual and textual data. The exploration from graph mining researchers and industrial practitioners is timely catching up just recently. There are many unexplored challenges and opportunities, which urges the establishment of an organized and open community to collaboratively advance the science behind it. The prospective participants of this workshop will include researchers and practitioners from both graph mining and federated learning communities, whose interests include, but are not limited to: graph analysis and mining, heterogeneous network modeling, complex data mining, large-scale machine learning, distributed systems, optimization, meta-learning, reinforcement learning, privacy, robustness, explainability, fairness, ethics, and trustworthiness. Carl Yang 0001, Xiaoxiao Li 0001, Nathalie Baracaldo, Neil Shah, Chaoyang He 0001, Lingjuan Lyu, Lichao Sun 0001, Amir Salman Avestimehr |
CIKM | 3 |
| 2022 | Keynote Talk - Federated Learning: The Hype, State-of-the-Art and Open ChallengesabstractThe popularity of machine learning models has dramatically increased in a large variety of applications that affect people's daily lifes, including product recommendations, healthcare predictions and critical applications. This wide availability has at the same time raised questions about the trustworthiness, security, and privacy implications of using these systems. While novel technologies and methodologies have been emerging to protect the privacy and security of AI Systems, there are still open challenges that need to be addressed by the research community. Over the past years, my research has focused on the creation of defenses to protect the machine learning pipeline and the design of privacy-aware methodologies to enable the training of accurate machine learning models without transmitting the data to a central place. In this talk, I will focus on data privacy covering a game-changing paradigm known as federated learning [4], which to some extend addresses privacy concerns and regulations that prevent the free transmission and sharing of information. Federated learning is a technology that enables multiple participants owning private data to collaboratively train a single machine learning model while maintaining their training data locally. This is in sharp contrast to traditional machine learning where all data needs to be in a central place. Some argue that federated learning is a privacy-by-design technology given that it does not require data to be transmitted to a central place. However, there are still privacy risks that are relevant in some scenarios. Novel inference attacks that take advantage of the federated learning process have been demonstrated in the literature, resulting in a variety of defenses that aim to reduce these risks. I will present some of these attacks and several cryptographical and differential privacy techniques to deter them, including [5,7,8]. The plethora of defenses is particularly interesting given their diverse threat models and the divergent set of privacy requirements they address. In this talk I will demystify them. I will also explain some challenges related to manipulation attacks [6,9] and machine learning fairness [1] in the context of federated learning. Finally, I will touch upon transparency issues and how to enable accountability for regulated industries [2,3] and vertical federated learning [7]. This talk will go through the security and privacy challenges and solutions in federated learning systems. Nathalie Baracaldo |
SACMAT | 1 |
| 2021 | LEGATO: A LayerwisE Gradient AggregaTiOn Algorithm for Mitigating Byzantine Attacks in Federated LearningabstractFederated learning has arisen as a mechanism to allow multiple participants to collaboratively train a model without sharing their data. In these settings, participants (workers) may not trust each other fully; for instance, a set of competitors may collaboratively train a machine learning model to detect fraud. The workers provide local gradients that a central server uses to update a global model. This global model can be corrupted when Byzantine workers send malicious gradients, which necessitates robust methods for aggregating gradients that mitigate the adverse effects of Byzantine inputs. Existing robust aggregation algorithms are often computationally expensive and only effective under strict assumptions. In this paper, we introduce LayerwisE Gradient AggregatTiOn (LEGATO), an aggregation algorithm that is, by contrast, scalable and generalizable. Informed by a study of layer-specific responses of gradients to Byzantine attacks, LEGATO employs a dynamic gradient reweighing scheme that is novel in its treatment of gradients based on layer-specific robustness. We show that LEGATO is more computationally efficient than multiple state-of-the-art techniques and more generally robust across a variety of attack settings in practice. We also demonstrate LEGATO's benefits for gradient descent convergence in the absence of an attack. Kamala Varma, Yi Zhou 0015, Nathalie Baracaldo, Ali Anwar 0001 |
CLOUD | 3 |
| 2021 | Curse or Redemption? How Data Heterogeneity Affects the Robustness of Federated LearningabstractData heterogeneity has been identified as one of the key features in federated learning but often overlooked in the lens of robustness to adversarial attacks. This paper focuses on characterizing and understanding its impact on backdooring attacks in federated learning through comprehensive experiments using synthetic and the LEAF benchmarks. The initial impression driven by our experimental results suggests that data heterogeneity is the dominant factor in the effectiveness of attacks and it may be a redemption for defending against backdooring as it makes the attack less efficient, more challenging to design effective attack strategies, and the attack result also becomes less predictable. However, with further investigations, we found data heterogeneity is more of a curse than a redemption as the attack effectiveness can be significantly boosted by simply adjusting the client-side backdooring timing. More importantly, data heterogeneity may result in overfitting at the local training of benign clients, which can be utilized by attackers to disguise themselves and fool skewed-feature based defenses. In addition, effective attack strategies can be made by adjusting attack data distribution. Finally, we discuss the potential directions of defending the curses brought by data heterogeneity. The results and lessons learned from our extensive experiments and analysis offer new insights for designing robust federated learning methods and systems. Syed Zawad, Ali Anwar 0001, Yi Zhou 0015, Nathalie Baracaldo, Yuan Tian 0001, Feng Yan 0001 |
AAAI | 6 |
| 2021 | The Design and Development of a Game to Study Backdoor Poisoning Attacks: The Backdoor GameabstractAI Security researchers have identified a new way crowdsourced data can be intentionally compromised. Backdoor attacks are a process through which an adversary creates a vulnerability in a machine learning model by ?poisoning?’ the training set by selectively mislabelling images containing a backdoor object. The model continues to perform well on standard testing data but misclassifies on the inputs that contain the backdoor chosen by the adversary. In this paper, we present the design and development of the Backdoor Game, the first game in which users can interact with different poisoned classifiers and upload their own images containing backdoor objects in an engaging way. We conduct semi-structured interviews with eight different participants who interacted with a first version of the Backdoor Game and deploy the game to Mechanical Turk users (N=68) to demonstrate how users interacted with the backdoor objects. We present results including novel types of interactions that emerged as a result of game play and design recommendations for the improvement of the system. The combined design, development and deployment of our system can help AI Security researchers to study this emerging concept, from determining the effectiveness of different backdoor objects to help compiling a collection of diverse and unique backdoor objects from the public, increasing the safety of future AI systems. Zahra Ashktorab, Casey Dugan, Aabhas Sharma, Dustin Ramsey Torres, Ingrid Lange, Benjamin Hoover, Heiko Ludwig, Bryant Chen, Nathalie Baracaldo, Werner Geyer |
IUI | 10 |
| 2020 | TiFL: A Tier-based Federated Learning SystemabstractFederated Learning (FL) enables learning a shared model acrossmany clients without violating the privacy requirements. One of the key attributes in FL is the heterogeneity that exists in both resource and data due to the differences in computation and communication capacity, as well as the quantity and content of data among different clients. We conduct a case study to show that heterogeneity in resource and data has a significant impact on training time and model accuracy in conventional FL systems. To this end, we propose TiFL, a Tier-based Federated Learning System, which divides clients into tiers based on their training performance and selects clients from the same tier in each training round to mitigate the straggler problem caused by heterogeneity in resource anddata quantity. To further tame the heterogeneity caused by non-IID (Independent and Identical Distribution) data and resources, TiFL employs an adaptive tier selection approach to update the tiering on-the-fly based on the observed training performance and accuracy. We prototype TiFL in a FL testbed following Google's FL architecture and evaluate it using the state-of-the-art FL benchmarks. Experimental evaluation shows that TiFL outperforms the conventional FL in various heterogeneous conditions. With the proposed adaptive tier selection policy, we demonstrate that TiFL achieves much faster training performance while achieving the same or better test accuracy across the board. Syed Zawad, Stacey Truex, Ali Anwar 0001, Nathalie Baracaldo, Yi Zhou 0015, Heiko Ludwig, Feng Yan 0001, Yue Cheng 0001 |
HPDC | 6 |
| 2019 | G-SIR: An Insider Attack Resilient Geo-Social Access Control FrameworkabstractInsider attacks are among the most dangerous and costly attacks to organizations. These attacks are carried out by individuals who are legitimately authorized to access the system. Preventing insider attacks is a daunting task. The recent proliferation of social media and mobile devices offer new opportunities to collect geo-social information that can help in detecting and deterring insider attacks. In particular, such geo-social information allows us to better understand the context and behavior of users. In this paper, we propose a Geo-Social Insider Threat Resilient Access Control Framework (G-SIR) to deter insider threats by including current and historic geo-social information as part of the access control decision process. We include policy constraints to manage the risks of colluding communities, proximity threats, and suspicious users while leveraging the presence of users around the requester to make an access decision. By examining users' geo-social behavior, we can detect those users whose access behavior deviates from the expected patterns; such suspicious behaviors can point to potential insider attackers who may deliberately or inadvertently carry out malicious activities. We use such information to establish how trustworthy a user is before granting access. We evaluate the G-SIR framework through extensive simulations and our results show that the proposed approach is efficient, scalable and effective. Nathalie Baracaldo, Balaji Palanisamy, James B. D. Joshi |
IEEE Trans. Dependable Secur. Comput. | 1 |
| 2018 | Complex Collaborative Physical Process Management: A Position on the Trinity of BPM, IoT and DA
Paul Grefen, Heiko Ludwig, Samir Tata, Remco M. Dijkman, Nathalie Baracaldo, Anna Wilbik, Tim D'Hondt |
PRO-VE | 5 |
| 2017 | rSLA: An Approach for Managing Service Level Agreements in Cloud EnvironmentsabstractCloud providers offer services at different levels of abstraction from infrastructure to applications. The quality of Cloud services is a key determinant of the overall service level a provider offers to its customers. Service Level Agreements (SLAs) are (1) crucial for Cloud customers to ensure that promised levels of services are met, (2) an important sales instrument and (3) a differentiating factor for providers. Cloud providers and services are often selected more dynamically than in traditional IT services, and as a result, SLAs need to be set up and their monitoring implemented to match the same speed. In this context, managing SLAs is complex: different Cloud providers expose different management interfaces and SLA metrics differ from one provider to another. In this paper, we will analyze how IT service quality has been defined and managed over time, discuss how to manage SLAs in today’s multi-layer, multi-sourced Cloud environments, and what to expect going forward. A particular focus will be made on the rSLA framework that enables fast setup of SLA monitoring in dynamic and heterogeneous Cloud environments. The rSLA framework is made up of three main components: the rSLA language to formally represent SLAs, the rSLA Service, which interprets the SLAs and implements the behavior specified in them, and a set of Xlets-lightweight, dynamically bound adapters to monitoring and controlling interfaces. rSLA has been tested in the context of a real pilot and found to reduce the client on-boarding process from months to weeks. Mohamed Mohamed 0001, Obinna Anya, Samir Tata, NagaPramod Mandagere, Nathalie Baracaldo, Heiko Ludwig |
Int. J. Cooperative Inf. Syst. | 5 |
| 2016 | Managing Service Quality at the Platform and Application Levels with rSLaabstractManaging service quality in heterogeneous Cloud environments is complex: different providers expose different management interfaces for monitoring and configuration actions that can occur at the infrastructure, platform and application levels. To manage Service Level Agreements (SLAs) in this context, we have developed the rSLA Manager for enabling fast setup of SLA monitoring in dynamic, heterogeneous Cloud environments. In this paper, we demonstrate the use of the rSLA Manager to manage the response time behavior of an image scaling service on the IBM Blue mix platform. We show how the system enables SLA violation reduction by facilitating platform-level adaptation (e.g., adding application instances) and application-level behavior configuration (e.g., changing image scaling quality) based on different service level objectives as well as current application configuration and workload. Samir Tata, Mohamed Mohamed 0001, Obinna Anya, Takashi Sakairi, NagaPramod Mandagere, Heiko Ludwig, Nathalie Baracaldo |
WETICE | 7 |
| 2014 | Geo-Social-RBAC: A Location-Based Socially Aware Access Control Framework
Nathalie Baracaldo, Balaji Palanisamy, James B. D. Joshi |
NSS | 1 |
| 2013 | Beyond accountability: using obligations to reduce risk exposure and deter insider attacksabstractRecently, the importance of including obligations as part of access control systems for privilege management, for example, in healthcare information systems, has been well recognized. In an access control system, an a posteriori obligation states which actions need to be performed by a user after he has accessed a resource. There is no guarantee that a user will fulfill a posteriori obligations. Not fulfilling these obligations may incur financial loss, or loss of goodwill and productivity to the organization. In this paper, we propose a trust-and-obligation based framework that reduces the risk exposure of an organization associated with a posteriori obligations. We propose a methodology to assign trust values to users to indicate how trustworthy they are with regards to fulfilling their obligations. When access requests that trigger a posteriori obligations are evaluated, the requesting users' trust values and the criticality of the associated obligations are used. Our framework detects and mitigates insider attacks and unintentional damages that may result from violating a posteriori obligations. Our framework also provides mechanisms to determine misconfigurations of obligation policies. We evaluate our framework through simulations and demonstrate its effectiveness. Nathalie Baracaldo, James B. D. Joshi |
SACMAT | 1 |
| 2013 | An adaptive risk management and access control framework to mitigate insider threats
Nathalie Baracaldo, James B. D. Joshi |
Comput. Secur. | 1 |
| 2012 | A trust-and-risk aware RBAC framework: tackling insider threatabstractInsider Attacks are one of the most dangerous threats organizations face today. An insider attack occurs when a person authorized to perform certain actions in an organization decides to abuse the trust, and harm the organization. These attacks may negatively impact the reputation of the organization, its productivity, and may produce losses in revenue and clients. Avoiding insider attacks is a daunting task. While it is necessary to provide privileges to employees so they can perform their jobs efficiently, providing too many privileges may backfire when users accidentally or intentionally abuse their privileges. Hence, finding a middle ground, where the necessary privileges are provided and malicious usage are avoided, is necessary. In this paper, we propose a framework that extends the role-based access control (RBAC) model by incorporating a risk assessment process, and the trust the system has on its users. Our framework adapts to suspicious changes in users' behavior by removing privileges when users' trust falls below a certain threshold. This threshold is computed based on a risk assessment process that includes the risk due to inference of unauthorized information. We use a Coloured-Petri net to detect inferences. We also redefine the existing role activation problem, and propose an algorithm that reduces the risk exposure. We present experimental evaluation to validate our work. Nathalie Baracaldo, James B. D. Joshi |
SACMAT | 1 |
| 2011 | A secure, constraint-aware role-based access control interoperation frameworkabstractWith the growing needs for and the benefits of sharing resources and information among different organizations, an interoperation framework that automatically integrates policies to facilitate such cross-domain sharing in a secure way is becoming increasingly important. To avoid security breaches, such policies must enforce the policy constraints of the individual domains. Such constraints may include temporal constraints that limit the times when the users can access the resources, and separation of duty (SoD) constraints. Existing interoperation solutions do not address such cross-domain temporal access control and SoDs requirements. In this paper, we propose a role-based framework to facilitate secure interoperation among multiple domains by ensuring the enforcement of temporal and SoD constraints of individual domains. To support interoperation, we do not modify the internal policies, as most of the current approaches do. We present experimental results to demonstrate our proposed framework is effective and easily realizable. Nathalie Baracaldo, Amirreza Masoumzadeh 0001, James B. D. Joshi |
NSS | 1 |