Explainability has emerged as a pillar of Trustworthy AI for ensuring safety in high-risk application domains. However, the incorporation of explainability to boost the transparency of black-box AI systems can inadvertently introduce unforeseen vulnerabilities. Previous research has drawn attention to privacy leakage, malicious or otherwise, from explainable interfaces leading to identification of individuals and exposure of sensitive personal information. Privacy preservation methods used in response to this leakage are found to adversely affect the utility of the system, including the degradation of model accuracy and explanation quality. The proposed thesis will examine the advancement of Privacy Enhancing Technologies (PETs) in Explainable AI (XAI) while ensuring that users remain at the core of the design process. The main objectives of this research are: (1) determining defenses for privacy attacks in XAI (2) building interpretable algorithms for private models and (3) examining user requirements for privacy preserving XAI. This research is expected to yield characteristics of privacy preserving XAI, guidelines and recommendations for effectively building privacy compliant XAI while considering the diverse needs of end users. The research outcomes will enable developers and researchers in designing XAI that is safe for deployment and considers the balance between privacy, explainability and utility.

Explainable artificial intelligence (XAI) plays a crucial role in mitigating the risks associated with the non-transparency of black-box artificial intelligence (AI) systems. However, despite its advantages, XAI methods have been shown to expose the privacy of individuals whose data are used to train or query the underlying models. Prior research has demonstrated privacy attacks that exploit explanations to infer sensitive personal information of individuals. At present, there is a lack of effective defenses against such privacy attacks targeting explanations, particularly when vulnerable XAI techniques are deployed in production environments or used in machine learning as a service systems. To address this gap, this study investigates the use of privacy enhancing technologies (PETs) as a defense mechanism against attribute inference attacks on explanations generated by feature-based XAI methods. We empirically evaluate three types of PETs, i.e., synthetic training data, differentially private training and noise addition, across two categories of feature-based XAI. Our findings reveal varying levels of effectiveness among the mitigation strategies, as well as trade-offs between privacy, utility and system performance. In the best scenario, integrating PETs into the explanation process reduced attack success by 49.47% while preserving model utility and explanation quality. Based on our evaluation, we propose strategies for effectively integrating PETs into XAI to maximize privacy protection and minimize the risk of sensitive information leakage.

Black-box Artificial Intelligence (AI) systems have achieved state-of-the-art accuracy in many problem domains in recent years. However, the lack of transparency of these systems is a bottleneck in their usage in high-risk applications which make automated decisions on individuals. Trustworthy AI proposes principles such as reliability, validity, privacy, fairness, and explainability among others, to mitigate risks from large-scale AI deployments in such domains. Explainable AI (XAI) is a technique of providing insights into the decision-making process of black-box systems thus enabling transparency. It plays a crucial role in communicating the rationale of automated decisions to relevant stakeholders. Though explainability is a highly desirable requirement, recent research has determined that explanations can introduce new privacy risks in AI systems. Researchers have demonstrated different types of privacy attacks on XAI deployed in production and cloud systems. Despite these risks, currently there is a lack of research into defenses for known privacy attacks in XAI. In this article, we contribute to this gap by proposing a defense mechanism for attribute inference attack on feature-based XAI. We empirically evaluate a well-known privacy preservation technique, namely, additive noise, and show its impact on privacy, explainability and utility. Our findings indicate that additive noise enables privacy while achieving faithful explanations and without compromising model utility.

Privacy attacks on machine learning (ML) models pose significant risks to individuals whose personal data is used for training or querying these models. Although concerns about the potential exposure of sensitive information through ML models continue to grow, existing safeguard mechanisms primarily focus on security threats, often neglecting privacy risks. In this paper, we examine existing tools to assess privacy risks of ML models and provide an overview of various privacy attacks and defense strategies. Given the lack of a comprehensive framework for assessing privacy vulnerabilities, we propose a generic framework for evaluating the privacy of ML systems and establish a set of tailored evaluation metrics for different types of privacy attacks. In addition, we develop a dedicated testbed to implement our framework and present experimental results that demonstrate the impact of various privacy attacks on different ML models.