Retrieval augmented generation (RAG) allows large language models to answer domain-specific questions by using external knowledge bases without training on this domain data or fine-tuning on its updates. This is especially promising for clinical tasks, as medical data tends to be dynamic, private, and distributed. By exposing the source documents that inform the response, RAG enables greater interpretability as well as reduced hallucination, both of which are crucial for safe deployment in healthcare. However, applying standard RAG to answer questions across patient data is complicated by strict access control to sensitive records enforced by privacy regulations such as HIPAA as well as the distributed nature of EHRs among institutions. We propose MediRAG, a clinical QA system that (i) supports a hierarchical design for federated document retrieval, and (ii) enables policy-based access control (PBAC) at retrieval time. For our experiments, we use the MIMIC-IV dataset, a publicly available EHR database that has been used in a wide array of research studies. We carry out a simulation of multiple federated hospitals and show that our scheme results in no loss of quality against a centralized baseline. We also evaluate performance with respect to key RAG metrics such as ROUGE, BLUE, context relevance, faithfulness and answer relevance and show that MediRAG is effective at clinical question answering across decentralized EHR documents while enforcing policies on sensitive data.

In the face of escalating threats from privacy attacks on machine learning models, we propose a system that can artificially generate data that imitates real data but doesn’t contain any sensitive or personally identifiable information. The generated data, called synthetic data, will have the same semantic and statistical distribution as the original dataset but provide privacy guarantees. Compared to previous works that dealt with either structured or unstructured data separately, our work develops a complete hybrid pipeline for generating private synthetic datasets from complex datasets that consist of both structured (numerical or categorical) and unstructured data. The private synthetic data generated can be analyzed by collaborators and third parties without increasing the risks of leakage of sensitive data. We evaluate our system on Yelp reviews and drug side-effects datasets and calculate metrics for both quality and privacy. We introduce a context-aware exposure metric to quantify context-dependent memorization and use it along with exposure to evaluate privacy. Our evaluations demonstrate that our system generates meaningful private synthetic datasets that achieve good performance in characteristic similarity, utility, as well as privacy. Given these results, the generated synthetic data can be used by data scientists, researchers, and developers to address challenges related to data privacy, scarcity, diversity, and model testing in a wide range of applications including healthcare, insurance, and financial systems that rely on sensitive data.