The rise of Gen-AI and LLMs often makes it difficult for users to trust retrieved results.The risk of IR systems using LLMs and being susceptible to misinformation can be tackled under the lens of explainability in AI.The topic of explainability in AI, machine learning (XAI) and information retrieval (IR) has been explored through various methods, yet few incorporate user feedback to adapt explanations.In this work, we present an XAI-driven extension to the classic relevance feedback model in IR, incorporating user feedback in the process of explaining the model behavior to the user.Our proposed model, RelEx, introduces XAI-specific elements, including key phrase vectors, text summaries, and contextual phrases combined with a neural ranker.RelEx interactively gathers user feedback, adapting search results based on the modified query and contextual vectors.We further introduce a novel additive similarity scheme that combines document similarity with key-phrase overlap.Retrieval performance is empirically evaluated on multiple benchmark datasets.In the absence of ground truth explanations, we assess explainability and assessability via user studies, where RelEx exhibit promising results.