Retrieval Augmented Generation (RAG) is increasingly employed in building Generative AI applications, yet their evaluation often relies on manual, trial-and-error processes. Automating this evaluation process involves generating test data to trigger failures involving context comprehension, data formatting, specificity, and content completeness. Random question-answer generation is insufficient. However, prior works rely on standard QA datasets, benchmarks and tactics that are not tailored to the specific domain requirements. Hence, current approaches and datasets do not trigger sufficiently broad and context-specific failures. In this paper, we introduce evaluation scenarios that describe the process of generating question-answer pairs from content indexed by RAG pipelines, and they are designed to trigger a wider range of failures and to simplify automation. This enables developers to identify and address weaknesses more effectively. We validate our approach on five open-source RAG pipelines using three datasets. Our approach triggers high failure rates, by generating prompts that combine multiple questions (up to 91% failure rate) highlighting the need for developers to prioritize handling such queries. We generated failure rates of 60% in an academic domain dataset and 53% and 64% in open-domain datasets. Compared to existing state-of-the-art methods, our approach triggers 77% more failures on average per RAG pipeline and 53% more failures on average per dataset, offering a mechanism to support developers to improve the RAG pipeline quality.

Experimentation plays an important role in the work of data scientists to explore unfamiliar problem domains, to answer questions from data, and to develop diverse machine learning applications. Good experimentation requires creativity, is based on prior results and informed from the literature. However, finding relevant information from online sources to guide experimentation causes inefficiencies for data scientists. The objective of this research is to help data scientists through the presentation of context aware ranked data science experiments, considering problem domain, development task and learning task. Data science experiments for this study were extracted from publicly available interactive notebooks and were manually annotated based on a taxonomy of data science techniques and a meta model of a data science experiment. Further, the ranking algorithm was developed for data science experiments for given problem domain and development task. As a result, a tool was developed to demonstrate context aware ranked data science experiments for given problem domains such as natural language processing, computer vision and time series and for development stages such as feature engineering and model selection. This study shows that tools and techniques can be designed to be aware of the data science context, in fact, much more so than for software engineering tools. This study supports these efforts by providing knowledge that can improve experimentation process of data scientists.

Data scientists, like software engineers, use search engines, code repositories, tutorials, and question and answer sites for finding code snippets. The objective of this study is to understand what information can be extracted from data science experiment repositories for quicker availability of relevant information when data scientists search for information. In this paper, we investigated a set of notebooks to identify recurring data science techniques for efficient information retrieval and easy adaptation from online solutions to support their search during experimentation. From the manual annotation of 57 natural language processing notebooks, a taxonomy on 106 data science techniques was developed, grouped by data science workflow stages. The preliminary evaluation shows that our constructed taxonomy is relevant to retrieve information that data scientists are searching for. Future work will continue to investigate the creation of a context aware code snippet engine designed for data scientists.