The adoption of artificial intelligence (AI) tools, specifically those based on large language models (LLMs), has quickly led to turbulent changes in software engineering practice. However, despite their potential, AI-based tools are often thrust into the development process with little consideration towards establishing trust, a key factor in the adoption of new tooling. With this work, we address this trust gap by focusing on the needs of software engineers to facilitate a trust-based, synergistic adoption framework in AI-assisted software engineering (AI4SE). We aim to identify the key factors influencing trust in AI4SE and to design a practical adoption framework that covers the concrete steps and best practices for a trust-based integration of AI4SE. Finally, we evaluate the designed framework by applying it to a real-world use case with a professional software development team. By treating trust as a core design goal, this research provides practical guidance to improve the developer experience and facilitate a seamless adoption of AI4SE tooling.

Searching for clues, gathering evidence, and reviewing case files are all techniques used by criminal investigators to draw sound conclusions and avoid wrongful convictions. Medicine, too, has a long tradition of evidence-based practice, in which administering a treatment without evidence of its efficacy is considered malpractice. Similarly, in software engineering (SE) research, we can develop sound methodologies and mitigate threats to validity by basing study design decisions on evidence. Echoing a recent call for the empirical evaluation of design decisions in program comprehension experiments, we conducted a 2-phases study consisting of systematic literature searches, snowballing, and thematic synthesis. We found out (1) which validity threat categories are most often discussed in primary studies of code comprehension, and we collected evidence to build (2) the evidence profiles for the three most commonly reported threats to validity. We discovered that few mentions of validity threats in primary studies (31 of 409) included a reference to supporting evidence. For the three most commonly mentioned threats, namely the influence of programming experience, program length, and the selected comprehension measures, almost all cited studies (17 of 18) did not meet our criteria for evidence. We show that for many threats to validity that are currently assumed to be influential across all studies, their actual impact may depend on the design and context of each specific study. Researchers should discuss threats to validity within the context of their particular study and support their discussions with evidence. The present paper can be one resource for evidence, and we call for more meta-studies of this type to be conducted, which will then inform design decisions in primary studies. Further, although we have applied our methodology in the context of program comprehension, our approach can also be used in other SE research areas to enable evidence-based experiment design decisions and meaningful discussions of threats to validity.

Background: Developers spend a lot of their time on understanding source code. Static code analysis tools can draw attention to code that is difficult for developers to understand. However, most of the findings are based on non-validated metrics, which can lead to confusion and code that is hard to understand not being identified.