Despite advances in high-performance Software Defect Prediction (SDP) models, practitioners remain hesitant to adopt them due to opaque decision-making and a lack of actionable insights. Recent research has applied various explainable AI (XAI) techniques to provide explainable and actionable guidance for SDP results to address these limitations, but the trustworthiness of such guidance for practitioners has not been sufficiently investigated. Practitioners may question the feasibility of implementing the proposed changes, and if these changes fail to resolve predicted defects or prove inaccurate, their trust in the guidance may diminish. In this study, we empirically evaluate the effectiveness of current XAI approaches for SDP across 32 releases of 9 large-scale projects, focusing on whether the guidance meets practitioners' expectations. Our findings reveal that their actionable guidance (i) does not guarantee that predicted defects are resolved; (ii) fails to pinpoint modifications required to resolve predicted defects; and (iii) deviates from the typical code changes practitioners make in their projects. These limitations indicate that the guidance is not yet reliable enough for developers to justify investing their limited debugging resources. We suggest that future XAI research for SDP incorporate feedback loops that offer clear rewards for practitioners' efforts, and propose a potential alternative approach utilizing counterfactual explanations.

Modern developers make new changes based on their understanding of the existing code context and review these changes by analyzing the modified code and its context (i.e., commits). If commits are difficult to comprehend, the likelihood of human errors increases, making it harder for practitioners to identify commits that might introduce unintended defects. Nevertheless, research on predicting defect-inducing commits based on the difficulty of understanding them has been limited. In this study, we present a novel approach NeuroJIT, that leverages the correlation between modern developers' neurophysiological and empirical reactions to different code segments and their code characteristics to find the features that can capture the understandability of each commit. We investigate the understandability features of NeuroJIT in three key aspects: (i) their correlation with defect-inducing risks; (ii) their differences from widely adopted features used to predict these risks; and (iii) whether they can improve the performance of just-in-time defect prediction models. Based on our findings, we conclude that neurophysiological and empirical understandability of commits can be a competitive predictor and provide more actionable guidance from a unique perspective on defect-inducing commits.

Recently, software defect prediction studies have attempted to make black-box defect prediction models explainable and actionable. State-of-the-art defect prediction studies have utilized various model-agnostic techniques derived from the explainable AI domain as key tools to make the predictions easier for practitioners to understand. However, it has not been sufficiently investigated whether there is inconsistent information within local explanations generated by different model-agnostic techniques when interpreting a defect prediction. If local explanations generated by heterogeneous model-agnostic techniques consist of different information, the derivable insights to understand and act upon defect predictions becomes less viable and it may cause ineffective or even incorrect defect corrections. In this research, we empirically analyzed 323,844 local explanations generated by three different model-agnostic techniques: (1) LIME, (2) SHAP, and (3) BreakDown. These local explanations were analyzed in terms of how the contributions of features were distributed, how the contributions were ranked, and whether the contributions were contradictory. We concluded that (i) different model-agnostic techniques provide practitioners with local explanations where average contributions of the top-ranked features are different; (ii) different model-agnostic techniques provide practitioners with local explanations consisting of different contribution rankings and inconsistent contribution directions of top-ranked features. Therefore, we recommend that practitioners should avoid using model-agnostic techniques interchangeably and must perform a multi-faceted manual validation when planning actions based on the local explanations.