Developers continuously interact in crowd-sourced community-based question-answer (Q&A) sites. Reportedly, ∼ 30% of all software professionals visit the most popular Q&A site StackOverflow (SO) every day. Software engineering (SE) research studies are also increasingly using SO data. To find out the trend, implication, impact, and future research potential utilizing SO data, a systematic mapping study needs to be conducted. Following a rigorous reproducible mapping study approach, from 18 reputed SE journals and conferences, we collected 384 SO-based research articles and categorized them into 10 facets (i.e., themes). We found that SO contributes to 85% of SE research compared with popular Q&A sites such as Quora, and Reddit. We found that 18 SE domains directly benefited from SO data whereas Recommender Systems , and API Design and Evolution domains use SO data the most (15% and 16% of all SO-based research studies, respectively). API Design and Evolution , and Machine Learning with/for SE domains have consistent upward publication. Deep Learning Bug Analysis and Code Cloning research areas have the highest potential research impact recently. With the insights, recommendations, and facet-based categorized paper list from this mapping study, SE researchers can find out potential research areas according to their interest to utilize large-scale SO data.

We conducted a survey of 135 software engineering (SE) practitioners to understand how they use Generative AI-based chatbots like ChatGPT for SE tasks. We find that they want to use ChatGPT for SE tasks like software library selection but often worry about the truthfulness of ChatGPT responses. We developed a suite of techniques and a tool called CID (ChatGPT Incorrectness Detector) to automatically test and detect the incorrectness in ChatGPT responses. CID is based on the iterative prompting to ChatGPT by asking it contextually similar but textually divergent questions (using an approach that utilizes metamorphic relationships in texts). The underlying principle in CID is that for a given question, a response that is different from other responses (across multiple incarnations of the question) is likely an incorrect response. In a benchmark study of library selection, we show that CID can detect incorrect responses from ChatGPT with an F1-score of 0.74 -- 0.75.