A/B testing, also referred to as online controlled experimentation or continuous experimentation, is a form of hypothesis testing where two variants of a piece of software are compared in the field from an end user’s point of view. A/B testing is widely used in practice to enable data-driven decision making for software development. While a few studies have explored different facets of research on A/B testing, no comprehensive study has been conducted on the state-of-the-art in A/B testing. Such a study is crucial to provide a systematic overview of the field of A/B testing driving future research forward. To address this gap and provide an overview of the state-of-the-art in A/B testing, this paper reports the results of a systematic literature review that analyzed primary studies. The research questions focused on the subject of A/B testing, how A/B tests are designed and executed, what roles stakeholders have in this process, and the open challenges in the area. Analysis of the extracted data shows that the main targets of A/B testing are algorithms, visual elements, and workflow and processes. Single classic A/B tests are the dominating type of tests, primarily based in hypothesis tests. Stakeholders have three main roles in the design of A/B tests: concept designer, experiment architect, and setup technician. The primary types of data collected during the execution of A/B tests are product/system data, user-centric data, and spatio-temporal data. The dominating use of the test results are feature selection, feature rollout, continued feature development, and subsequent A/B test design. Stakeholders have two main roles during A/B test execution: experiment coordinator and experiment assessor. The main reported open problems are related to the enhancement of proposed approaches and their usability. From our study we derived three interesting lines for future research: strengthen the adoption of statistical methods in A/B testing, improving the process of A/B testing, and enhancing the automation of A/B testing.

When modeling topics from chat messages of developer instant messaging communication, individual chat messages are short text documents. Our study aims at understanding how short text topic models perform with conversations from developer instant messaging. We applied four models to nine Gitter chat rooms (with sizes ranging from ≈100 to ≈160,000 messages). To assess the quality of topics and identify the best performing models, we compared topics based on four metrics for topic coherence. Furthermore, for a subset of Gitter chat rooms we used two human-based assessments: intrusion tasks with 18 experts analyzing 40 topics each, and topic naming (assigning a name to a topic that summarizes its main concept) with eight additional experts naming 60 topics each. Models performed differently in terms of coherence metrics and human assessment depending on the corpus (small, medium or large chat room). Our findings offer recommendations for the selection and use of short text topic models with developer chat messages based on characteristics of models and their performance with different sizes of corpora, and based on different strategies to assess topic quality.

Software developers use instant messaging (e.g., Slack, Gitter) to collaboratively discuss software engineering problems and solutions. This communication takes place in chat rooms that generally contain a description of the main topic of discussion and the messages exchanged. To analyze whether and how the knowledge accumulated in these chat rooms is relevant to other developers, we first need to understand the themes discussed in these chat rooms. In this paper, we used thematic analysis to manually identify software engineering themes in the description of 87 chat rooms of Gitter, an instant messaging tool for software developers. Then, we checked whether these themes also occur in 184 public chat rooms of Slack, another instant messaging tool. We identified 47 themes in Gitter chat rooms, and regarding the applicability of themes, we could relate 36 of our themes to 173 Slack chat rooms. Our results indicate that, in the context of our study, chat rooms in developer instant messaging communication are mostly about software development technologies and practices rather than development processes. Furthermore, most chat rooms are topic- rather than project-related (e.g., a chat room used by developers of a particular software development project).

Abstract Topic modeling using models such as Latent Dirichlet Allocation (LDA) is a text mining technique to extract human-readable semantic “topics” (i.e., word clusters) from a corpus of textual documents. In software engineering, topic modeling has been used to analyze textual data in empirical studies (e.g., to find out what developers talk about online), but also to build new techniques to support software engineering tasks (e.g., to support source code comprehension). Topic modeling needs to be applied carefully (e.g., depending on the type of textual data analyzed and modeling parameters). Our study aims at describing how topic modeling has been applied in software engineering research with a focus on four aspects: (1) which topic models and modeling techniques have been applied, (2) which textual inputs have been used for topic modeling, (3) how textual data was “prepared” (i.e., pre-processed) for topic modeling, and (4) how generated topics (i.e., word clusters) were named to give them a human-understandable meaning. We analyzed topic modeling as applied in 111 papers from ten highly-ranked software engineering venues (five journals and five conferences) published between 2009 and 2020. We found that (1) LDA and LDA-based techniques are the most frequent topic modeling techniques, (2) developer communication and bug reports have been modelled most, (3) data pre-processing and modeling parameters vary quite a bit and are often vaguely reported, and (4) manual topic naming (such as deducting names based on frequent words in a topic) is common.

Software development requires many different types of knowledge, such as knowledge about software development processes, practices and techniques, and about the domain of an application. Software, developers often share knowledge in informal communication channels (e.g., instant messaging tools, e-mails, or online forums). Considering that this informal communication contains knowledge that may be potentially relevant for other developers and given that this knowledge is not necessarily captured and formally documented for reuse, in this work we propose (a) exploring whether developer communication (via instant messaging) is a suitable source of reusable software engineering knowledge; (b) investigating how to identify that knowledge using data mining; (c) and analysing through action research how to present it to developers in a useful way for reuse. The envisioned theories and solutions approaches will analyze existing software development data captured in communication, rather than data that were captured and stored specifically to be reused.