Recent research has shown feasibility of an interactive pair-programming conversational agent, but implementing such an agent poses three challenges: a lack of benchmark datasets, absence of software engineering specific labels, and the need to understand developer conversations. To address these challenges, we conducted a Wizard of Oz study with 14 participants pair programming with a simulated agent and collected 4,443 developer-agent utterances. Based on this dataset, we created 26 software engineering labels using an open coding process to develop a hierarchical classification scheme. To understand labeled developer-agent conversations, we compared the accuracy of three state-of-the-art transformer-based language models, BERT, GPT-2, and XLNet, which performed interchangeably. In order to begin creating a developer-agent dataset, researchers and practitioners need to conduct resource intensive Wizard of Oz studies. Presently, there exists vast amounts of developer-developer conversations on video hosting websites. To investigate the feasibility of using developer-developer conversations, we labeled a publicly available developer-developer dataset (3,436 utterances) with our hierarchical classification scheme and found that a BERT model trained on developer-developer data performed ~10% worse than the BERT trained on developer-agent data, but when using transfer-learning, accuracy improved. Finally, our qualitative analysis revealed that developer-developer conversations are more implicit, neutral, and opinionated than developer-agent conversations. Our results have implications for software engineering researchers and practitioners developing conversational agents.

Pair programming conversational agents demand vast amounts of data for training. Recently a benchmark dataset of developer-developer and developer-agent pair programming conversations, from lab studies (8,324 utterances) was released for training natural language unit of a pair programming conversational agent. Unfortunately, this dataset is limited to a single domain and language. To investigate if it was feasible to utilize already available pair programming conversations from online video hosting platforms (i.e. YouTube), we collected five Youtube videos (roughly 350 minutes) with 4,822 utterances. We used transformer-based language model BERT to compare the lab studies with online videos. We found that a transfer learning approach, first training BERT on online videos and then fine-tuning with specific developer-agent data, resulted in the best performance.

While pair programming conversational agents have the potential to change the current landscape of programming, they require vast amounts of diverse data to train. However, due to gender gaps in the Computer Science field, it is difficult to obtain data involving women in pair programming scenarios; this may result in a bias in a future agent. Furthermore, previous research has highlighted differences between men and women in problem solving, communication, creativity, and leadership styles, which are critical for the success of pair collaboration. Therefore, it is crucial to understand how the agent’s performance is affected by the gender composition of training datasets. Using the transformer-based language model BERT, we created a natural language understanding (NLU) model for our future agent, and tested its intent classification performance when alternately trained and tested on datasets composed entirely of either men or women. We found that the model’s performance was significantly higher when trained and tested on men datasets, indicating the presence of gender bias within the NLU model of a future agent.