Recent research has shown feasibility of an interactive pair-programming conversational agent, but implementing such an agent poses three challenges: a lack of benchmark datasets, absence of software engineering specific labels, and the need to understand developer conversations. To address these challenges, we conducted a Wizard of Oz study with 14 participants pair programming with a simulated agent and collected 4,443 developer-agent utterances. Based on this dataset, we created 26 software engineering labels using an open coding process to develop a hierarchical classification scheme. To understand labeled developer-agent conversations, we compared the accuracy of three state-of-the-art transformer-based language models, BERT, GPT-2, and XLNet, which performed interchangeably. In order to begin creating a developer-agent dataset, researchers and practitioners need to conduct resource intensive Wizard of Oz studies. Presently, there exists vast amounts of developer-developer conversations on video hosting websites. To investigate the feasibility of using developer-developer conversations, we labeled a publicly available developer-developer dataset (3,436 utterances) with our hierarchical classification scheme and found that a BERT model trained on developer-developer data performed ~10% worse than the BERT trained on developer-agent data, but when using transfer-learning, accuracy improved. Finally, our qualitative analysis revealed that developer-developer conversations are more implicit, neutral, and opinionated than developer-agent conversations. Our results have implications for software engineering researchers and practitioners developing conversational agents.

Pair programming conversational agents demand vast amounts of data for training. Recently a benchmark dataset of developer-developer and developer-agent pair programming conversations, from lab studies (8,324 utterances) was released for training natural language unit of a pair programming conversational agent. Unfortunately, this dataset is limited to a single domain and language. To investigate if it was feasible to utilize already available pair programming conversations from online video hosting platforms (i.e. YouTube), we collected five Youtube videos (roughly 350 minutes) with 4,822 utterances. We used transformer-based language model BERT to compare the lab studies with online videos. We found that a transfer learning approach, first training BERT on online videos and then fine-tuning with specific developer-agent data, resulted in the best performance.