Abstract Large language models (LLMs) are rapidly reshaping software development, but their impact across the full software development lifecycle is underexplored. Existing work tends to focus on isolated activities such as code generation or testing, leaving open questions about how LLMs affect developers, processes, products, and the broader software ecosystem. We address this gap through semi-structured interviews with sixteen early-adopter software professionals who integrated LLM-based tools into their day-to-day work in early to mid-2023. We treat these interviews as early empirical evidence and compare participants’ accounts with recent work on LLMs in software engineering, noting which early patterns persist or shift. Using thematic analysis, we organize our findings around four dimensions: people, process, product, and society. Developers reported substantial productivity gains from reducing mundane tasks, streamlining search, and accelerating debugging, but also described a productivity-quality paradox: they frequently discarded generated code and shifted effort from writing code to critically evaluating and integrating it. LLM use was highly phase-dependent, with strong uptake in implementation and debugging but limited influence on requirements gathering and collaborative work. Participants developed new competencies to use LLMs effectively, including prompt engineering strategies, multi-layered verification, and security-conscious integration to protect proprietary data. They also anticipated changes in hiring expectations, team practices, and computing education, while emphasizing that human judgment and foundational software engineering skills remain essential. Our findings, consistent with evidence from large-scale studies, offer actionable implications for developers, organizations, educators, and tool designers seeking to integrate LLMs responsibly into professional software practice.

Remote collaboration software mediates modern knowledge work, yet existing inspection methods often overlook inequities in remote teamwork. We introduce RemoteCollabEval, a new HCI inspection method for systematically surfacing inclusivity issues in synchronous remote collaboration software. RemoteCollabEval consists of: (1) six novel research-based facets illustrated by two personas grounded in social identity theory to capture the interdependent nature of teamwork for dominant and under-served users, and (2) a specialized walkthrough to help practitioners identify inclusivity issues and actionable interface design fixes. We evaluated RemoteCollabEval against the standard Groupware Walkthrough in a controlled study with 29 HCI students across 10 teams. Participants inspected Zoom and Replit, followed by a survey and semi-structured group discussion. RemoteCollabEval surfaced about six times more inclusivity issues than the standard approach and was perceived as comprehensive and well-designed. Our contributions include: (1) a validated method for identifying inclusivity issues in collaboration software and (2) teamwork-specific facets that provide a foundation for more equitable design practices in distributed teams.

Modern knowledge work is increasingly collaborative, especially in information-intensive domains such as crisis response, scientific discovery, and software engineering. Software engineering epitomizes these trends through practices like pair programming and collaborative debugging. Yet existing computational models of information foraging remain individual-centric, leaving teams without support for social foraging—leveraging partners’ actions and communication to navigate complex projects. We introduce PFIS-T, a predictive computational model of social information foraging. Building on the PFIS model family, it integrates implicit cues from teammates’ recent navigation and explicit cues from synchronous communication to predict a programmer’s next action. We evaluated PFIS-T with ten three-person debugging teams, finding that it substantially outperforms the strongest individual baseline, PFIS3, predicting 81.5% of navigations and improving accuracy by 16.7%. These results show how predictive models can operationalize social foraging and point to opportunities for collaborative IDEs and interactive systems that adaptively surface social trails to improve coordination and awareness.

Women researchers in computing face persistent, career-longchallenges that contribute to broader gender disparities in the field. While prior work has examined barriers faced by women in computing, little is known about the lived experiences of women researchers, specifically those navigating careers across academia and industry. To address this gap, we conducted in-depth, semi-structured interviews with ten novice and experienced women researchers across three continents. Using inductive analysis and open coding of transcribed interviews, we identified thirteen career-related challenges, organized into three overarching themes: absence of belonging, experiences of implicit bias, and increased emotional burden. Participants described heightened gender awareness in computing environments, which contributed to feelings of isolation and hindered professional growth. Our findings highlight the need for targeted interventions to support the career trajectories of women researchers, foster inclusive research communities, and ultimately reduce the gender gap in computing.

Remote pair programming is widely used in software development, but no research has examined how race affects these interactions between developers. We embarked on this study due to the historical underrepresentation of Black developers in the tech industry, with White developers comprising the majority. Our study involved 24 experienced developers, forming 12 gender-balanced same- and mixed-race pairs. Pairs collaborated on a programming task using the think-aloud method, followed by individual retrospective interviews. Our findings revealed elevated productivity scores for mixed-race pairs, with no differences in code quality between same- and mixed-race pairs. Mixed-race pairs excelled in task distribution, shared decision-making, and role-exchange but encountered communication challenges, discomfort, and anxiety, shedding light on the complexity of diversity dynamics. Our study emphasizes race’s impact on remote pair programming and underscores the need for diverse tools and methods to address racial disparities for collaboration.

This study compares the information foraging behavior of developers on two prominent platforms, StackOverflow and GitHub, which are widely used for code hosting and question and answer purposes. Understanding how developers seek and retrieve information is crucial for designing effective interfaces. In a gender and expertise-balanced study involving 12 developers, we utilized Information Foraging Theory to analyze their foraging behavior. Our findings revealed contrasting patterns, with women spending 30% more time and utilizing 21.24% more cues on GitHub, while men utilized 55% more time and 19.7% more cues on StackOverflow. These insights have significant implications for optimizing website design and information presentation to enhance the efficiency and effectiveness of developers' information seeking processes.

Remote pair programming is a popular software development method that lacks evidence on how race may affect pair dynamics. Past computer science studies demonstrated how race impacts various fields such as AI, education, politics and jobs. A complex history of interracial interactions in the United States has led to differences in collaborative styles. We recruited 12 professional developers and investigated how same-and mixed-race pairs (Black-White) coordinated during remote pair programming interactions. Our results revealed that Black developers in mixed-race pairs were more democratic while in same-race pairs were more authoritative.

Recent research has shown feasibility of an interactive pair-programming conversational agent, but implementing such an agent poses three challenges: a lack of benchmark datasets, absence of software engineering specific labels, and the need to understand developer conversations. To address these challenges, we conducted a Wizard of Oz study with 14 participants pair programming with a simulated agent and collected 4,443 developer-agent utterances. Based on this dataset, we created 26 software engineering labels using an open coding process to develop a hierarchical classification scheme. To understand labeled developer-agent conversations, we compared the accuracy of three state-of-the-art transformer-based language models, BERT, GPT-2, and XLNet, which performed interchangeably. In order to begin creating a developer-agent dataset, researchers and practitioners need to conduct resource intensive Wizard of Oz studies. Presently, there exists vast amounts of developer-developer conversations on video hosting websites. To investigate the feasibility of using developer-developer conversations, we labeled a publicly available developer-developer dataset (3,436 utterances) with our hierarchical classification scheme and found that a BERT model trained on developer-developer data performed ~10% worse than the BERT trained on developer-agent data, but when using transfer-learning, accuracy improved. Finally, our qualitative analysis revealed that developer-developer conversations are more implicit, neutral, and opinionated than developer-agent conversations. Our results have implications for software engineering researchers and practitioners developing conversational agents.

Stack Overflow is the most popular platform for developers to exchange knowledge on software engineering tasks. Developers spend large portions of their time trying to find relevant questions and answers. However, Stack Overflow contains tremendous amounts of noisy and redundant questions and answers. This becomes costly in terms of time as well as cognitive load on developers. To understand information-seeking behavior, we used Information Foraging Theory (IFT). We conducted a gender-balanced, think-aloud study with 12 participants (6 students and 6 professionals) where they completed a debugging task. We investigated the cues associated with participants’ actions through the lens of IFT.

Pair programming conversational agents demand vast amounts of data for training. Recently a benchmark dataset of developer-developer and developer-agent pair programming conversations, from lab studies (8,324 utterances) was released for training natural language unit of a pair programming conversational agent. Unfortunately, this dataset is limited to a single domain and language. To investigate if it was feasible to utilize already available pair programming conversations from online video hosting platforms (i.e. YouTube), we collected five Youtube videos (roughly 350 minutes) with 4,822 utterances. We used transformer-based language model BERT to compare the lab studies with online videos. We found that a transfer learning approach, first training BERT on online videos and then fine-tuning with specific developer-agent data, resulted in the best performance.

While pair programming conversational agents have the potential to change the current landscape of programming, they require vast amounts of diverse data to train. However, due to gender gaps in the Computer Science field, it is difficult to obtain data involving women in pair programming scenarios; this may result in a bias in a future agent. Furthermore, previous research has highlighted differences between men and women in problem solving, communication, creativity, and leadership styles, which are critical for the success of pair collaboration. Therefore, it is crucial to understand how the agent’s performance is affected by the gender composition of training datasets. Using the transformer-based language model BERT, we created a natural language understanding (NLU) model for our future agent, and tested its intent classification performance when alternately trained and tested on datasets composed entirely of either men or women. We found that the model’s performance was significantly higher when trained and tested on men datasets, indicating the presence of gender bias within the NLU model of a future agent.

GitHub is the largest code hosting platform for version control and collaboration. With presence of numerous files and folders within each repository, developers have to sacrifice their time to sift through them during debugging. We investigated developers’ debugging behavior with a think-aloud study involving 12 participants and reported cues and strategies utilized by them.

We operationalized information foraging theory for Stack Overflow and built a semi-supervised model to recommend optimal information to the developers.

From automated customer support to virtual assistants, conversational agents have transformed everyday interactions, yet despite phenomenal progress, no agent exists for programming tasks. To understand the design space of such an agent, we prototyped PairBuddy—an interactive pair programming partner—based on research from conversational agents, software engineering, education, human-robot interactions, psychology, and artificial intelligence. We iterated PairBuddy’s design using a series of Wizard-of-Oz studies. Our pilot study of six programmers showed promising results and provided insights toward PairBuddy’s interface design. Our second study of 14 programmers was positively praised across all skill levels. PairBuddy’s active application of soft skills—adaptability, motivation, and social presence—as a navigator increased participants’ confidence and trust, while its technical skills—code contributions, just-in-time feedback, and creativity support—as a driver helped participants realize their own solutions. PairBuddy takes the first step towards an Alexa-like programming partner.

Pair programming has a documented history of benefits, such as increased code quality, productivity, self-efficacy, knowledge transfer, and reduced gender gap. Research uncovered problems with pair programming related to scheduling, collocating, role imbalance, and power dynamics. We investigated the trade-offs of substituting a human with an agent to simultaneously provide benefits and alleviate obstacles in pair programming. We conducted gender-balanced studies with human-human pairs in a remote lab with 18 programmers and Wizard-of-Oz studies with 14 programmers, then analyzed results quantitatively and qualitatively. Our comparative analysis of the two studies showed no significant differences in productivity, code quality, and self-efficacy. Further, agents facilitated knowledge transfer; however, unlike humans, agents were unable to provide logical explanations or discussions. Human partners trusted and showed humility towards agents. Our results demonstrate that agents can act as effective pair programming partners and open the way towards new research on conversational agents for programming.

Remote pair programming research indicates benefits for CS students, increasing productivity, code quality, teamwork, knowledge management, and morale. The COVID-19 pandemic increased the prevalence of remote pair programming. Gender gaps persist in CS classes and workplaces, which may negatively impact the way pairs coordinate, communicate, and collaborate. To understand these effects, we conducted a large-scale survey to investigate differences between men and women as well as same- and mixed-gender pairs. The survey questions were adapted from established literature on gender differences in the fields of education, communication, management, human-robotic interaction, and human-computer interaction. Quantitative analysis of the survey data using ANOVA and pairwise t-tests indicated that women participants reported their men partners made gender-based assumptions about them, and felt dominated and interrupted with men partners. Men participants felt their men partners were more rude and gave more negative feedback than women partners. Further, qualitative analysis of interviews gave insights to several challenges CS students face in same and mixed-gender pairs when programming remotely. Our findings have implications for researchers, practitioners, and educators to promote gender inclusivity in collaborative environments.

Established research on pair programming reveals benefits, including increasing communication, creativity, self-efficacy, and promoting gender inclusivity. However, research has reported limitations such as finding a compatible partner, scheduling sessions between partners, and resistance to pairing. Further, pairings can be affected by predispositions to negative stereotypes. These problems can be addressed by replacing one human member of the pair with a conversational agent. To investigate the design space of such a conversational agent, we conducted a controlled remote pair programming study. Our analysis found various creative problem-solving strategies and differences in conversational styles. We further analyzed the transferable strategies from human-human collaboration to human-agent collaboration by conducting a Wizard of Oz study. The findings from the two studies helped us gain insights regarding design of a programmer conversational agent. We make recommendations for researchers and practitioners for designing pair programming conversational agent tools.

Remote pair programming encapsulates the benefits of well-researched (co-located) pair programming. However, its effectiveness is hindered by challenges including pair incompatibility, imbalanced roles, and inclinations to work alone. Recent research has explored pedagogical methods to alleviate these challenges, but none have considered the integration of machine learning agents to facilitate remote pair programming. Therefore, we investigated the capabilities of popular text classification algorithms on identifying three facets of pair programming: dialogue acts, creativity stages, and pair programming roles. We collected a dataset of 3,436 utterances from a lab study of 18 pair programmers in a simulated remote environment. We found that pair programming dialogue poses a challenge as it is often unpremeditated and inadequately structured. Despite this, the accuracy of our machine learning classifier was improved by the choice of contextual dialogue features. Our results have implications for facilitating pair programming in global software development and online computer science education.

Online CS education shows many gender-inclusivity problems in practice and through tools, and little has been done to address this. In CS classrooms, pair programming has been shown to significantly help women and men understand and appreciate programming concepts, and to help close gender gaps. Unfortunately, pair programming is not well supported in online CS education, especially when one of the pair members is a woman. Our overall objective is to address this gap by investigating the following question: How can we bring the educational benefits of pair programming to online CS students in gender-equitable ways? In this paper, we empirically investigate whether and how technology-mediated remote pair programming hinders online students of same- and mixed-gender pairs. Based on our results, we propose refining personas and the cognitive walkthrough to include leadership styles and preferences for pair-programming roles. We further recommend features for online CS educational tools to promote gender-inclusiveness.

Programmers reuse code to increase their productivity, which leads to large fragments of duplicate or near-duplicate code in the code base. The current code clone detection techniques for finding semantic clones utilize Program Dependency Graphs (PDG), which are expensive and resource-intensive. PDG and other clone detection techniques utilize code and have completely ignored the comments - due to ambiguity of English language, but in terms of program comprehension, comments carry the important domain knowledge. We empirically evaluated the accuracy of detecting clones with both code and comments on a JHotDraw package. Results show that detecting code clones in the presence of comments, Latent Dirichlet Allocation (LDA), gave 84% precision and 94% recall, while in the presence of a PDG, using GRAPLE, we got 55% precision and 29% recall. These results indicate that comments can be used to find semantic clones. We recommend utilizing comments with LDA to find clones at the file level and code with PDG for finding clones at the function level. These findings necessitate a need to reexamine the assumptions regarding semantic clone detection techniques.

Technical and social competencies are highly desirable for a protean developer. Managers make hiring decisions based on developer's contributions to online peer production sites like GitHub and Stack Overflow. These sites provide ample history regarding developers' technical and social skills. Although these histories are utilized by hiring tools to help managers make their hiring decisions, little is known empirically how developers' social skills affect their technical skills and vice versa. Without such knowledge, tools, research, and training might be flawed. We present an in-depth empirical study investigating the correlation between the technical and social skills of developers. Our quantitative analysis of factors influencing the social skills of developers compared with factors affecting their technical skills indicates that better collaboration competency skills are associated with enhanced coding abilities as well as the quality of code.

Foraging among similar variants of the same artifact is a common activity, but computational models of Information Foraging Theory (IFT) have not been developed to take such variants into account. Without being able to computationally predict people's foraging behavior with variants, our ability to harness the theory in practical ways--such as building and systematically assessing tools for people who forage different variants of an artifact--is limited. Therefore, in this paper, we introduce a new predictive model, PFIS-V, that builds upon PFIS3, the most recent of the PFIS family of modeling IFT in programming situations. Our empirical results show that PFIS-V is up to 25% more accurate than PFIS3 in predicting where a forager will navigate in a variationed information space.

Foraging among too many variants of the same artifact can be problematic when many of these variants are similar. This situation, which is largely overlooked in the literature, is commonplace in several types of creative tasks, one of which is exploratory programming. In this paper, we investigate how novice programmers forage through similar variants. Based on our results, we propose a refinement to Information Foraging Theory (IFT) to include constructs about variation foraging behavior, and propose refinements to computational models of IFT to better account for foraging among variants.

Managers are increasingly using online contributions to make hiring decisions. However, it is nontrivial to find the relevant information of candidates in large online, global communities. We present Visual Resume, a novel tool that aggregates information on contributions across two different types of peer production sites (a code hosting site and a technical Q&A forum). Visual Resume displays summaries of developers' contributions, and allows easy access to contribution details. It also facilitates pairwise comparisons of candidates through a card-based design. Our study, involving participants from global organizations or corporations that draw from the global community, indicates that Visual Resume facilitated hiring decisions, both technical and soft skills were important when making these decisions.

Programming is a creative task and is generally exploratory in nature. Often, end-user programmers (non-professional) indulge in opportunistically creating their programs. To facilitate learning and reuse of code, most end-user programming environments provides online repositories. While the provision of programs in repositories helps support end-user programming to an extent, finding and reusing an appropriate program variant is a challenging task. In this paper, we explore the reuse behavior of end-user programmers using Information Foraging Theory. We conducted an empirical study of eight end-user programmers, qualitatively analyzed their information-seeking behavior while reusing program variants, and report new cue types and strategies specific to end-user programmers.

Many systems are designed to help novices who want to learn programming, but few support those who are not interested in learning (more) programming. This paper targets the subset of end-user programmers (EUPs) in this category. We present a set of principles on how to help EUPs like this learn just a little when they need to overcome a barrier. We then instantiate the principles in a prototype and empirically investigate the principles in two studies: a formative think-aloud study and a pair of summer camps attended by 42 teens. Among the surprising results were the complementary roles of implicitly actionable hints versus explicitly actionable hints, and the importance of both context-free and context-sensitive availability. Under these principles, the camp participants required significantly less in-person help than in a previous camp to learn the same amount of material in the same amount of time.

End users with little formal programming background are creating software in many different forms, including spreadsheets, web macros, and web mashups. Web mashups are particularly popular because they are relatively easy to create, and because many programming environments that support their creation are available. These programming environments, however, provide no support for tracking versions or provenance of mashups. We believe that versioning support can help end users create, understand, and debug mashups. To investigate this belief, we have added versioning support to a popular wire-oriented mashup environment, Yahoo! Pipes. Our enhanced environment, which we call “Pipes Plumber,” automatically retains versions of pipes and provides an interface with which pipe programmers can browse histories of pipes and retrieve specific versions. We have conducted two studies of this environment: an exploratory study and a larger controlled experiment. Our results provide evidence that versioning helps pipe programmers create and debug mashups. Subsequent qualitative results provide further insights into the barriers faced by pipe programmers, the support for reuse provided by our approach, and the support for debugging provided.

Programming for the web can be an intimidating task, particularly for non-professional ("end-user") programmers. Mashup programming environments attempt to remedy this by providing support for such programming. It is well known, however, that mashup programmers create applications that contain bugs. Furthermore, mashup programmers learn from examples and reuse other mashups, which causes bugs to propagate to other mashups. In this paper we classify the bugs that occur in a large corpus of Yahoo! Pipes mashups. We describe support we have implemented in the Yahoo! Pipes environment to provide automatic error detection techniques that help mashup programmers localize and correct these bugs. We present the results of a think-aloud study comparing the experiences of end-user mashup programmers using and not using our support. Our results show that our debugging enhancements do help these programmers localize and correct bugs more effectively and efficiently.

End-user programming environments provide central repositories where users can execute and store their programs. However, these environments do not provide facilities by which users can keep track of the variations that they create for their programs. In the professional world, software developers use variation management for code reuse, program understanding, change traceability, debugging and maintenance.

Web active end users often coalesce web information using web mashups. Web contents, however, tend to evolve frequently, and along with the black box nature of visual languages this complicates the process of debugging mashups. While debugging, end users need to locate faults within the code and then find a way to correct them; this process requires them to seek information related to web page content and behavior. In this paper, using an information foraging theory lens, we qualitatively study the debugging behaviors of 16 web-active end users. Our results show that the stronger scents available within mashup programming environments can improve users' foraging success. Our results lead to a new model for debugging activities framed in terms of information foraging theory, and to a better understanding of ways in which end-user programming environments can be enhanced to better support debugging.

Web mashup environments provide a way for users to combine data from web applications and services to create new content. Currently, these environments do not provide support for tracking the development histories of mashups. We have thus added configuration management support to the Yahoo! Pipes mashup environment. We describe this support, and provide results of an experiment studying the ability of programmers to create and debug mashups in its presence. Our results show that versioning support can help both groups of users do both tasks better.