Users of feature-rich tools like Excel often miss more efficient workflows, repeating tedious steps and making avoidable errors. Current approaches to helping them require either manual prompting, which is effortful for users, or automated logging, which is limiting for developers. We present InvisibleMentor, a system inspired by over-the-shoulder learning: it observes what users do, then shows them how to do it better. To do this, InvisibleMentor analyzes screen recordings with a vision-language model to reconstruct actions and context, then uses a large language model to generate vision-grounded task reflection, structured suggestions grounded in observed behavior. In a user study, participants found InvisibleMentor’s suggestions more clear, more relevant, and more useful than those from a prompt-based assistant, demonstrating that AI can do more than automate away work—it can help users master it.

Software Engineering Agents (SWE agents) can autonomously perform development tasks on benchmarks like SWE Bench, but still face challenges when tackling complex and ambiguous real-world tasks. Consequently, SWE agents are often designed to allow interactivity with developers, enabling collaborative problem-solving. To understand how developers collaborate with SWE agents and the barriers they face in such interactions, we observed 19 developers using an in-IDE agent to resolve 33 open issues in repositories to which they had previously contributed. Participants successfully resolved about half of these issues, with those solving issues incrementally having greater success than those using a one-shot approach. Participants who actively collaborated with the agent and iterated on its outputs were also more successful, though they faced challenges in trusting the agent’s responses and collaborating on debugging and testing. Our findings suggest that to facilitate successful collaborations, both SWE agents and developers should actively contribute to tasks throughout all stages of the software development process. SWE agents can enable this by challenging and engaging in discussions with developers, rather than being conclusive or sycophantic.

Spreadsheet programming is challenging. Programmers use spreadsheet programming knowledge (e.g., formulas) and problem-solving skills to combine actions into complex tasks. Advancements in large language models have introduced language agents that observe, plan, and perform tasks, showing promise for spreadsheet creation. We present TableTalk, a spreadsheet programming agent embodying three design principles—scaffolding, flexibility, and incrementality—derived from studies with seven spreadsheet programmers and 85 Excel templates. TableTalk guides programmers through structured plans based on professional workflows, generating three potential next steps to adapt plans to programmer needs. It uses pre-defined tools to generate spreadsheet components and incrementally build spreadsheets. In a study with 20 programmers, TableTalk produced higher-quality spreadsheets 2.3 times more likely to be preferred than the baseline. It reduced cognitive load and thinking time by 12.6%. From this, we derive design guidelines for agentic spreadsheet programming tools and discuss implications on spreadsheet programming, end-user programming, AI-assisted programming, and human-agent collaboration.

Prior research shows that the GenderMag method can help identify and address usability barriers that are more likely to affect women software users than men. However, the evidence for the effectiveness of GenderMag is limited to small lab studies. In this case study, by combining self-reported gender data from tens of thousands of users of an internal code review tool with software logs data gathered over a five-year period, we quantitatively show that GenderMag helped a team at Google (a) correctly identify discoverability as a usability barrier more likely to affect women than men, and (b) increase discoverability by 2.4x while also achieving gender parity. That is, compared to men using the original code review tool, women and men using the system redesigned with GenderMag were both 2.4x more likely to discover the "Suggest Edit" feature at any given time. Thus, this paper contributes the first large-scale evidence of the effectiveness of GenderMag in the field.

Online platforms and communities are a critical part of modern software engineering, yet are often affected by human biases. While previous studies investigated human biases and their potential harms against the efficiency and fairness of online communities, they have mainly focused on the open source andQ & Aplatforms, such asGitHubandStack Overflow, but overlooked the audience-focused online platforms for delivering programming and SE-related technical articles, where millions of software engineering practitioners share, seek for, and learn from high-quality software engineering articles (i.e.,technical articlesfor SE). Furthermore, most of the previous work has revealed gender and race bias, but we have little knowledge about the effect of age on software engineering practice. In this paper, we propose to investigate the effect of authors’ demographic information (gender and age) on the evaluation of technical articles on software engineering and potential behavioral differences among participants. We conducted a survey-based and controlled human study and collected responses from 540 participants to investigate developers’ evaluation of technical articles for software engineering. By controlling the gender and age of the author profiles of technical articles for SE, we found that raters tend to have more positive content depth evaluations for younger male authors when compared to older male authors and that male participants conduct technical article evaluations faster than female participants, consistent with prior study findings. Surprisingly, different from other software engineering evaluation activities (e.g., code review, pull request, etc.), we did not find a significant difference in the genders of authors on the evaluation outcome of technical articles in SE.

Teams that build software are largely demographically homogeneous. Without diversity, homogeneous perspectives dominate how, why, and for whom software is designed. To understand how teams can successfully build and sustain diversity, we interviewed 11 engineers and 9 managers from some of the most gender and racially diverse teams at Google, a large software company. Qualitatively analyzing the interviews, we found shared approaches to recruiting, hiring, and promoting an inclusive environment, all of which create a positive feedback loop. Our findings produce actionable practices that every member of the team can take to increase diversity by fostering a more inclusive software engineering environment.

Code review is an essential task for modern software engineers, where the author of a code change assigns other engineers the task of providing feedback on the author's code. In this paper, we investigate the task of code review through the lens of equity, the proposition that engineers should share reviewing responsibilities fairly. Through this lens, we quantitatively examine gender inequities in code review load at Google. We found that, on average, women perform about 25% fewer reviews than men, an inequity with multiple systemic antecedents, including authors' tendency to choose men as reviewers, a recommender system's amplification of human biases, and gender differences in how reviewer credentials are assigned and earned. Although substantial work remains to close the review load gap, we show how one small change has begun to do so.

Understanding what affects software developer productivity can help organizations choose wise investments in their technical and social environment. But the research literature either focuses on what correlates with developer productivity in ecologically valid settings or focuses on what causes developer productivity in highly constrained settings. In this paper, we bridge the gap by studying software developers at Google through two analyses. In the first analysis, we use panel data with 39 productivity factors, finding that code quality, technical debt, infrastructure tools and support, team communication, goals and priorities, and organizational change and process are all causally linked to self-reported developer productivity. In the second analysis, we use a lagged panel analysis to strengthen our causal claims. We find that increases in perceived code quality tend to be followed by increased perceived developer productivity, but not vice versa, providing the strongest evidence to date that code quality affects individual developer productivity.

The software industry lacks gender diversity. Recent research has suggested that a toxic working culture is to blame. Studies have found that communications in software repositories directed towards women are more negative in general. In this study, we use a destructive criticism lens to examine gender differences in software code review feedback. Software code review is a practice where code is peer reviewed and negative feedback is often delivered. We explore differences in perceptions, frequency, and impact of destructive criticism across genders. We surveyed 93 software practitioners eliciting perceived reactions to hypothetical scenarios (or vignettes) where participants are asked to imagine receiving either constructive or destructive criticism. In addition, the survey collected general opinions on feedback obtained during software code review as well as the frequency that participants give and receive destructive criticism. We found that opinions on destructive criticism vary. Women perceive destructive criticism as less appropriate and are less motivated to continue working with the developer after receiving destructive criticism. Destructive criticism is fairly common with more than half of respondents having received nonspecific negative feedback and nearly a quarter having received inconsiderate negative feedback in the past year. Our results suggest that destructive criticism in code review could be a contributing factor to the lack of gender diversity observed in the software industry.

Code review is a powerful technique to ensure high quality software and spread knowledge of best coding practices between engineers. Unfortunately, code reviewers may have biases about authors of the code they are reviewing, which can lead to inequitable experiences and outcomes. In principle, anonymous author code review can reduce the impact of such biases by withholding an author's identity from a reviewer. In this paper, to understand the engineering effects of using author anonymous code review in a practical setting, we applied the technique to 5217 code reviews performed by 300 software engineers at Google. Our results suggest that during anonymous author code review, reviewers can frequently guess authors’ identities; that focus is reduced on reviewer-author power dynamics; and that the practice poses a barrier to offline, high-bandwidth conversations. Based on our findings, we recommend that those who choose to implement anonymous author code review should reveal the time zone of the author by default, have a break-the-glass option for revealing author identity, and reveal author identity directly after the review.

Organizations have a variety of options to help their software developers become their most productive selves, from modifying office layouts, to investing in better tools, to cleaning up the source code. But which options will have the biggest impact? Drawing from the literature in software engineering and industrial/organizational psychology to identify factors that correlate with productivity, we designed a survey that asked 622 developers across 3 companies about these productivity factors and about self-rated productivity. Our results suggest that the factors that most strongly correlate with self-rated productivity were non-technical factors, such as job enthusiasm, peer support for new ideas, and receiving useful feedback about job performance. Compared to other knowledge workers, our results also suggest that software developers' self-rated productivity is more strongly related to task variety and ability to work remotely.

During code review, developers critically examine each others' code to improve its quality, share knowledge, and ensure conformance to coding standards. In the process, developers may have negative interpersonal interactions with their peers, which can lead to frustration and stress; these negative interactions may ultimately result in developers abandoning projects. In this mixed-methods study at one company, we surveyed 1,317 developers to characterize the negative experiences and cross-referenced the results with objective data from code review logs to predict these experiences. Our results suggest that such negative experiences, which we call "pushback", are relatively rare in practice, but have negative repercussions when they occur. Our metrics can predict feelings of pushback with high recall but low precision, making them potentially appropriate for highlighting interactions that may benefit from a self-intervention.

For modern data analytics, practices from software development are increasingly necessary to manage data, but they must be incorporated alongside other statistical and scientific skills. Therefore, we ask: how does a community recontextualize software development through the unique pressures of their work? To answer this, we explore the analytic community around baseball, or sabermetrics. To discover software development's place in the search for robust statistical insight in sports, we interview 10 participants in the sabermetric community and survey over 120 more data analysts, both in baseball and not. We explore how their work lives at the intersection of science and entertainment, and as a consequence, baseball data serves as an accessible yet deep subject to practice analytic skills. Software development exists within an iterative research process that cycles between defining rigorous statistical methods and preserving the flexibility to chase interesting problems. In this question-driven process, members of the community inhabit several overlapping roles of intentional work, in which software development can become the priority to support research and statistical infrastructure, and we discuss the way that the community can foster the balance of these skills.

Diversity, including gender diversity, is valued by many software development organizations, yet the field remains dominated by men. One reason for this lack of diversity is gender bias. In this paper, we study the effects of that bias by using an existing framework derived from the gender studies literature.We adapt the four main effects proposed in the framework by posing hypotheses about how they might manifest on GitHub,then evaluate those hypotheses quantitatively. While our results how that effects of gender bias are largely invisible on the GitHub platform itself, there are still signals of women concentrating their work in fewer places and being more restrained in communication than men.

Maintaining awareness of useful tools is a substantial challenge for developers. Physical newsletters are a simple technique to inform developers about tools. In this paper, we evaluate such a technique, called Testing on the Toilet, by performing a mixed-methods case study. We first quantitatively evaluate how effective this technique is by applying statistical causal inference over six years of data about tools used by thousands of developers. We then qualitatively contextualize these results by interviewing and surveying 382 developers, from authors to editors to readers. We found that the technique was generally effective at increasing software development tool use, although the increase varied depending on factors such as the breadth of applicability of the tool, the extent to which the tool has reached saturation, and the memorability of the tool name.

While using security tools to resolve security defects, software developers must apply considerable effort. Success depends on a developer's ability to interact with tools, ask the right questions, and make strategic decisions. To build better security tools and subsequently help developers resolve defects more accurately and efficiently, we studied the defect resolution process-from the questions developers ask to their strategies for answering them. In this paper, we report on an exploratory study with novice and experienced software developers. We equipped them with Find Security Bugs, a security-oriented static analysis tool, and observed their interactions with security vulnerabilities in an open-source system that they had previously contributed to. We found that they asked questions not only about security vulnerabilities, associated attacks, and fixes, but also questions about the software itself, the social ecosystem that built the software, and related resources and tools. We describe the strategic successes and failures we observed and how future tools can leverage our findings to encourage better strategies.

Without usable and accurate documentation of how to use an API, developers can find themselves deterred from reusing relevant code. In C++, one place developers can find documentation is in a header file. When information is missing, they may look at the corresponding implementation code. To understand what's missing from C++ API documentation and the factors influencing whether it will be fixed, we conducted a mixed-methods study involving two experience sampling surveys with hundreds of developers at the moment they visited implementation code, interviews with 18 of those developers, and interviews with 8 API maintainers. In many cases, updating documentation may provide only limited value for developers, while requiring effort maintainers don't want to invest. We identify a set of questions maintainers and tool developers should consider when improving API-level documentation.

Open-source software (OSS) often evolves from volunteer contributions, so OSS development teams must cooperate with their communities to attract new developers. However, in view of the myriad ways that developers interact over platforms for OSS development, observers of these communities may have trouble discerning, and thus learning from, the successful patterns of developer-to-team interactions that lead to eventual team acceptance. In this work, we study project communities on GitHub to discover which forms of software contribution characterize developers who begin as development team outsiders and eventually join the team, in contrast to developers who remain team outsiders. From this, we identify and compare the forms of contribution, such as pull requests and several forms of discussion comments, that influence whether new developers join OSS teams, and we discuss the implications that these behavioral patterns have for the focus of designers and educators.

Compilers primarily give feedback about problems to developers through the use of error messages. Unfortunately, developers routinely find these messages to be confusing and unhelpful. In this paper, we postulate that because error messages present poor explanations, theories of explanation---such as Toulmin's model of argument---can be applied to improve their quality. To understand how compilers should present explanations to developers, we conducted a comparative evaluation with 68 professional software developers and an empirical study of compiler error messages found in Stack Overflow questions across seven different programming languages.

Ethical decisions in software development can substantially impact end-users, organizations, and our environment, as is evidenced by recent ethics scandals in the news. Organizations, like the ACM, publish codes of ethics to guide software-related ethical decisions. In fact, the ACM has recently demonstrated renewed interest in its code of ethics and made updates for the first time since 1992. To better understand how the ACM code of ethics changes software-related decisions, we replicated a prior behavioral ethics study with 63 software engineering students and 105 professional software developers, measuring their responses to 11 ethical vignettes. We found that explicitly instructing participants to consider the ACM code of ethics in their decision making had no observed effect when compared with a control group. Our findings suggest a challenge to the research community: if not a code of ethics, what techniques can improve ethical decision making in software engineering?

The popularity of end-user programming has lead to diverse end-user development environments. Despite accurate and efficient tools available in such environments, end-user programmers often manually complete tasks. What are the consequences of rejecting these tools? In this paper, we answer this question by studying end-user programmers completing four tasks with and without tools. In analyzing 111 solutions to each of these tasks, we observe that neither tool use nor tool rejection was consistently more accurate or efficient. In some cases, tool users took nearly twice as long to solve problems and over-relied on tools, causing errors in 95% of solutions. Compared to manual task completion, the primary benefit of tool use was narrowing the kinds of errors that users made. We also observed that partial tool use can be worse than no tool use at all.

In integrated development environments, developers receive compiler error messages through a variety of textual and visual mechanisms, such as popups and wavy red underlines. Although error messages are the primary means of communicating defects to developers, researchers have a limited understanding on how developers actually use these messages to resolve defects. To understand how developers use error messages, we conducted an eye tracking study with 56 participants from undergraduate and graduate software engineering courses at our university. The participants attempted to resolve common, yet problematic defects in a Java code base within the Eclipse development environment. We found that: 1) participants read error messages and the difficulty of reading these messages is comparable to the difficulty of reading source code, 2) difficulty reading error messages significantly predicts participants' task performance, and 3) participants allocate a substantial portion of their total task to reading error messages (13%-25%). The results of our study offer empirical justification for the need to improve compiler error messages for developers.

We present the concept of Just-In-Time (JIT) static analysis that interleaves code development and bug fixing in an integrated development environment. Unlike traditional batch-style analysis tools, a JIT analysis tool presents warnings to code developers over time, providing the most relevant results quickly, and computing less relevant results incrementally later. In this paper, we describe general guidelines for designing JIT analyses. We also present a general recipe for transforming static data-flow analyses to JIT analyses through a concept of layered analysis execution. We illustrate this transformation through CHEETAH, a JIT taint analysis for Android applications. Our empirical evaluation of CHEETAH on real-world applications shows that our approach returns warnings quickly enough to avoid disrupting the normal workflow of developers. This result is confirmed by our user study, in which developers fixed data leaks twice as fast when using CHEETAH compared to an equivalent batch-style analysis.

To help users gain awareness of tools and features available in applications, recommender systems can automatically suggest useful tools. Such systems aim to present recommendations just like users would recommend tools to one another, but little is known about the nature of these user-to-user recommendations. This paper explores user-to-user recommendations through a study of 13 pairs of software users performing data analysis tasks. We found that users were more likely to adopt tools when they were receptive to the recommendation, but did not find the recommendations were any more likely to be effective when they contained other characteristics such as politeness, persuasiveness, or referred to observable tools. These findings suggest that, for example, automated systems should avoid recommending obscure and unfamiliar tools, but making recommendations politely is not a critical design goal.

Code review, where developers manually inspect one another's code changes, improves software quality and transfers knowledge in a team. Unfortunately, tools that support code review treat behavior-preserving changes, or refactorings, and behavior-altering changes, or non-refactorings, the same way, so developers have to spend effort differentiating between the two before they can evaluate the impact of a change set. In this paper, we describe a formative study of 35 developers that motivates the need for separating refactorings from non-refactorings during code review. Then, we present a refactoring-aware code review tool, called ReviewFactor, that differentiates between refactoring and non-refactoring, and allows developers to focus on one of them at a time. Finally, a case study of two open source projects suggests that ReviewFactor detects refactorings in 39% of the commits, and identifies 4.6% of the total lines of code change as refactorings. Our results also show that the precision and recall of ReviewFactor's refactoring detection algorithm are 92.5% and 94.2%, respectively.

Program navigation is a critical task for software developers. State-of-the-art tools have been shown to support effective program navigation strategies, and do so by adding widgets, secondary views, and visualizations to the screen. In this work, we build on prior work by exploring what types of navigation can be supported with relatively few interface elements. To that end, we designed and implemented a prototype tool, named Flower, that supports structural program navigation while maintaining a minimalistic interface. Flower enables developers to simultaneously navigate control flow and data flow within the Eclipse Integrated Development Environment. Based on a preliminary evaluation with eight programmers, Flower succeeds when call graphs contained relatively few branches, but was strained by complex program structures.

Quick Fixes as implemented by IDEs today prioritize the speed of applying the fix as a primary criteria for success. In this paper, we argue that when tools over-optimize this criteria, such tools neglect other dimensions that are important to successfully applying a fix, such as being able to explore the design space of multiple fixes. This is especially true in cases where a fix only partially implements the intention of the developer. In this paper, we implement an extension to the FindBugs defect finding tool, called FixBugs, an interactive resolution approach within the Eclipse development environment that prioritizes other design criteria to the successful application of suggested fixes. Our empirical evaluation method of 12 developers suggests that FixBugs enables developers to explore alternative designs and balances the benefits of manual fixing with automated fixing, without having to compromise in either effectiveness or efficiency. Our analytic evaluation method with six usability experts identified trade-offs between FixBugs and Quick Fix, and suggests ways in which FixBugs and Quick Fix can offer complementary capabilities to better support developers.

Software engineering researchers have a tendency to be optimistic about the future. Though useful, optimism bias bolsters unrealistic expectations towards desirable outcomes. We argue that explicitly framing software engineering research through pessimistic futures, or dystopias, will mitigate optimism bias and engender more diverse and thought-provoking research directions. We demonstrate through three pop culture dystopias, Battlestar Galactica, Fallout 3, and Children of Men, how reflecting on dystopian scenarios provides research opportunities as well as implications, such as making research accessible to non-experts, that are relevant to our present.

Program analysis tools use notifications to communicate with developers, but previous research suggests that developers encounter challenges that impede this communication. This paper describes a qualitative study that identifies 10 kinds of challenges that cause notifications to miscommunicate with developers. Our resulting notification communication theory reveals that many challenges span multiple tools and multiple levels of developer experience. Our results suggest that, for example, future tools that model developer experience could improve communication and help developers build more accurate mental models.

Spreadsheet environments offer many functions to manipulate data, which users can combine into complex formulae. However, for both researchers and practitioners who want to study formulae to improve spreadsheet practices, anticipating these combinations is difficult. Therefore, we developed Perquimans, a tool that analyzes spreadsheet collections to visualize patterns of function combination as an interactive tree, representing both the most common and most anomalous patterns of formula construction and their contexts. Using spreadsheets from the Enron corpus, we conduct a case study and a user study to explore Perquimans' various applications, such as those in flexible smell detection and spreadsheet education.

Programming environments and game environments share many of the same characteristics, such as requiring their users to understand strategies and solve difficult challenges. Yet, only game designers have been able to capitalize on methods that are consistently able to keep their users engaged. Consequently, software engineers have been increasingly interested in understanding how these game experiences can be transferred to programming experiences, a process termed gamification. In this perspective paper, we offer a formal argument that gamification as applied today is predominately narrow, placing emphasis on the reward aspects of game mechanics at the expense of other important game elements, such as framing. We argue that more authentic game experiences are possible when programming environments are re-conceptualized and assessed as holistic, serious games. This broad gamification enables us to more effectively apply and leverage the breadth of game elements to the construction and understanding of programming environments.

Decades of psychology and sociology research demonstrate that humans change cognitively and socially as they age, so it stands to reason that software developers likewise undergo changes that may affect their work. In this paper, we investigate age-related differences in software developers through the lens of open source software, which is built in communities that tend not to include older software developers. We report on the results of a qualitative panel discussion, then quantitatively analyze such veteran developers' activities on StackOverflow, to understand why few veteran software developers' participate in open source, and how their contributions to software development differ from their younger peers. Our results suggest that veterans' are less motivated by social interactions than their younger peers. Our results also suggest that veterans could contribute a broader knowledge of software development than their younger peers, as well as knowledge of old technologies that can be applied to newer technologies.

Developers who use version control are expected to produce systematic commit histories that show well-defined steps with logical forward progress. Existing version control tools assume that developers also write code systematically. Unfortunately, the process by which developers write source code is often evolutionary, or as-needed, rather than systematic. Our contribution is a fragment-oriented concept called Commit Bubbles that will allow developers to construct systematic commit histories that adhere to version control best practices with less cognitive effort, and in a way that integrates with their as-needed coding workflows.

Spreadsheets are used extensively in business processes around the world and as such, are a topic of research interest. Over the past few years, many spreadsheet studies have been performed on the EUSES spreadsheet corpus. While this corpus has served the spreadsheet community well, the spreadsheets it contains are mainly gathered with search engines and might therefore not represent spreadsheets used in companies. This paper presents an analysis of a new dataset, extracted from the Enron email archive, containing over 15,000 spreadsheets used within the Enron Corporation. In addition to the spreadsheets, we also present an analysis of the associated emails, where we look into spreadsheet-specific email behavior. Our analysis shows that 1) 24% of Enron spreadsheets with at least one formula contain an Excel error, 2) there is little diversity in the functions used in spreadsheets: 76% of spreadsheets in the presented corpus use the same 15 functions and, 3) the spreadsheets are substantially more smelly than the EUSES corpus, especially in terms of long calculation chains. Regarding the emails, we observe that spreadsheets 1) are a frequent topic of email conversation with 10% of emails either referring to or sending spreadsheets and 2) the emails are frequently discussing errors in and updates to spreadsheets.

Spreadsheets are perhaps the most ubiquitous form of end-user programming software. This paper describes a corpus, called Fuse, containing 2,127,284 URLs that return spreadsheets (and their HTTP server responses), and 249,376 unique spreadsheets, contained within a public web archive of over 26.83 billion pages. Obtained using nearly 60,000 hours of computation, the resulting corpus exhibits several useful properties over prior spreadsheet corpora, including reproducibility and extendability. Our corpus is unencumbered by any license agreements, available to all, and intended for wide usage by end-user software engineering researchers. In this paper, we detail the data and the spreadsheet extraction process, describe the data schema, and discuss the trade-offs of Fuse with other corpora.

Grounded theory is an important research method in empirical software engineering, but it is also time consuming, tedious, and complex. This makes it difficult for researchers to assess if threats, such as missing themes or sample bias, have inadvertently materialized. To better assess such threats, our new idea is that we can automatically extract knowledge from social news websites, such as Hacker News, to easily replicate existing grounded theory research --- and then compare the results. We conduct a replication study on static analysis tool adoption using Hacker News. We confirm that even a basic replication and analysis using social news websites can offer additional insights to existing themes in studies, while also identifying new themes. For example, we identified that security was not a theme discovered in the original study on tool adoption. As a long-term vision, we consider techniques from the discipline of knowledge discovery to make this replication process more automatic.

Even though different developers have varying levels of expertise, the tools in one developer's integrated development environment (IDE) behave the same as the tools in every other developers' IDE. In this paper, we propose the idea of automatically customizing development tools by modeling what a developer knows about software concepts. We then sketch three such ``bespoke'' tools and describe how development data can be used to infer what a developer knows about relevant concepts. Finally, we describe our ongoing efforts to make bespoke program analysis tools that customize their notifications to the developer using them.

Security tools can help developers answer questions about potential vulnerabilities in their code. A better understanding of the types of questions asked by developers may help toolsmiths design more effective tools. In this paper, we describe how we collected and categorized these questions by conducting an exploratory study with novice and experienced software developers. We equipped them with Find Security Bugs, a security-oriented static analysis tool, and observed their interactions with security vulnerabilities in an open-source system that they had previously contributed to. We found that they asked questions not only about security vulnerabilities, associated attacks, and fixes, but also questions about the software itself, the social ecosystem that built the software, and related resources and tools. For example, when participants asked questions about the source of tainted data, their tools forced them to make imperfect tradeoffs between systematic and ad hoc program navigation strategies.

Security tools could help developers find critical vulnerabilities, yet such tools remain underused. We surveyed developers from 14 companies and 5 mailing lists about their reasons for using and not using security tools. The resulting thirty-nine predictors of security tool use provide both expected and unexpected insights. As we expected, developers who perceive security to be important are more likely to use security tools than those who do not. But that was not the strongest predictor of security tool use, it was instead developers' ability to observe their peers using security tools.

While there are a variety of existing tools to help detect security vulnerabilities in code, they are seldom used by developers due to the time or security expertise required. We are investigating techniques integrated within the IDE to help developers detect and mitigate security vulnerabilities. In this paper, we examine using interactive annotation for access control vulnerabilities. We evaluated whether developers could indicate access control logic using interactive annotation and understand the vulnerabilities reported as a result. Our study indicates that developers can easily find and annotate access control logic but can struggle to use our tool to trace the cause of the vulnerability. Our results provide design guidance for improving the interaction and communication of such security tools with developers.

When software engineers fix bugs, they may have several options as to how to fix those bugs. Which fix they choose has many implications, both for practitioners and researchers: What is the risk of introducing other bugs during the fix? Is the bug fix in the same code that caused the bug? Is the change fixing the cause or just covering a symptom? In this paper, we investigate alternative fixes to bugs and present an empirical study of how engineers make design choices about how to fix bugs. We start with a motivating case study of the Pex4Fun environment. Then, based on qualitative interviews with 40 engineers working on a variety of products, data from six bug triage meetings, and a survey filled out by 326 Microsoft engineers and 37 developers from other companies, we found a number of factors, many of them non-technical, that influence how bugs are fixed, such as how close to release the software is. We also discuss implications for research and practice, including how to make bug prediction and localization more accurate.

The human element is often considered the weakest element in security. Although many kinds of humans interact with systems that are designed to be secure, one particular type of human is especially important, the security information worker. Security information workers include software developers, system administrators, and intelligence analysts. This workshop aims to develop and stimulate discussion about security information workers.

Security tools can help developers build more secure software systems by helping developers detect or fix security vulnerabilities in source code. However, developers do not always use these tools. In this paper, we investigate a number of social factors that impact developers' adoption decisions, based on a multidisciplinary field of research called diffusion of innovations. We conducted 42 one-on-one interviews with professional software developers, and our results suggest a number of ways in which security tool adoption depends on developers' social environments and on the channels through which information about tools is communicated. For example, some participants trusted developers with strong reputations on the Internet as much as they trust their colleagues for information about security tools.

Refactoring, the practice of applying behavior-preserving changes to existing code, can enhance the quality of software systems. Refactoring tools can automatically perform and check the correctness of refactorings. However, even when developers have these tools, they still perform about 90% of refactorings manually, which is error-prone. To address this problem, we propose a technique called GhostFactor separating transformation and correctness checking: we allow the developer to transform code manually, but check the correctness of her transformation automatically. We implemented our technique as a Visual Studio plugin, then evaluated it with a human study of eight software developers; GhostFactor improved the correctness of manual refactorings by 67%.

Video games make up an important part of the software industry, yet the software engineering community rarely studies video games. This imbalance is a problem if video game development differs from general software development, as some game experts suggest. In this paper we describe a study with 14 interviewees and 364 survey respondents. The study elicited substantial differences between video game development and other software development. For example, in game development, “cowboy coders” are necessary to cope with the continuous interplay between creative desires and technical constraints. Consequently, game developers are hesitant to use automated testing because of these tests’ rapid obsolescence in the face of shifting creative desires of game designers. These differences between game and non-game development have implications for research, industry, and practice. For instance, as a starting point for impacting game development, researchers could create testing tools that enable game developers to create tests that assert flexible behavior with little up-front investment.

Although software development involves making numerous decisions amongst alternatives, the design and implementation choices made typically become invisible; what a developer sees in the project's artifacts are the end result of all of the decisions. What if, instead, all of the choices made were tracked and it was easy for a developer to revisit a point where a decision was made and choose another alternative? What if the development environment could detect and suggest alternative choices? What if it was easy and low-cost to try another path? We explore the idea of speculative reprogramming that could support a what-if environment for the programming stages of software development.

Self-explanation is one cognitive strategy through which developers comprehend error notifications. Self-explanation, when left solely to developers, can result in a significant loss of productivity because humans are imperfect and bounded in their cognitive abilities. We argue that modern IDEs offer limited visual affordances for aiding developers with self-explanation, because compilers do not reveal their reasoning about the causes of errors to the developer. The contribution of our paper is a foundational set of visual annotations that aid developers in better comprehending error messages when compilers expose their internal reasoning. We demonstrate through a user study of 28 undergraduate Software Engineering students that our annotations align with the way in which developers self-explain error notifications. We show that these annotations allow developers to give significantly better self-explanations when compared against today's dominant visualization paradigm, and that better self-explanations yield better mental models of notifications. The results of our work suggest that the diagrammatic techniques developers use to explain problems can serve as an effective foundation for how IDEs should visually communicate to developers.

Developers often start programming tasks by searching for relevant code in their local codebase. Previous research suggests that 88% of manually-composed queries retrieve no relevant results. Many searches fail because existing search tools depend solely on string matching with a manually-composed query, which cannot find semantically-related code. To solve this problem, researchers proposed query recommendation techniques to help developers compose queries without the extensive knowledge of the codebase under search. However, few of these techniques are empirically evaluated by the usage data from real-world developers. To fill this gap, we studied several query recommendation techniques by extending Sando and conducting a longitudinal field study. Our study shows that over 30% of all queries were adopted from recommendation; and recommended queries retrieved results 7% more often than manual queries.

As a software system evolves, the system's codebase constantly changes, making it difficult for developers to answer such questions as who is knowledgeable about particular parts of the code or who needs to know about changes made. In this article, we show that an externalized model of a developer's individual knowledge of code can make it easier for developers to answer such questions. We introduce a degree-of-knowledge model that computes automatically, for each source-code element in a codebase, a real value that represents a developer's knowledge of that element based on a developer's authorship and interaction data. We present evidence that shows that both authorship and interaction data of the code are important in characterizing a developer's knowledge of code. We report on the usage of our model in case studies on expert finding, knowledge transfer, and identifying changes of interest. We show that our model improves upon an existing expertise-finding approach and can accurately identify changes for which a developer should likely be aware. We discuss how our model may provide a starting point for knowledge transfer but that more refinement is needed. Finally, we discuss the robustness of the model across multiple development sites.

Using static analysis tools for automating code inspections can be beneficial for software engineers. Such tools can make finding bugs, or software defects, faster and cheaper than manual inspections. Despite the benefits of using static analysis tools to find bugs, research suggests that these tools are underused. In this paper, we investigate why developers are not widely using static analysis tools and how current tools could potentially be improved. We conducted interviews with 20 developers and found that although all of our participants felt that use is beneficial, false positives and the way in which the warnings are presented, among other things, are barriers to use. We discuss several implications of these results, such as the need for an interactive mechanism to help developers fix defects.

When software engineers fix bugs, they may have several options as to how to fix those bugs. Which fix they choose has many implications, both for practitioners and researchers: What is the risk of introducing other bugs during the fix? Is the bug fix in the same code that caused the bug? Is the change fixing the cause or just covering a symptom? In this paper, we investigate alternative fixes to bugs and present an empirical study of how engineers make design choices about how to fix bugs. Based on qualitative interviews with 40 engineers working on a variety of products, data from 6 bug triage meetings, and a survey filled out by 326 engineers, we found a number of factors, many of them non-technical, that influence how bugs are fixed, such as how close to release the software is. We also discuss several implications for research and practice, including ways to make bug prediction and localization more accurate.

Software engineering researchers develop great techniques consisting of practices and tools that improve efficiency and quality of software development. Prior work evaluates developers' use of techniques such as Test-Driven-Development and refactoring by measuring actions in the development environment. What we still lack is a method to communicate effectively and motivate developers to adopt best practices and tools. This work proposes a game-like system to motivate adoption while continuously measuring developers' use of more efficient development techniques.

Becoming an expert at programming is thought to take an estimated 10,000 hours of deliberate practice. But what happens after that? Do programming experts continue to develop, do they plateau, or is there a decline at some point? A diversity of opinion exists on this matter, but many seem to think that aging brings a decline in adoption and absorption of new programming knowledge. We develop several research questions on this theme, and draw on data from StackOverflow (SO) to address these questions. The goal of this research is to support career planning and staff development for programmers by identifying age-related trends in SO data. We observe that programmer reputation scores increase relative to age well into the 50's, that programmers in their 30's tend to focus on fewer areas relative to those younger or older in age, and that there is not a strong correlation between age and scores in specific knowledge areas.

Although useful and widely available, refactoring tools are underused. One cause of this underuse is that a developer sometimes fails to recognize that she is going to refactor before she begins manually refactoring. To address this issue, we conducted a formative study of developers' manual refactoring process, suggesting that developers' reliance on “chasing error messages” when manually refactoring is an error-prone manual refactoring strategy. Additionally, our study distilled a set of manual refactoring workflow patterns. Using these patterns, we designed a novel refactoring tool called BeneFactor. BeneFactor detects a developer's manual refactoring, reminds her that automatic refactoring is available, and can complete her refactoring automatically. By alleviating the burden of recognizing manual refactoring, BeneFactor is designed to help solve the refactoring tool underuse problem.

The wide variety of software development tools available today have a great potential to improve the way developers make software, but that potential goes unfulfilled when developers are not aware of useful tools. In this paper, I introduce the idea of continuous social screencasting, a novel mechanism to help developers gain awareness of relevant tools by enabling them to learn remotely and asychronously from their peers. The idea builds on the strength of several existing techniques that developers already use for discovering new tools, including screencasts and online social networks.

Software developers interact with the development environments they use by issuing commands that execute various programming tools, from source code formatters to build tools. However, developers often only use a small subset of the commands offered by modern development environments, reducing their overall development fluency. In this paper, we use several existing command recommender algorithms to suggest new commands to developers based on their existing command usage history, and also introduce several new algorithms. By running these algorithms on data submitted by several thousand Eclipse users, we describe two studies that explore the feasibility of automatically recommending commands to software developers. The results suggest that, while recommendation is more difficult in development environments than in other domains, it is still feasible to automatically recommend commands to developers based on their usage history, and that using patterns of past discovery is a useful way to do so.

As a research community, we currently know very little about the challenges faced by blind software developers. Without knowing what those challenges are, the community cannot effectively address these challenges. In this paper, we describe the first exploratory empirical study, where we conducted eight interviews with blind software developers to identify aspects of software development that are a challenge. Our results suggest that visually impaired software developers face challenges, for instance, when using screen readers to look up information when writing code. We discuss a variety of implications, including that blind software developers need additional support in discovering relevant software development tools.

Refactoring tools, common to many integrated development environments, can help programmers to restructure their code. These tools sometimes refuse to restructure the programmer's code, instead giving the programmer a textual error message that she must decode if she wishes to understand the reason for the tool's refusal and what corrective action to take. This paper describes a graphical alternative to textual error messages called Refactoring Annotations. It reports on two experiments, one using an integrated development environment and the other using paper mockups, that show that programmers can use Refactoring Annotations to quickly and accurately understand the cause of refactoring errors.

Refactoring is widely practiced by developers, and considerable research and development effort has been invested in refactoring tools. However, little has been reported about the adoption of refactoring tools, and many assumptions about refactoring practice have little empirical support. In this paper, we examine refactoring tool usage and evaluate some of the assumptions made by other researchers. To measure tool usage, we randomly sampled code changes from four Eclipse and eight Mylyn developers and ascertained, for each refactoring, if it was performed manually or with tool support. We found that refactoring tools are seldom used: 11 percent by Eclipse developers and 9 percent by Mylyn developers. To understand refactoring practice at large, we drew from a variety of data sets spanning more than 39,000 developers, 240,000 tool-assisted refactorings, 2,500 developer hours, and 12,000 version control commits. Using these data, we cast doubt on several previously stated assumptions about how programmers refactor, while validating others. Finally, we interviewed the Eclipse and Mylyn developers to help us understand why they did not use refactoring tools and to gather ideas for future research.

Computer users rely on software tools to work effectively and efficiently, but it is difficult for users to be aware of all the tools that might be useful to them. While there are several potential technical solutions to this difficulty, we know little about social solutions, such as one user telling a peer about a tool. To explore these social solutions in one particular domain, we describe a series of interviews with 18 programmers in industry that explore how tool discovery takes place. These interviews provide a rich set of qualitative data that give us detailed insights into how programmers discover tools. One finding was that, while programmers believe that discovery from peers is effective, they actually discover tools from peers relatively infrequently. Another finding was that programmers can effectively discover tools from their peers both in a co-located and remote settings. We describe several implications of our findings, such as that discovery from peers can be enhanced by improving programmers' ability to communicate openly and concisely about tools.

Commercial software development teams have limited time available to focus on improvements to their software. These teams need a way to quickly identify areas of the source code that would benefit from improvement, as well as quantifiable data to defend the selected improvements to management. Past research has shown that mining configuration management systems for change information can be useful in determining faulty areas of the code. We present a tool named Code Hot Spot, which mines change records out of Microsoft's TFS configuration management system and creates a report of hot spots. Hot spots are contiguous areas of the code that have higher values of metrics that are indicators of faulty code. We present a study where we use this tool to study projects at ABB to determine areas that need improvement. The resulting data have been used to prioritize areas for additional code reviews and unit testing, as well as identifying change prone areas in need of refactoring.

Support for generic programming was added to the Java language in 2004, representing perhaps the most significant change to one of the most widely used programming languages today. Researchers and language designers anticipated this addition would relieve many long-standing problems plaguing developers, but surprisingly, no one has yet measured whether generics actually provide such relief. In this paper, we report on the first empirical investigation into how Java generics have been integrated into open source software by automatically mining the history of 20 popular open source Java programs, traversing more than 500 million lines of code in the process. We evaluate five hypotheses, each based on assertions made by prior researchers, about how Java developers use generics. For example, our results suggest that generics do not significantly reduce the number of type casts and that generics are usually adopted by a single champion in a project, rather than all committers.

Refactoring is the process of changing the structure of code without changing its meaning, and is a frequent practice among developers. Although programmers refactor frequently, they usually do not use refactoring tools to automate this process. We argue that the need to recall the name of a refactoring before the appropriate tool can be invoked makes it unnecessarily hard to initiate a refactoring with a tool. Conventional ways of initiating a tool also make it hard to transition from novice tool user to expert tool user. The contribution of this paper is a memorable mapping from gestures to refactorings, and an implementation of that mapping in the form of marking menus. In the first reported experiment to explore the effect of the position of items in marking menus on people's ability to infer the location of those items, we asked 16 programmers to complete a paper-based evaluation of our mapping. The results suggest that programmers can infer the gesture that will invoke the appropriate refactoring tool, even if they do not know the name of the refactoring. We also illustrate how marking menus might be used for refactoring during development with two other small studies.

The size and high rate of change of source code comprising a software system make it difficult for software developers to keep up with who on the team knows about particular parts of the code. Existing approaches to this problem are based solely on authorship of code. In this paper, we present data from two professional software development teams to show that both authorship and interaction information about how a developer interacts with the code are important in characterizing a developer's knowledge of code. We introduce the degree-of-knowledge model that computes automatically a real value for each source code element based on both authorship and interaction information. We show that the degree-of-knowledge model can provide better results than an existing expertise finding approach and also report on case studies of the use of the model to support knowledge transfer and to identify changes of interest.

Much of what we know about how programmers refactor in the wild is based on studies that examine just a few software projects. Researchers have rarely taken the time to replicate these studies in other contexts or to examine the assumptions on which they are based. To help put refactoring research on a sound scientific basis, we draw conclusions using four data sets spanning more than 13 000 developers, 240 000 tool-assisted refactorings, 2500 developer hours, and 3400 version control commits. Using these data, we cast doubt on several previously stated assumptions about how programmers refactor, while validating others. For example, we find that programmers frequently do not indicate refactoring activity in commit logs, which contradicts assumptions made by several previous researchers. In contrast, we were able to confirm the assumption that programmers do frequently intersperse refactoring with other program changes. By confirming assumptions and replicating studies made by other researchers, we can have greater confidence that those researchers' conclusions are generalizable.

Refactoring, the process of changing the structure of code without changing its behavior, can be semi-automated with the help of tools. However, many tools do a poor job of communicating errors triggered by the refactoring process. This poor communication causes programmers to refactor slowly, conservatively, and incorrectly. In this paper we demonstrate problems with current refactoring tools, characterize three new tools to assist in refactoring, and describe a user study that compares these new tools against existing tools. The results of the study show that the speed, accuracy, and user satisfaction can be significantly increased The new tools have inspired a set of usability recommendations that we hope will help build a new generation of programmer-friendly refactoring tools.