Robots are increasingly present in human spaces, such as for conducting deliveries in hospitals, interacting with visitors at museums, and stocking items in warehouses. To ensure the seamless integration of robots into these spaces, a new role in human-robot interaction is emerging—the robot wrangler, namely an individual who is responsible for setting up, overseeing, and troubleshooting the robot. To understand the needs of this stakeholder, we conducted a scoping review that uncovered a typology of robot wrangling across the research literature, and discovered that wrangling is an umbrella term that collapses a highly complex and heterogeneous space of activities, often rendering this labor difficult to characterize and support. To further clarify and understand robot wrangling, we then reflected on our own firsthand and imagined experiences as robot wranglers within our own respective domains. Guided by the scoping review and our reflections, we devise a series of design implications for supporting wranglers directly as individuals and as members of a wider service ecology.

Debugging is an essential yet often tedious part of the software development process. Omniscient debuggers have long aimed to make debugging easier by recording execution traces, enabling more direct debugging interactions. Although the concept of omniscient debugging has been explored extensively in research, it has seen limited adoption in industry until recently. The emergence of new commercial tools like Replay presents an opportunity to reevaluate the impact of omniscient debugging. In this paper, we conducted a controlled experiment with 20 participants with a commercial omniscient debugger, Replay, and a traditional debugger, Chrome DevTools. We investigated whether the omniscient debugger improved developer productivity and how it influenced debugging behavior. We coded developers’ navigation, rerun, and runtime value collection behaviors and summarized their debugging strategies. Our results show that developers with the omniscient debugger were not more successful or faster than those using the traditional debugger. Omniscient debugger users reran the program less, but there was no significant difference in the number of files or functions they explored or the number of runtime values they collected. Omniscient debugger users faced navigation and runtime value collection challenges, which may have hindered their effectiveness. Our results suggest that commercial omniscient debuggers must include more of the high-level support for interacting with traces found in research prototypes to successfully help developers in debugging tasks.

Microtask programming enables software engineers such as freelancers and part-time employees to contribute to software projects even when they can not spend much time on them. It decomposes software design into small, self-contained specifications. The decomposed specifications enable them to complete implementation and review task in a short time. In this paper, we empirically investigate the time required for software engineers to complete microtasks in an industrial setting and explore their perceptions of microtask programming by investigating two industrial projects using it. The projects were carried out in different companies and differed in the employment of the engineers. One contracted 9 freelancers, and the other asked for 8 part-time contributions from employees at work on other projects. We conducted a survey and a focus group with the engineers. Based on the development data of the case studies, we found that almost all microtasks were completed in less than four pomodoro repetitions, namely about two hours in the pomodoro technique. These data shows that engineers who cannot work full-time on a project can undertake microtasks if they can spare one-third of their work day. We also examine how engineers who are employees experience microtask programming similarly and differently from freelancers.

Decision-making is a key software engineering skill. Developers constantly make choices throughout the software development process, from requirements to implementation. While prior work has studied developer decision-making, the choices made while choosing what solution to write in code remain understudied. In this mixed-methods study, we examine the phenomenon where developers select one specific way to implement a behavior in code, given many potential alternatives. We call these decisions implementation design decisions. Our mixed-methods study includes 46 survey responses and 14 semi-structured interviews with professional developers about their decision types, considerations, processes, and expertise for implementation design decisions. We find that implementation design decisions, rather than being a natural outcome from higher levels of design, require constant monitoring of higher level design choices, such as requirements and architecture. We also show that developers have a consistent general structure to their implementation decision-making process, but no single process is exactly the same. We discuss the implications of our findings on research, education, and practice, including insights on teaching developers how to make implementation design decisions.

When software defects occur, developers begin the debugging process by formulating hypotheses to explain the cause. These hypotheses guide the investigation process, determining which evidence developers gather to accept or reject the hypothesis, such as parts of the code and program state developers examine. However, existing debugging techniques do not offer support in finding relevant hypotheses, leading to wasted time testing hypotheses and examining code that ultimately does not lead to a fix. To address this issue, we introduce a new type of debugging tool, the hypothesis-based debugger, and an implementation of this tool in Hypothesizer. Hypothesis-based debuggers support developers from the beginning of the debugging process by finding relevant hypotheses until the defect is fixed. To debug using Hypothesizer, developers first demonstrate the defect, generating a recording of the program behavior with code execution, user interface events, network communications, and user interface changes. Based on this information and the developer’s descriptions of the symptoms, Hypothesizer finds relevant hypotheses, analyzes the code to identify relevant evidence to test the hypothesis, and generates an investigation plan through a timeline view. This summarizes all evidence items related to the hypothesis, indicates whether the hypothesis is likely to be true by showing which evidence items were confirmed in the recording, and enables the developer to quickly check evidence in the recording by viewing code snippets for each evidence item. A randomized controlled experiment with 16 professional developers found that, compared to traditional debugging tools and techniques such as breakpoint debuggers and Stack Overflow, Hypothesizer dramatically improved the success rate of fixing defects by a factor of five and decreased the time to debug by a factor of three.

Welcome to the 2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). We proudly continue VL/HCC's tradition as the premier international forum for research on how people learn, express, and understand computational ideas and on the languages, tools, and interventions that can aid people in doing so. Our program this year was diverse as usual, and we also encouraged submissions and keynote talks around low-code / no-code programming systems, especially given the recent popularity of AI coding assistance tools based on large language models (LLMs) such as GitHub Copilot, ChatGPT, and others. With these new technologies developing at such a rapid pace, it is an exciting time to be learning and doing programming.

A key goal of software engineering research is to improve the environments, tools, languages, and techniques programmers use to efficiently create quality software. Successfully designing these tools and demonstrating their effectiveness involves engaging with tool users—software engineers. Researchers often want to conduct user studies of software engineers to collect direct evidence. However, running user studies can be difficult, and researchers may lack solution strategies to overcome the barriers, so they may avoid user studies. To understand the challenges researchers face when conducting programmer user studies, we interviewed 26 researchers. Based on the analysis of interview data, we contribute (i) a taxonomy of 18 barriers researchers encounter; (ii) 23 solution strategies some researchers use to address 8 of the 18 barriers in their own studies; and (iii) 4 design ideas, which we adapted from the behavioral science community, that may lower 8 additional barriers. To validate the design ideas, we held an in-person all-day focus group with 16 researchers.

In many domains, strategic knowledge is documented and shared through checklists and handbooks. In software engineering, however, developers rarely share strategic knowledge for approaching programming problems, in contrast to other artifacts and despite its importance to productivity and success. To understand barriers to sharing, we simulated a programming strategy knowledge-sharing platform, asking experienced developers to articulate a programming strategy and others to use these strategies while providing feedback. Throughout, we asked strategy authors and users to reflect on the challenges they faced. Our analysis revealed that developers could share strategic knowledge. However, they struggled in choosing a level of detail and understanding the diversity of the potential audience. While authors required substantial feedback, users struggled to give it and authors to interpret it. Our results suggest that sharing strategic knowledge differs from sharing code and raises challenging questions about how knowledge-sharing platforms should support search and feedback.

Developers building web applications constantly face challenges, particularly in working with complex APIs. In response, developers often turn to Stack Overflow, offering a window into the programming barriers developers face. We examined 301 posts on Stack Overflow related to front-end web development and systematically characterized the challenges present in these posts. We found that most challenges reflected not a request for new code or an explanation of an error message but a request about how a specific code snippet might be edited to make its behavior as desired. Many challenges also reflected an underlying need to gather information about how specific code idioms are implemented within a framework or library. We identified 28 barriers developers face in front-end web development. Our findings suggest opportunities for facilitating more effective interactions with complex APIs through new types of programming content and tools that better address barriers in working with code idioms.

As developers program and debug, they continuously edit and run their code, a behavior known as edit-run cycles. While techniques such as live programming are intended to support this behavior, little is known about the characteristics of edit-run cycles themselves. To bridge this gap, we analyzed 28 hours of programming and debugging work from 11 professional developers which encompassed over three thousand development activities. We mapped activities to edit or run steps, constructing 581 debugging and 207 programming edit-run cycles. We found that edit-run cycles are frequent. Developers edit and run the program, on average, 7 times before fixing a defect and twice before introducing a defect. Developers waited longer before again running the program when programming than debugging, with a mean cycle length of 3 minutes for programming and 1 minute for debugging. Most cycles involved an edit to a single file after which a developer ran the program to observe the impact on the final output. Edit-run cycles which included activities beyond edit and run, such as navigating between files, consulting resources, or interacting with other IDE features, were much longer, with a mean length of 5 minutes, rather than 1.5 minutes. We conclude with a discussion of design recommendations for tools to enable more fluidity in edit-run cycles.

Developers rely heavily on resources to find technical insights on how to use languages, APIs, and platforms, seeking help from Stack Overflow, GitHub, meetups, blogs, live streams, forums, documentation, and more. However, there is one kind of knowledge for which resources are hard to find: strategic knowledge. In contrast to technical knowledge, strategic knowledge provides insight into how to approach problem-solving. Prior work has demonstrated that developers can make use of written strategies to improve their problem-solving. However, there is currently no way for developers to share, curate, and search for this knowledge at scale. To address this gap, we contribute HowToo, a platform for sharing, finding, and using programming strategies. Its key insight is that there are many different approaches to the same problem, and developers may need different strategies depending on their situation. In a longitudinal evaluation with more than 30 students in a project-based software engineering course, we found that: 1) students viewed HowToo as complementary to technical resources; 2) students viewed strategies as helping them be more systematic and complete in their work; 3) HowToo helped students be more confident in their problem solving; 4) when students were under time pressure, they were less inclined to use HowToo to structure their work, as being mindful required them to slow down.

Good documentation offers the promise of enabling developers to easily understand design decisions. Unfortunately, in practice, design documents are often rarely updated, becoming inaccurate, incomplete, and untrustworthy. A better solution is to enable developers to write down design rules which are checked against code for consistency. But existing rule checkers require learning specialized query languages or program analysis frameworks, creating a barrier to writing project-specific rules. We introduce two new techniques for authoring design rules: snippet-based authoring and semi-natural-language authoring. In snippet-based authoring, developers specify characteristics of elements to match by writing partial code snippets. In semi-natural language authoring, a textual representation offers a representation for understanding design rules and resolving ambiguities. We implemented these approaches in RulePad. To evaluate RulePad, we conducted a between-subjects study with 14 participants comparing RulePad to the PMD Designer, a utility for writing rules in a popular rule checker. We found that those with RulePad were able to successfully author 13 times more query elements in significantly less time and reported being significantly more willing to use RulePad in their everyday work.

A critical issue in software development projects in IT service companies is finding the right people at the right time. By enabling assignments of tasks to people to be more fluid, the use of crowdsourcing approaches within a company offers a potential solution to this challenge. Inside a company, as multiple system development projects are ongoing separately, developers with slack time on one project might use this time to contribute to other projects. In this paper, we report on a case study of the application of crowdsourcing within an industrial web application system development project in a large telecommunications company. Developers worked with system specifications which were organized into a set of microtasks, offering a set of short and self-contained descriptions. When crowd workers in other projects had slack time, they fetched and completed microtasks. Our results offer initial evidence for the potential value of microtask programming in increasing the fluidity of team assignments within a company. Crowd contributors to the project were able to onboard and contribute to a new project in less than 2 hours. After onboarding, the crowd workers were together able to successfully implement a small program which contained only a small number of defects. Interview and survey data gathered from project participants revealed that crowd workers reported that they perceived onboarding costs to be reduced and did not experience issues with the reduced face to face communication, but experienced challenges with motivation.

When working in large and complex codebases, developers face challenges using Find Usages to understand how to reuse classes and methods. To better understand these challenges, we conducted a small exploratory study with 4 participants. We found that developers often wasted time reading long lists of similar usages or prematurely focused on a single usage. Based on these findings, we hypothesized that clustering usages by the similarity of their surrounding context might enable developers to more rapidly understand how to use a function. To explore this idea, we designed and implemented Find Unique Usages, which extracts usages, computes a diff between pairs of usages, generates similarity scores, and uses these scores to form usage clusters. To evaluate this approach, we conducted a controlled experiment with 12 participants. We found that developers with Find Unique Usages were significantly faster, completing their task in 35% less time.

As developers debug, developers formulate hypotheses about the cause of the defect and gather evidence to test these hypotheses. To better understand the role of hypotheses in debugging, we conducted two studies. In a preliminary study, we found that, even with the benefit of modern internet resources, incorrect hypotheses can cause developers to investigate irrelevant information and block progress. We then conducted a controlled experiment where 20 developers debugged and recorded their hypotheses. We found that developers have few hypotheses, two per defect. Having a correct hypothesis early strongly predicted later success. We also studied the impact of two debugging aids: fault locations and potential hypotheses. Offering fault locations did not help developers formulate more correct hypotheses or debug more successfully. In contrast, offering potential hypotheses made developers six times more likely to succeed. These results demonstrate the potential of future debugging tools that enable finding and sharing relevant hypotheses.

One way to teach programming problem solving is to teach explicit, step-by-step strategies. While prior work has shown these to be effective in controlled settings, there has been little work investigating their efficacy in classrooms. We conducted a 5-week case study with 17 students aged 15-18, investigating students' sentiments toward two strategies for debugging and code reuse, students' use of scaffolding to execute these strategies, and associations between students' strategy use and their success at independently writing programs in class. We found that while students reported the strategies to be valuable, many had trouble regulating their choice of strategies, defaulting to ineffective trial and error, even when they knew systematic strategies would be more effective. Students that embraced the debugging strategy completed more features in a game development project, but this association was mediated by other factors, such as reliance on help, strategy self-efficacy, and mastery of the programming language used in the class. These results suggest that teaching of strategies may require more explicit instruction on strategy selection and self-regulation.

In live-streamed programming, developers broadcast their development work on open source projects using streaming media such as YouTube or Twitch. Sessions are first announced by a developer acting as the streamer, inviting other developers to join and interact as watchers using chat. To better understand the characteristics, motivations, and challenges in live-streamed programming, we analyzed 20 hours of live-streamed programming videos and surveyed 7 streamers about their experiences. The results reveal that live-streamed programming shares some of the characteristics and benefits of pair programming, but differs in the nature of the relationship between the streamer and watchers. We also found that streamers are motivated by knowledge sharing, socializing, and building an online identity, but face challenges with tool limitations and maintaining engagement with watchers. We discuss the implications of these findings, identify limitations with current tools, and propose design recommendations for new forms of tools to better supporting live-streamed programming.

Developers authoring HTML documents define elements following patterns which establish and reflect the visual structure of a document, such as making all images in a footer the same height by applying a class to each. To surface these patterns to developers and support developers in authoring consistent with these patterns, we propose a mixed human-AI technique for creating code patterns. Patterns are first learned from individual HTML documents through a decision tree, generating a representation which developers may view and edit. Code patterns are used to offer developers autocomplete suggestions, list examples, and flag violations. To evaluate our technique, we conducted a user study in which 24 participants wrote, edited, and corrected HTML documents. We found that our technique enabled developers to edit and correct documents more quickly and create, edit, and correct documents more successfully.

Good documentation has long been argued to be key to helping developers write code more quickly and consistently with design decisions, but is left largely disconnected from code. We propose a method for active documentation, where design decisions are made explicit as design rules and checked against code. Developers can discover how to follow a design rule by navigating to examples in their codebase. After editing code, developers receive immediate feedback about which design rules are satisfied and which are violated, notifying developers who miss design decisions about the existence of these design decisions. We implemented our approach in a prototype tool and conducted a user study. Compared to developers using a traditional design document, developers working in an unfamiliar codebase with active documentation were faster and more successful, using active documentation to learn how to follow design decisions through examples and receive immediate feedback on their changes.

Traditional forms of Crowdsourcing such as open source software development harness crowd contributions to democratize the creation of software. However, potential contributors must first overcome joining barriers forcing casually committed contributors to spend days or weeks onboarding and thereby reducing participation. To more effectively harness potential contributions from the crowd, we propose a method for programming in which work occurs entirely through microtasks, offering contributors short, self-contained tasks such as implementing part of a function or updating a call site invoking a function to match a change made to the function. In microtask programming, microtasks involve changes to a single artifact, are automatically generated as necessary by the system, and nurture quality through iteration. A study examining the feasibility of microtask programming to create small programs found that developers were able to complete 1008 microtasks, onboard and submit their first microtask in less than 15 minutes, complete all types of microtasks in less than 5 minutes on average, and create 490 lines of code and 149 unit tests. The results demonstrate the potential feasibility as well as revealing a number of important challenges to address to successfully scale microtask programming to larger and more complex programs.

Crowdsourcing is increasingly revolutionizing the ways in which software is engineered. Programmers increasingly crowdsource answering their questions through Q&A sites. Non-programmers may contribute human-intelligence to development projects, by, for example, usability testing software or even play games with a purpose to implicitly construct formal specifications. Crowdfunding helps to democratize decisions about what software to build. Software engineering researchers may even benefit from new opportunities to evaluate their work with real developers by recruiting developers from the crowd. CSI- SE will inform the software engineering community of current techniques and trends in crowdsourcing, discuss the application of crowdsourcing to software engineering to date, and identify new opportunities to apply crowdsourcing to solve software engineering problems.

One form of crowdsourcing is the competition, which poses an open call for competing solutions. Commercial systems such as TopCoder have begun to explore the application of competitions to software development, but have important limitations diminishing the potential benefits drawn from the crowd. In particular, they employ a model of independent work that ignores the opportunity for designs to arise from the ideas of multiple designers. In this paper, we examine the potential for software design competitions to incorporate recombination, in which competing designers are given the designs of others and encouraged to use them to revise their own designs. To explore this, we conducted two software design competitions in which participants were asked to produce both an initial and a revised design, drawing on lessons learned from the crowd. We found that, in both competitions, all participants borrowed ideas and most improved the quality of their designs. Our findings demonstrate the potential benefits of recombination in software design and suggest several ways in which software design competitions can be improved.

Crowdsourcing has had extraordinary success in solving a diverse set of problems, ranging from digitization of libraries and translation of the Internet, to scientific challenges such as classifying elements in the galaxy or determining the 3D shape of an enzyme. By leveraging the power of the masses, it is feasible to complete tasks in mere days and sometimes even hours, and to take on tasks that were previously impossible because of their sheer scale. Underlying the success of crowdsourcing is a common theme - the microtask. By breaking down the overall task at hand into microtasks providing short, self-contained pieces of work, work can be performed independently, quickly, and in parallel - enabling numerous and often untrained participants to chip in. This paper puts forth a research agenda, examining the question of whether the same kinds of successes that microtask crowdsourcing is having in revolutionizing other domains can be brought to software development. That is, we ask whether it is possible to push well beyond the open source paradigm, which still relies on traditional, coarse-grained tasks, to a model in which programming proceeds through microtasks performed by vast numbers of crowd developers.

Programming today regularly involves searching for source code online, whether through a general search engine such as Google or a specialized code search engine such as SearchCode, Ohloh, or GitHub. Searching typically is an iterative process, with developers adjusting the keywords they use based on the results of the previous query. However, searching in this manner is not ideal, because just using keywords places limits on what developers can express as well as the overall interaction that is required. Based on the observation that the results from one query create a con-text in which a next is formulated, we present CodeExchange, a new code search engine that we developed to explicitly leverage this context to support fluid, expressive reformulation of queries. We motivate the need for CodeExchange, highlight its key design decisions and overall architecture, and evaluate its use in both a field deployment and a laboratory study.

Programming work is inherently interdependent, requiring developers to share and coordinate decisions that crosscut the structure of code. This is particularly challenging for programming in a microtasking context, in which developers are assumed to be transient and thus cannot rely on traditional learning and coordination mechanisms such as an extended onboarding process and code ownership. In this paper, we explore scaffolding coordination and knowledge sharing through a question and answer system, structuring project knowledge and coordination into questions and answers. To investigate its potential for enabling coordination in a microtask setting, we implemented a Q&A system for microtask programming work and conducted a user study where a crowd used it to coordinate their work on a software project over a 30-hour period. The results reveal both the potential for the use of Q&A systems for within project coordination and challenges that this approach brings.

Whiteboard sketches play a crucial role in software development, helping to support groups of designers in reasoning about a software design problem at hand. However, little is known about these sketches and how they support design ‘in the moment’, particularly in terms of the relationships among sketches, visual syntactic elements within sketches, and reasoning activities. To address this gap, we analyzed 14 hours of design activity by eight pairs of professional software designers, manually coding over 4000 events capturing the introduction of visual syntactic elements into sketches, focus transitions between sketches, and reasoning activities. Our findings indicate that sketches serve as a rich medium for supporting design conversations. Designers often use general-purpose notations. Designers introduce new syntactic elements to record aspects of the design, or re-purpose sketches as the design develops. Designers constantly shift focus between sketches, using groups of sketches together that contain complementary information. Finally, sketches play an important role in supporting several types of reasoning activities (mental simulation, review of progress, consideration of alternatives). But these activities often leave no trace and rarely lead to sketch creation. We discuss the implications of these and other findings for the practice of software design at the whiteboard and for the creation of new electronic software design sketching tools.

Whiteboards serve an important role in supporting informal design, providing a fluid and flexible medium for collaborative design. Interactive whiteboards offer the potential for enhanced support for manipulating content, managing sketches, and distributed work, but little is known about how this support affects the practice of informal design. To understand the opportunities and challenges, we first conducted a literature review, identifying 14 behaviors that occur during informal design. We then designed an interactive whiteboard system to support all of these behaviors and deployed the system to three groups of designers. Through usage logs and interviews, we examined the effects of interactivity on whiteboard use across a wide spectrum of design behaviors, identifying ways in which interactive whiteboards support the practices used in physical whiteboards and where they enable designers to work more effectively.

Microtask crowdsourcing organizes complex work into workflows, decomposing large tasks into small, relatively independent microtasks. Applied to software development, this model might increase participation in open source software development by lowering the barriers to contribu-tion and dramatically decrease time to market by increasing the parallelism in development work. To explore this idea, we have developed an approach to decomposing programming work into microtasks. Work is coordinated through tracking changes to a graph of artifacts, generating appropriate microtasks and propagating change notifications to artifacts with dependencies. We have implemented our approach in CrowdCode, a cloud IDE for crowd development. To evaluate the feasibility of microtask programming, we performed a small study and found that a small crowd of 12 workers was able to successfully write 480 lines of code and 61 unit tests in 14.25 person-hours of time.

The use of a studio approach - a hands-on teaching method that emphasizes in-class discussion and activities - is becoming an increasingly accepted method of teaching within software engineering. In such studios, emphasis is placed not only on the artifacts to be produced, but also on the process used to arrive at those artifacts. In this paper, we introduce Calico, a sketch-based collaborative software design tool, and discuss how it supports the delivery of a studio approach to software design education. We particularly describe our experiences with Calico in Software Design I, a course aimed at introducing students to the early, creative phases of software design. Our results show that Calico enabled students to work effectively in teams on their design problems, quickly developing, refining, and evaluating their designs.

Code completion menus have replaced standalone API browsers for most developers because they are more tightly integrated into the development workflow. Refinements to the code completion menu that incorporate additional sources of information have similarly been shown to be valuable, even relative to standalone counterparts offering similar functionality. In this paper, we describe active code completion, an architecture that allows library developers to introduce interactive and highly-specialized code generation interfaces, called palettes, directly into the editor. Using several empirical methods, we examine the contexts in which such a system could be useful, describe the design constraints governing the system architecture as well as particular code completion interfaces, and design one such system, named Graphite, for the Eclipse Java development environment. Using Graphite, we implement a palette for writing regular expressions as our primary example and conduct a small pilot study. In addition to showing the feasibility of this approach, it provides further evidence in support of the claim that integrating specialized code completion interfaces directly into the editor is valuable to professional developers.

Developers navigate and reason about call graphs throughout investigation and debugging activities. This is often difficult: developers can spend tens of minutes answering a single question, get lost and disoriented, and erroneously make assumptions, causing bugs. To address these problems, we designed a new form of interactive call graph visualization - REACHER. Instead of leaving developers to manually traverse the call graph, REACHER lets developers search along control flow. The interactive call graph visualization encodes a number of properties that help developers answer questions about causality, ordering, type membership, repetition, choice, and other relationships. And developers remain oriented while navigating. To evaluate REACHER'S benefits, we conducted a lab study in which 12 participants answered control flow questions. Compared to an existing IDE, participants with REACHER were over 5 times more successful in significantly less time. All enthusiastically preferred REACHER, with many positive comments.

In this paper, we propose a complementary technique called active code completion. When the developer invokes the code completion menu, the editor looks for a palette definition associated with the type of the expression being entered. If found, an option to use this palette is added to the code completion menu. When the developer selects this option, source code is not inserted immediately. Instead, the palette definition takes control of the code completion interface. The developer can then interact with this interface to provide parameters and other information related to her intent, and receive immediate feedback about the effect these choices will have on the object's behavior. When the developer indicates that she is satisfied with these choices, the palette generates code that is inserted at the cursor.

A reachability question is a search across feasible paths through a program for target statements matching search criteria. In three separate studies, we found that reachability questions are common and often time consuming to answer. In the first study, we observed 13 developers in the lab and found that half of the bugs developers inserted were associated with reachability questions. In the second study, 460 professional software developers reported asking questions that may be answered using reachability questions more than 9 times a day, and 82% rated one or more as at least somewhat hard to answer. In the third study, we observed 17 developers in the field and found that 9 of the 10 longest activities were associated with reachability questions. These findings suggest that answering reachability questions is an important source of difficulty understanding large, complex codebases.

Little is known about how developers think about design during code modification tasks or how experienced developers' design knowledge helps them work more effectively. We performed a lab study in which thirteen developers worked for 3 hours under-standing the design of a 54 KLOC open source application. Par-ticipants had from 0 to 10.5 years of industry experience and were grouped into three "experts" and ten "novices." We observed that participants spent their time seeking, learning, critiquing, explain-ing, proposing, and implementing facts about the code such as "getFoldLevel has effects". These facts served numerous roles, such as suggesting changes, constraining changes, and predicting the amount of additional investigation necessary to make a change. Differences between experts and novices included that the experts explained the root cause of the design problem and made changes to address it, while novice changes addressed only the symptoms. Experts did not read more methods but also did not visit some methods novices wasted time understanding. Experts talked about code in terms of abstractions such as "caching" while novices more often described code statement by statement. Ex-perts were able to implement a change faster than novices. Experts perceived problems novices did not and were able to explain facts novices could not. These findings have interesting implications for future tools.

To understand developers' typical tools, activities, and practices and their satisfaction with each, we conducted two surveys and eleven interviews. We found that many problems arose because developers were forced to invest great effort recovering implicit knowledge by exploring code and interrupting teammates and this knowledge was only saved in their memory. Contrary to expectations that email and IM prevent expensive task switches caused by face-to-face interruptions, we found that face-to-face communication enjoys many advantages. Contrary to expectations that documentation makes understanding design rationale easy, we found that current design documents are inadequate. Contrary to expectations that code duplication involves the copy and paste of code snippets, developers reported several types of duplication. We use data to characterize these and other problems and draw implications for the design of tools for their solution.