Documentation enables sharing knowledge between the developers of a technology and its users. Creating quality documents, however, is challenging: Documents must satisfy the needs of a large audience without being overwhelming for individuals. We address this challenge with a new document format named Casdoc. Casdoc documents are interactive resources centered around code examples for programmers. Explanations of the code elements are presented as annotations that the readers reveal based on their needs. We evaluated Casdoc in a field study with over 300 participants who used 126 documents as part of a software design course. During the study, the majority of participants adopted Casdoc instead of a baseline format and used interactive annotations to reveal additional information about the code example. Although participants collectively viewed the majority of the documents’ content, they individually revealed a minority of the annotations they saw. We gathered insights into five aspects of Casdoc that can be applied to other formats and highlighted five lessons learned to improve navigability in online documents.

Unit tests must be readable to help developers understand and evolve production code. Most existing test quality metrics assess test code’s ability to detect bugs. Few metrics focus on test code’s readability. One standard approach to improve readability is the consistent application of conventions. We investigated test convention consistency as a dimension of test quality. We formalized test suite consistency as the extent to which alternatives are used within a code base and introduce two complementary metrics to capture this extent. We elaborated a catalog of over 30 test conventions for the Java language organized in 10 convention classes that group mutual alternatives. We developed tool support to detect occurrences of conventions, compute consistency metrics over a test suite, and view occurrences of conventions in the corresponding code. We applied our tools to study the consistency of the test suites of 20 large open source Java projects. The study validates the design of the test convention classes, provides descriptive statistics on the range of consistency values for 10 different convention classes, and enables us to link observed changes in consistency values to specific events in the change history of our target systems, thus providing evidence of the construct validity of the metrics. We conclude that analyzing test suite consistency via static analysis shows promise as a practical approach to help improve test suite quality.

Reflecting on the limitations of a study is a crucial part of the research process. In software engineering studies, this reflection is typically conveyed through discussions of study limitations or threats to validity. In current practice, such discussions seldom provide sufficient insight to understand the rationale for decisions taken before and during the study, and their implications. We revisit the practice of discussing study limitations and threats to validity and identify its weaknesses. We propose to refocus this practice of self-reflection to a discussion centered on the notion of trade-offs . We argue that documenting trade-offs allows researchers to clarify how the benefits of their study design decisions outweigh the costs of possible alternatives. We present guidelines for reporting trade-offs in a way that promotes a fair and dispassionate assessment of researchers’ work.

A large number of tutorials for popular software development technologies are available online, and those about the same technology vary widely in their presentation. We studied the design of tutorials in the software documentation landscape for five popular programming languages: Java, C#, Python, Javascript, and Typescript. We investigated the extent to which tutorial pages, i.e.resources, differ and report statistics of variations in resource properties. We developed a framework for characterizing resources based on theirdistinguishing attributes, i.e. properties that vary widely for the resource, relative to other resources. Additionally, we propose that a resource can be represented by itsresource style, i.e. the combination of its distinguishing attributes. We discuss three techniques for characterizing resources based on our framework, to capture notable and relevant content and presentation properties of tutorial pages. We apply these techniques on a data set of 2551 resources to validate that our framework identifies valid and interpretable styles. We contribute this framework for reasoning about the design of resources in the online software documentation landscape.

When working on a project, software developers must be familiar with computing concepts, standards, and technologies related to the project. We present a novel approach, called Scode, to automatically identify those concepts using the project's documentation. Scode combines entity linking and network analysis techniques specialized for the software development domain. In addition to concepts explicitly mentioned in the documentation, Scode can retrieve implicit concepts related to the project's domain. Concepts identified by Scode have a recognized meaning that is consistent across projects. We compared Scode to different baselines and found that it is more effective at mapping projects to a consistent concept space.

Code examples are of great value to programmers trying to learn an unfamiliar API. Effective code examples are often surrounded with plain text explanations of the relevant concepts, techniques, and API elements involved in the example. However, authoring concise yet complete explanations is a challenging balancing act. To address this challenge, we propose Casdoc, a novel authoring technique and presentation format for annotated code examples. Casdoc-formatted code examples are HTML documents designed to embed unobtrusive explanations into the code. They thus contain more explanations to address the varying needs of a larger audience, without disrupting individual readers with information they already know. Explanations are split into short annotations and organized into an intuitive tree-like structure, thus supporting a streamlined authoring process. We used Casdoc to produce 105 Java code examples as part of the course material for an undergraduate computer science course. Students preferred the new format over traditional code examples. Their interaction with code examples suggests that the intuitive structure of Casdoc annotations reduces the need for navigation aids such as search fields.

Software engineering is knowledge-intensive and requires software developers to continually search for knowledge, often on community question answering platforms such as Stack Overflow. Such information sharing platforms do not exist in isolation, and part of the evidence that they exist in a broader software documentation ecosystem is the common presence of hyperlinks to other documentation resources found in forum posts. With the goal of helping to improve the information diffusion between Stack Overflow and other documentation resources, we conducted a study to answer the question of how and why documentation is referenced in Stack Overflow threads. We sampled and classified 759 links from two different domains, regular expressions and Android development, to qualitatively and quantitatively analyze the links’ context and purpose, including attribution, awareness, and recommendations. We found that links on Stack Overflow serve a wide range of distinct purposes, ranging from citation links attributing content copied into Stack Overflow, over links clarifying concepts using Wikipedia pages, to recommendations of software components and resources for background reading. This purpose spectrum has major corollaries, including our observation that links to documentation resources are a reflection of the information needs typical to a technology domain. We contribute a framework and method to analyze the context and purpose of Stack Overflow links, a public dataset of annotated links, and a description of five major observations about linking practices on Stack Overflow. Those observations include the above-mentioned purpose spectrum, its interplay with documentation resources and applications domains, and the fact that links on Stack Overflow often lack context in form of accompanying quotes or summaries. We further point to potential tool support to enhance the information diffusion between Stack Overflow and other documentation resources.

Software projects capture redundant information in various kinds of artifacts, as specifications from the source code are also tested and documented. Such redundancy provides an opportunity to reduce development effort by supporting the joint generation of different types of artifacts. We introduce a tool-supported technique, called DScribe, that allows developers to combine unit test and documentation templates, and to invoke these templates to generate documentation and unit tests. DScribe supports the detection and replacement of outdated documentation, and the use of templates can encourage extensive test suites with a consistent style. Our evaluation of 835 specifications revealed that 85 percent were not tested or correctly documented, and DScribe could be used to automatically generate 97 percent of the tests and documentation. An additional study revealed that tests generated by DScribe are more focused and readable than those written by human testers or generated by state-of-the-art automated techniques.

When contributors to a software project leave, the knowledge they hold may become lost, thus impacting code quality and team productivity. Although well-known strategies can be used to mitigate knowledge loss, these strategies have to be tailored to their target context to be effective. To help software development organizations mitigate turnover-induced knowledge loss, we sought to better understand the different contexts in which developers experience this knowledge loss, and the resulting implications. We conducted qualitative interviews with 27 professional developers and managers from three different companies that provide software products and services. Leveraging the experience of these practitioners, we contribute a framework for characterizing turnover-induced knowledge loss and descriptions of the implications of knowledge loss, synthesized into 20 observations. These observations about knowledge loss in practice are organized into four themes, validated by the participants, and discussed within the context of the research literature in software engineering.

Posts on on-line forums where programmers look for information often include links to Wikipedia when it can be assumed the reader will not be familiar with the linked terms. A Wikipedia article will thus often be the first exposure to a new computing concept for a novice programmer. We conducted an exploratory study with 18 novice programmers by asking them to read a Wikipedia article on a common computing concept that was new to them, while using the think-aloud protocol. We performed a qualitative analysis of the session transcripts to better understand the experience of the novice programmer learning a new computing concept using Wikipedia. We elicited five themes that capture this experience: Concept Confusion, Need for Examples, New Terminology, Trivia Clutter, and Unfamiliar Notation. We conclude that Wikipedia is not well suited as a resource for the opportunistic learning of new computing concepts, and we recommend adapting information sharing practices in on-line programmer communities to better account for the learning needs of the users.

Informal language and the absence of a standard taxonomy for software technologies make it difficult to reliably analyze technology trends on discussion forums and other on-line venues. We propose an automated approach called Witt for the categorization of software technologies (an expanded version of the hypernym discovery problem). Witt takes as input a phrase describing a software technology or concept and returns a general category that describes it (e.g., integrated development environment), along with attributes that further qualify it (commercial, php, etc.). By extension, the approach enables the dynamic creation of lists of all technologies of a given type (e.g., web application frameworks). Our approach relies on Stack Overflow and Wikipedia, and involves numerous original domain adaptations and a new solution to the problem of normalizing automatically-detected hypernyms. We compared Witt with six independent taxonomy tools and found that, when applied to software terms, Witt demonstrated better coverage than all evaluated alternative solutions, without a corresponding degradation in false positive rate.

This artifact is a data set generated as part of a study on the threats of aggregating software repository data, which includes information derived from the GitHub repositories of eight open-source projects.

This artifact is a data set generated as part of a study on the threats of aggregating software repository data, which includes information derived from the GitHub repositories of eight open-source projects.

In large software projects, tacit knowledge of the system is threatened by developer turnover. When a developer leaves the project, their knowledge may be lost if the other developers do not understand the design decisions made by the leaving developer. Understanding the source code written by leaving developers thus becomes a burden for their successors. In a previous paper, Rigby et al. reported on a case study of turnover-induced knowledge loss in two large projects, Chromium and a project at Avaya, using risk evaluation methods usually applied to financial systems. They found that the two projects were susceptible to large knowledge losses that are more than three times the average loss. We report on a replication of their study on the Chromium project, as well as seven other large and medium-sized open source projects. We also extended theirwork by studying two variations of the knowledge loss metric, as well as the location and persistence of abandoned files. We found that all projects had a similar knowledge loss probability distribution, but extreme knowledge loss can be more severe than those originally discovered in Chromium and the project at Avaya. We also found that, in the systems under study, abandoned files often remained in the system for long periods.

We advocate for a paradigm shift in supporting the information needs of developers, centered around the concept of automated on-demand developer documentation. Currently, developer information needs are fulfilled by asking experts or consulting documentation. Unfortunately, traditional documentation practices are inefficient because of, among others, the manual nature of its creation and the gap between the creators and consumers. We discuss the major challenges we face in realizing such a paradigm shift, highlight existing research that can be leveraged to this end, and promote opportunities for increased convergence in research on software documentation.

Code fragments posted in answers on Q&A forums can form an important source of developer knowledge. However, effective reuse of code fragments found online often requires information other than the code fragment alone. We report on the results of a survey-based study to investigate to what extent developers perceive Stack Overflow code fragments to be self-explanatory. As part of the study, we also investigated the types of information missing from fragments that were not self-explanatory. We find that less than half of the Stack Overflow code fragments in our sample are considered to be self-explanatory by the 321 participants who answered our survey, and that the main issues that negatively affect code fragment understandability include incomplete fragments, code quality, missing rationale, code organization, clutter, naming issues, and missing domain information. This study is a step towards understanding developers' information needs as they relate to code fragments, and how these needs can be addressed.

Refactoring is a common software development practice and many simple refactorings can be performed automatically by tools. Identifier renaming is a widely performed refactoring activity. With tool support, rename refactorings can rely on the program structure to ensure correctness of the code transformation. Unfortunately, the textual references to the renamed identifier present in the unstructured comment text cannot be formally detected through the syntax of the language, and are thus fragile with respect to identifier renaming. We designed a new rule-based approach to detect fragile comments. Our approach, called Fraco, takes into account the type of identifier, its morphology, the scope of the identifier and the location of comments. We evaluated the approach by comparing its precision and recall against hand-annotated benchmarks created for six target Java systems, and compared the results against the performance of Eclipse's automated in-comment identifier replacement feature. Fraco performed with near-optimal precision and recall on most components of our evaluation data set, and generally outperformed the baseline Eclipse feature. As part of our evaluation, we also noted that more than half of the total number of identifiers in our data set had fragile comments after renaming, which further motivates the need for research on automatic comment refactoring.

Generating and maintaining REST API documentation with usage examples can be a time consuming and expensive process for evolving APIs. Most REST API documentation tools focus on automating the documentation of the API objects, but require manual effort for capturing usage examples. Consequently, REST API developers need to know the cost vs. benefit of providing usage examples in the documentation to prioritize the documentation efforts. To this end, we have performed a controlled study with 26 experienced software engineers to understand problems that REST API client developers face while using an API without usage examples. We found that REST API client developers face productivity problems with using correct data types, data formats, required HTTP headers and request body when documentation lacks usage examples. By following the REST API documentation suggestions from this paper, REST API developers can reduce the errors, improve success rate and satisfaction of API client developers.

This paper reports on an interview-based study of 18 authors of different chapters of the two-volume book "Architecture of Open-Source Applications". The main contributions are a synthesis of the process of authoring essay-style documents (ESDs) on software architecture, a series of observations on important factors that influence the content and presentation of architectural knowledge in this documentation form, and a set of recommendations for readers and writers of ESDs on software architecture. We analyzed the influence of three factors in particular: the evolution of a system, the community involvement in the project, and the personal characteristics of the author. This study provides the first systematic investigation of the creation of ESDs on software architecture. The observations we collected have implications for both readers and writers of ESDs, and for architecture documentation in general.

Software developers need access to different kinds of information which is often dispersed among different documentation sources, such as API documentation or Stack Overflow. We present an approach to automatically augment API documentation with "insight sentences" from Stack Overflow---sentences that are related to a particular API type and that provide insight not contained in the API documentation of that type. Based on a development set of 1,574 sentences, we compare the performance of two state-of-the-art summarization techniques as well as a pattern-based approach for insight sentence extraction. We then present SISE, a novel machine learning based approach that uses as features the sentences themselves, their formatting, their question, their answer, and their authors as well as part-of-speech tags and the similarity of a sentence to the corresponding API documentation. With SISE, we were able to achieve a precision of 0.64 and a coverage of 0.7 on the development set. In a comparative study with eight software developers, we found that SISE resulted in the highest number of sentences that were considered to add useful information not found in the API documentation. These results indicate that taking into account the meta data available on Stack Overflow as well as part-of-speech tags can significantly improve unsupervised extraction approaches when applied to Stack Overflow data.

Although design plays a central role in software development, the information produced in this activity is often left to progressively evaporate as the result of software evolution, loss of artifacts, or the fading of related knowledge held by the development team. This paper introduces the concept of sustainability for software design, and calls for its integration into the existing catalog of design quality attributes. Applied to software design, sustainability conveys the idea that a particular set of design decisions and their rationale can be succinctly reflected in the host technology and/or described in documentation in a way that is checkable for conformance with the code and generally resistant to evaporation. The paper discusses the relation between sustainability and existing research areas in software engineering, and highlights future research challenges related to sustainable software design.

Current software projects deal with exceptions in implementation and maintenance phases without a clear definition of exception handling policies. We call an exception handling policy the set of design decisions that govern the use of exceptions in a software project. Without an explicit exception handling policy, developers can remain unaware of the originally intended use of exceptions. In this paper, we present Exception Handling Policies Language (EPL), a domain-specific language to specify and verify exception handling policies. The evaluation of EPL was based on a user-centric observational study and case studies. The user-centric study was performed to observe how potential users of the language actually use it. With this study, we could better understand the trade-offs related to different language design decisions based on concrete and well-documented observations and experiences reported by participants. We identified some language characteristics that hindered its use and that motivated new language constructs. In addition, we performed case studies with one open-source project and two industry-strength systems to investigate how specifying and verifying exception handling policies may assist in detecting exception handling problems. The results show that violations of exception handling policies help to indicate potential faults in the exception handling code.

Many software development tasks require developers to quickly learn a subset of an Application Programming Interface (API). API learning resources are crucial for helping developers learn an API, but the knowledge relevant to a particular topic of interest may easily be scattered across different documents, which makes finding the necessary information more challenging. This paper proposes an approach to discovering tutorial sections that explain a given API type. At the core of our approach, we classify fragmented tutorial sections using supervised text classification based on linguistic and structural features. Experiments conducted on five tutorials show that our approach is able to discover sections explaining an API type with precision between 0.69 and 0.87 (depending on the tutorial) when trained and tested on the same tutorial. When trained and tested across tutorials, we obtained a precision between 0.74 and 0.94 and lower recall values.

To help developers navigate documentation, we introduce Task Nav, a tool that automatically discovers and indexes task descriptions in software documentation. With Task Nav, we conceptualize tasks as specific programming actions that have been described in the documentation. Task Nav presents these extracted task descriptions along with concepts, code elements, and section headers in an auto-complete search interface. Our preliminary evaluation indicates that search results identified through extracted task descriptions are more helpful to developers than those found through other means, and that they help bridge the gap between documentation structure and the information needs of software developers. Video: https://www.youtube.com/watch?v=opnGYmMGnqY.

Knowledge management plays a central role in many software development organizations. While much of the important technical knowledge can be captured in documentation, there often exists a gap between the information needs of software developers and the documentation structure. To help developers navigate documentation, we developed a technique for automatically extracting tasks from software documentation by conceptualizing tasks as specific programming actions that have been described in the documentation. More than 70 percent of the tasks we extracted from the documentation of two projects were judged meaningful by at least one of two developers. We present TaskNavigator, a user interface for search queries that suggests tasks extracted with our technique in an auto-complete list along with concepts, code elements, and section headers. We conducted a field study in which six professional developers used TaskNavigator for two weeks as part of their ongoing work. We found search results identified through extracted tasks to be more helpful to developers than those found through concepts, code elements, and section headers. The results indicate that task descriptions can be effectively extracted from software documentation, and that they help bridge the gap between documentation structure and the information needs of software developers.

Code examples are an important source for answering questions about software libraries and applications. Many usage contexts for code examples require them to be distilled to their essence: e.g., when serving as cues to longer documents, or for reminding developers of a previously known idiom. We conducted a study to discover how code can be summarized and why. As part of the study, we collected 156 pairs of code examples and their summaries from 16 participants, along with over 26 hours of think-aloud verbalizations detailing the decisions of the participants during their summarization activities. Based on a qualitative analysis of this data we elicited a list of practices followed by the participants to summarize code examples and propose empirically-supported hypotheses justifying the use of specific practices. One main finding was that none of the participants exclusively extracted code verbatim for the summaries, motivating abstractive summarization. The results provide a grounded basis for the development of code example summarization and presentation technology.

Developer documentation helps developers learn frameworks and libraries, yet developing and maintaining accurate documentation requires considerable effort and resources. Contributors who work on developer documentation often need to manually track all changes in the code, determine which changes are significant enough to document, and then, adapt the documentation. We propose AdDoc, a technique that automatically discovers documentation patterns, i.e., coherent sets of code elements that are documented together, and that reports violations of these patterns as the code and the documentation evolves. We evaluated our approach in a retrospective analysis of four Java open source projects and found that at least 50 percent of all the changes in the documentation were related to existing documentation patterns. Our technique allows contributors to quickly adapt existing documentation, so that they can focus their documentation effort on the new features.

To access the knowledge contained in developer communication, such as forum posts, it is useful to determine automatically the code elements referred to in the discussions. We propose a novel traceability recovery approach to extract the code elements contained in various documents. As opposed to previous work, our approach does not require an index of code elements to find links, which makes it particularly well-suited for the analysis of informal documentation. When evaluated on 188 StackOverflow answer posts containing 993 code elements, the technique performs with average 0.92 precision and 0.90 recall. As a major refinement on traditional traceability approaches, we also propose to detect which of the code elements in a document are salient, or germane, to the topic of the post. To this end we developed a three-feature decision tree classifier that performs with a precision of 0.65-0.74 and recall of 0.30-0.65, depending on the subject of the document.

Current research in software engineering has mostly focused on the retrieval accuracy aspect but little on the presentation aspect of code examples, e.g., how code examples are presented in a result page. We investigate the feasibility of summarizing code examples for better presenting a code example. Our algorithm based on machine learning could approximate summaries in an oracle manually generated by humans with a precision of 0.71. This result is promising as summaries with this level of precision achieved the same level of agreement as human annotators with each other.

Reading reference documentation is an important part of programming with application programming interfaces (APIs). Reference documentation complements the API by providing information not obvious from the API syntax. To improve the quality of reference documentation and the efficiency with which the relevant information it contains can be accessed, we must first understand its content. We report on a study of the nature and organization of knowledge contained in the reference documentation of the hundreds of APIs provided as a part of two major technology platforms: Java SDK 6 and .NET 4.0. Our study involved the development of a taxonomy of knowledge types based on grounded methods and independent empirical validation. Seventeen trained coders used the taxonomy to rate a total of 5,574 randomly sampled documentation units to assess the knowledge they contain. Our results provide a comprehensive perspective on the patterns of knowledge in API documentation: observations about the types of knowledge it contains and how this knowledge is distributed throughout the documentation. The taxonomy and patterns of knowledge we present in this paper can be used to help practitioners evaluate the content of their API documentation, better organize their documentation, and limit the amount of low-value content. They also provide a vocabulary that can help structure and facilitate discussions about the content of APIs.

Frameworks and libraries offer reusable and customizable functionality through Application Programming Interfaces (APIs). Correctly using large and sophisticated APIs can represent a challenge due to hidden assumptions and requirements. Numerous approaches have been developed to infer properties of APIs, intended to guide their use by developers. With each approach come new definitions of API properties, new techniques for inferring these properties, and new ways to assess their correctness and usefulness. This paper provides a comprehensive survey of over a decade of research on automated property inference for APIs. Our survey provides a synthesis of this complex technical field along different dimensions of analysis: properties inferred, mining techniques, and empirical results. In particular, we derive a classification and organization of over 60 techniques into five different categories based on the type of API property inferred: unordered usage patterns, sequential usage patterns, behavioral specifications, migration mappings, and general information.

Large frameworks and libraries require extensive developer learning resources, such as documentation and mailing lists, to be useful. Maintaining these learning resources is challenging partly because they are not explicitly linked to the frameworks' API, and changes in the API are not reflected in the learning resources. Automatically recovering traceability links between an API and learning resources is notoriously difficult due to the inherent ambiguity of unstructured natural language. Code elements mentioned in documents are rarely fully qualified, so readers need to understand the context in which a code element is mentioned. We propose a technique that identifies code-like terms in documents and links these terms to specific code elements in an API, such as methods. In an evaluation study with four open source systems, we found that our technique had an average recall and precision of 96%.

The increasing size of APIs and the increase in the number of APIs available imply developers must frequently learn how to use unfamiliar APIs. To identify the types of questions developers want answered when working with unfamiliar APIs and to understand the difficulty they may encounter answering those questions, we conducted a study involving twenty programmers working on different programming tasks, using unfamiliar APIs. Based on the screen captured videos and the verbalization of the participants, we identified twenty different types of questions programmers ask when working with unfamiliar APIs, and provide new insights to the cause of the difficulties programmers encounter when answering questions about the use of APIs. The questions we have identified and the difficulties we observed can be used for evaluating tools aimed at improving API learning, and in identifying areas of the API learning process where tool support is missing, or could be improved.

Software reuse through Application Programming Interfaces (APIs) is an integral part of software development. The functionality offered by an API is not always accessed uniformly throughout the lifetime of a client program. We propose Temporal API Usage Pattern Mining to detect API usage patterns in terms of their time of introduction into client programs. We detect concepts as distinct groups of API functionality from the change history of a client program. We locate those concepts in the client change history and detect temporal usage patterns, where a pattern contains a set of concepts that were added into the client program in a specific temporal order. We investigated the properties of temporal API usage patterns through a multiple-case study of three APIs and their use in up to 19 client software projects. Our technique was able to detect a number of valuable patterns in two out of three of the APIs investigated. Further investigation showed some patterns to be relatively consistent between clients, produced by multiple developers, and not trivially derivable from program structure or API documentation.

Numerous techniques involve mining change data captured in software archives to assist engineering efforts, for example to identify components that tend to evolve together. We observed that important changes to software artifacts are sometimes accompanied by numerous non-essential modifications, such as local variable refactorings, or textual differences induced as part of a rename refactoring. We developed a tool-supported technique for detecting non-essential code differences in the revision histories of software systems. We used our technique to investigate code changes in over 24,000 change sets gathered from the change histories of seven long-lived open-source systems. We found that up to 15.5% of a system's method updates were due solely to non-essential differences. We also report on numerous observations on the distribution of non-essential differences in change history and their potential impact on change-based analyses.

Programmers performing a change task must understand the existing software in addition to performing the actual change. This process is likely to be affected by characteristics of the task. We investigated whether the nature of a task has any relationship with when a programmer edits code during a programming session. We characterized differences in editing behaviour with three types of editing styles: edit-first, edit-last, and edit-throughout. We based our analysis on the interaction history of over 4000 programming sessions collected as part of the development history of open source projects. Our results showed that an enhancement task (as opposed to a bug fix) was less likely to be associated with a high fraction of source code edit events at the beginning of the programming session. To our surprise, we also found that the presence of a stack trace in a bug report did not significantly effect the editing style of the programming session.

Software reuse through Application Programming Interfaces (APIs) is an integral part of software development. As developers write client programs, their understanding and usage of APIs change over time. Can we learn from long-term changes in how developers work with APIs in the lifetime of a client program? We propose Temporal API Usage Mining to detect significant changes in API usage. We describe a framework to extract detailed models representing addition and removal of calls to API methods over the change history of a client program. We apply machine learning technique to these models to semi-automatically infer temporal API usage patterns, i.e., coherent addition of API calls at different phases in the life-cycle of the client program.

In the course of a framework’s evolution, changes ranging from a simple refactoring to a complete rearchitecture can break client programs. Finding suitable replacements for framework elements that were accessed by a client program and deleted as part of the framework’s evolution can be a challenging task. We present a recommendation system, SemDiff, that suggests adaptations to client programs by analyzing how a framework was adapted to its own changes. In a study of the evolution of one open source framework and three client programs, our approach recommended relevant adaptive changes with a high level of precision. In a second study of the evolution of two frameworks, we found that related change detection approaches were better at discovering systematic changes and that SemDiff was complementary to these approaches by detecting non-trivial changes such as when a functionality is imported from an external library.

When developers join a software development project, they find themselves in a project landscape, and they must become familiar with the various landscape features. To better understand the nature of project landscapes and the integration process, with a view to improving the experience of both newcomers and the people responsible for orienting them, we performed a grounded theory study with 18 newcomers across 18 projects. We identified the main features that characterize a project landscape, together with key orientation aids and obstacles, and we theorize that there are three primary factors that impact the integration experience of newcomers: early experimentation, internalizing structures and cultures, and progress validation.

The goal of this one-day workshop is to bring together researchers and practitioners with interest and experience in the theory, elaboration, and evaluation of concepts, techniques, and tools for providing recommendations to developers, managers, and other stakeholders involved in software engineering tasks.

Developer documentation helps developers learn frameworks and libraries. To better understand how documentation in open source projects is created and maintained, we performed a qualitative study in which we interviewed core contributors who wrote developer documentation and developers who read documentation. In addition, we studied the evolution of 19 documents by analyzing more than 1500 document revisions. We identified the decisions that contributors make, the factors influencing these decisions and the consequences for the project. Among many findings, we observed how working on the documentation could improve the code quality and how constant interaction with the projects' community positively impacted the documentation.

Abstract During software maintenance tasks, developers often spend a valuable amount of effort investigating source code. This effort can be reduced if tools are available to help developers navigate the source code effectively. We studied to what extent developers can benefit from information contained in clusters of change sets to guide their investigation of a software system. We defined change clusters as groups of change sets that have a certain amount of elements in common. Our analysis of 4200 change sets for seven different systems and covering a cumulative time span of over 17 years of development showed that less than one in five tasks overlapped with change clusters. Furthermore, a detailed qualitative analysis of the results revealed that only 13% of the clusters associated with applicable change tasks were likely to be useful. We conclude that change clusters can only support a minority of change tasks, and should only be recommended if it is possible to do so at minimal cost to the developers. Copyright © 2009 John Wiley & Sons, Ltd.

Source code duplication, commonly known as code cloning , is considered an obstacle to software maintenance because changes to a cloned region often require consistent changes to other regions of the source code. Research has provided evidence that the elimination of clones may not always be practical, feasible, or cost-effective. We present a clone management approach that describes clone regions in a robust way that is independent from the exact text of clone regions or their location in a file, and that provides support for tracking clones in evolving software. Our technique relies on the concept of abstract clone region descriptors (CRDs), which describe clone regions using a combination of their syntactic, structural, and lexical information. We present our definition of CRDs, and describe a clone tracking system capable of producing CRDs from the output of different clone detection tools, notifying developers of modifications to clone regions, and supporting updates to the documented clone relationships. We evaluated the performance and usefulness of our approach across three clone detection tools and five subject systems, and the results indicate that CRDs are a practical and robust representation for tracking code clones in evolving software.

Research has provided evidence that type usage in source files is correlated with the risk of failure of software components. Previous studies that investigated the correlation between type usage and component failure assigned equal blame to all the types imported by a component with a failure history, regardless of whether a type is used in the component, or associated to its failures. A failure-prone component may use a type, but it is not always the case that the use of this type has been responsible for any of its failures. To gain more insight about the correlation between type usage and component failure, we introduce the concept of a failure-associated type to represent the imported types referenced within methods fixed due to failures. We conducted two studies to investigate the tradeoffs between the equal-blame approach and the failure-associated type approach. Our results indicate that few of the types or packages imported by a failure-prone component are associated with its failures - less than 25% of the type imports, and less than 55% of the packages whose usage were reported to be highly correlated with failures by the equal-blame approach, were actually correlated with failures when we looked at the failure-associated types.

As a framework evolves, changes in its application programming interface (API) can break client programs that extend the framework. Repairing a client program can be a challenging task because developers need to understand the context surrounding the API change. This paper describes SemDiff, a tool that recommends replacements for framework methods that were accessed by a client program and deleted during the evolution of the framework. SemDiff recommends replacements for non-trivial changes undiscovered by other change-detection techniques and also enables developers to look at the context of the changes that led to the deletion of a framework method.

Planning a complex software modification task imposes a high cognitive burden on developers, who must juggle navigating the software, understanding what they see with respect to their task, and deciding how their task should be performed given what they have discovered. Pragmatic reuse tasks, where source code is reused in a white-box fashion, is an example of a complex and error-prone modification task: the developer must plan out which portions of a system to reuse, extract the code, and integrate it into their own system. In this paper we present a recommendation system that automates some aspects of the planning process undertaken by developers during pragmatic reuse tasks. In a retroactive evaluation, we demonstrate that our technique was able to provide the correct recommendation 64% of the time and was incorrect 25% of the time. Our case study suggests that developer investigative behaviour is positively influenced by the use of the recommendation system.

Software projects often rely on third-party libraries made accessible through Application Programming Interfaces (APIs). We have observed many cases where APIs are used in ways that are not the most effective. We developed a technique and tool support to automatically detect such patterns of API usage in software projects. The main hypothesis underlying our technique is that client code imitating the behavior of an API method without calling it may not be using the API effectively because it could instead call the method it imitates. Our technique involves analyzing software systems to detect cases of API method imitations. In addition to warning developers of potentially re-implemented API methods, we also indicate how to improve the use of the API. Applying our approach on 10 Java systems revealed over 400 actual cases of potentially suboptimal API usage, leading to many improvements to the quality of the code we studied.

In the course of a framework's evolution, changes ranging from a simple refactoring to a complete rearchitecture can break client programs. Finding suitable replacements for framework elements that were accessed by a client program and deleted as part of the framework's evolution can be a challenging task. We present a recommendation system, SemDiff, that suggests adaptations to client programs by analyzing how a framework adapts to its own changes. In a study of the evolution of the Eclipse JDT framework and three client programs, our approach recommended relevant adaptive changes with a high level of precision, and detected non-trivial changes typically undiscovered by current refactoring detection techniques.

Code clones are generally considered to be an obstacle to software maintenance. Research has provided evidence that it may not always be practical, feasible, or cost-effective to eliminate certain clone groups through refactoring. This paper describes CloneTracker, an Eclipse plug-in that provides support for tracking code clones in evolving software. With CloneTracker, developers can specify clone groups they wish to track, and the tool will automatically generate a clone model that is robust to changes to the source code, and can be shared with other collaborators of the project. When future modifications intersect with tracked clones, CloneTracker will notify the developer, provide support to consistently apply changes to a corresponding clone region, and provide support for updating the clone model. CloneTracker complements existing techniques by providing support for reusing knowledge about the location of clones in source code, and support for keeping track of clones when refactoring is not desirable.

Software maintenance tasks typically involve an important amount of program investigation effort on the part of software developers. To what extent can we benefit from prior program investigation activities to decrease this effort? To investigate this question, we studied the revision history of two systems to determine how knowledge derived from prior investigation activities could have been reused to support other change tasks. Our initial investigation used a tool, ConcernDetector, that can recommend sets of program elements associated with a high-level concern when elements in the set overlap with elements currently being modified. We discovered that simple overlap-based techniques for retrieving prior investigation knowledge have important limitations, and that effective reuse of prior program investigation knowledge requires analyses that can partially infer the nature and intent of a task.

Code clones are generally considered harmful in software development, and the predominant approach is to try to eliminate them through refactoring. However, recent research has provided evidence that it may not always be practical, feasible, or cost-effective to eliminate certain clone groups. We propose a technique for tracking clones in evolving software. Our technique relies on the concept of abstract clone region descriptors (CRD), which describe clone regions within methods in a robust way that is independent from the exact text of the clone region or its location in a file. We present our definition of CRDs, and describe a complete clone tracking system capable of producing CRDs from the output of a clone detection tool, notify developers of modifications to clone regions, and support the simultaneous editing of clone regions. We report on two experiments and a case study conducted to assess the performance and usefulness of our approach.

The investigation of a software system prior to a modification task often constitutes an important fraction of the overall effort associated with the task. We present Suade, an Eclipse plug-in to automatically generate suggestions for software investigation. The goal of Suade is to increase the efficiency with which developers explore the source code by recommending locations that are likely to be relevant to the task. Based on a context of software elements (fields and methods) explicitly specified by a developer, Suade automatically generates other elements that are likely to be relevant given the context, by analyzing the topology of structural dependencies in a software system.

Programmers need tools to help explore large software systems when performing software evolution tasks. A variety of tools have been created to improve the effectiveness of such exploration. The usefulness of these tools has been argued largely on the basis of case studies, small narrowly-focussed experiments, or non-human-based experiments. In this paper, we report on a more rigorously controlled study of three specialized software exploration tools in which professional programmers used the tools to plan complex change tasks to a medium-sized code base. We found that the tools had little apparent effect; the effects observed instead appear to be dominated by individual styles and strategies of the programmers and characteristics of the tasks. In addition to presenting the results of the study, this paper introduces the use of two experimental evaluation aids: the NASA task load index (TLX) for assessing task difficulty and distance profiles for assessing the to which programmers remain on-track.

As part of the evolution of software systems, effort is often invested to discover in what parts of the source code a feature (or other concern) is implemented. Unfortunately, knowledge about a concern's implementation can become invalid as the system evolves. We propose to mitigate this problem by automatically inferring structural patterns among the elements identified as relevant to a concern's implementation. We then document the inferred patterns as rules that can be checked as the source code evolves. Checking whether structural patterns hold across different versions of a system enables the automatic identification of new elements related to a documented concern. We implemented our technique for JAVA in an Eclipse plug-in called ISIS and applied it to a number of concerns. With a case study spanning 34 versions of the development history of an open-source system, we show how our approach supports the tracking of a concern's implementation through modifications such as extensions and refactorings

Before performing a modification task, a developer usually has to investigate the source code of a system to understand how to carry out the task. Discovering the code relevant to a change task is costly because it is a human activity whose success depends on a large number of unpredictable factors, such as intuition and luck. Although studies have shown that effective developers tend to explore a program by following structural dependencies, no methodology is available to guide their navigation through the thousands of dependency paths found in a nontrivial program. We describe a technique to automatically propose and rank program elements that are potentially interesting to a developer investigating source code. Our technique is based on an analysis of the topology of structural dependencies in a program. It takes as input a set of program elements of interest to a developer and produces a fuzzy set describing other elements of potential interest. Empirical evaluation of our technique indicates that it can help developers quickly select program elements worthy of investigation while avoiding less interesting ones.

A software modification task often addresses several concerns . A concern is anything a stakeholder may want to consider as a conceptual unit, including features, nonfunctional requirements, and design idioms. In many cases, the source code implementing a concern is not encapsulated in a single programming language module, and is instead scattered and tangled throughout a system. Inadequate separation of concerns increases the difficulty of evolving software in a correct and cost-effective manner. To make it easier to modify concerns that are not well modularized, we propose an approach in which the implementation of concerns is documented in artifacts, called concern graphs. Concern graphs are abstract models that describe which parts of the source code are relevant to different concerns. We present a formal model for concern graphs and the tool support we developed to enable software developers to create and use concern graphs during software evolution tasks. We report on five empirical studies, providing evidence that concern graphs support views and operations that facilitate the task of modifying the code implementing scattered concerns, are cost-effective to create and use, and robust enough to be used with different versions of a software system.

Programming languages provide various mechanisms to support information hiding. One problem with information hiding, however, is that providing a stable interface behind which to hide implementation details involves fixing in advance the services offered through the interface. We introduce a flexible approach to define and manage interfaces to achieve separation of concerns in evolving software. Our approach involves explicitly specifying interface and implementation classes for individual concerns, and automatically classifying implementation classes based on their relation to the interface. Our approach is supported by JMantlet, a tool that provides advanced interface management within an integrated development environment. We report on a case study of a large system that provides evidence that flexible interface management is desirable and adequately supported by our approach

The association between the description of a concern (e.g., a feature) and the code that implements it is valuable information that can degrade as the code of a system evolves. We present a study of the evolution of the implementation of a concern in 33 versions of an open-source text editor. We represented the implementation of the concern using concern graphs, a model that was designed to be resilient to source code evolution. The study showed how the concern graph model supports tracking a concern's implementation in an evolving system, as well as inferring high-level past changes and assessing the stability of the concern's implementation

Repeated changes to a software system can introduce small weaknesses such as unplanned dependencies between different parts of the system. While such problems usually go undetected, their cumulative effect can result in a noticeable decrease in the quality of a system. We present an approach to warn developers about increased coupling between the (potentially scattered) implementation of different features. Our automated approach can detect sections of the source code contributing to the increased coupling as soon as software changes are tested. Developers can then inspect the results to assess whether the quality of their changes is adequate. We have implemented our approach for C++ and integrated it with the development process of a proprietary 3D graphics software. We report on our evaluation of the approach in the field, and on a study showing that, for files in the target system, causing increases in feature coupling is a significant predictor of future modifications due to bug fixes.

Many software engineering activities are organized around the idea of concerns. Separation of concerns is a basic tenet of software engineering intended to facilitate the development and evolution of software systems. Unfortunately, separation of concerns is not always possible in practice, and concerns often end up scattered and tangled. The goal of the MACS workshop is to bring together researchers and practitioners with interest and experience in techniques for modeling and analyzing the realization of concerns in software systems.

Before performing a modification task, a developer usually has to investigate the source code of a system to understand how to carry out the task. Discovering the code relevant to a change task is costly because it is an inherently human activity whose success depends on a large number of unpredictable factors, such as intuition and luck. Although studies have shown that effective developers tend to explore a program by following structural dependencies, no methodology is available to guide their navigation through the typically hundreds of dependency paths found in a non-trivial program. In this paper, we propose a technique to automatically propose and rank program elements that are potentially interesting to a developer investigating source code. Our technique is based on an analysis of the topology of structural dependencies in a program. It takes as input a set of program elements of interest to a developer and produces a fuzzy set describing other elements of potential interest. Empirical evaluation of our technique indicates that it can help developers quickly select program elements worthy of investigation while avoiding less interesting ones.

Prior to performing a software change task, developers must discover and understand the subset of the system relevant to the task. Since the behavior exhibited by individual developers when investigating a software system is influenced by intuition, experience, and skill, there is often significant variability in developer effectiveness. To understand the factors that contribute to effective program investigation behavior, we conducted a study of five developers performing a change task on a medium-size open source system. We isolated the factors related to effective program investigation behavior by performing a detailed qualitative analysis of the program investigation behavior of successful and unsuccessful developers. We report on these factors as a set of detailed observations, such as evidence of the phenomenon of inattention blindness by developers skimming source code. In general, our results support the intuitive notion that a methodical and structured approach to program investigation is the most effective.

Developers working on existing programs repeatedly have to address concerns, or aspects, that are not well modularized in the source code comprising a system. In such cases, a developer has to first locate the implementation of the concern in the source code comprising the system, and then document the concern sufficiently to be able to understand it and perform the actual change task.

When performing a program evolution task, developers typically spend a significant amount of effort investigating and reinvestigating source code. To reduce this effort, we propose a technique to automatically infer the essence of program investigation activities as a set of concern descriptions. The concern descriptions produced by our technique list methods and fields of importance in the context of the investigation of an object-oriented system. A developer can rely on this information to perform the change task at hand, or at a later stage for a change that involves the same concerns. The technique involves applying an algorithm to a transcript of a program investigation session. The transcript lists which pieces of source code were accessed by a developer when investigating a program and how the different pieces of code were accessed. We applied the technique to data obtained from program investigation activities for five subjects involved in two different program evolution tasks. The results show that relevant concerns can be identified with a manageable level of noise.

Exception-handling mechanisms in modern programming languages provide a means to help software developers build robust applications by separating the normal control flow of a program from the control flow of the program under exceptional situations. Separating the exceptional structure from the code associated with normal operations bears some consequences. One consequence is that developers wishing to improve the robustness of a program must figure out which exceptions, if any, can flow to a point in the program. Unfortunately, in large programs, this exceptional control flow can be difficult, if not impossible, to determine.In this article, we present a model that encapsulates the minimal concepts necessary for a developer to determine exception flow for object-oriented languages that define exceptions as objects. Using these concepts, we describe why exception-flow information is needed to build and evolve robust programs. We then describe Jex, a static analysis tool we have developed to provide exception-flow information for Java systems based on this model. The Jex tool provides a view of the actual exception types that might arise at different program points and of the handlers that are present. Use of this tool on a collection of Java library and application source code demonstrates that the approach can be helpful to support both local and global improvements to the exception-handling structure of a system.

Existing approaches that are available to help software developers locate and manage the scattered implementation of concerns use a representation based on lines of source code. Because they do not explicitly express program structure, concern representations based on source code have inherent limitations when finding, describing and analyzing concerns. To address these problems, we are trying to find a concern representation that captures as close as possible the amount and precision level of information that software developers need to efficiently plan a change, and that supports queries that can provide this information. As a proposed solution, we introduced the Concern Graph representation that abstracts the implementation details of a concern and makes explicit the relationships between different parts of the concern. The Concern Graph abstraction has also been designed to allow an obvious and expensive mapping back to the corresponding source code. To investigate the practical tradeoffs related to this approach, we have built the Feature Exploration and Analysis Tool (FEAT) that allows a developer to navigate over an extracted model of a Java program, to build up the subset of the model that corresponds to a concern of interest, and to analyze the relationships of that concern to the code base.

Many maintenance tasks address concerns, or features, that are not well modularized in the source code comprising a system. Existing approaches available to help software developers locate and manage scattered concerns use a representation based on lines of source code, complicating the analysis of the concerns. In this paper, we introduce the Concern Graph representation that abstracts the implementation details of a concern and makes explicit the relationships between different parts of the concern. The abstraction used in a Concern Graph has been designed to allow an obvious and inexpensive mapping back to the corresponding source code. To investigate the practical tradeoffs related to this approach, we have built the Feature Exploration and Analysis tool (FEAT) that allows a developer to manipulate a concern representation extracted from a Java system, and to analyze the relationships of that concern to the code base. We have used this tool to find and describe concerns related to software change tasks. We have performed case studies to evaluate the feasibility, usability, and scalability of the approach. Our results indicate that Concern Graphs can be used to document a concern for change, that developers unfamiliar with Concern Graphs can use them effectively, and that the underlying technology scales to industrial-sized programs.

Most software systems are inflexible. Reconfiguring a system's modules to add or to delete a feature requires substantial effort. This inflexibility increases the costs of building variants of a system, amongst other problems. New languages and tools that are being developed to provide additional support for separating concerns show promise to help address this problem. However applying these mechanisms requires determining how to enable a feature to be separated from the codebase. We investigate this problem through an exploratory study conducted in the context of two existing systems: gnu.regexp and jFTPd. The study consisted of applying three different separation of concern mechanisms: Hyper/J/sup TM/ AspectJ/sup TM/ and a lightweight, lexically-based approach, to separate features in the two packages. We report on the study, providing contributions in two areas. First, we characterize the effect different mechanisms had on the structure of the codebase. Second, we characterize the restructuring process required to perform the separations. These characterizations can help researchers to elucidate how the mechanisms may be best used, tool developers to design support to aid the separation process, and early adopters to apply the techniques.

Exception handling mechanisms are intended to help developers build robust systems. Although an exception handling mechanism provides a basis for structuring source code dealing with unusual situations, little information is available to help guide a developer in the appropriate application of the mechanism. In our experience, this lack of guidance leads to complex exception structures. In this paper, we reflect upon our experiences using the Java exception handling mechanism. Based on these experiences, we discuss two issues we believe underlie the difficulties encountered: exceptions are a global design problem, and exception sources are often dif-ficult to predict in advance. We then describe a design approach, based on work by Litke for Ada programs, which we have used to simplify exception structure in existing Java programs.