Documentation enables sharing knowledge between the developers of a technology and its users. Creating quality documents, however, is challenging: Documents must satisfy the needs of a large audience without being overwhelming for individuals. We address this challenge with a new document format named Casdoc. Casdoc documents are interactive resources centered around code examples for programmers. Explanations of the code elements are presented as annotations that the readers reveal based on their needs. We evaluated Casdoc in a field study with over 300 participants who used 126 documents as part of a software design course. During the study, the majority of participants adopted Casdoc instead of a baseline format and used interactive annotations to reveal additional information about the code example. Although participants collectively viewed the majority of the documents’ content, they individually revealed a minority of the annotations they saw. We gathered insights into five aspects of Casdoc that can be applied to other formats and highlighted five lessons learned to improve navigability in online documents.

Unit tests must be readable to help developers understand and evolve production code. Most existing test quality metrics assess test code’s ability to detect bugs. Few metrics focus on test code’s readability. One standard approach to improve readability is the consistent application of conventions. We investigated test convention consistency as a dimension of test quality. We formalized test suite consistency as the extent to which alternatives are used within a code base and introduce two complementary metrics to capture this extent. We elaborated a catalog of over 30 test conventions for the Java language organized in 10 convention classes that group mutual alternatives. We developed tool support to detect occurrences of conventions, compute consistency metrics over a test suite, and view occurrences of conventions in the corresponding code. We applied our tools to study the consistency of the test suites of 20 large open source Java projects. The study validates the design of the test convention classes, provides descriptive statistics on the range of consistency values for 10 different convention classes, and enables us to link observed changes in consistency values to specific events in the change history of our target systems, thus providing evidence of the construct validity of the metrics. We conclude that analyzing test suite consistency via static analysis shows promise as a practical approach to help improve test suite quality.

Reflecting on the limitations of a study is a crucial part of the research process. In software engineering studies, this reflection is typically conveyed through discussions of study limitations or threats to validity. In current practice, such discussions seldom provide sufficient insight to understand the rationale for decisions taken before and during the study, and their implications. We revisit the practice of discussing study limitations and threats to validity and identify its weaknesses. We propose to refocus this practice of self-reflection to a discussion centered on the notion of trade-offs . We argue that documenting trade-offs allows researchers to clarify how the benefits of their study design decisions outweigh the costs of possible alternatives. We present guidelines for reporting trade-offs in a way that promotes a fair and dispassionate assessment of researchers’ work.

When working on a project, software developers must be familiar with computing concepts, standards, and technologies related to the project. We present a novel approach, called Scode, to automatically identify those concepts using the project's documentation. Scode combines entity linking and network analysis techniques specialized for the software development domain. In addition to concepts explicitly mentioned in the documentation, Scode can retrieve implicit concepts related to the project's domain. Concepts identified by Scode have a recognized meaning that is consistent across projects. We compared Scode to different baselines and found that it is more effective at mapping projects to a consistent concept space.

Code examples are of great value to programmers trying to learn an unfamiliar API. Effective code examples are often surrounded with plain text explanations of the relevant concepts, techniques, and API elements involved in the example. However, authoring concise yet complete explanations is a challenging balancing act. To address this challenge, we propose Casdoc, a novel authoring technique and presentation format for annotated code examples. Casdoc-formatted code examples are HTML documents designed to embed unobtrusive explanations into the code. They thus contain more explanations to address the varying needs of a larger audience, without disrupting individual readers with information they already know. Explanations are split into short annotations and organized into an intuitive tree-like structure, thus supporting a streamlined authoring process. We used Casdoc to produce 105 Java code examples as part of the course material for an undergraduate computer science course. Students preferred the new format over traditional code examples. Their interaction with code examples suggests that the intuitive structure of Casdoc annotations reduces the need for navigation aids such as search fields.

Software projects capture redundant information in various kinds of artifacts, as specifications from the source code are also tested and documented. Such redundancy provides an opportunity to reduce development effort by supporting the joint generation of different types of artifacts. We introduce a tool-supported technique, called DScribe, that allows developers to combine unit test and documentation templates, and to invoke these templates to generate documentation and unit tests. DScribe supports the detection and replacement of outdated documentation, and the use of templates can encourage extensive test suites with a consistent style. Our evaluation of 835 specifications revealed that 85 percent were not tested or correctly documented, and DScribe could be used to automatically generate 97 percent of the tests and documentation. An additional study revealed that tests generated by DScribe are more focused and readable than those written by human testers or generated by state-of-the-art automated techniques.

Informal language and the absence of a standard taxonomy for software technologies make it difficult to reliably analyze technology trends on discussion forums and other on-line venues. We propose an automated approach called Witt for the categorization of software technologies (an expanded version of the hypernym discovery problem). Witt takes as input a phrase describing a software technology or concept and returns a general category that describes it (e.g., integrated development environment), along with attributes that further qualify it (commercial, php, etc.). By extension, the approach enables the dynamic creation of lists of all technologies of a given type (e.g., web application frameworks). Our approach relies on Stack Overflow and Wikipedia, and involves numerous original domain adaptations and a new solution to the problem of normalizing automatically-detected hypernyms. We compared Witt with six independent taxonomy tools and found that, when applied to software terms, Witt demonstrated better coverage than all evaluated alternative solutions, without a corresponding degradation in false positive rate.

This artifact is a data set generated as part of a study on the threats of aggregating software repository data, which includes information derived from the GitHub repositories of eight open-source projects.

This artifact is a data set generated as part of a study on the threats of aggregating software repository data, which includes information derived from the GitHub repositories of eight open-source projects.

In large software projects, tacit knowledge of the system is threatened by developer turnover. When a developer leaves the project, their knowledge may be lost if the other developers do not understand the design decisions made by the leaving developer. Understanding the source code written by leaving developers thus becomes a burden for their successors. In a previous paper, Rigby et al. reported on a case study of turnover-induced knowledge loss in two large projects, Chromium and a project at Avaya, using risk evaluation methods usually applied to financial systems. They found that the two projects were susceptible to large knowledge losses that are more than three times the average loss. We report on a replication of their study on the Chromium project, as well as seven other large and medium-sized open source projects. We also extended theirwork by studying two variations of the knowledge loss metric, as well as the location and persistence of abandoned files. We found that all projects had a similar knowledge loss probability distribution, but extreme knowledge loss can be more severe than those originally discovered in Chromium and the project at Avaya. We also found that, in the systems under study, abandoned files often remained in the system for long periods.