Software engineers are crowdsourcing answers to their everyday challenges on Q&A forums (e.g., Stack Overflow) and more recently in public chat communities such as Slack, IRC, and Gitter. Many software-related chat conversations contain valuable expert knowledge that is useful for both mining to improve programming support tools and for readers who did not participate in the original chat conversations. However, most chat platforms and communities do not contain built-in quality indicators (e.g., accepted answers, vote counts). Therefore, it is difficult to identify conversations that contain useful information for mining or reading, i.e., conversations of post hoc quality. In this article, we investigate automatically detecting developer conversations of post hoc quality from public chat channels. We first describe an analysis of 400 developer conversations that indicate potential characteristics of post hoc quality, followed by a machine learning-based approach for automatically identifying conversations of post hoc quality. Our evaluation of 2,000 annotated Slack conversations in four programming communities (python, clojure, elm, and racket) indicates that our approach can achieve precision of 0.82, recall of 0.90, F-measure of 0.86, and MCC of 0.57. To our knowledge, this is the first automated technique for detecting developer conversations of post hoc quality.

More than ever, developers are participating in public chat communities to ask and answer software development questions. With over ten million daily active users, Slack is one of the most popular chat platforms, hosting many active channels focused on software development technologies, e.g., python, react. Prior studies have shown that public Slack chat transcripts contain valuable information, which could provide support for improving automatic software maintenance tools or help researchers understand developer struggles or concerns.

The standard approach to applying text retrieval models to code repositories is to train models on documents representing program elements. However, code changes lead to model obsolescence and to the need to retrain the model from the latest snapshot. To address this, we previously introduced an approach that trains a model on documents representing changesets from a repository and demonstrated its feasibility for feature location. In this paper, we expand our work by investigating: a second task (developer identification), the effects of including different changeset parts in the model, the repository characteristics that affect the accuracy of our approach, and the effects of the time invariance assumption on evaluation results. Our results demonstrate that our approach is as accurate as the standard approach for projects with most changes localized to a subset of the code, but less accurate when changes are highly distributed throughout the code. Moreover, our results demonstrate that context and messages are key to the accuracy of changeset-based models and that the time invariance assumption has a statistically significant effect on evaluation results, providing overly-optimistic results. Our findings indicate that our approach is a suitable alternative to the standard approach, providing comparable accuracy while eliminating retraining costs.

Modern software development communities are increasingly social. Popular chat platforms such as Slack host public chat communities that focus on specific development topics such as Python or Ruby-on-Rails. Conversations in these public chats often follow a Q&A format, with someone seeking information and others providing answers in chat form. In this paper, we describe an exploratory study into the potential use-fulness and challenges of mining developer Q&A conversations for supporting software maintenance and evolution tools. We designed the study to investigate the availability of information that has been successfully mined from other developer communications, particularly Stack Overflow. We also analyze characteristics of chat conversations that might inhibit accurate automated analysis. Our results indicate the prevalence of useful information, including API mentions and code snippets with descriptions, and several hurdles that need to be overcome to automate mining that information.

Interaction data, gathered from developers' daily clicks and key presses in the IDE, has found use in both empirical studies and in recommendation systems for software engineering. We observe that this data has several characteristics, common across IDEs:

Modern software development is social and relies on many online resources and tools. In this paper, we study opportunistic code reuse from the Web, e.g., when developers copy code snippets from popular Q&A sites and paste them into their projects. Our focus is the behavior of developers following opportunistic code reuse, which reveals the success or failure of the action. We study developer behavior via a large, representative dataset of micro-interactions in the IDE. Our analysis of developer behavior exhibited in this dataset confirms laboratory study observations that code reuse from the Web is followed by heavy editing, in some cases by a rapid undo, and rarely by the execution of tests.

A study of how method roles evolve during the lifetime of a software system is presented. Evolution is examined by analyzing when the stereotype of a method changes. Stereotypes provide a high-level categorization of a method's behavior and role, and also provide insight into how a method interacts with its environment and carries out tasks. The study covers 50 open-source systems and 6 closed-source systems. Results show that method behavior with respect to stereotype is highly stable and constant over time. Overall, out of all the history examined, only about 10% of changes to methods result in a change in their stereotype. Examples of methods that change stereotype are further examined. A select number of these types of changes are indicators of code smells.

Abstract Text retrieval–based feature location techniques (FLTs) use information from the terms present in documents in classes and methods. However, relevant terms originating from certain locations (eg, method names) often comprise only a small part of the entire method lexicon. Feature location techniques should benefit from techniques that make greater use of this information. The primary objective of this study was to investigate how weighting terms from different locations in source code can improve a latent Dirichlet allocation (LDA)‐based FLT. We conducted an empirical study of 4 subject software systems and 372 features. For each subject system, we trained 1024 different LDA models with new weighting schemes applied to leading comments, method names, parameters, body comments, and local variables. We conducted both a quantitative and qualitative analysis to identify the effects of using the weighting schemes on the performance of the LDA‐based FLT. We evaluated weighting schemes based on mean reciprocal rank and spread of effectiveness measures. In addition, we conducted a factorial analysis to identify which locations have a main impact on the results of the FLT. We then examined the effects of adding information from class comments, class names, and fields to the top 10 configurations for each system. This results in an additional 640 different LDA models for each system. From our results, we identified a significant effect in the performance of an LDA‐based weighting configuration when applying our weighting schemes to the LDA‐based FLT. Furthermore, we found that adding information from each method's containing class can improve the effectiveness of an LDA‐based FLT. Finally, we identified a set of recommendations for identifying better weighting schemes for LDA.

While early software command recommender systems drew negative user reaction, recent studies show that users of unusually complex applications will accept and utilize command recommendations. Given this new interest, more than a decade after first attempts, both the recommendation generation (backend) and the user experience (frontend) should be revisited. In this work, we focus on recommendation generation. One shortcoming of existing command recommenders is that algorithms focus primarily on mirroring the short-term past,-i.e., assuming that a developer who is currently debugging will continue to debug endlessly. We propose an approach to improve on the state of the art by modeling future task context to make better recommendations to developers. That is, the approach can predict that a developer who is currently debugging may continue to debug OR may edit their program. To predict future development commands, we applied Temporal Latent Dirichlet Allocation, a topic model used primarily for natural language, to software development interaction data (i.e., command streams). We evaluated this approach on two large interaction datasets for two different IDEs, Microsoft Visual Studio and ABB Robot Studio. Our evaluation shows that this is a promising approach for both predicting future IDE commands and producing empirically-interpretable observations.

Programmers need documentation to comprehend software, but they often lack the time to write it. Thus, programmers must prioritize their documentation effort to ensure that sections of code important to program comprehension are thoroughly explained. In this paper, we explore the possibility of automatically prioritizing documentation effort. We performed two user studies to evaluate the effectiveness of static source code attributes and textual analysis of source code towards prioritizing documentation effort. The first study used open-source API Libraries while the second study was conducted using closed-source industrial software from ABB. Our findings suggest that static source code attributes are poor predictors of documentation effort priority, whereas textual analysis of source code consistently performed well as a predictor of documentation effort priority.

Due to the high number and cost of interruptions at work, several approaches have been suggested to reduce this cost for knowledge workers. These approaches predominantly focus either on a manual and physical indicator, such as headphones or a closed office door, or on the automatic measure of a worker's interruptibilty in combination with a computer-based indicator. Little is known about the combination of a physical indicator with an automatic interruptibility measure and its long-term impact in the workplace. In our research, we developed the FlowLight, that combines a physical traffic-light like LED with an automatic interruptibility measure based on computer interaction data. In a large-scale and long-term field study with 449 participants from 12 countries, we found, amongst other results, that the FlowLight reduced the interruptions of participants by 46%, increased their awareness on the potential disruptiveness of interruptions and most participants never stopped using it.

Many software development teams collect product defect reports, which can either be manually submitted or automatically created from product logs. Periodically, the teams use the collected defect reports to prioritize which defect to address next. We present a set of behavior-based metrics that can be used in this process. These metrics are based on the insight that development teams can estimate user inconvenience from user and application behavior in interaction logs. To estimate user inconvenience, the behavior metrics capture important user and application behavior after exceptions (the defects of interest in our case). We validated these metrics through a survey of how developers would incorporate the behavior metrics into their prioritization decisions. We found that developers change their priority of investigating an exception about 31% of the time after including the behavior metrics in the priority decision. These findings provide evidence that behavior metrics provide a promising advance towards prioritizing application exceptions.

Spreadsheets are ubiquitous. Thus, it is important to understand the challenges faced by spreadsheet users in practice. To better understand these challenges, we surveyed ABB employees and then interviewed a cross-section of survey respondents. We used a two-phase coding process to classify the challenges they described. Our survey findings demonstrate that practices in our single-company setting are consistent with practices in broader settings. Our interviews revealed both individual and organizational challenges. For instance, individual participants described data pipeline challenges related to importing data from external sources or storing and archiving spreadsheet data. Further, participants' collective responses revealed challenges pertaining to knowledge distribution within the organization. We outline possible interventions to address these challenges. Our results will help guide researchers and tool designers in addressing the practical challenges facing spreadsheet users.

A large corpora of software-related documents is available on the Web, and these documents offer the unique opportunity to learn from what developers are saying or asking about the code snippets that they are discussing. For example, the natural language in a bug report provides information about what is not functioning properly in a particular code snippet. Previous research has mined information about code snippets from bug reports, emails, and Q&A forums. This paper describes an exploratory study into the kinds of information that is embedded in different software-related documents. The goal of the study is to gain insight into the potential value and difficulty of mining the natural language text associated with the code snippets found in a variety of software-related documents, including blog posts, API documentation, code reviews, and public chats.

Background: The relevance of ESEM research to industry practitioners is key to the long-term health of the conference. Aims: The goal of this work is to understand how ESEM research is perceived within the practitioner community and provide feedback to the ESEM community ensure our research remains relevant. Method: To understand how practitioners perceive ESEM research, we replicated previous work by sending a survey to several hundred industry practitioners at a number of companies around the world. We asked the survey participants to rate the relevance of the research described in 156 ESEM papers published between 2011 and 2015. Results: We received 9,941 ratings by 437 practitioners who labeled ideas as Essential, Worth-while, Unimportant, or Unwise. The results showed that overall, industrial practitioners find the work published in ESEM to be valuable: 67% of all ratings were essential or worthwhile. We found no correlation between citation count and perceived relevance of the papers. Through a qualitative analysis, we also identified a number of research themes on which practitioners would like to see an increased research focus. Conclusions: The work published in ESEM is generally relevant to industrial practitioners. There are a number of topics for which those practitioners would like to see additional research undertaken.

Maintaining unit test cases is important during the maintenance and evolution of a software system. In particular, automatically documenting these unit test cases can ameliorate the burden on developers maintaining them. For instance, by relying on up-to-date documentation, developers can more easily identify test cases that relate to some new or modified functionality of the system. We surveyed 212 developers (both industrial and open-source) to understand their perspective towards writing, maintaining, and documenting unit test cases. In addition, we mined change histories of C# software systems and empirically found that unit test methods seldom had preceding comments and infrequently had inner comments, and both were rarely modified as those methods were modified. In order to support developers in maintaining unit test cases, we propose a novel approach - UnitTestScribe - that combines static analysis, natural language processing, backward slicing, and code summarization techniques to automatically generate natural language documentation of unit test cases. We evaluated UnitTestScribe on four subject systems by means of an online survey with industrial developers and graduate students. In general, participants indicated that UnitTestScribe descriptions are complete, concise, and easy to read.

This paper describes a case study of using developer activity logs as indicators of a program comprehension effort by analyzing temporal sequences of developer actions (e.g., navigation and edit actions). We analyze developer activity data spanning 109,065 events and 69 hours of work on a medium-sized industrial application. We examine potential correlations between different measures of developer activity, code change metrics and code smells to gain insight into questions that could direct future technical debt interest estimation. To gain more insights into the data, we follow our analysis with commit message analysis and a developer interview. Our results indicate that developer activity as an estimate of program comprehension effort is correlated with both change proneness and static metrics for code smells.

Deep learning models can infer complex patterns present in natural language text. Relative to n-gram models, deep learning models can capture more complex statistical patterns based on smaller training corpora. In this paper we explore the use of a particular deep learning model, document vectors (DVs), for feature location. DVs seem well suited to use with source code, because they both capture the influence of context on each term in a corpus and map terms into a continuous semantic space that encodes semantic relationships such as synonymy. We present preliminary results that show that a feature location technique (FLT) based on DVs can outperform an analogous FLT based on latent Dirichlet allocation (LDA) and then suggest several directions for future work on the use of deep learning models to improve developer effectiveness in feature location.

Feature location is a program comprehension activity in which a developer inspects source code to locate the classes or methods that implement a feature of interest. Many feature location techniques (FLTs) are based on text retrieval models, and in such FLTs it is typical for the models to be trained on source code snapshots. However, source code evolution leads to model obsolescence and thus to the need to retrain the model from the latest snapshot. In this paper, we introduce a topic-modeling-based FLT in which the model is built incrementally from source code history. By training an online learning algorithm using changesets, the FLT maintains an up-to-date model without incurring the non-trivial computational cost associated with retraining traditional FLTs. Overall, we studied over 600 defects and features from 4 open-source Java projects. We also present a historical simulation that demonstrates how the FLT performs as a project evolves. Our results indicate that the accuracy of a changeset-based FLT is similar to that of a snapshot-based FLT, but without the retraining costs.

Systematic Literature Reviews (SLRs) are an important tool used by software engineering researchers to summarize the state of knowledge about a particular topic. Currently, SLR authors must perform the difficult, time-consuming task in largely manual fashion. To identify barriers faced by SLR authors, we conducted an interactive community workshop prior to ESEM'13. Workshop participants generated a total of 100 ideas that, through group discussions, formed 37 composite barriers to the SLR process. Further analysis reveals the barriers relate to latent themes regarding the SLR process, primary studies, the practitioner community, and tooling. This paper describes the barriers identified during the workshop along with a ranking of those barriers that is based on votes by workshop attendees. The paper concludes by describing the impact of these barriers on three important constituencies: SLR Methodology Researchers, SLR Authors and SLR consumers.

Software maintenance tasks such as feature location and traceability link recovery are search-oriented. Most of the recently proposed approaches for automation of search-oriented tasks are based on a traditional text retrieval (TR) model in which documents are unstructured representations of text and queries consist only of keywords. Because source code has structure, approaches based on a structured retrieval model may yield improved performance. Indeed, Saha et al. Recently proposed a feature location technique based on structured retrieval that offers improved performance relative to a technique based on traditional TR. Although they use abstract syntax tree (AST) information to structure documents, they nonetheless use content-only (keyword) queries to retrieve documents. In this paper we propose an approach to source code search using AST information to structure queries in addition to documents. Such queries, known as content and structure (CAS) queries, allow developers to search for source code entities based not only on content relevance, but also on structural similarity. After introducing the structured retrieval model, we provide examples that illustrate the trade-off between the simplicity of content-only queries and the power of CAS queries.

Issue tracking software of large software projects receive a large volume of issue reports each day. Each of these issues is typically triaged by hand, a time consuming and error prone task. Additionally, issue reporters lack the necessary understanding to know whether their issue has previously been reported. This leads to issue trackers containing a lot of duplicate reports, adding complexity to the triaging task.

Conducting a systematic literature review (SLR) is difficult and time-consuming for an experienced researcher, and even more so for a novice graduate student. With a better understanding of the most common difficulties in the SLR process, mentors will be better prepared to guide novices through the process. This understanding will help researchers have more realistic expectations of the SLR process and will help mentors guide novices through its planning, execution, and documentation phases. Consequently, the objectives of this work are to identify the most difficult and time-consuming phases of the SLR process. Using data from two sources - 52 responses to an online survey sent to all authors of SLRs published in software engineering venues and qualitative experience reports from 8 PhD students who conducted SLRs as part of a course - we identified specific difficulties related to each phase of the SLR process. Our findings highlight the importance of planning, teamwork, and mentoring by an experienced researcher throughout the process. The paper also identifies implications for the teaching of the SLR process.

Many recent feature location techniques (FLTs) apply text retrieval (TR) techniques to corpora built from text embedded in source code. Term weighting is a standard preprocessing step in TR and is used to adjust the importance of a term within a document or corpus. Common term weighting schemes such as tf-idf may not be optimal for use with source code, because they originate from a natural language context and were designed for use with unstructured documents. In this paper we propose a new approach to term weighting in which term weights are assigned using the structural information from the source code. We then evaluate the proposed approach by conducting an empirical study of a TR-based FLT. In all, we study over 400 bugs and features from five open source Java systems and find that structural term weighting can cause a statistically significant improvement in the accuracy of the FLT.

During software evolution a developer must investigate source code to locate then understand the entities that must be modified to complete a change task. To help developers in this task, Haiduc et al. proposed text summarization based approaches to the automatic generation of class and method summaries, and via a study of four developers, they evaluated source code summaries generated using their techniques. In this paper we propose a new topic modeling based approach to source code summarization, and via a study of 14 developers, we evaluate source code summaries generated using the proposed technique. Our study partially replicates the original study by Haiduc et al. in that it uses the objects, the instruments, and a subset of the summaries from the original study, but it also expands the original study in that it includes more subjects and new summaries. The results of our study both support the findings of the original and provide new insights into the processes and criteria that developers use to evaluate source code summaries. Based on our results, we suggest future directions for research on source code summarization.

StackOverflow (SO) contributors are recognized by reputation scores. Earning a high reputation score requires technical expertise and sustained effort. We analyzed the SO data from four perspectives to understand the dynamics of reputation building on SO. The results of our analysis provide guidance to new SO contributors who want to earn high reputation scores quickly. In particular, the results indicate that the following activities can help to build reputation quickly: answering questions related to tags with lower expertise density, answering questions promptly, being the first one to answer a question, being active during off peak hours, and contributing to diverse areas.

SUMMARY Detection of code clones — similar or identical source code fragments — is of concern both to researchers and to practitioners. An analysis of the clone detection results for a single source code version provides a developer with information about a discrete state in the evolution of the software system. However, tracing clones across multiple source code versions permits a clone analysis to consider a temporal dimension. Such an analysis of clone evolution can be used to uncover the patterns and characteristics exhibited by clones as they evolve within a system. Developers can use the results of this analysis to understand the clones more completely, which may help them to manage the clones more effectively. Thus, studies of clone evolution serve a key role in understanding and addressing issues of cloning in software. In this paper, we present a systematic review of the literature on clone evolution. In particular, we present a detailed analysis of 30 relevant papers that we identified in accordance with our review protocol. The review results were organized to address three research questions. Through our answers to these questions, we present the methods that researchers have used to study clone evolution, the patterns that researchers have found evolving clones to exhibit, and the evidence that researchers have established regarding the extent of inconsistent change undergone by clones during software evolution. Overall, the review results indicate that whereas researchers have conducted several empirical studies of clone evolution, there are contradictions among the reported findings, particularly regarding the lifetimes of clone lineages and the consistency with which clones are changed during software evolution. We identify human‐based empirical studies and classification of clone evolution patterns as two areas that are in particular need of further work. Copyright © 2011 John Wiley & Sons, Ltd.

In this paper we present an approach for the class integration test order problem in aspect-oriented programs. Several approaches have been proposed for aspect-oriented systems, but the proposed approach is the first, to our best knowledge, to consider the indirect impact of aspects. This approach relies on a genetic algorithm and can reduce the testing efforts when many methods are indirectly impacted by aspects. We detail the algorithm and then discuss its parameters. The approach has been implemented for Aspect J systems, and to validate it, has been applied to a motivating example.

Exploring linguistic topics in source code is a program comprehension activity that shows promise in helping a developer to become familiar with an unfamiliar software system. Examining ownership in source code can reveal complementary information, such as who to contact with questions regarding a source code entity, but the relationship between linguistic topics and ownership is an unexplored area. In this paper we combine software repository mining and topic modeling to measure the ownership of linguistic topics in source code. We conduct an exploratory study of the relationship between linguistic topics and ownership in source code using 10 open source Java systems. We find that classes that belong to the same linguistic topic tend to have similar ownership characteristics, which suggests that conceptually related classes often share the same owner(s). We also find that similar topics tend to share the same ownership characteristics, which suggests that the same developers own related topics.

Testing is a key skill for computer science students to acquire during their studies. To determine how well students are learning this skill, we conducted an empirical study in two offerings of a senior-level computer science course. The goal of the study was to determine whether students would be able to create a small, complete test suite for a simple program. The students created a test suite first without the aid of a coverage tool and then with the aid of a coverage tool. The results indicate that without a coverage tool, students achieved significantly less than 100% statement, branch or condition coverage. When provided with a code coverage tool, students increased coverage levels. Still, examination of the test suites indicated that they were significantly larger than the minimum required. These results indicate that students cannot conduct adequate testing of even a small program. To provide context for our results, we provide a literature survey summarizing various techniques proposed for teaching testing in the computer science curriculum. We discuss each technique, its strengths, and its weaknesses.

Much recent research effort has been devoted to designing efficient code clone detection techniques and tools. However, there has been little human-based empirical study of developers as they use the outputs of those tools while performing maintenance tasks. This paper describes a study that investigates the usefulness of code clone information for performing a bug localization task. In this study 43 graduate students were observed while identifying defects in both cloned and non-cloned portions of code. The goal of the study was to understand how those developers used clone information to perform this task. The results of this study showed that participants who first identified a defect then used it to look for clones of the defect were more effective than participants who used the clone information before finding any defects. The results also show a relationship between the perceived efficacy of the clone information and effectiveness in finding defects. Finally, the results show that participants who had industrial experience were more effective in identifying defects than those without industrial experience.

A dissatisfaction appears to permeate the process of educating computer science students. Both students and instructors seem uninspired in the classroom, resulting in many attempts to enliven, freshen, and improve the experience. These attempts show efficacy, but the pace of improvement is slow. 100 Problems (100P) is an innovative guided discovery curriculum in which students are freed from the classroom and instead work on 100 concept- and research-related problems throughout their undergraduate careers. The 100 problems guide the students to discover the fundamental knowledge and skills required of a graduate of the degree program. Each student is free to create an individualized mode of learning and discovery. As such, the curriculum fosters deep learning among students and challenges students' intellectual growth. In this paper we introduce the 100P curriculum, describe the 100P course format and our experiences offering courses in this format, and report our early findings.

In this paper we present an empirical study of relationships between three source code lexicons: the identifier, comment, and literal lexicons. We conjecture that shared and unique properties of these lexicons for the given subject system can inform the configuration of a source code retrieval technique for a particular software understanding activity or software evolution task. Thus, we seek to discover these lexicon properties, and so we investigate five lexicon measures that consider term frequency, term density, and term provenance.

Recommended patrol routes can be used by organizations such as police agencies, emergency medical responders, and taxi services whose agents patrol roadway segments at proper times to assist or deter their target events. The creation of optimal complementary patrol routes for multiple agents targeting temporal event hotspots and minimizing travel distance is an NP-hard combinatorial problem that belongs to a class of problems known as the vehicle routing problem with time windows (VRPTW). Traffic safety patrol routing problems share many characteristics of VRPTW problems but differ in ways that prevent the application of existing solutions. In our approach, nondeterministic patrol routing algorithms are used to specify the movements of simulated mobile agents on a roadway system. Nondeterminism is critical in the traffic safety patrol routing domain, as rigidity and predictability can negatively impact the effectiveness of law enforcement agents' efforts.This paper addresses the problem of expressing, executing, evaluating, and engaging patrol routing algorithms that target event hotspots on roadways. The patrol algorithms are first expressed using Turn, which is our extensible domain-specific language (DSL) created for this purpose. Algorithms specified using Turn syntax are then executed in a custom simulation environment. Utilizing predefined metrics, users evaluate the resulting patrol routes to ensure that the criteria of interest in a given patrol context are met. Acceptable patrol routes are then engaged by end users via a web-based geographic information system (GIS) portal. To demonstrate the applicability and efficacy of our approach, we present two illustrative case studies.

Many software development tools that assist with tasks such as testing and maintenance are specific to a particular development language and require a parser for that language. Because a grammar is required to develop a parser, construction of these software development tools is dependent upon the availability of a grammar for the development language. However, a grammar is not always available for a language and, in these cases, acquiring a grammar is the most difficult, costly, and time-consuming phase of tool construction. In this paper, we describe a methodology for grammar recovery from a hard-coded parser. Our methodology is comprised of manual instrumentation of the parser, a technique for automatic grammar recovery from parse trees, and a semi-automatic metrics-guided approach to refactoring an iterative grammar to obtain a recursive grammar. We present the results of a case study in which we recover and refactor a grammar from version 4.0.0 of the GNU C++ parser and then refactor the recovered grammar using our metrics-guided approach. Additionally, we present an evaluation of the recovered and refactored grammar by comparing it to the ISO C++98 grammar.