Abstract Self-Admitted Technical Debt, or SATD, is a self-admission of technical debt present in a software system. The presence of SATD in software systems negatively affects developers, therefore, managing and addressing SATD is crucial for software engineering. To effectively manage SATD, developers need to estimate its priority and assess the effort required to fix the described technical debt. About a quarter of descriptions of SATD in software systems express some form of negativity or negative emotions when describing technical debt. In this paper, we report on an experiment conducted with 59 respondents to study whether negativity expressed in the description of SATD actually affects the prioritization of SATD. The respondents are a mix of professional developers and students, and in the experiment, we asked participants to prioritize four vignettes: two expressing negativity and two expressing neutral sentiment. To ensure the vignettes were realistic, they were based on existing SATD extracted from a dataset. We find that negativity causes between one-third and half of developers to prioritize SATD in which negativity is expressed as having more priority. Developers affected by negativity when prioritizing SATD are twice as likely to increase their estimation of urgency and 1.5 times as likely to increase their estimation of importance and effort for SATD compared to the likelihood of decreasing these prioritization scores. Our findings show how developers actively use negativity in SATD to determine how urgently a particular instance of technical debt should be addressed. However, our study also describes a gap in the actions and belief of developers. Even if 33% to 50% use negativity to prioritize SATD, 67% of developers believe that using negativity as a proxy for priority is unacceptable. Therefore, we would not recommend using negativity as a proxy for priority. However, we also recognize it might be unavoidable that negativity is expressed by developers to describe technical debt.

Technical debt (TD) refers to the accumulation of negative consequences resulting from sub-optimal solutions during software development. A recent paper by Edbert et al. studied the difference between security-related TD questions, and security-related non- TD questions on Stack Overflow (SO). One of the characteristics under investigation is the sentiment expressed in these two categories as sentiment provides insight into developers' attitudes and emotions toward security-related TD. To this end, Edbert et al. used a general-purpose, off-the-shelf, sentiment analysis tool. However, previous research has shown that general-purpose off-the-shelf sentiment tools are potentially unreliable when applied to software engineering texts. Therefore, we replicate the study by Edbert et al. using state-of-the-art sentiment analysis tools purpose-built and fine-tuned on SE data, to understand whether and how tool-choice influences the obtained results. We consider both shallow (Senti4SD) and deep learning (BERT4SentiSE) tools. To further understand the differences between shallow and deep-learning sentiment analysis tools, we perform a qualitative analysis into the underlying reasons for tools disagreement. We identify five categories of disagreements: misunderstanding context, courtesy phrases, subjective sentiment, brevity, and divergent examples. Our results are relevant to academics, reiterating the relevance of careful selection of tools used to perform sentiment analysis. Furthermore, the results are relevant to users and developers of sentiment analysis tools, as they inform tool selection dependent on the application domain, and provide insight into optimization of the pre-processing steps. Finally, our study shows that retraining sentiment analysis tools with identical data fails to resolve fundamental inconsis-tencies between how certain types of language, such as courtesy phrases, are classified.

Abstract Sentiment analysis has been used to study aspects of software engineering, such as issue resolution, toxicity, and self-admitted technical debt. To address the peculiarities of software engineering texts, sentiment analysis tools often consider the specific technical lingo practitioners use. To further improve the application of sentiment analysis, there have been two recommendations: Using pre-trained transformer models to classify sentiment and replacing non-natural language elements with meta-tokens. In this work, we benchmark five different sentiment analysis tools (two pre-trained transformer models and three machine learning tools) on 2 gold-standard sentiment analysis datasets. We find that pre-trained transformers outperform the best machine learning tool on only one of the two datasets, and that even on that dataset the performance difference is a few percentage points. Therefore, we recommend that software engineering researchers should not just consider predictive performance when selecting a sentiment analysis tool because the best-performing sentiment analysis tools perform very similarly to each other (within 4 percentage points). Meanwhile, we find that meta-tokenization does not improve the predictive performance of sentiment analysis tools. Both of our findings can be used by software engineering researchers who seek to apply sentiment analysis tools to software engineering data.

Software bots fulfill an important role in collective software development, and their adoption by developers promises increased productivity. Past research has identified that bots that communicate too often can irritate developers, which affects the utility of the bot. However, it is not clear what other properties of human-bot collaboration affect developers' preferences, or what impact these properties might have. The main idea of this paper is to explore characteristics affecting developer preferences for interactions between humans and bots, in the context of GitHub pull requests. We carried out an exploratory sequential study with interviews and a subsequent vignette-based survey. We find developers generally prefer bots that are personable but show little autonomy, however, more experienced developers tend to prefer more autonomous bots. Based on this empirical evidence, we recommend bot developers increase configuration options for bots so that individual developers and projects can configure bots to best align with their own preferences and project cultures.

Open-Source Software (OSS) projects have adopted various automations for repetitive tasks in recent years. One common type of automation in OSS is bots. In this exploratory case study, we seek to understand how the adoption of one particular bot (ASFBot) by the Apache Software Foundation (ASF) impacts the discussions in the issue-trackers of these projects. We use the SmartShark dataset to investigate whether the ASFBot affects (i) human comments mentioning pull requests and fixes in issue comments and (ii) the general human comment rate on issues. We apply a regression discontinuity design (RDD) on nine ASF projects that have been active both before and after the ASFBot adoption. Our results indicate (i) an immediate decrease in the number of median comments mentioning pull requests and fixes after the bot adoption, but the trend of a monthly decrease in this comment count is reversed, and (ii) no effect in the number of human comments after the bot adoption. We make an effort to gather first insights in understanding the impact of adopting the ASFBot on the commenting behavior of developers who are working on ASF projects.

In software engineering the role of human aspects is an important one, especially as developers indicate that they experience a wide range of emotions while developing software. Within software engineering researchers have sought to understand the role emotions and sentiment play in the development of software by studying issues, pull-requests and commit messages. To detect sentiment, automated tools are used, and in this doctoral thesis we plan to study the use of these sentiment analysis tools, their applications, best practices for their usage and the effect of non-natural language on their performance. In addition to studying the application of sentiment analysis tools, we also aim to study self-admitted technical debt and bots in software engineering, to understand why developers express sentiment and what they signal when they express sentiment. Through studying both the application of sentiment analysis tools and the role of sentiment in software engineering, we hope to provide practical recommendations for both researchers and developers.

Abstract Self-Admitted Technical Debt (SATD) consists of annotations—typically, but not only, source code comments—pointing out incomplete features, maintainability problems, or, in general, portions of a program not-ready yet. The way a SATD comment is written, and specifically its polarity, may be a proxy indicator of the severity of the problem and, to some extent, of the priority with which it should be addressed. In this paper, we study the relationship between different types of SATD comments in source code and their polarity, to understand in which circumstances (and why) developers use negative or rather neutral comments to highlight an SATD. To address this goal, we combine a manual analysis of 1038 SATD comments from a curated dataset with a survey involving 46 professional developers. First of all, we categorize SATD content into its types. Then, we study the extent to which developers express negative sentiment in different types of SATD as a proxy for priority, and whether they believe this can be considered as an acceptable practice. Finally, we look at whether such annotations contain additional details such as bug references and developers’ names/initials. Results of the study indicate that SATD comments are mainly used for annotating poor implementation choices ( $\simeq $ ≃ 41%) and partially implemented functionality ( $\simeq $ ≃ 22%). The latter may depend from “waiting” for other features being implemented, and this makes SATD comments more negatives than in other cases. Around 30% of the survey respondents agree on using/interpreting negative sentiment as a proxy for priority, while 50% of them indicate that it would be better to discuss SATD on issue trackers and not in the source code. However, while our study indicates that open-source developers use links to external systems, such as bug identifiers, to annotate high-priority SATD, better tool support is required for SATD management.

Opinion mining, sometimes referred to as sentiment analysis, has gained increasing attention in software engineering (SE) studies. SE researchers have applied opinion mining techniques in various contexts, such as identifying developers’ emotions expressed in code comments and extracting users’ critics toward mobile apps. Given the large amount of relevant studies available, it can take considerable time for researchers and developers to figure out which approaches they can adopt in their own studies and what perils these approaches entail. We conducted a systematic literature review involving 185 papers. More specifically, we present (1) well-defined categories of opinion mining-related software development activities, (2) available opinion mining approaches, whether they are evaluated when adopted in other studies, and how their performance is compared, (3) available datasets for performance evaluation and tool customization, and (4) concerns or limitations SE researchers might need to take into account when applying/customizing these opinion mining techniques. The results of our study serve as references to choose suitable opinion mining tools for software development activities and provide critical insights for the further development of opinion mining techniques in the SE domain.

Several bot detection algorithms have recently been discussed in the literature, as software bots that perform maintenance tasks have become more popular in recent years. State-of-the-art techniques detect bots based on a binary classification, where a GitHub account is either a human or a bot. However, this conceptualisation of bot detection as an account-level binary classification problem fails to account for ‘mixed accounts’, accounts that are shared between a human and a bot, and that therefore exhibit both bot and human activity. By using binary classification models for bot detection, researchers might hence mischaracterize both human and bot behavior in software maintenance. This calls for conceptualisation of bot detection through a comment-level classification. However, the single such approach solely investigates a small number of mixed account comments. The nature of mixed accounts on GitHub is thus yet unknown, and the absence of appropriate datasets make this a difficult problem to study. In this paper, we investigate three comment-level classification models and we evaluate these classifiers on a manually labeled dataset of mixed accounts. We find that the best classifiers based on these classification models achieve a precision and recall between 88% and 96%. However, even the most accurate comment-level classifier cannot accurately detect mixed accounts; rather, we find that textual content alone, or textual content combined with templates used by bots, are very effective features for the detection of both bot and mixed accounts. Our study calls for more accurate bot detection techniques capable of identifying mixed accounts, and as such supporting more refined insights in software maintenance activities performed by humans and bots on social coding sites.

Self-Admitted Technical Debt (SATD) represents the admission, made through source code comments or other channels, of portions of a program being poorly implemented, containing provisional solutions or, in general, simply being not ready yet. To better understand developers' habits in SATD annotation, and possibly support their exploitation in tool support, this paper provides an in-depth analysis of the content provided in SATD comments, and the expressed sentiment. We manually inspect and classify 1038 instances from an existing dataset, grouping them along a taxonomy composed of 41 categories (of which 9 top-level ones), identifying their sentiment, and the presence of external references such as author names or issue IDs. Results of our study indicate that (i) the SATD content is crosscutting along life-cycle dimensions identified in previous work, (ii) comments related to functional problems or on-hold SATD are generally more negative than poor implementation choices or partially implemented functionality, and (iii) despite observations from previous literature, only a minority of SATD comments leverage external references.

The adoption of Continuous Integration (CI) has been shown multiple benefits for software engineering practices related to build, test and dependency management. However, the impact of CI on the social aspects of software development has been overlooked so far. Specifically, we focus on studying the impact of CI on a paradigmatic socio-technical activity within the software engineering domain, namely code reviews. Indeed, one might expect that the introduction of CI allows reviewers to focus on more challenging aspects of software quality that could not be assessed using CI. To assess validity of this expectation we conduct an exploratory study of code reviews in 685 GitHub projects that have adopted Travis-CI, the most popular CI-service on GitHub. We observe that with the introduction of CI, pull requests are being discussed less. On average CI saves up to one review comment per pull request. This decrease in amount of discussion, however, cannot be explained by the decrease in the number of updates of the pull requests. This means that in presence of CI developers perform the same amount of work by communicating less, giving rise to the idea of CI as a silent helper.

Swift is a new programming language developed by Apple as a replacement to Objective-C. It features a sophisticated error handling (EH) mechanism that provides the kind of separation of concerns afforded by exception handling mechanisms in other languages, while also including constructs to improve safety and maintainability. However, Swift also inherits a software development culture stemming from Objective-C being the de-facto standard programming language for Apple platforms for the last 15 years. It is, therefore, a priori unclear whether Swift developers embrace the novel EH mechanisms of the programming language or still rely on the old EH culture of Objective-C even working in Swift.