Context: Most current code vulnerability detection tools provide only a binary classification (vulnerable/non-vulnerable) with little to no additional context. This paper explores the impact of providing explanations for vulnerabilities alongside code labelled as vulnerable. Objective: We investigate the influence of explanations on the ability of software practitioners to confirm such labelled code as actually vulnerable (i.e., a true positive vulnerability) and to fix such vulnerable code correctly. Method: We surveyed 99 software practitioners to establish their use of code-vulnerability detection tools and to evaluate the impact of explanations on their behaviour towards code labelled as vulnerable in a series of coding exercises. Participants were presented with four forms of explanation: vulnerable lines , vulnerability type , short-form text , and long-form text . Results: Software practitioners performed better at confirming and fixing code vulnerabilities when presented with any of the four forms of explanation. Although practitioners stated a preference for long-form text explanations, they achieved the highest confirmation and fixing performance with short-form text explanations. Practitioners also indicated willingness to accept modest drops in detection precision and recall if richer explanations were provided, and their preferences for explanation types and performance trade-offs varied according to where a detection tool is used in the software-development pipeline. Conclusions: Vulnerability-detection and prediction tools should provide explanatory output and allow different explanation types tailored to their deployment stage in the development workflow. Few current tools provide any explanations, and none identified in this study provide text-based explanations.

Vulnerabilities in code should be detected and patched quickly to reduce the time in which they can be exploited. There are many automated approaches to assist developers in detecting vulnerabilities, most notably Static Application Security Testing (SAST) tools. However, no single tool detects all vulnerabilities and so relying on any one tool may leave vulnerabilities dormant in code. In this study, we use a manually curated dataset to evaluate four SAST tools on production code with known vulnerabilities. Our results show that the vulnerability detection rates of individual tools range from 11.2% to 26.5%, but combining these four tools can detect 38.8% of vulnerabilities. We investigate why SAST tools are unable to detect 61.2% of vulnerabilities and identify missing vulnerable code patterns from tool rule sets. Based on our findings, we create new rules for Semgrep, a popular configurable SAST tool. Our newly configured Semgrep tool detects 44.7% of vulnerabilities, more than using a combination of tools, and a 181% improvement in Semgrep’s detection rate.

Static application Security Testing (SAST) tools are an established means of detecting vulnerabilities early in development. Previous studies have reported low detection rates from SAST tools and recommend either combining SAST tools or configuring rule sets to detect more vulnerabilities. However, while previous work suggests that developers rarely combine or configure any of the Automatic Static Analysis Tools (ASATs) they use, it is currently unclear whether SAST tools are used directly “out of the box”. To understand how developers use SAST tools, we performed a large-scale survey involving 1,263 developers. We pre-screened developers to establish their SAST use and found that only 20% (204/1,003) used SAST tools. Of those developers who did use SAST tools, we found a large number did not use multiple tools (59%), did not configure tools (54%) or did neither (40%). Our results suggest that more work is needed to help developers combine and configure tools, since doing so is likely to detect significantly more vulnerabilities.

In this paper, we re-analyse the data from a previous study of Straubinger et al., which asked 284 industrial IT staff about their views on testing. In that study, as well as developers, the dedicated role of tester was included in the data - both roles were treated as the same role. In this paper, we posit that the two roles (i.e., developer and tester) are so very different that we should analyse each role separately; testers will have unique insights into testing, so separating their views and experiences from developers is important. To this end, we analyse six of the same research questions as the original study, using separate developer and tester data. Results showed that for almost every question we re-visited, testers differed in their opinions from developers, whether on the type of testing they did, measures of code quality, effort to write tests and motivation for testing.

In this short paper, we analyse "gender" perspectives from a survey of three hundred and seventy-eight industry developers on two aspects of IT industry developer practice: bugs and Automatic Program Repair. We also explore questions of how developers view their job satisfaction. Our key motivation was to show whether there was a difference in the way that males and females viewed these three important concepts. From a total of thirteen survey questions analysed, only two showed any statistical difference between the responses of females compared to males. Those differences were found exclusively in the job satisfaction part of the survey. In terms of the way that male or female developers think about technical activities per se and diversity and inclusivity more generally, we therefore have the paradoxical issue of whether gender comparisons have any basis. We all think the same way about technical-oriented activities, so perhaps we need to stop trying to find differences and division.

The study of human values in software engineering (SE) is increasingly recognised as a fundamental human-centric issue of SE decision making. However, values studies in SE still face a number of issues, including the difficulty of eliciting values in a systematic and structured way, the challenges of measuring and tracking values over time, and the lack of practice-based understanding of values among software practitioners. This paper aims to help address these issues by: 1) outlining a research framework that supports a systematic approach to values elicitation, analysis, and understanding; 2) introducing tools and techniques that help elicit and measure values during SE decision making processes in a systematic way; and 3) applying such tools to a month-long research sprint co-designed with an industry partner and conducted with 27 software practitioners. The case study builds on lessons from an earlier pilot (12 participants) and combines in-situ observations with the use of two values-informed tools: the Values Q-Sort (V-QS), and the Values-Retro. The V-QS adapts instruments from values research to the SE context, the Values-Retro adapts existing SE techniques to values theory. We distil implications for research and practice in ten lessons learned.

Automatic program repair (APR) is a rapidly advancing field of software engineering that aims to supplement or replace manual bug fixing with an automated tool. For APR to be successfully adopted in industry, it is vital that APR tools respond to developer needs and preferences. However, very little research has considered developers' general attitudes to APR or developers' current bug fixing practices (the activity APR aims to replace). This paper responds to this gap by reporting on a survey of 386 software developers about their bug finding and fixing practices and experiences, and their instinctive attitudes towards APR. We find that bug finding and fixing is not necessarily as onerous for developers as has often been suggested, being rated as more satisfying than developers' general work. The fact that developers derive satisfaction and benefit from bug fixing indicates that APR adoption is not as simple as APR replacing an unwanted activity. When it comes to potential APR approaches, we find a strong preference for developers being kept in the loop (for example, choosing between different fixes or validating fixes) as opposed to a fully automated process. This suggests that advances in APR should be careful to consider the agency of the developer, as well as what information is presented to developers alongside fixes. It also indicates that there are key barriers related to trust that would need to be overcome for full scale APR adoption, supported by the fact that even those developers who stated that they were positive about APR listed several caveats and concerns. We find very few statistically significant relationships between particular demographic variables (for example, developer experience, age, education) and key attitudinal variables, suggesting that developers' instinctive attitudes towards APR are little influenced by experience level but are held widely across the developer community.

Automatic program repair (APR) offers significant potential for automating some coding tasks. Using APR could reduce the high costs historically associated with fixing code faults and deliver significant benefits to software engineering. Adopting APR could also have profound implications for software developers’ daily activities, transforming their work practices. To realise the benefits of APR it is vital that we consider how developers feel about APR and the impact APR may have on developers’ work. Developing APR tools without consideration of the developer is likely to undermine the success of APR deployment. In this paper, we critically review how developers are considered in APR research by analysing how human factors are treated in 260 studies from Monperrus’s Living Review of APR. Over half of the 260 studies in our review were motivated by a problem faced by developers (e.g., the difficulty associated with fixing faults). Despite these human-oriented motivations, fewer than 7% of the 260 studies included a human study. We looked in detail at these human studies and found their quality mixed (for example, one human study was based on input from only one developer). Our results suggest that software developers are often talkedaboutin APR studies, but are rarely talkedwith. A more comprehensive and reliable understanding of developer human factors in relation to APR is needed. Without this understanding, it will be difficult to develop APR tools and techniques which integrate effectively into developers’ workflows. We recommend a future research agenda to advance the study of human factors in APR.

In this short paper, we explore the Pareto principle, sometimes known as the “80-20” rule as part of the refactoring process. We explore five frequently applied refactorings, namely extract method, extract variable, rename variable, rename method and change variable type from a data set of forty open-source systems and nearly two hundred thousand refactorings. We address two key research questions. Firstly, do 80% of “buggy” refactorings (where a refactoring has induced a bug fix) arise from just 20% of commits and, secondly, does the same rule apply to “non-buggy” refactorings when applied to the same systems? To facilitate our analysis, we used refactoring and bug data from a study by Di Penta et al. Results showed that refactorings inducing bugs were clustered around a more concentrated set of commits than refactorings that did not induce bugs. One refactoring ‘change variable type’ stood out - it almost conformed to an 80-20 rule. The take-away message is, as the saying goes, that too much of a “good” thing [refactoring] could actually be a “bad” thing.

This paper reports on qualitative research into automatic program repair (APR) at Bloomberg. Six focus groups were conducted with a total of seventeen participants (including both developers of the APR tool and developers using the tool) to consider: the development at Bloomberg of a prototype APR tool (Fixie); developers’ early experiences using the tool; and developers’ perspectives on how they would like to interact with the tool in future. APR is developing rapidly and it is important to understand in greater detail developers' experiences using this emerging technology. In this paper, we provide in-depth, qualitative data from an industrial setting. We found that the development of APR at Bloomberg had become increasingly user-centered, emphasising how fixes were presented to developers, as well as particular features, such as customisability. From the focus groups with developers who had used Fixie, we found particular concern with the pragmatic aspects of APR, such as how and when fixes were presented to them. Based on our findings, we make a series of recommendations to inform future APR development, highlighting how APR tools should 'start small', be customisable, and fit with developers' workflows. We also suggest that APR tools should capitalise on the promise of repair bots and draw on advances in explainable AI.

Automatic Program Repair (APR) has been proposed to help developers and reduce the time spent repairing programs. Recent APR tools have applied learned templates (fix patterns) to fix code using knowledge from fixes successfully applied in the past. However, there is still no general agreement on the representation of fix patterns, making their application and comparison with a baseline difficult. As a consequence, it is also difficult to expand fix patterns and further enable APR. We automatically generate fix patterns from similar fixes and compare the generated fix patterns against a state-of-the-art taxonomy. Our automated approach splits fixes into smaller, method-level chunks and calculates their similarity. A threshold-based clustering algorithm groups similar chunks and finds matches with state-of-the-art fix patterns. In our evaluation, we present 33 clusters whose fix patterns were generated from the fixes of 835 Defects4J bugs. Of those 33 clusters, 22 matched a state-of-the-art taxonomy with good agreement. The remaining 11 clusters were thematically analysed and generated new fix patterns that expanded the taxonomy. Our new fix patterns should enable APR researchers and practitioners to expand their tools to fix a greater range of bugs in the future.

Within Computer Science (CS) education, women have long been underrepresented. In the UK, women make up less than 20% of CS students at A-Level and undergraduate level. The lack of diversity within CS has become well-studied, often through quantitative surveys of female CS students or evaluations of different pedagogical or cultural interventions aimed to increase diversity and inclusivity. However, there have been far fewer studies that explore the experiences of female CS students at a more in-depth, qualitative level. This paper reports on the results of 15 in-depth semi-structured interviews with first year female CS undergraduate students at a UK university. Several of our findings are consistent with much existing research, such as the prevalence of gendered CS stereotypes and a lack of confidence. However, we also find two key strategies by which female Computer Science students navigate their minority experience that have not been given much prior attention. Firstly, we find that female CS students find it hard to articulate their minority experience without utilising the linguistic device of hedging. Asked about their experiences as a minority, participants would often mention a negative feeling in relation to this situation, followed quickly by a diminishing clause- 'it's a bit weird, but it's ok'. Secondly, participants tended to individualise the problem, stressing the importance of their own individual responsibility to fit into CS and succeed, despite having sometimes experienced discrimination or sexism. We conclude by considering the implications of these findings for educators.

Background: Human values, such as prestige, social justice, and financial success, influence software production decision-making processes. While their subjectivity makes some values difficult to measure, their impact on software motivates our research. Aim: To contribute to the scientific understanding and the empirical investigation of human values in Software Engineering (SE). Approach: Drawing from social psychology, we consider values as mental representations to be investigated on three levels: at a system (L1), personal (L2), and instantiation level (L3). Method: We design and develop a selection of tools for the investigation of values at each level, and focus on the design, development, and use of the Values Q-Sort. Results: From our study with 12 software practitioners, it is possible to extract three values `prototypes' indicative of an emergent typology of values considerations in SE. Conclusions: The Values Q-Sort generates quantitative values prototypes indicating values relations (L1) as well as rich personal narratives (L2) that reflect specific software practices (L3). It thus offers a systematic, empirical approach to capturing values in SE.