Existing research has demonstrated promising results when applying large language models (LLMs) to detect security vulnerabilities in source code. However, these studies have been exclusively evaluated on benchmarks from open-source systems, using publicly known vulnerabilities that are likely part of the LLMs’ training data. This raises concerns that reported performance metrics may be inflated due to data contamination, providing a misleading view of the models’ actual capabilities.In this paper, we quantify this effect with a case study that evaluates five frontier LLMs on two carefully curated datasets: CWE-Bench-Java (an open-source dataset) and TS-Vuls (based on a closed-source commercial codebase). To provide a second angle, we also split CWE-Bench-Java by CVE record date to explore temporal contamination based on LLM knowledge cutoff dates.Our results reveal that the average F1 score dropped by approximately 20 percentage points when comparing the open-source to the closed-source dataset. Additionally, the precision drops from 56% to 34% on average, which is statistically significant (p < 0.05) for four of five models. This declining trend is consistent across all tested LLMs and metrics. In contrast, the results for the temporal split on the open-source data are inconclusive, suggesting that using a knowledge cutoff may reduce but does not ensure the elimination of contamination effects.Although our study is based on a single closed-source system and thus not generalizable, these findings provide the first empirical evidence that evaluating LLM-based vulnerability detection on open-source benchmarks may lead to overly optimistic results. This motivates the inclusion of closed-source datasets in future LLM evaluations.

Grown software systems often contain code that is not necessary anymore. Such unnecessary code wastes resources during development and maintenance, for example, when preparing code for migration or certification. Running a profiler may reveal code that is not used in production, but it is often time-consuming to obtain representative data in this way. We investigate to what extent a static analysis approach, which is based on code stability and code centrality, is able to identify unnecessary code and whether its recommendations are relevant in practice. To study the feasibility and usefulness of our approach, we conducted a study involving 14 open-source and closed-source software systems. As there is no perfect oracle for unnecessary code, we compared recommendations for unnecessary code with historical cleanups, runtime usage data, and feedback from 25 developers of five software projects. Our study shows that recommendations generated from stability and centrality information point to unnecessary code that cannot be identified by dead code detectors. Developers confirmed that 34% of recommendations were indeed unnecessary and deleted 20% of the recommendations shortly after our interviews. Overall, our results suggest that static analysis can provide quick feedback on unnecessary code and is useful in practice.

Recent empirical studies identified an interest of software developers in high-level usage information, i.e. Why and how end users employ a software application. Furthermore, recent empirical work found that developers of interactive applications put themselves in the role of users by interacting with the user interface during program comprehension. This paper presents an exploratory case study investigating these two user perspectives in detail. The study focuses on information needs regarding software usage and developers in the role of users during program comprehension. 21 developers from six software companies were observed during program comprehension tasks and interviewed. The resulting observation protocols and interview minutes were analyzed using coding. We found that developers are interested in information about use cases and user behavior, user goals and user needs, failure reproduction steps, and application domain concepts. But such information is rarely available to them during program comprehension. This mismatch indicates a potential to improve program comprehension practices by capturing such information and providing it to developers. Furthermore, we found that developers interact with the user interface of an interactive application to reproduce failures, to find relevant source code, to test changes, to trigger the debugger, and to familiarize with an unknown part of the application. Also, developers conceptually map elements of the user interface to source code, data structures, and algorithms. We call this behavior "UI-based comprehension" and argue that it is part of a broader comprehension strategy together with comprehension activities like reading source code or debugging.

Software developers make various decisions when implementing software.For instance, they decide on how to implement an algorithm most efficiently or in which way to process user input.When code is revisited during maintenance, the underlying decisions need to be understood and possibly adjusted to the current situation.Common documentation approaches like JavaDoc neither cover knowledge related to decisions explicitly, nor are they integrated closely with knowledge management.In consequence, decision knowledge is rarely documented and therefore inaccessible, especially when developers have left the team.So, effective maintenance is hindered.We have developed an annotation model for decision knowledge and integrated it with the knowledge management tool UNICASE.The approach enables developers to document decisions within code without tool switches to lower their documentation effort.Afterwards, maintainers can exploit the embedded decision knowledge and follow links to external knowledge.This paper presents the approach and evaluation results of a first case study, which indicate its practicability.

Bug reports submitted by users and crash reports collected by crash reporting tools often lack information about reproduction steps, i.e. the steps necessary to reproduce a failure. Hence, developers have difficulties to reproduce field failures and might not be able to fix all reported bugs. We present an approach to automatically extract failure reproduction steps from user interaction traces. We capture interactions between a user and a WIMP GUI using a capture/replay tool. Then, we extract the minimal, failure-inducing subsequence of captured interaction traces. We use three algorithms to perform this extraction: Delta Debugging, Sequential Pattern Mining, and a combination of both. Delta Debugging automatically replays subsequences of an interaction trace to identify the minimal, failure-inducing subsequence. Sequential Pattern Mining identifies the common subsequence in interaction traces inducing the same failure. We evaluated our approach in a case study. We injected four bugs to the code of a mail client application, collected interaction traces of five participants trying to find these bugs, and applied the extraction algorithms. Delta Debugging extracted the minimal, failure-inducing interaction subsequence in 90% of all cases. Sequential Pattern Mining produced failure-inducing interaction sequences in 75% of all cases and removed on average 93% of unnecessary interactions, potentially enabling manual analysis by developers. Both algorithms complement each other because they are applicable in different contexts and can be combined to improve performance.

Research in program comprehension has evolved considerably over the past decades. However, only little is known about how developers practice program comprehension in their daily work. This article reports on qualitative and quantitative research to comprehend the strategies, tools, and knowledge used for program comprehension. We observed 28 professional developers, focusing on their comprehension behavior, strategies followed, and tools used. In an online survey with 1,477 respondents, we analyzed the importance of certain types of knowledge for comprehension and where developers typically access and share this knowledge. We found that developers follow pragmatic comprehension strategies depending on context. They try to avoid comprehension whenever possible and often put themselves in the role of users by inspecting graphical interfaces. Participants confirmed that standards, experience, and personal communication facilitate comprehension. The team size, its distribution, and open-source experience influence their knowledge sharing and access behavior. While face-to-face communication is preferred for accessing knowledge, knowledge is frequently shared in informal comments. Our results reveal a gap between research and practice, as we did not observe any use of comprehension tools and developers seem to be unaware of them. Overall, our findings call for reconsidering the research agendas towards context-aware tool support.

Support of end-user needs is an important success factor for a software application. In order to optimize the support of end-user needs, developers have to be aware of them and their evolution over time. But a communication gap between developers and users leads to ignorance of developers about how users use their application. Also, developer assumptions about user behavior are rarely tested and corrected if they are wrong. Consequently, many software applications have a mediocre support of user needs and user problems as well as changes in user needs are detected rather late. In this paper, we present a research agenda addressing this problem by comparing use case descriptions to monitored user actions. More specifically, we propose to monitor user actions using instrumentation, detect the current use case of a user using machine learning, and compare use case steps to monitored user actions. By detecting differences between both, we identify mismatches between user behavior and developer assumptions reflected in use case descriptions. Those mismatches can serve as starting points to identify software improvements, to test the use case specification and identify updates, and to revise training programs. Finally, we sketch a plan to evaluate our approach.

The first step to comprehend and fix a software bug is usually to reproduce the corresponding failure. Reproducing a failure requires information about steps to reproduce, i.e. the steps necessary to make a failure occur in the development environment. In case of an application with a user interface, steps to reproduce consist of the interactions between a user and the application that precede the failure. Unfortunately, bug reports typically lack this information. Users are either unaware of its importance to developers, are unable to describe it, or simply do not have time to report it. In this paper, we present a simple but effective and resource efficient approach to monitor interactions between users and their applications selectively at a high level of abstraction, e.g. editing operations and commands. This minimizes the monitoring overhead and enables developers to analyze user interaction traces. We map monitored interactions to a taxonomy of user interactions to help developers comprehend user behavior. Further, we present the Timeline Tool that visualizes monitored interaction traces preceding failures. To evaluate our approach we conducted an experiment with 12 participants and asked them to reproduce bug reports from an open-source project. We found that developers are able to derive steps to reproduce from monitored interaction traces. In particular, inexperienced developers profit from the Timeline Tool, as they are able to reproduce failures that they cannot reproduce without it. The monitoring overhead is rather small (approx. 5% CPU and 2-5% memory) and users feel it does not influence their work in a negative way.

Software maintenance and support services are key factors to the customer perception of software product quality. The overall goal of FastFix is to provide developers with a real-time maintenance environment that increases efficiency and reduces costs, improving accuracy in identification of failure causes and facilitating their resolution. To achieve this goal, FastFix observes application execution and user interaction at runtime. We give an overview of the functionality of FastFix and present one of its main application scenarios.

Detecting the current activity of developers and problems they are facing is a prerequisite for a context-aware assistance and for capturing developers' experiences during their work. We present an approach to detect the current activity of software developers and if they are facing a problem. By observing developer actions like changing code or searching the web, we detect whether developers are locating the cause of a problem, searching for a solution, or applying a solution. We model development work as recurring problem solution cycle, detect developer's actions by instrumenting the IDE, translate developer actions to observations using ontologies, and infer developer activities by using Hidden Markov Models. In a preliminary evaluation, our approach was able to correctly detect 72% of all activities. However, a broader more reliable evaluation is still needed.

Research in program comprehension has considerably evolved over the past two decades. However, only little is known about how developers practice program comprehension under time and project pressure, and which methods and tools proposed by researchers are used in industry. This paper reports on an observational study of 28 professional developers from seven companies, investigating how developers comprehend software. In particular we focus on the strategies followed, information needed, and tools used. We found that developers put themselves in the role of end users by inspecting user interfaces. They try to avoid program comprehension, and employ recurring, structured comprehension strategies depending on work context. Further, we found that standards and experience facilitate comprehension. Program comprehension was considered a subtask of other maintenance tasks rather than a task by itself. We also found that face-to-face communication is preferred to documentation. Overall, our results show a gap between program comprehension research and practice as we did not observe any use of state of the art comprehension tools and developers seem to be unaware of them. Our findings call for further careful analysis and for reconsidering research agendas.

There is a semantic gap between simple but high-level action instructions like “Pick up the cup with the right hand” and low-level robot descriptions that model, for example, the structure and kinematics of a robot's manipulator. Currently, programmers bridge this gap by mapping abstract instructions to parametrized algorithms and rigid body parts of a robot within their control programs. By linking descriptions of robot components, i.e. sensors, actuators and control programs, via capabilities to actions in an ontology we equip robots with knowledge about themselves that allows them to infer the required components for performing a given action. Thereby a robot that is instructed by an end-user, a programmer, or even another robot to perform a certain action, can assess itself whether it is able and how to perform the requested action. This self-knowledge for robots could considerably change the way of robot control, robot interaction, robot programming, and multi-robot communication.