Large Language Models have demonstrated a remarkable capability in natural language and program generation and software development. However, the source code generated by the LLMs does not always meet quality requirements and may fail to compile. Therefore, many studies evolve into agents that can reason about the problem before generating the source code for the solution. The goal of this paper is to study the degree to which such agents benefit from access to software development tools, in our case, a gcc compiler. We conduct a computational experiment on the RosettaCode dataset, on 699 programming tasks in C. We evaluate how the integration with a compiler shifts the role of the language model from a passive generator to an active agent capable of iteratively developing runnable programs based on feedback from the compiler. We evaluated 16 language models with sizes ranging from small (135 million) to medium (3 billion) and large (70 billion). Our results show that access to a compiler improved the compilation success by 5.3 to 79.4 percentage units in compilation without affecting the semantics of the generated program. Syntax errors dropped by 75 %, and errors related to undefined references dropped by 87 % for the tasks where the agents outperformed the baselines. We also observed that in some cases, smaller models with a compiler outperform larger models with a compiler. We conclude that it is essential for LLMs to have access to software engineering tools to enhance their performance and reduce the need for large models in software engineering, such as reducing our energy footprint.

Over the last years, user feedback has become a valuable source for requirements elicitation. Software vendors increasingly rely on user feedback to collect product issues and feature requests, discover requirements and monitor the overall sentiment of the users about a product. While the analysis of user feedback for requirements elicitation has revealed that feedback can contain helpful information for the product team, collecting valuable, informative, and actionable feedback is still challenging: User feedback is often vague, emotional, or missing important information, such as contextual information, to actually support a product team. Information describing the context of the reported feedback, such as the device model and software version, plays an essential role in increasing its value [1], [2]. Without a given context, reported issues can be complex to understand, reproduce, and address.

Context: Electroencephalograms (EEG) have been used in a few cases in the context of software engineering (SE). EEGs allow capturing emotions and cognitive functioning. Such human factors have already shown to be important to understand software engineering tasks. Therefore, it is essential to gain experience in the community to utilize EEG as a research tool. Objective: To report experiences of using EEG in the context of a software engineering education (review of master theses proposals). We provide our reflections and lessons learned of (1) how to plan an EEG study, (2) how to conduct and execute (e.g., tools), (3) how to analyze. Method: We carried out an experiment using an EEG headset to measure the participants' attention rate. The experiment task includes reviewing three master thesis project plans. Results: We describe how we evolved our understanding of experimentation practices to collect and analyze psychological and cognitive data. We also provide a set of lessons learned regarding the application of EEG technology for research. Conclusions: We believe that that EEG could benefit software engineering research to collect cognitive information under certain conditions. The lessons learned reported here should be used as inputs for future experiments in software engineering, where human aspects are of interest.

Context: Software evolution ensures that software systems in use stay up to date and provide value for end-users. However, it is challenging for requirements engineers to continuously elicit needs for systems used by heterogeneous end-users who are out of organisational reach. Objective: We aim at supporting continuous requirements elicitation by combining user feedback and usage monitoring. Online feedback mechanisms enable end-users to remotely communicate problems, experiences, and opinions, while monitoring provides valuable information about runtime events. It is argued that bringing both information sources together can help requirements engineers to understand end-user needs better. Method/Tool: We present FAME, a framework for the combined and simultaneous collection of feedback and monitoring data in web and mobile contexts to support continuous requirements elicitation. In addition to a detailed discussion of our technical solution, we present the first evidence that FAME can be successfully introduced in real-world contexts. Therefore, we deployed FAME in a web application of a German small and medium-sized enterprise (SME) to collect user feedback and usage data. Results/Conclusion: Our results suggest that FAME not only can be successfully used in industrial environments but that bringing feedback and monitoring data together helps the SME to improve their understanding of end-user needs, ultimately supporting continuous requirements elicitation.

Companies are interested in knowing how users experience and perceive their products. Quality of Experience (QoE) is a measurement that is used to assess the degree of delight or annoyance in experiencing a software product. To assess QoE, we have used a feedback tool integrated into a software product to ask users about their QoE ratings and to obtain information about their rationales for good or bad QoEs. It is known that requests for feedback may disturb users; however, little is known about the subjective reasoning behind this disturbance or about whether this disturbance negatively affects the QoE of the software product for which the feedback is sought. In this paper, we present a mixed qualitative-quantitative study with 35 subjects that explore the relationship between feedback requests and QoE. The subjects experienced a requirement-modeling mobile product, which was integrated with a feedback tool. During and at the end of the experience, we collected the users’ perceptions of the product and the feedback requests. Based on the users’ rational for being disturbed by the feedback requests, such as “early feedback,” “interruptive requests,” “frequent requests,” and “apparently inappropriate content,” we modeled feedback requests. The model defines feedback requests using a set of five-tuple variables: “task,” “timing” of the task for issuing the feedback requests, user’s “expertise-phase” with the product, the “frequency” of feedback requests about the task, and the “content” of the feedback request. Configuration of these parameters might drive the participants’ perceived disturbances. We also found that the disturbances generated by triggering user feedback requests have negligible impacts on the QoE of software products. These results imply that software product vendors may trust users’ feedback even when the feedback requests disturb the users.

Feedback communication channels allow end-users to express their needs, which can be considered in software development and evolution. Although feedback gathering and analysis have been identified as an important topic and several researchers have started their investigation, information is scarce on how software companies currently elicit end-user feedback. In this study, we explore the experiences of software companies with respect to feedback gathering. The results of a case study and online survey indicate two sides of the same coin: on the one hand, most software companies are aware of the relevance of end-user feedback for software evolution and provide feedback channels, which allow end-users to communicate their needs and problems. On the other hand, the quantity and quality of the feedback received varies. We conclude that software companies still do not fully exploit the potential of end-user feedback for software development and evolution.

Runtime monitoring and assessment of software products, features, and requirements allow product managers and requirement engineers to verify the implemented features or requirements, and validate the user acceptance. Gaining insight into software quality and impact of the quality on user facilitates interpretation of quality against users' acceptance and vice versa. The insight also expedites root cause analysis and fast evolution in the case of threatening the health and sustainability of the software. Several studies have proposed automated monitoring solutions and assessment, however, none of the studies introduces a solution for a joint assessment of software quality and quality impact on users. In this research, we study the relation between software quality and the impact of quality on Quality of Experience (QoE) of users to support the assessment of software products, features, and requirements. We propose a Quality-Impact assessment method based on a joint analysis of software quality and user feedback. As an application of the proposed method in requirement engineering, the joint analysis guides verification and validation of functional and quality requirements as well as capturing new requirements. The study follows a design science approach to design Quality-Impact method artifact. The Quality-Impact method has been already designed and validated in the first design cycle. However, next design cycles will contribute to clarify problems of the initial design, refine and validate the proposed method. This paper presents the concluded results and explains future studies for the follow up of the Ph.D. research.

Quality requirements, an important class of non-functional requirements, are inherently difficult to elicit. Particularly challenging is the definition of good-enough quality. The problem cannot be avoided though, because hitting the right quality level is critical. Too little quality leads to churn for the software product. Excessive quality generates unnecessary cost and drains the resources of the operating platform. To address this problem, we propose to elicit the specific relationships between software quality levels and their impacts for given quality attributes and stakeholders. An understanding of each such relationship can then be used to specify the right level of quality by deciding about acceptable impacts. The quality-impact relationships can be used to design and dimension a software system appropriately and, in a second step, to develop service level agreements that allow re-use of the obtained knowledge of good-enough quality. This paper describes an approach to elicit such quality-impact relationships and to use them for specifying quality requirements. The approach has been applied with user representatives in requirements workshops and used for determining Quality of Service (QoS) requirements based the involved users' Quality of Experience (QoE). The paper describes the approach in detail and reports early experiences from applying the approach.

SaaS cloud computing, in contrast to packaged products, enables permanent contact between users of a software product and the product-owning company. When planning the development and evolution of a software product, a product manager depends on reliable information about feature attractiveness. So far, planning decisions were based on stakeholder opinion and the customer's willingness to buy. Whether or not a feature actually is used was out of consideration. Analytics that measure the interaction between users and the SaaS gives product managers unprecedented access to information about product usage. To understand whether and how SaaS analytics can be used for product planning decision, we performed 17 in-depth interviews with experienced managers of SaaS products and analyzed the results analyzed with a mixed-method strategy. The empirical results characterize the relevance of a broad range of analytics for product planning decisions, and the strengths and limitations of an analytics-based product planning approach.