Fairness—the absence of unjustified bias—is a core principle in the development of Artificial Intelligence (AI) systems, yet it remains difficult to assess and enforce. Current approaches to fairness testing in large language models (LLMs) often rely on manual evaluation, fixed templates, deterministic heuristics, and curated datasets, making them resource-intensive and difficult to scale. This work aims to lay the groundwork for a novel, automated method for testing fairness in LLMs, reducing the dependence on domain-specific resources and broadening the applicability of current approaches. Our approach, Meta-Fair, is based on two key ideas. First, we adopt metamorphic testing to uncover bias by examining how model outputs vary in response to controlled modifications of input prompts, defined by metamorphic relations (MRs). Second, we propose exploiting the potential of LLMs for both test case generation and output evaluation, leveraging their capability to generate diverse inputs and classify outputs effectively. The proposal is complemented by three open-source tools supporting LLM-driven generation, execution, and evaluation of test cases. We report the findings of several experiments involving 12 pre-trained LLMs, 14 MRs, 5 bias dimensions, and 7.9K automatically generated test cases. The results show that Meta-Fair is effective in uncovering bias in LLMs, achieving an average precision of 92% and revealing biased behaviour in 29% of executions. Additionally, LLMs prove to be reliable and consistent evaluators, with the best-performing models achieving F1-scores of up to 0.79. Although non-determinism affects consistency, these effects can be mitigated through careful MR design. This work highlights the feasibility and potential of integrating metamorphic testing with LLM-driven test generation and assessment. While challenges remain to ensure broader applicability, the results indicate a promising path towards an unprecedented level of automation in LLM testing.

The SaaS paradigm has popularized the usage of pricings, allowing providers to offer a wide range of subscription possibilities. This creates a vast configuration space for customers, enabling them to choose the features, limits and guarantees that best suit their needs. Regardless of the reasons, changes in pricings are frequent, and are increasing their complexity. Therefore, for those responsible for the development and operation of SaaS, it would be ideal to minimize the time required to transfer changes in SaaS pricing to the software and underlying infrastructure, without compromising quality and reliability. We call this pricing-driven self-adaptation, and this work explores the extent of industry support for it. First, after analyzing 240 pricings from 37 different SaaS over seven years, we reveal a trend of exponentially increasing complexity, mainly driven by a sustained increase in the number of add-ons. Second, acknowledging feature toggling as a promising technique for enabling pricing-driven self-adaptation, we evaluate 18 existing solutions to assess their suitability. However, results reveal a gap between their capabilities and the requirements imposed by the growing complexity of pricings. In light of these results, establishing a standard for pricing serialization and advancing automation in pricing-driven self-adaptation emerge as key steps toward reducing the time-to-market of SaaS pricing updates.

Large Language Models (LLMs) have recently gained significant attention due to their ability to understand and generate sophisticated human-like content. However, ensuring their safety is paramount as they might provide harmful and unsafe responses. Existing LLM testing frameworks address various safety-related concerns (e.g., drugs, terrorism, animal abuse) but often face challenges due to unbalanced and obsolete datasets. In this paper, we present ASTRAL, a tool that automates the generation and execution of test cases (i.e., prompts) for testing the safety of LLMs. First, we introduce a novel black-box coverage criterion to generate balanced and diverse unsafe test inputs across a diverse set of safety categories as well as linguistic writing characteristics (i.e., different style and persuasive writing techniques). Second, we propose an LLM-based approach that leverages Retrieval Augmented Generation (RAG), few-shot prompting strategies and web browsing to generate up-to-date test inputs. Lastly, similar to current LLM test automation techniques, we leverage LLMs as test oracles to distinguish between safe and unsafe test outputs, allowing a fully automated testing approach. We conduct an extensive evaluation on well-known LLMs, revealing the following key findings: i) GPT3.5 outperforms other LLMs when acting as the test oracle, accurately detecting unsafe responses, and even surpassing more recent LLMs (e.g., GPT-4), as well as LLMs that are specifically tailored to detect unsafe LLM outputs (e.g., LlamaGuard); ii) the results confirm that our approach can uncover nearly twice as many unsafe LLM behaviors with the same number of test inputs compared to currently used static datasets; and iii) our black-box coverage criterion combined with web browsing can effectively guide the LLM on generating up-to-date unsafe test inputs, significantly increasing the number of unsafe LLM behaviors.

Requirements Elicitation (RE) is a crucial software engineering skill that involves interviewing a client and then devising a software design based on the interview results. We propose conditioning a large language model to play the role of the client during a chat-based interview. We evaluate our approach in a study (n=120) using both a qualitative survey and quantitative observations about participants' work. Our positive findings suggest a new way to practice critical RE skills in a scalable and realistic manner without the overhead of arranging live interviews.

Most of the attention on GenAI in computing education has focused on programming-centric tasks, such as code generation, giving feedback on code, or providing synthetic programming partners. Yet in advanced software engineering and project courses, interpersonal skills such as team meetings or customer interviews are equally important but difficult and instructor-intensive to teach realistically. GenAI presents the possibility of scaffolding the teaching of some of these practices by enabling exercises in which students develop the ability to investigate a topic by iteratively asking questions to find a solution. The goal is to create scenarios in which students train to interact with humans in real-world situations, simulating these interactions in a controlled, guided environment. These simulations could help students practice and refine ''soft skills,'' such as teamwork and interviewing, by mimicking the types of exchanges and problem-solving they would encounter in professional environments. This approach allows learners to engage in realistic communication exercises, improving their ability to handle complex, interpersonal tasks through repeated practice with AI-guided feedback. As an example, we envision examples that include requirements elicitation with customers, development team meetings, and discussion with potential investors, to name just a few.

Mutation testing drives the creation and improvement of test cases by evaluating their ability to identify synthetic faults. Over the past decades, the technique has gained popularity in academic circles. In practice, however, little is known about its adoption and use. While there are some pilot studies applying mutation testing in industry, the overall usage of mutation testing among developers remains largely unexplored. To fill this gap, this paper presents the results of a qualitative study among open-source developers on the use of mutation testing. Specifically, we report the results of a survey of 104 contributors to open-source projects using a variety of mutation testing tools. The findings of our study provide helpful insights into the use of mutation testing in practice, including its main benefits and limitations. Overall, we observe a high degree of satisfaction with mutation testing across different programming languages and mutation testing tools. Developers find the technique helpful for improving the quality of test suites, detecting bugs, and improving code maintainability. Popularity, usability, and configurability emerge as key factors for the adoption of mutation tools, whereas performance stands overwhelmingly as their main limitation. These results lay the groundwork for new research contributions and tools that meet the needs of developers and boost the widespread adoption of mutation testing.

Background. Mindfulness is a meditation technique whose main goal is educating attention by focusing only on one thing at a time, usually breathing. Mindfulness practices improve concentration and attention, being particularly valuable in demanding and high-stress work settings, such as those found in software companies. A family of five controlled experiments on the impact of mindfulness on future and current software engineers' performance has been carried out in six years, whose participants practiced mindfulness daily for several weeks. Aims. This work has a twofold purpose, to present the fifth experiment in the series and to summarize the lessons learned across the family of experiments. The fifth experiment was carried out at INPRO, a public software company in Seville (Spain), in order to evaluate whether software workers improve their performance and some psychological factors, i.e. attention awareness, techno-stress and well-being, compared to a control group. Method. Employees of two departments (Development and Operation) were recruited to participate in the study. Mindfulness (the treatment) was applied to 24 subjects who attended mindfulness sessions daily for six weeks, while the other 27 subjects were the control group. For all subjects, such psychological factors were measured using questionnaires, whereas performance was measured using INPRO's task management systems. Results. Findings have shown significant differences in terms of attention awareness and techno-stress levels between the practitioners who practised mindfulness and those who did not. Benefits on the perceived well-being have also been reported by participants after the continued practise of mindfulness. Regarding the performance, the analysis depicts inconclusive results, probably due to the small size of the sample, a problem which was accentuated by significant variability in the kinds of tasks performed in both departments. Conclusions. Mindfulness practice has yielded significant benefits in the series of experiments, such as performance in the academy and psychological factors in the industry. Nevertheless, its impact on performance in software companies requires further research, since the limited data availability on subjects' performance has led to a small sample size, ultimately posing challenges in drawing dependable conclusions.

Purpose: Mindfulness is a meditation technique whose main goal involves maintaining a calm mind and training attention by focusing only on a single thing (the support) at a time; this support is usually the practitioner's breathing. The practice of mindfulness aims to improve concentration and attention, which has proven useful in knowledge-intensive and stressful work environments like technological companies. This article aims to find empirical evidence on the positive effect of the practice of mindfulness on a sample of 56 helpdesk employees working for a consulting and information technology company (Accenture) with respect to: i) their attention awareness; ii) a set of key performance indicators (KPIs); and iii) the perceived benefits of mindfulness. Method: Of the 56 recruited employees, 29 worked as managers, and 27 worked as agents answering phone calls to solve software issues of the main information system of the Andalusian Health Service, a public organization with more than 115,000 employees. Mindfulness (the treatment) was applied to 26 subjects, while the other 30 subjects were the control group. For all subjects, their attention awareness was measured using the MAAS scale. Results: Both helpdesk managers and agents significantly improved their attention awareness with respect to the control group. Regarding organizational KPIs, in general, no evidence of significant differences between groups was detected, apart from the fact that the number of phone calls answered was significantly lower in the mindfulness group, probably due to a longer call duration caused by a deliberate better attention to the customer, but without degrading any other KPI. With respect to the perceived benefits of the treatment, the questionnaires show relevant improvements perceived by most employees after practicing mindfulness.Conclusions: We confirm that mindfulness improves attention awareness and benefits the working and personal life of helpdesk employees. However, further research is needed to identify a clear impact on productivity.

Context. Mindfulness is a meditation technique whose main goal is keeping the mind calm and educating attention by focusing only on one thing at a time, usually breathing. The reported benefits of its continued practice can be of interest for Software Engineering students and practitioners, especially in tasks like conceptual modeling, in which concentration and clearness of mind are crucial.Goal. In order to evaluate whether Software Engineering students enhance their conceptual modeling performance after several weeks of mindfulness practice, a series of three controlled experiments were carried out at the University of Seville during three consecutive academic years (2013–2016) involving 130 students.Method. In all the experiments, the subjects were divided into two groups. While the experimental group practiced mindfulness, the control group was trained in public speaking as a placebo treatment. All the subjects developed two conceptual models based on a transcript of an interview, one before and another one after the treatment. The results were compared in terms of conceptual modeling quality (measured as effectiveness, i.e., the percentage of model elements correctly identified) and productivity (measured as efficiency, i.e., the number of model elements correctly identified per unit of time).Results. The statistically significant results of the series of experiments revealed that the subjects who practiced mindfulness developed slightly better conceptual models (their quality was 8.16 percent higher) and they did it faster (they were 46.67 percent more productive) than the control group, even if they did not have a previous interest in meditation.Conclusions. The practice of mindfulness improves the performance of Software Engineering students in conceptual modeling, especially their productivity. Nevertheless, more experimentation is needed in order to confirm the outcomes in other Software Engineering tasks and populations.

A Software Product Line (SPL) is a set of products built from a number of features, the set of valid products being defined by a feature model. Typically, it does not make sense to test all products defined by an SPL and one instead chooses a set of products to test (test selection) and, ideally, derives a good order in which to test them (test prioritisation). Since one cannot know in advance which products will reveal faults, test selection and prioritisation are normally based on objective functions that are known to relate to likely effectiveness or cost. This article introduces a new technique, the grid-based evolution strategy (GrES), which considers several objective functions that assess a selection or prioritisation and aims to optimise on all of these. The problem is thus a many-objective optimisation problem. We use a new approach, in which all of the objective functions are considered but one (pairwise coverage) is seen as the most important. We also derive a novel evolution strategy based on domain knowledge. The results of the evaluation, on randomly generated and realistic feature models, were promising, with GrES outperforming previously proposed techniques and a range of many-objective optimisation algorithms.

Web Application Programming Interfaces (APIs) specify how to access services and data over the network, typically using Web services. Web APIs are rapidly proliferating as a key element to foster reusability, integration, and innovation, enabling new consumption models such as mobile or smart TV apps. Companies such as Facebook, Twitter, Google, eBay or Netflix receive billions of API calls every day from thousands of different third-party applications and devices, which constitutes more than half of their total traffic.

Model transformations play a cornerstone role in Model-Driven Engineering (MDE), as they provide the essential mechanisms for manipulating and transforming models. The correctness of software built using MDE techniques greatly relies on the correctness of model transformations. However, it is challenging and error prone to debug them, and the situation gets more critical as the size and complexity of model transformations grow, where manual debugging is no longer possible. Spectrum-Based Fault Localization (SBFL) uses the results of test cases and their corresponding code coverage information to estimate the likelihood of each program component (e.g., statements) of being faulty. In this article we present an approach to apply SBFL for locating the faulty rules in model transformations. We evaluate the feasibility and accuracy of the approach by comparing the effectiveness of 18 different state-of-the-art SBFL techniques at locating faults in model transformations. Evaluation results revealed that the best techniques, namely Kulcynski2 , Mountford , Ochiai , and Zoltar , lead the debugger to inspect a maximum of three rules to locate the bug in around 74% of the cases. Furthermore, we compare our approach with a static approach for fault localization in model transformations, observing a clear superiority of the proposed SBFL-based method.

Web Application Programming Interfaces (APIs) allow systems to interact with each other over the network. Modern Web APIs often adhere to the REST architectural style, being referred to as RESTful Web APIs. RESTful Web APIs are decomposed into multiple resources (e.g., a video in the YouTube API) that clients can manipulate through HTTP interactions. Testing Web APIs is critical but challenging due to the difficulty to assess the correctness of API responses, i.e., the oracle problem. Metamorphic testing alleviates the oracle problem by exploiting relations (so-called metamorphic relations) among multiple executions of the program under test. In this paper, we present a metamorphic testing approach for the detection of faults in RESTful Web APIs. We first propose six abstract relations that capture the shape of many of the metamorphic relations found in RESTful Web APIs, we call these Metamorphic Relation Output Patterns (MROPs). Each MROP can then be instantiated into one or more concrete metamorphic relations. The approach was evaluated using both automatically seeded and real faults in six subject Web APIs. Among other results, we identified 60 metamorphic relations (instances of the proposed MROPs) in the Web APIs of Spotify and YouTube. Each metamorphic relation was implemented using both random and manual test data, running over 4.7K automated tests. As a result, 11 issues were detected (3 in Spotify and 8 in YouTube), 10 of them confirmed by the API developers or reproduced by other users, supporting the effectiveness of the approach.

Context. Many reports support the fact that some psycho--social aspects of software engineers are key factors for the quality of the software development process and its resulting products. Based on the experience of some of the authors after more than a year of practising mindfulness---a meditation technique aimed to increase clearness of mind and awareness---we guessed that it could be interesting to empirically evaluate whether mindfulness affects positively not only the behaviour but also the professional performance of software engineers.