Abstract Context In recent years, the multi-class classification of non-functional requirements has seen improvements through the use of Machine Learning algorithms. However, challenges such as data scarcity and class imbalance persist, particularly for languages other than English, such as Spanish. Objective This study aims to analyze the performance metrics of Machine Learning algorithms for classifying non-functional requirements translated into and originally written in Spanish. It evaluates the effectiveness of dataset balancing techniques and conducts cross-dataset validation to assess the generalizability of the models. Method A dataset balancing process was conducted using a combination of oversampling and undersampling techniques. Six algorithms were trained in two experiments using a hyperparameter tuning process, employing two different datasets: PROMISE_exp_translated and the newly PROMISE_exp_balanced . The best-performing models were further tested on unseen data to evaluate their generalizability. Results Logistic Regression and Naive Bayes demonstrated superior performance on the translated dataset, achieving f1-scores of 82% and 81%, respectively. Although overall performance decreased on the balanced dataset, specific underrepresented classes such as Portability and Fault Tolerance benefited from the balancing process. Conclusion Shallow Machine Learning algorithms are effective for classifying Spanish non-functional requirements, particularly when addressing data imbalance. The study highlights the importance of dataset balancing in improving classification performance for specific classes and provides insights into the challenges of generalizing models across datasets.

Background: Ensemble models play an important role in integrating multiple classifiers in a wide range of applications, such as medical diagnosis, sentiment analysis, and financial market trends. In Requirements Engineering (RE), automatic requirements classification can be improved by the utilization of these models. Aims: This paper analyses the performance metrics of voting and stacking ensemble models for requirements classification prediction. Moreover, a cross-dataset validation was performed for the meta-models generated using the stacking ensemble method. Methods: Some previously trained base models and two datasets of software requirements written in Spanish (translated PROMISE_exp and ReSpa dataset) were used to build the ensemble models. Results: The results indicate that the stacking model achieved a weighted F1-score of 0.828 using Support Vector Machine (SVM) and Multi-layer Perceptron (MLP) for translated PROMISE_exp dataset. For the ReSpa dataset, the stacking model achieved a weighted F1-score of 0.890 using Logistic Regression (LR). Conclusion: This study confirms a slight improvement in the performance of binary requirements classification using stacking ensemble methods over voting and most individual base models. Moreover, combining all models outperforms combinations that include only Shallow ML or DL models.

Machine Learning (ML) algorithms have become a powerful instrument in software requirements classification. Nevertheless, most of the research focusing on requirements is in English, with less attention to other languages. Given a lack of datasets in Spanish, we created a new dataset from a collection of requirements from final degree projects from the University of A Coruña. In this paper, we investigate which combinations of text vectorization techniques with ML algorithms perform best for requirements classification in a Spanish dataset. We found that SVM with TF-IDF gives the highest f1-score (0.95 and 0.79 for functional and non-functional classification).

[Context] Research on the professional occupation of Requirements Engineering (RE) indicated soft skills as very important in employers' search for RE specialists. A very long list of soft skills came out of prior research, with proficiency in English being ranked as the Number One skill. [Question/problem] To what extent, however, are these skills deemed critical in RE in practice, and also in making hiring decisions in RE, and why? Which skills are to be ranked higher in regard to hiring decisions, and which lower? In this study, we compare results based on analyses of job ads with practitioners' perceptions in real-life organizations. [Methodology] We executed two focus groups (FGs) with RE practitioners from industry, as part of the REFSQ 2018 conference. [Principal ideas/results] The soft skills from previous studies were discussed in depth with the FGs participants regarding where and why these skills are needed or sometimes are even unnecessary. Here, we report a ranked list of soft skills, among which communication skills were considered as the most important. Next, some of the skills ranked as the most important in job ads, are not needed everywhere. [Contribution] The results lead to a more differentiated view on those soft skills that requirements engineers need in practice, and the reasons for these skills' importance.

Background: A software sustainability model is commonly defined in terms of social, economic, technical and environmental dimensions. A key challenge is its characterization regarding sustainability requirements and their dependencies since it depends on, and varies with, the type of software system, and changes in operational contexts. Aims: With the purpose of facilitating the definition of a context-dependent software sustainability model, we introduce nichesourcing as a type of distributed problem-solving paradigm, named crowdsourcing. Method: First a set of nichesourcing requirements were identified, and an existing nichesourcing methodology, originally used in the cultural heritage domain was analyzed with respect to its applicability in Software Engineering. Results: As a result, we propose an adapted methodology for nichesourcing the knowledge on sustainability requirements and their dependencies. Strengths and challenges for crowdsourced software engineering are also identified.

Background. Software sustainability is defined in terms of multiple and interdependent dimensions (economic, social, technical and environmental). Preliminary initiatives have investigated the contribution of certain quality attributes to sustainability dimensions. Problem. Despite these valuable efforts, the characterization of software sustainability is still a key challenge. This entails how sustainability can be embraced in the design of software systems by identifying the relevant software quality attributes (QAs) and their dependencies. Both attributes and dependencies vary heavily with amongst others the type of software system and its operational context. Aim and Method. We followed a multiple case study research method with the main objective of investigating the applicability of the sustainability model in different contexts. We aim also to enrich our model, by means of identifying missing quality attributes or new contributions to the sustainability dimensions. We selected two software projects as cases of our study, where each one was independently conducted in specific situations. Results. The results of the study show that the relevant quality requirements identified in both projects (cases) are covered by most of the QAs related to the social (82%) and technical (83%) dimensions. Moreover, some QAs that were not addressed in the corresponding projects, their relevance like context completeness, and flexibility were acknowledged. These results suggest that the software sustainability model could support the identification of relevant QAs. The case study also contributed to identify QAs that had not been considered in the economic, technical and social dimensions of the sustainability-quality model.

While modern software development technologies enhance the capabilities of model-based/driven development, they introduce challenges for testers such as how to perform early testing at model level to ensure the quality of the model. In this context, we have developed an early testing technique supported by the CoSTest tool to validate requirements at model level. In this paper we describe an empirical evaluation of CoSTest with respect to its effectiveness in terms of its fault detection and test suite adequacy. This evaluation is carried out by model-level mutation testing using first order mutants (created by injection of a single fault) and high order mutants (containing more than one fault) with seven conceptual schemas (of different sizes) that represent the functionality of different software systems in different domains. Our findings show that the tests generated by CoSTest are effective at killing a large number of mutants. However, there are also some fault types (e.g. delete the references to a class attribute or an operation call in a constraint) that our test suites were not able to detect. CoSTest was more effective in terms of detecting fault types using high order mutants that first order mutants. Thus, CoSTest's effectiveness is affected by the mutant type tested.

We present CoSTest, a tool that supports the validation of Conceptual Schemas by using testing. The tool implements techniques for transforming instantiations from a Requirements Model into test case implementations by supporting a Model-driven architecture.

In Model-Driven Development (MDD), defects are managed at the level of conceptual models because the other artefacts are generated from them, such as more refined models, test cases and code. Although some studies have reported on defect types at model level, there still does not exist a clear and complete overview of the defect types that occur at the abstraction level. This paper presents a systematic mapping study to identify the model defect types reported in the literature and determine how they have been detected. Among the 282 articles published in software engineering area, 28 articles were selected for analysis. A total of 226 defects were identified, classified and their results analysed. For this, an appropriate defect classification scheme was built based on appropriate dimensions for models in an MDD context.

[Context] Automated test case design and execution at the GUI level of applications is not a fact in industrial practice. Tests are still mainly designed and executed manually. In previous work we have described TESTAR, a tool which allows to set-up fully automatic testing at the GUI level of applications to find severe faults such as crashes or non-responsiveness. [Method] This paper aims at the evaluation of TESTAR with an industrial case study. The case study was conducted at SOFTEAM, a French software company, while testing their Modelio SaaS system, a cloud-based system to manage virtual machines that run their popular graphical UML editor Modelio. [Goal] The goal of the study was to evaluate how the tool would perform within the context of SOFTEAM and on their software application. On the other hand, we were interested to see how easy or difficult it is to learn and implant our academic prototype within an industrial setting. [Results] The effectiveness and efficiency of the automated tests generated with TESTAR can definitely compete with that of the manual test suite. [Conclusions] The training materials as well as the user and installation manual of TESTAR need to be improved using the feedback received during the study. Finally, the need to program Java-code to create sophisticated oracles for testing created some initial problems and some resistance. However, it became clear that this could be solved by explaining the need for these oracles and compare them to the alternative of more expensive and complex human oracles. The need to raise consciousness that automated testing means programming solved most of the initial problems.

Several initiatives have been proposed in recent years with the purpose of investigating how to tackle sustainability issues within software systems and software engineering. However, there is a lack of attention from the research community in exploring sustainability as an emergent quality property of the software systems, especially when they are becoming more and more distributed, heterogeneous, and dynamic. In order to involve a more international research community, and provide an open forum for researchers and practitioners working on green and sustainable measurement and metrics, an interactive workshop is planned within the IWSM-Mensura conference.

[Context] Replication Data Management (RDM) aims at enabling the use of data collections from several itera-tions of an experiment. However, there are several major chal-lenges to RDM from integrating data models and data from em-pirical study infrastructures that were not designed to cooperate, e.g., data model variation of local data sources. [Objective] In this paper we analyze RDM needs and evaluate conceptual RDM approaches to support replication researchers. [Method] We adapted the ATAM evaluation process to (a) analyze RDM use cases and needs of empirical replication study research groups and (b) compare three conceptual approaches to address these RDM needs: central data repositories with a fixed data model, heterogeneous local repositories, and an empirical ecosystem. [Results] While the central and local approaches have major issues that are hard to resolve in practice, the empirical ecosys-tem allows bridging current gaps in RDM from heterogeneous data sources. [Conclusions] The empirical ecosystem approach should be explored in diverse empirical environments.

[Context] Numerous combinatorial testing techniques are available for generating test cases. However, many of them are never used in practice. [Objective] Considering that learn ability plays a vital role in initial adoption or rejection of a technology, in this paper we aim to investigate the learnability of a combinatorial testing tool in an industrial environment. [Method] A case study research method was designed and conducted, by including i) the definition of learnability measures for test cases models built using a combinatorial testing tool. ii) A training program was also implemented. iii) Qualitative and quantitative evaluation based on a three-level strategy was carried out (Reaction, Learning, and Performance). [Results] At the first level, the tool was perceived as easy to learn by the trainees (from a five-point ordinal scale). However, at the second level, during hands-on learning, it changed slightly: According to the working diaries, there were major difficulties. At third level, analyzing the learning curve of each trainee, we observe that semantic errors made per each subject were reduced slightly over the time.

This paper aims at evaluating a set of automated tools of the FITTEST EU project within an industrial case study. The case study was conducted at the IBM Research lab in Haifa, by a team responsible for building the testing environment for future development versions of an IBM system management product. The main function of that product is resource management in a networked environment. This case study has investigated whether current IBM Research testing practices could be improved or complemented by using some of the automated testing tools that were developed within the FITTEST EU project. Although the existing Test Suite from IBM Research (TSibm) that was selected for comparison is substantially smaller than the Test Suite generated by FITTEST (TSfittest), the effectiveness of TSfittest, measured by the injected faults coverage is significantly higher (50% vs 70%). With respect to efficiency, by normalizing the execution times, we found the TSfittest runs faster (9.18 vs. 6.99). This is due to the fact that the TSfittest includes shorter tests. Within IBM Research and for the testing of the target product in the simulated environment: the FITTEST tools can increase the effectiveness of the current practice and the test cases automatically generated by the FITTEST tools can help in more efficient identification of the source of the identified faults. Moreover, the FITTEST tools have shown the ability to automate testing within a real industry case.

Usability is a quality attribute related to effectiveness, efficiency and satisfaction of the end-users when they interact with a system. There are sound usability design guidelines that aim to optimize the system usability. However, it is difficult to work with them if there is not a previous background in the Human-Computer Interaction area. Moreover, in the Software Engineering community, there are not specific methods to capture usability requirements. Usually, usability requirements are gathered with other non-functional requirements such as security or reliability, even though the goals of these non-functional requirements are not the same as the goals of usability. This problem arises when we aim to include the usability requirements capture in a Model Driven Development (MDD) process, where an effective capture technique with an unambiguous notation is needed to represent these requirements. In this paper, we propose a new method to capture usability requirements at the early stages of the software development process in such a way that non-experts in usability can use it. In a first step, experts in usability organize several interface design guidelines and usability guidelines in a tree structure. Next, these guidelines are shown to the analyst (non-expert in usability) through questions that she/he must ask the end-users. Answers to these questions mark the path through the tree structure. At the end of the process, if we gather all the end-user's answers, we have the usability requirements. These usability requirements could be transformed into a conceptual model of any existing MDD method by means of model to model transformations.

This short paper summarizes and discusses the result of an iterative construction and evaluation of a checklist for writing and reading reports about experimental and observational research.

In this paper we analyse possible reasons for the relatively low use of theories in software engineering (SE) papers found by Hannay et al. [16]. We provide an initial explanation in terms of properties of theories, test this by analyzing 32 of the 40 theories reviewed by Hannay et al., and then revise our analysis based on this test. Our analysis revealed that background theories from other disciplines are context-free and make idealizing assumptions, which make it easier for them to be (re)used across settings. Theories built in SE are mid-range and context-sensitive, and make less idealizing assumptions. This is normal for engineering theories, but it does make them harder to (re)use across settings. We also found that background theories from other disciplines usually provide explanations for phenomena in terms of mechanisms, whereas SE theories are statistical models of phenomena observed in an experiment, which also makes them harder to (re)use across settings. We end the paper with a recommendation of bottom-up development of theories about mechanisms in software engineering projects by doing case studies.

Usability is currently a key feature for developing quality systems. A system that satisfies all the functional requirements can be strongly rejected by end-users if it presents usability problems. End-users demand intuitive interfaces and an easy interaction in order to simplify their work. The first step in developing usable systems is to determine whether a system is or is not usable. To do this, there are several proposals for measuring the system usability. Most of these proposals are focused on the final system and require a large amount of resources to perform the evaluation (end-users, video cameras, questionnaires, etc.). Usability problems that are detected once the system has been developed involve a lot of reworking by the analyst since these changes can affect the analysis, design, and implementation phases. This paper proposes a method to minimize the resources needed for the evaluation and reworking of usability problems. We propose an early usability evaluation that is based on conceptual models. The analyst can measure the usability of attributes that depend on conceptual primitives. This evaluation can be automated taking as input the conceptual models that represent the system abstractly.

Nowadays several Computer-Aided Software Engineering environments exploit Model-Driven Engineering (MDE) techniques in order to generate a single user interface for a given computing platform or multi-platform user interfaces for several computing platforms simultaneously. Therefore, there is a need to assess the usability of those generated user interfaces, either taken in isolation or compared to each other. This paper describes an MDE approach that generates multi-platform graphical user interfaces (e.g., desktop, web) that will be subject to an exploratory controlled experiment. The usability of user interfaces generated for the two mentioned platforms and used on multiple display devices (i.e., standard size, large, and small screens) has been examined in terms of satisfaction, effectiveness and efficiency. An experiment with a factorial design for repeated measures was conducted for 31 participants, i.e., postgraduate students and professors selected by convenience sampling. The data were collected with the help of questionnaires and forms and were analyzed using parametric and non-parametric tests such as ANOVA with repeated measures and Friedman's test, respectively. Efficiency was significantly better in large screens than in small ones as well as in the desktop platform rather than in the web platform, with a confidence level of 95%. The experiment also suggests that satisfaction tends to be better in standard size screens than in small ones. The results suggest that the tested MDE approach should incorporate enhancements in its multi-device/platform user interface generation process in order to improve its generated usability.

This paper reports on an experiment that investigates the predictability of software project size from software product size. The predictability research problem is analyzed at the stage of early requirements by accounting the size of functional requirements as well as the size of non-functional requirements. The experiment was carried out with 55 graduate students in Computer Science from Concordia University in Canada. In the experiment, a functional size measure and a project size measure were used in building estimation models for sets of web application development projects. The results show that project size is predictable from product size. Further replications of the experiment are, however, planed to obtain more results to confirm or disconfirm our claim.

This paper describes an empirical mapping study, which was designed to identify what aspects of software requirement specifications (SRS) are empirically evaluated, in which context, and by using which research method. On the basis of 46 identified and categorized primary studies, we found that understandability is the most commonly evaluated aspect of SRS, experiments are the most commonly used research method, and the academic environment is where most empirical evaluation takes place.

Requirements engineering (RE) is a relatively young discipline, and still many advances have been achieved during the last decades. In particular, numerous RE methods have been proposed. However, there is a growing concern for empirical validations that assess RE proposals and statements. This paper is related to the evaluation of the quality of functional requirements specifications, focusing on completeness and granularity. To do this, several concepts related to conceptual model quality are presented; these concepts lead to the definition of metrics that allow measuring certain aspects of a requirements model quality (e.g. degree of functional encapsulations completeness with respect to a reference model, number of functional fragmentation errors). A laboratory experiment with master students has been carried out, in order to compare (using the proposed metrics) two RE approaches; namely, Use Cases and Communication Analysis. Results indicate greater quality (in terms of completeness and granularity) when communication analysis guidelines are followed. Moreover, interesting issues arise from experimental results, which invite further research.

Traditionally, usability has been evaluated taking into account the user's satisfaction when interacting with the software system. However, in a Model-Driven Development (MDD) process, where conceptual models are the main resource for software system generation, the usability can potentially be evaluated at earlier stages. This work goes one step further proposing that certain usability attributes, specifically internal understandability attributes, can be measured from Conceptual Models. This work presents an empirical study carried out to evaluate the proposal. The goal of this study is to evaluate whether the value measured using our proposal is related to the understandability value perceived by the end user. From the analysis of the empirical results obtained, several weaknesses of the proposal are stated.