Experiments are a commonly used method of research in software engineering (SE). Researchers report their experiments following detailed guidelines. However, researchers do not, in the field of test-driven development (TDD) at least, specify how they operationalized the response variables and, particularly, the measurement process. This article has three aims: (i) identify the response variable operationalization components in TDD experiments that study external quality; (ii) study their influence on the experimental results; (iii) determine if the experiment reports describe the measurement process components that have an impact on the results. We used two-part sequential mixed methods research. The first part of the research adopts a quantitative approach applying a statistical analysis of the impact of the operationalization components on the experimental results. The second part follows with a qualitative approach applying a systematic mapping study (SMS). The test suites, intervention types and measurers have an influence on the measurements and results of the statistical analysis of TDD experiments in SE. The test suites have a major impact on both the measurements and the results of the experiments. The intervention type has less impact on the results than on the measurements. While the measurers have an impact on the measurements, this is not transferred to the experimental results. On the other hand, the results of our SMS confirm that TDD experiments do not usually report either the test suites, the test case generation method, or the details of how external quality was measured. A measurement protocol should be used to ensure that the measurements made by different measurers are similar. It is necessary to report the test cases, the experimental task and the intervention type in order to be able to reproduce the measurements and statistical analyses, as well as to replicate experiments and build dependable families of experiments.

Context: The usability software quality characteristic aims to improve system user performance. In a previous study, we found evidence of the impact of a set of usability features from the viewpoint of users in terms of efficiency, effectiveness and satisfaction. However, the impact level appears to depend on the usability feature and suggest priorities with respect to their implementation depending on how they promote user performance.Objectives: We use a family of three experiments to increase the precision and generalization of the results in the baseline experiment and provide findings regarding the impact on user performance of the Abort Operation, Progress Feedback and Preferences usability mechanisms.Method: We conduct two replications of the baseline experiment in academic settings. We analyse the data of 366 experimental subjects and apply aggregation (meta-analysis) procedures.Results: We find that the Abort Operation and Preferences usability mechanisms appear to improve system usability a great deal with respect to efficiency, effectiveness and user satisfaction.Conclusions: We find that the family of experiments further corroborates the results of the baseline experiment. Most of the results are statistically significant, and, because of the large number of experimental subjects, the evidence that we gathered in the replications is sufficient to outweigh other experiments.

Context:Recent developments in natural language processing have facilitated the adoption of chatbots in typically collaborative software engineering tasks (such as diagram modelling). Families of experiments can assess the performance of tools and processes and, at the same time, alleviate some of the typical shortcomings of individual experiments (e.g., inaccurate and potentially biased results due to a small number of participants).Objective:Compare the usability of a chatbot for collaborative modelling (i.e., SOCIO) and an online web tool (i.e., Creately).Method:We conducted a family of three experiments to evaluate the usability of SOCIO against the Creately online collaborative tool in academic settings.Results:The student participants were faster at building class diagrams using the chatbot than with the online collaborative tool and more satisfied with SOCIO. Besides, the class diagrams built using the chatbot tended to be more concise —albeit slightly less complete.Conclusion:Chatbots appear to be helpful for building class diagrams. In fact, our study has helped us to shed light on the future direction for experimentation in this field and lays the groundwork for researching the applicability of chatbots in diagramming.

The General Data Protection Regulation (GDPR) is the core of digital privacy legislation across Europe (EU), and it applies to processing carried out by organisations operating within and outside the EU that offer goods or services to individuals in the EU, including software products.Nevertheless, software teams are generally unaware of the legal requirements for personal data protection and its application throughout the software life cycle.In this paper, we propose a comprehensive guidance to integrate compliance with GDPR requirements within the Unified Software Development Process (UP) across the entire lifetime.

Recent natural language processing developments have facilitated the adoption of chatbots in typically collaborative software engineering tasks.Families of experiments can overcome limitations in terms of the sample size of individual experiments.To experimentally evaluate the usability of a chatbot for collaborative modelling (i.e., SOCIO) and tackle some of the typical shortcomings of individual experiments, we conducted a family of three experiments to evaluate the usability of SOCIO against the Creately online collaborative tool.Results show that the participants were more satisfied with the chatbot than with the online collaborative tool and that they also created class diagrams faster using the chatbot.We conclude that chatbots may be helpful for creating class diagrams.

Context: The interest in developing chatbots is on the rise as the usability evaluation is an essential step in the chatbot development process; the number of experimental studies of chatbot usability has grown as well.Objective: Aggregating and concluding the features and metrics used to evaluate the usability of chatbots in experiments, to identify the state of the art of chatbots usability experimentation.Method: A systematic mapping study has been conducted, searching in five scientific databases.Results: Of 363 papers, 14 papers with experiments were selected as the primary studies.The published works in this area were initiated in 2018.Control tools are applied commonly in experiments.Various advantages and shortages of chatbot usability experiments were revealed, for example, most of the experiments do not provide raw data, and only one of the identified works replicated the experiment.Conclusions: An increased interest in usability experimentation of chatbots is observed in recent years.The chatbot usability experiment should be more replicable to improve the reliability of experimental results.

Modelling is a fundamental activity in software engineering, which is often performed in collaboration. For this purpose, on-line tools running on the cloud are frequently used. However, recent advances in Natural Language Processing have fostered the emergence of chatbots, which are increasingly used for all sorts of software engineering tasks, including modelling. To evaluate to what extent chatbots are suitable for collaborative modelling, we conducted an experimental study with 54 participants, to evaluate the usability of a modelling chatbot called SOCIO, comparing it with the on-line tool Creately. We employed a within-subjects cross-over design of 2 sequences and 2 periods. Usability was determined by attributes of efficiency, effectiveness, satisfaction and quality of the results. We found that SOCIO saved time and reduced communication effort over Creately. SOCIO satisfied users to a greater extent than Creately, while in effectiveness results were similar. With respect to diagram quality, SOCIO outperformed Creately in terms of precision, while solutions with Creately had better recall and perceived success. However, in terms of accuracy and error scores, both tools were similar.

Background: The use of chatbots has increased considerably in recent years.These are used in different areas and by a wide variety of users.Due to this fact, it is essential to incorporate usability in their development.Aim: Our objective is to identify the state of the art in chatbot usability and applied human-computer interaction techniques, to analyze how to evaluate chatbots usability.Method: A systematic mapping study has been conducted, searching the main scientific databases.The search retrieved 170 citations and 19 articles were retained as primary studies.Results: The works were categorized according to four criteria: usability techniques, usability characteristics, research methods and type of chatbots.Conclusions: Chatbot usability is a very incipient field of research, where the published studies are mainly surveys, usability tests, and rather informal experimental studies.Hence, it becomes necessary to perform more formal experiments to measure user experience, and exploit these results to provide usability-aware design guidelines.

Background: The use of chatbots has increased considerably in recent years. These are used in different areas and by a wide variety of users. Due to this fact, it is essential to incorporate usability in their development. Aim: Our objective is to identify the state-of-the-art in chatbot usability and applied human–computer interaction techniques, to analyze how to evaluate chatbot usability. Method: We have conducted a systematic mapping study, by searching the main scientific databases. The search retrieved 170 references and 21 articles were retained as primary studies. Results: The works were categorized according to four criteria: usability techniques, usability characteristics, research methods and type of chatbots. Conclusions: Chatbot usability is still a very incipient field of research where the published studies are mainly surveys, usability tests, and rather informal experimental studies. Hence, it becomes necessary to perform more formal experiments to measure user experience, and exploit these results to provide usability-aware design guidelines.

As a result of the growth of non-developer users of OSS applications, usability has over the last ten years begun to attract the interest of the open source software (OSS) community. The OSS community has some special characteristics (such as worldwide geographical distribution of both users and developers and missing resources) which are an obstacle to the direct adoption of many usability techniques as specified in the human-computer interaction (HCI) field. The aim of this research is to adapt and evaluate the feasibility of applying four usability techniques: user profiles, personas, direct observation and post-test information to four OSS projects from the viewpoint of the development team. The applied research method was a multiple case study of the following OSS projects: Quite Universal Circuit Simulator, PSeInt, FreeMind and OpenOffice Writer. We formalized the application procedure of each of the adapted usability techniques. We found that either there were no procedures for adopting usability techniques in OSS or they were not fully systematized. Additionally, we identified the adverse conditions that are an obstacle to their adoption in OSS and propose the special adaptations required to overcome the obstacles. To avoid some of the adverse conditions, we created web artefacts (online survey, wiki and forum) that are very popular in the OSS field. It is necessary to adapt usability techniques for application in OSS projects considering their idiosyncrasy. Additionally, we found that there are obstacles (for example, number of participant users, biased information provided by developers) to the application of the techniques. Despite these obstacles, it is feasible to apply the adapted techniques in OSS projects.

The growth in the number of non-developer open source software (OSS) application users and the escalating use of these applications have both created a need for, and interest in, developing usable OSS.OSS communities are unclear about which techniques to use in each activity of the development process.The aim of our research is to adopt the visual brainstorming usability technique in the HistoryCal OSS project and determine the feasibility of adapting the technique for application.To do this, we participated as volunteers in the HistoryCal project.We used the case study research method to investigate technique application and community participation.As a result, we identified adverse conditions that were an obstacle to technique application and modified the technique to make it applicable.We can conclude from our experience that these changes were helpful for applying the technique, although it was not easy to recruit OSS users to participate in usability technique application.

Background: Interest in the integration of the agile software development process (ASDP) and user-centred design (UCD) has grown over the last decade. However, there are not many papers that study this issue holistically and uncover the current state of this integration. Aim: This study sets out to answer the following research question: What is the current state of integration between agile processes and usability? Method: We conduct a systematic mapping study (SMS). A SMS is a form of systematic literature review that aims to identify and classify the research papers published about a specified issue. Results: We retrieved a total of 161 primary studies, which we categorized according to four criteria: process, practice, team and technology integration. The largest group refers to process integration with a total of 76 primary studies. Judging by the increase in the number of publications over the last decade, the integration of human-computer interaction (HCI) activities and techniques into the ASDP is an issue of notable interest. Conclusions: There are very few papers that study the literature holistically and report the current state of usability in the ASDP. Additionally, there are no formalized proposals for adopting usability techniques in ASDP.

We present a methodology for creating hypermedia materials derived from face-to-face tutoring sessions between a tutor and graduate students. To create the hypermedia materials, the tutor and the student used a smart pen which allowed to record the conversation and digitalize the notes being taken. The production of hypermedia material is based on the use of visual representations and text to help students go from concrete to abstract thinking and vice versa. We point out that hypermedia materials are audio-visual narratives (i.e., dynamic graphics, diagrams) that facilitate the representation of co-constructed shared knowledge and let participants navigate between oral and textual information. This methodology allows the production of individualized material without investing additional time in editing and designing. The hypermedia based tutoring (HBT) model is highly valued by students since it helps them to go over the discussions with the tutor and review the thinking process that both constructed during the session. HBT becomes a creative form of communicating and representing information that challenges the tutor and student to develop new skills and ways of thinking. The model that we propose here requires to change traditional tutor and student roles and to create learning experiences that do not overlook students' needs.

Context: Over the last decade there has been a growing interest in the integration of agile software development process (ASDP) and user-centred design (UCD).However, there are no papers that study which usability techniques related to requirements engineering are being adopted in the ASDP, and there are no formalized proposals for their adoption.Objective: Identify which techniques related to requirements engineering activities are being adopted in the ASDP and determine how they are being adopted.Method: We have conducted a systematic mapping study (SMS) to retrieve the literature reporting the application of usability techniques in the ASDP.We analysed these techniques using a catalogue of techniques compiled by software engineering researchers.We then determined the manner in which the techniques that are being used in the ASDP were adopted.Results: The agile community is very much interested in adopting usability techniques.The most used techniques are Personas, contextual inquiry and prototyping.Conclusions: This research offers an overview of the adoption of usability techniques related to requirements engineering in ASDPs and reports how they are being adopted.We found that some of the techniques are being adapted for adoption.

Usability is a software system quality attribute. Although software engineers originally considered usability to be related exclusively to the user interface, it was later found to affect the core functionality of software applications. As of then, proposals for addressing usability at different stages of the software development cycle were researched. The objective of this paper is to present three reusable solutions at detailed design and programming level in order to effectively implement the Abort Operation, Progress Feedback and Preferences usability functionalities in web applications. To do this, an inductive research method was applied. We developed three web applications including the above usability functionalities as case studies. We looked for commonalities across the implementations in order to induce a general solution. The elements common to all three developed applications include: application scenarios, functionalities, responsibilities, classes, methods, attributes and code snippets. The findings were specified as an implementation-oriented design pattern and as programming patterns in three languages. Additional case studies were conducted in order to validate the proposed solution. The independent developers used the patterns to implement different applications for each case study. As a result, we found that solutions specified as patterns can be reused to develop web applications.

The research about the formalization and conduction of controlled experiments in software engineering has reported important insights and guidelines for their organization. However, the computational support to formalize and execute controlled experiments still requires deeper investigation. In this context, this paper presents an empirical study that evaluates a domain-specific language (DSL) proposed to formalize controlled experiments in software engineering. The language is part of a model-driven approach that allows the generation of executable workflows for the experiment participants, according to the statistical design of the experiment. Our study involves the modeling of 16 software engineering experiments to analyze the completeness and expressiveness of the investigated DSL when specifying different controlled experiments. The results highlight several limitations of the DSL that affect the formalization and execution of experiments. These outcomes were used to extend and improve the evaluated DSL. Finally, the improved version of the language was used to model the same experiments in order to illustrate the benefits of the proposed improvements.

Abstract?Background: There is no globally accepted open source software development process to define how open source software is developed in practice. A process description is important for coordinating all the software development activities involving both people and technology. Aim: The research question that this study sets out to answer is: What activities do open source software process models contain? The activity groups on which it focuses are Concept Exploration, Software Requirements, Design, Maintenance and Evaluation. Method: We conduct a systematic mapping study (SMS). A SMS is a form of systematic literature review that aims to identify and classify available research papers concerning a particular issue. Results: We located a total of 29 primary studies, which we categorized by the open source software project that they examine and by activity types (Concept Exploration, Software Requirements, Design, Maintenance and Evaluation). The activities present in most of the open source software development processes were Execute Tests and Conduct Reviews, which belong to the Evaluation activities group. Maintenance is the only group that has primary studies addressing all the activities that it contains. Conclusions: The primary studies located by the SMS are the starting point for analyzing the open source software development process and proposing a process model for this community. The papers in our paper pool that describe a specific open source software project provide more regarding our research question than the papers that talk about open source software development without referring to a specific open source software project.

This paper analyses the relationships between personality, team processes, task characteristics, product quality and satisfaction in software development teams. The data analysed here were gathered from a sample of 35 teams of students (105 participants) from a Spanish university. These teams applied an adaptation of an agile methodology, eXtreme Programming (XP), to develop the same software product.

Abstract This paper is based on the premise that people's behavioural competencies or characteristics of professional conduct influence the effectiveness and efficiency with which they perform a predetermined role in the software process. We propose a capabilities‐oriented process model that includes traditional elements of the software process (activities, products, techniques, people and roles) and the original element of this paper (capabilities). With the aim of adding behavioural competencies to the process model, we define the capability–person and capability–role relationships involved in software development. Additionally, we propose two procedures that are based on each of these relationships: a procedure that can be used to determine the capabilities of the members of a development team; and a procedure that can be used to assign people to perform roles depending on their capabilities and the capabilities demanded by the roles. Finally, the person–capabilities–role relationship has been empirically validated. The results yielded by this experiment confirm the hypothesis that assigning people to roles according to their capabilities and the capabilities demanded by the role improves software development. Copyright © 2004 John Wiley & Sons, Ltd.

In this paper an empiric validation of a Person to Role Allocation Process is presented. In this process the allocation of persons to fulfill roles is made according to the capacities that the persons possess and those required by the roles in the software process. A set of experiments are carried out dealing with the development of the Initiation, Planning and Estimation Process, Domain Study Process, Requirements Analysis Process and Design Process of eight projects. It was proved that the estimated time deviation, as well as the errors found in the technical reviews of requirements specification, were less when the persons fulfilling the roles of planning engineer, domain analyst, requirements specificator and designer were allocated, according to the proposed process, considering the set of critical human capacities.

In this paper a culture-centered multilevel software process cycle model (MPCM) is presented. This model interrelates the socio-cultural, scientific/technological and paradigmatological environments. The proposed model is composed of three environments made up of the ecological universe, engineering, management, development and evaluation levels which represent the process cycle, and the lines of “what”, “who” and “how”. MPCM considers cultural, social, organizational and personnel-related issues, normally absent in the traditional software process models. A cultural comprehensive procedure for the realization of the engineering level is developed. The different stages and their products of the Cultural Procedure of the MPCM are represented. This procedure is proposed in order to model the organization, to reveal its culture and to determine the abilities that a person (user or developer) has to reallocate him/her in the software process.

Software engineering (SE) and knowledge engineering (KE) develop software systems using different construction process models. Because of the growing complexity of the problems to be solved by computers, the conventional systems (CS) and knowledge-based systems (KBS) software process is at present passing through a period of integration. In this paper, we propose a software process model applicable to both CS and KBS. The model designed is declarative, that is, it indicates what is done to build a software system. Its goal is to provide software and knowledge engineers with a techno-conceptual tool to develop systems comprising both traditional and knowledge-based software.