Experiments are a commonly used method of research in software engineering (SE). Researchers report their experiments following detailed guidelines. However, researchers do not, in the field of test-driven development (TDD) at least, specify how they operationalized the response variables and, particularly, the measurement process. This article has three aims: (i) identify the response variable operationalization components in TDD experiments that study external quality; (ii) study their influence on the experimental results; (iii) determine if the experiment reports describe the measurement process components that have an impact on the results. We used two-part sequential mixed methods research. The first part of the research adopts a quantitative approach applying a statistical analysis of the impact of the operationalization components on the experimental results. The second part follows with a qualitative approach applying a systematic mapping study (SMS). The test suites, intervention types and measurers have an influence on the measurements and results of the statistical analysis of TDD experiments in SE. The test suites have a major impact on both the measurements and the results of the experiments. The intervention type has less impact on the results than on the measurements. While the measurers have an impact on the measurements, this is not transferred to the experimental results. On the other hand, the results of our SMS confirm that TDD experiments do not usually report either the test suites, the test case generation method, or the details of how external quality was measured. A measurement protocol should be used to ensure that the measurements made by different measurers are similar. It is necessary to report the test cases, the experimental task and the intervention type in order to be able to reproduce the measurements and statistical analyses, as well as to replicate experiments and build dependable families of experiments.

Context:Recent developments in natural language processing have facilitated the adoption of chatbots in typically collaborative software engineering tasks (such as diagram modelling). Families of experiments can assess the performance of tools and processes and, at the same time, alleviate some of the typical shortcomings of individual experiments (e.g., inaccurate and potentially biased results due to a small number of participants).Objective:Compare the usability of a chatbot for collaborative modelling (i.e., SOCIO) and an online web tool (i.e., Creately).Method:We conducted a family of three experiments to evaluate the usability of SOCIO against the Creately online collaborative tool in academic settings.Results:The student participants were faster at building class diagrams using the chatbot than with the online collaborative tool and more satisfied with SOCIO. Besides, the class diagrams built using the chatbot tended to be more concise —albeit slightly less complete.Conclusion:Chatbots appear to be helpful for building class diagrams. In fact, our study has helped us to shed light on the future direction for experimentation in this field and lays the groundwork for researching the applicability of chatbots in diagramming.

Recent natural language processing developments have facilitated the adoption of chatbots in typically collaborative software engineering tasks.Families of experiments can overcome limitations in terms of the sample size of individual experiments.To experimentally evaluate the usability of a chatbot for collaborative modelling (i.e., SOCIO) and tackle some of the typical shortcomings of individual experiments, we conducted a family of three experiments to evaluate the usability of SOCIO against the Creately online collaborative tool.Results show that the participants were more satisfied with the chatbot than with the online collaborative tool and that they also created class diagrams faster using the chatbot.We conclude that chatbots may be helpful for creating class diagrams.

Context: The interest in developing chatbots is on the rise as the usability evaluation is an essential step in the chatbot development process; the number of experimental studies of chatbot usability has grown as well.Objective: Aggregating and concluding the features and metrics used to evaluate the usability of chatbots in experiments, to identify the state of the art of chatbots usability experimentation.Method: A systematic mapping study has been conducted, searching in five scientific databases.Results: Of 363 papers, 14 papers with experiments were selected as the primary studies.The published works in this area were initiated in 2018.Control tools are applied commonly in experiments.Various advantages and shortages of chatbot usability experiments were revealed, for example, most of the experiments do not provide raw data, and only one of the identified works replicated the experiment.Conclusions: An increased interest in usability experimentation of chatbots is observed in recent years.The chatbot usability experiment should be more replicable to improve the reliability of experimental results.

Modelling is a fundamental activity in software engineering, which is often performed in collaboration. For this purpose, on-line tools running on the cloud are frequently used. However, recent advances in Natural Language Processing have fostered the emergence of chatbots, which are increasingly used for all sorts of software engineering tasks, including modelling. To evaluate to what extent chatbots are suitable for collaborative modelling, we conducted an experimental study with 54 participants, to evaluate the usability of a modelling chatbot called SOCIO, comparing it with the on-line tool Creately. We employed a within-subjects cross-over design of 2 sequences and 2 periods. Usability was determined by attributes of efficiency, effectiveness, satisfaction and quality of the results. We found that SOCIO saved time and reduced communication effort over Creately. SOCIO satisfied users to a greater extent than Creately, while in effectiveness results were similar. With respect to diagram quality, SOCIO outperformed Creately in terms of precision, while solutions with Creately had better recall and perceived success. However, in terms of accuracy and error scores, both tools were similar.

Background: The use of chatbots has increased considerably in recent years.These are used in different areas and by a wide variety of users.Due to this fact, it is essential to incorporate usability in their development.Aim: Our objective is to identify the state of the art in chatbot usability and applied human-computer interaction techniques, to analyze how to evaluate chatbots usability.Method: A systematic mapping study has been conducted, searching the main scientific databases.The search retrieved 170 citations and 19 articles were retained as primary studies.Results: The works were categorized according to four criteria: usability techniques, usability characteristics, research methods and type of chatbots.Conclusions: Chatbot usability is a very incipient field of research, where the published studies are mainly surveys, usability tests, and rather informal experimental studies.Hence, it becomes necessary to perform more formal experiments to measure user experience, and exploit these results to provide usability-aware design guidelines.

Background: The use of chatbots has increased considerably in recent years. These are used in different areas and by a wide variety of users. Due to this fact, it is essential to incorporate usability in their development. Aim: Our objective is to identify the state-of-the-art in chatbot usability and applied human–computer interaction techniques, to analyze how to evaluate chatbot usability. Method: We have conducted a systematic mapping study, by searching the main scientific databases. The search retrieved 170 references and 21 articles were retained as primary studies. Results: The works were categorized according to four criteria: usability techniques, usability characteristics, research methods and type of chatbots. Conclusions: Chatbot usability is still a very incipient field of research where the published studies are mainly surveys, usability tests, and rather informal experimental studies. Hence, it becomes necessary to perform more formal experiments to measure user experience, and exploit these results to provide usability-aware design guidelines.

As a result of the growth of non-developer users of OSS applications, usability has over the last ten years begun to attract the interest of the open source software (OSS) community. The OSS community has some special characteristics (such as worldwide geographical distribution of both users and developers and missing resources) which are an obstacle to the direct adoption of many usability techniques as specified in the human-computer interaction (HCI) field. The aim of this research is to adapt and evaluate the feasibility of applying four usability techniques: user profiles, personas, direct observation and post-test information to four OSS projects from the viewpoint of the development team. The applied research method was a multiple case study of the following OSS projects: Quite Universal Circuit Simulator, PSeInt, FreeMind and OpenOffice Writer. We formalized the application procedure of each of the adapted usability techniques. We found that either there were no procedures for adopting usability techniques in OSS or they were not fully systematized. Additionally, we identified the adverse conditions that are an obstacle to their adoption in OSS and propose the special adaptations required to overcome the obstacles. To avoid some of the adverse conditions, we created web artefacts (online survey, wiki and forum) that are very popular in the OSS field. It is necessary to adapt usability techniques for application in OSS projects considering their idiosyncrasy. Additionally, we found that there are obstacles (for example, number of participant users, biased information provided by developers) to the application of the techniques. Despite these obstacles, it is feasible to apply the adapted techniques in OSS projects.

The growth in the number of non-developer open source software (OSS) application users and the escalating use of these applications have both created a need for, and interest in, developing usable OSS.OSS communities are unclear about which techniques to use in each activity of the development process.The aim of our research is to adopt the visual brainstorming usability technique in the HistoryCal OSS project and determine the feasibility of adapting the technique for application.To do this, we participated as volunteers in the HistoryCal project.We used the case study research method to investigate technique application and community participation.As a result, we identified adverse conditions that were an obstacle to technique application and modified the technique to make it applicable.We can conclude from our experience that these changes were helpful for applying the technique, although it was not easy to recruit OSS users to participate in usability technique application.

Background: Interest in the integration of the agile software development process (ASDP) and user-centred design (UCD) has grown over the last decade. However, there are not many papers that study this issue holistically and uncover the current state of this integration. Aim: This study sets out to answer the following research question: What is the current state of integration between agile processes and usability? Method: We conduct a systematic mapping study (SMS). A SMS is a form of systematic literature review that aims to identify and classify the research papers published about a specified issue. Results: We retrieved a total of 161 primary studies, which we categorized according to four criteria: process, practice, team and technology integration. The largest group refers to process integration with a total of 76 primary studies. Judging by the increase in the number of publications over the last decade, the integration of human-computer interaction (HCI) activities and techniques into the ASDP is an issue of notable interest. Conclusions: There are very few papers that study the literature holistically and report the current state of usability in the ASDP. Additionally, there are no formalized proposals for adopting usability techniques in ASDP.

Context: Over the last decade there has been a growing interest in the integration of agile software development process (ASDP) and user-centred design (UCD).However, there are no papers that study which usability techniques related to requirements engineering are being adopted in the ASDP, and there are no formalized proposals for their adoption.Objective: Identify which techniques related to requirements engineering activities are being adopted in the ASDP and determine how they are being adopted.Method: We have conducted a systematic mapping study (SMS) to retrieve the literature reporting the application of usability techniques in the ASDP.We analysed these techniques using a catalogue of techniques compiled by software engineering researchers.We then determined the manner in which the techniques that are being used in the ASDP were adopted.Results: The agile community is very much interested in adopting usability techniques.The most used techniques are Personas, contextual inquiry and prototyping.Conclusions: This research offers an overview of the adoption of usability techniques related to requirements engineering in ASDPs and reports how they are being adopted.We found that some of the techniques are being adapted for adoption.

Abstract?Background: There is no globally accepted open source software development process to define how open source software is developed in practice. A process description is important for coordinating all the software development activities involving both people and technology. Aim: The research question that this study sets out to answer is: What activities do open source software process models contain? The activity groups on which it focuses are Concept Exploration, Software Requirements, Design, Maintenance and Evaluation. Method: We conduct a systematic mapping study (SMS). A SMS is a form of systematic literature review that aims to identify and classify available research papers concerning a particular issue. Results: We located a total of 29 primary studies, which we categorized by the open source software project that they examine and by activity types (Concept Exploration, Software Requirements, Design, Maintenance and Evaluation). The activities present in most of the open source software development processes were Execute Tests and Conduct Reviews, which belong to the Evaluation activities group. Maintenance is the only group that has primary studies addressing all the activities that it contains. Conclusions: The primary studies located by the SMS are the starting point for analyzing the open source software development process and proposing a process model for this community. The papers in our paper pool that describe a specific open source software project provide more regarding our research question than the papers that talk about open source software development without referring to a specific open source software project.