Context Model-Based Testing (MBT) was first introduced in 1970′s, and has the potential to improve efficiency and effectiveness of testing. However, its adoption—especially for web applications—has been hindered by the effort required to manually design MBT models, and keep them updated. Objective Based on the above challenge in a real industrial context, this study introduces an automated approach to reduce that effort by reverse engineering MBT models from clickstream data captured during users' interaction with web applications. Method We have developed and present in this paper an open-source tool, named MBTModelGenerator , which logs user interactions via a lightweight JavaScript module in the front-end, and transmits them to a REST API backend. These interactions are then transformed into directly executable MBT models in the input format of an open-source MBT tool named GraphWalker . Results The tool was evaluated on two representative open-source web applications, Spring PetClinic and a Task Manager web app, and is under evaluation in several large-scale industrial testing projects. The generated MBT models accurately reflected user navigation flows and could be executed in the GraphWalker MBT tool without any manual changes. Using the tool has significantly reduced the effort of MBT model design by more than 90%, while still allowing test engineers to inspect and refine the generated models for completeness. Conclusion Our approach facilitates lightweight adoption of MBT by automating model generation, which is the most effort intensive phase of MBT. To ensure correctness and completeness, the generated models should still be reviewed by test engineers —but that effort remains substantially lower than designing MBT models from scratch. The tool is in active industrial use and available as open-source for reuse and further development.

Abstract Industry 4.0 changes traditional manufacturing relationships from isolated optimized cells to fully integrated data and product flows across borders with its technological pillars. However, the transition to Industry 4.0 is not a straightforward journey in which organizations need assistance. A well‐known approach that can be utilized during the early phases of the transition is to assess the capability of the organization. Maturity models are frequently used to improve capability. In this systematic literature review (SLR), we analyzed 22 maturity and readiness models based on 10 criteria: year, type, focus, structure, research methodology followed during the design of models, base frameworks, tool support, community support, objectivity, and extent of usage in practice. Our SLR provides a well‐defined comparison for organizations to choose and apply available models. This SLR showed that (1) there is no widely accepted maturity/readiness model for Industry 4.0, as well as no international standard; (2) only a few models have received positive feedback from the industry, whereas most do not provide any practical usage information; and (3) the objectivity of the assessment method is controversial in most of the models. We have also identified a number of issues as open research areas for assessing readiness and maturity models for Industry 4.0.

Realizing desired properties “by construction” is a highly appealing goal in the design of safety-critical embedded systems. As verification and validation tasks in this domain are often both challenging and time-consuming, the by-construction paradigm is a promising solution to increase design productivity and reduce design errors. In the XANDAR project, partners from industry and academia develop a toolchain that will advance current development processes by employing a modelbased X-by-Construction (XbC) approach. XANDAR defines a development process, metamodel extensions, a library of safety and security patterns, and investigates many further techniques for design automation, verification, and validation. The developed toolchain will use a hypervisor-based platform, targeting future centralized, AI-capable high-performance embedded processing systems. It is co-developed and validated in both an avionics use case for situation perception and pilot assistance as well as an automotive use case for autonomous driving.

The next generation of networked embedded systems (ES) necessitates rapid prototyping and high performance while maintaining key qualities like trustworthiness and safety. However, development of safety-critical ES suffers from complex software (SW) toolchains and engineering processes. Moreover, the current trend in autonomous systems, which relies on Machine Learning (ML) and AI applications when combined with fail-operational requirements renders the Verification and Validation (V&V) of these new systems a challenging endeavor. Prime examples are Advanced Driver-Assistance Systems (ADAS) that are prone to various safety/security vulnerabilities. The XANDAR project aims at developing a mature SW toolchain (from requirements analysis to the actual code integration on target including V&V) fulfilling the needs of industry for rapid prototyping of interoperable and autonomous ES. Starting from a model-based system architecture, XANDAR will leverage automatic model synthesis and software parallelization techniques to achieve specific non-functional requirements setting the foundation for a novel (real-time, safety-, and security)-by-Construction paradigm.

Abstract Software engineering (SE) research should be relevant to industrial practice. There have been regular discussions in the SE community on this issue since the 1980’s, led by pioneers such as Robert Glass. As we recently passed the milestone of “50 years of software engineering”, some recent positive efforts have been made in this direction, e.g., establishing “industrial” tracks in several SE conferences. However, many researchers and practitioners believe that we, as a community, are still struggling with research relevance and utility. The goal of this paper is to synthesize the evidence and experience-based opinions shared on this topic so far in the SE community, and to encourage the community to further reflect and act on the research relevance. For this purpose, we have conducted a Multi-vocal Literature Review (MLR) of 54 systematically-selected sources (papers and non peer-reviewed articles). Instead of relying on and considering the individual opinions on research relevance, mentioned in each of the sources, the MLR aims to synthesize and provide the “holistic” view on the topic. The highlights of our MLR findings are as follows. The top three root causes of low relevance, discussed in the community, are: (1) Researchers having simplistic views (or wrong assumptions) about SE in practice; (2) Lack of connection with industry; and (3) Wrong identification of research problems. The top three suggestions for improving research relevance are: (1) Using appropriate research approaches such as action-research; (2) Choosing relevant (practical) research problems; and (3) Collaborating with industry. By synthesizing all the discussions on this important topic so far, this paper aims to encourage further discussions and actions in the community to increase our collective efforts to improve the research relevance. Furthermore, we raise the need for empirically-grounded and rigorous studies on the relevance problem in SE research, as carried out in other fields such as management science.

Retrieving and mining practitioners’ self-reports of their professional experience of software practice could provide valuable evidence for research. The authors are, however, unaware of any existing reviews of research conducted in this area. The authors reviewed and classified previous research, and identified insights into the challenges research confronts when retrieving and mining practitioners’ self-reports of their experience of software practice. They conducted an exploratory review to identify and classify 42 studies. They analysed a selection of those studies for insights on challenges to mining professional experience. They identified only one directly relevant study. Even then this study concerns the software professional's emotional experiences rather than the professional's reporting of behaviour and events occurring during software practice. They discussed the challenges concerning: the prevalence of professional experience; definitions, models and theories; the sparseness of data; units of discourse analysis; annotator agreement; evaluation of the performance of algorithms; and the lack of replications. No directly relevant prior research appears to have been conducted in this area. They discussed the value of reporting negative results in secondary studies. There are a range of research opportunities but also considerable challenges. They formulated a set of guiding questions for further research in this area.

Context:Knowledge level and productivity of the software engineering (SE) workforce are the subject of regular discussions among practitioners, educators, and researchers. There have been many efforts to measure and improve the knowledge gap between SE education and industrial needs. Objective:Although the existing efforts for aligning SE education and industrial needs have provided valuable insights, there is a need for analyzing the SE topics in a more “fine-grained” manner; i.e., knowing that SE university graduates should know more about requirements engineering is important, but it is more valuable to know the exact topics of requirements engineering that are most important in the industry. Method:We achieve the above objective by assessing the knowledge gaps of software engineers by designing and executing an opinion survey on levels of knowledge learned in universities versus skills needed in industry. We designed the survey by using the SE knowledge areas (KAs) from the latest version of the Software Engineering Body of Knowledge (SWEBOK v3), which classifies the SE knowledge into 12 KAs, which are themselves broken down into 67 subareas (sub-KAs) in total. Our analysis is based on (opinion) data gathered from 129 practitioners, who are mostly based in Turkey. Results:Based on our findings, we recommend that educators should include more materials on software maintenance, software configuration management, and testing in their SE curriculum. Based on the literature as well as the current trends in industry, we provide actionable suggestions to improve SE curriculum to decrease the knowledge gap.

Abstract Context Software testing is an important and costly software engineering activity in the industry. Despite the efforts of the software testing research community in the last several decades, various studies show that still many practitioners in the industry report challenges in their software testing tasks. Objective To shed light on industry's challenges in software testing, we characterize and synthesize the challenges reported by practitioners. Such concrete challenges can then be used for a variety of purposes, eg, research collaborations between industry and academia. Method Our empirical research method is opinion survey. By designing an online survey, we solicited practitioners' opinions about their challenges in different testing activities. Our dataset includes data from 72 practitioners from eight different countries. Results Our results show that test management and test automation are considered the most challenging among all testing activities by practitioners. Our results also include a set of 104 concrete challenges in software testing that may need further investigations by the research community. Conclusion We conclude that the focal points of industrial work and academic research in software testing differ. Furthermore, the paper at hand provides valuable insights concerning practitioners' “pain” points and, thus, provides researchers with a source of important research topics of high practical relevance.

Video game development is a complex endeavor, often involving complex software, large organizations, and aggressive release deadlines. Several studies have reported that periods of “crunch time” are prevalent in the video game industry, but there are few studies on the effects of time pressure. We conducted a survey with participants of the Global Game Jam (GGJ), a 48-h hackathon. Based on 198 responses, the results suggest the following: iterative brainstorming is the most popular method for conceptualizing initial requirements; continuous integration, minimum viable product, scope management, version control, and stand-up meetings are frequently applied development practices; regular communication, internal playtesting, and dynamic and proactive planning are the most common quality assurance activities; and familiarity with agile development has a weak correlation with perception of success in the GGJ. We conclude that GGJ teams rely on ad hoc approaches to the development and face-to-face communication, and recommend some complementary practices with limited overhead. Furthermore, as our findings are similar to recommendations for software startups, we posit that game jams and the startup scene share contextual similarities. Finally, we discuss the drawbacks of systemic “crunch time” and argue that game jam organizers are in a good position to problematize the phenomenon.

Research collaboration between industry and academia supports improvement and innovation in industry and helps ensure the industrial relevance of academic research. However, many researchers and practitioners in the community believe that the level of joint industry-academia collaboration (IAC) projects in Software Engineering (SE) research is relatively low, creating a barrier between research and practice. The goal of the empirical study reported in this paper is to explore and characterize the state of IAC with respect to industrial needs, developed solutions, impacts of the projects and also a set of challenges, patterns and anti-patterns identified by a recent Systematic Literature Review (SLR) study. To address the above goal, we conducted an opinion survey among researchers and practitioners with respect to their experience in IAC. Our dataset includes 101 data points from IAC projects conducted in 21 different countries. Our findings include: (1) the most popular topics of the IAC projects, in the dataset, are: software testing, quality, process, and project managements; (2) over 90% of IAC projects result in at least one publication; (3) almost 50% of IACs are initiated by industry, busting the myth that industry tends to avoid IACs; and (4) 61% of the IAC projects report having a positive impact on their industrial context, while 31% report no noticeable impacts or were “not sure”. To improve this situation, we present evidence-based recommendations to increase the success of IAC projects, such as the importance of testing pilot solutions before using them in industry. This study aims to contribute to the body of evidence in the area of IAC, and benefit researchers and practitioners. Using the data and evidence presented in this paper, they can conduct more successful IAC projects in SE by being aware of the challenges and how to overcome them, by applying best practices (patterns), and by preventing anti-patterns.

Software engineering researchers have, over the years, proposed different critical success factors (CSFs) which are believed to be critically correlated with the success of software projects. To conduct an empirical investigation into the correlation of CSFs with success of software projects, we adapt and extend in this work an existing contingency fit model of CSFs. To archive the above objective, we designed an online survey and gathered CSF-related data for 101 software projects in the Turkish software industry. Among our findings is that the top three CSFs having the most significant associations with project success were: (1) team experience with the software development methodologies, (2) team’s expertise with the task, and (3) project monitoring and controlling. A comprehensive correlation analysis between the CSFs and project success indicates positive associations between the majority of the factors and variables, however, in most of the cases at non-significant levels. By adding to the body of evidence in this field, the results of the study will be useful for a wide audience. Software managers can use the results to prioritize the improvement opportunities in their organizations w.r.t. the discussed CSFs. Software engineers might use the results to improve their skills in different dimensions, and researchers might use the results to prioritize and conduct follow-up in-depth studies on those factors.

Regression testing is needed when a software or the environment hosting that software changes. Motivated by a real-world industrial need in the context of a large financial (banking) corporation in Turkey, the authors and their colleagues developed and introduced an automated regression testing infrastructure for automated testing of one of the main mobile applications of the company. Before this project, regression testing was conducted manually which incurred a lot of costs and was by nature subjective. We report in this paper our experience in "transitioning" from manual to automated regression testing, and in developing and introducing a set of large automated test suites (more than 16 KLOC in total), using best practices in state-of-the art and -practice, and to report its observed benefits by conducting cost-benefit analysis. The project was conducted based on the principles of case-study and "action research" in which the real industrial needs drove the research. Among the best practices that we used are the followings: (1) modularity in test code, (2) creating test-specific libraries, and (3) separating test data from test logic. By serving as a success story and experience report in development and introduction of automated test suites in an industrial setting, this paper adds to the body of evidence in this area and it aims at sharing both technical (e.g., using automated test patterns) and process aspects (e.g., test process improvement) from our project with other practitioners and researchers.

To systematically collect evidence and to structure a given area in software engineering (SE), Systematic Literature Reviews (SLR) and Systematic Mapping (SM) studies have become common. Data extraction is one of the main phases (activities) when conducting an SM or an SLR, whose objective is to extract required data from the primary studies and to accurately record the information researchers need to answer the questions of the SM/SLR study. Based on experience in a large number of SM/SLR studies, we and many other researchers have found the data extraction in SLRs to be time consuming and error-prone, thus raising the real need for heuristics and guidelines for effective and efficient data extraction in these studies, especially to be learnt by junior and young researchers. As a 'guideline' paper, this paper contributes a synthesized list of challenges usually faced during SLRs' data extraction phase and the corresponding solutions (guidelines). For our synthesis, we consider two data sources: (1) the pool of 16 SLR studies in which the authors have been involved in, as well as (2) a review of challenges and guidelines in the existing literature. Our experience in utilizing the presented guidelines in the near past have helped our junior colleagues to conduct data extractions more effectively and efficiently.

Research collaboration between industry and academia supports improvement and innovation in industry and helps to ensure industrial relevance in academic research. However, many researchers and practitioners believe that the level of joint industry-academia collaboration (IAC) in software engineering (SE) research is still relatively low, compared to the amount of activity in each of the two communities. The goal of the empirical study reported in this paper is to exploratory characterize the state of IAC with respect to a set of challenges, patterns and anti-patterns identified by a recent Systematic Literature Review study. To address the above goal, we gathered the opinions of researchers and practitioners w.r.t. their experiences in IAC projects. Our dataset includes 47 opinion data points related to a large set of projects conducted in 10 different countries. We aim to contribute to the body of evidence in the area of IAC, for the benefit of researchers and practitioners in conducting future successful IAC projects in SE. As an output, the study presents a set of empirical findings and evidence-based recommendations to increase the success of IAC projects.

The level of industry-academia collaboration (IAC) in software engineering in general and in software testing in particular is quite low. Many researchers and practitioners are not collaborating with the "other side" to solve industrial problems. To shed light on the above issue and to characterize precisely what industry wants from academia in software testing, we solicited practitioners' opinions on their challenges in different testing activities and also the particularly relevant topics that they want the research community to work on. This short paper aims to draw the community's attention to the important issue of strengthening IAC with the hope of more IAC in software testing in the areas of most importance to the industry.

Background: Choosing the right software test automation tool is not trivial, and recent industrial surveys indicate lack of right tools as the main obstacle to test automation. Aim: In this paper, we study how practitioners tackle the problem of choosing the right test automation tool. Method: We synthesize the "voice" of the practitioners with a grey literature review originating from 53 different companies. The industry experts behind the sources had roles such as "Software Test Automation Architect", and "Principal Software Engineer". Results: Common consensus about the important criteria exists but those are not applied systematically. We summarize the scattered steps from individual sources by presenting a comprehensive process for tool evaluation with 12 steps and a total of 14 different criteria for choosing the right tool. Conclusions: The practitioners tend to have general interest in and be influenced by related grey literature as about 78% of our sources had at least 20 backlinks (a reference comparable to a citation) while the variation was between 3 and 759 backlinks. There is a plethora of different software testing tools available, yet the practitioners seem to prefer and adopt the widely known and used tools. The study helps to identify the potential pitfalls of existing processes and opportunities for comprehensive tool evaluation.

To cope with growing complexity of embedded software, modeling has become popular. The usage of models in embedded software industry and the relevant practices usually vary since the purposes of diagram development and usage differ. Since a large variety of software modeling practices used in embedded software industry, it is important to understand its state-of-the-practice and its usage degree while investigating the relations between its attributes (e.g., modeling rigor, purpose, code correspondence, stakeholder, medium used while modeling, etc). To achieve this, we have designed and conducted a survey in our earlier work. In this paper, we present a conceptual model of development and usage for software modeling that is based on the findings of this survey and incorporates expert opinions. The conceptual model, which characterizes the attributes of a diagram development and usage, will help to express the meaning of terms used by domain experts to discuss the problems and find the relationships between these attributes.

Software and system development faces numerous challenges of rapidly changing markets. To address such challenges, companies and projects design and adopt specific development approaches by combining well-structured comprehensive methods and flexible agile practices. Yet, the number of methods and practices is large, and available studies argue that the actual process composition is carried out in a fairly ad-hoc manner. The present paper reports on a survey on hybrid software development approaches. We study which approaches are used in practice, how different approaches are combined, and what contextual factors influence the use and combination of hybrid software development approaches. Our results from 69 study participants show a variety of development approaches used and combined in practice. We show that most combinations follow a pattern in which a traditional process model serves as framework in which several fine-grained (agile) practices are plugged in. We further show that hybrid software development approaches are independent from the company size and external triggers. We conclude that such approaches are the results of a natural process evolution, which is mainly driven by experience, learning, and pragmatism.

In the context of an industry-academia collaboration in the scope of test process improvement, there was a need to assess the usage, usefulness and quality of defect reports. The objective was to assess the extent to which each field of a defect report (e.g., product name, version number) is read/used by developers, ensuring that efforts spent into writing those fields are worthwhile, and also to pinpoint improvement areas in defect reports to increase developers' effectiveness in fixing defects. To address those needs, we conducted a questionnaire survey which gathered input from 38 software developers in the context of a large Turkish software and systems company providing global solutions in the areas of defense and IT. Analysis of survey results helped us to determine the usage and usefulness of various defect report fields and also to pinpoint the necessary improvement areas in defect reports, out of which an improvement effort has already started.

Systematic Literature Reviews (SLR) may not provide insight into the "state of the practice" in SE, as they do not typically include the "grey" (non-published) literature. A Multivocal Literature Review (MLR) is a form of a SLR which includes grey literature in addition to the published (formal) literature. Only a few MLRs have been published in SE so far. We aim at raising the awareness for MLRs in SE by addressing two research questions (RQs): (1) What types of knowledge are missed when a SLR does not include the multivocal literature in a SE field? and (2) What do we, as a community, gain when we include the multivocal literature and conduct MLRs? To answer these RQs, we sample a few example SLRs and MLRs and identify the missing and the gained knowledge due to excluding or including the grey literature. We find that (1) grey literature can give substantial benefits in certain areas of SE, and that (2) the inclusion of grey literature brings forward certain challenges as evidence in them is often experience and opinion based. Given these conflicting viewpoints, the authors are planning to prepare systematic guidelines for performing MLRs in SE.

The global software industry and the Software Engineering (SE) academia are two large communities. However, unfortunately, the level of joint industry-academia collaborations (IAC) in SE is still relatively very low, compared to the amount of activity in each of the two communities. Selecting the right topic for a new IAC has been reported to be challenging and often a deal-maker or-breaker for the start of IACs. Motivated by the above need, the goal of this paper is to propose experience-based guidelines from our 10+ software testing IACs in the past several years in Canada and Turkey to effectively and efficiently select right topics for IACs in software testing (also easily generalizable to other areas of SE), for the benefit of SE researchers and practitioners in starting new IACs. The experience and evidence supporting the guidelines in this paper are drawn from the authors' past projects and also seven on-going software-testing projects in the context of a large Turkish software and systems company. The topic-selection process has involved interaction with company representatives in the form of both multiple group discussions and separate face-to-face meetings while utilizing grounded-theory to find (converge to) topics which would be 'interesting' and useful from both industrial and academic perspectives. To increase the success of our topic selection process, we also utilized two other sources of information from the literature: (1) a set of four fitness criteria for topic selection in industry experiments, and (2) challenges and best practices for IAC, specific to project inception, as synthesized in a recent systematic literature review. We believe the results of this paper would be helpful for other researchers and practitioners not only in software testing but also in software engineering in general in increasing their chances of success in project inception and topic selection phase.

Abstract Software test processes are complex and costly. To reduce testing effort without compromising effectiveness and product quality, automation of test activities has been adopted as a popular approach in software industry. However, because test automation usually requires substantial upfront investments, automation is not always more cost‐effective than manual testing. To support decision‐makers in finding the optimal degree of test automation in a given project, we recently proposed a process simulation model using the System Dynamics modeling technique and used the simulation model in the context of a case study with a software company in Calgary, Canada. With the help of the simulation model, we were able to evaluate the performance of test processes with varying degrees of automation of test activities and help testers choose the most optimal cases. The goal of the earlier study was to investigate how the simulation model can help decision‐makers decide whether and to what degree the company should automate their test processes. In this article, we present further details of the System Dynamics model, its usage scenarios and examples of simulation experiments independent from a specific company context. Copyright © 2015 John Wiley & Sons, Ltd.

Abstract In this paper, the modeling of developers’ assignment to bugs (DAB) is studied. The problem is modeled both as a single objective (minimize bug fix time) and as a bi‐objective (minimize bug fix time and cost) combinatorial optimization problem. Two models of developer assignment are considered where in the first model a single developer is assigned per bug (single developer model), while in the second model a single developer is assigned for each competency area of a bug (individual competency model). The latter model is proposed in this paper. For the single developer model, GA@DAB, an existing genetic algorithm‐based approach, is extended to support precedence among bugs. For the individual competency model of DAB, one genetic algorithm‐based approach (Competence@DAB) and one nondominated sorting genetic algorithm II‐based approach (CompetenceMulti2@DAB ) are proposed to generate solutions minimizing time and minimizing both time and cost, respectively. The performance of the proposed approaches was evaluated for 2040 bugs of 19 open‐source milestone projects from the Eclipse platform. Our results and analysis show that the proposed individual competency model is far better than the single developer model, with average bug fix time reduction of 39.7% across all projects. Copyright © 2016 John Wiley & Sons, Ltd.

Although citation counts are often considered a measure of academic impact, they are criticized for failing to evaluate impact as intended. In this paper we propose that software engineering citations may be classified according to how the citation is used by the author of the citing paper, and that through this classification of citation behaviour it is possible to achieve a more refined understanding of the cited paper's impact. Our objective in this work is to conduct an initial evaluation using the citation behaviour taxonomy proposed by Bornmann and Daniel. We independently classified citations to ten highly-cited papers published at the International Symposium on Empirical Software Engineering and Measurement (ESEM). The degree to which classifications were consistent between researchers was analyzed in order to assess the clarity of Bornmann and Daniel's taxonomy. We found poor to fair agreement between researchers even though the taxonomy was perceived as relatively easy to apply for the majority of citations. We were nevertheless able to identify clear differences in the profile of citation behaviors between the cited papers. We conclude that an improved taxonomy is required if classification is to be reliable, and that a degree of automation would improve reliability as well as reduce the time taken to make a classification.

Context: The pool of papers published in ESEM. Objective: To utilize citation analysis and automated topic analysis to characterize the SE research literature over the years focusing on those papers published in ESEM. Method: We collected data from Scopus database consisting of 513 ESEM papers. For thematic analysis, we used topic modeling to automatically generate the most probable topic distributions given the data. Results: Nearly 42% of the papers have not been cited at all but the effect seems to wear off as time passes. Using text mining of article titles and abstracts, we found that currently the most popular research topics in the ESEM community are: systematic reviews, testing, defects, cost estimation, and team work. Conclusions: While this study analyzes the paper pool of the ESEM symposium, the approach can easily be applied to any other sub-set of SE papers to conduct large scale studies. Due to large volumes of research in SE, we suggest using the automated analysis of bibliometrics as we have done in this paper.

Software test processes are complex and costly. To reduce testing effort without compromising effectiveness and product quality, automation of test activities has been adopted as a popular approach in software industry. However, since test automation usually requires substantial upfront investments, automation is not always more cost-effective than manual testing. To support decision-makers in finding the optimal degree of test automation in a given project, we propose in this paper a simulation model using the System Dynamics (SD) modeling technique. With the help of the simulation model, we can evaluate the performance of test processes with varying degrees of automation of test activities and help testers choose the most optimal cases. As the case study, we describe how we used our simulation model in the context of an Action Research (AR) study conducted in collaboration with a software company in Calgary, Canada. The goal of the study was to investigate how the simulation model can help decision-makers decide whether and to what degree the company should automate their test processes. As a first step, we compared the performances of the current fully manual testing with several cases of partly automated testing as anticipated for implementation in the partner company. The development of the simulation model as well as the analysis of simulation results helped the partner company to get a deeper understanding of the strengths and weaknesses of their current test process and supported decision-makers in the cost effective planning of improvements of selected test activities.

Context: Software documentation is an integral part of any software development process. However, software practitioners are often concerned about the lack of usage and quality of documentation in practice. Unfortunately, in many projects, practitioners find that software documentation artifacts are outdated, incomplete and sometimes not beneficial. Objective: Motivated by the needs of NovAtel Inc. (NovAtel), a world-leading company of GPS software systems, we propose in this paper an approach to analyze the usage and quality of software documentation in development and maintenance phases. Method: The approach incorporates inputs from automated analysis (e.g., mining of project's data) and also experts' opinion extracted from survey-based questionnaire. The approach has been designed based on the "action-research" approach and in close collaboration between industry and academia. Results: To evaluate the feasibility and usefulness of the proposed approach, we have applied it in an industrial setting and results are presented in this paper. One of the results is that, in the context of our case-study, usage of documentation for an implementation purpose is higher than the usage for maintenance purposes. Conclusion: It is concluded that the usage of documentation differs for various purposes and it depends on the type of the information needs as well as the task to be completed (e.g. development and maintenance). In addition, we identify the most important and relevant quality attributes which are critical to improving documentation quality.

This paper reports the results and some challenges we discovered during the design and execution of a pilot experiment with 21 bachelor students aimed at investigating the effect of documentation accuracy during software maintenance and evolution activities. As documentation we considered: a high level system functionality description and UML documents. Preliminary results indicate a benefit of +15% in terms of efficiency (computed as number of correct tasks per minute) when a more accurate documentation is used. The discovered challenging aspects to carefully consider in future executions of the experiment are as follows: selecting "the right" documentation artefacts, maintenance tasks and documentation versions, verifying that the subjects really used the documentation during the experiment and measuring documentation-code alignment.

Scientific and engineering research is heavily dependent on effective development and use of software artifacts. Many of these artifacts are produced by the scientists themselves, rather than by trained software engineers. To address the challenges in this area, a research community often referred to as "Development of Scientific Software" has emerged in the last few decades. As this research area has matured, there has been a sharp increase in the number of papers and results made available, and it has thus become important to summarize and provide an overview about those studies. Through a systematic mapping and bibliometrics study, we have reviewed 130 papers in this area. We present the results of our study in this paper. Also we have made the mapping data available on an online repository which is planned to be updated on a regular basis. The results of our study seem to suggest that many software engineering techniques and activities are being used in the development of scientific software. However, there is still a need for further exploration of the usefulness of specific software engineering techniques (e.g., regarding software maintenance, evolution, refactoring, re(v)-engineering, process and project management) in the scientific context. It is hoped that this article will help (new) researchers get an overview of the research space and help them to understand the trends in the area.

Bibliometric rankings are quite common in the field of software engineering. For example, there are a series of ranking repeated every year which identify the top researchers and institutions at the international level in the field. There are also other studies to determine the most cited articles in software engineering journals, the most popular research topics in this area, or identify the top researchers and institutions in regional levels. However, there exists no existing bibliometric quantitative analysis of publications in the area of software engineering (SE), including relative and absolute growth in the number of all SE publications as well as an analysis among countries. This is the main goal and motivation of this article. Besides, this study intends to provide an overall quantitative trend of the software engineering papers, and compare that trend to research output in other areas of science. The bibliometric study reported in this paper is motivated by the fact that understanding the amount of geographical research contributions to the field of software engineering can help identify different countries’ level of commitment to support research activities in this area over years. We analyze how the contribution levels of top-ranked countries have changed over the years and how SE compares to other disciplines of engineering and science. Among the most interesting findings of this study are: (1) Over 40 years, in total about 60% of the SE literature has been contributed by only 7% of all countries, (2) the SE research output of different countries does not necessarily correlate with their GDPs, (3) the share of contributions to the SE discipline by the American researchers has declined from 71.43% (in 1980) to 14.90% (in 2008), and (4) China is the country with the biggest share growth in the number of publications (from 0.82% of the entire SE publications in 1991 to 13.82% in 2009).

We report in this case-study paper our experience and success story with a practical approach and tool for unit regression testing of a SCADA (Supervisory Control and Data Acquisition) software. The tool uses a black-box specification of the units under test to automatically generate NUnit test code. We then improved the test suite by white-box and mutation testing. The approach and tool were developed in an action-research project to test a commercial large-scale SCADA system called Rocket.

Software development takes place in a dynamic context of frequently changing technologies, and limited resources. Development teams worldwide are under ever-increasing pressure to deliver trustworthy software products more quickly and with higher levels of quality. At the same time, global competition is forcing software development organizations to cut costs by rationalizing processes, outsourcing part or all of their activities, reusing existing software in new or modified applications, and evolving existing systems to meet new needs, whereas still minimizing the risk of projects failing to deliver. To address these difficulties, new or modified processes are emerging including lean and agile methods, plan-based product line development, and increased integration with systems engineering processes. A common understanding is that software trustworthiness is a synthetic evaluation for software quality, and is characterized by the extent that software meet its requirements, include cost, and schedule requirements. Many software process researchers have worked in this area to solve the existing challenges. From 1984 to 1996, the International Software Process Workshops attracted many researchers and practitioners. Then came the International Conference on the Software Process, (ICSP, from 1991 until 1996), the International Workshop on Software Process Simulation and Modeling (ProSim, from 1998 until 2006), and the Software Process Workshop (SPW, in 2005 and 2006). ProSim and SPW were held together in 2006 and merged in 2007 to form the new International Conference on Software Process. In 2009, the third ICSP conference was held in Vancouver, Canada, on May 16–17, 2009, and was co-located with the ICSE 2009 (the International Conference on Software Engineering). The theme of ICSP 2009 was ‘Processes to Develop Trustworthy Software’. There were 96 submissions to the conference that were received from 26 different countries and regions including Australia, Austria, Brazil, Canada, Chile, China, Colombia, Finland, France, Germany, Greece, India, Ireland, Italy, Japan, Mexico, New Zealand, Pakistan, Russia, Singapore, Spain, Sweden, Turkey, United Kingdom, United States of America, and Serbia. After a rigorous review process, 33 papers were accepted for the proceedings of the ICSP 2009. After the conference, the organization committee invited 5 of the 33 papers to submit the extended versions of their papers to a special issue of the International Journal of Software Process: Improvement and Practice on the topic of ‘Processes to Develop Trustworthy Software’. The call-for-paper was broadcasted in the SE-World mailing list as well and submission was open to all the community at large. From the pool of submissions to the special issue, seven papers were accepted and went through several revision stages. The remainder of this guest editors' introduction provides an overview to the articles in this special issue. In their article ‘Hybrid Modeling and Simulation for Trustworthy Software Process Management: A Stakeholder-oriented Approach’, Xu Bai, Liguo Huang, He Zhang, and Supannika Koolmanojwong present a software process modeling method that integrates the stakeholder-oriented approach and hybrid simulation techniques. The stakeholder-oriented approach addresses the various perspectives of stakeholders during process modeling and simulation. The hybrid process simulation combines microlevel discrete process models with the macrolevel continuous process models to capture process dynamics. The article ‘Towards Trustworthy Software Process Models: An Exploratory Study on Transformable Process Modeling’ by He Zhang, Barbara Kitchenham, and Ross Jeffery presents a trustworthy process modeling method that integrates trustworthy process management framework and transformable process modeling. It undertakes an exploratory study on transforming a quantitative continuous (System Dynamics) process model into a qualitative form for simulation. The results present the different capabilities and performance between these two modeling paradigms, as well as the possible benefits and interesting perspectives. Reda Bendraou, Jean-Marc Jezéquél, and Franck Fleurey present a framework that combines Aspect and Model-Driven Engineering approach to support process modeling, simulation and execution, in their article ‘Achieving Process Modeling and Execution through the Combination of Aspect and Model-Driven Engineering Approaches’. This framework is based upon UML4SPM, a UML2.0-based language for software process, modeling, is an executable meta programming language, which precisely defines the runtime behaviors of each element of meta-model, and then formalized the execution semantics at meta-model level. Qi Li, Ye Yang, Mingshu Li, Qing Wang, Barry Boehm, and Chenyong Hu in their article ‘Improving Software Testing Process: Feature Prioritization to Make Winners of Success-Critical Stakeholders’ propose a multi-objective feature prioritization strategy for testing planning and controlling, which aligns the internal testing process with value objectives coming from customers and markets. They also present a case study to show how a real-life business project uses the method to plan, manage, and improve its testing process, and make customer satisfaction increasing. Dan Houston, Douglas Buettner, and Myron Hecht describe a dynamic COQUALMO in their article ‘Defectivity Profiling with Dynamic COQUALMO: An Explication and Product Quality Retrospective’, which offers quantitative support though a defectivity profile. The profile can be combined with COQUALMO and calibrated over time, and used for defect estimation. The authors also demonstrate the model's use in a project retrospective. Mubarak Mohammad and Vangalur Alagar think that for trustworthy software development, the process should maintain a chain of evidence to confirm all the trustworthy properties are preserved in every related activity. They introduce a new process for a rigorous component-centered development of trustworthy systems in their article ‘A Component-Based Development Process for Trustworthy Systems’. The proposed process includes five parallel interrelated tracks including component development, component assessment, component reuse, and component-based system development and prescribes specific activities. Software quality is dependent considerably on the quality of the development processes. But unfortunately, many software providers have to deliver trustworthy software in a changing environment. In his article ‘Dynamic Capability for Trustworthy Software Development’, Lan Cao discusses how software development process achieve capability to pursue the conflicting with process stable predictability and dynamic flexibility. A case study of large military software development project has been studied in the article to examine the presented method. Last but not least, we would like to thank all the authors and reviewers of this special issue. We also would like to thank Darren Dalcher, David Raffo and Dietmar Pfahl for their support and help during the reviewing ‘process’ of this special issue. July 2010 Vahid Garousi (University of Calgary, Calgary, Canada) and Qing Wang (Chinese Academy of Sciences, Beijing, China) Guest Editors, Special Issue on ‘Processes to Develop Trustworthy Software’ Including the invited papers from the International Conference on the Software Process (ICSP) 2009

In order to effectively teach software engineering students how to solve real-world problems, if possible, students should have the chance of working with and testing “real-world” industrial software systems during their courses. In a previous article, we presented a comprehensive software-testing lab exercise repository in which real software systems and test tools were incorporated to give students the chance of learning industry-standard tools (such as JUnit and IBM Rational Functional Tester). As the next step in our on-going efforts to improve the learning experience of students in testing courses, we have incorporated “real-world” industrial testing projects in a graduate-level software testing course in the past three years (2008-2010). The experience and the outcomes of these industrial-caliber projects have been very satisfying to the stakeholders. We report in this article some details about those projects and also discuss the opportunities, challenges, and lessons learned in those projects.

Development, testing and maintenance of software for embedded systems is a complex task. Analysis of the traceability between different software artifacts (e.g., source code, test code and requirements) is an enabling capability for better development, testing and maintenance of software systems. However, there is a general lack of tool support for automating traceability analysis for embedded systems. We demonstrate in this paper how to extend an existing unit test and test coverage framework to produce a hard-ware assisted tool framework capable of automatically deriving and visualizing traceability links between source code and test code artifacts. To demonstrate the applicability and usefulness of the framework, we report the application of the framework on realistic embedded software for vehicle gear transmission control built and deployed on the Analog Devices Blackfin ® ADSP-5XX family of DSP processors.

Abstract In a previous article, a stress testing methodology was reported to detect network traffic‐related Real‐Time (RT) faults in distributed RT systems based on the design UML model of a System Under Test (SUT). The stress methodology, referred to as Test LOcation‐driven Stress Testing (TLOST), aimed at increasing the chances of RT failures (violations in RT constraints) associated with a given stress test location (an network or a node under test). As demonstrated and experimented in this article, although TLOST is useful in stress testing different test locations (nodes and network, it does not guarantee to target (test) all RT constraints in an SUT. This is because the durations of message sequences bounded by some RT constraints might never be exercised (covered) by TLOST. A complementary stress test methodology is proposed in this article, which guarantees to target (cover) all RT constraints in an SUT and detect their potential RT faults (if any). Using a case study, this article shows that the new complementary methodology is capable of targeting the RT faults not detected by the previous test methodology. Copyright © 2009 John Wiley & Sons, Ltd.

In order to effectively teach software testing students how to test real-world software, the software tools, exercises, and lab projects chosen by testing educators should be practical and realistic. However, there are not many publicly-available realistic testing courseware for software testing educators to adapt and customize. Even for the existing testing lab exercises developed and/or used by the educators, there are various drawbacks, e.g.: (1) They are not usually kept up-to-date with the most recent testing tools and technologies, e.g., JUnit, (2) They are not built based on realistic/real-world Systems Under Test (SUTs), but rather use ¿toy¿ examples (SUTs). The above needs were the main motives for the author and his team at the University of Calgary to modernize the lab exercises of an undergraduate software testing course. This paper presents the designed lab courseware, and the experiences learned from using the courseware in the University of Calgary. It is hoped (and expected) that other software testing educators start to use this laboratory courseware and find it useful for their instruction and training needs.

This article is a brief overview of the current state of the software testing education in the Canadian and American universities. In doing so, the authors hope to pinpoint the strengths, and areas for improvement and to encourage a systematic software testing curriculum development. We also present some recommendations for new software testing educators.

Software organizations have typically de-emphasized the importance of software testing. In an earlier study in 2004, our colleagues reported the results of an Alberta-wide regional survey of software testing techniques in practice. Five years after that first study, the authors felt it is time to replicate the survey and analyze what has changed and what not from 2004 to 2009. This study was conducted during the summer of 2009 by surveying software organizations in the Canadian province of Alberta. The survey results reveal important and interesting findings about software testing practices in Alberta, and point out what has changed from 2004 to 2009 and what not. Note that although our study is conducted in the province of Alberta, we have compared the results to few international similar studies, such as the ones conducted in the US, Turkey, Hong Kong and Australia, The study should thus be of interest to all testing professionals world-wide. Among the findings are the followings: (1) almost all companies perform unit and system testing with a slight increase since 2004, (2) automation of unit, integration and systems tests has increased sharply since 2004, (3) more organization are using observations and expert opinion to conduct usability testing, (4) the choices of test-case generation mechanisms have not changed much from 2004, (5) JUnit and IBM Rational tools are the most widely used test tools, (6) Alberta companies still face approximately the same defect-related economic issues as do companies in other jurisdictions, (7) Alberta software firms have improved their test automation capability since 2004, but there is still some room for improvement, and (8) compared to 2004, more companies are spending more effort on pre-release testing.

Genetic algorithms (GAs) have been applied previously to UML-driven stress test requirements generation with the aim of increasing chances of discovering faults relating to network traffic in distributed real-time systems. However, since evolutionary algorithms are heuristic, their performance can vary across multiple executions, which may affect robustness and scalability. To address this, we present the design and technical detail of a UML-driven, GA-based stress test requirements generation tool, together with its empirical analysis. The main goal is to analyze and improve the applicability, efficiency, and effectiveness and also to validate the design choices of the GA used in the tool. Findings of the empirical evaluation reveal that the tool is robust and reasonably scalable when it is executed on large-scale experimental design models. The study also reveals the main bottlenecks and limitations of the tools, e.g., there is a performance bottleneck when the system under test has a large number of sequence diagrams which could be triggered independently from each other. In addition, issues specific to stress testing, e.g., the impact of variations in task arrival pattern types, reveal that the tool generally generates effective test requirements, although the features of those test requirements might be different in different runs (e.g., different stress times from the test start time might be chosen). While the use of evolutionary algorithms to generate software test cases has been widely reported, the extent, depth, and detail of the empirical findings presented in this paper are novel and suggest that the proposed approach is effective and efficient in generating stress test requirements. It is hoped that the findings of this empirical study will help other SBSE researchers with the empirical evaluation of their own techniques and tools.

Measurement and detection of redundancy in test suites attempt to achieve test minimization which in turn can help reduce test maintenance costs, and to also ensure the integrity of test cases. Test suite reduction based on coverage information has been discussed in many previous works. However, the applications of such techniques on real test suites and realistic measurements of redundancy have not yet been experimented thoroughly. To address such a need, we formulate in this paper two experimental metrics for coverage-based measurement of test redundancy in the context of JUnit test suites. We then evaluate the approach by measuring the redundancy of four real Java projects. The automated measures are compared with manual redundancy decisions (performed through an inspection by a tester). The results and lessons learned are interesting and somewhat surprising in that besides they show usefulness of coverage information, they present a set of shortcomings (in terms of precision) for the simplistic coverage-based redundancy measurement approach as discussed in the literature. The root-cause analysis of our observations identify several key lessons learned which should help the testing researchers and practitioners in devising better techniques for more precise measurement of test redundancy.

There is an increasing demand for the multi-agent systems (MAS) in the automation of manufacturing systems. However, similar to other distributed systems, autonomous agents' interaction in the automated manufacturing systems (AMS) can potentially lead to runtime behavioral failures including deadlock. Deadlocks can cause major financial consequences by negatively affecting the production cost and time. Therefore, a multi-agent manufacturing system should be monitored against the unwanted emergent behaviors such as deadlocks. In this paper, we propose a monitoring technique for deadlock detection in multi-agent manufacturing system based on the MAS design models. In this technique, the MAS is instrumented with a dedicated communication protocol to use the potential deadlock information derived from the design models to propagate deadlock detection query messages. The technique is able to reduce the message communication overhead among the agents by limiting the number of agents that the deadlock detection query messages should be initiated to.

Evolutionary testing denotes the use of evolutionary algorithms, e.g., Genetic Algorithms (GAs), to support various test automation tasks. Since evolutionary algorithms are heuristics, their performance and output efficiency can vary across multiple runs. Therefore, there is a strong need to empirically investigate the capacity of evolutionary test techniques to achieve the desired objectives (e.g., generate stress test cases) and their scalability in terms of the complexity of the System Under Test (SUT), the inputs, and the control parameters of the search algorithms. In a previous work, we presented a GA-based UML-driven, stress test technique aimed at increasing chances of discovering faults related to network traffic in distributed real-time software. This paper reports a carefully-designed empirical study which was conducted to analyze and improve the applicability, efficiency and effectiveness of the above GA-based stress test technique. Detailed stages and objectives of the empirical analysis are reported. The findings of the study are furthermore used to better calibrate the parameters of the GA-based stress test technique.

In a previous work, we reported and experimented with a stress testing methodology to detect network traffic- related real-time (RT) faults in distributed real-time systems (DRTSs) based on the design UML models. The stress methodology, referred to as time-shifting stress test methodology (TSSTM), aimed at increasing chances of discovering RT faults originating from network traffic overloads in DRTSs. The TSSTM uses the UML 2.0 model of a system under test (SUT), augmented with timing information, and is based on an analysis of the control flow in UML sequence diagrams. In order to devise deterministic test requirements (from time point of view) that yield the maximum stress test scenario in terms of network traffic in a SUT, the TSSTM methodology requires that the timing information of messages in sequence diagrams is available and as precise as possible. In reality, however, the timing information of messages is not always available and precise. As we demonstrate using a case study in this work, the effectiveness of the stress test cases generated by TSSTM is very sensitive to such time uncertainty. In other words, TSSTM might generate imprecise and not necessarily maximum stressing test cases in the presence of such time uncertainty and, thus, it might not be very effective in revealing RT faults. To address the above limitation of TSSTM, we present in this article a modified testing methodology which can be used to stress test systems when the timing information of messages is imprecise or unpredictable. The stress test results of applying the new test methodology to a prototype DRTS indicate that, in the presence of uncertainty in timing information of messages, the new methodology is more effective in detecting RT faults when compared to our previous methodology (i.e., TSSTM) and also test cases based on an operational profile.

The increasing demand for multi-agent systems (MAS) in the software industry has led to development of several agent oriented software engineering (AOSE) methodologies. The autonomous agents' interaction in a dynamic software environment can potentially lead to runtime behavioral failures such as deadlock. Therefore, the MAS environment should be tested and monitored against the unwanted emergent behaviors. The AOSE methodologies usually do not cover monitoring and testing. On the other hand model-based software development practices such as the Unified Modeling Languages (UML) are commonly used in practice and are equipped with a rich set of model based testing and monitoring tools. In this paper, we propose a conversion tool to help MAS engineers use UML-based monitoring and testing tools to test and monitor MAS design and analysis artifacts created by multi-agent software engineering (MaSE) as one of the most powerful and famous AOSE methodologies.

A stress test methodology aimed at increasing chances of discovering faults related to network traffic in distributed systems is presented. The technique uses the UML 2.0 model of the distributed system under test, augmented with timing information, and is based on an analysis of the control flow in sequence diagrams. It yields stress test requirements that are made of specific control flow paths along with time values indicating when to trigger them. Different variants of our stress testing technique already exist (they stress different aspects of a distributed system) and we focus here on one variant that is designed to identify and to stress test the system at the instant when data traffic on a network is maximal. Using a real-world distributed system specification, we design and implement a prototype distributed system and describe, for that particular system, how the stress test cases are derived and executed using our methodology. The stress test results indicate that the technique is significantly more effective at detecting network traffic-related faults when compared to test cases based on an operational profile.