While software-developing organizations increasingly write test cases and continuously run them to find defects and ensure quality, the associated maintenance and execution costs increase over time. According to practitioners in the software industry, more test cases is not the solution, which calls for other methods and tools that can complement traditional test cases. Based on a cross-company workshop with 30 participants, three focus groups with 19 participants, and a validation workshop with 30 participants, this paper presents insights from practitioners at six large-scale companies, describing experiences from real industrial settings with regards to testing of software units and components. As every tool is also a cost for the organization, this paper presents a structured and holistic approach based on grounded theory, where methods and tools are evaluated and selected from four categories, representing test activities with different purposes: Optimize feedback loops, Improve test efficiency, Improve test suite effectiveness, and Technical debt management. The participants in the study also described how methods and tools used within their companies were often not well described in the company’s ways-of-working, which caused frustration and confusion. Feedback from the continuous integration and delivery pipeline must be understandable (describing what the problem is), actionable (describing what the developer needs to do), and explained (describing and motivating why this is important). The findings from this study provide actionable strategies for how testing of units and components can be conducted in a more structured way, which could help companies cut costs, save time and identify problems earlier.

Testing of a large-scale and complex software system requires many types of knowledge, skills and personality traits. Contrasting the idea of a perfect all-round tester, this paper presents the Testing Hopscotch model with six complementary profiles, and the key characteristics considered as most relevant for each profile. The model is based on 60 interviews with engineers from three large-scale companies in different industry segments. The Testing Hopscotch model was well received by three focus groups including 22 participants, which further strengthens the validity of the model.

Abstract The characteristics of the test environment are of vital importance to its ability to support the organizations testing objectives. This paper seeks to address the need for a structured and reliable approach, which can be used by companies and other organizations to optimize their test environments in each individual case. The reported study included a series of interviews with 30 individuals, a series of focus groups with in total 31 individuals and a cross‐company workshop with 30 participants from five large‐scale companies, operating in different industry segments. The study resulted in a list of success factors, including not only characteristics and capabilities existing within a test environment (intrinsic success factors) but also properties not inherent to the test environment, but still vital for a successfully implemented test environment (extrinsic success factors). This distinction is important, as the root causes differ and as addressing them requires distinct approaches—not only of technology but also of organization, communication and collaboration. We find that successful implementations of test environments for large‐scale software systems depend primarily on how they support the company's business strategy, test organization and product testability (extrinsic success factors). Based on this, test environments can then be optimized to improve test environment capabilities, usability and stability (intrinsic success factors). The list of intrinsic and extrinsic success factors was well received by all five companies included in the study, supporting that the intrinsic and extrinsic success factors for test environments can be applied to a large segment of the software industry.

Based on multiple series of interviews and workshops, this paper presents the MaLET model – a representation of the typical evolution path for companies who successfully adopt exploratory testing. The model provides a step-by-step approach to systematically improve exploratory testing over time, and shows how and why capabilities may regress. The MaLET model was validated through a series of interviews with 20 interviewees from eight case study companies in separate industry segments. The interviews also revealed examples on improvement initiatives that had failed in the companies, showing that improvement initiatives tend to fail if they are not planned in an order corresponding to the maturity levels in the model. The MaLET model was well received by the interviewees during the validation, describing the model as sound, relevant and useful in practice.

Continuous integration and delivery are well-established paradigms in the software development community. With these continuous practices come many challenges; while some of these challenges are immediate and well documented in literature, others may be revealed only after sustained application of these practices in large-scale and complex contexts. Based on researcher observations and interviews with 22 senior professionals from four companies, all with significant but varied experiences of continuous practices, we present a set of underreported challenges with continuous practices observed in multiple industry settings. Through a grounded theory approach we construct the Tapco model — Test Automation Progression in Continuous Practices — identifying two distinct ways in which companies progress towards continuous delivery, onto which the studied cases are mapped. This model is then validated by presenting it to the four studied companies, operating in disparate industry segments, and to three additional industry cases, letting them evaluate its relevancy, accuracy and novelty. We find that the model provides industry professionals with essential guidance on how to avoid common pitfalls, as well as an understanding of their causes and possible remediation.

This paper identifies three improvement areas related to system design and architecture, where an organization can change to better support continuous integration and continuous delivery: “The product's architecture”, “Ways to work with system design and architecture”, and “The role of the architect”. The three improvement areas are based on a literature review, two series of interviews and a cross-company workshop with three case-study companies. Furthermore, the paper proposes three actionable strategies corresponding to the three identified improvement areas: “Systems with a modular and loosely coupled architecture”, “A balanced approach where system design and architecture is focused on the system's most important characteristics”, and “Architects shifting perspective from control to facilitation”.

Abstract Based on 25 interviews with participants from four case study companies that develop large‐scale software embedded systems, this paper presents the Test Activity Stakeholders (TAS) model. The TAS model shows how the continuous integration and delivery pipeline can be designed to include test activities that support four stakeholder interests: “Check changes,” “Secure stability,” “Measure progress,” and “Verify compliance.” The model is developed to show how each of the stakeholder interests are best supported by unit/component tests or system tests, by automated testing or manual testing and by tests executed in simulated environments or on real hardware. The TAS model may serve as a starting point for companies when evaluating and designing their continuous integration and delivery pipeline. The validation of the TAS model included twelve individuals from three case study companies. The validation showed that the TAS model is actionable and useful in practice, enabling the identification of stakeholders and showing where improvement efforts should be focused in order to support all stakeholder interests in the continuous integration and delivery pipeline.

Keeping the build fast is often stated as an important prerequisite for continuous integration, and is also widely discussed in literature. But which importance does the capacity of the build system actually have in relation to developer behavior? Based on metrics and interview results from a large-scale industry project, we present the factors that according to the developers themselves affect how often they deliver software to the mainline. We show that the developer will deliver less frequently if the delivery processes is time-consuming, if it's too complicated to deliver or if there is no evident value in delivering often to the mainline. Behind these three main themes, we also present a range of sub-categories such as architecture, test activities and administration. The build system capacity is one of several factors which, if not considered, could result in undesired continuous integration behaviors - but other factors should be seen as at least as important.

The EMFIS model allows companies to explicate a representation of the organization's current situation regarding continuous integration impediments, and visualizes what the organization must focus on in order to enable more frequent integration of software. The model is used to perform an assessment of twelve factors, where the ratings from participants representing the developers are summarized separately from ratings from participants representing the enablers (responsible for processes, development tools, test environments etc.). The EMFIS model is based on semi-structured interviews with 20 developers from two companies which develop large-scale software systems, and a literature review that included 74 research papers and four books. The model has been validated in workshops and interviews, which in total included 28 individuals in five different companies. The model was well received during the validation, and was appreciated for its simplicity and its ability to show differences in rating between developers and enablers.

DevOps and continuous practices are attracting steadily growing attention by both practitioners and researchers in the software engineering community. The terms are often used inconsistently, interchangeably and with unclear meaning, however. Taking the position that this ambiguity and miscommunication renders the community great harm, in that it impedes our ability to critically appraise these practices, their effects and interplay between them, we analyze how published literature on both continuous practices and DevOps treat the terms. Based on this analysis, along with statements by often cited sources in the community as well as personal experience from researching and practicing these concepts, we propose guidelines to help authors reduce ambiguity in their publications. Additionally, definitions designed to reflect mainstream interpretation while disentangling the terms from one another are presented.

Based on interviews with 20 developers from two case study companies that develop large-scale software-intensive embedded systems, this paper presents the main factors that affect how often developers deliver software to the mainline. Further on, the paper describes the continuous integration behaviors in projects where up to 1,000 developers commit to the same mainline. The main factors that could enable more frequent integration of software are: "Activity planning and execution", "System thinking", "Speed" and "Confidence through test activities". Behind these main themes we also present a wide range of sub-categories ("Modular and loosely coupled architecture", "Test selection" etc) which summarizes what the developers themselves see as the continuous integration impediments in large-scale industry projects.