Modern release engineering practices provide multiple benefits for software companies, but organizations have struggled when trying to adopt the most advanced practices, such as continuous delivery. It is not known in which contexts the most advanced practices are applicable and what can be achieved by adopting them. In this study, we discuss the effect of the organizational context on adopted release engineering practices and what outcomes are achieved with the practices. We study two organizational contexts: the startup and the large mature company context. The effect of the product context is mitigated by studying two case organizations with similar products, a rare research opportunity. We performed 18 interviews with various roles in the case organizations. The number of production environments, the number of customers, the control over the production environment, the available resources, the organization size and the distribution of the organization affected the release engineering practices and the ability to release frequently. Having less internal verification and more customer verification enabled fast feedback and customer experimentation in the startup context, but increased the number of production defects. However, having more internal verification in the large mature company context surprisingly did not prevent production defects. The organizational context had a large effect on how achievable modern release engineering practices, such as continuous delivery, were. In the startup context, the lack of resources was the main factor hindering the improvement of release engineering practices, while in the large mature company context, the number of stakeholders and products were the main factors.

Team-level retrospectives are widely used in agile and lean software development, yet little is known about what is actually discussed during retrospectives or their outcomes. In this paper, we synthesise the outcomes of sprint retrospectives in a large, distributed, agile software development organisation. This longitudinal case study analyses data from 37 team-level retrospectives for almost 3 years. We report the outcomes of the retrospectives, their perceived importance for process improvement and relatVed action proposals. Most discussions were related to topics close to and controllable by the team. However, the discussions might suffer from participant bias, and in cases where they are not supported by hard evidence, they might not reflect reality, but rather the sometimes strong opinions of the participants. Some discussions were related to topics that could not be resolved at the team level due to their complexity. Certain topics recurred over a long period of time, either reflecting issues that can and have been solved previously, but that recur naturally as development proceeds, or reflecting waste since they cannot be resolved or improved on by the team due to a lack of controllability or their complexity. For example, the discussion on estimation accuracy did not reflect the true situation and improving the estimates was complicated. On the other hand, discussions on the high number of known bugs recurred despite effective improvements as development proceeded.

Context: Continuous delivery is a software development discipline in which software is always kept releasable. The literature contains instructions on how to adopt continuous delivery, but the adoption has been challenging in practice. Objective: In this study, a systematic literature review is conducted to survey the faced problems when adopting continuous delivery. In addition, we identify causes for and solutions to the problems. Method: By searching five major bibliographic databases, we identified 293 articles related to continuous delivery. We selected 30 of them for further analysis based on them containing empirical evidence of adoption of continuous delivery, and focus on practice instead of only tooling. We analyzed the selected articles qualitatively and extracted problems, causes and solutions. The problems and solutions were thematically synthesized into seven themes: build design, system design, integration, testing, release, human and organizational and resource. Results: We identified a total of 40 problems, 28 causal relationships and 29 solutions related to adoption of continuous delivery. Testing and integration problems were reported most often, while the most critical reported problems were related to testing and system design. Causally, system design and testing were most connected to other themes. Solutions in the system design, resource and human and organizational themes had the most significant impact on the other themes. The system design and build design themes had the least reported solutions. Conclusions: When adopting continuous delivery, problems related to system design are common, critical and little studied. The found problems, causes and solutions can be used to solve problems when adopting continuous delivery in practice.

Context: In continuous delivery, the aim is that every feature passes through the integration and deployment pipeline, resulting in an immediately deployable product. This practice has been proposed to accelerate value delivery, improve software quality and increase developer productivity.

Context: Continuous delivery (CD) is a development practice for decreasing the time-to-market by keeping software releasable all the time. Adopting CD within a stage-gate managed development process might be useful, although scientific evidence of such adoption is not available. In a stage-gate process, new releases pass through stages and gates protect low-quality output from progressing. Large organizations with stage-gate processes are often hierarchical and the adoption can be either top-down, driven by the management, or bottom-up, driven by the development unit.

Signal processing software is characterized by a heavy emphasis on arithmetic calculations and the lack of complicated control structures, placing specific constraints on which testing techniques are applicable and how signal processing software can be efficiently tested. In this paper, we analyze the unique characteristics of signal processing software from the testing viewpoint and propose applied techniques for tackling the verification challenges of such software. We propose a testing method for the signal processing context and provide examples of its application to an FIR and a second order IIR filter. This method extends the applicability of algebraic testing and domain testing methods to signal processing software. The developed method applies to linear systems and can be further extended to take nonlinearities in the tested system into account.

Root cause analysis (RCA) is a recommended practice in retrospectives and cause–effect diagram (CED) is a commonly recommended technique for RCA. Our objective is to evaluate whether CED improves the outcome and perceived utility of RCA. We conducted a controlled experiment with 11 student software project teams by using a single factor paired design resulting in a total of 22 experimental units. Two visualization techniques of underlying causes were compared: CED and a structural list of causes. We used the output of RCA, questionnaires, and group interviews to compare the two techniques. In our results, CED increased the total number of detected causes. CED also increased the links between causes, thus, suggesting more structured analysis of problems. Furthermore, the participants perceived that CED improved organizing and outlining the detected causes. The implication of our results is that using CED in the RCA of retrospectives is recommended, yet, not mandatory as the groups also performed well with the structural list. In addition to increased number of detected causes, CED is visually more attractive and preferred by retrospective participants, even though it is somewhat harder to read and requires specific software tools.

We present a field study on how testers use knowledge while performing exploratory software testing (ET) in industrial settings. We video recorded 12 testing sessions in four industrial organizations, having our subjects think aloud while performing their usual functional testing work. Using applied grounded theory, we analyzed how the subjects performed tests and what type of knowledge they utilized. We discuss how testers recognize failures based on their personal knowledge without detailed test case descriptions. The knowledge is classified under the categories of domain knowledge, system knowledge, and general software engineering knowledge. We found that testers applied their knowledge either as a test oracle to determine whether a result was correct or not, or for test design, to guide them in selecting objects for test and designing tests. Interestingly, a large number of failures, windfall failures, were found outside the actual focus areas of testing as a result of exploratory investigation. We conclude that the way exploratory testers apply their knowledge for test design and failure recognition differs clearly from the test-case-based paradigm and is one of the explanatory factors of the effectiveness of the exploratory testing approach.

There is a recognized disconnect between testing research and industry practice, and more studies are needed on understanding how testing is conducted in real-world circumstances instead of demonstrating the superiority of specific methods. Recent literature indicates that testing is a cross-cutting activity that involves various organizational roles rather than the sole involvement of specialized testers. This research empirically investigates how testing involves employees in varying organizational roles in software product companies. We studied the organization and values of testing using an exploratory case study methodology through interviews, defect database analysis, workshops, analyses of documentation, and informal communications at three software product companies. We analyzed which employee groups test software in the case companies, and how many defects they find. Two companies organized testing as a team effort, and one company had a specialized testing group because of its different development model. We found evidence that testing was not an action conducted only by testing specialists. Testing by individuals with customer contact and domain expertise was an important validation method. We discovered that defects found by developers had the highest fix rates while those revealed by specialized testers had the lowest. The defect importance was susceptible to organizational competition of resources (i.e., overvaluing defects of reporter’s own products or projects). We conclude that it is important to understand the diversity of individuals participating in software testing and the relevance of validation from the end users’ viewpoint. Future research is required to evaluate testing approaches for diverse organizational roles. Finally, to improve defect information, we suggest increasing automation in defect data collection.

Objective: We studied what are the characteristics of high performing software testers in the industry. Method: We conducted an exploratory case study, collecting data through recorded interviews of one development manager and three testers in each of the three companies, analysis of the defect database, and informal communication within our research partnership with the companies. Results: We found that experience, reflection, motivation and personal characteristics were the top level themes. Experience related to the domain, e.g. processes of the customer, and on the other hand, specialized technical skills, e.g. performance testing, were seen more important than skills of test case design and test planning.

We present the results of a qualitative observation study on the manual testing practices in four software development companies. Manual testing practices are seldom studied, and based on the literature we conjecture that they have a strong effect on the effectiveness of manual testing. We observed testing sessions of 11 software professionals performing system level functional testing. As a result we identified 22 manual testing practices that we classified into 9 test session strategies and 13 detailed test execution techniques. Many of the identified techniques were based on similar ideas as traditional test case design techniques. However, the subjects applied these techniques during manual testing without separate test design phase. The results indicate that software professionals use a wide set of strategies and techniques when performing manual testing. Testers seem to need and use techniques even if applying exploratory testing.

This paper presents a controlled experiment comparing the defect detection efficiency of exploratory testing (ET) and test case based testing (TCT). While traditional testing literature emphasizes test cases, ET stresses the individual tester's skills during test execution and does not rely upon predesigned test cases. In the experiment, 79 advanced software engineering students performed manual functional testing on an open-source application with actual and seeded defects. Each student participated in two 90-minute controlled sessions, using ET in one and TCT in the other. We found no significant differences in defect detection efficiency between TCT and ET. The distributions of detected defects did not differ significantly regarding technical type, detection difficulty, or severity. However, TCT produced significantly more false defect reports than ET. Surprisingly, our results show no benefit of using predesigned test cases in terms of defect detection efficiency, emphasizing the need for further studies of manual testing.