Software developers and testers have to work together to achieve the goals of software development projects. In globally distributed software projects the development and testing are often scattered across multiple locations forming virtual teams. Further, the distributed projects are so complex that none of team members can possibly possess all the knowledge about the project individually. During testing in such teams, developers and testers need to coordinate and communicate frequently. However, coordination is affected by the availability of the project information, which is distributed among different project members and organizational structures. Many companies are facing decisions about how to apply agile methods in their distributed projects. These companies are often motivated by the opportunities of solving the coordination and communication difficulties associated with global software development. In this paper we investigate the communication between testers and developers in two teams from two software companies performing continuous agile testing in a distributed setting. We describe four communication practices used by the team: handover through issue tracker system, formal meetings, written communication and coordination by mutual adjustment. We also discuss communication between testers and developers in collocated versus distributed testers and developers. We have found that early participation of the testers is very important to the success of the handover between testers and developers. The communication between developers and testers is not sufficiently effective through written communication and that it changes depending on the type of the tasks and experience of the testers.

Small, self-directed teams are central in agile development. This article investigates the effect of teamwork quality on team performance, learning and work satisfaction in agile software teams, and whether this effect differs from that of traditional software teams. A survey was administered to 477 respondents from 71 agile software teams in 26 companies and analyzed using structural equation modeling. A positive effect of teamwork quality on team performance was found when team members and team leaders rated team performance. In contrast, a negligible effect was found when product owners rated team performance. The effect of teamwork quality on team members´ learning and work satisfaction was strongly positive, but was only rated by the team members. Despite claims of the importance of teamwork in agile teams, this study did not find teamwork quality to be higher than in a similar survey on traditional teams. The effect of teamwork quality on team performance was only marginally greater for the agile teams than for the traditional teams.

Virtual teams, with a high level of interdependence and cooperation among team members, are the building block of successful global software organizations. While becoming agile helps on communication and collaboration, such teams meet several challenges in the form of cultural differences, language barriers, national traditions, different values and norms, lack of face-to-face communication, time-zone differences, and difficulties in building and maintaining trust. A successful agile virtual team needs to have the right structure, but equally important is the ability to improve as a team, to become self-managing with shared decision-making and shared leadership. It takes a long time to form such a team, and expert coaching is needed. We describe and discuss how one team leader coached and improved a global virtual agile team at a large savings and insurance company over a period of one year. Because the team members had overlapping working hours the team was able to base coordination on mutual adjustment and frequent feedback. Social software and face-to-face meetings were important factors to achieve this. By involving the remote developers in the strategy of the product, enabling everyone to pick their own tasks, and focusing on continuous learning, knowledge sharing and team build activities, the team members became highly motivated and self-managing.

In globally distributed software projects the testing expertise may be scattered across multiple locations. We describe and discuss a globally distributed agile project at DNV GL Software, a multinational provider of software for a safer, smarter and greener future in the energy, process and maritime industries. DNV GL Software is headquartered in Norway. The project is distributed across two locations with 12 team members in Norway and three testers in China. In a distributed agile team with little overlap in working hours the challenge is to coordinate tasks and test activities in a way that makes coordination and communication efficient. DNV GL Software believes in including the remote testers as part of the agile team, enabling self-managing, cross-functional virtual teams that are capable of taking the full responsibility for implementing and verifying one entire feature. To support the communication between testers in China and the rest of the team in Norway, the team needs a shared understanding of the goal of a release and how to collaborate. We conducted interviews with the team and representatives from different roles in the organization, and we performed retrospectives with the team. In this article we describe how continuous testing based on continuous and frequent feedback ensures knowledge sharing and safeguarding the quality of the system under development. We found the following enablers for a successful virtual agile team: coordination by mutual adjustment, dedicated testers and low turnover, shifting working hours, and self-management and autonomy. Non-technical factors, such as socio-technical and organizational factors, have a significant influence on the way software testing is performed in an agile virtual team. To be successful the organization needs to invest in bringing the remote testers closer to the rest of the team, as part of the virtual team.

Agile software development represents a new approach for planning and managing software projects. It puts less emphasis on up-front plans and strict control and relies more on informal collaboration, coordination, and learning. This briefing provides a characterization and definition of agile project management based on extensive studies of large-scale industrial projects. It explains the circumstances behind the change from traditional management with its focus on direct supervision and standardization of work processes, to the newer, agile focus on self-managing teams, including its opportunities and benefits, but also its complexity and challenges. The main focus of the briefing is the four principles of agile project management: minimum critical specification, autonomous teams, redundancy, and feedback and learning. The briefing is intended for researchers, practitioners and educators in software engineering, especially project managers. For researchers, an updated state of the art will be uncovered, and the presentation will be based on current best evidence. For practitioners, principles, processes, and key success factors will be outlined and a successful large-scale case study of agile project management will be presented. For educators, the briefing will provide the basis for developing course material.

Skilled workers are crucial to the success of software development. The current practice in research and industry for assessing programming skills is mostly to use proxy variables of skill, such as education, experience, and multiple-choice knowledge tests. There is as yet no valid and efficient way to measure programming skill. The aim of this research is to develop a valid instrument that measures programming skill by inferring skill directly from the performance on programming tasks. Over two days, 65 professional developers from eight countries solved 19 Java programming tasks. Based on the developers' performance, the Rasch measurement model was used to construct the instrument. The instrument was found to have satisfactory (internal) psychometric properties and correlated with external variables in compliance with theoretical expectations. Such an instrument has many implications for practice, for example, in job recruitment and project allocation.

Context: Several text books and papers published between 2000 and 2002 have attempted to introduce experimental design and statistical methods to software engineers undertaking empirical studies. Objective: This paper investigates whether there has been an increase in the quality of human-centric experimental and quasi-experimental journal papers over the time period 1993 to 2010. Method: Seventy experimental and quasi-experimental papers published in four general software engineering journals in the years 1992-2002 and 2006-2010 were each assessed for quality by three empirical software engineering researchers using two quality assessment methods (a questionnaire-based method and a subjective overall assessment). Regression analysis was used to assess the relationship between paper quality and the year of publication, publication date group (before 2003 and after 2005), source journal, average coauthor experience, citation of statistical text books and papers, and paper length. The results were validated both by removing papers for which the quality score appeared unreliable and using an alternative quality measure. Results: Paper quality was significantly associated with year, citing general statistical texts, and paper length (p <; 0.05). Paper length did not reach significance when quality was measured using an overall subjective assessment. Conclusions: The quality of experimental and quasi-experimental software engineering papers appears to have improved gradually since 1993.

Context: Code smells are assumed to indicate bad design that leads to less maintainable code. However, this assumption has not been investigated in controlled studies with professional software developers. Aim: This paper investigates the relationship between code smells and maintenance effort. Method: Six developers were hired to perform three maintenance tasks each on four functionally equivalent Java systems originally implemented by different companies. Each developer spent three to four weeks. In total, they modified 298 Java files in the four systems. An Eclipse IDE plug-in measured the exact amount of time a developer spent maintaining each file. Regression analysis was used to explain the effort using file properties, including the number of smells. Result: None of the 12 investigated smells was significantly associated with increased effort after we adjusted for file size and the number of changes; Refused Bequest was significantly associated with decreased effort. File size and the number of changes explained almost all of the modeled variation in effort. Conclusion: The effects of the 12 smells on maintenance effort were limited. To reduce maintenance effort, a focus on reducing code size and the work practices that limit the number of changes may be more beneficial than refactoring code smells.

Context is a central concept in empirical software engineering. It is one of the distinctive features of the discipline and it is an in-dispensable part of software practice. It is likely responsible for one of the most challenging methodological and theoretical problems: study-to-study variation in research findings. Still, empirical software engineering research is mostly concerned with attempts to identify universal relationships that are independent of how work settings and other contexts interact with the processes important to software practice. The aim of this paper is to provide an overview of how context affects empirical research and how empirical software engineering research can be better 'contextualized' in order to provide a better understanding of what works for whom, where, when, and why. We exemplify the importance of context with examples from recent systematic reviews and offer recommendations on the way forward.

Ethnography is about the adoption of a cultural lens to observe and interpret events, actions, and behaviors, ensuring that they are placed in a relevant and meaningful context. Using this approach, it is possible to capture and analyze software development practices. Our aims are to illustrate the use of an ethnographic approach in a case study of agile software development adoption, to discuss the methodological challenges involved, and to provide support to others who conduct ethnographic studies of software practice. An ethnographic case study was conducted, employing participant observation, interviews, and document analysis. Difficulties and decisions were recorded and compared with those encountered in the literature. Finally, key challenges and guidelines to tackle them were discussed and documented. We identified five key challenges of applying ethnography to the study of software practices: (a) working in collaboration with and having something to offer to the participating company; (b) the insider/outsider dynamic of participant observation; (c) the balance between participant listening and participant observation; (d) the researcher's relationship with the participants; and (e) the rigor in qualitative work that involves the dilemma of the contextualization to be sufficiently broad and detailed. This study shows that ethnographic methods are indispensible when trying to understand software practice, and that the fundamental challenge for the researcher is to balance the role of participant observer with rigorous fieldwork.

During systematic literature reviews it is necessary to assess the quality of empirical papers. Current guidelines suggest that two researchers should independently apply a quality checklist and any disagreements must be resolved. However, there is little empirical evidence concerning the effectiveness of these guidelines. This paper investigates the three techniques that can be used to improve the reliability (i.e. the consensus among reviewers) of quality assessments, specifically, the number of reviewers, the use of a set of evaluation criteria and consultation among reviewers. We undertook a series of studies to investigate these factors. Two studies involved four research papers and eight reviewers using a quality checklist with nine questions. The first study was based on individual assessments, the second study on joint assessments with a period of inter-rater discussion. A third more formal randomised block experiment involved 48 reviewers assessing two of the papers used previously in teams of one, two and three persons to assess the impact of discussion among teams of different size using the evaluations of the “teams” of one person as a control. For the first two studies, the inter-rater reliability was poor for individual assessments, but better for joint evaluations. However, the results of the third study contradicted the results of Study 2. Inter-rater reliability was poor for all groups but worse for teams of two or three than for individuals. When performing quality assessments for systematic literature reviews, we recommend using three independent reviewers and adopting the median assessment. A quality checklist seems useful but it is difficult to ensure that the checklist is both appropriate and understood by reviewers. Furthermore, future experiments should ensure participants are given more time to understand the quality checklist and to evaluate the research papers.

The skills of software developers are important to the success of software projects. Also, when studying the general effect of a tool or method, it is important to control for individual differences in skill. However, the way skill is assessed is often ad hoc, or based on unvalidated methods. According to established test theory, validated tests of skill should infer skill levels from well-defined performance measures on multiple, small, representative tasks. In this respect, we show how time and quality, which are often analyzed separately, can be combined as task performance and subsequently be aggregated as an approximation of skill. Our results show significant positive correlations between our proposed measures of skill and other variables, such as seniority, lines of code written, and self-evaluated expertise. The method for combining time and quality is a promising first step to measuring programming skill in both industry and research settings.

Thematic analysis is an approach that is often used for identifying, analyzing, and reporting patterns (themes) within data in primary qualitative research. 'Thematic synthesis' draws on the principles of thematic analysis and identifies the recurring themes or issues from multiple studies, interprets and explains these themes, and draws conclusions in systematic reviews. This paper conceptualizes the thematic synthesis approach in software engineering as a scientific inquiry involving five steps that parallel those of primary research. The process and outcome associated with each step are described and illustrated with examples from systematic reviews in software engineering.

Synthesis of case studies is different from synthesis of purely quantitative studies, for example, in that sampling and analysis in primary studies have been carried out differently, and that primary results are of a different nature. The objective of this research is to identify what challenges should be considered when choosing and using a method for synthesis of case studies. We collected experience from independent synthesis of two published case studies (on trust in outsourcing) by two teams, one team applied cross-case analysis, the other team applied thematic synthesis. The two teams reached both supporting and complimentary conclusions. Identified challenges relate to the goals and research questions of the cases to be synthesized, the number of case studies, temporal and spatial variations, and access to raw data.

Synthesizing the evidence from a set of studies that spans many countries and years, and that incorporates a wide variety of research methods and theoretical perspectives, is probably the single most challenging task of performing a systematic review. In this paper, we perform a tertiary review to assess the types and methods of research synthesis in systematic reviews in software engineering. Almost half of the 31 studies included in our review did not contain any synthesis; of the ones that did, two thirds performed a narrative or a thematic synthesis. The results show that, despite the focus on systematic reviews, there is, currently, limited attention to research synthesis in software engineering. This needs to change and a repertoire of synthesis methods needs to be an integral part of systematic reviews to increase their significance and utility for research and practice.

Context: The authors wanted to assess whether the quality of published human-centric software engineering experiments was improving. This required a reliable means of assessing the quality of such experiments. Aims: The aims of the study were to confirm the usability of a quality evaluation checklist, determine how many reviewers were needed per paper that reports an experiment, and specify an appropriate process for evaluating quality. Method: With eight reviewers and four papers describing human-centric software engineering experiments, we used a quality checklist with nine questions. We conducted the study in two parts: the first was based on individual assessments and the second on collaborative evaluations. Results: The inter-rater reliability was poor for individual assessments but much better for joint evaluations. Four reviewers working in two pairs with discussion were more reliable than eight reviewers with no discussion. The sum of the nine criteria was more reliable than individual questions or a simple overall assessment. Conclusions: If quality evaluation is critical, more than two reviewers are required and a round of discussion is necessary. We advise using quality criteria and basing the final assessment on the sum of the aggregated criteria. The restricted number of papers used and the relatively extensive expertise of the reviewers limit our results. In addition, the results of the second part of the study could have been affected by removing a time restriction on the review as well as the consultation process.

Although Scrum is an important topic in software engineering and information systems, few longitudinal industrial studies have investigated the effects of Scrum on software quality, in terms of defects and defect density, and the quality assurance process. In this paper we report on a longitudinal study in which we have followed a project over a three-year period. We compared software quality assurance processes and software defects of the project between a 17-month phase with a plan-driven process, followed by a 20-month phase with Scrum. The results of the study did not show a significant reduction of defect densities or changes of defect profiles after Scrum was used. However, the iterative nature of Scrum resulted in constant system and acceptance testing and related defect fixing, which made the development process more efficient in terms of fewer surprises and better control of software quality and release date. In addition, software quality and knowledge sharing got more focus when using Scrum. However, Scrum put more stress and time pressure on the developers, and made them reluctant to perform certain tasks for later maintenance, such as refactoring.

Systematic reviews are only as good as the evidence they are based on. It is important, therefore, that users of systematic reviews know how much confidence they can place in the conclusions and recommendations arising from such reviews. In this paper we present an overview of some of the most influential systems for assessing the quality of individual primary studies and for grading the overall strength of a body of evidence. We also present an example of the use of such systems based on a systematic review of empirical studies of agile software development. Our findings suggest that the systems used in other disciplines for grading the strength of evidence for and reporting of systematic reviews, especially those that take account of qualitative and observational studies are of particular relevance for software engineering.

Systematic reviews are one of the key building blocks of evidence-based software engineering. Current guidelines for such reviews are, for a large part, based on standard meta-analytic techniques. However, such quantitative techniques have only limited applicability to software engineering research. In this paper, therefore, we describe our experience with an approach to combine diverse study types in a systematic review of empirical research of agile software development.

A total of 295 junior, intermediate, and senior professional Java consultants (99 individuals and 98 pairs) from 29 international consultancy companies in Norway, Sweden, and the UK were hired for one day to participate in a controlled experiment on pair programming. The subjects used professional Java tools to perform several change tasks on two alternative Java systems with different degrees of complexity. The results of this experiment do not support the hypotheses that pair programming in general reduces the time required to solve the tasks correctly or increases the proportion of correct solutions. On the other hand, there is a significant 84 percent increase in effort to perform the tasks correctly. However, on the more complex system, the pair programmers had a 48 percent increase in the proportion of correct solutions but no significant differences in the time taken to solve the tasks correctly. For the simpler system, there was a 20 percent decrease in time taken but no significant differences in correctness. However, the moderating effect of system complexity depends on the programmer expertise of the subjects. The observed benefits of pair programming in terms of correctness on the complex system apply mainly to juniors, whereas the reductions in duration to perform the tasks correctly on the simple system apply mainly to intermediates and seniors. It is possible that the benefits of pair programming will exceed the results obtained in this experiment for larger, more complex tasks and if the pair programmers have a chance to work together over a longer period of time

Empirically based theories are generally perceived as foundational to science. However, in many disciplines, the nature, role and even the necessity of theories remain matters for debate, particularly in young or practical disciplines such as software engineering. This article reports a systematic review of the explicit use of theory in a comprehensive set of 103 articles reporting experiments, from of a total of 5,453 articles published in major software engineering journals and conferences in the decade 1993-2002. Of the 103 articles, 24 use a total of 40 theories in various ways to explain the cause-effect relationship(s) under investigation. The majority of these use theory in the experimental design to justify research questions and hypotheses, some use theory to provide post hoc explanations of their results, and a few test or modify theory. A third of the theories are proposed by authors of the reviewed articles. The interdisciplinary nature of the theories used is greater than that of research in software engineering in general. We found that theory use and awareness of theoretical issues are present, but that theory-driven research is, as yet, not a major issue in empirical software engineering. Several articles comment explicitly on the lack of relevant theory. We call for an increased awareness of the potential benefits of involving theory, when feasible. To support software engineering researchers who wish to use theory, we show which of the reviewed articles on which topics use which theories for what purposes, as well as details of the theories' characteristics

Understanding how to implement software process improvement (SPI) successfully is arguably the most challenging issue facing the SPI field today. The SPI literature contains many case studies of successful companies and descriptions of their SPI programs. However, the research efforts to date are limited and inconclusive and without adequate theoretical and psychometric justification. This paper extends and integrates models from prior research by performing an empirical investigation of the key factors for success in SPI. A quantitative survey of 120 software organizations was designed to test the conceptual model and hypotheses of the study. The results indicate that success depends critically on six organizational factors, which explained more than 50 percent of the variance in the outcome variable. The main contribution of the paper is to increase the understanding of the influence of organizational issues by empirically showing that they are at least as important as technology for succeeding with SPI and, thus, to provide researchers and practitioners with important new insights regarding the critical factors of success in SPI.

Our objective is to describe how software engineering might benefit from an evidence-based approach and to identify the potential difficulties associated with the approach. We compared the organisation and technical infrastructure supporting evidence-based medicine (EBM) with the situation in software engineering. We considered the impact that factors peculiar to software engineering (i.e. the skill factor and the lifecycle factor) would have on our ability to practice evidence-based software engineering (EBSE). EBSE promises a number of benefits by encouraging integration of research results with a view to supporting the needs of many different stakeholder groups. However, we do not currently have the infrastructure needed for widespread adoption of EBSE. The skill factor means software engineering experiments are vulnerable to subject and experimenter bias. The lifecycle factor means it is difficult to determine how technologies will behave once deployed. Software engineering would benefit from adopting what it can of the evidence approach provided that it deals with the specific problems that arise from the nature of software engineering.

Existing software engineering and organization development literature acknowledges that there are fundamental operational differences between small and large organizations. Despite this recognition, there has been no attempt to verify whether small and large software organizations implement software process improvement (SPI) programs differently in order to advance their businesses.This study examines whether an organization's size affects its SPI implementation strategy and the degree of SPI success. Based on an extensive literature review of critical factors of quality management, organizational learning, and SPI, a survey questionnaire was developed and data on the implementation of six organizational factors and the resulting organizational performance was collected through a mail survey of 120 software organizations. The findings show that small organizations reported that they implement SPI elements as effectively as large organizations, and in turn, achieve high organizational performance. The main lesson to be learned from this study is that to implement SPI at least as effectively as their large counterparts, small software organizations should capitalize on their relative strengths in employee participation and exploration of new knowledge.

Most quality and software process improvement frameworks emphasize written (i.e. formal) documentation to convey recommended work practices. However, there is considerable skepticism among developers to learn from and adhere to prescribed process models. The latter are often perceived as overly "structured" or implying too much "control". Further, what is relevant knowledge has often been decided by "others"---often the quality manager. The study was carried out in the context of a national software process improvement program in Norway for small- and medium-sized companies to assess the attitude to formalized knowledge and experience sources. The results show that developers are rather skeptical at using written routines, while quality and technical managers are taking this for granted. This is an explosive combination. The conclusion is that formal routines must be supplemented by collaborative, social processes to promote effective dissemination and organizational learning. Trying to force a (well-intended) quality system down the developers' throats is both futile and demoralizing. The wider implications for quality and improvement work is that we must strike a balance between the "disciplined" or "rational" and the "creative" way of working.