Background: Software engineering research articles should make precise claims regarding their contribution, so that practitioners can decide when they might be interested and researchers can better recognize (1) whether the given research is valid, (2) which published works to use as stepping stones for their own research (and which not), and (3) where additional research is required. In particular, articles should spell out what assumptions were made at each research step. Question: Can we identify recurring patterns of assumptions that are not spelled out? Method: This is a position paper. It formulates impressions, but does not present concrete evidence. Results: Assumptions that are wrong or assumptions that are risky and not explicit threaten the integrity of the scientific record. There are several recurring types of such assumptions. The frequency of these problems is currently unknown. Conclusion: The software engineering research community should become more conscious and more explicit with respect to the assumptions that underlie individual research works.

Background: Despite a lot of research on the effectiveness of Pair Programming (PP), the question when it is useful or less useful remains unsettled.

Context: While successful conventional software development regularly employs separate testing staff, there are successful agile teams with as well as without separate testers. Question: How does successful agile development work without separate testers? What are advantages and disadvantages? Method: A case study, based on Grounded Theory evaluation of interviews and direct observation of three agile teams; one having separate testers, two without. All teams perform long-term development of parts of e-business web portals. Results: Teams without testers use a quality experience work mode centered around a tight field-use feedback loop, driven by a feeling of responsibility, supported by test automation, resulting in frequent deployments. Conclusion: In the given domain, hand-overs to separate testers appear to hamper the feedback loop more than they contribute to quality, so working without testers is preferred. However, Quality Experience is achievable only with modular architectures and in suitable domains.

Context.Several empirical studies have explored the benefits of software design patterns, but their collective results are highly inconsistent. Resolving the inconsistencies requires investigating moderators—i.e., variables that cause an effect to differ across contexts.Objectives.Replicate a design patterns experiment at multiple sites and identify sufficient moderators to generalize the results across prior studies.Methods.We perform a close replication of an experiment investigating the impact (in terms of time and quality) of design patterns (Decorator and Abstract Factory) on software maintenance. The experiment was replicated once previously, with divergent results. We execute our replication at four universities—spanning two continents and three countries—using a new method for performing distributed replications based on closely coordinated, small-scale instances (“joint replication”). We perform two analyses: 1) apost-hocanalysis of moderators, based on frequentist and Bayesian statistics; 2) ana priorianalysis of the original hypotheses, based on frequentist statistics.Results.The main effect differs across the previous instances of the experiment and across the sites in our distributed replication. Our analysis of moderators (including developer experience and pattern knowledge) resolves the differences sufficiently to allow for cross-context (and cross-study) conclusions. The final conclusions represent 126 participants from five universities and 12 software companies, spanning two continents and at least four countries.Conclusions.The Decorator pattern is found to be preferable to a simpler solution during maintenance, as long as the developer has at least some prior knowledge of the pattern. For Abstract Factory, the simpler solution is found to be mostly equivalent to the pattern solution. Abstract Factory is shown to require a higher level of knowledge and/or experience than Decorator for the pattern to be beneficial.

Context: General knowledge transfer is often considered a valuable effect or side-effect of pair programming, but even more important is its role for the success of the pair programming session itself: The partners often need to explain an idea to carry the process forward. Goal: Understand the mechanisms at work when knowledge is transferred during a pair programming session; provide practical advice for constructive behavior. Method: Qualitative data analysis of recordings of actual industrial pair programming sessions. Results: Some pairs are much more efficient in their knowledge transfer than others. These pairs manage to (1) not attempt to explain multiple things at once, (2) not lose sight of a topic, (3) clarify difficult points in stages. Conclusions: Pair programming requires skill beyond software development skill. To be able to identify knowledge needs and then push such knowledge to or pull it from the partner successfully is one aspect of such skill. We characterize a number of its elements.

The RESER workshop provides a venue in which empirical software engineering researchers can discuss the theoretical foundations and methods of replication, as well as present the results of specific replicated studies.

The classical definition of pair programming (PP) describes it via two obvious roles: driver (the person currently having the keyboard) and observer (the other, alternatively called navigator). Although prior research has found some assumptions regarding these roles to be false, so far no alternative PP role model took hold. Instead, most PP research tacitly assumes the classical model to be true and thus PP to be no more difficult than solo programming. We perform qualitative research (using Grounded Theory Methodology) to find a more realistic role model, and have uncovered a suprising complexity: There are more than two roles, they are assumed and unassumed gradually, multiple roles can be held by one person at the same time, and some of their facets are subtle. Mastering this complexity requires specific PP skills beyond mere programming and communication skills. By ignoring such skills, previous PP studies (in particular the controlled experiments) have investigated a rather mixed bag of situations, which explains their heterogeneous results. The emerging result is that qualitative research on the PP process will lead to constructive behavioral advice (process patterns) for pair members and to more meaningful designs for quantitative PP research.

Empirical evidence on emergent properties of different web development platforms when used in a non-trivial setting is rare to non-existent. In this paper we report on an experiment called Plat_Forms 2011 where teams of professional software developers implemented the same specification of a small to medium sized web application using different web development platforms, with 3 to 4 teams per platform. We define platforms by the main programming language used, in our case Java, Perl, PHP, or Ruby. In order to find properties that are similar within a web development platform but different across platforms, we analyzed several characteristics of the teams and their solutions, such as completeness, robustness, structure and aspects of the team's development process. We found certain characteristics that can be attributed to the platforms used but others that cannot. Our findings also indicate that for some characteristics the programming language might not be the best attribute by which to define the platform anymore.

Background: For developing Web-based applications, there exist several competing and widely used technological platforms (consisting of a programming language, framework(s), components, and tools), each with an accompanying development culture and style. Research question: Do Web development projects exhibit emergent process or product properties that are characteristic and consistent within a platform, but show relevant substantial differences across platforms or do team-to-team individual differences outweigh such differences, if any? Such a property could be positive (i.e., a platform advantage), negative, or neutral, and it might be unobvious which is which. Method: In a nonrandomized, controlled experiment, framed as a public contest called “Plat_Forms,” top-class teams of three professional programmers competed to implement the same requirements for a Web-based application within 30 hours. Three different platforms (Java EE, PHP, or Perl) were used by three teams each. We compare the resulting nine products and process records along many dimensions, both external (usability, functionality, reliability, security, etc.) and internal (size, structure, modifiability, etc.). Results: The various results obtained cover a wide spectrum: First, there are results that many people would have called “obvious” or “well known,” say, that Perl solutions tend to be more compact than Java solutions. Second, there are results that contradict conventional wisdom, say, that our PHP solutions appear in some (but not all) respects to be actually at least as secure as the others. Finally, one result makes a statement we have not seen discussed previously: Along several dimensions, the amount of within-platform variation between the teams tends to be smaller for PHP than for the other platforms. Conclusion: The results suggest that substantial characteristic platform differences do indeed exist in some dimensions, but possibly not in others.

The RESER 2010 workshop provides a venue in which empirical Software Engineering researchers may present and discuss theoretical foundations and methods of replication, as well as the results of replicated studies.

Many small and medium-sized systems have little or no design documentation, which makes program understanding during maintenance enormously more difficult when performed by outsiders. Thus, if only minimal design documentation is available, which form should it take to maximize its usefulness? We suggest that it is helpful if the documentation describes a tour through the source code, leading the user directly to relevant details. This work reports an evaluation of this conceptual idea in the form of a controlled experiment with 59 student subjects working on a difficult program understanding task in the context of the 27 KLOC JHotDraw graphics framework. One group received a plain text documentation, the other received tour-structured documentation which they navigated by using an Eclipse plugin called JTourBus that we constructed for the experiment. The results indicate that program understanding can be achieved somewhat faster (albeit not more correctly) with JTourBus than with a plain text document.

abaXX. components was one of the first AN software products fully based on Enterprise JavaBeans/spl trade/ (EJB) technology. We describe the evolution of its architecture as it moved from simply taking the initial EJB hype for the truth, through several intermediate stages, to using EJB simply as one of several encapsulated implementation techniques. So far, the public perception of how to use EJB properly evolved along a similar path, lagging 6 to 12 months behind.

Programs whose parallelism stems from multiple recursion form an interesting subclass of parallel programs with many practical applications. The highly irregular shape of many recursion trees makes it difficult to obtain good load balancing with small overhead. We present a system, called REAPAR, that executes recursive C programs in parallel on SMP machines. Based on data from a single profiling run of the program, REAPAR selects a load-balancing strategy that is both effective and efficient and it generates parallel code implementing that strategy. The performance obtained by REAPAR on a diverse set of benchmarks matches that published for much more complex systems requiring high-level problem-oriented explicitly parallel constructs. A case study even found REAPAR to be competitive to handwritten (low-level, machine-oriented) thread-parallel code.

Using design patterns is claimed to improve programmer productivity and software quality. Such improvements may manifest both at construction time (in faster and better program design) and at maintenance time (in faster and more accurate program comprehension). The paper focuses on the maintenance context and reports on experimental tests of the following question: does it help the maintainer if the design patterns in the program code are documented explicitly (using source code comments) compared to a well-commented program without explicit reference to design patterns? Subjects performed maintenance tasks on two programs ranging from 360 to 560 LOC including comments. The experiments tested whether pattern comment lines (PCL) help during maintenance if patterns are relevant and sufficient program comments are already present. This question is a challenge for the experimental methodology: A setup leading to relevant results is quite difficult to find. We discuss these issues in detail and suggest a general approach to such situations. A conservative analysis of the results supports the hypothesis that pattern-relevant maintenance tasks were completed faster or with fewer errors if redundant design pattern information was provided. The article provides the first controlled experiment results on design pattern usage and it presents a solution approach to an important class of experiment design problems for experiments regarding documentation.

We present a software system, called Tunserver, which recognizes a musical tune whistled by the user, finds it in a database, and returns its name, composer, and other information. Such a service is useful for track retrieval at radio stations, music stores, etc., and is also a step toward the long-term goal of communicating with a computer much like one would with a human being. Tuneserver is implemented as a public Java-based WWW service with a database of approximately 10,000 motifs. Tune recognition is based on a highly error-resistant encoding, proposed by Parsons, that uses only the direction of the melody, ignoring the size of intervals as well as rhythm. We present the design and implementation of the tune recognition core, outline the design of the Web service, and describe the results obtained in an empirical evaluation of the new interface, including the derivation of suitable system parameters, resulting performance figures, and an error analysis.

The personal software process is a process improvement methodology aimed at individual software engineers. It claims to improve software quality (in particular defect content), effort estimation capability, and process adaptation and improvement capabilities. We have tested some of these claims in an experiment comparing the performance of participants who had just previously received a PSP course to a different group of participants who had received other technical training instead. Each participant of both groups performed the same task. We found the following positive effects: the PSP group estimated their productivity (though not their effort) more accurately, made fewer trivial mistakes, and their programs performed more careful error-checking; further, the performance variability was smaller in the PSP group in various respects. However, the improvements are smaller than the PSP proponents usually assume, possibly due to the low actual usage of PSP techniques in the PSP group. We conjecture that PSP training alone does not automatically realize the PSP's potential benefits (as seen in some industrial PSP success stories) when programmers are left alone with motivating themselves to actually use the PSP techniques.

Software design patterns package proven solutions to recurring design problems in a form that simplifies reuse. We are seeking empirical evidence whether using design patterns is beneficial. In particular, one may prefer using a design pattern even if the actual design problem is simpler than that solved by the pattern, i.e., if not all of the functionality offered by the pattern is actually required. Our experiment investigates software maintenance scenarios that employ various design patterns and compares designs with patterns to simpler alternatives. The subjects were professional software engineers. In most of our nine maintenance tasks, we found positive effects from using a design pattern: either its inherent additional flexibility was achieved without requiring more maintenance time or maintenance time was reduced compared to the simpler alternative. In a few cases, we found negative effects: the alternative solution was less error-prone or required less maintenance time. Overall, we conclude that, unless there is a clear reason to prefer the simpler solution, it is probably wise to choose the flexibility provided by the design pattern because unexpected new requirements often appear. We identify several questions for future empirical research.

Domain-specific constraints can be exploited to implement compiler optimizations that are not otherwise feasible. Compilers for neural network learning algorithms can achieve near-optimal colocality of data and processes and near-optimal balancing of load over processors, even for dynamically irregular problems. This is impossible for general programs, but restricting programs to the neural algorithm domain allows for the exploitation of domain-specific properties. The operations performed by neural algorithms are broadcasts, reductions, and object-local operations only; the load distribution is regular with respect to the (perhaps irregular) network topology; changes of network topology occur only from time to time. A language, compilation techniques, and a compiler implementation on the MasPar MP-1 are described and quantitative results for the effects of various optimizations used in the compiler are shown. Conservative experiments with weight pruning algorithms yield performance improvements of 27 percent due to load balancing and 195 percent improvement is achieved due to data locality, both compared to unoptimized versions. Two other optimizations-connection allocation and selecting the number of replicates-speed programs up by about 50 percent and 100 percent, respectively. This work can be viewed as a case study in exploiting domain-specific information; some of the principles presented here may apply to other domains as well.

Type checking is considered an important mechanism for detecting programming errors, especially interface errors. This report describes an experiment to assess the defect-detection capabilities of static, intermodule type checking. The experiment uses ANSI C and Kernighan & Ritchie (K&R) C. The relevant difference is that the ANSI C compiler checks module interfaces (i.e., the parameter lists calls to external functions), whereas K&R C does not. The experiment employs a counterbalanced design in which each of the 40 subjects, most of them CS PhD students, writes two nontrivial programs that interface with a complex library (Motif). Each subject writes one program in ANSI C and one in K&R C. The input to each compiler run is saved and manually analyzed for defects. Results indicate that delivered ANSI C programs contain significantly fewer interface defects than delivered K&R C programs. Furthermore, after subjects have gained some familiarity with the interface they are using, ANSI C programmers remove defects faster and are more productive (measured in both delivery time and functionality implemented).