Abstract Indentation is an old technique that emphasizes elements in source code using white spaces or tabs. But while this technique has been taught and applied for decades, evidence for its effectiveness is weak: up to 2022 relatively few experiments can be found – the present authors are only aware of one single experiment that revealed an effect of indentation and reported the effect size, but even in that experiment the effect of indentation was found to be weak. The situation changed recently, where an experiment was published that suddenly revealed a strong and large effect of indentation in control flows on reaction time. However, although the experiment provided an initial indication of a possible cause for the difference between indented and non-indented code (the length of the code that can be skipped in indented code), the evidence for this indicator was rather weak. The present paper presents a formal model on the differences in reading times (measured in terms of reaction times) between indented and non-indented code. Based on that model a controlled experiment on generated tasks was designed and executed on 27 participants (undergraduate students, PhD students, professionals). The experiment (again) confirms a strong (p < .001) and large ( $$\eta _{p}^{2}$$ ηp2 = .198, $$\frac{M_{Non-Indented}}{M_{Indented}}$$ MNon-IndentedMIndented = 2.13) effect of indentation. Furthermore, it confirms that the larger the skippable code is, the larger is the difference between reading time of indented and non-indented code (p = .001, $$\eta _{p}^{2}$$ ηp2 = .072). I.e., the experiment goes beyond the point “indented vs non-indented code” and explains the difference by revealing a factor that controls this difference. However, although the previous statements holds true for the whole sample in the experiment, this effect could only be shown for a subset of individual participants.

Java's Stream API, that massively makes use of lambda expressions, permits a more declarative way of defining operations on collections in comparison to traditional loops. While experimental results suggest that the use of the Stream API has measurable benefits with respect to code readability (in comparison to loops), a remaining question is whether it has other implications. And one of such implications is, for example, tooling in general and debugging in particular because of the following: While the traditional loop-based approach applies filters one after another to single elements, the Stream API applies filters on whole collections. In the meantime there are dedicated debuggers for the Stream API, but it remains unclear whether such a debugger (on the Stream API) has a measurable benefit in comparison to the traditional stepwise debugger (on loops). The present papers introduces a controlled experiment on the debugging of filter operations using a stepwise debugger versus a stream debugger. The results indicate that under the experiment's settings the stream debugger has a significant ($\mathrm{p} < .001$) and large, positive effect$(\eta_{p}^{2}=.899;\ \frac{M_{stepwise}}{M_{stream}} \sim 204\%)$. However, the experiment reveals that additional factors interact with the debugger treatment such as whether or not the failing object is known upfront. The mentioned factor has a strong and large disordinal interaction effect with the debugger ($\mathrm{p} < .001; \eta_{p}^{2}=.928$): In case an object is known upfront that can be used to identify a failing filter, the stream debugger is even less efficient than the stepwise debugger$(\frac{M_{stepwise}}{M_{stream}}\sim 72\%)$. Hence, while we found overall a positive effect of the stream debugger, the answer whether or not debugging is easier on loops or streams cannot be answered without taking the other variables into account. Consequently, we see a contribution of the present paper not only in the comparison of different debuggers but in the identification of additional factors.

Java introduced in version 8 with the Stream API means to operate on collections using lambda expressions. Since then, this API is an alternative way to handle collections in a more declarative manner instead of the traditional, imperative style using loops. However, whether the Stream API is beneficial in comparison to loops in terms of usability is unclear. The present paper introduces a randomized control trial (RCT) on the understandability of collection operations performed on 20 participants with the dependent variables response time and correctness. As tasks, subjects had to determine the results for collection operations (either defined with the Stream API or with loops). The results indicate that the Stream API has a significant (p<.001) and large [EQUATION] positive effect on the response times. Furthermore, the usage of the Stream API caused significantly less errors. And finally, the participants perceived their speed with the Stream API higher compared to the loop-based code and the participants considered the code based on the Stream API as more readable. Hence, while existing studies found a negative effect of declarative constructs (in terms of lambda expressions) on the usability of a main stream programming language, the present study found the opposite: the present study gives evidence that declarative code on collections using the Stream API based on lambda expressions has a large, positive effect in comparison to traditional loops.

Abstract In Java, lambda expressions (LEs) were introduced at a time where the similar language construct anonymous inner class (AIC) already existed for years. But while LEs became quite popular in mainstream programming languages in general, their usability is hardly studied. From the Java perspective the need to study the relationship between LEs and AICs was and is quite obvious, because both language constructs co-exist. However, it is quite usual that new language constructs are introduced although they are not or hardly studied using scientific methods – and an often heard argument from programming language designers is that the effort or the costs for the application of the scientific method on language constructs is too high. The present paper contributes in two different ways. First, with respect to LEs in comparison to AICs, this paper presents two N-of-1 studies (i.e. randomized control trials executed on a single subject) where LEs and AICs are used as listeners in Java code. Both experiments had two similar and rather simple tasks (“count the number of parameters”, respectively “count the number of used parameters”) with the dependent variable being reaction time. The first experiment used the number of parameters, the second the number of used parameters as the controlled, independent variable (in addition to the technique LE and AIC). Other variables (LOC, etc.) were randomly generated within given boundaries. The main result of both experiments is that LEs without type annotations require less reading time (phs .2, reduction of reaction time of at most 35%). The results are based on 9,600 observations (one N-of-1 trial with eight replications). This gives evidence that the readability of LEs without type annotations improves the readability of code. However, the effect seems to be so small, that we do not expect this to have a larger impact on daily programming. Second, we see the contribution of this paper in the application of N-of-1 trials. Such experiments require relatively low effort in the data selection but still permit to analyze results in a non-subjective way using commonly accepted analysis techniques. Additionally, they permit to increase the number of selected data points in comparison to traditional multi–subject experiments. We think that researchers should take such experiments into account before planning and executing larger experiments.

Test-Driven Code Review (TDR) is a code review practice in which a reviewer inspects a patch by examining the changed test code before the changed production code. Although this practice has been mentioned positively by practitioners in informal literature and interviews, there is no systematic knowledge of its effects, prevalence, problems, and advantages. In this paper, we aim at empirically understanding whether this practice has an effect on code review effectiveness and how developers' perceive TDR. We conduct (i) a controlled experiment with 93 developers that perform more than 150 reviews, and (ii) 9 semi-structured interviews and a survey with 103 respondents to gather information on how TDR is perceived. Key results from the experiment show that developers adopting TDR find the same proportion of defects in production code, but more in test code, at the expenses of fewer maintainability issues in production code. Furthermore, we found that most developers prefer to review production code as they deem it more critical and tests should follow from it. Moreover, general poor test code quality and no tool support hinder the adoption of TDR. Public preprint: [https: //doi.org/10.5281/zenodo.2551217], data and materials: [https:// doi.org/10.5281/zenodo.2553139].

Although the readability of source code plays an important role in software construction, not many studies are available that do actually compare the impact of different notations on the readability of source code. Among the huge set of possible languages to be studied, one language is frequently used in education as well as in practice: regular expressions. This paper introduces a randomized controlled trial performed on 22 participants that compares two different notations for regular expressions: a textual and a graphical one. It shows that a regular expression's length as well as the notation (textual vs. graphical) has a strong effect on the regular expression's readability while the participants' background showed no measurable effect. On average, the time required by the participants to answer questions about shortest words using the textual representation was almost three times higher than the time it took for answering such questions using the graphical representation.

Lambda functions have become prevalent in mainstream programming languages, as they are increasingly introduced to widely used object oriented programming languages such as Java and C++. Some in the scientific literature argue that this feature increases programmer productivity, ease of reading, and makes parallel programming easier. Others are less convinced, citing concerns that the use of lambdas makes debugging harder. This thesis describes the design, execution and results of an experiment to test the impact of using lambda functions compared to the iterator design pattern for iteration tasks, as a first step in evaluating these claims. The approach is a randomized controlled trial, which focuses on the percentage of tasks completed, number of compiler errors, the percentage of time to fix such errors, and the amount of time it takes to complete programming tasks correctly. The overall goal is to investigate, if lambda functions have an impact on the ability of developers to complete tasks, or the amount of time taken to complete them. Additionally, it is tested if developers introduce more errors while using lambda functions and if fixing errors takes them more time. Lastly, the impact of experience level on productivity is evaluated by comparing the performance of participants from different levels of experience with one-another. Participants were assigned one of five levels based on their progress in the computer science major. The five levels were freshman, sophomore, junior, senior and professional. The professional level was assigned to individuals out of school with 5 or more years of professional experience. Results show that the impact of using lambdas, as opposed to iterators, on the number of tasks completed was significant. The impact on time to completion, number of errors introduced during the experiment and times spent fixing errors were also found to be significant. The analysis of the difference between the different levels of experience also shows a significant difference. The percentage of time spent on fixing compilation errors was 56.37% for the lambda group while it was 44.2% for the control group with 3.5% of the variance being explained by the group difference. 45.7% of the variance in the sample was explained by the difference between the level of education. Therefore, this study suggests that earlier findings that student results are comparable with the results of professional developers are to be considered carefully.

There are a number of experiments that show a benefit for statically typed programming languages. However, it is unclear whether these results are mainly driven by the expectations that developers do benefit from static type systems, i.e. whether the results reflect on the experimenters' bias. From that perspective, it seems consequent to design an experiment that tries to reveals the opposite: to enforce a benefit for dynamically typed languages. This paper describes an experiment that tries to enforce such a benefit for dynamically typed languages in an experimental setting. Four (quite artificial) tasks were designed from which the experimenters expected to measure a clear benefit for dynamically typed languages. However, only in two cases such a benefit could be measured. What is even more interesting is that two other tasks (again, explicitly designed to show a benefit for dynamically typed languages) showed the opposite.

Recent empirical studies that compared static and dynamic type systems on API usability showed a positive impact of static type systems on developer productivity in most cases. Nevertheless, it is unclear how large this effect is in comparison to other factors. One obvious factor in programming is tooling: It is commonly accepted that modern IDEs have a large positive impact on developers, although it is not clear which parts of modern IDEs are responsible for that. One possible---and for most developers obvious candidate---is code completion. This paper describes a 2x2 randomized trial that compares JavaScript and Microsoft's statically typed alternative TypeScript with and without code completion in MS Visual Studio. While the experiment shows (in correspondence to previous experiments) a large positive effect of the statically typed language TypeScript, the code completion effect is not only marginal, but also just approaching statistical significance. This seems to be an indicator that the effect of static type systems is larger than often assumed, at least in comparison to code completion.

Programming language research in the last decades was mainly driven by mathematical methods (such as formal semantics, correctness proofs, type soundness proofs, etc.) or run-time arguments based on benchmark tests. This happened despite the frequent discussion over programming language usability. We have now been through decade after decade of one language after another domainating the field, forcing companies to switch languages and migrate libraries. Now that Javascript seems to be the next language to dominate, people start to ask old questions anew. The first goal of this talk is to discuss why the application of empirical methods is (still) relatively rare in PL research, and to discuss what could be done in empirical methods to make them a substantial part of PL research. The second goal is to speculate about the possible effects that concrete empirical knowledge could have had on the programming language community. For example, what would have happened to programming languages if current knowledge would have been available 30 years ago? What if knowledge about programming languages from the year 2050 would be available today?

When developers use Application Programming Interfaces (APIs), they often rely on documentation to assist their tasks. In previous studies, we reported evidence indicating that static type systems acted as a form of implicit documentation, benefiting developer productivity. Such implicit documentation is easier to maintain, given it is enforced by the compiler, but previous experiments tested users without any explicit documentation. In this paper, we report on a controlled experiment and an exploratory study comparing the impact of using documentation and a static or dynamic type system on a development task. Results of our study both confirm previous findings and show that the benefits of static typing are strengthened with explicit documentation, but that this was not as strongly felt with dynamically typed languages.

Several studies have concluded that static type systems offer an advantage over dynamic type systems for programming tasks involving the discovery of a new API. However, these studies did not take into account modern IDE features; the advanced navigation and code completion techniques available in modern IDEs could drastically alter their conclusions. This study describes an experiment that compares the usage of an unknown API using Java and Groovy using the IDE Eclipse. It turns out that the previous finding that static type systems improve the usability of an unknown API still holds, even in the presence of a modern IDE.

In recent years, the programming language design community has engaged in rigorous debate on the role of empirical evidence in the design of general purpose programming languages. Some scholars contend that the language community has failed to embrace a form of evidence that is non-controversial in other disciplines (e.g., medicine, biology, psychology, sociology, physics, chemistry), while others argue that a science of language design is unrealistic. While the discussion will likely persist for some time, we begin here a systematic evaluation of the use of empirical evidence with human users, documenting, paper-by-paper, the evidence provided for human factors decisions, beginning with 359 papers from the workshops PPIG, Plateau, and ESP. This preliminary work provides the following contributions: an analysis of the 1) overall quantity and quality of empirical evidence used in the workshops, and of the 2) overall significant challenges to reliably coding academic papers. We hope that, once complete, this long-term research project will serve as a practical catalog designers can use when evaluating the impact of a language feature on human users.

Type systems that permit developers to express themselves more precisely are one of the primary topics in programming language research, as well as in industrial software development. While it seems plausible that an expressive static type system increases developer productivity, there is little empirical evidence for or against this hypothesis. Generic types in Java are an example: as an extension of Java's original type system, some claim that Java 1.5 improves the type system's "expressiveness." Even if this claim is true, there exists little empirical evidence that claimed expressiveness leads to a measurable increase in developer productivity. This paper introduces an experiment where generic types (in comparison to raw types) have been evaluated in three different directions: (1) the documentation impact on undocumented APIs, (2) the time required for fixing type errors, and (3) the extensibility of a generic type hierarchy. The results of the experiment suggest that generic types improve documentation and reduce extensibility -- without revealing a difference in the time required for fixing type errors.

Programming experience is an important confounding parameter in controlled experiments regarding program comprehension. In literature, ways to measure or control programming experience vary. Often, researchers neglect it or do not specify how they controlled it. We set out to find a well-defined understanding of programming experience and a way to measure it. From published comprehension experiments, we extracted questions that assess programming experience. In a controlled experiment, we compare the answers of 128 students to these questions with their performance in solving program-comprehension tasks. We found that self estimation seems to be a reliable way to measure programming experience. Furthermore, we applied exploratory factor analysis to extract a model of programming experience. With our analysis, we initiate a path toward measuring programming experience with a valid and reliable tool, so that we can control its influence on program comprehension.

Static type systems play an essential role in contemporary programming languages. Despite their importance, whether static type systems influence human software development capabilities remains an open question. One frequently mentioned argument for static type systems is that they improve the maintainability of software systems - an often used claim for which there is no empirical evidence. This paper describes an experiment which tests whether static type systems improve the maintainability of software systems. The results show rigorous empirical evidence that static type are indeed beneficial to these activities, except for fixing semantic errors.

Abstract Although the study of static and dynamic type systems plays a major role in research, relatively little is known about the impact of type systems on software development. Perhaps one of the more common arguments for static type systems in languages such as Java or C++ is that they require developers to annotate their code with type names, which is thus claimed to improve the documentation of software. In contrast, one common argument against static type systems is that they decrease flexibility, which may make them harder to use. While these arguments are found in the literature, rigorous empirical evidence is lacking. We report on a controlled experiment where 27 subjects performed programming tasks on an undocumented API with a static type system (requiring type annotations) as well as a dynamic type system (which does not). Our results show that for some tasks, programmers had faster completion times using a static type system, while for others, the opposite held. We conduct an exploratory study to try and theorize why.

Static type systems are essential in computer science. However, there is hardly any knowledge about the impact of type systems on the resulting piece of software. While there are authors that state that static types increase the development speed, other authors argue the other way around. A previous experiment suggests that there are multiple factors that play a role for a comparison of statically and dynamically typed language. As a follow-up, this paper presents an empirical study with 21 subjects that compares programming tasks performed in Java and Groovy - programming tasks where the number of expected type casts vary in the statically typed language. The result of the study is, that the dynamically typed group solved the complete programming tasks significantly faster for most tasks - but that for larger tasks with a higher number of type casts no significant difference could be found.

Aspect-oriented programming (AOP) is commonly assumed to be a technique which improves the resulting software with respect to modularity. However, previous empirical experiments suggest that AOP is with respect to development or maintenance time either a technique without a measurable benefit or a technique with a measurable negative effect. A possible reason why previous experiments were not able to show such a benefit is, that those experiments did not consider situations where AOP has its strength: situations where modules need to be frequently changed. In those situations AOP might be able to compensate a possible higher initial development effort. This paper describes an empirical, socio-technical study with Java and AspectJ where developers needed to perform changes on their code base multiple times. It shows that frequent changes in the crosscutting code which do not change the concern's underlying structure compensate an initial higher development time for those concerns. But it also shows that changes, which do alter the concern's structure again result in higher development times when using AOP.

Many language constructs have been brought forth by research in aspect-oriented software development which permit a succinct and abstract specification of join point selections (aka pointcuts). These language constructs are believed to improve the comprehensibility of the point cuts in comparison to their manually implemented counterparts. The case of comprehensibility gets undecided, though, if two notations permit to specify join point selection constraints in a likewise succinct and abstract manner. This paper reports on a controlled experiment which compares two notations to specify point cuts, i.e. Trace matches and Join Point Designation Diagrams, with respect to their ability to facilitate the comprehension of data constraints in join point selections. Two comprehension tasks are investigated on a basis of 28 point cuts in a three-factorial within-subject design with 35 participants. The experiment results show that JPDDs improve over Trace matches in most cases.

Although static type systems are an essential part in teach-ing and research in software engineering and computer science, there is hardly any knowledge about what the impact of static type systems on the development time or the resulting quality for a piece of software is. On the one hand there are authors that state that static type systems decrease an application's complexity and hence its development time (which means that the quality must be improved since developers have more time left in their projects). On the other hand there are authors that argue that static type systems increase development time (and hence decrease the code quality) since they restrict developers to express themselves in a desired way. This paper presents an empirical study with 49 subjects that studies the impact of a static type system for the development of a parser over 27 hours working time. In the experiments the existence of the static type system has neither a positive nor a negative impact on an application's development time (under the conditions of the experiment).

Research in the area of programming languages has different facets -- from formal reasoning about new programming language constructs (such as type soundness proofs for new type systems) over inventions of new abstractions, up to performance measurements of virtual machines. A closer look into the underlying research methods reveals a distressing characteristic of programming language research: developers, which are the main audience for new language constructs, are hardly considered in the research process. As a consequence, it is simply not possible to state whether a new construct that requires some kind of interaction with the developer has any positive impact on the construction of software. This paper argues for appropriate research methods in programming language research that rely on studies of developers -- and argues that the introduction of corresponding empirical methods not only requires a new understanding of research but also a different view on how to teach software science to students.

Aspect-oriented software development is an approach which addresses the construction of software artifacts that traditional software engineering constructs fail to modularize: the so-called crosscutting concerns. However, although aspect-orientation claims to permit a better modularization of crosscutting concerns, it is still not clear whether the development time for such crosscutting concerns is increased or decreased by the application of aspect-oriented techniques. This paper addresses this issue by an experiment which compares the development times of crosscutting concerns using traditional composition techniques and aspect-oriented composition techniques using the object-oriented programming language Java and the aspect-oriented programming language AspectJ. In that way, the experiment reveals opportunities and risks caused by aspect-oriented programming techniques in comparison to object-oriented ones.

The specification of join point selections (also known as "pointcuts") is a major design issue in Aspect-Oriented Software Development. Aspect-oriented systems generally provide specific language constructs (subsumed by the term "pointcut language") for specifying such a join point selection. Pointcut languages differ widely with respect to their syntax and semantics. Consequently, developers familiar with one specific language can hardly benefit from this knowledge when designing and implementing pointcuts in another language. This implies that developers working with different aspect-oriented languages can hardly communicate their design to each other, and knowledge about aspect-oriented design can hardly be transferred among developers developing in different languages. In order to overcome this problem, we present novel specification means based on the UML to represent diverse ways of join point selections — without relying on language-specific syntax and semantics. Instead, the proposed language constructs are able to express join point selections in a variety of different aspect-oriented programming languages.