Contains fulltext : 333701.pdf (Publisher’s version ) (Open Access)

Code comprehension has been shown to be challenging and important for a positive learning outcome. Students don't always understand the code they write. This has been exacerbated by the advent of large language models that automatically generate code that may or may not be correct. Now students don't just have to understand their own code, but they have to be able to critically analyze automatically generated code as well. To help students with code comprehension, instructors often use notional machines. Notional machines are used not only by instructors to explain code, but also in activities or exam questions given to students. Traditionally, these questions involve code that was not written by students. However, asking questions to students about their own code (Questions on Learners' Code, QLCs) has been shown to strengthen their code comprehension. This poster presents an approach to combine notional machines and QLCs to automatically generate personalized questions about learners' code based on notional machines. Our aim is to understand whether notional machine-based QLCs are effective. We conducted a pilot study with 67 students to test our approach, and we plan to conduct a comprehensive empirical evaluation to study its effectiveness.

Programming is increasingly being taught in high schools and in non-CS majors at universities. However, the interests of this wide population are often different from those of students who decided to study computer science. Programming graphics is a great way to cater to these interests and motivate students to learn programming with examples that go beyond the more traditional "Hello World" or mathematical exercises. In this workshop, we demonstrate how to use PyTamaro, a Python library designed to teach programming with graphics to novices. PyTamaro's design removes the need to use and explain sophisticated programming language features and directs learners' attention towards fundamental concepts of programming. The workshop combines unplugged activities and Python programming. Graphics are first created using colored paper cutouts and cards that represent PyTamaro functions; then, the composition is written as a Python program. The programming part will be carried out on the PyTamaro web platform, which also contains more than a hundred activities of varying levels of difficulty and on different themes that instructors can directly use or take inspiration from.

Knowledge about misconceptions is an important element of pedagogical content knowledge. The computing education research community collected a large body of research on misconceptions, using a diverse set of definitions and approaches. Inspired by this prior work, we present an actionable definition of misconceptions, focused on the area most commonly studied: programming and programming languages. We then introduce an organizational structure for collections of programming language misconceptions. We study how existing collections fit our organization, and we present a curated inventory of programming language misconceptions that aims to follow our definition and structure. Our inventory goes beyond traditional programming misconception collections. It connects misconceptions to the authoritative specifications of languages, to places they may be triggered in textbooks, to research papers that discuss them, and it provides support for integrating programming language misconceptions into educational platforms.

To understand, debug, and predict the performance of complex software systems, we develop the concept of probabilistic performance annotations. In essence, we annotate components (e.g., methods) with a relation between a measurable performance metric, such as running time, and one or more features of the input or the state of that component. We use two forms of regression analysis: regression trees and mixture models. Such relations can capture non-trivial behaviors beyond the more classic algorithmic complexity of a component. We present a method to derive such annotations automatically by generalizing observed measurements. We illustrate the use of our approach on three complex systems---the ownCloud distributed storage service; the MySQL database system; and the x264 video encoder library and application---producing non-trivial characterizations of the performance. Notably, we isolate a performance regression and identify the root cause of a second performance bug in MySQL.

A notional machine is a pedagogic device to assist the understanding of some aspect of programs or programming. It is typically used to support explaining a programming construct, or the user-understandable semantics of a program. For example, a variable is like a box with a label, and assignment copies or moves a value into that box. This working group will capture examples of notional machines from actual pedagogical practice, as expressed in textbooks (or other teaching materials) or used in the classroom. We will interview at least 30 teachers about their experience with, and perceptions of, the use of notional machines in teaching. Using the interviews, we will work on devising and refining a form to characterise essential features of notional machines. We will also attempt to relate them to each other to describe potential learning sequences or progressions. The working group report will contain descriptions of notional machines used at different levels in education, in different countries, by many teachers. Capturing and Characterising Notional Machines Sally Fincher, Johan Jeuring, Craig S Miller Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). ITiCSE 2020,,Trondheim, Norway © 2020 Copyright held by the owner/author(s). 978-1-4503-0000-0/18/06...$15.00 https://doi.org/10.1145/1234567890 The resulting catalogue of notional machines will allow a teacher to select a machine for a particular use, permit comparison between them, and provide a starting point for further categorization and analysis of notional machines. Additionally, we will make more theoretical explorations. We will explore a variety of presentational formats, examining what is necessary and what superfluous; we will look for dimensions of comparison and will examine how notional machines are instantiated across the discipline. We argue that the creation and use of notional machines is potentially a signature pedagogy for computing [1] and that creating and using notional machines represents a certain level of pedagogic sophistication that might be an indicator of pedagogic content knowledge (PCK).

The main goal of a static type system is to prevent certain kinds of errors from happening at run time. A type system is formulated as a set of constraints that gives any expression or term in a program a well-defined type. Yet mainstream programming languages are endowed with type systems that provide the means to circumvent their constraints through casting. We want to understand how and when developers escape the static type system to use dynamic typing. We empirically study how casting is used by developers in more than seven thousand Java projects. We find that casts are widely used (8.7% of methods contain at least one cast) and that 50% of casts we inspected are not guarded locally to ensure against potential run-time errors. To help us better categorize use cases and thus understand how casts are used in practice, we identify 25 cast-usage patterns---recurrent programming idioms using casts to solve a specific issue. This knowledge can be: (a) a recommendation for current and future language designers to make informed decisions (b) a reference for tool builders, e.g., by providing more precise or new refactoring analyses, (c) a guide for researchers to test new language features, or to carry out controlled programming experiments, and (d) a guide for developers for better practices.

Developers of performance sensitive production software are in a dilemma: performance regression tests are too costly to run at each commit, but skipping the tests delays and complicates performance regression detection. Ideally, developers would have a system that predicts whether a given commit is likely to impact performance and suggests which tests to run to detect a potential performance regression. Prior approaches towards this problem require static or dynamic analyses that limit their generality and applicability. This paper presents an approach that is simple and general, and that works surprisingly well for real applications.

Program comprehension tools are often developed for a specific programming language. Developing such a tool from scratch requires significant effort. In this paper, we report on our experience developing a language-independent framework that enables the creation of program comprehension tools, specifically tools gathering insight from deep dynamic analysis, with little effort. Our framework is language independent, because it is built on top of Truffle, an open-source platform, developed in Oracle Labs, for implementing dynamic languages in the form of AST interpreters. Our framework supports the creation of a diverse variety of program comprehension techniques, such as query, program slicing, and back-in-time debugging, because it is centered around a powerful information-flow tracking engine. Tools developed with our framework get access to the information-flow through a program execution. While it is possible to develop similarly powerful tools without our framework, for example by tracking information-flow through bytecode instrumentation, our approach leads to information that is closer to source code constructs, thus more comprehensible by the user. To demonstrate the effectiveness of our framework, we applied it to two of Truffle-based languages namely Simple Language and TruffleRuby, and we distill our experience into guidelines for developers of other Truffle-based languages who want to develop program comprehension tools for their language.

Learning a programming language is hard. Students need to acquire three types of skills: (1) understand the new language concepts, (2) interpret pieces of code that use those concepts, and (3) write pieces of code involving those concepts. In this paper, we present an approach to help with the second type of skill. There are tools that explain/visualize the execution by stepping through the code. However, such tools suffer from two types of problems. First, the granularity of the stepping is coarse, hiding the intermediate steps in evaluating expressions. Second, for a single statement corresponding to a step of the execution, the order of the evaluation of source code constructs is not aligned with the order the constructs appear in the source code. To fix those, we combine compile-time with run-time information to automatically produce intuitive explanations of code. At compile time, we generate the explanations by traversing the AST. The runtime information provides the execution map and runtime values. We applied our idea to the Java version of Online Python Tutor, a web-based program visualization tool. Each explanation is complemented by highlighting the piece of the source code to which it corresponds, being spoken by the system, and a tree structure visualizing the evaluation of the involved expressions. The fact that the generation of explanations is syntax driven makes the result close to what a human tutor would provide.

Given different ways to fix a failure in a program run, you may want to fix it such that future runs of the same program with other inputs do not show any effect of that fixed bug. We present InfectoMeter, a tool which provides a map of the source code for a programmer such that each statement in the source code is colored based on the impact it has on the rest of the execution. A darker line represents the point where the fix can have a higher impact, but it does not say anything about where the bug might happen. The strategy is inspired from the notion of infection in medical science. Given an infection, it can propagate throughout the whole system and affect different parts. So to fix the system, one needs to focus on the source of infection, rather than a specific element of the system. InfectoMeter implements this idea. It takes as input a unit test and provides colored source code, such that each source line's color represents the impact it has on the rest of the execution.

Authentic problem solving tasks in digital environments are often open-ended with ill-defined pathways to a goal state. Scaffolds and formative feedback during this process help learners develop the requisite skills and understanding, but require assessing the problem-solving process. This paper describes a hybrid approach to assessing process at scale in the context of the use of computational thinking practices during programming. Our approach combines hypothesis-driven analysis, using an evidence-centered design framework, with discovery-driven data analytics. We report on work-in-progress involving novices and expert programmers working on Blockly games.

Heterogeneous programming complicates software development. We present CLOP, a platform that embeds code targeting heterogeneous compute devices in a convenient and clean way, allowing unobstructed data flow between the host code and the devices, reducing the amount of source code by an order of magnitude. The CLOP compiler uses the standard facilities of the D programming language to generate code strictly at compile-time. In this paper we describe the CLOP language and the CLOP compiler implementation.

Java is a safe language. Its runtime environment provides strong safety guarantees that any Java application can rely on. Or so we think. We show that the runtime actually does not provide these guarantees---for a large fraction of today's Java code. Unbeknownst to many application developers, the Java runtime includes a "backdoor" that allows expert library and framework developers to circumvent Java's safety guarantees. This backdoor is there by design, and is well known to experts, as it enables them to write high-performance "systems-level" code in Java. For much the same reasons that safe languages are preferred over unsafe languages, these powerful---but unsafe---capabilities in Java should be restricted. They should be made safe by changing the language, the runtime system, or the libraries. At the very least, their use should be restricted. This paper is a step in that direction. We analyzed 74 GB of compiled Java code, spread over 86,479 Java archives, to determine how Java's unsafe capabilities are used in real-world libraries and applications. We found that 25% of Java bytecode archives depend on unsafe third-party Java code, and thus Java's safety guarantees cannot be trusted. We identify 14 different usage patterns of Java's unsafe capabilities, and we provide supporting evidence for why real-world code needs these capabilities. Our long-term goal is to provide a foundation for the design of new language features to regain safety in Java.

Geometry-related software is increasingly important in computational science and visual computing. Engineering such software is particularly challenging due to the size and complexity of the data it operates on. In this paper we present VESTIGE, a framework that employs visualization to address that challenge. VESTIGE targets four software engineering activities: (1) visualization-guided development, (2) monitoring and bug detection, (3) test oracle generation, and (4) debugging. We present five scenarios from our real-life experience as developers of geometry-related software that show how VESTIGE helps to improve the software development process. Integrating VESTIGE into the development workflow takes little effort and can have significant benefits.

Research has shown that correctly conducting and analysing computer performance experiments is difficult. This paper investigates what is necessary to conduct successful computer performance evaluation by attempting to repeat a prior experiment: the comparison between two Linux schedulers.

In this paper we compare different approaches to maintain shadow state for heap objects in Java. We identify dynamic analyses that need to shadow heap objects, and we consider their requirements. We describe three very different approaches in detail: using a hash map, heap tagging, and injecting shadow fields.

Traditional profilers identify where a program spends most of its resources. They do not provide information about why the program spends those resources or about how resource consumption would change for different program inputs. In this paper we introduce the idea of algorithmic profiling. While a traditional profiler determines a set of measured cost values, an algorithmic profiler determines a cost function. It does that by automatically determining the "inputs" of a program, by measuring the program's "cost" for any given input, and by inferring an empirical cost function.

Profilers help developers to find and fix performance problems. But do they find performance bugs -- performance problems that real users actually notice? In this paper we argue that -- especially in the case of interactive applications -- traditional profilers find irrelevant problems but fail to find relevant bugs.

Abstract The performance of a program often varies significantly over the course of the program's run. Thus, to understand the performance of a program it is valuable to look not just at end‐to‐end metrics (e.g. total number of cache misses) but also the time‐varying performance of the program. Unfortunately, analyzing time‐varying performance is both cumbersome and difficult. This paper makes three contributions, all geared toward helping others in working with traces. First, it describes a system, the TraceAnalyzer, designed specifically for working with performance traces; a performance trace captures the time‐varying performance of a program run. Second, it describes lessons that we have learned from many years of working with these traces. Finally, it uses a case study to demonstrate how we have used the TraceAnalyzer to understand a performance anomaly. Copyright © 2010 John Wiley & Sons, Ltd.

Many computer systems are interactive in some way, that means they are used by human users. A human user perceives the performance of a computer system primarily in terms of its response time. If a system does not respond to a user's input, such as a key press, a mouse motion, or a gesture on a touch screen, within roughly 100 ms, a user perceives the system as sluggish. Developers of interactive software and systems thus are interested in keeping response times below this perceptibility threshold. Existing tools allow the measurement of interactive response times, and the tracing of application behavior. In this paper we present a tool, LagAlyzer, which analyzes and visualizes the information gathered by such latency measurement tools. LagAlyzer enables the characterization of perceptible lag, for example by quantifying to what degree perceptible performance was caused by synchronization bottlenecks, by garbage collection, by the runtime libraries, or by the application. We use LagAlyzer to characterize the perceptible latency found in commonly used interactive Java applications. We believe that this is the first study giving insight into why interactive Java applications sometimes are perceived as sluggish.

When designers of Java runtime systems evaluate the performance of their systems for the purpose of running clientside Java applications, they normally use the Dacapo and SPEC JVM benchmark suites. However, when users of those Java runtime systems run client applications, they usually run interactive applications such as Eclipse or NetBeans. In this paper we study whether this mismatch is a problem: Do the prevalent Java client-side benchmark suites faithfully represent the characteristics of real-world Java client applications? To answer this question we characterize benchmarks and applications using three kinds of metrics: static metrics, architecture-independent dynamic metrics, and hardware performance counters. We find that real-world applications significantly differ from existing benchmarks. Our finding indicates that the current benchmark suites should be augmented to more faithfully represent the large segment of interactive applications.

Performance analysts profile their programs to find methods that are worth optimizing: the "hot" methods. This paper shows that four commonly-used Java profilers (xprof , hprof , jprofile, and yourkit) often disagree on the identity of the hot methods. If two profilers disagree, at least one must be incorrect. Thus, there is a good chance that a profiler will mislead a performance analyst into wasting time optimizing a cold method with little or no performance improvement.

Abstract Modern systems are enormously complex; many applications today comprise millions of lines of code, make extensive use of software frameworks, and run on complex, multi‐tiered, run‐time systems. Understanding the performance of these applications is challenging because it depends on the interactions between the many software and the hardware components. This paper describes and evaluates an interactive and iterative methodology, temporal vertical profiling, for understanding the performance of applications. There are two key insights behind temporal vertical profiling. First, we need to collect and reason across information from multiple layers of the system before we can understand an application's performance. Second, application performance changes over time and thus we must consider the time‐varying behavior of the application instead of aggregate statistics. We have developed temporal vertical profiling from our own experience of analyzing performance anomalies and have found it very helpful for methodically exploring the space of hardware and software components. By representing an application's behavior as a set of metrics, where each metric is represented as a time series, temporal vertical profiling provides a way to reason about performance across system layers, regardless of their level of abstraction, and independent of their semantics. Temporal vertical profiling provides a methodology to explore a large space of metrics, hundreds of metrics even for small benchmarks, in a systematic way. Copyright © 2010 John Wiley & Sons, Ltd.

This paper presents a surprising result: changing a seemingly innocuous aspect of an experimental setup can cause a systems researcher to draw wrong conclusions from an experiment. What appears to be an innocuous aspect in the experimental setup may in fact introduce a significant bias in an evaluation. This phenomenon is called measurement bias in the natural and social sciences.

Many experimental performance evaluations depend on accurate measurements of the cost of executing a piece of code. Often these measurements are conducted using infrastructures to access hardware performance counters. Most modern processors provide such counters to count micro-architectural events such as retired instructions or clock cycles. These counters can be difficult to configure, may not be programmable or readable from user-level code, and can not discriminate between events caused by different software threads. Various software infrastructures address this problem, providing access to per-thread counters from application code. This paper constitutes the first comparative study of the accuracy of three commonly used measurement infrastructures (perfctr, perfmon2, and PAPI) on three common processors (Pentium D, Core 2 Duo, and AMD ATHLON 64 X2).

We show two severe problems with the state of the art in empirical computer system performance evaluation, observer effect and measurement context bias, and we outline the path toward a solution.

Performance analysts commonly use trace-based data containing hardware and software metrics to understand performance. The trace data is generated by instrumenting the code to increment a counter when an event occurs and to collect hardware and software metrics in a trace. Unfortunately, the act of collecting a trace can perturb the behavior that the trace is frying to capture. In this paper, we gain an understanding of perturbation due to measurement instrumentation of the system. We identify two mechanisms to quantify perturbation: inner and outer perturbation. Using inner perturbation, a performance analyst can determine when a run is perturbed by collecting too much information. Using outer perturbation, the performance analyst can determine if she can use the data from multiple runs as if the data were all from a single run. Our evaluation of these mechanisms lead to two results. First, we are surprised to find that even with minimal instrumentation overhead, which increased instructions executed by less than 3%, high perturbation resulted, which prevented one from correctly reasoning about metrics within a trace or across traces. Second, the instrumentation of different software metrics interact in subtle, and not always obvious, ways making the impact of instrumentation on perturbation difficult, if not impossible, to predict. Finally, we outline a methodology for collecting data while avoiding perturbation. When inner perturbation occurs, the performance analyst can spread out the data collection over multiple runs. When outer perturbation occurs, she can try different strategies for spreading out the data collection over multiple runs.

The performance of computer systems varies over the course of their execution. A system may perform well during some parts of its execution and poorly during others. To understand why a system behaves in this way performance analysts need to study its time-varying behavior. Fortunately, modern microprocessors support hardware performance monitors which enable performance analysts to collect time-varying metrics with relative ease. Unfortunately, even though modern microprocessors can collect hundreds of metrics, they can collect only a few of these metrics simultaneously. Prior work has proposed time-interpolation techniques for circumventing this limitation. Time interpolation collects different metrics at different points in time, either within the same trace (multiplexing) or in different traces (trace alignment), and interpolates the results to allow reasoning across all metrics at the same points in time. This paper introduces and uses a novel approach for evaluating time interpolation techniques. This evaluation leads to insights that improve both multiplexing and trace-alignment. Specifically, this paper (i) improves the effectiveness and applicability of the best performing trace alignment technique in prior work; and (ii) introduces criteria that performance analysts can use to determine whether or not to trust multiplexing or trace alignment results for their particular situation. Finally, this paper evaluates time interpolation techniques by exploring their performance in a wide variety of situations and on programs written in two different programming languages, C and Java, and on two different architectures, Pentium 4 and POWER4.

For many performance analysis problems, the ability to reason across traces is invaluable. However, due to non-determinism in the OS and virtual machines, even two identical runs of an application yield slightly different traces. For example, it is unlikely that two identical runs of an application will suffer context switches at exactly the same points. These sorts of variations across traces make it difficult to reason across traces. This paper describes and evaluates an algorithm, dynamic time warping (DTW) that can be used to align traces, thus enabling us to reason across traces. While DTW comes from prior work our use of DTW is novel. Also we describe and evaluate an enhancement to DTW that significantly improves the quality of its alignments. Our results show that for applications whose performance varies significantly over time, DTW does a great job at aligning the traces. For applications whose performance stays largely constant for significant periods of time, the original DTW does not perform well; however, our enhanced DTW performs much better.

Real-time systems have reached a level of complexity beyond the scaling capability of the low-level or restricted languages traditionally used for real-time programming.While Metronome garbage collection has made it practical to use Java to implement real-time systems, many challenges remain for the construction of complex real-time systems, some specific to the use of Java and others simply due to the change in scale of such systems.The goal of our current research is the creation of a comprehensive Java-based programming environment and methodology for the creation of complex real-time systems. Our goals include construction of a provably correct real-time garbage collector capable of providing worst case latencies of 100 μs, capable of scaling from sensor nodes up to large multiprocessors; specialized programming constructs that retain the safety and simplicity of Java, and yet provide sub-microsecond latencies; the extension of Java's "write once, run anywhere" principle from functional correctness to timing behavior; on-line analysis and visualization that aids in the understanding of complex behaviors; and a principled probabilistic analysis methodology for bounding the behavior of the resulting systems.While much remains to be done, this paper describes the progress we have made towards these goals.

Last year at OOPSLA we presented a methodology, vertical profiling, for understanding the performance of object-oriented programs. The key insight behind this methodology is that modern programs run on top of many layers (virtual machine, middleware, etc) and thus we need to collect and combine information from all layers in order to understand system performance. Although our methodology was able to explain previously unexplained performance phenomena, it was extremely labor intensive. In this paper we describe and evaluate techniques for automating two significant activities of vertical profiling: trace alignment and correlation. Trace alignment aligns traces obtained from separate runs so that one can reason across the traces. We are not aware of any prior approach that effectively and automatically aligns traces. Correlation sifts through hundreds of metrics to find ones that have a bearing on a performance anomaly of interest. In prior work we found that statistical correlation was only sometimes effective. We have identified highly-effective approaches for both activities.For aligning traces we explore dynamic time warping, and for correlation we explore eight correlators based on statistical correlation, distance measures, and piecewise linear segmentation. Although we explore these activities in the context of vertical profiling, both activities are widely applicable in the performance analysis area.

Sampling has been successfully used to identify performance optimization opportunities. We would like to apply similar techniques to check program correctness. Unfortunately, sampling provides poor coverage of infrequently executed code, where bugs often lurk. We describe an adaptive profiling scheme that addresses this by sampling executions of code segments at a rate inversely proportional to their execution frequency. To validate our ideas, we have implemented SWAT, a novel memory leak detection tool. SWAT traces program allocations/ frees to construct a heap model and uses our adaptive profiling infrastructure to monitor loads/stores to these objects with low overhead. SWAT reports 'stale' objects that have not been accessed for a 'long' time as leaks. This allows it to find all leaks that manifest during the current program execution. Since SWAT has low runtime overhead (‹5%), and low space overhead (‹10% in most cases and often less than 5%), it can be used to track leaks in production code that take days to manifest. In addition to identifying the allocations that leak memory, SWAT exposes where the program last accessed the leaked data, which facilitates debugging and fixing the leak. SWAT has been used by several product groups at Microsoft for the past 18 months and has proved effective at detecting leaks with a low false positive rate (‹10%).

Object-oriented programming languages provide a rich set of features that provide significant software engineering benefits. The increased productivity provided by these features comes at a justifiable cost in a more sophisticated runtime system whose responsibility is to implement these features efficiently. However, the virtualization introduced by this sophistication provides a significant challenge to understanding complete system performance, not found in traditionally compiled languages, such as C or C++. Thus, understanding system performance of such a system requires profiling that spans all levels of the execution stack, such as the hardware, operating system, virtual machine, and application.

While caches are effective at avoiding most main-memory accesses, the few remaining memory references are still expensive. Even one cache miss per one hundred accesses can double a program's execution time. To better tolerate the data-cache miss latency, architects have proposed various speculation mechanisms, including load-value prediction. A load-value predictor guesses the result of a load so that the dependent instructions can immediately proceed without having to wait for the memory access to complete. To use the prediction resources most effectively, speculation should be restricted to loads that are likely to miss in the cache and that are likely to be predicted correctly. Prior work has considered hardware- and profile-based methods to make these decisions. Our work focuses on making these decisions at compile time. We show that a simple compiler classification is effective at separating the loads that should be speculated from the loads that should not. We present results for a number of C and Java programs and demonstrate that our results are consistent across programming languages and across program inputs.