The Standard Performance Evaluation Corporation (SPEC) CPU benchmark has been widely used as a measure of computing performance for decades. The SPEC is an industry-standardized, CPU-intensive benchmark suite and the collective data provide a proxy for the history of worldwide CPU and system performance. Past efforts have not provided or enabled answers to questions such as, how has the SPEC benchmark suite evolved empirically over time and what micro-architecture artifacts have had the most influence on performance?—have any micro-benchmarks within the suite had undue influence on the results and comparisons among the codes?—can the answers to these questions provide insights to the future of computer system performance? To answer these questions, we detail our historical and statistical analysis of specific hardware artifacts (clock frequencies, core counts, etc.) on the performance of the SPEC benchmarks since 1995. We discuss in detail several methods to normalize across benchmark evolutions. We perform both isolated and collective sensitivity analyses for various hardware artifacts and we identify one benchmark (libquantum) that had somewhat undue influence on performance outcomes. We also present the use of SPEC data to predict future performance.

Cybersecurity education has been emphasized by several national organizations in the United States, including the National Academy of Engineering, which recognizes securing cyberspace as one of the 14 Engineering Grand Challenges. To prepare students for such challenges and to enhance cybersecurity education opportunities at our large research university, we implemented an NSF-funded cybersecurity education project. This project is a collaborative effort between faculty and graduate students in the Engineering Education, Computer Science (CS) and Computer Engineering (CPE) departments at a major US research university. In this effort, we integrated cybersecurity learning modules into multiple existing core CS and CPE courses following Jerome Bruner's spiral-theory model, which has previously been used to reformulate several academic curricula. In this paper, we present our cybersecurity curriculum initiative, describe the spiral-theory based process we developed to implement the curriculum and provide an in-depth description of four reusable cybersecurity learning modules that we developed. A core tenet of spiral theory holds to revisit topics as students advance through their curriculum. This work applies this approach to Cybersecurity education by carefully designing the learning objectives of the modules and its contents. For evaluating these learning modules we implemented pre and post-tests to assess students' technical knowledge, their perceptions towards the modules' learning objectives, and how it influenced their motivation to learn cybersecurity. Our findings are overwhelmingly positive and the students' feedback has helped us improve these learning modules. Since its inception, our initiative has educated more than $2\,000$ students and is currently being used to revise the affected courses' syllabi.

Computer systems form the backbone of computing from very small, mobile devices to the huge datacenters that power the digital economy. These systems often exhibit large degrees of variability in their performance that is little understood, but such variability threatens to severely diminish the effectiveness of critical systems upon which society relies. Funded by a large NSF grant, the VarSys project at Virginia Tech researches the sources of variability in computer systems and develops methods to overcome it. We believe it is crucial to raise awareness of the phenomena surrounding variability in computer systems at the undergraduate level. Towards this end, we are connecting the research techniques developed as part of this NSF award to ongoing classroom projects in a core systems course. Our key insight is to expose students to the phenomenon as it occurs in the systems software modules (e.g. a memory allocator, a fork-join thread pool) they are themselves developing in the course. We have implemented a web-based system that allows students to submit their own systems-level code to a specialized cluster which then benchmarks it while systematically varying a number of ordinal and categorical variables. These variables reflect environmental factors that can influence the performance of complex systems. Students are then presented with a visual statistical analysis of the results and asked to interpret those results. We have successfully deployed this system in 2 semesters to over 250 students and collected student data about their experience with this system and are documenting our progress towards these important learning objectives.

The current emphasis on cybersecurity worldwide, demonstrates the importance of this topic. This poster describes a unique NSF funded project that aims to create cybersecurity education opportunities at Virginia Tech (VT). It is a collaborative effort among faculty and graduate students in the Engineering Education, Computer Science (CS), Electrical and Computer Engineering (includes two majors, Electrical Engineering (EE) and Computer Engineering (CPE)) Departments, and the Hume Center in the College of Engineering at VT. The goal is to integrate cybersecurity modules into eight required CS and CPE courses, from freshman to junior levels, utilizing Jeremy Bruner's spiral-theory-based1,2 curriculum model. A spiraling theme of "handling threats to software for securing information" is chosen that will be returned to repeatedly as learners advance in their knowledge and intellectual capacity. Cybersecurity goals of the Confidentiality, Integrity, Availability, Authenticity, Anonymity, Assurance (CIA/AAA) triad, as appropriate for various academic levels, are adopted to develop the cybersecurity modules. Each module engages students in an authentic activity that reinforces the cybersecurity concepts. The project includes an engineering education research component, which is focused on evaluating the effectiveness of the curriculum in enhancing students' knowledge, skills, and motivation in cybersecurity concepts. The first year of the project has been completed by introducing cybersecurity modules into four courses (CS: Introduction to Software Design, and Software Design and Data Structures, and CPE: Engineering Problem Solving with C++, and Data Structures and Algorithms) impacting ~1600 students. The details of curriculum development, implementation and, preliminary findings of the research will be presented.

Following an exhaustive set of experiments, we identify slowdowns in I/O performance that occur when processor power and frequency are increased. Our initial analyses indicate slowdowns are more likely to occur and more acute when the number of parallel I/O threads increases and the variability between runs is high. We use a micro benchmark-driven methodology to simplify isolation of the root causes of I/O performance loss. We classify the observed performance loss into two categories: file synchronization and file write delays. We introduce LUC, a runtime system to Limit the Unintended Consequences of power scaling and dynamically improve I/O performance. We demonstrate the effectiveness of the LUC system running on two platforms for two critical parallel transaction-oriented workloads including a mail server (vermeil) and online transaction processing (lotp).

Most operating systems provide protection and isolation to user processes, but not to critical system components such as device drivers or other system code. Consequently, failures in these components often lead to system failures. VirtuOS is an operating system that exploits a new method of decomposition to protect against such failures. VirtuOS exploits virtualization to isolate and protect vertical slices of existing OS kernels in separate service domains. Each service domain represents a partition of an existing kernel, which implements a subset of that kernel's functionality. Unlike competing solutions that merely isolate device drivers, or cannot protect from malicious and vulnerable code, VirtuOS provides full protection of isolated system components. VirtuOS's user library dispatches system calls directly to service domains using an exceptionless system call model, avoiding the cost of a system call trap in many cases.

Virtualization is a powerful technique used for variety of application domains, including emerging cloud environments that provide access to virtual machines as a service. Because of the interaction of virtual machines with multiple underlying software and hardware layers, the analysis of the performance of applications running in virtualized environments has been difficult. Moreover, performance analysis tools commonly used in native environments were not available in virtualized environments, a gap which our work closes.

Memory leaks are caused by software programs that prevent the reclamation of memory that is no longer in use. They can cause significant slowdowns, exhaustion of available storage space and, eventually, application crashes. Detecting memory leaks is challenging because real-world applications are built on multiple layers of software frameworks, making it difficult for a developer to know whether observed references to objects are legitimate or the cause of a leak. We present a graph mining solution to this problem wherein we analyze heap dumps to automatically identify subgraphs which could represent potential memory leak sources. Although heap dumps are commonly analyzed in existing heap profiling tools, our work is the first to apply a graph grammar mining solution to this problem. Unlike classical graph mining work, we show that it suffices to mine the dominator tree of the heap dump, which is significantly smaller than the underlying graph. Our approach identifies not just leaking candidates and their structure, but also provides aggregate information about the access path to the leaks. We demonstrate several synthetic as well as real-world examples of heap dumps for which our approach provides more insight into the problem than state-of-the-art tools such as Eclipse's MAT.

Pattern-based Representation (PBR) is a novel approach to improving the performance of Sparse Matrix-Vector Multiply (SMVM) numerical kernels. Motivated by our observation that many matrices can be divided into blocks that share a small number of distinct patterns, we generate custom multiplication kernels for frequently recurring block patterns. The resulting reduction in index overhead significantly reduces memory bandwidth requirements and improves performance. Unlike existing methods, PBR requires neither detection of dense blocks nor zero filling, making it particularly advantageous for matrices that lack dense nonzero concentrations. SMVM kernels for PBR can benefit from explicit prefetching and vectorization, and are amenable to parallelization. We present sequential and parallel performance results for PBR on two current multicore architectures, which show that PBR outperforms available alternatives for the matrices to which it is applicable.

Pintos is an instructional operating system, complete with documentation and ready-made, modular projects that introduce students to the principles of multi-programming, scheduling, virtual memory, and filesystems. By allowing students to run their work product on actual hardware, while simultaneously benefiting from debugging and dynamic analysis tools provided in simulated and emulated environments, Pintos increases student engagement. Unlike tailored versions of commercial or open source OS such as Linux, Pintos is designed from the ground up from an educational perspective. It has been used by multiple institutions for a number of years and is available for wider use.

VT-ASOS is a framework for holistic and continuous customization of system software on HPC systems. The framework leverages paravirtualization technology. VT-ASOS extends the Xen hypervisor with interfaces, mechanisms, and policies for supporting application-specific resource management schemes on many-core systems, while retaining the advantages of virtualization, including protection, performance isolation, and fault tolerance. We outline the VTASOS framework and present results from a preliminary prototype, which enables static customization of scheduler parameters and runtime adaptation of parallel virtual machines.

Iterative solutions of sparse problems often achieve only a small fraction of the peak theoretical performance on modern architectures. This problem is highly challenging because sparse matrix storage schemes require data to be accessed irregularly, which leads to massive cache misses. Furthermore, the inner loop of typical sparse matrix operations accesses only a small and variable amount of data, which not only leads to low utilization of floating point registers, but also prevents optimization techniques that improve instruction level parallelism (ILP), such as unroll and jam. Although a general solution to this problem has not been found, significant performance improvements can be made for at least one important special case, namely large ensemble computations, which run the same application repeatedly on different data sets. In this paper, we present the Operation Stacking Framework (OSF), which runs multiple sparse problems simultaneously, stacking their data and solving them as one, thus improving both cache and ILP utilization. Programmers can use stacked solvers transparently in their applications. Moreover, OSF provides an API that makes it simple to convert existing solvers such as the conjugate gradient (CG) and generalized minimal residual (GMRES) methods into a stacked form. Our experimental results show that stacking can reduce the number of L2 misses by 25% to 44%, resulting in performance improvements of up to 1.95x with an average of 1.60x for stacked CG and GMRES algorithms on a single CPU.

Convenience, reliability, and effectiveness of automatic memory management have long been established in modern systems and programming languages such as Java. The timeliness requirements of real-time systems, however, impose specific demands on the operational parameters of the garbage collector. The memory requirements of real-time tasks must be accommodated with a predictable impact on the timeline, and under the purview of the scheduler. Utility accrual is a method of dynamic overload scheduling that is designed to respond to CPU overload conditions by producing a schedule that heuristically maximize a predefined metric of utility. There also exists in such systems the possibility of memory overload situations in which the cumulative memory demand exceeds the amount of memory available. This paper presents a utility accrual algorithm for uniprocessor CPU and garbage collection scheduling that addresses memory overload conditions. By tightly linking CPU and memory allocation, the scheduler can appropriately respond to overload along both dimensions. This scheduler is the first of its kind to enable the use of automatic memory management in a utility accrual system. Experimental results using actual Java application profiles indicate the viability of this model

Single-language runtime systems, in the form of Java virtual machines, are widely deployed platforms for executing untrusted mobile code. These runtimes provide some of the features that operating systems provide: interapplication memory protection and basic system services. They do not, however, provide the ability to isolate applications from each other. Neither do they provide the ability to limit the resource consumption of applications. Consequently, the performance of current systems degrades severely in the presence of malicious or buggy code that exhibits ill-behaved resource usage. We show that Java runtime systems can be extended to support processes , and that processes can provide robust and efficient support for untrusted applications.We have designed and built KaffeOS, a Java runtime system that provides support for processes. KaffeOS isolates processes and manages the physical resources available to them: CPU and memory. Unlike existing Java virtual machines, KaffeOS can safely terminate processes without adversely affecting the integrity of the system, and it can fully reclaim a terminated process's resources. Finally, KaffeOS requires no changes to the Java language. The novel aspects of the KaffeOS architecture include the application of a user/kernel boundary as a structuring principle for runtime systems, the employment of garbage collection techniques for resource management and isolation, and a model for direct sharing of objects between untrusted applications. The difficulty in designing KaffeOS lay in balancing the goals of isolation and resource management against the goal of allowing direct sharing of objects.For the SpecJVM benchmarks, the overhead that our KaffeOS prototype incurs ranges from 0% to 25%, when compared to the open-source JVM on which it is based. We consider this overhead acceptable for the safety that KaffeOS provides. In addition, our KaffeOS prototype can scale to run more applications than running multiple JVMs. Finally, in the presence of malicious or buggy code that engages in a denial-of-service attack, KaffeOS can contain the attack, remove resources from the attacked applications, and continue to provide robust service to other clients.

In this paper, we examine the energy consumption of a state-of-the-art pocket computer. Using a data acquisition system, we measure the energy consumption of the Itsy Pocket Computer, developed by Compaq Computer Corporation's Palo Alto Research Labs. We begin by showing that the energy usage characteristics of the Itsy differ markedly from that of a notebook computer. Then, since we expect that flexible software environments will become increasingly prevalent on pocket computers, we consider applications running in a Java environment. In particular, we explain some of the Java design tradeoffs applicable to pocket computers, and quantify their energy costs. For the design options we considered and the three workloads we studied, we find a maximum change in energy use of 25%.

Implementing new operating systems is tedious, costly, and often impractical except for large projects.The Flux OSKit addresses this problem in a novel way by providing clean, well-documented OS components designed to be reused in a wide variety of other environments, rather than defining a new OS structure.The OSKit uses unconventional techniques to maximize its usefulness, such as intentionally exposing implementation details and platform-specific facilities.Further, the OSKit demonstrates a technique that allows unmodified code from existing mature operating systems to be incorporated quickly and updated regularly, by wrapping it with a small amount of carefully designed "glue" code to isolate its dependencies and export wellrdefined interfaces.The OSKit uses this technique to incorporate over 230,000 lines of stable code including device drivers, file systems, and network protocols.Our experience demonstrates that this approach to component software structure and reuse has a surprisingly large impact in the OS implementation domain.Four real-world examples show how the OSKit is catalyzing research and development in operating systems and programming languages.

No abstract available.