Today cloud companies offer fully managed Spark services. This has made it easy to onboard new customers but has also increased the volume of users and their workload sizes. However, both cloud providers and users lack the tools and time to optimize these massive workloads. To solve this problem, we designed SparkCruise that can help understand and optimize workload instances by adding a workload-driven feedback loop to the Spark query optimizer. In this paper, we present our approach to collecting and representing Spark query workloads and use it to improve the overall performance on the workload, all without requiring any access to user data. These methods scale with the number of workloads and apply learned feedback in an online fashion. We explain one specific workload optimization developed for computation reuse. We also share the detailed analysis of production Spark workloads and contrast them with the corresponding analysis of TPC-DS benchmark. To the best of our knowledge, this is the first study to share the analysis of large-scale production Spark SQL workloads.

Orchestrating data transfers between CPUs and a coprocessor manually is cumbersome, particularly for multi-dimensional arrays and other data structures with multi-level pointers, which are common in scientific computations. This work describes a system that includes both compile-time and runtime solutions for this problem, with the overarching goal of improving programmer productivity while maintaining performance.

Applications executing on multicore processors can now easily offload computations to many core processors, such as Intel Xeon Phi coprocessors. However, it requires high levels of expertise and effort to tune such offloaded applications to realize high-performance execution. Previous efforts have focused on optimizing the execution of offloaded computations on many core processors. However, we observe that the data transfer overhead between multicore and many core processors, and the limited device memories of many core processors often constrain the performance gains that are possible by offloading computations. In this paper, we present three source-to-source compiler optimizations that can significantly improve the performance of applications that offload computations to many core processors. The first optimization automatically transforms offloaded codes to enable data streaming, which overlaps data transfer between multicore and many core processors with computations on these processors to hide data transfer overhead. This optimization is also designed to minimize the memory usage on many core processors, while achieving the optimal performance. The second compiler optimization re-orders computations to regularize irregular memory accesses. It enables data streaming and factorization on many core processors, even when the memory access patterns in the original source codes are irregular. Finally, our new shared memory mechanism provides efficient support for transferring large pointer-based data structures between hosts and many core processors. Our evaluation shows that the proposed compiler optimizations benefit 9 out of 12 benchmarks. Compared with simply offloading the original parallel implementations of these benchmarks, we can achieve 1.16x-52.21x speedups.

Work division between the processor and accelerator is a common theme in modern heterogenous computing. Recent efforts (such as LEO and OpenAcc) provide directives that allow the developer to mark code regions in the original application from which offloadable tasks can be generated by the compiler. Auto-tuners and runtime schedulers work with the options (i.e., offloadable tasks) generated at compile time, which is limited by the directives specified by the developer. There is no provision for offload restructuring.

MapReduce has emerged as one of the most popular programming models for data parallel enterprise applications. Despite advances in runtime, the opportunities for optimizing MapReduce applications remain largely unexplored. In this paper, we present a framework for performing holistic compiler optimizations on legacy MapReduce applications. We have identified and implemented two optimizations and evaluated them with a set of Hadoop applications on a cluster of Xeon servers. Our experiments show that performance gains of more than 3X can be achieved without user involvement.

Intel MIC (Many Integrated Core) is the first x86-based coprocessor architecture aimed at accelerating multi-core HPC applications. In the most common usage model, parallel code sections are offloaded to the MIC coprocessor using LEO (Language Extensions for Offload). The developer is responsible for identifying and specifying offloadable code regions, managing data transfers between the CPU and MIC and optimizing the application for performance, which requires some amount of effort and experimentation. In this paper, we present Apricot, an optimizing compiler and productivity tool for x86-compatible many-core coprocessors (such as Intel MIC) that minimizes developer effort by (i) automatically inserting LEO clauses for parallelizable code regions, (ii) selectively offloading some of the code regions to the coprocessor at runtime based on a cost model that we have developed, (iii) applying a set ofoptimizations for minimizing the data communication overhead and improving overall performance. Apricot is intended to assist programmers in porting existing multi-core applications and writing new ones to take advantage of the many-core coprocessor, while maximizing overall performance. Experiments with SpecOMP and NAS Parallel benchmarks show that Apricot can successfully transform OpenMP applications to run on the MIC coprocessor with good performance gains.

In this paper, we propose a system for context- aware battery management that warns the user when it detects that the phone battery can run out before the next charging opportunity is encountered. At the heart of this system, are algorithms that predict: (1) when the next charging opportunity will be available, (2) how much call-time will be required by the user in the interim, and (3) how long the battery will last if the current set of applications continue to execute. We propose algorithms that process user's location traces and call-logs for making some of these predictions. We also propose a technique to predict battery consumption of applications. We present the design of the system and demonstrate its feasibility by experimentally showing that each of the prediction algorithms can perform with fairly high accuracy.

This paper presents a framework for preserving location privacy without affecting location accuracy. In this framework, services migrate a piece of code to a trusted server, which is assumed to have location information of all the interesting subjects. The code executes on the trusted server, reads location information and sends back results. We introduce Non-inference, a novel information-flow control model that guarantees that the code does not leak exact location information. We discuss the design, implementation and evaluation of a static program analysis technique that enforces non-inference for location based services

In this paper we introduce the inverted browser, a novel approach to enable mobile users to view content from their personal devices on public displays. The inverted browser is a network service to start and control a browser that is then used to view the content. In contrast to a traditional Web browser, which runs on the client device and pulls content from a server, content is pushed to the inverted browser from a personal data source upon user input. This approach allows a wide variety of personal content to be viewed by facilitating symbiotic relationships between mobile devices and intelligent displays in the environment. Our initial inverted browser prototype is based on a Web services wrapper around a traditional Web browser. Our experiments show that the inverted browser approach is superior to other solutions in terms of user convenience, ease of use, energy consumption, and privacy, but interaction latencies need improvement.

The integration of Bluetooth service discovery protocol (SDP), and GPRS Internet connectivity into phones provides a simple yet powerful infrastructure for accessing services in nomadic environments. In this paper, we discuss the design and implementation of SDIPP, a protocol for provisioning services on smart phones. Although several service discovery protocols have been proposed earlier, such as SLP, Jini, UPnP, Salutation, they all have their own infrastructure requirements and target audiences, Bluetooth SDP is an on-the-fly service discovery protocol. However, it is not nearly as powerful as its counterparts. SDIPP works by augmenting Bluetooth SDP with Web access and personalization. Payment of services has been overlooked in the protocols proposed earlier. SDIPP provides a novel protocol for anonymous payment, based on the idea of Millicent scrips. We have implemented a few services to illustrate our protocol. We report on our experiences and experimental results. In particular, we analyze and provide an application level solution to the Bluetooth inquiry clash problem that was discovered in the process.

Recent advances in wireless technology allow Java-enabled devices, such as Smart Phones and PDAs, to create mobile ad hoc networks, over which distributed applications can be executed. Although Java shields the programmers from the heterogeneity of the hardware platforms, a common middleware architecture is needed to support a cooperative execution environment in these networks. In this paper, we present a portable runtime system for smart messages (SMs), a middleware architecture based on execution migration, that we designed and implemented on top of an unmodified Java virtual machine. To facilitate portability, we have designed a lightweight migration mechanism based on Java bytecode instrumentation. This mechanism is suitable for mobile ad hoc networks where limited bandwidth and mobility impose constraints on the amount of data transferred. The experimental results for applications executed over wireless networks of HP iPAQs demonstrate the feasibility of our portable runtime system.