There is an emerging class of enterprise applications in areas such as log data analysis, information discovery, and social media marketing that involve analytics over large volumes of unstructured and semi-structured data. These applications are leveraging new analytics platforms based on the MapReduce framework and its open source Hadoop implementation. While this trend has engendered work on high-level data analysis languages, NoSQL data stores, workflow engines etc., there has been very little attention to the challenges of deploying analytic workflows into production for continuous operation. In this paper, we argue that an essential platform component for enabling continuous production analytic workflows is an analytics store. We highlight five key requirements that impact the design of such a store: (i) efficient incremental operations, (ii) flexible storage model for hierarchical data, (iii) snapshot support (iv) object-level incremental updates, and (v) support for handling change sets. We describe the design of SASH, a scalable analytics store that we have developed on top of HBase to address these requirements. Using the workload from a production workflow that powers search within IBM's intranet and extranet, we demonstrate orders of magnitude improvement in IO performance using SASH.

Migrating production applications from physical datacenters to a virtualized environment is becoming essential to reduce operational costs. In order to avoid direct access to production systems, migration from disk snapshots is preferable. Model based migration approaches are not suitable for this purpose since they need to perform configuration discovery on production systems. Recent approaches that rely upon an isolated network require special setup and are limited to handling network configurations only. In this paper we present a framework for instantiating an application from disk snapshots in a virtualized environment. Our approach comprises of formally specifying the knowledge about product configurations and compiling the knowledge into a plan. The plan is capable of discovering application configurations in an inconsistent configuration setup. A key feature of our framework is that knowledge is specified once per product and re-used across applications that use the product. We have implemented a prototype of our approach and evaluated it on a real world application. The evaluation demonstrates the feasibility of our approach for migration using disk snapshots.

Loosely-coupled composite enterprise applications based on modern Web technologies are becoming increasingly popular. While composing such applications is appealing for a number of reasons, the distributed nature of the applications makes problem determination difficult. Stringent service level agreements in these environments require rapid localization of failing and poorly performing services. We present in this paper a method that performs real-time transaction level problem determination by tracking synchronous transaction flows in HTTP based composite enterprise applications. Our method relies on instrumentation of service requests and responses to transmit downstream path and monitoring information in realtime. Further, our method applies change-point based techniques on monitored information at the point of origin of a transaction, and quickly detects anomalies in the performance of invoked services. Since our method performs transaction level monitoring, it avoids the pitfalls associated with techniques that use aggregate performance metrics. Additionally, since we use change-point based techniques to detect problems, our method is more robust than error-prone static threshold based techniques.

Clustered enterprise middleware systems employing dynamic workload scheduling are susceptible to a variety of application malfunctions that can manifest themselves in a counterintuitive fashion and cause debilitating damage. Until now, diagnosing problems in that domain involves investigating log files and configuration settings and requires in-depth knowledge of the middleware architecture and application design. This paper presents a method for problem determination using change point detection techniques and problem signatures consisting of a combination of changes (or absence of changes) in different metrics. We implemented this approach on a clustered middleware system and applied it to the detection of the storm drain condition: a debilitating problem encountered in clustered systems with counterintuitive symptoms. Our experimental results show that the system detects 93% of storm drain faults with no false positives.

Java is becoming an important platform for memory-constrained consumer devices such as PDAs and cellular phones, because it provides safety and portability. Since Java uses garbage collection, efficient garbage collectors that run in constrained memory are essential. Typical collection techniques used on these devices are mark-sweep and mark-compact. Mark-sweep collectors can provide good throughput and pause times but suffer from fragmentation. Mark-compact collectors prevent fragmentation, have low space overheads, and provide good throughput. However, they can suffer from long pause times. Copying collectors can provide higher throughput than either of these techniques, but because of their high space overhead, they previously were unsuitable for memory-constrained devices. This paper presents MC2 (Memory-Constrained Copying), a copying generational garbage collector that meets the needs of memory-constrained devices with soft real-time requirements. MC2 has low space over-head and tight space bounds, prevents fragmentation, provides good throughput, and yields short pause times. These qualities make MC2 attractive for other environments, including desktops and servers.

Copying garbage collectors have a number of advantages over non-copying collectors, including cheap allocation and avoiding fragmentation. However, in order to provide completeness (the guarantee to reclaim each garbage object eventually), standard copying collectors require space equal to twice the size of the maximum live data for a program. We present a mark-copy collection algorithm (MC) that extends generational copying collection and significantly reduces the heap space required to run a program. MC reduces space overhead by 75--85% compared with standard copying garbage collectors, increasing the range of applications that can use copying garbage collection. We show that when MC is given the same amount of space as a generational copying collector, it improves total execution time of Java benchmarks significantly in tight heaps, and by 5--10% in moderate size heaps. We also compare the performance of MC with a (non-generational) mark-sweep collector and a hybrid copying/mark-sweep generational collector. We find that MC can run in heaps comparable in size to the minimum heap space required by mark-sweep. We also find that for most benchmarks MC is significantly faster than mark-sweep in small and moderate size heaps. When compared with the hybrid collector, MC improves total execution time by about 5% for some benchmarks, partly by increasing the speed of execution of the application code.