Software discovery is a key management function to ensure that systems are free of vulnerabilities, comply with licensing requirements, and support advanced search for systems containing given software. Today, software is predominantly discovered through querying package management tools, or using rules that check for file metadata or contents. These approaches are inadequate as not every software is installed through package managers, and agile development practices lead to frequent deployment of software. Other approaches to software discovery use machine learning methods requiring training phase, or require maintaining knowledge bases. Columbus uses the knowledge of the software packaging practices that evolved over time, and uses the information embedded in the file system impression created by a software package to discover it. Columbus is able to discover software in 92% of all official Docker images. Further, Columbus can be used in problem diagnosis and drift detection situations to compare two different systems, or to determine the evolution of a system overtime.

Software-defined networks (SDNs) only logically centralize the control plane. In reality, SDN controllers are distributed entities, which may exhibit different behavior on event triggers. We identify several classes of faults that afflict an SDN controller cluster and demonstrate them on two enterprise SDN controllers, ONOS and OpenDaylight. We present JURY, a system to validate controller activities in a clustered SDN deployment, involving topological and forwarding state, without imposing any restrictions on the controller behavior. Our evaluation shows that JURY requires minimal changes to the SDN controllers for deployment, and is capable of validating controller actions in near real time with low performance overheads.

With majority of the world's data and computation handled by cloud-based systems, cloud management stacks such as Apache's CloudStack, VMware's vSphere and OpenStack have become an increasingly important component in cloud software. However, like every other complex distributed system, these cloud stacks are susceptible to faults, whose root cause is often hard to diagnose. We present HANSEL, a system that leverages non-intrusive network monitoring to expedite root cause analysis of such faults manifesting in OpenStack operations. HANSEL is fast and accurate, and precise even under conditions of stress.

As businesses move their critical IT operations to multi-tenant cloud data centers, it is becoming increasingly important to provide network performance guarantees to individual tenants. Due to the impact of network congestion on the performance of many common cloud applications, recent work has focused on enabling network reservation for individual tenants. Current network reservation methods, however, do not gracefully degrade in the presence of network over subscriptions that may frequently occur in a cloud environment. In this context, for a shared data center network, we introduce Network Satisfaction Ratio (NSR) as a measure of the satisfaction derived by a tenant from a given network reservation. NSR is defined as the ratio of the actual reserved bandwidth to the desired bandwidth of the tenant. Based on NSR, we present a novel network reservation mechanism that can admit time-varying tenant requests and can fairly distribute any degradation in the NSR among the tenants in presence of network over subscription. We evaluate the proposed method using both synthetic network traffic trace and representative data center traffic trace generated by running a reduced data center job trace in a small test bed. The evaluation shows that our method adapts to changes in network reservations, and it provides significant and fair improvement in NSR when the data center network is oversubscribed.

Human error has been identified one of the major factors behind system outages and network downtime in a number of previous research papers and surveys. Gartner statistics show that almost 40% of unplanned application downtime is caused due to operator errors such as unintentional changes to network configuration resulting in a network outage, patch installations, service restart, etc. Yet, system admin activities on production IT systems are rarely properly logged and monitored. Existing tools to track user activities either produce too much information without any hints of a potential outage scenario or too little information to be useful in a meaningful way. In this paper, we describe the design and implementation of iTrack - a framework for monitoring user activities and correlating them with system data. iTrack makes use of commonly available native monitoring and diagnostic utilities on operating systems to monitor systems events as well as system admin activity, correlates these two sets of information and categorizes the activity as potentially abnormal or harmful based on its impact on the system in terms of file system, network and process activities. We demonstrate the usefulness of iTrack through several use cases and real world examples such as detecting and diagnosing system outages in real time, conducting post mortem analysis of outages, and maintaining audit logs. Our experimental evaluation of iTrack confirms that its monitoring overhead in terms of CPU time, activity completion time and data generated is within the tolerance range of most production systems. In cases, where the overhead was found to be unacceptable, we detect the underlying cause and provide solutions. These solutions improve performance by up to 20% to 90%, in terms of managed server and iTrack server CPU utilization, respectively and by up to 2 times in terms of completion time of certain system admin activities on the managed server.

Most enterprises today run their applications on virtual machines (VMs). VM mobility - both live and offline, can provide enormous flexibility and also bring down OPEX (Operational Expenditure) costs. However, both live and offline migration of VMs is still limited to within a local network because of the complexities associated with cross subnet live and offline migration. These complexities mainly arise from the hierarchical addressing used by various layer 3 routing protocols. For cross data center VM mobility, virtualization vendors require that the network configuration of the new data center where a VM migrates must be similar to that of the old data center. This severely restricts wide spread use of VM migration across data center networks. For offline migration, the above limitations can be overcome by reconfiguring IP addresses for the migrated VMs. However, even this effort is non-trivial and time consuming as these IP addresses are embedded in various configuration files inside these VMs. As enterprises grow and new data centers emerge in different geographic locations, there is a need to interconnect these data centers in a way that allows seamless VM mobility. In this context, we present CrossRoads - a network fabric that provides layer agnostic and seamless live and offline VM mobility across multiple data centers. We leverage software defined networking and implement an OpenFlow based prototype of CrossRoads. CrossRoads extends the idea of location independence based on pseudo addresses proposed in recent research to work with a control plane overlay of OpenFlow network controllers in various data centers. We evaluate CrossRoads on an innovative testbed that leverages nested virtualization to emulate two data centers. Our results confirm that CrossRoads has negligible performance overhead as compared to a Default layer 2 network - its average performance was no worse than 2.3% as compared to Default fabric across all experiments. In some experiments, it even outperformed the Default by up to 30%.

Various studies have shown that OS jitter can degrade parallel program performance considerably at large processor counts. Most sources of system jitter fall broadly into 5 categories - user space processes, kernel threads, interrupts, SMT interference and hypervisor activity. Solutions to OS jitter typically consist of a combination of techniques such as synchronization of jitter across nodes (co-scheduling or gang scheduling) and use of microkernels. Both techniques present several drawbacks. Multicore and Multithreaded systems present opportunities to handle OS jitter. They have multiple cores and threads, some of which can be used for handling OS jitter, while the application threads run on remaining cores and threads. However, they are also prone to risks such as inter-thread cache interference and process migration. In this paper, we present a holistic approach that aims to reduce jitter caused by various sources of jitter by utilizing the additional threads or cores in a system. Our approach handles jitter through reduction of kernel threads, intelligent interrupt handling, and switching of hardware SMT thread priorities. This helps in reducing jitter experienced by application threads in the user space, at the kernel level, and at the hardware level. We make use of existing features available in the Linux kernel and Power Architecture as well make enhancements to the Linux kernel. We demonstrate the efficacy of our techniques by reducing jitter on two different platforms and operating system versions. In the first case our approach helps in reducing periodic jitter that improves both average and worst case performance of a simulated parallel application. In the second case our approach helps in reducing infrequent very large jitter that helps the worst case performance of a real parallel application. Our experimental results show up to 30% reduction in slowdown in the average case at 16K OS images and up to 50% reduction in slowdown in the worst case at 8 OS images using this approach as compared to a baseline configuration.

Various studies have pointed out the debilitating effects of OS jitter on the performance of parallel applications on large clusters such as the ASCI Purple and the Mare Nostrum at Barcelona Supercomputing Center. These clusters use commodity OSes such as AIX and Linux respectively. The biggest hindrance in evaluating any technique to mitigate jitter is getting access to such large scale production HPC systems running a commodity OS. An earlier attempt aimed at solving this problem was to emulate the effects of OS jitter on more widely available and jitter-free systems such as BlueGene/L. In this paper, we point out the shortcomings of previous such approaches and present the design and implementation of an emulation framework that helps overcome those shortcomings by using innovative techniques. We collect jitter traces on a commodity OS with a given configuration, under which we want to study the scaling behavior. These traces are then replayed on a jitter-free system to predict scalability in presence of OS jitter. The application of this emulation framework to predict scalability is illustrated through a comparative scalability study of an off-the-shelf Linux distribution with a minimal configuration (runlevel 1) and a highly optimized embedded Linux distribution, running on the IO nodes of BlueGene/L. We validate the results of our emulation both on a single node as well as on a real cluster. Our results indicate that an optimized OS along with a technique to synchronize jitter can reduce the performance degradation due to jitter from 99% (in case of the off-the-shelf Linux without any synchronization) to a much more tolerable level of 6% (in case of highly optimized BlueGene/L IO node Linux with synchronization) at 2048 processors. Furthermore, perfect synchronization can give linear scaling with less than 1% slowdown, regardless of the type of OS used. However, as the jitter at different nodes starts getting desynchronized, even with a minor skew across nodes, the optimized OS starts outperforming the off-the-shelf OS.

Understanding the behavior and impact of various sources of Operating System Jitter (OS Jitter) is important not only for tuning a system for HPC applications, but also for the ongoing efforts to create light-weight versions of commercial operating systems such as Linux, that can be used on compute nodes of large scale HPC systems. In this paper, we present a tool that helps in identifying sources of OS Jitter in a commodity operating system such as Linux and measures the impact of OS Jitter through fine grained kernel instrumentation. Our methodology comprises of running a user-level micro-benchmark and measuring the latencies experienced by the benchmark. We then associate each latency to operating system daemons and interrupts using data obtained from kernel instrumentation. We present experimental results that help identify the biggest contributors to the total OS Jitter perceived by an application on a commodity operating system such as Linux. Our results revealed that while 63% of the total jitter comes from timer interrupts, the rest comes from various system daemons and interrupts, most of which can be easily eliminated. The tool presented in this paper can also be used to tune “out of the box” commodity operating systems as well as to detect new sources of operating system jitter that get introduced as software get installed and upgraded on a tuned system.

Clustered enterprise middleware systems employing dynamic workload scheduling are susceptible to a variety of application malfunctions that can manifest themselves in a counterintuitive fashion and cause debilitating damage. Until now, diagnosing problems in that domain involves investigating log files and configuration settings and requires in-depth knowledge of the middleware architecture and application design. This paper presents a method for problem determination using change point detection techniques and problem signatures consisting of a combination of changes (or absence of changes) in different metrics. We implemented this approach on a clustered middleware system and applied it to the detection of the storm drain condition: a debilitating problem encountered in clustered systems with counterintuitive symptoms. Our experimental results show that the system detects 93% of storm drain faults with no false positives.

A composite service is typically specified using a language such as BPEL4WS and orchestrated by a single coordinator node in a centralized manner. The coordinator receives the client request, makes the required data transformations and invokes the component Web services as per the specification. However, in certain scenarios businesses might want to impose restrictions on access to the data they provide or the source from which they can accept data. Centralized orchestration can lead to violation of these data flow constraints as the central coordinator has access to the input and output data of all the component Web services. In many cases existing methods of data encryption and authentication are not sufficient to handle such constraints. These data flow constraints, thus, present obstacles for composite Web service orchestration. In this paper we propose a solution for orchestrating composite Web services under data flow constraints. The solution is based on decentralized orchestration, in which a composite Web service is broken into a set of partitions, one partition per component Web service. To overcome data flow constraints, each partition is executed within the same domain as the corresponding component Web service and hence, has the same access rights. However, there are, in general, many ways to decentralize a composite Web service. We apply a rule based filtering mechanism to choose a set of partitions that does not violate the specified dataflow constraints.

Abstract The growth of the Internet and the advent of the computational Grid have made it possible to develop and deploy advanced computational collaboratories. These systems build on high‐end computational resources, communication technologies and enabling services underlying the Grid, and provide seamless and collaborative access to resources, applications and data. Combining these focused collaboratories and allowing them to interoperate has many advantages and can lead to truly collaborative, multidisciplinary and multi‐institutional problem solving. However, integrating these collaboratories presents significant challenges, as each of these collaboratories has a unique architecture and implementation, and builds on different enabling technologies. This paper investigates the issues involved in integrating collaboratories operating on the Grid. It then presents the design and implementation of a prototype middleware substrate to enable a peer‐to‐peer integration of and global access to multiple, geographically distributed instances of the DISCOVER computational collaboratory. An experimental evaluation of the middleware substrate is presented. Copyright © 2002 John Wiley & Sons, Ltd.

The growth of the Internet and the advent of the computational "Grid" have made it possible to develop and deploy advanced computational collaboratories. These systems build on high-end computational resources and communication technologies underlying the Grid, and provide seamless and collaborative access to particular resources, services or applications. Integrating these "focused" collaboratories presents significant challenges. Key among these is the design and development of robust middleware support that addresses scalability, service discovery, security and access control, and interaction and collaboration management for consistent access. The authors first investigate the architecture of such a middleware that enables global (Web-based) access to collaboratories. They then present the design and implementation of a middleware substrate that enables a peer-to-peer integration of and global (collaborative) access to geographically distributed instances of the DISCOVER computational collaboratory for interaction and steering.

Abstract This paper presents the design, implementation, and deployment of the DISCOVER Web‐based computational collaboratory. Its primary goal is to bring large distributed simulations to the scientists'/engineers' desktop by providing collaborative Web‐based portals for monitoring, interaction and control. DISCOVER supports a three‐tier architecture composed of detachable thin‐clients at the front‐end, a network of interaction servers in the middle, and a control network of sensors, actuators, and interaction agents at the back‐end. The interaction servers enable clients to connect and collaboratively interact with registered applications using a browser. The application control network enables sensors and actuators to be encapsulated within, and directly deployed with the computational objects. The application interaction gateway manages overall interaction. It uses Java Native Interface to create Java proxies that mirror computational objects and allow them to be directly accessed at the interaction server. Security and authentication are provided using customizable access control lists and SSL‐based secure servers. Copyright © 2001 John Wiley & Sons, Ltd.