Fulfilling the IT Service Management proposition of providing IT as a service aligned with the company business presents many interesting challenges. Among these we consider the challenge of integrating Business Performance Management (BPM) of critical applications with the systems management and provisioning of the IT infrastructure that supports the organization’s business processes.

Modeling, assembling, deploying and managing composite applications built using service oriented architectures (SOA) present many interesting challenges. Among these challenges, we consider two: service deployment and security management for access control. In this application session, these challenges are explored in the context of a prototype banking application: "Jivaro". The Jivaro prototype has features such as automated and configurable business processes using BPEL and manual tasks, multitenancy using virtual portals, and security using LDAP. We focus on two specific management pain points: 1. the difficulty in deploying multiple SOA components into multiple target middleware containers, and 2. inconsistency in access control policies for multi-tier applications. For addressing the deployment issues, we present a real-wo rid deployment scenario involving the use of ANT tasks and scripting interfaces. For addressing inconsistencies in access control policies, we present a solution using the XACML standard, a common authorization model developed as an extension of the Java authorization contract for containers and a common policy store and policy administration point. We compare and contrast current access control policies for J2EE containers and databases with the proposed new common authorization model. We also compare separate access control policy stores versus the proposed solution for a common store. Our aggregated role based authorization model provides consistent access control policies that complement single sign-on and identity propagation schemes. This model also touches upon issues surrounding role and policy based management, specifically regarding the potential of combining security policy administrator roles for different tiers.

Dynamic computing environments that support a changing application set and load are now a reality. Within these environments the decision to add or remove resources, must be made in a timely manner, while at the same time they must avoid excessive resource rebalancing. When the overhead incurred by the resource reallocation process is significant, we should be confident that additional resources are necessary before initiating an allocation process. This process is complicated by the instability of application content and frequent changes in average request processing time. Load and response time thresholds need to be dynamically and automatically adjusted if they are to remain effective. We investigate automated methods for selecting and setting thresholds, estimating load, and analyzing response time. Specifically, these methods estimate the load on a set of application servers, and set appropriate resource allocation thresholds based on the relationship between projected site response times and server load. We describe a solution to the problem of determining the maximum load over multiple servers under changing conditions, which include both changing traffic and application characteristics.

We present Neptune - the resource director of Océano, a policy driven fabric management system that dynamically reconfigures resources in a computing utility cluster. Neptune implements an on-line control mechanism subject to policy-based performance and resource configuration objectives. Neptune reassigns servers and bandwidth among a set of service domains, based on pre-defined policy, in response to workload changes. It builds and executes a reconfiguration plan through a planning framework, breaking reconfiguration objectives into individual tasks delegated to set of lower level resource managers. We describe an example decision policy algorithm that we implemented and demonstrated in an 80 server multi-domain computing utility.

Oceano is management software for a eUtility infrastructure capable of providing cost-effective, autonomic resource allocation for multiple customer or application domains, in response to existent performance and availability conditions. The control layer of Oceano provides mechanisms to manage resources. This layer consists of a resource director and a set of resource managers. The resource director formulates resource configurations and coordinates their execution through resource managers. These resource managers maintain restore state and carry out detailed configuration tasks. This paper describes the Oceano resource management model and its server and application deployment resource managers. A prototype of Oceano has been developed and deployed on an 80 server platform, and has been tested with multiple domains and applications.

This paper describes GulfStream, a scalable distributed software system designed to address the problem of managing the network topology in a multi-domain server farm. In particular, it addresses the following core problems: topology discovery and verification, and failure detection. Unlike most topology discovery and failure detection systems which focus on the nodes in a cluster, GulfStream logically organizes the network adapters of the server farm into groups. Each group contains those adapters that can directly exchange messages. GulfStream dynamically establishes a hierarchy for reporting network topology and availability of network adapters. We describe a prototype implementation of GulfStream on a 55 node heterogeneous server farm interconnected using switched fast Ethernet.

Oceano is a prototype of a highly available, scaleable, and manageable infrastructure for an e-business computing utility. It enables multiple customers to be hosted on a collection of sequentially shared resources. The hosting environment is divided into secure domains, each supporting one customer. These domains are dynamic: the resources assigned to them may be augmented when the load increases and reduced when load dips. This dynamic resource allocation enables flexible service level agreements (SLAs) with customers in an environment where peak loads are an order of magnitude greater than the normal steady state.

Yemanja is a model-based event correlation engine for multi-layer fault diagnosis. It targets complex propagating fault scenarios, and can smoothly correlate low-level network events with high-level application performance alerts related to quality of service violations. Entity-models that represent devices or abstract components encapsulate entity behavior. Distantly associated entities are not explicitly aware of each other, and communicate through event propagation chains. Yemanja's state-based engine supports generic scenario definitions, prioritization of alternate solutions, integrated problem-state and device testing, and simultaneous analysis of overlapping problems. The system of correlation rules was developed based on device, layer, and dependency analysis, and reveals the layered structure of computer networks. The primary objectives of this research include the development of reusable, configuration independent, correlation scenarios; adaptability and the extensibility of the engine to match the constantly changing topology of a multi-domain server farm; and the development of a concise specification language that is relatively simple yet powerful.

Process groups in distributed applications and services rely on failure detectors to detect process failures completely, and as quickly, accurately, and scalably as possible, even in the face of unreliable message deliveries. In this paper, we look at quantifying the optimal scalability, in terms of network load, (in messages per second, with messages having a size limit) of distributed, complete failure detectors as a function of application-specified requirements. These requirements are 1) quick failure detection by some non-faulty process, and 2) accuracy of failure detection. We assume a crash-recovery (non-Byzantine) failure model, and a network model that is probabilistically unreliable (w.r.t. message deliveries and process failures). First, we characterize, under certain independence assumptions, the optimum worst-case network load imposed by any failure detector that achieves an application's requirements. We then discuss why traditional heart beating schemes are inherently unscalable according to the optimal load. We also present a randomized, distributed, failure detector algorithm that imposes an equal expected load per group member. This protocol satisfies the application defined constraints of completeness and accuracy, and speed of detection on an average. It imposes a network load that differs frown the optimal by a sub-optimality factor that is much lower than that for traditional distributed heartbeating schemes. Moreover, this sub-optimality factor does not vary with group size (for large groups).

The Enterprise Management Protocol (TEMP) and its affiliated infrastructure provide a highly scalable, transport protocol-independent base for enabling the management of networked devices, systems and, most importantly, applications. The functionality and benefits of TEMP are illustrated through a case study of the remote management of Narwhal Client Agents. Narwhal is a technology that provides a local (i.e., client-resident) intermediary broker that can intelligently route the traffic among intermediaries. Its benefits include: (1) improved availability of intermediary services; (2) load sharing requests across several intermediaries; (3) bypassing intermediaries whenever possible; and (4) remote administrative control enabling the implementation of domain-specific policies to utilize the shared limited networking resources.

Network Dispatcher (ND) is a software tool that "routes" TCP connections to multiple TCP servers that share their workload. It exports a set of virtual IP addresses that are concealed and shared by the servers. It implements a novel dynamic load-sharing algorithm for allocation of TCP connections among servers according to their real-time load and responsiveness. ND forwards packets to the servers without performing any TCP/IP header translations, consequently outgoing server-to-client packets are not handled, and can follow a separate network route to the clients. Its allocation method was proven to be efficient in live tests, supporting Internet sites that served millions of TCP connections per hour. This paper describes the load management features of ND.

As the global Internet traffic increases, many popular sites are often unable to serve their TCP/IP workload, particularly during peak periods of activity. For example, Web servers for sports events are often swamped by requests during and after games. To address this problem, many sites allocate multiple server hosts to concurrently handle the incoming requests. To support workload sharing, they need a method to distribute the requests among the servers. Since network traffic is self-similar, with waves of heavy traffic at peak times, this requires dynamic feedback control. In this presentation we analyse several solutions to this scaling problem (client-based and DNS-based), and show some of their deficiencies. We then present our preferred method, which is based on IP packet forwarding and is transparent to both clients and servers. We implemented a TCP/IP load-management tool, NetDispatcher, that enables scalable, heterogeneous TCP/IP server clusters that can handle millions of TCP connections per hour. NetDispatcher does not perform any TCP/IP header translations. Hence, outgoing server-to-client packets need no processing and can follow a separate network route to the clients, resulting in improved bandwidth utilization and lower latency. Netdispatcher transparently handles server failures. NetDispatcher failures are handled by a shadow node without loosing active connection.

This paper introduces a novel approach to distributed computing based on delegation-agents, and describes its applications to decentralize network management. Delegation agents are programs that can be dispatched to remote processes, dynamically linked and executed under local or remote control. Unlike scripted agents, delegation agent programs may be written in arbitrary languages, interpreted or compiled. They can thus be more broadly applied to handle such tasks as real-time monitoring, analysis and control of network resources. Distributed management by delegation (MbD) uses this to manage remote elements and domains. MbD provides a paradigm for distributed, flexible, scalable and robust network management that overcomes the key limitations of current centralized management schemes.

An integrated system design for debugging distributed programs written in concurrent high-level languages is described. A variety of user-interface, monitoring, and analysis tools integrated around a uniform process model are provided. Because the tools are language-based, the user does not have to deal with low-level implementation details of distribution and concurrency, and instead can focus on the logic of the program in terms of language-level objects and constructs. The tools provide facilities for experimentation with process scheduling, environment simulation, and nondeterministic selections. Presentation and analysis of the program's behavior are supported by history replay, state queries, and assertion checking. Assertions are formulated in linear time temporal logic, which is a logic particularly well suited to specify the behavior of distributed programs. The tools are separated into two sets. The language-specific tools are those that directly interact with programs for monitoring of and on-line experimenting with distributed programs. The language-independent tools are those that support off-line presentation and analysis of the monitored information. This separation makes the system applicable to a wide range of programming languages. In addition, the separation of interactive experimentation from off-line analysis provides for efficient exploitation of both user time and machine resources. The implementation of a debugging facility for OCCAM is described.

Concert, a high-level-language approach to programming heterogeneous distributed systems, is described. The Concert model introduces a small set of language extensions into conventional procedural languages. These language extensions support a cooperative peer process model which addresses in the distributed environment the same issues addressed by language semantics in the conventional environment. The Concert implementation provides layered support for these language extensions, bridging a different source of heterogeneity at each layer. A prototype Concert system currently includes C programs running on OS/2 on multiple PS/2s communicating via calls with one another as well as with PL/I programs running on VM/370.>