In recent years, much of the discussion involving “smart grids” has implicitly involved only the distribution side, notably advanced metering. However, today's electric systems have many challenges that also involve the rest of the system. An enabling technology for improving the power system, which has emerged in recent years, is the ability to measure coherent, real-time data. In this paper, we describe major challenges facing electrical generation and transmission today that availability of these measurements can help address. We overview applications using coherent, real-time measurements that are in use today or proposed by researchers. Specifically, we describe, normalize, and then quantitatively compare key factors for these power applications that influence how the delivery system should be planned, implemented, and managed. These factors include whether a person or computer is in the loop and (for both inputs and outputs) latency, rate, criticality, quantity, and geographic scope. From this, we abstract the baseline communications requirements of a data delivery system supporting these applications and suggest implementation guidelines to achieve them. Finally, we overview the state of the art in the supporting computer science areas of overlay networking and distributed computing (including middleware) and analyze gaps in commercial middleware products, utility standards, and issues that limit low-level network protocols from meeting these requirements when used in isolation.

GridStat is a QoS-managed middleware framework designed to replace the power grid's aging, inflexible, and slow data communications system. GridStat is a specialization of the publish-subscribe paradigm that takes advantage of the semantics of periodic updates of cached sensor data to provide data delivery with a per-subscriber rate, latency, and redundant paths. While GridStat is well-suited for delivery of sensor data over a wide-area network, its baseline one-way mechanisms are not suitable for round-trip invocations such as setting an actuator or calling between control centers. In this paper we present the design, implementation and experimental evaluation of Ratatoskr, a tunable remote procedure call mechanism that builds on the QoS semantics of GridStat and supports three kinds of redundancy. Additionally, user-defined pre- and post-condition predicates over GridStat status variables are built into the call semantics.

The power grid is not only a network interconnecting generators and loads through a transmission and distribution system, but is overlaid with a communication and control system that enables economic and secure operation. This multilayered infrastructure has evolved over many decades utilizing new technologies as they have appeared. This evolution has been slow and incremental, as the operation of the power system consisting of vertically integrated utilities has, until recently, changed very little. The monitoring of the grid is still done by a hierarchical design with polling for data at scanning rates in seconds that reflects the conceptual design of the 1960s. This design was adequate for vertically integrated utilities with limited feedback and wide-area controls; however, the thesis of this paper is that the changing environment, in both policy and technology, requires a new look at the operation of the power grid and a complete redesign of the control, communication and computation infrastructure. We provide several example novel control and communication regimes for such a new infrastructure.

Embedded systems have become commonplace in recent years, and are increasingly being networked Middleware offers many advantages to the distributed application programmer, yet there exist very few middleware frameworks for the low end of the embedded systems market. In this paper we describe MicroQoSCORBA. It represents a fundamental, bottom-up rethinking of what middleware can and should support for resource-constrained devices. This middleware is tailorable, with a fine degree of granularity, to both the device and the application program's constraints. We describe the multiple Quality of Service domains that MicroQoSCORBA supports, and present an evaluation of our working framework.

Building dependable distributed systems from commercial off-the-shelf components is of growing practical importance. For both cost and production reasons, there is interest in approaches and architectures that facilitate building such systems. The AQuA architecture is one such approach; its goal is to provide adaptive fault tolerance to CORBA applications by replicating objects. The AQuA architecture allows application programmers to request desired levels of dependability during applications' runtimes. It provides fault tolerance mechanisms to ensure that a CORBA client can always obtain reliable services, even if the CORBA server object that provides the desired services suffers from crash failures and value faults. AQuA includes a replicated dependability manager that provides dependability management by configuring the system in response to applications' requests and changes in system resources due to faults. It uses Maestro/Ensemble to provide group communication services. It contains a gateway to intercept standard CORBA IIOP messages to allow any standard CORBA application to use AQuA. It provides different types of replication schemes to forward messages reliably to the remote replicated objects. All of the replication schemes ensure strong, data consistency among replicas. This paper describes the AQuA architecture and presents, in detail, the active replication pass-first scheme. In addition, the interface to the dependability manager and the design of the dependability manager replication are also described. Finally, we describe performance measurements that were conducted for the active replication pass-first scheme, and we present results from our study of fault detection, recovery, and blocking times.

Voting is the process of combining multiple replies from replicated servers into a single reply. Data fusion is similar to but more general than voting. In data fusion, the input sources are not necessarily replicated servers, hence the inputs exhibit greater variance. Data fusion is a fundamental building block in distributed systems. It occurs in diverse contexts such as consensus, sensor networks, intrusion detection, and hierarchical resource monitoring, among others. This paper describes Mr. Fusion, a framework that provides data fusion in middleware. The heart of Mr. Fusion is a Fusion Core module that provides mechanisms for programming a wide variety of data fusion algorithms. Another part is a Fusion Status Service that monitors low-level outputs from the Fusion Core and alerts subscribers to divergent values or timings. The implementation borrows techniques from data warehousing and data mining.

Intrusion tolerant systems provide high-integrity and high-availability services to their clients in the face of successful attacks from an adversary. The Intrusion Tolerant Distributed Object Systems (ITDOS) research project is developing an architecture for a heterogeneous intrusion tolerant distributed object system. ITDOS integrates a Byzantine Fault Tolerant multicast protocol into an open-source CORBA ORB to provide intrusion tolerant middleware. This foundation allows up to f simultaneous Byzantine failures of replicated servers in a system of at least 3f+1 replicas. Voting on unmarshalled CORBA messages allows heterogeneous application implementations for a given service, allowing for greater diversity in implementation and greater survivability. Symmetric encryption session keys generated by distributed pseudo-random function techniques provide confidential client-server communications. This paper overviews the ITDOS architecture, discusses some of the challenging technical issues related to intrusion tolerance in heterogeneous middleware systems, and offers views on future areas of work.

Middleware is a class of software systems above the operating system which is becoming widely used for programming distributed systems. Voting is a fundamental operation when distributed systems involve replicated components. However support for voting in middleware is very limited. The paper describes issues involved with supporting voting, and more general data fusion, in middleware. We describe the Voting Virtual Machine (VVM) architecture, which can be embedded in different middleware substrates such as .NET and CORBA. We also describe its companion Voting Definition Language (VDL), which allows for portable description of voting algorithms. The VVM and VDL, together with the external VVM voting manager provide for voting transparency and adaptive voting. Finally, we describe how the simple "byte-by-byte" value comparison schemes used in other voting middleware as well as in byzantine fault tolerant multicast systems do not work in the face of the heterogeneity inherent in distributed systems.

Distributed applications that share a dynamically changing state are increasingly being deployed in wide-area environments. Such applications must access the state in a consistent manner, but the consistency requirements vary significantly from other systems. For example, shared memory models, such as sequential consistency, focus on the ordering of operations, and the same level of consistency is provided to each process. In interactive distributed applications, the timeliness of updates becoming effective could be an extremely important consistency requirement, and it could be different across different users. We propose a system that provides both non-timed and time-sensitive read and write operations for dynamic shared state. For example, a timed read can be used by a process to read a recently written value, whereas a timed write can make a new value available to all readers within a certain amount of time. We develop a consistency model that precisely defines the semantics of timed and non-tinted read and write operations. A protocol that implements this model is also presented. We also describe an implementation and some performance measurements.

A common class of wide-area distributed applications remotely collect time-varying data and send it to consumers around the network. Some examples of these include network management, stock ticker data and event logs. The environment in which these applications must operate often dictates the schemes for disseminating the data between the writers and the readers. If the transport channel can be optimized to match the application's behavior patterns and the network resource constraints, sufficient improvements in application-level quality of service (QoS) can be achieved. The PASS (Piecewise Asynchronous Sample Service) system addresses this problem by using a flexible system of interconnected servers. PASS servers are distributed geographically around the network and are connected to readers and writers using the CORBA protocol. The forwarding policies used by the servers and the server interconnections can be customized for each application. Thus, PASS acts like an application-level multicast service with variable forwarding policies. PASS has been used to disseminate the up/down status of a large number of devices to a network management system. The PASS forwarding policy used very little network bandwidth while responding to failures in half a network round-trip time.

As networks and the use of communications within applications continue to grow and find more uses, so too does the demand for more control and manageability of various "system properties" through middleware. An important component supporting an integrated property architecture is the concept of an object gateway, which is a quality-of-service (QoS)-aware element transparently inserted at the transport layer between clients and objects to provide managed communication behavior for the particular property being supported. In this paper, we introduce the concept of a QoS-oriented gateway to integrate a variety of QoS enforcement and implementation mechanisms controlling the underlying distributed interactions. We discuss the functions performed by such a component in achieving the desired overall end-to-end QoS, and the design considerations underlying our current implementation. We conclude with experiences to date with two variations of the gateway: one controlling managed latency and throughput using bandwidth allocation, and one controlling dependability through the coordination of object replicas.

Distributed applications are difficult to build and maintain and are even more difficult when the applications are distributed over wide-area networks. Distributed Object Computing middleware has emerged to simplify the building of distributed applications by hiding implementation details behind functional interfaces. However, critical applications have non-functional requirements, such as real-time performance, dependability, or security, that are as important as the functional requirements, but are also hidden by the middleware. Because current distributed object middleware doesn't support these aspects of critical applications, application developers often find themselves bypassing the distributed object systems, effectively gaining little or no advantage from the middleware. We have developed Quality Objects (QuO), a framework for including Quality of Service (QoS) in distributed object applications. QuO supports the specification of QoS contracts between clients and service providers, runtime monitoring of contracts, and adaptation to changing system conditions. A crucial aspect of QuO is a suite of Quality Description Languages for describing states of QoS, system elements that need to be monitored to measure the current QoS, and notification and adaptation to trigger when the state of QoS in the system changes. This paper gives a brief overview of QuO and describes the syntax and semantics of CDL, the component of QDL for describing QoS contracts.

Dependable distributed systems are difficult to build. This is particularly true if they have dependability requirements that change during the execution of an application, and are built with commercial off-the-shelf hardware. In that case, fault tolerance must be achieved using middleware software, and mechanisms must be provided to communicate the dependability requirements of a distributed application to the system and to adapt the system's configuration to try to achieve the desired dependability. The AQuA architecture allows distributed applications to request a desired level of availability using the Quality Objects (QuO) framework and includes a dependability manager that attempts to meet requested availability levels by configuring the system in response to outside requests and changes in system resources due to faults. The AQuA architecture uses the QuO runtime to process and invoke availability requests, the Proteus dependability manager to configure the system in response to faults and availability requests, and the Ensemble protocol stack to provide group communication services. Furthermore, a CORBA interface is provided to application objects using the AQuA gateway. The gateway provides a mechanism to translate between process-level communication, as supported by Ensemble, and IIOP messages, understood by Object Request Brokers. Both active and passive replication are supported, and the replication type to use is chosen based on the performance and dependability requirements of particular distributed applications.

Linda is a language for programming parallel applications whose most notable feature is a distributed shared memory called tuple space. While suitable for a wide variety of programs, one shortcoming of the language as commonly defined and implemented is a lack of support for writing programs that can tolerate failures in the underlying computing platform. This paper describes FT-Linda, a version of Linda that addresses this problem by providing two major enhancements that facilitate the writing of fault-tolerant applications: stable tuple spaces and atomic execution of tuple space operations. The former is a type of stable storage in which tuple values are guaranteed to persist across failures, while the latter allows collections of tuple operations to be executed in an all-or-nothing fashion despite failures and concurrency. The design of these enhancements is presented in detail and illustrated by examples drawn from both the Linda and fault-tolerance domains. An implementation of FT-Linda for a network of workstations is also described. The design is based on replicating the contents of stable tuple spaces to provide failure resilience and then updating the copies using atomic multicast. This strategy allows an efficient implementation in which only a single multicast message is needed for each atomic collection of tuple space operations.>