Over the past one and half years, we have been using RDMA over commodity Ethernet (RoCEv2) to support some of Microsoft's highly-reliable, latency-sensitive services. This paper describes the challenges we encountered during the process and the solutions we devised to address them. In order to scale RoCEv2 beyond VLAN, we have designed a DSCP-based priority flow-control (PFC) mechanism to ensure large-scale deployment. We have addressed the safety challenges brought by PFC-induced deadlock (yes, it happened!), RDMA transport livelock, and the NIC PFC pause frame storm problem. We have also built the monitoring and management systems to make sure RDMA works as expected. Our experiences show that the safety and scalability issues of running RoCEv2 at scale can all be addressed, and RDMA can replace TCP for intra data center communications and achieve low latency, low CPU overhead, and high throughput.

This paper describes a hierarchical scheme for scheduling periodic messages of distributed real-time applications in a cluster of PCs connected by a Myrinet. The scheme allows the developer of each real-time application to validate the schedulability of the application as if it were running alone on slower dedicated processors and network. A simple acceptance test is provided. According to the test, the operating system can determine whether to admit a new real-time system application into a system without having to perform detailed global schedulability analysis. Simulation results show that the hierarchical scheduling scheme outperforms the corresponding one-level heuristic algorithms. An open system prototype on a Myrinet cluster PCs running Windows NT uses this hierarchical scheme.

This paper extends the two-level hierarchical scheme in (Deng et al., 1997) for scheduling independently developed real-time applications with non-real-time applications in an open environment. The environment allows the schedulability of each real-time application to be validated independently of other applications in the system. The extended scheme removes the following two restrictions of the scheme: real-time applications that are scheduled preemptively must consist solely of periodic tasks; and applications must not share global resources (i.e., resources used by more than one application). Consequently, the extended scheme can accommodate a much broader spectrum of real-time applications.

This paper describes PERTS, a prototyping environment for real-time systems. PERTS provides a comprehensive set of design, validation, and simulation tools that are built on recent theoretical advances. It also provides software modules that implement wellknown and novel real-time scheduling algorithms, task and resource assignment algorithms, and resource access control protocols. Together, the tools and software modules support the systematic and rigorous evaluation of new designs, experimentation with alternative scheduling and resource management strategies, and the evaluation and validation of the resultant real-time system.

PERTS is a prototyping environment for real-time systems. It contains schedulers and resource access protocols for time-critical applications, together with a comprehensive set of tools for the analysis, validation, and evaluation of real-time systems built on the scheduling paradigms supported by these building blocks. This paper describes the underlying models of real-time systems supported by PERTS, as well as its capabilities and intended use. A key component is the schedulability analyzer. The basic version of this system of tools supports the validation and evaluation of real-time systems built on the framework of the periodic-task model. This system of tools is now available.>