High-availability softwarized networks, while offering tremendous flexibility, typically require distributed implementations to ensure resilience and scalability. For example, in software-defined networking (SDN), the control plane constitutes the “network brain,” and is implemented in a distributed fashion to avoid a single point of failure, while sophisticated algorithms are used to ensure that the distributed controller instances operate together as a logically centralized entity. Distributed consensus algorithms such as Raft are used in leading open source distributed SDN controller implementations, such as ONOS and ODL, to guarantee strong consistency of critical replicated data and provide resiliency under failures. We demonstrate the vulnerability of SDN distributed controller software (ONOS) to host mobility-based DDoS attacks, and show that (bursty) DDoS attacks and (intermittent) overload in network demands trigger a form of software “babble” that causes SDN to violate key assumptions of Raft, resulting in significant unavailability of critical communication between the control plane and data plane switches due to Raft behaviors. We propose BabbleResistantRaft, a “babble-resistant” variant of Raft that ensures safety, liveness, and stability under these types of attacks and network conditions, and demonstrate the effectiveness of BabbleResistantRaft through our implementation extending the open-source pysyncobj Raft library.

Software rejuvenation is a technique of proactive fault tolerance that designs the system for periodic reboots. This paper contains three patterns of which Software Rejuvenation is the first. The second, Count the Black Sheep, provides a solution to the problem of knowing precisely what fault has activated. This information is needed to recover quickly and to be able to provide effective long-term treatment. N-Version Programming completes this collection. Multiple versions of an implementation are created to prevent incorrect understanding of the requirements causing operational failures. All the versions will be executed simultaneously and the correct output chosen.

Telecommunications systems achieve high levels of reliability by implementing detection and recovery mechanisms with high coverage. With the trend towards the use of more COTS components in these systems the choices available for the systems detection and recovery mechanisms are more limited. An escalating recovery model with varying coverage factors and recovery durations is developed to provide insight into high availability design alternatives for commercial products. This work extends our previous examination of escalating detection by considering recovery.

VeriSoft is a tool for systematically exploring the state spaces of systems composed of several concurrent processes executing arbitrary code written in full-fledged programming languages such as C or C++. The state space of a concurrent system is a directed graph that represents the combined behavior of all concurrent components in the system. By exploring its state space, VeriSoft can automatically detect coordination problems between the processes of a concurrent system.We report in this paper our analysis with VeriSoft of the "Heart-Beat Monitor" (HBM), a telephone switching application developed at Lucent Technologies. The HBM of a telephone switch determines the status of different elements connected to the switch by measuring propagation delays of messages transmitted via these elements. This information plays an important role in the routing of data in the switch, and can significantly impact switch performance.We discuss the steps of our analysis of the HBM using VeriSoft. Because no modeling of the HBM code is necessary with this tool, the total elapsed time before being able to run the first tests was on the order of a few hours, instead of several days or weeks that would have been needed for the (error-prone) modeling phase required with traditional model checkers or theorem provers.We then present the results of our analysis. Since VeriSoft automatically generates, executes and evaluates thousands of tests per minute and has complete control over nondeterminism, our analysis revealed HBM behavior that is virtually impossible to detect or test in a traditional lab-testing environment. Specifically, we discovered flaws in the existing documentation on this application and unexpected behaviors in the software itself. These results are being used as the basis for the redesign of the HBM software in the next commercial release of the switching software.