This article is an eight-year retrospective on development priorities for RocksDB, a key-value store developed at Facebook that targets large-scale distributed systems and that is optimized for Solid State Drives (SSDs). We describe how the priorities evolved over time as a result of hardware trends and extensive experiences running RocksDB at scale in production at a number of organizations: from optimizing write amplification, to space amplification, to CPU utilization. We describe lessons from running large-scale applications, including that resource allocation needs to be managed across different RocksDB instances, that data formats need to remain backward- and forward-compatible to allow incremental software rollouts, and that appropriate support for database replication and backups are needed. Lessons from failure handling taught us that data corruption errors needed to be detected earlier and that data integrity protection mechanisms are needed at every layer of the system. We describe improvements to the key-value interface. We describe a number of efforts that in retrospect proved to be misguided. Finally, we describe a number of open problems that could benefit from future research.

The usage and number of available video conferencing (VC) applications are rising as the high-bandwidth, low latency networks on which they depend become increasingly prevalent. Since VC applications support real-time human interaction, problems with performance that impair interactivity are social issues. Currently, performance measurements cannot easily be obtained due to the proprietary nature of VC applications, however, such measurements would be useful because they enable researchers to easily evaluate the performance impact of architectural and design decisions, quantitatively compare VC applications, and determine service level agreement (SLA) compliance. In this paper, we present a tool called Av Cloak that is capable of measuring several key performance metrics in proprietary VC applications: mouth-to-ear latency and jitter, capture-to-display latency and jitter, and audio-visual synchronization skew. AvCloak takes these measurements by wrapping ("cloaking") the VC application's audio/video inputs/outputs and transmitting timestamp data through them. At the sender side, AvCloak synthesizes media data encoding timestamps and feeds them to the VC application's media inputs, while at the receiver side, AvCloak decodes timestamps from the VC application's media outputs. Since AvCloak interacts with the target VC application only through its media inputs and outputs, it treats the target application as a black box and is thus applicable to arbitrary VC applications. We provide extensive analyses to measure AvCloak's overhead and show how to improve accuracy in measurements using two popular VC applications: Skype and Google+ Hangouts.

One type of Peer-to-Peer (P2P) live streaming has not yet been significantly investigated, namely topologies that provide many-to-many, interactive connectivity. Exemplar applications of such P2P systems include spatial audio services for networked virtual environments (NVEs) and distributed online games. Numerous challenging problems have to be overcome-among them providing low delay, resilience to churn, effective load balancing, and rapid convergence-in such dynamic environments. We propose a novel P2P overlay dissemination mechanism, termed IDM, that can satisfy such demanding real-time requirements. Our target application is to provide spatialized voice support in multiplayer NVEs, where each bandwidth constrained peer potentially communicates with all other peers within its area-of-interest (AoI). With IDM each peer maintains a set of partners, termed helpers, which may act as stream forwarders. We prove analytically that the system reachability is maximized when the loads of helpers are balanced proportionally to their network capacities. We then propose a game-theoretic algorithm that balances the loads of the peers in a fully distributed manner. Of practical importance in dynamic systems, we prove that our algorithm converges to an approximately balanced state from any prior state in rapid O(\log \log n) time, where n is the number of users. We further evaluate our technique with simulations and show that it can achieve near optimal system reachability and satisfy the tight latency constraints of interactive audio under conditions of churn, avatar mobility, and heterogeneous user access network bandwidth.