Resource isolation is a fundamental requirement in datacenter environments. However, our production experience in Meta’s large-scale datacenters shows that existing IO control mechanisms for block storage are inadequate in containerized environments. IO control needs to provide proportional resources to containers while taking into account the hardware heterogeneity of storage devices and the idiosyncrasies of the workloads deployed in datacenters. The speed of modern SSDs requires IO control to execute with low-overheads. Furthermore, IO control should strive for work conservation, take into account the interactions with the memory management subsystem, and avoid priority inversions that lead to isolation failures. To address these challenges, this paper presents IOCost, an IO control solution that is designed for containerized environments and provides scalable, work-conserving, and low-overhead IO control for heterogeneous storage devices and diverse workloads in datacenters. IOCost performs offline profiling to build a device model and uses it to estimate device occupancy of each IO request. To minimize runtime overhead, it separates IO control into a fast per-IO issue path and a slower periodic planning path. A novel work-conserving budget donation algorithm enables containers to dynamically share unused budget. We have deployed IOCost across the entirety of Meta’s datacenters comprised of millions of ma- chines, upstreamed IOCost to the Linux kernel, and open-sourced our device-profiling tools. IOCost has been running in production for two years, providing IO control for Meta’s fleet. We describe the design of IOCost and share our experience deploying it at scale.

Capacity reservation is a common offering in public clouds and on-premise infrastructure. However, no prior work provides capacity reservation with SLO guarantees that takes into account random and correlated hardware failures, datacenter maintenance, and heterogeneous hardware. In this paper, we describe how Facebook's region-scale Resource Allowance System (RAS) addresses these issues and provides guaranteed capacity. RAS uses a capacity abstraction called reservation to represent a set of servers dynamically assigned to a logical cluster. We take a two-level approach to scale resource allocation to all datacenters in a region, where a mixed-integer-programming solver continuously optimizes server-to-reservation assignments off the critical path, and a traditional container allocator does real-time placement of containers on servers in a reservation. As a relatively new component of Facebook's 10-year old cluster manager Twine, RAS has been running in production for almost two years, continuously optimizing the allocation of millions of servers to thousands of reservations. We describe the design of RAS and share our experience of deploying it at scale.

We present Taiji, a new system for managing user traffic for large-scale Internet services that accomplishes two goals: 1) balancing the utilization of data centers and 2) minimizing network latency of user requests.

Distributed actor systems are widely used for developing interactive scalable cloud services, such as social networks and on-line games. By modeling an application as a dynamic set of lightweight communicating "actors", developers can easily build complex distributed applications, while the underlying runtime system deals with low-level complexities of a distributed environment.

As the Internet becomes an important part of the infrastructure our society depends on, it is crucial to construct networks that are able to work even when part of the network is compromised. This paper presents the first practical intrusion-tolerant network service, targeting high-value applications such as monitoring and control of global clouds and management of critical infrastructure for the power grid. We use an overlay approach to leverage the existing IP infrastructure while providing the required resiliency and timeliness. Our solution overcomes malicious attacks and compromises in both the underlying network infrastructure and in the overlay itself. We deploy and evaluate the intrusion-tolerant overlay implementation on a global cloud spanning East Asia, North America, and Europe, and make it publicly available.

Networks with homogeneous routing nodes are constantly at risk as any vulnerability found against a node could be used to compromise all nodes. Introducing diversity among nodes can be used to address this problem. With few variants, the choice of assignment of variants to nodes is critical to the overall network resiliency. We present the Diversity Assignment Problem (DAP), the assignment of variants to nodes in a network, and we show how to compute the optimal solution in medium-size networks. We also present a greedy approximation to DAP that scales well to large networks. Our solution shows that a high level of overall network resiliency can be obtained even from variants that are weak on their own. We provide a variation of our problem that matches the specific communication requirements of applications run over the network (e.g., Paxos and BFT). Also, we analyze the loss in resiliency when optimally assigning variants based on inaccurate information about compromises.

Networks with homogeneous routing nodes are constantly at risk as any vulnerability found against a node could be used to compromise all nodes. Introducing diversity among nodes can be used to address this problem. With few variants, the choice of assignment of variants to nodes is critical to the overall network resiliency. We present the Diversity Assignment Problem (DAP), the assignment of variants to nodes in a network, and we show how to compute the optimal solution in medium-size networks. We also present a greedy approximation to DAP that scales well to large networks. Our solution shows that a high level of overall network resiliency can be obtained even from variants that are weak on their own. For real-world systems that grow incrementally over time, we provide an online version of our solution. Lastly, we provide a variation of our solution that is tunable for specific applications (e.g., BFT).

Recent work in defending against pollution attacks for intra-flow network coding systems proposed a null spaces based algebraic approach which has a smaller computation cost than previous pollution defenses. The approach requires the source to distribute keys periodically, but in order to scale involves forwarder nodes in the creation of new keys and their distribution. As a result the key distribution is secure only in specific network topologies such as those created by large-scale peer to peer systems, and is not secure in wireless networks where such topologies do not exist. We propose Split Null Keys, which splits the keys such that only a small portion of the key is updated periodically. The small updates allow for a scalable key distribution scheme that does not involve forwarder nodes in creating keys and thus does not rely its security on constraints imposed on the network topology. We prove that our scheme is secure despite splitting the key and we show that when compared with existing defenses our scheme imposes lower communication and computation overhead, is resilient to colluding adversaries, and does not require time synchronization.

Multihop wireless networks gain higher performance by using network coding. However, using network coding also introduces new attacks such as the well-studied pollution attacks and less-studied entropy attacks. Unlike in pollution attacks where an attacker injects polluted packets (i.e., packets that are not linear combinations of the packets sent by the source), in entropy attacks an attacker creates non-innovative packets (i.e., packets that contain information already known by the system). In both cases the result is a severe degradation of the system performance. In this paper, we identify two variants of entropy attacks (local and global) and show that while they share some characteristics with pollution attacks and selective forwarding, none of the techniques proposed to defend against such attacks are applicable to entropy attacks because the packets look legitimate and the packet forwarding is stealthy in nature. We propose and evaluate several defenses that vary in detection capabilities and overhead.

The increasing availability of low-cost battery-operated wireless cameras has motivated the deployment of large-scale Wireless Multimedia Sensor Networks (WMSNs) which can be leveraged for gathering disparate views of events from multiple perspectives. Such multi-perspective coverage not only provides better visual knowledge about the events but also helps reduce occlusions in many critical applications. Different than traditional k-coverage in Wireless Sensor Networks (WSNs), multi-perspective coverage computation considers the orientation of cameras in addition to their locations. In this paper, we first introduce a new metric which can measure multi-perspective coverage for a particular region from a given number of perspectives. Using this metric, we then propose camera placement techniques based on binary integer programming and heuristics to achieve full multi-perspective coverage with the least camera count. Finally, to be used as a baseline, we come up with a formula which can analytically compute the multi-perspective coverage for a given network of randomly placed cameras in a certain region. We evaluated the performance of these camera placement approaches (e.g., integer programming, heuristic and random) in terms of coverage and number of cameras needed under different number of perspectives.

With the increasing interest in the deployment of wireless multimedia sensor networks (WMSNs), new challenges arouse with effective use of camera sensors to provide maximized event coverage with the least amount of redundancy in the collected multimedia data. Given that the processing and transmission of multimedia data are costly in terms of energy, camera sensors should only be actuated when an event is detected within their vicinity. While achieving maximum coverage with such actuation is desirable, multiple camera sensors' field-of- view (FoV) can be covering the same spots and thus redundant multimedia data can unnecessarily be sent to the base-station. In this paper, assuming camera sensors with fixed orientation, we propose a low-cost distributed actuation scheme which strives to turn on the least number of camera sensors to avoid possible redundancy in the multimedia data while still providing the necessary event coverage. The basic idea of this distributed scheme is the collaboration of camera sensors that have heard from scalar sensors about an occurring event in order to minimize the possible coverage overlaps among their FoVs. The scheme requires only 1-hop information for camera sensors and its messaging overhead is negligible. Through simulation, we show how the distributed scheme performs with respect to the cases when all the cameras within the vicinity or the region are actuated and assess the performance under various conditions.