The open radio access network (O-RAN) architecture provides enhanced opportunities for integrating machine learning in 5G/6G resource management by decomposing RAN functionalities. Yet, generic learning mechanisms either do not fully exploit the disaggregated non-real-time and near-real-time RAN controllers or ignore the potential elasticity of application demands, another degree of freedom in managing RAN resources. We introduce a two-timescale framework aimed at optimizing users’ long-term total QoS. Rather than reactive resource allocation, our approach proactively modifies multi-resource user demands using congestion indicators, prior to enforcing any allocation rules. Addressing the issue of insufficient user feedback on individual resource utilities, we employ a bandit-feedback version of the combinatorial multi-armed bandit framework to deduce resource-specific signals. Also, to compensate for insufficient and infrequent feedback, we’ve developed an algorithm that gleans side information from live network traffic to refine predictions on user resource sensitivities. This streamlines the algorithm’s optimality convergence and leverages the two-tier O-RAN controller structure. We validate our algorithms’ efficacy through analysis and 5G usage experiments, revealing our proposed method improves application utility by 13-60%, throughput by 8-19%, and reduces latency by 10-18%.

Many geo-distributed services at web-scale companies still rely on databases (DBs) primarily optimized for single-site performance. At AT&T this is exemplified by services in the network control plane that rely on third-party software that uses DBs like MariaDB and PostgreSQL, which do not provide strict serializability across sites without a significant performance impact. Moreover, it is often impractical for these services to re-purpose their code to use newer DBs optimized for geo-distribution. In this paper, a novel drop-in solution for DB clustering across sites called Metric is presented that can be used by services without changing a single line of code. Metric leverages the single-site performance of an existing service's DB and combines it with a cross-site clustering solution based on an entry-consistent redo log that is specifically tailored for geo-distribution. Detailed correctness arguments are presented and extensive evaluations with various benchmarks show that Metric outperforms other solutions for the access patterns in our production use-cases where service replicas access different tables on different sites. In particular, Metric achieves up to 56% less latency and 5.2x higher throughput than MariaDB and PostgreSQL clustering, and up to 90% less latency and 26x higher throughput than CockroachDB and TiDB, systems that are designed to support geo-distribution.

Recent studies have reported that big data analytics clusters, such as Hadoop, can create substantial power peaks, bringing instability and inflexibility issues to the power grid. Substantial power peaks also lead to high penalty charges from electric utility companies, accounting for more than 30 percent of the electricity bill for a cluster operator according to empirical studies. To this end, in this paper, we present a framework that schedules computing jobs in large-scale data analytics clusters to mitigate power peaks. The scheduling model captures important properties of modern distributed data analytics clusters, including bundled resource provisioning and job-to-task decomposition with distributed processing. The scheduling problem is formulated as a nonlinear integer program. Its solution is derived by decomposing it into two classes of sub-problems and solving each class with an exact and efficient solution method. As a direct application, we detail the implementation of our proposed scheduling framework on a Hadoop cluster, and demonstrate its efficacy by extensive trace-driven simulations based on the CloudSim simulator.

Finding nodes which match certain criteria, based on potentially highly dynamic information, is a critical need in many distributed systems, ranging from cloud management, to network service deployments, to emerging IoT applications. With the increasing scale, dynamicity, and richness of data, existing systems, which typically implement a custom solution based around message queues where nodes push status to a central database, are ill-suited for this purpose. In this paper, we present FOCUS, a general and scalable service which easily integrates into existing and emerging systems to provide this fundamental capability. FOCUS utilizes a gossip-based protocol for nodes to organize into groups based on attributes and current value. With this approach, nodes need not synchronize with a central database, and instead the FOCUS service only needs to query the sub-set of nodes which have the potential to positively match a given query. We show FOCUS's flexibility through an operational example of complex querying for Virtual Network Functions instantiation over cloud sites, and illustrate its ease of integration by replacing the push-based approach in OpenStack's placement service. Our evaluation demonstrates a 5-15× reduction in bandwidth consumption and an ability to scale much better than existing approaches.

Distributed stream big data analytics platforms have emerged to tackle the continuously generated data streams. In stream big data analytics, the data processing workflow is abstracted as a directed graph referred to as a topology. Data are read from the storage and processed tuple by tuple, and these processing results are updated dynamically. The performance of a topology is evaluated by its throughput. This paper proposes an efficient resource allocation scheme for a heterogeneous stream big data analytics cluster shared by multiple topologies, in order to achieve max-min fairness in the utilities of the throughput for all the topologies. We first formulate a novel resource allocation problem, which is a mixed 0-1 integer program. The NP-hardness of the problem is rigorously proven. To tackle this problem, we transform the non-convex constraint to several linear constraints using linearization and reformulation techniques. Based on the analysis of the problem-specific structure and characteristics, we propose an approach that iteratively solves the continuous problem with a fixed set of discrete variables optimally, and updates the discrete variables heuristically. Simulations show that our proposed resource allocation scheme remarkably improves the max-min fairness in utilities of the topology throughput, and is low in computational complexity.

Given a set of files that show a certain degree of similarity, we consider a novel problem of performing data redundancy elimination across a set of distributed worker nodes in a shared-nothing in-memory big data analytic system. The redundancy elimination scheme is designed in a manner that is: (i) space-efficient: the total space needed to store the files is minimized and, (ii) access-isolation: data shuffling among server is also minimized. In this paper, we first show that finding an access-efficient and space optimal solution is an NP-Hard problem. Following this, we present the file partitioning algorithms that locate access-efficient solutions in an incremental manner with minimal algorithm time complexity (polynomial time). Our experimental verification on multiple data sets confirms that the proposed file partitioning solution is able to achieve compression ratio close to the optimal compression performance achieved by a centralized solution.

A cloud computing cluster equipped with a deadline-aware job scheduler faces fairness and efficiency challenges when greedy users falsely advertise the urgency of their jobs. Penalizing such untruthfulness without demotivating users from using the cloud service calls for advanced mechanism design techniques that work together with deadline-aware job scheduling. We propose a Bayesian incentive compatible pricing mechanism based on matching by replica-surrogate valuation functions. User valuations can be discovered by the mechanism, even when the users themselves do not fully understand their own valuations. Furthermore, users who are charged a Bayesian incentive compatible price have no reason to lie about the urgency of their jobs. The proposed mechanism achieves multiple desired truthful properties such as Bayesian incentive compatibility and ex-post individual rationality. We implement the proposed pricing mechanism. Through experiments in a Hadoop cluster with real-world datasets, we show that our prototype is capable of suppressing untruthful behavior from users.

We address the problem of scheduling jobs with utilities that depend solely upon their completion-times in a shared cloud that imposes considerable uncertainty on the jobs' runtime. However, it is very hard to estimate the jobs' runtime in a shared cloud where jobs are often delayed due to reasons such as slow I/O performance and variations in memory availability. Unlike prior works, we acknowledge that runtime estimates are often erroneous and instead shift the burden of robustness to the job scheduler. Specifically, we present a scheduling problem that jointly accounts for: (i) job utilities specified as functions of their completion-time, and (ii) uncertainty in the jobs' runtime. Our proposed solution to this problem achieves lexicographic max-min fairness among the job utilities. We implement this as a robust scheduler, named RUSH, for YARN in Hadoop. Our experiments, using real-world data sets, illustrate RUSH's efficacy when compared with other commonly used schedulers.

Consolidating virtual machine workload is a unique feature of cloud computing platforms that greatly reduces the operating cost of the cloud data center. Correctly consolidating VMs' workloads for a large scale cloud computing platform is nontrivial because a shortsighted scheme may save some cost in one aspect but becomes expensive in other aspects being neglected. In this paper, we present a framework that automates the VM consolidation process to improve the VMs and servers assignment whenever such improvement is possible. The proposed VM consolidation framework can achieve a balance among multiple administrative objectives (e.g., power cost, network cost) during the VM consolidation process. The solution method of solving the VM consolidation problem is designed based on the powerful and efficient semi quasi M-convex optimization framework. The proposed algorithm can also produce VM consolidation solutions that require minimal system reconfigurations (e.g., VM migrations, turning on/off servers). More importantly, the proposed algorithm can be implemented distributedly so that the scalability of the proposed framework is greatly improved. As a result, the proposed framework is efficient, scalable and highly practical.

There is an increasing need for cloud service performance that can be tailored to customer requirements. In the context of jobs submitted to cloud computing clusters, a crucial requirement is the specification of job completion-times. A natural way to model this specification, is through client/job utility functions that are dependent on job completion-times. We present a method to allocate and schedule heterogeneous resources to jointly optimize the utilities of jobs in a cloud. Specifically: (i) we formulate a completion-time optimal resource allocation (CORA) problem to apportion cluster resources across the jobs that enforces max-min fairness among job utilities, and (ii) starting with an integer programming problem, we perform a series of steps to transform it into an equivalent linear programming problem, and (iii) we implement the proposed framework as a utility-aware resource scheduler in the widely used Hadoop data processing framework, and finally (iv) through extensive experiments with real-world datasets, we show that our prototype achieves significant performance improvement over existing resource-allocation policies.

Frequency domain packet scheduling (FDPS) problem is one of the crucial elements in the 3rd generation partnership project (3GPP) long term evolution (LTE) system, which has a tremendous impact on the LTE system performance. However, optimal resource allocation through FDPS is a complicated and challenging task when multiple input and multiple output (MIMO)-orthogonal frequency division multiple access (OFDMA) technology is adopted. In this paper, we propose a new FDPS scheme for the 3GPP LTE downlink channels that optimally assigns time-frequency resources to users based on the proportional fairness metric. The proposed scheme jointly considers multiple system constraints imposed by the 3GPP LTE standard that arise in the MIMO mode selection and modulation and coding scheme (MCS) selection. Such scheduling problem is considered to be among the most difficult optimization problems. Our major contribution in this paper is that we model the FDPS problem as a unique binary linear programming problem and prove that the problem is totally unimodular. With the total unimodularity property, we are able to solve the complicated FDPS problem using efficient linear programming problem solvers. Simulation results show that our proposed scheme outperforms the existing state-of-the-art MIMO proportional fair scheduling algorithms used in the FDPS problem.

One of the most attractive features that computing clouds can offer is that the data center maintenance cost can be reduced by consolidating the workload generated by the virtual machines (VMs) into a few powerful physical servers. In this paper, the problem of maximally consolidating heterogeneous virtual machines into servers while protecting the service level agreement of each virtual machine is investigated. A generic and robust VM workload consolidation framework is proposed. The proposed framework achieves several attractive properties like low maintenance cost, low algorithm complexity, adjustable resource provisioning aggressiveness, and good scalability.

While the overall bandwidth of peer-to-peer live video streaming system scales automatically as peers collectively contribute the bandwidth, each peer also demands to download at the specified video playback rate so as to play the video smoothly. Therefore, a fundamental problem arisen is how to balance the bandwidth supply and demand in the peer-to-peer system to enjoy peers with the best video quality. To address this problem, we propose a fully distributed peer-to-peer video streaming framework which automatically adapts the network towards full bandwidth utilization. Our design possesses two unique features. First, a speciallink-level homogenousoverlay network is formed in which all the overlay links approach to have an identical bandwidth value. With such a feature, video flowing through the overlay links will not encounter any bottlenecks, and peers can thus achieve the guaranteed downloading rates. Second, based on the peer downloading rate observed locally at the streaming server, the server can adaptively adjust the video playback rate so that peers can achieve the best video quality with full bandwidth utilization. The effectiveness of our framework is verified through extensive simulations.