Cellular service providers (CSPs) require predicting the network performance for various reasons such as analyzing the impact of planned configuration changes and large-scale events on the network. Although network configurations are widely considered as key predictors of performance, we claim that they are insufficient for accurately predicting cellular network performance. The cellular networks are impacted by unmeasured external factors (e.g., weather, called latents), therefore, the performance prediction based solely on configurations may result in confounding effects. We show that the Mobility, Access, and Traffic (MAT) metrics should be considered in addition as network performance predictors. Using a large dataset collected from a live cellular network, we validate the claim and show the benefit of using MAT metrics for accurate performance prediction.

Cellular service providers (CSPs) aim to optimize network performance and enhance user experience by tuning network configurations. However, this process often requires continuous live network testing, which incurs significant operational costs. In this paper, we focus on predicting the impact of configuration changes using historical data, thereby reducing the need for live network tests. A key challenge in developing such a model is accounting for unobserved external factors (e.g., weather, referred to as latents) that can introduce confounding effects between configurations and performance metrics. To address this, we employ intermediate network metrics, called Mobility, Access, and Traffic (MAT) metrics, which are influenced by both configurations and latents, and in turn, affect performance metrics. We introduce Configuration Impact Prediction Analysis Toolkit (CIPAT), a novel two-stage toolkit developed using a comprehensive real-world dataset from live LTE networks. Our evaluation demonstrates that CIPAT enables CSPs to predict the performance impact of proposed configuration changes with up to 86% accuracy and 85% efficacy, thereby reducing the operational costs associated with configuration tuning.

Managing network configuration and improving service experience effectively is essential for cellular service providers (CSPs). This is challenging because of cellular networks' large scale and complexity, the wide variety of configuration parameters, and the performance impact tradeoffs resulting across multiple metrics and geographical locations. This paper focuses on learning and using network contexts to recommend performance-improving configurations. While learning contexts, one must carefully account for the configuration parameter dependency, performance impact confusion that can arise due to co-occurring unrelated changes, and uneven change deployment distribution across locations. We present a new solution Chroma that addresses the above challenges. Using real-world data collected from a large operational LTE and 5G cellular service provider, we thoroughly evaluate and demonstrate the efficacy of Chroma. We successfully trial Chroma on an operational cellular network and highlight its benefits in practical settings.

Cellular service operators frequently tune the network configuration to optimize coverage, support seamless handovers, minimize channel interference, and improve the service performance experience to the end-users. Tuning such a complicated network is highly challenging because of the many configuration parameters, evolving complexity of cellular networks, and diverse requirements across voice, video, and data applications. Any misconfigurations or even poor settings can significantly negatively impact service quality. In this paper, we propose a new approach Aurora that derives best practices knowledge from exploration of the massive existing configuration in the network and uses conformity-based recommendation with performance-based filtering to improve cellular service. We implemented and evaluated Aurora using data from a very large LTE and 5G cellular service provider. Our operational experience over the last one year highlights the benefits of Aurora and exposes exciting research opportunities and challenges in configuration tuning and performance management.

Networks undergo continuous changes to introduce new services and improve existing ones. Network change deployment involves carefully deciding when each change activity will be executed and who will be executing the change. This is a complex process because each service group has to plan its activities following a set of operational and technological constraints. Besides, multiple groups may be working on the same or dependent nodes at the same time, and they must coordinate their deployment plans. If they do not co-ordinate, conflicting change execution could result in unexpected impacts. Traditionally, change deployment has been a tedious and time-consuming task. To address this, we propose an innovative solution Zapper that aims for minimal human effort to coordinate the changes, minimal risk to service quality, and efficient plans to rapidly deploy the changes. Zapper maps change scheduling constraints into mathematical equations and then uses optimization algorithms to generate conflict-free change plans that satisfy all constraints across service groups. We have deployed Zapper at a large service provider and it is being used regularly by the network operations teams for more than two years to schedule over 4.5 million change activities.

Change management has been a long-standing challenge for network operations. The large scale and diversity of networks, their complex dependencies, and continuous evolution through technology and software updates combined with the risk of service impact create tremendous challenges to effectively manage changes. In this paper, we use data from a large service provider and experiences of their operations teams to highlight the need for quick and easy adaptation of change management capabilities and keep up with the continuous network changes. We propose a new framework CORNET (COmposition fRamework for chaNge managEmenT) with key ideas of modularization of changes into building blocks, flexible composition into change workflows, change plan optimization, change impact verification, and automated translation of high-level change management intent into low-level implementations and mathematical models. We demonstrate the effectiveness of CORNET using real-world data collected from 4G and 5G cellular networks and virtualized services such as VPN and SDWAN running in the cloud as well as experiments conducted on a testbed of virtualized network functions. We also share our operational experiences and lessons learned from successfully using CORNET within a large service provider network over the last three years.

Cellular service providers add carriers in the network in order to support the increasing demand in voice and data traffic and provide good quality of service to the users. Addition of new carriers requires the network operators to accurately configure their parameters for the desired behaviors. This is a challenging problem because of the large number of parameters related to various functions like user mobility, interference management and load balancing. Furthermore, the same parameters can have varying values across different locations to manage user and traffic behaviors as planned and respond appropriately to different signal propagation patterns and interference. Manual configuration is time-consuming, tedious and error-prone, which could result in poor quality of service. In this paper, we propose a new data-driven recommendation approach Auric to automatically and accurately generate configuration parameters for new carriers added in cellular networks. Our approach incorporates new algorithms based on collaborative filtering and geographical proximity to automatically determine similarity across existing carriers. We conduct a thorough evaluation using real-world LTE network data and observe a high accuracy (96%) across a large number of carriers and configuration parameters. We also share experiences from our deployment and use of Auric in production environments.

Performance impacts are commonly observed in cellular networks and are induced by several factors, such as software upgrade and configuration changes. The variability in traffic patterns across different granularities can lead to impact cancellation or dilution. As a result, performance impacts are hard to capture if not aggregated over problematic features. Analyzing performance impact across all possible feature combinations is too expensive. On the other hand, the set of features that causes issues is unpredictable due to the highly dynamic and heterogeneous cellular networks. In this paper, we propose a novel algorithm that dynamically explores those network feature combinations that are likely to have problems by using a summary structure Sketch. We further design a neural network based algorithm to localize root cause. We achieve high scalability in neural network by leveraging the Lattice and Sketch structure. We demonstrate the effectiveness of our impact detection and diagnosis through extensive evaluation using data collected from a major tier-1 cellular carrier in US and synthetic traces.

Traffic migration is a common procedure performed by operators during planned maintenance and unexpected incidents to prevent/reduce service disruptions. However, current practices of traffic migration often couple operators' intentions (e.g. device upgrades) with network setups (e.g. load-balancers), resulting in poor re-usability and substantial operational complexities. Our study of 205 Methods of Procedure (MOPs) from a major U.S. carrier suggests that generalizing traffic migration with a unified model is feasible. Such generalization along with SDN's automation capability is key to scalable and flexible management of traffic, especially for virtualized network functions with unprecedented scale, heterogeneity, and fast iteration. In this paper, we propose Egret, a generic traffic migration system that simplifies traffic management for physical and virtual network functions. Egret (1) hides intricate implementation details from operators with generic intention-based interfaces, and (2) modularizes common traffic migration procedures to enable plug-and-play by developers and vendors. Leveraging a novel mask-based abstraction of traffic migration jobs, Egret can further simplify reverse traffic migration and enable job interleaving.

Cellular service providers continuously deploy changes in their network in the form of new software releases, service feature introductions, configuration changes, equipment re-homes, firmware upgrades, and topology modifications. It is important to carefully assess the impact of these changes on service performance to validate expected behaviors and take mitigation actions in a timely fashion in case of any unexpected degradation. The diverse nature of the network changes, complex interactions across different layers of the cellular network, and the rapid evolution of the network make it challenging to accurately conduct the assessment. In this paper, we present the design and implementation of our system that enables rigorous, effortless and timely assessment of performance around network changes. We share our lessons learned from the deployment in an operational cellular network over the last eight years.

Cellular networks are constantly evolving due to frequent changes in radio access and end user equipment technologies, dynamic applications and associated trafflc mixes. Network upgrades should be performed with extreme caution since millions of users heavily depend on the cellular networks for a wide range of day to day tasks, including emergency and alert notifications. Before upgrading the entire network, it is important to conduct field evaluation of upgrades. Field evaluations are typically cumbersome and can be time consuming; however if done correctly they can help alleviate a lot of the deployment issues in terms of service quality degradation. The choice and number of field test locations have significant impacts on the time-to-market as well as confidence in how well various network upgrades will work out in the rest of the network. In this paper, we propose a novel approach — Reflection to automatically determine where to conduct the upgrade field tests in order to accurately identify important features that affect the upgrade. We demonstrate the effectiveness of Reflection using extensive evaluation based on real traces collected from a major US cellular network as well as synthetic traces.

Cellular service providers continuously upgrade their network software on base stations to introduce new service features, fix software bugs, enhance quality of experience to users, or patch security vulnerabilities. A software upgrade typically requires the network element to be taken out of service, which can potentially degrade the service to users. Thus, the new software is deployed across the network using a rolling upgrade model such that the service impact during the roll-out is minimized. A sequential roll-out guarantees minimal impact but increases the deployment time thereby incurring a significant human cost and time in monitoring the upgrade. A network-wide concurrent roll-out guarantees minimal deployment time but can result in a significant service impact. The goal is to strike a balance between deployment time and service impact during the upgrade. In this paper, we first present our findings from analyzing upgrades in operational networks and discussions with network operators and exposing the challenges in rolling software upgrades. We propose a new framework Concord to effectively coordinate software upgrades across the network that balances the deployment time and service impact. We evaluate Concord using real-world data collected from a large operational cellular network and demonstrate the benefits and tradeoffs. We also present a prototype deployment of Concord using a small-scale LTE testbed deployed indoors in a corporate building.

Cellular network providers collect and use a wide variety of data for assessing the service quality experienced by their smartphone users. The data is essential for tasks ranging from event detection, problem diagnosis, impact analysis, coverage and capacity planning, load balancing, and performance optimization. For example, service quality measurements and data from drive-by tests provide useful and detailed information about different aspects of quality of service such as dropped calls due to handovers or radio interference. However, a major challenge for effective service quality management in operational setup is the presence of missing or unavailable data. Furthermore, the cellular data is inherently multidimensional, i.e. is a function of several variables such as location, device type, and time. Motivated by recent advances in handling multidimensional data, we propose to use tensor algebraic models and methods for cellular data prediction. The main idea is to model the data as a low rank tensor and use a rank constrained interpolation for data prediction. We focus on two recently proposed algebraic models employing two different notions of tensor rank. We test and compare the performance of the two approaches on real-world data sets collected from an operational cellular network and indicate the regimes in which one method is superior to the other. Based on these observations the proposed algorithm chooses the best of the two approaches using cross-validation.

Smartphone manufacturers often release software upgrades to their users for improving service performance, patching security vulnerabilities, enhancing device stability, fixing bugs, increasing battery life, or even enriching the graphical user interface. It is crucial to monitor the smartphones after software upgrades to either confirm their expected impacts, or quickly identify any undesirable behaviors. In this paper, we focus on automatically detecting the software upgrades on smartphones and analyzing their service performance impacts. The complex interactions between the software on the smartphones and the cellular networks make it hard to differentiate if the impacts are smartphone-centric, or network-centric. We propose a new approach, SSM (Smartphone Specific Monitoring) for conducting pre/post impact analysis of multiple service performance metrics across smartphones aggregated by their type, make, model and network locations. Using one-year worth of operational network data, we demonstrate the effectiveness of SSM in accurately detecting and diagnosing the performance impact of smartphone software upgrades.

As smartphone users increasingly rely on cellular networks to access voice, video, and web applications, guaranteeing good performance and high availability is more important than ever. Historically, managing cellular network configuration has been a manual, error-prone process; recently, automated solutions such as SON (Self-Organizing Networks) controllers are being deployed for dynamic tuning of network configuration to improve end-user service performance under dynamic network and traffic conditions. SON automates many aspects of cellular network configuration, but it is nonetheless susceptible to software bugs and expected traffic changes that could result in sub-optimal performance. In this paper, we propose a capability (Veracity) to analyze and quantify the performance effects of SON actions. Assessing the effects of SON control is difficult because of the dynamic nature of SON and the dependency of end-user performance on factors such as radio channel quality, mobility and traffic load. Veracity addresses these using model-driven impact detection and quantification. Our evaluation using data collected from an operational cellular network demonstrates that Veracity is accurate. Veracity is now being used by the service providers' field operation teams for the assessment of SON effectiveness in arenas and stadiums.

Load on cellular towers is one of the key metrics that cellular service providers monitor as part of their operational and management tasks. Increased load on the towers can lead to congestion, which in turn can severely degrade quality of service perceived by users. Hence, it is of great interest to cellular service providers to minimize the maximum load at cell towers, and thereby minimize chances of congestion in the event of a sudden increase in load due to user demand changes. This goal can be achieved by proactive load balancing among neighboring cell towers, i.e., proactively identify opportunities to balance the load through re-binding of users from heavily loaded cell towers to lightly loaded neighboring towers. In this paper, we propose a new tool Libra to effectively assess the impact of load balancing related parameter changes. Libra provides an objective measure of the degree of load imbalance across multiple network locations and identifies if the measure improves or degrades after parameter changes. Our evaluation of Libra using real-world data collected from a large cellular provider demonstrates its effectiveness in accurately capturing the degree of imbalance at multiple cell towers.

Cellular networks are constantly evolving due to frequent changes in radio access and end user equipment technologies, applications, and traffic. Network upgrades should be performed with extreme caution since millions of users heavily depend on the cellular networks. Before upgrading the entire network, it is important to conduct field evaluation of upgrades.The choice and number of field test locations have significant impact on the time-to-market and confidence in how well various network upgrades will work out in the rest of the network. We propose a novel approach -- Reflection to automatically determine where to conduct the upgrade field tests to accurately identify important features that affect the upgrade and predict for the performance of untested locations. We demonstrate its effectiveness using real traces collected from a major US cellular network as well as synthetic traces.

Planned upgrades in cellular networks occur every day, may often need to be performed on weekdays, and can potentially degrade service for customers. In this paper, we explore the problem of tuning network configurations in order to mitigate any potential impact due to a planned upgrade which takes the base station off-air. The objective is to recover the loss in service performance or coverage which would have occurred without any modifications. To our knowledge, impact mitigation for planned base station downtimes has not been explored before in the literature. The primary contribution of this work is a proactive approach based on a predictive model that uses operational data of user density distributions and path loss (rather than idealized analytical models of these) to quickly estimate the best power and tilt configuration of neighboring base stations that enables high recovery. A secondary contribution is an approach to minimize synchronized handovers. These ideas, embodied in a capability called Magus, enables us to recover up to 76% of the potential performance loss due to planned upgrades in some cases for a large US mobile network, and this recovery varies as a function of base station density. Moreover, Magus is able to reduce synchronized handovers by a factor of 8.

Cellular network service providers often have to conduct small scale testing in the operational network before a change (e.g., a new feature) is fully rolled out across the entire network. This is referred to as the First Field Application (FFA). However, assessing the effectiveness of FFA changes is challenging because of overlapping external factors: seasonality (foliage, leaves budding), weather (rain, snow, hurricanes, storms), traffic pattern changes due to big events (e.g., games at stadiums, students returning to school after holidays), and network events such as outages or other maintenance activities in different regions. In this paper, we first highlight the technical challenges in assessing the service performance impact of changes in operational cellular networks. We then propose Litmus, a new approach based on a spatial dependency model for robust assessment of changes. We evaluate the effectiveness of Litmus using real-world data from operational cellular networks (GSM, UMTS and LTE). Our operational experiences demonstrate accurate inferences of the service performance impact of changes in the field.

Service quality in operational IP networks can be impacted due to planned or unplanned maintenance. During any maintenance activity, the responsibility of the operations team is to complete the work order and perform a check-up to ensure there are no unexpected service disruptions. Once the maintenance is complete, it is crucial to continuously monitor the network and look for any performance impacts. What operations lack today are effective tools to rapidly detect maintenance induced performance changes. The large scale and heterogeneity of network elements and performance metrics makes the problem extremely challenging.

Cloud service providers use replication across geographically distributed data centers to improve end-to-end performance as well as to offer high reliability under failures. Content replication often involves the transfer of huge data sets over the wide area network and demands high backbone transport capacity. In this paper, we discuss how a Globally Reconfigurable Intelligent Photonic Network (GRIPhoN) between data centers could improve operational flexibility for cloud service providers. The proposed GRIPhoN architecture is an extension of earlier work [34] and can provide a bandwidth-on-demand service ranging from low data rates (e.g., 1 Gbps) to high data rates (e.g., 10-40 Gbps). The inter-data center communication network which is currently statically provisioned could be dynamically configured based on demand. Today's backbone optical networks can take several weeks to provision a customer's private line connection. GRIPhoN would enable cloud operators to dynamically set up and tear down their connections (sub-wavelength or wavelength rates) within a few minutes. GRIPhoN also offers cost-effective restoration capabilities at wavelength rates and automated bridge-and-roll of private line connections to minimize the impact of planned maintenance activities.

In large-scale IPTV systems, it is essential to maintain high service quality while providing a wider variety of service features than typical traditional TV. Thus service quality assessment systems are of paramount importance as they monitor the user-perceived service quality and alert when issues occurs. For IPTV systems, however, there is no simple metric to represent user-perceived service quality and Quality of Experience (QoE). Moreover, there is only limited user feedback, often in the form of noisy and delayed customer calls. Therefore, we aim to approximate the QoE through a selected set of performance indicators in a proactive (i.e., detect issues before customers reports to call centers) and scalable fashion.

Networks continue to change to support new applications, improve reliability and performance and reduce the operational cost. The changes are made to the network in the form of upgrades such as software or hardware upgrades, new network or service features and network configuration changes. It is crucial to monitor the network when upgrades are made because they can have a significant impact on network performance and if not monitored may lead to unexpected consequences in operational networks. This can be achieved manually for a small number of devices, but does not scale to large networks with hundreds or thousands of routers and extremely large number of different upgrades made on a regular basis.

Network resiliency is crucial to IP network operations. Existing techniques to recover from one or a series of failures do not offer performance predictability and may cause serious congestion. In this paper, we propose Resilient Routing Reconfiguration (R3), a novel routing protection scheme that is (i) provably congestion-free under a large number of failure scenarios; (ii) efficient by having low router processing overhead and memory requirements; (iii) flexible in accommodating different performance requirements (e.g., handling realistic failure scenarios, prioritized traffic, and the trade-off between performance and resilience); and (iv) robust to both topology failures and traffic variations. We implement R3 on Linux using a simple extension of MPLS, called MPLS-ff. We then conduct extensive Emulab experiments and simulations using realistic network topologies and traffic demands. Our results show that R3 achieves near-optimal performance and is at least 50% better than the existing schemes under a wide range of failure scenarios.

IPTV is increasingly being deployed and offered as a commercial service to residential broadband customers. Compared with traditional ISP networks, an IPTV distribution network (i) typically adopts a hierarchical instead of mesh-like structure, (ii) imposes more stringent requirements on both reliability and performance, (iii) has different distribution protocols (which make heavy use of IP multicast) and traffic patterns, and (iv) faces more serious scalability challenges in managing millions of network elements. These unique characteristics impose tremendous challenges in the effective management of IPTV network and service. In this paper, we focus on characterizing and troubleshooting performance issues in one of the largest IPTV networks in North America. We collect a large amount of measurement data from a wide range of sources, including device usage and error logs, user activity logs, video quality alarms, and customer trouble tickets. We develop a novel diagnosis tool called Giza that is specifically tailored to the enormous scale and hierarchical structure of the IPTV network. Giza applies multi-resolution data analysis to quickly detect and localize regions in the IPTV distribution hierarchy that are experiencing serious performance problems. Giza then uses several statistical data mining techniques to troubleshoot the identified problems and diagnose their root causes. Validation against operational experiences demonstrates the effectiveness of Giza in detecting important performance issues and identifying interesting dependencies. The methodology and algorithms in Giza promise to be of great use in IPTV network operations.

Chronic network conditions are caused by performance impairing events that occur intermittently over an extended period of time. Such conditions can cause repeated performance degradation to customers, and sometimes can even turn into serious hard failures. It is therefore critical to troubleshoot and repair chronic network conditions in a timely fashion in order to ensure high reliability and performance in large IP networks. Today, troubleshooting chronic conditions is often performed manually, making it a tedious, time-consuming and error-prone process.

In this paper, we describe the design and evaluation of a scheduler (referred to as Everest) for allocating processors to services in high performance, multi-service routers. A scheduler for such routers is required to maximize the number of packets processed within a given delay tolerance, while isolating the performance of services from each other. The design of such a scheduler is novel and challenging because of three domain-specific characteristics: (1) difficult-to-predict and high packet arrival rates, (2) small delay tolerances of packets, and (3) significant overheads for switching allocation of processors from one service to another. These characteristics require that the scheduler be agile and wary simultaneously. Whereas agility enables the scheduler to react quickly to fluctuations in packet arrival rates, wariness prevents the scheduler from wasting computational resources in unnecessary context switches. We demonstrate that by balancing agility and wariness, Everest, as compared to conventional schedulers, reduces by more than an order of magnitude the average delay and the percentage of packets that experience delays greater than their tolerance. We describe a prototype implementation of Everest on Intel's IXP2400 network processor

Availability is a critical issue in modern distributed systems. While many techniques and protocols for preventing denial of service (DoS) attacks have been proposed and deployed in recent years, formal methods for analyzing and proving them correct have not kept up with the state of the art in DoS prevention. This paper proposes a new protocol for preventing malicious bandwidth consumption, and demonstrates how game-based formal methods can be successfully used to verify availability-related security properties of network protocols. We describe two classes of DoS attacks aimed at bandwidth consumption and resource exhaustion, respectively. We then propose our own protocol, based on a variant of client puzzles, to defend against bandwidth consumption, and use the JFKr key exchange protocol as an example of a protocol that defends against resource exhaustion attacks. We specify both protocols as alternating transition systems (ATS), state their security properties in alternating-time temporal logic (ATL) and verify them using MOCHA, a model checker that has been previously used to analyze fair exchange protocols.

Sensor networks include nodes with limited computation and communication capabilities. One of the basic functions of sensor networks is to sense and transmit data to the end users. The resource constraints and security issues pose a challenge to information aggregation in large sensor networks. Bootstrapping keys is another challenge because public key cryptosystems are unsuitable for use in resource-constrained sensor networks. In this paper, we propose a solution by dividing the problem in two domains. First, we present a protocol for establishing cluster keys in sensor networks using verifiable secret sharing. We chose elliptic curve cryptosystems for security because of their smaller key size, faster computations and reductions in processing power. Second, we develop a secure data aggregation and verification (SecureDAV) protocol that ensures that the base station never accepts faulty aggregate readings. An integrity check of the readings is done using Merkle hash trees, avoiding over-reliance on the cluster-heads.