This paper demonstrates that end-to-end active measurements can give invaluable insights into the nature and characteristics of packet loss in cellular networks. We conduct a large-scale measurement study of packet loss in four UMTS networks. The study is based on active measurements from hundreds of measurement nodes over a period of one year. We find that a significant fraction of loss occurs during pathological and normal Radio Resource Control (RRC) state transitions. The remaining loss exhibits pronounced diurnal patterns and shows a relatively strong correlation between geographically diverse measurement nodes. Our results indicate that the causes of a significant part of the remaining loss lie beyond the radio access network.

Mobile broadband networks play an increasingly important role in society, and there is a strong need for independent assessments of their robustness and performance. A promising source of such information is active end-to-end measurements. It is, however, a challenging task to go from individual measurements to an assessment of network reliability, which is a complex notion encompassing many stability and performance related metrics. This paper presents a framework for measuring the user-experienced reliability in mobile broadband networks. We argue that reliability must be assessed at several levels, from the availability of the network connection to the stability of application performance. Based on the proposed framework, we conduct a large-scale measurement study of reliability in 5 mobile broadband networks. The study builds on active measurements from hundreds of measurement nodes over a period of 10 months. The results show that the reliability of mobile broadband networks is lower than one could hope: more than 20% of connections from stationary nodes are unavailable more than 10 minutes per day. There is, however, a significant potential for improving robustness if a device can connect simultaneously to several networks. We find that in most cases, our devices can achieve 99.999% ("five nines") connection availability by combining two operators. We further show how both radio conditions and network configuration play important roles in determining reliability, and how external measurements can reveal weaknesses and incidents that are not always captured by the operators' existing monitoring tools.

Over the last decade, the Internet has grown at a tremendous speed in both size and complexity. Nowadays, a large number of important services – for instance e-commerce, healthcare and many others – depend on the availability of the underlying network. Clearly, service interruptions due to network problems may have a severe impact. On the long way towards the Future Internet, the complexity will grow even further. Therefore, new ideas and concepts must be evaluated thoroughly, and particularly in realistic, real-world Internet scenarios, before they can be deployed for production networks. For this purpose, various testbeds – for instance PlanetLab, GpENI or G-Lab – have been established and are intensively used for research. However, all of these testbeds lack the support for so-called multi-homing. Multi-homing denotes the connection of a site to multiple Internet service providers, in order to achieve redundancy. Clearly, with the need for network availability, there is a steadily growing demand for multi-homing. The idea of the NorNet Core project is to establish a Future Internet research testbed with multi-homed sites, in order to allow researchers to perform experiments with multi-homed systems. Particular use cases for this testbed include realistic experiments in the areas of multi-path routing, load balancing, multi-path transport protocols, overlay networks and network resilience. In this paper, we introduce the NorNet Core testbed as well as its architecture.

We present Nornet Edge (NNE), a dedicated infrastructure for measurements and experimentation in mobile broadband networks. NNE is unprecedented in size, consisting of more than 400 measurement nodes geographically distributed all over Norway. Each measurement node is a Linux-based embedded computer, and is connected to multiple mobile broadband providers. In addition, NNE includes an extensive backend system for deploying and managing experiments and collecting data. NNE makes it possible to run long-term measurement experiments to assess and compare quality and performance across different network operators on a national scale. Particular focus is put on allowing experiments to run in parallel on multiple network connections, and on collecting rich context information related to the experiments. In this paper we give a detailed presentation of NNE, and describe three different measurement experiments that illustrate how the infrastructure can be used. We also provide a roadmap for further development of NNE.

Some network applications have requirements that exceed the service levels offered by the best-effort Internet. Several network-layer Quality of Service architectures with extended service levels have been designed, but the massive scale and distributed nature of the Internet have prohibited their wide deployment. It now seems clear that the special needs of demanding applications must be met through other approaches. This paper describes how incrementally-deployed innovative solutions at the network level can contribute to an improved service for a particular type of applications, namely high-quality, wide-area video conferencing. We have built a global IP network, aiming to give users a better video conferencing experience primarily through packet loss reduction in transport networks. The key concept in our approach is a well-provisioned network-layer overlay, combined with geography-based "cold potato" BGP routing. Through an extensive set of experiments we show how our design choices impact routing and data plane behavior in the network, and demonstrate that we are able to significantly reduce packet loss compared to wide-area transport through global transit providers.

A prime objective of fault tolerant routing methods is the availability of multiple routing options at each hop. The methods that are currently implemented in IP networks, such as Equal-Cost Multi-Path (ECMP) and Loop Free Alternates (LFA), share the following four properties: First, they work with a hop-by-hop forwarding strategy optimized for the fault free case. Second, they do not require information associated with network faults included in the packet header. Third, they do not form forwarding loops, even under multiple failures in the network. Finally, they are compatible with standard link state routing protocols. However, ECMP and LFA give very poor fault coverage; in most cases fewer than 50% of primary next-hops are protected when using typical link weight settings. This paper presents a new routing method that combines the concept of permutation routing with interface-specific forwarding. Our method results in a routing strategy that strictly adheres to the four stated design properties. Through experiments we show that we protect more than 97% of the primary next-hops for all tested ISP networks.

P2P networking is a popular technology for sharing large files efficiently without using powerful servers. The goal for a P2P file sharing application is to minimize the download time experienced by the peers. However, the efficiency of a P2P network comes at the expense of excessive network utilization. ISPs want to control the network utilization of P2P applications, and in particular it is desirable to minimize the inter-ISP traffic. The often contradicting interests of applications and network owners have led to the search for an ISP-friendly P2P application that reduces inter-ISP traffic while maintaining the high file-sharing efficiency. Existing solutions propose to localize most of the P2P traffic within the ISP, while randomly selecting a few external peers to ensure the global spread of content. However, these proposals disregard the inherent randomness that will naturally exist in P2P systems. In this paper, we analyze the different sources of inherent randomness in ISP-friendly P2P systems, and show that they can have a significant impact on performance, and therefore must be taken into account when designing new solutions.

The scalability limitations of BGP have been a major concern lately. An important aspect of this issue is the rate of routing updates (churn) that BGP routers must process. This paper presents an analysis of the evolution of churn in four networks at the backbone of the Internet over a period of seven years and eight months, using BGP update traces from the RouteViews project. The churn rate varies widely over time and between networks. Instead of descriptive “black-box” statistical analysis, we take an exploratory data analysis approach attempting to understand the reasons behind major observed characteristics of the churn time series. We find that duplicate announcements are a major churn contributor, responsible for most large spikes. Remaining spikes are mostly caused by routing incidents that affect a large number of prefixes simultaneously. More long-term intense periods of churn, on the other hand, are caused by misconfigurations or other special events at or close to the monitored autonomous system (AS). After filtering pathologies and effects that are not related to the long-term evolution of churn, we analyze the remaining “baseline” churn and find that it is increasing at a rate that is similar to the growth of the number of ASs.

This paper develops a new resilient multipath routing technique, referred to as SPT-DAG, that has the following characteristics: (1) provides multipath routing over directed acyclic graphs (DAG); (2) the shortest path tree is guaranteed to be part of the DAG; and (3) provides guaranteed recovery from single link failures. We develop polynomial time algorithm to compute the SPT-DAG and a routing protocol to forward packets over the SPT-DAG using one overhead bit in the packet. We consider different load-balancing approaches for forwarding a packet when the packet has not encountered a failure. Through extensive flow-based simulations, we show that the SPT-DAG performs significantly better than those approaches that exclusively target load balancing or resiliency.

In order to reduce the number of BGP updates that routers need to process, it is common to rate-limit such updates using a timer that specifies the minimum time between two consecutive updates for a given destination prefix. Rate-limiting plays an important role in determining the number of routing updates that are generated after a routing event, and the time it takes before the network converges to a new stable state. Still, there are few guidelines for how rate-limiting timers should be configured in order to achieve the desired convergence properties. This work takes a first step in this direction, by exploring how different rate-limiting implementations and configurations affect the resulting churn level in a live BGP session. Measurements are performed on multiple parallel BGP sessions to a stub AS, configured with and without rate-limiting timers. We find that the daily rate of updates is reduced by two thirds when configuring the timer to the default value recommended by BGP standards. We further investigate different rate-limiting implementations and configurations using the measured BGP update patterns on emulated BGP sessions, and find that increasing the rate-limiting timer gives a logarithmic decrease in churn. Finally, using BGP update traces from RouteViews, we present the first empirical model that quantifies the impact of rate-limiting in terms of churn reduction given the observed arrival pattern of BGP updates.

The scalability limitations of BGP have been a major concern in the networking community lately. An important issue in this respect is the rate of routing updates (churn) that BGP routers must process. This paper presents an analysis of the evolution of churn in four networks in the backbone of the Internet over the last six years, using update traces from the Routeviews project. The churn rate varies widely over time and between networks, and cannot be understood through "black-box'' statistical analysis. Instead we take a different approach with a focus on investigating the underlying reasons for BGP churn evolution. Through our analysis we are able to identify and isolate the main reasons behind many of the anomalies in the churn time series. We find that duplicate announcements is a major churn contributor, and responsible for most large spikes in the churn time series. Other intense periods of churn are caused by misconfigurations or other special events in or close to the monitored AS, and hence limiting these is an important mean to limit churn. We then analyze the remaining "baseline'' churn, and find that it is increasing with a rate much slower than the increase in the routing table size.

The scalability of BGP routing is a major concern for the Internet community. Scalability is an issue in two different aspects: increasing routing table size, and increasing rate of BGP updates. In this paper, we focus on the latter. Our objective is to characterize the churn increase experienced by ASes in different levels of the Internet hierarchy as the network grows. We look at several "what-if" growth scenarios that are either plausible directions in the evolution of the Internet or educational corner cases, and investigate their scalability implications and interaction with different failure types. Our findings explain the dramatically different impact of multihoming and peering on BGP scalability, highlight negative and positive effects of multihoming on churn and reachability, and identify which topological growth scenarios will lead to faster churn increase for different failure types.

This paper develops novel mechanisms for recovering from failures in IP networks with proactive backup path calculations and Internet Protocol (IP) tunneling. The primary scheme provides resilience for up to two link failures along a path. The highlight of the developed routing approach is that a node reroutes a packet around the failed link without the knowledge of the second link failure. The proposed technique requires three protection addresses for every node, in addition to the normal address. Associated with every protection address of a node is a protection graph. Each link connected to the node is removed in at least one of the protection graphs, and every protection graph is guaranteed to be two-edge-connected. The network recovers from the first failure by tunneling the packet to the next-hop node using one of the protection addresses of the next-hop node; the packet is routed over the protection graph corresponding to that protection address. We prove that it is sufficient to provide up to three protection addresses per node to tolerate any arbitrary two link failures in a three-edge-connected graph. An extension to the basic scheme provides recovery from single-node failures in the network. It involves identification of the failed node in the packet path and then routing the packet to the destination along an alternate path not containing the failed node. The effectiveness of the proposed techniques were evaluated by simulating the developed algorithms over several network topologies.

We propose a routing and load-balancing approach with the primary goal of being robust to sudden topological changes and significant traffic matrix variations. The proposed method load-balances traffic over several routes in an adaptive way based on its local view of the load in the network. The focus is on robustness and simplicity, rather than optimality, and so it does not rely on a given traffic matrix, nor it is tuned to a specific topology. Instead, we aim to achieve a satisfactory routing under a wide range of traffic and topology scenarios based on each node's independent operation. The scheme avoids the instability risks of previous load-responsive routing schemes, it does not load the control plane with congestion-related signaling, and it can be implemented on top of existing routing protocols. In this paper, we present the proposed scheme, discuss how it aims to meet the objectives of robustness and load-responsiveness, and evaluate its performance under diverse traffic loads and topological changes with flow-level simulations.

This paper develops a novel mechanism for recovering from dual link failures in IP networks. The highlight of the developed routing approach is that a node re-routes a packet around the failed link without the knowledge of the second link failure. The proposed technique requires three protection addresses for every node, in addition to the normal address. Associated with every protection address of a node is a protection graph. Each link connected to the node is removed in at least one of protection graphs and every protection graph is guaranteed to be two-edge connected. The network recovers from the first failure by tunneling the packet to the next-hop node using one of the protection addresses of the next-hop node; and the packet is routed over the protection graph corresponding to that protection address. We prove that it is sufficient to provide up to three protection addresses per node to tolerate any arbitrary two link failures in a three-edge connected graph. We evaluate the effectiveness of the proposed technique over several network topologies.

Multi-topology routing is an increasingly popular IP network management concept that allows transport of different traffic types over disjoint network paths. The concept is of particular interest for implementation of IP fast reroute (IP FRR). The authors have previously proposed an IP FRR scheme based on multi-topology routing called Multiple Routing Configurations (MRC). MRC supports guaranteed, instantaneous recovery from any single link or node failure in biconnected networks as well as from many combined failures, provided sufficient bandwidth on the surviving links. Furthermore, in MRC different failures result in routing over different network topologies, which gives a good control of the traffic distribution in the networks after a failure

The scalability of BGP routing is a major concern for the Internet community. Scalability is an issue in two different aspects: increasing routing table size, and increasing rate of BGP updates. In this paper, we focus on the latter. Our objective is to characterize the churn increase experienced by ASes in different levels of the Internet hierarchy as the network grows. We look at several what-if growth scenarios that are either plausible directions in the evolution of the Internet or educational corner cases, and investigate their scalability implications. In addition, we examine the effect of the BGP update rate-limiting timer (MRAI), considering both major variations with which it has been deployed. Our findings explain the dramatically different impact of multi-homing and peering on BGP scalability, identify which topological growth scenarios will lead to faster churn increase, and emphasize the importance of not rate-limiting explicit withdrawals (despite what RFC-4271 recently required).

Multi-topology routing is an increasingly popular IP network management concept that allows transport of different traffic types over disjoint network paths. The concept is of particular interest for implementation of IP fast reroute (IP FRR). First, it can support guaranteed, instantaneous recovery from any link or node failure. Second, different failures result in routing over different network topologies, which augments the parameter space for load distribution optimizations. Multiple routing configurations (MRC) is the state-of-the-art IP FRR scheme based on multi-topology routing today. In this paper we present a new, enhanced IP FRR scheme which we call ldquorelaxed MRCrdquo (rMRC). rMRC simplifies the topology construction and increases the routing flexibility in each topology. According to our experimental evaluation, rMRC has several benefits compared to MRC. The number of backup topologies required to provide protection against the same set of failures is reduced, hence reducing state in routers. In addition, the backup paths are shorter, and the link utilization is significantly better.

The slow convergence of IGP routing protocols after a topology change has led to several proposals for proactive recovery schemes in IP networks. These proposals are limited to guaranteeing loop-free connectivity after a link or node failure, and do not take into account the resulting load distribution in the network. This can lead to congestion and packet drops. In this work, we show how a good load distribution can be achieved in pure IP networks immediately after a link failure, when multiple routing configurations (MRC) is used as a fast recovery mechanism. This paper is the first attempt to improve the load balancing when a proactive recovery scheme is used. Unlike load balancing methods used with normal IP rerouting, our method does not compromise on the routing performance in the failure free case. Our method is evaluated using simulations on several real and synthetically generated network topologies. The evaluation shows that our method yields good routing performance, making it feasible to use MRC to handle transient network failures.

Abstract — As the Internet takes an increasingly central role in our communications infrastructure, the slow convergence of routing protocols after a network failure becomes a growing problem. To assure fast recovery from link and node failures in IP networks, we present a new recovery scheme called Multiple Routing Configurations (MRC). MRC is based on keeping additional routing information in the routers, and allows packet forwarding to continue on an alternative output link immediately after the detection of a failure. Our proposed scheme guarantees recovery in all single failure scenarios, using a single mechanism to handle both link and node failures, and without knowing the root cause of the failure. MRC is strictly connectionless, and assumes only destination based hop-by-hop forwarding. It can be implemented with only minor changes to existing solutions. In this paper we present MRC, and analyze its performance with respect to scalability, backup path lengths, and load distribution after a failure. I.

Most existing methods for network recovery are often complex and seldom used by network administrators. In this paper we present a novel approach for global and local recovery named resilient routing layers (RRL). The method is supported by algorithms, but also simple enough for a network administrator to implement by hand for reasonably sized networks. The idea in our approach is that for each node in the network there is a topology subset called a "safe layer", which can handle any traffic affected by a fault in the node itself, or any of its links. We demonstrate that our approach performs well compared to other comparable methods in a wide range of different network topologies. Particularly, we demonstrate RRLs performance for what are assumed to be the weakest parameters for our method, i.e., backup-path lengths and state information overhead. We discuss implementation issues of RRL, and demonstrate its applicability to MPLS networks.

Resilient packet ring (RPR, IEEE std. 802.17-2004) is a recent networking standard developed by the IEEE LAN/MAN working group. RPR is an insertion buffer, dual ring technology, utilizing a back pressure based fairness algorithm to distribute bandwidth when congestion occurs. In its attempt to control a set of nodes' sending behavior over a congested link, the RPR fairness algorithm suffers from two severe performance deficiencies. The first concerns how the node closest to a congested link calculates a fair rate estimate, the second deficiency relates to the method used to distribute this fair rate estimate to nodes upstream from the congested node. In this paper, we analyze these deficiencies and propose improvements to resolve them.

Resilient packet ring (RPR) is a new standard, designated IEEE standard number 802.17, for MAN and WAN dual ring topologies. RPR uses the buffer insertion principle as a basis for its medium access control protocol. In this paper, we analyze parts of the aggressive mode of the RPR fairness protocol. We look at a congested node, and utilize control systems theory to analyze the stability of the associated fairness algorithm. In particular, we discuss how the settings of the two important parameters ageCoef and IpCoef influence the stability of an RPR-network. At the end of the paper we present simulated scenarios in order to illustrate our results.

We present a novel scheme for network recovery, named resilient routing layers (RRL). Our proposed scheme is based on calculating fully connected topology subsets, termed layers, which are used to forward traffic in case of a network failure. For the purpose of this work, the layers are created to protect against link failures only. RRL keeps pre-calculated backup routing information in the network stations. This allows local response to network failures, which gives recovery in the order of milliseconds. The main strengths of our approach are its flexibility, as it is independent of the network technology used, and its simplicity, as it offers the network operator a simple and coherent view of the resources available after a link failure. We also show that our scheme scales well for networks of several hundred nodes.

Resilient packet ring (RPR, IEEE 802.17) is designed with a protection mechanism aiming at restoring traffic on the ring within 50 ms in case of a link or node failure. However, the total experienced disruption time often exceeds this if strict order delivery of packets is required, since a 40 ms (configurable from 10 ms to 100 ms) topology stabilisation timer is used to avoid packet reordering. In this article, we present three alternative ways to avoid packet reordering for strict order traffic in RPR networks. The three suggested methods are discussed analytically, and simulations are used to compare their performance. Our simulation results suggest that the new methods give up to 90% reduction in packet loss during a failure situation compared to the current RPR standard