Directed hypergraphs are vital for modeling complex polyadic relationships in domains such as discrete mathematics, computer science, network security, and systems modeling. However, their inherent complexity often impedes effective visualization and analysis, particularly for large graphs. This paper introduces a novel Transitivity Preserving Projection (TPP) to address the limitations of the computationally intensive Basu and Blanning projection (BBP), which can paradoxically increase complexity by flattening transitive relationships. TPP offers a minimal and complete representation of relationships within a chosen subset of elements, capturing only irreducible dominant metapaths to ensure the smallest set of edges while preserving all essential transitive and direct connections. This approach significantly enhances visualization by reducing edge proliferation and maintains the integrity of the original hypergraph’s structure. We develop an efficient algorithm leveraging the set-trie data structure, reducing the computational complexity from an exponential number of metapath searches in BBP to a linear number of metapath searches with polynomial-time filtering, enabling scalability for real-world applications. Experimental results demonstrate TPP’s superior performance, completing projections in seconds on graphs where BBP fails to terminate within 24 hours. By providing a minimal yet complete view of relationships, TPP supports applications in network security and supply chain analysis, offering a clearer, more efficient framework for hypergraph simplification and analysis.

IoT devices are increasingly being implicated in cyber-attacks, raising community concern about the risks they pose to critical infrastructure, corporations, and citizens. In order to reduce this risk, the IETF is pushing IoT vendors to develop formal specifications of the intended purpose of their IoT devices, in the form of a Manufacturer Usage Description (MUD), so that their network behavior in any operating environment can be locked down and verified rigorously. This article aims to assist IoT manufacturers in developing and verifying MUD profiles, while also helping adopters of these devices to ensure they are compatible with their organizational policies and track device network behavior using their MUD profile. Our first contribution is to develop a tool that takes the traffic trace of an arbitrary IoT device as input and automatically generates the MUD profile for it. We contribute our tool as open source, apply it to 28 consumer IoT devices, and highlight insights and challenges encountered in the process. Our second contribution is to apply a formal semantic framework that not only validates a given MUD profile for consistency, but also checks its compatibility with a given organizational policy. We apply our framework to representative organizations and selected devices, to demonstrate how MUD can reduce the effort needed for IoT acceptance testing. Finally, we show how operators can dynamically identify IoT devices using known MUD profiles and monitor their behavioral changes in their network.

Reliable network-policy specification requires abstractions that can naturally model policies together with rigorous formal foundations to reason about these policies. Current specifications satisfy one of these requirements or the other, but not both. A Metagraph is a generalized graph-theoretic structure that overcomes this limitation. They are a natural way of expressing high-level end-to-end network policies. The rich formal foundations provided by metagraph algebra help analyze important network-policy properties such as reachability, redundancy and consistency. These features make metagraphs a clear choice for modeling and reasoning about policies in Formally-Verifiable Policy-Defined Networking (FV-PDN): a network-programming paradigm which has verifiability built-in. In this article, we demonstrate the use of metagraphs in policy specification by modeling and analyzing real policies from a large university network. We show their benefit in FV-PDN by developing a prototype solution which automatically refines metagraph-based high-level policies to device configurations and deploys them to an SDN-based emulated network.

Analysing narratives through their social networks is an expanding field in quantitative literary studies. Manually extracting a social network from any narrative can be time consuming, so automatic extraction methods of varying complexity have been developed. However, the effect of different extraction methods on the resulting networks is unknown. Here we model and compare three extraction methods for social networks in narratives: manual extraction, co-occurrence automated extraction and automated extraction using machine learning. Although the manual extraction method produces more precise results in the network analysis, it is highly time consuming. The automatic extraction methods yield comparable results for density, centrality measures and edge weights. Our results provide evidence that automatically-extracted social networks are reliable for many analyses. We also describe which aspects of analysis are not reliable with such a social network. Our findings provide a framework to analyse narratives, which help us improve our understanding of how stories are written and evolve, and how people interact with each other.

Large-scale entity resolution (ER) techniques have received tremendous attention due to the emergence of data processing within organizations and governments. The traditional ER process requires pairwise comparisons between each record when identifying records belong to the same entity, which is computationally prohibitive for large databases. With many existing indexing techniques to address this issue, it remains an open research question. We propose a landmarks-based indexing algorithm to reduce the possible pairwise comparisons of non-matches. The blocks are determined based on pre-selected records called landmarks in a multidimensional Euclidean space. The pair-wise comparisons only within these blocks reduce the search space immensely. Our method is scalable for big data entity resolution as it has O(n) insertion and query complexity.

Pinpointing autonomous systems which deploy specific inter-domain techniques such as Route Flap Damping (RFD) or Route Origin Validation (ROV) remains a challenge today. Previous approaches to detect per-AS behavior often relied on heuristics derived from passive and active measurements. Those heuristics, however, often lacked accuracy or imposed tight restrictions on the measurement methods.

Various Online Social Network (OSN) based applications depend on the interactions between users to disseminate information and recruit more users. The temporal evolution of adoption or cascade process of new products, applications or ideas is important to advertisers, OSN operators and application developers. Interactions between users are represented by massive directed graphs, so graph sampling methods were proposed to capture their properties. Existing graph sampling methods, such as a simple random walk, however, are ill- suited for capturing this and other dynamic properties of the graph. We propose ForestStream, a measurement method that relies on a combination of sampling and streaming with the goal of capturing the statistical properties of cascades in OSN graphs. We demonstrate our method's accuracy over existing methods in inferring the cascade statistics, with a low memory usage.

The traffic matrix (TM) is a chief input in many network design and planning applications. In this paper, we propose a model, called the spherically additive noise model (SANM). In conjunction with iterative proportional fitting (IPF), it enables fast generation of synthetic TMs around a predicted TM. We analyze SANM and IPF's action on the model to show theoretical guarantees on asymptotic convergence, in particular, convergence to the well-known gravity model.

Firewalls are a crucial element of any modern day business; they protect data and resources in a communications network from unauthorised access. In particular domains, such as SCADA networks, there are guidelines for firewall configuration, but currently there are no automated means to test compliance. Our research tackles this from first principles: we ask how firewall policies can be described at a high-level, independent of firewall-vendor and network minutiae. The semantic foundations we propose allow us to compare network-wide firewall policies and check if they are equivalent; or one is contained in the other in meaningful ways. These foundations also enable policy change-impact analysis and help identify functional discrepancies between multiple policy designs from users in distinct policy sub-domains (e.g., SCADA engineers, Corporate admins).

A common requirement in policy specification languages is the ability to map policies to the underlying network devices. Doing so, in a provably correct way, is important in a security policy context, so administrators can be confident of the level of protection provided by the policies for their networks. Existing policy languages allow policy composition but lack formal semantics to allocate policy to network devices. Our research tackles this from first principles: we ask how network policies can be described at a high-level, independent of vendor and network minutiae. We identify the algebraic requirements of the policy-mapping process and propose semantic foundations to formally verify if a policy is implemented by the correct set of policy-arbiters. We show the value of our proposed algebras in maintaining concise network-device configurations by applying them to real-world networks.

A common goal in network-management is security. Reliable security requires confidence in the level of protection provided. But, many obstacles hinder reliable security management; most prominent is the lack of built-in verifiability in existing management paradigms. This shortfall makes it difficult to provide assurance that the expected security outcome is consistent pre- and post-deployment. Our research tackles the problem from first principles: we identify the verifiability requirements of robust security management, evaluate the limitations of existing paradigms and propose a new paradigm with verifi- ability built in: Formally-Verifiable Policy-Defined Networking (FV-PDN). In particular, we pay attention to firewalls which protect network data and resources from unauthorised access. We show how FV-PDN can be used to configure firewalls reliably in mission critical networks to protect them from cyber attacks.

Firewall configuration is an important activity for any modern day business. It is particularly a critical task for the supervisory control and data acquisition (SCADA) networks that control power stations, water distribution, factory automation, etc. Lack of automation tools to assist with this critical task has resulted in unoptimised, error prone configurations that expose these networks to cyber attacks. Automation can make designing firewall configurations more reliable and their deployment increasingly cost-effective. Best practices have been proposed by the industry for developing high-level security policy (e.g., ANSI/ISA 62443-1-1). But these best practices lack specification in several key aspects needed to allow a firewall to be automatically configured. For instance, the standards are vague on how firewall management policies should be captured at a high-level using its specifications. In this paper, we uncover these missing pieces and propose extensions. We apply our extended best-practice specification to real-world firewall case studies to achieve multiple objectives: 1) to evaluate the usefulness of the refined best-practice in the automated specification of firewalls and 2) to illustrate that even in simple cases, SCADA networks are often insecure due to their misconfigured firewalls.

Traffic matrices describe the volume of traffic between a set of sources and destinations within a network. These matrices are used in a variety of tasks in network planning and traffic engineering, such as the design of network topologies. Traffic matrices naturally possess complex spatiotemporal characteristics, but their proprietary nature means that little data about them is available publicly, and this situation is unlikely to change.

Subscription fraud, i.e., customers signing up to a service with no intent to pay, causes significant losses in the telecommunication industry. Telecom operators have developed strategies to identify those fraudsters, but fraudsters tend to migrate from one carrier to another. Data sharing between telecoms would increase fraud detection rates, but phone records are protected by law and telecom operators might be reluctant to share information about fraudsters because they see it as giving a competitive advantage. We propose several protocols to enable fraud detection across multiple databases without revealing additional information. We also propose a model to generate phone records, with which we evaluate how the choice of parameters affects detection performance. We show feasibility, performance and costs with implementations of our protocols.

Network topology synthesis seeks methods to generate large numbers of example network topologies primarily for use in simulation. It is a topic that has received much attention over the years, underlying which is a conflict between randomness and design. Random graphs are appealing because they are simple and avoid the messy details that plague real networks. However real networks are messy, because network operators design their networks in the context of complex technological constraints, costs, and goals. When random models have been used they often produce patently unrealistic networks that only match a few artificial connectivity statistics of real networks: the features that make the network useful and interesting are ignored. At best a network divorced from context is a purely mathematical object with no meaning or utility. At worst it can be completely misleading. However, design alone cannot generate an ensemble of networks with the variability needed in simulation. We need to balance design and randomness in a way that generates reasonable networks with given characteristics and predictable variability. This paper presents such a method, Combined Optimization and Layered Design (COLD), incorporating randomness and design principles to create ensembles of PoP-level synthetic networks.

Traffic matrices are used in many network engineering tasks, for instance optimal network design. Unfortunately, measurements of these matrices are error-prone, a problem that is exacerbated when they are extrapolated to provide the predictions used in planning. Practical network design and management should consider sensitivity to such errors, but although robust optimisation techniques exist, it seems they are rarely used, at least in part because of the difficulty in generating an ensemble of admissible traffic matrices with a controllable error level. We address this problem in our paper by presenting a fast and flexible technique of generating synthetic traffic matrices. We demonstrate the utility of the method by presenting a methodology for robust network design based on adaptation of the mean-risk analysis concept from finance.

Emulated networks and systems, where router and server software are run in virtual environments, allow network operators and researchers to perform experiments at large scale more economically than in testbeds. Running real code provides a greater level of realism than simulation.

Security of routing protocols is a critical issue, as shown by the increasing number of attacks on the Internet's routing infrastructure. One often overlooked aspect of security is privacy. In the context of a routing protocol we mean the ability of a router to keep information such as its routing policies private. BGP does this to some extent through design. An Autonomous System's policies are not explicitly revealed to other participants in the routing protocol. Nevertheless, BGP still reveals a great deal of information about the Internet and its participants. We propose a privacy-preserving routing protocol called STRIP that reveals very little information to participants in the protocol. For instance, participants can find shortest-paths to destinations in the network without ever learning the path lengths. Such privacy could be useful for a range of reasons: preserving the proprietary information captured in a routing policy, or preventing an attacker from gaining valuable information about the network. We show the feasibility, performance, and costs of STRIP with simulations and implementations of the protocol.

The prevalence of hidden Markov models (HMMs) in various applications of statistical signal processing and communications is a testament to the power and flexibility of the model. In this paper, we link the identifiability problem with tensor decomposition, in particular, the Canonical Polyadic decomposition. Using recent results in deriving uniqueness conditions for tensor decomposition, we are able to provide a necessary and sufficient condition for the identification of the parameters of discrete time finite alphabet HMMs. This result resolves a long standing open problem regarding the derivation of a necessary and sufficient condition for uniquely identifying an HMM. We then further extend recent preliminary work on the identification of HMMs with multiple observers by deriving necessary and sufficient conditions for identifiability in this setting.

Loss measurements are widely used in today's networks. There are existing standards and commercial products to perform these measurements. The missing element is a rigorous statistical methodology for their analysis. Indeed, most existing tools ignore the correlation between packet losses and severely underestimate the errors in the measured loss ratios. In this paper, we present a rigorous technique for analyzing performance measurements, in particular, for estimating confidence intervals of packet loss measurements. The task is challenging because Internet packet loss ratios are typically small and the packet loss process is bursty. Our approach, SAIL, is motivated by some simple observations about the mechanism of packet losses. Packet losses occur when the buffer in a switch or router fills, when there are major routing instabilities, or when the hosts are overloaded, and so we expect packet loss to proceed in episodes of loss, interspersed with periods of successful packet transmission. This can be modeled as a simple on/off process, and in fact, empirical measurements suggest that an alternating renewal process is a reasonable approximation to the real underlying loss process. We use this structure to build a hidden semi-Markov model (HSMM) of the underlying loss process and, from this, to estimate both loss ratios and confidence intervals on these loss ratios. We use both simulations and a set of more than 18 000 hours of real Internet measurements (between dedicated measurement hosts, PlanetLab hosts, Web and DNS servers) to cross-validate our estimates and show that they are better than any current alternative.

Most large attacks on the Internet are distributed. As a result, such attacks are only partially observed by any one Internet service provider (ISP). Detection would be significantly easier with pooled observations, but privacy concerns often limit the information that providers are willing to share. Multi-party secure distributed computation provides a means for combining observations without compromising privacy. In this paper, we show the benefits of this approach, the most notable of which is that combinations of observations solve identifiability problems in existing approaches for detecting network attacks.

Detection of malicious traffic and network health problems would be much easier if ISPs shared their data. Unfortunately, they are reluctant to share because doing so would either violate privacy legislation or expose business secrets. However, secure distributed computation allows calculations to be made using private data, without leaking this data. This paper presents such a method, allowing multiple parties to jointly infer a Hidden Markov Model (HMM) for traffic and/or user behaviour in order to detect anomalies. We extend prior work on HMMs in network security to include observations from multiple ISPs and develop secure protocols to infer the model parameters without revealing the private data. We implement a prototype of the protocols, and our experiments with the prototype show its has a reasonable computational and communications overhead, making it practical for adoption by ISPs.

We present a methodology that brings simplicity to large and complex test labs by using abstraction. The networking community has appreciated the value of large scale test labs to explore complex network interactions, as seen in projects such as PlanetLab, GENI, DETER, Emulab, and SecSI. Virtualization has enabled the creation of many more such labs. However, one problem remains: it is time consuming, tedious and error prone to setup and configure large scale test networks. Separate devices need to be configured in a coordinated way, even in a virtual lab.

Most large attacks on the Internet are distributed. As a result, such attacks are only partially observed by any one Internet Service Provider (ISP). Detection would be significantly easier with pooled observations, but privacy concerns often limit the information that providers are willing to share. Multi-party secure distributed computation provides a means for combining observations without compromising privacy. In this letter, we show the benefits of this approach, the most notable of which is that combinations of observations solve identifiability problems in existing approaches for detecting network attacks.

Despite advances in measurement technology, it is still challenging to reliably compile large-scale network datasets. For example, because of flaws in the measurement systems or difficulties posed by the measurement problem itself, missing, ambiguous, or indirect data are common. In the case where such data have spatio-temporal structure, it is natural to try to leverage this structure to deal with the challenges posed by the problematic nature of the data. Our work involving network datasets draws on ideas from the area of compressive sensing and matrix completion, where sparsity is exploited in estimating quantities of interest. However, the standard results on compressive sensing are: 1) reliant on conditions that generally do not hold for network datasets; and 2) do not allow us to exploit all we know about their spatio-temporal structure. In this paper, we overcome these limitations with an algorithm that has at its heart the same ideas espoused in compressive sensing, but adapted to the problem of network datasets. We show how this algorithm can be used in a variety of ways, in particular on traffic data, to solve problems such as simple interpolation of missing values, traffic matrix inference from link data, prediction, and anomaly detection. The elegance of the approach lies in the fact that it unifies all of these tasks and allows them to be performed even when as much as 98% of the data is missing.

In this paper we discuss why a simple network topology inference algorithm based on network co-occurrence measurements and a Markov random walk model for routing enables perfect topology reconstruction, despite the seeming model mismatch to real network routing.

Network design, as it is currently practiced, involves putting devices together to create a network. However, a network is more than the sum of its parts, both in terms of the services it provides, and the potential for bugs. Devices are important, but their combination into a network should follow from expression of high-level policy, not the minutiae of network device configuration. Ideally we want to consider the network as a whole object. In this paper we develop generalized graph products that allow the mathematical design of a network in terms of small subgraphs that directly express business policy. The result is a flexible algebraic description of networks suitable for manipulation and proof. The approach is more than just design - it allows for analysis of existing networks providing an understanding of the policies used in their construction, something which can be difficult if the original designers no longer work on that network. We apply the approach to several real world networks to demonstrate how it can provide insight, and improve design.

Experiments on diverse topics such as network measurement, management and security are routinely conducted using empirical flow export traces. However, the availability of empirical flow traces from operational networks is limited and frequently comes with significant restrictions. Furthermore, empirical traces typically lack critical meta-data (e.g., labeled anomalies) which reduce their utility in certain contexts. In this paper, we describe fs: a first-of-its-kind tool for automatically generating representative flow export records as well as basic SNMP-like router interface counts. fs generates measurements for a target network topology with specified traffic characteristics. The resulting records for each router in the topology have byte, packet and flow characteristics that are representative of what would be seen in a live network. fs also includes the ability to inject different types of anomalous events that have precisely defined characteristics, thereby enabling evaluation of proposed attack and anomaly detection methods. We validate fs by comparing it with the ns-2 simulator, which targets accurate recreation of packet-level dynamics in small network topologies. We show that data generated by fs are virtually identical to what are generated by ns-2, except over small time scales (below 1 second). We also show that fs is highly efficient, thus enabling test sets to be created for large topologies. Finally, we demonstrate the utility of fs through an assessment of anomaly detection algorithms, highlighting the need for flexible, scalable generation of network-wide measurement data with known ground truth.

Traffic matrix describes the traffic volumes traversing the network from the input nodes to the exit nodes over a measured period. Such a matrix is very hard, if not intractable, to be obtained for a large network. We apply a new technique to analyze the traffic matrix by use of diffusion wavelet in this paper. It is shown that diffusion wavelet can do an efficient multi-resolution analysis on TM. The original TM can be reconstructed by choosing the diffused traffic in a particular level. This paper also shows there are a lot of potential applications by use of diffusion wavelet-based analysis on traffic matrix.

No abstract available.

The study of network topology has attracted a great deal of attention in the last decade, but has been hampered by a lack of accurate data. Existing methods for measuring topology have flaws, and arguments about the importance of these have overshadowed the more interesting questions about network structure. The Internet Topology Zoo is a store of network data created from the information that network operators make public. As such it is the most accurate large-scale collection of network topologies available, and includes meta-data that couldn't have been measured. With this data we can answer questions about network structure with more certainty than ever before - we illustrate its power through a preliminary analysis of the PoP-level topology of over 140 networks. We find a wide range of network designs not conforming as a whole to any obvious model.

Formally, the Internet inter-domain routing system is a collection of networks, their policies, peering relationships and organizational affiliations, and the addresses they advertize. It also includes components like Internet exchange points. By its very definition, each and every aspect of this system is impacted by BGP, the de-facto standard inter-domain routing protocol. The element of this inter-domain routing system that has attracted the single-most attention within the research community has been the "inter-domain topology". Unfortunately, almost from the get go, the vast majority of studies of this topology, from definition, to measurement, to modeling and analysis, have ignored the central role of BGP in this problem. The legacy is a set of specious findings, unsubstantiated claims, and ill-conceived ideas about the Internet as a whole. By presenting a BGP-focused state-of-the-art treatment of the aspects that are critical for a rigorous study of this inter-domain topology, we demystify in this paper many "controversial" observations reported in the existing literature. At the same time, we illustrate the benefits and richness of new scientific approaches to measuring, modeling, and analyzing the inter-domain topology that are faithful to the BGP-specific nature of this problem domain.

The ability to detect unexpected events in large networks can be a significant benefit to daily network operations. A great deal of work has been done over the past decade to develop effective anomaly detection tools, but they remain virtually unused in live network operations due to an unacceptably high false alarm rate. In this paper, we seek to improve the ability to accurately detect unexpected network events through the use of BasisDetect, a flexible but precise modeling framework. Using a small dataset with labeled anomalies, the BasisDetect framework allows us to define large classes of anomalies and detect them in different types of network data, both from single sources and from multiple, potentially diverse sources. Network anomaly signal characteristics are learned via a novel basis pursuit based methodology. We demonstrate the feasibility of our BasisDetect framework method and compare it to previous detection methods using a combination of synthetic and real-world data. In comparison with previous anomaly detection methods, our BasisDetect methodology results show a 50% reduction in the number of false alarms in a single node dataset, and over 65% reduction in false alarms for synthetic network-wide data.

In this paper we present a rigorous technique for estimating confidence intervals of packet loss measurements. Our approach is motivated by simple observations that the loss process can be modelled as an alternating renewal process. We use this structure to build a Hidden Semi-Markov Model (HSMM) for the measurement process, and from this estimate both loss rates, and their confidence intervals. We use both simulations and a set of more than 18000 hours of real Internet measurements (between dedicated measurement hosts, PlanetLab hosts, web and DNS servers) to cross-validate our estimates, and show that they are significantly more accurate than any current alternative.

Reachability is thought of as the most basic service provided by today's Internet. Unfortunately, this does not imply that the community has a deep understanding of it. Researchers and operators rely on two views of reachability: control/routing- and data-plane measurements, but both types of measurements suffer from biases and limitations. In this paper, we illustrate some of these biases, and show how to design controlled experiments which allow us to "see" through the limitations of previous measurement techniques. For example, we discover the extent of default routing and its impact on reachability. This explains some of the previous unexpected results from studies that compared control- and data-plane measurements.

Routing oscillation is highly detrimental. It can decrease performance and lead to a high level of update churn placing unnecessary workload on router the problem is distributed between many providers. However, iBGP --- the routing protocol used to distribute routes inside a single Autonomous System --- has also been shown to oscillate. Despite the fact that iBGP is configured by a single provider according to apparently straight forward rules, more than eight years of research has not solved the problem of iBGP oscillation. Various solutions have been proposed but they all lack critical features: either they are complicated to implement, restrict routing flexibility, or lack guarantees of stability. In this paper we propose a very simple adaptation to the BGP decision process. Despite its simplicity and negligible cost we prove algebraically that it prevents iBGP oscillation. We extend the idea to provide routing flexibility, such as respecting the MED attribute, without sacrificing network stability.

Many basic network engineering tasks (e.g., traffic engineering, capacity planning, anomaly detection) rely heavily on the availability and accuracy of traffic matrices. However, in practice it is challenging to reliably measure traffic matrices. Missing values are common. This observation brings us into the realm of compressive sensing, a generic technique for dealing with missing values that exploits the presence of structure and redundancy in many real-world systems. Despite much recent progress made in compressive sensing, existing compressive-sensing solutions often perform poorly for traffic matrix interpolation, because real traffic matrices rarely satisfy the technical conditions required for these solutions.

What does a childpsilas search of a large, complex cartoon for the eponymous character (Waldo) have to do with Internet routing? Network operators also search complex datasets, but Waldo is the least of their worries. Routing oscillation is a much greater concern. Networks can be designed to avoid routing oscillation, but the approaches so far proposed unnecessarily reduce the configuration flexibility. More importantly, apparently minor changes to a configuration can lead to instability. Verification of network stability is therefore an important task, but unlike the childpsilas search, this problem is NP hard. Until now, no practical method was available for large networks. In this paper, we present an efficient algorithm for proving stability of iBGP, or finding the potential oscillatory modes, and demonstrate its efficacy by applying it to the iBGP configuration of a large Tier-2 AS.

Reverse engineering of the Internet is a valuable activity. Apart from providing scientific insight, the resulting datasets are invaluable in providing realistic network scenarios for other researchers. The Rocketfuel project attempted this process, but it is surprising how little effort has been made to validate its results. This paper concentrates on validating a particular inference methodology used to obtain link weights on a network. There is a basic difficulty in assessing the accuracy of such inferences in that a non-unique set of link-weights may produce the same routing, and so simple measurements of accuracy (even where ground truth data are available) do not capture the usefulness of a set of inferred weights. We propose a methodology based on predictive power to assess the quality of the weight inference. We used this to test Rocketfuel's algorithm, and our tests suggest that it is reasonably good particularly on certain topologies, though it has limitations when its underlying assumptions are incorrect.

Most research on traffic matrices (TM) has focused on finding models that help with inference, but not with other important tasks such as synthesis of TMs, traffic prediction, or anomaly detection. In this paper we approach the problem of a general model for traffic matrices, and argue that such a model must be sparse, i.e., have a small number of parameters in comparison to the size of the TM. A Multi-Resolution Analysis (MRA) of TMs can provide such a sparse representation. The Diffusion Wavelet (DW) transform is a good choice as a MRA tool here, because it inherently adapts to the structure of the underlying network. The paper describes our construction of the two-dimensional version of the DW transform and shows how to use it for our proposed MRA of TMs. The results obtained with operational networks confirm the sparseness of the DW-based TM analysis approach and its applicability to other TM-related tasks.

Study of the Internet's high-level structure has for some time intrigued scientists. The AS-graph (showing interconnections between Autonomous Systems) has been measured, studied, modelled and discussed in many papers over the last decade. However, the quality of the measurement data has always been in question. It is by now well known that most measurements of the AS-graph are missing some set of links. Many efforts have been undertaken to correct this, primarily by increasing the set of measurements, but the issue remains: how much is enough? When will we know that we have enough measurements to be sure we can see all (or almost all) of the links. This paper aims to address the problem of estimating how many links are missing from our measurements. We use techniques pioneered in biostatistics and epidemiology for estimating the size of populations (for instance of fish or disease carriers). It is rarely possible to observe entire populations, and so sampling techniques are used. We extend those techniques to the domain of the AS-graph. The key difference between our work and the biological literature is that all links are not the same, and so we build a stratified model and specify an EM algorithm for estimating its parameters. Our estimates suggest that a very significant number of links (many of thousands) are missing from standard route monitor measurements of the AS-graph. Finally, we use the model to derive the number of monitors that would be needed to see a complete AS-graph with high-probability. We estimate that 700 route monitors would see 99.9% of links.

Energy-constraint is a crucial problem in wireless sensor networks (WSNs). Many sensor node (SN) placement schemes and routing protocols are proposed to address this problem. In this paper, we first present how to place SNs by use of a minimal number to maximize the coverage area when the communication radius of the SN is not less than the sensing radius, which results in the application of regular topology to WSNs deployment. With nodes placed at an equal distance and equipped with an equal power supply, we discuss the energy imbalance problem and then give the mathematical formulation for maximizing network lifetime in grid-based WSNs. The formulation shows the problem of maximizing network lifetime is a non-linear programming problem and NP-hard even in the 1-D case. We discuss several heuristic solutions and show that the halving shift data collection scheme is the best solution among them. We also generalize the maximizing network lifetime problem to the randomly-deployed WSNs which shows the significance of our mathematical formulation for this crucial problem.

Wireless ad hoc networks provide a useful communications infrastructure for the mobile battlefield. In this paper we apply and develop passive radio frequency signal strength monitoring and packet transmission time profiling techniques, to characterise and reconstruct an encrypted wireless network's topology. We show that by using signal strength measurements from three or more wireless probes and by assuming the use of carrier sense multiple access with collision avoidance, for physical layer control, we can produce a representation of a wireless network's logical topology and in some cases reconstruct the physical topology. Smoothed Kalman filtering is used to track the reconstructed topology over time, and in conjunction with a weighted least squares template fitting technique, enables the profiling of the individual network nodes and the characterisation of their transmissions.

An understanding of the topological structure of the Internet is needed for quite a number of networking tasks, e. g., making decisions about peering relationships, choice of upstream providers, inter-domain traffic engineering. One essential component of these tasks is the ability to predict routes in the Internet. However, the Internet is composed of a large number of independent autonomous systems (ASes) resulting in complex interactions, and until now no model of the Internet has succeeded in producing predictions of acceptable accuracy.We demonstrate that there are two limitations of prior models: (i) they have all assumed that an Autonomous System (AS) is an atomic structure - it is not, and (ii) models have tended to oversimplify the relationships between ASes. Our approach uses multiple quasi-routers to capture route diversity within the ASes, and is deliberately agnostic regarding the types of relationships between ASes. The resulting model ensures that its routing is consistent with the observed routes. Exploiting a large number of observation points, we show that our model provides accurate predictions for unobserved routes, a first step towards developing structural mod-els of the Internet that enable real applications.

Active probes of network performance represent samples of the underlying performance of a system. Some effort has gone into considering appropriate sampling patterns for such probes, i.e., there has been significant discussion of the importance of sampling using a Poisson process to avoid biases introduced by synchronization of system and measurements. However, there are unanswered questions about whether Poisson probing has costs in terms of sampling efficiency, and there is some misinformation about what types of inferences are possible with different probe patterns. This paper provides a quantitative comparison of two different sampling methods. This paper also shows that the irregularity in probing patterns is useful not just in avoiding synchronization, but also in determining frequency-domain properties of a system. This paper provides a firm basis for practitioners or researchers for making decisions about the type of sampling they should use in a particular applications, along with methods for the analysis of their outputs

This paper considers the basic problem of "how accurate can we make Internet performance measurements". The answer is somewhat counter-intuitive in that there are bounds on the accuracy of such measurements, no matter how many probes we can use in a given time interval, and thus arises a type of Heisenberg inequality describing the bounds in our knowledge of the performance of a network. The results stem from the fact that we cannot make independent measurements of a system's performance: all such measures are correlated, and these correlations reduce the efficacy of measurements. The degree of correlation is also strongly dependent on system load. The result has important practical implications that reach beyond the design of Internet measurement experiments, into the design of network protocols.

Traffic matrices are required inputs for many IP network management tasks, such as capacity planning, traffic engineering, and network reliability analysis. However, it is difficult to measure these matrices directly in large operational IP networks, so there has been recent interest in inferring traffic matrices from link measurements and other more easily measured data. Typically, this inference problem is ill-posed, as it involves significantly more unknowns than data. Experience in many scientific and engineering fields has shown that it is essential to approach such ill-posed problems via "regularization". This paper presents a new approach to traffic matrix estimation using a regularization based on "entropy penalization". Our solution chooses the traffic matrix consistent with the measured data that is information-theoretically closest to a model in which source/destination pairs are stochastically independent. It applies to both point-to-point and point-to-multipoint traffic matrix estimation. We use fast algorithms based on modern convex optimization theory to solve for our traffic matrices. We evaluate our algorithm with real backbone traffic and routing data, and demonstrate that it is fast, accurate, robust, and flexible.

The ability to provide different Quality of Service (QoS) guarantees to traffic from different applications is a highly desired feature for many IP network operators, particularly for enterprise networks. Although various mechanisms exist for providing QoS in the network, QoS is yet to be widely deployed. We believe that a key factor holding back widespread QoS adoption is the absence of suitable methodologies/processes for appropriately mapping the traffic from different applications to different QoS classes. This is a challenging task, because many enterprise network operators who are interested in QoS do not know all the applications running on their network, and furthermore, over recent years port-based application classification has become problematic. We argue that measurement based automated Class of Service (CoS) mapping is an important practical problem that needs to be studied.

IP forwarding anomalies, triggered by equipment failures, implementation bugs, or configuration errors, can significantly disrupt and degrade network service. Robust and reliable detection of such anomalies is essential to rapid problem diagnosis, problem mitigation, and repair. We propose a simple, robust method that integrates routing and traffic data streams to reliably detect forwarding anomalies. The overall method is scalable, automated and self-training. We find this technique effectively identifies forwarding anomalies, while avoiding the high false alarms rate that would otherwise result if either stream were used unilaterally.

The desire to better understand global BGP dynamics has motivated several studies using active measurement techniques, which inject announcements and withdrawals of prefixes from the global routing domain. From these one can measure quantities such as the BGP convergence time. Previously, the route injection infrastructure of such experiments has either been temporary in nature, or its use has been restricted to the experimenters. The routing research community would benefit from a permanent and public infrastructure for such active probes. We use the term BGP Beacon to refer to a publicly documented prefix having global visibility and a published schedule for announcements and withdrawals. A BGP Beacon is to be used for the ongoing study of BGP dynamics, and so should be supportedwith a long-term commitment. We describe several BGP Beacons thathave been set up at various points in the Internet. We then describe techniques for processing BGP updates when a BGP Beacon is observed from a BGP monitoring point such as Oregon's Route Views. Finally, we illustrate the use of BGP Beacons in the analysis of convergence delays, route flap damping, and update inter-arrival times.

Traffic engineering and traffic matrix estimation are often treated as separate fields, even though one of the major applications for a traffic matrix is traffic engineering. In cases where a traffic matrix cannot be measured directly, it may still be estimated from indirect data (such as link measurements), but these estimates contain errors. Yet little thought has been given to the effects of inexact traffic estimates on traffic engineering. In this paper we consider how well traffic engineering works with estimated traffic matrices in the context of a specific task; namely that of optimizing network routing to minimize congestion, measured by maximum link-utilization. Our basic question is: how well is the real traffic routed if the routing is only optimized for an estimated traffic matrix? We compare against optimal routing of the real traffic using data derived from an operational tier-1 ISP. We find that the magnitude of errors in the traffic matrix estimate is not, in itself, a good indicator of the performance of that estimate in route optimization. Likewise, the optimal algorithm for traffic engineering given knowledge of the real traffic matrix is no longer the best with only the estimated traffic matrix as input. Our main practical finding is that the combination of a known traffic matrix estimation technique and a known traffic engineering technique can get close to the optimum in avoiding congestion for the real traffic. We even demonstrate stability in the sense that routing optimized on data from one day continued to perform well on subsequent days. This stability is crucial for the practical relevance to off-line traffic engineering, as it can be performed by ISPs today.

Traffic matrices are required inputs for many IP network management tasks: for instance, capacity planning, traffic engineering and network reliability analysis. However, it is difficult to measure these matrices directly, and so there has been recent interest in inferring traffic matrices from link measurements and other more easily measured data. Typically, this inference problem is ill-posed, as it involves significantly more unknowns than data. Experience in many scientific and engineering fields has shown that it is essential to approach such ill-posed problems via "regularization". This paper presents a new approach to traffic matrix estimation using a regularization based on "entropy penalization". Our solution chooses the traffic matrix consistent with the measured data that is information-theoretically closest to a model in which source/destination pairs are stochastically independent. We use fast algorithms based on modern convex optimization theory to solve for our traffic matrices. We evaluate the algorithm with real backbone traffic and routing data, and demonstrate that it is fast, accurate, robust, and flexible.

We consider the performance of estimated tra#c matrices in tra#c engineering. More precisely, we first optimize the routing in an IP backbone to minimize congestion with the estimated tra#c matrix. We then test the performance of the resulting routing on the real tra#c matrix.

A matrix giving the traffic volumes between origin and destination in a network has tremendously potential utility for network capacity planning and management. Unfortunately, traffic matrices are generally unavailable in large operational IP networks. On the other hand, link load measurements are readily available in IP networks. In this paper, we propose a new method for practical and rapid inference of traffic matrices in IP networks from link load measurements, augmented by readily available network and routing configuration information. We apply and validate the method by computing backbone-router to backbone-router traffic matrices on a large operational tier-1 IP network -- a problem an order of magnitude larger than any other comparable method has tackled. The results show that the method is remarkably fast and accurate, delivering the traffic matrix in under five seconds.

Understanding the variability of Internet traffic in backbone networks is essential to better plan and manage existing networks, as well as to design next generation networks. However, most traffic analyses that might be used to approach this problem are based on detailed packet or flow level measurements, which are usually not available throughout a large network. As a result there is a poor understanding of backbone traffic variability, and its impact on network operations (e.g. on capacity planning or traffic engineering).This paper introduces a metric for measuring backbone traffic variability that is grounded on simple but powerful traffic theory. What sets this metric apart, however, is that we present a method for making practical measurements of the metric using widely available SNMP traffic measurements. Furthermore, we use a novel method to overcome the major limitation of SNMP measurements -- that they only provide link statistics. The method, based on a "gravity model", derives an approximate traffic matrix from the SNMP data. In addition to simulations, we use more than 1 year's worth of SNMP data from an operational IP network of about 1000 nodes to test our methods. We also delve into the degree and sources of variability in real backbone traffic, providing insight into the true nature of traffic variability.

Open Shortest Path First (OSPF) is widely deployed in IP networks to manage intra-domain routing. OSPF is a link-state protocol, in which routers reliably flood "Link State Advertisements" (LSAs), enabling each to build a consistent, global view of the routing topology. Reliable performance hinges on routing stability, yet the behavior of large operational OSPF networks is not well understood. In this paper, we provide a case study on the eharacteristics and dynamics of LSA traffic for a large enterprise network. This network consists of several hundred routers, distributed in tens of OSPF areas, and connected by LANs and private lines. For this network, we focus on LSA traffic and analyze: (a) the class of LSAs triggered by OSPF's soft-state refresh, (b) the class of LSAs triggered by events that change the status of the network, and (c) a class of "duplicate" LSAs received due to redundancy in OSPF's reliable LSA flooding mechanism. We derive the baseline rate of refresh-triggered LSAs automatically from network configuration information. We also investigate finer time scale statistical properties of this traffic, including burstiness, periodicity, and synchronization. We discuss root causes of event-triggered and duplicate LSA traffic, as well as steps identified to reduce this traffic (e.g., localizing a failing router or changing the OSPF configuration).

One of the most significant findings of traffic measurement studies over the last decade has been the observed self-similarity in packet network traffic. Subsequent research has focused on the origins of this self-similarity, and the network engineering significance of this phenomenon. This paper reviews what is currently known about network traffic self-similarity and its significance. We then consider a matter of current research, namely, the manner in which network dynamics (specifically, the dynamics of transmission control protocol (TCP), the predominant transport protocol used in today's Internet) can affect the observed self-similarity. To this end, we first discuss some of the pitfalls associated with applying traditional performance evaluation techniques to highly-interacting, large-scale networks such as the Internet. We then present one promising approach based on chaotic maps to capture and model the dynamics of TCP-type feedback control in such networks. Not only can appropriately chosen chaotic map models capture a range of realistic source characteristics, but by coupling these to network state equations, one can study the effects of network dynamics on the observed scaling behavior We consider several aspects of TCP feedback, and illustrate by examples that while TCP-type feedback can modify the self-similar scaling behavior of network traffic, it neither generates it nor eliminates it.

An on-line version of the Abry-Veitch (see IEEE GLOBECOM'98, Sydney, Australia,p.3716-21, 1998) wavelet-based estimator of the Hurst parameter is presented. It has very low memory and computational requirements and scales naturally to arbitrarily high data rates, enabling its use in real-time applications such as admission control, and avoiding the need to store huge data sets for off-line analysis. The performance of the estimator as a function of the length of data processed is demonstrated using simulated data. An implementation for 10-Mb/s Ethernet based on standard hardware supporting sampling rates of 1 data point per millisecond is described, and results of its operation presented, as is an implementation for 155-Mb/s asynchronous transfer mode networks. Finally we illustrate the power of on-line measurements by collecting measurements over a period of five months, and using them to look for diurnal trends in scaling properties of the data.

The bottleneck in many telecommunication systems has often been modeled by an M/G/1 queueing system with priorities. While the probability generating function (PGF) for the occupancy distribution of each traffic class can be readily obtained, the occupancy distributions have been obtainable only rarely. However, the occupancy distribution is of great importance, particularly in those cases where the moments are not all finite. We present a method of obtaining the occupancy distribution from the PGF and demonstrate its validity by obtaining the occupancy distributions for a number of cases, including those with regularly varying service time distributions.

Previous measurements of various types of network traffic have shown evidence consistent with long-range dependence and self-similarity. However, an alternative explanation for these measurements is non-stationarity. Standard estimators of LRD parameters such as the Hurst parameter H assume stationarity and are susceptible to bias when this assumption does not hold. Hence LRD may be indicated by these estimators when none is present, or alternatively LRD taken to be non-stationarity. The Abry-Veitch (see IEEE Trans. on on Info. Theory, vol.44, no.1, p.2-15, 1998) joint estimator has much better properties when a time-series is non-stationary. In particular the effect of polynomial trends in data may be intrinsically eliminated from the estimates of LRD parameters. This paper investigates the behavior of the AV estimator when there are non-stationarities in the form of a level shift in the mean and/or the variance of a process. We examine cases where the change occurs both gradually or as a single jump discontinuity, and also examine the effect of the size of the shift. In particular we show that although a jump discontinuity may cause bins in the estimates of the H, the bias is negligible except when the jump is sharp, and large compared with the standard deviation of the process. We explain these effects and suggest how any introduced errors might be minimized. We define a broad class of non-stationary LRD processes so that LRD remains well defined under time varying mean and variance. The results are tested by applying the estimator to a real data set which contains a clear non-stationary event falling within this class.

The interest sparked by observations of long-range dependent traffic in real networks has lead to a revival of interest in non-standard queueing systems. One such queueing system is the M/G/1 queue where the service-time distribution has infinite variance. The known results for such systems are asymptotic in nature, typically providing the asymptotic form for the tail of the workload distribution, simulation being required to learn about the rest of the distribution. Simulation however performs very poorly for such systems due to the large impact of rare events. We provide a method for numerically evaluating the entire distribution for the number of customers in the M/G/1 queue with power-law tail service-time. The method is computationally efficient and shown to be accurate through careful simulations. It can be directly extended to other queueing systems and more generally to many problems where the inversion of probability generating functions complicated by power-laws is at issue. Through the use of examples we study the limitations of simulation and show that information on the tail of the queue-length distribution is not always sufficient to answer significant performance questions. We also derive the asymptotic form of the number of customers in the system in the case of a service-time distribution with a regularly varying tail (e.g. infinite variance) and thus illustrate the techniques required to apply the method in other contexts.