Microservices architecture, while fostering scalability and maintainability, introduces difficulties in anomaly detection and root cause analysis due to its distributed nature. Numerous methods have been proposed to tackle this problem by incorporating automation into the recovery process. However, these methods often lack proper evaluation due to the limitations of current benchmarks, which are typically bound by synchronous communication and lack real-world complexity. To address this gap, this study proposes Hexagon, an open-source toolkit for generating asynchronous benchmark microservices with configurable complexity, specifically focusing on interaction patterns. Hexagon provides insights into how asynchronous communication within microservices influences their resilience against anomaly propagation, allowing the evaluation of microservice compositions based on their ease of recovery. An exploratory experiment demonstrates the impact of asynchronous communication on anomaly propagation patterns, highlighting the need for such realistic benchmarks in this domain.

Microservices architecture offers significant scalability and flexibility, becoming the standard for large-scale web applications. However, its distributed nature introduces inherent uncertainty, particularly in service communication, leading to unpredictable behavior under varying load and failure conditions. This uncertainty poses challenges in maintaining system reliability and performance [1] .

Owing to the critical role of the domain name system (DNS), its query log data are utilized for various network monitoring purposes. With the diversification of network services, these data have become increasingly complex, making mining useful information challenging. DNS query log data can be considered as the superposition of two types of communication patterns: groups of domains accessed simultaneously (e.g., ad servers and content delivery network (CDN) servers) and time-series access patterns based on user behavior characteristics (e.g., access trends during the night). However, previous studies have not focused on extracting both access patterns hidden in the data. This study proposes a method that extracts both patterns of accessed domains and temporal access patterns as user communication behaviors from DNS query log data and predicts future accesses based on these patterns. The proposed method first aggregates similar fully qualified domain names (FQDNs) associated with the same service. We then present temporal regularized nonnegative tensor factorization (TR-NTF) that extracts both access patterns from a third-order tensor expressing DNS query log data and enables prediction. We evaluate the proposed method using synthetic and actual data and demonstrate that it successfully extracts hidden communication patterns and achieves sufficient prediction accuracy.

This paper introduces a method to estimate latent traffic from its origin to destination based on the link packet loss rate and traffic volume. Using the estimated latent traffic, this paper also shows that we can compute the appropriate link capacity and route of packet transfer. Observed traffic might deviate from the original traffic demand and become latent when the traffic passes through congested links because of transmission control protocol (TCP) congestion control and behavioral change in the users and/or applications owing to a degraded quality of experience (QoE). The latent traffic is actualized when the congested link's capacity is improved. When link provisioning is based on observed traffic, actualized traffic might cause new congestion at other links. Thus, network providers need to estimate the origin-destination (OD) original traffic demand for network planning. Although estimation of original traffic has been researched, the estimation was only for links. In this paper, we propose a method to estimate latent origin-destination traffic by combining and expanding techniques. One approach estimates the actualized OD traffic and loss rate from the actualized traffic and packet loss rate of links. The other method estimates the latent traffic demand. Then, using the estimated value, the link capacity and routing matrix are optimized. We evaluated our method through simulation and confirmed that congestion could be avoided by capacity provisioning based on estimated latent traffic, while provisioning based on observed traffic retained the congestion. The combined method can avoid congestion with a 23% increment of capacity compared to capacity provisioning only.

We propose a root-cause diagnosis method for finding equipment suffering from rare failures in a communication network. Although many studies have been conducted on root cause diagnosis for finding failed equipment using a Bayesian Network or other methods, there has not been sufficient research into finding rare-failure equipment. Current methods are mainly focused on typical-failure equipment and cannot find rare-failure equipment. This is because rare failures have two features;unexpected causal relations and observation errors. To adapt rare- failure features, we propose a method that consists of an extended causal model and an extended inference algorithm with dynamic modification of the causal relations and observation statuses in a Bayesian Network. We experimentally evaluated its effectiveness.

Network functions virtualization (NFV) enables telecommunications service providers to provide various network services by flexibly combining multiple virtual network functions (VNFs). To provide such services with carrier-grade quality, an NFV controller must optimally allocate such VNFs into physical networks and servers, taking into account combination(s) of objective functions and constraints for each metric defined for each VNF type. The NFV controller should also be extendable, i.e., new metrics should be able to be added. One approach for NFV control to optimize allocations is to construct an algorithm that simultaneously solves the combined optimization problem. However, this algorithm is not extendable because the problem formulation needs to be rebuilt every time, e.g., a new metric is added. Another approach involves using an extendable network-control architecture that coordinates multiple control algorithms specified for individual metrics. However, to the best of our knowledge, no method has been developed to optimize allocations through this kind of coordination. In this paper, we propose an extendable NFV-integrated control method by coordinating multiple control algorithms. We also propose an efficient coordination algorithm based on reinforcement learning. Finally, we evaluate the effectiveness of the proposed method through simulations.

For Internet service providers to efficiently use network resources, they need to conduct traffic engineering to dynamically control traffic routes to accommodate traffic with limited network resources. The performance of traffic engineering depends on the accuracy of traffic prediction. However, the volume of network traffic has been changing drastically in recent years due to the growth of various types of network services, making traffic prediction increasingly difficult. Our simple ideas to overcome this challenge are to separate traffic into predictable and unpredictable parts and to apply different control policies to predictable and unpredictable traffic. To promote these ideas, we use software-defined networking technology, particularly Open-Flow, that can control macroflows defined by any combination of L2-L4 packet header information such as 5-tuple. In this paper, we therefore propose the macroflow-generating method for separating traffic into predictable macroflows that have little traffic variation and unpredictable macroflows that have large traffic variation within a limited flow table size. We also propose a macroflow-based traffic engineering scheme that uses different routing policies in accordance with traffic predictability. Simulation evaluation results suggest that our proposed scheme can reduce the maximum link load in a network at the most congested time by 34% and the average link load in a network on average by 11% compared with the current traffic engineering schemes.

In current large scale networks, troubleshooting has become more complicated task due to the diversification in the causes of network failures. The increase in the operational costs has become a serious problem. Thus, manualization of the troubleshooting process also becomes important task though it is time-consuming. We propose a method that automatically extracts a workflow for troubleshooting using multiple trouble tickets. Our method extracts an operator's actions from free-format texts and aligns relative sentences between multiple trouble tickets. Finally, we show a novel approach to visualizing a workflow by mining conditional branches using clustering. We validated our method using real trouble ticket data captured from a network operation and showed that it can extract the workflow to identify the cause of failure.

With the growth of services in IP networks, network operators are required to perform proactive operation that quickly detects the signs of critical failures and prevents future problems. Network log data, including router syslog, are rich sources for such operations. However, it has become impossible to find genuinely important logs that lead to serious problems due to the large volume and complexity of log data. We propose a log analysis system for proactive detection of failures. Our key observation is that the abnormality of logs depends on not just the keywords in the messages (e.g. ERROR, FAIL), but generation patterns such as burstiness. Our system consists of three functions: (i) extracting log templates automatically and quickly from a massive amount of unstructured log data; (ii) constructing log feature vectors to characterize the generation patterns of logs; and (iii) using a supervised machine learning approach to associate failures with the log data that appeared before them. We validated our system using real log data collected from a large network and determined its effectiveness.

Popularity ranking of Internet services is an important metric for network operators, because it enables mid- to-long term planning of their network facilities and root cause analysis for unexpected traffic. The service-oriented traffic monitoring is much helpful to infer the popularity, hence it has been gathering much attention from both researchers and practitioners. Lately, service identification of a given flow has become very difficult due to the rapid growth of CDNs and/or encrypted traffic, while some research works employed preceding DNS traffic as a hint. However, because of its cache mechanism, the DNS message count deviates from the actual number of flows, which can greatly degrade the ranking reliability. We propose a theoretical model for inferring the user's number of accesses per domain name by exploiting the characteristics of the DNS message count. To the best of our knowledge, this paper is the first attempt to formulate the effect of user's stub resolvers; previous studies were focused on analyzing the effect of cache servers. We evaluated the precision of our model with a real dataset of traffic of thousands of users. By analyzing the top-50 domain names by the number of users, we can infer the number of flows within a 24% error rate on average in 42 out of 50 FQDNs.

Traffic engineering (TE) plays an essential role in deciding routes that effectively use network resources. This is particularly important when one considers the increasing time variation of Internet traffic such as streaming and cloud services. Traffic engineering with traffic prediction is one approach to stably accommodating time-varying traffic. This approach calculates routes from predicted traffic to avoid congestion, but predictions may include errors that instead cause congestion. We propose a prediction-based traffic engineering method that is robust to prediction errors by considering the probability distribution of predicted traffic. Our approach is based on a control-theoretic approach called stochastic model predictive control. Routes are calculated using a probability distribution of prediction errors so that the occurrence probability of congestion is lower than an operator-specified level. By considering the multi-step future dynamics of traffic, the routes are changed gradually to avoid route oscillation. We also show a relaxation method for unreliable far-future probabilistic constraints to avoid overly conservative route changes. Through simulations using backbone network traffic traces, we demonstrate that our method can accommodate most traffic variations under a given target link capacity without sudden large routes changes.

Although the use of software-defined networking (SDN) enables routes of packets to be controlled with finer granularity (down to the individual flow level) by using traffic engineering (TE) and thereby enables better balancing of the link loads, the corresponding increase in the number of states that need to be managed at routers and controller is problematic in large-scale networks. Aggregating flows into macro flows and assigning routes by macro flow should be an effective approach to solving this problem. However, when macro flows are constructed as TE targets, variations of traffic rates in each macro flow should be minimized to improve route stability. We propose two methods for generating macro flows: one is based on a greedy algorithm that minimizes the variation in rates, and the other clusters micro flows with similar traffic variation patterns into groups and optimizes the traffic ratio of extracted from each cluster to aggregate into each macro flow. Evaluation using traffic demand matrixes for 48 hours of Internet2 traffic demonstrated that the proposed methods can reduce the number of TE targets to about 1/50 ~ 1/400 without degrading the link-load balancing effect of TE.

Understanding the impacts and patterns of network events such as link flaps or hardware errors is crucial for diagnosing network anomalies. In large production networks, analyzing the log messages that record network events has become a challenging task due to the following two reasons. First, the log messages are composed of unstructured text messages generated by vendor-specific rules. Second, network equipment such as routers, switches, and RADIUS severs generate various log messages induced by network events that span across several geographical locations, network layers, protocols, and services. In this paper, we have tackled these obstacles by building two novel techniques: statistical template extraction (STE) and log tensor factorization (LTF). STE leverages a statistical clustering technique to automatically extract primary templates from unstructured log messages. LTF aims to build a statistical model that captures spatial-temporal patterns of log messages. Such spatial-temporal patterns provide useful insights into understanding the impacts and root cause of hidden network events. This paper first formulates our problem in a mathematical way. We then validate our techniques using massive amount of network log messages collected from a large operating network. We also demonstrate several case studies that validate the usefulness of our technique.

Traffic engineering with traffic prediction is one approach to accommodate time-varying traffic without frequent route changes. In this approach, the routes are calculated so as to avoid congestion based on the predicted traffic. The accuracy of the traffic prediction however has large impacts on this approach. Especially, if the predicted traffic amount is significantly less than the actual traffic, the congestion may occur. In this paper, we propose the traffic prediction methods suitable to the traffic engineering. In our method, we perform preprocessing before the prediction in order to predict the periodical variation accurately. Moreover, we consider the confidence interval for the prediction error and the variation excluded by the preprocessing to avoid the congestion caused by the temporal traffic variation. In this paper, we discuss three preprocessing approaches; the trend component, the lowpass filter, and the envelope. Through simulation, we clarify that the preprocessing by the trend component or the lowpass filter increases the accuracy of the prediction. In addition, considering the confidence interval achieves the lower link utilization within a fixed control period.

One problem with active measurement is that, while it is suitable for measuring time-average network performance, it is difficult to measure per-flow quality of service (QoS), which is defined as the average over packets in the flow. To achieve such per-flow QoS measurement, the authors proposed a new technique, called the change- of- measure-based passive/active monitoring (CoMPACT Monitor), which is based on the change-of-measure framework in probability/measure theory and transforms actively obtained information by using passively monitored data. This technique enables us to concurrently measure one-way delay information about individual users, applications, and organizations in detail in a lightweight manner. This paper presents the mathematical formulation for the CoMPACT Monitor and verifies that it works well under some weak conditions. In addition, we investigate its characteristics regarding several implementation issues through simulation and actual network experiments. The results reveal that our technique provides highly qualified estimates involving only a limited amount of extra traffic from active probes.

We investigate the detection accuracy of network anomalies when we use flow statistics obtained through packet sampling. We have already shown, through a case study based on measurement data, that network anomalies generating a huge number of small flows, such as network scans or SYN flooding, become hard to detect when we perform packet sampling. In this paper, we first develop an analytical model that enables us to quantitatively evaluate the effect of packet sampling on the detection accuracy and then investigate why detection accuracy worsens when the packet sampling rate decreases. In addition, we show that, even with a low sampling rate, spatially partitioning the monitored traffic into groups makes it possible to increase the detection accuracy. We also develop a method of determining an appropriate number of partitioned groups and show its effectiveness.

A method of estimating TCP flow-rates of sampled flows through packet sampling is described in this paper. We use sequence numbers of sampled packets, which make it possible to improve markedly the accuracy of estimating the flow rates. Using an analytical model, we investigate how to set parameters such as packet sampling probability used in this method of estimation. As a remarkable result, we show that the estimation accuracy improves as the sampling probability decreases. Using measured data, we also show that this method gives accurate estimations. We also show that this estimation method enables us to detect performance degradation at the TCP flow level.

Many studies of social networks have recently been published. Interest in topological structures, such as scale-free characteristics, has been particularly strong. In this paper, we focus on the analysis of macro traffic data in a communications network of cellular phone users as a way of investigating large-scale social networks. Behaviors of information exchange between pairs of cellular phone users are reflected in traffic data, which thus reflects interesting features of social networks. We analyze the relationship between the number of customers and the volume of traffic with a view to finding clues about the structure of social networks among the very large set of potential customers. We then demonstrate some interesting features that our analysis reveals: a scale-free topology of human relations, their cluster structures, and behaviors of user-dynamics. In addition, we consider the relationship between traffic volume and the number of customers depending on the situation.

In this paper, we propose a method of detecting TCP performance degradation using only bottleneck-link utilization statistics. The variance of link utilization normally increases as the mean link-utilization increases. However, because link-utilization has a maximum of 100%, as the mean approaches 100%, the variance decreases to zero. In this paper, using the M/G/R processor sharing model, we relate this phenomenon to the behavior of flows. We also show that by using this relationship, we can detect TCP performance degradation using the mean and variance of link utilization. Particularly, with this method, a network operator can determine whether or not the degradation originates from the congestion of his/her own network. Because our method requires us to measure link utilization only, the cost of performance management can be greatly decreased compared with the conventional method, which requires dedicated equipment to measure the network performance.

We develop a method of dimensioning and managing the bandwidth of a link on which TCP flows from access links are aggregated. To do this, we extend the application of the processor-sharing queue model to TCP performance evaluation by using flow statistics. To handle various factors that affect actual TCP behavior, besides the access-link bandwidth, such as round-trip time, window-size, and other bottlenecks, we extend the model by replacing the access-link bandwidth with the actual file-transfer speed of a flow under a low utilization of the aggregation link. We only use the number of active flows and the link utilization to estimate the file-transfer speed. Unlike previous studies, the extended model based on the actual transfer speed does not require any assumptions/predeterminations about file-size, packet-size, and round-trip times, etc. Using the extended model, we predict the TCP performance when the link utilization increases. We also show a method of dimensioning the bandwidth needed to maintain TCP performance. We show the effectiveness of our method through simulation analysis.

We previously proposed a change-of-measure based performance measurement method which combines active and passive measurement to estimate user-experienced performance. We also applied this method to packet-delay estimation. We apply this method to loss-rate estimation. Because packet loss rarely occurs in current networks, its measurement usually requires a huge number of probe packets, which imposes a non-negligible load on the networks. We propose a loss-rate estimation method which requires significantly fewer probe packets. In our proposed method, the correlation between delay and loss is measured in advance, and at the time of measurement, the time-averaged loss rate is estimated by using the delay of probe packets and the correlation. We have also applied our change-of-measure framework to estimating the loss rate in user packets by using this time-averaged loss rate. We prove that the mean square error in our method is lower than the simple loss measurement which is estimated by dividing the number of lost packets by the total number of sent packets. We evaluate our method through simulations and actual measurements and find that it can estimate below 10/sup -3/ packet loss rate with only 900 probe packets.

We propose a method of IP traffic management where the quality of TCP performance at a bottleneck link is estimated from monitored data on the behavior of the number of active flows versus utilization of the link, each of which is easy to measure. Our method is based on the characteristics that (i) TCP performance remains constant until the level of link utilization exceeds some threshold value, but becomes degraded when the utilization exceeds this value and (ii) the number of active flows increases linearly with utilization of the link up to the same value, above which the increase becomes nonlinear. Though this threshold may vary from network to network, our method requires neither predetermination of a threshold, on the basis of assumed traffic conditions, nor direct measurement of TCP performance.

In this paper, we are studying the capacity dimensioning of virtual private networks (VPN) access-links for elastic traffic, such as the Web or ftp. Under the assumption that the core-VPN network is provisioned with sufficient large capacity, the capacity management of the VPN access link is a matter of bandwidth-sharing for elastic traffic of the two bottleneck links, the ingress and egress access links, the processor -sharing model gives a simple formulae of mean transfer time, but in our case, the value may be less than the actual transfer time. In contrast, max-min fair sharing provides an accurate sharing model which is similar to the TCP, but it is difficult to obtain a closed form of performance statistics. We propose a closed form approximation for a max-min fair sharing model, in a specific but realistic topology, by investigating the difference between the max-min and the processor sharing model. Using the approximation, we are performing the capacity dimensioning of VPN access links.

To make a scalable and lightweight QoS monitoring system, we have proposed a new QoS monitoring technique, change-of-measure based passive/active monitoring (CoMPACT monitor), which is based on change-of-measure framework and is an active measurement transformed by using passively monitored data. This technique enables us to measure detailed QoS information for individual users, applications, and organizations, in a scalable and lightweight manner. In this paper, we present the mathematical foundation of CoMPACT monitor. In addition, we show its characteristics through simulations in terms of typical implementation issues for inferring the delay distributions. The results show that CoMPACT monitor gives accurate QoS estimations with only a small amount of extra traffic for active measurement.

We propose a performance measurement method that uses both active and passive measurement data to infer the actual performance seen by users. With this method, the performance for individual users, organizations or applications can also be estimated. An actual implementation of the proposed method is examined through simulation. We also evaluated the estimation accuracy with respect to both the measurement interval and the number of measurements.

In the last decade two major solutions, namely IntServ and DiffServ have been introduced to empower IP networks with quality of service (QoS) capabilities. Among these two the DiffServ, which exhibits a better scalability and is the starting point for this paper is based on a per hop shaping of the traffic. The nodes control independently the flows without knowledge about the network state or/and the limitations suffered by the flows at other hops. Thus, violation of service differentiation can also occur. To correct this inefficiency a network-wide proportional service model is proposed. After presenting the theoretical argumentation and the architecture, we also provide an algorithm that computes the shaping factor needed to sustain our architecture. We use simulation experiments to validate our proposal.