Network Traffic Monitoring and Analysis (NTMA) represents a key component for network management, especially to guarantee the correct operation of large-scale networks such as the Internet. As the complexity of Internet services and the volume of traffic continue to increase, it becomes difficult to design scalable NTMA applications. Applications such as traffic classification and policing require real-time and scalable approaches. Anomaly detection and security mechanisms require to quickly identify and react to unpredictable events while processing millions of heterogeneous events. At last, the system has to collect, store, and process massive sets of historical data for post-mortem analysis. Those are precisely the challenges faced by general big data approaches: Volume, Velocity, Variety, and Veracity. This survey brings together NTMA and big data. We catalog previous work on NTMA that adopt big data approaches to understand to what extent the potential of big data is being explored in NTMA. This survey mainly focuses on approaches and technologies to manage the big NTMA data, additionally briefly discussing big data analytics (e.g., machine learning) for the sake of NTMA. Finally, we provide guidelines for future work, discussing lessons learned, and research directions.

Monitoring the Quality of Experience (QoE) undergone by cellular network customers has become paramount for cellular ISPs, who need to ensure high quality levels to limit customer churn due to quality dissatisfaction. This paper tackles the problem of QoE monitoring, assessment and prediction in cellular networks, relying on end-user device (i.e., smart-phone) QoS passive traffic measurements and QoE crowdsourced feedback. We conceive different QoE assessment models based on supervised machine learning techniques, which are capable to predict the QoE experienced by the end user of popular smartphone apps (e.g., YouTube and Facebook), using as input the passive in-device measurements. Using a rich QoE dataset derived from field trials in operational cellular networks, we benchmark the performance of multiple machine learning based predictors, and construct a decision-tree based model which is capable to predict the per-user overall experience and service acceptability with a success rate of 91% and 98% respectively To the best of our knowledge, this is the first paper using end-user, in-device passive measurements and machine learning models to predict the QoE of smartphone users in operational cellular networks.

YoMoApp (YouTube Monitoring App) is an Android app to monitor mobile YouTube video streaming on both application- and network-layer. Additionally, it allows to collect subjective Quality of Experience (QoE) feedback of end users. During the development of the app, the stable versions of YoMoApp were already available in the Google Play Store, and the app was downloaded, installed, and used on many devices to monitor streaming sessions. As the app was not advertised in special campaigns or used for dedicated QoE studies, the monitored streaming sessions of this period compose the data set of a large unsupervised field study. The collected data set is evaluated to characterize current mobile YouTube streaming on both application and network layers. Furthermore, the problems and methodology to obtain QoE results from such unsupervised field study are discussed together with the actual QoE results. Correlations between QoE factors are investigated, and the QoE of clusters of similar streaming sessions is analyzed.

Network security represents a keystone to ISPs, who need to cope with an increasing number of network attacks that put the network's integrity at risk. The high-dimensionality of network data provided by current network monitoring systems opens the door to the massive application of machine learning approaches to improve the detection and classification of network attacks. In this paper we devise a novel attacks detection and classification technique based on semi-supervised Machine Learning (ML) algorithms to automatically detect and diagnose network attacks with minimal training, and compare its performance to that achieved by other well-known supervised learning detectors. The proposed solution is evaluated using real network measurements coming from the WIDE backbone network, using the well-known MAWILab dataset for attacks labeling.

Despite a large body of literature and methods devoted to the analysis of network traffic, the automatic detection and classification of network traffic anomalies still represents a major issue for network operators. The problem becomes even more challenging for cellular ISPs, both due to the ever growing number of connected devices and to the constant deployment of new applications and services prone to performance issues. In this paper we tackle this problem using Machine Learning (ML) approaches: in particular, we devise a system based on Neural Networks to unveil the relations between several monitored traffic features and network anomalies impacting a large number of customers in an operational cellular network. By training a model based on Random Neural Networks (RNN), we provide a fast and accurate anomaly detector and classifier, capable to pinpoint anomalies without assuming any specific traffic model or particular network behavior. The proposed solution is evaluated using synthetically generated data from an operational cellular ISP, drawn from real traffic statistics to resemble the real cellular network traffic. Our RNN model is capable to detect and classify different classes of anomalies with high accuracy and low false alarm rates, even when the volume of such anomalies is small.

Internet access through cellular networks is rapidly growing, driven by the great success of the mobile apps paradigm and the overwhelming popularity of social-related multimedia services such as YouTube, Facebook, or even WhatsApp. Understanding the functioning, performance, and traffic generated by these applications is paramount for ISPs, especially for cellular operators, who must manage the huge surge of volume and number of users with the constraints and challenges of cellular networks. In this paper, we study important networking aspects of three popular applications in cellular networks: YouTube, Facebook, and WhatsApp. Our evaluations span the content delivery networks hosting these services, their traffic characteristics, and their performance. The analysis is performed on top of real cellular network traffic monitored at the nationwide cellular network of a major European ISP. Due to privacy issues and given the huge amount of data generated by these applications as well as the large number of monitored customers, the analysis has been done in an online fashion, using a customized big data analytics (BDA) platform called DBStream. We overview DBStream and discuss other potential solutions currently available for traffic monitoring and analysis of big networking data. To the best of our knowledge, this is the first paper providing a complete analysis of popular services in cellular networks, using BDA platforms.

The DNS protocol has proved to be a valuable means for identifying and dissecting large-scale anomalies in omnipresent Over The Top (OTT) Internet services. In this paper, we present and evaluate a framework for detecting and diagnosing traffic anomalies via DNS traffic analysis. Detection of such anomalies is achieved by monitoring different DNS-related symptomatic features, flagging a warning as soon as one or more of them show a significant change. The investigation of the root causes for such deviations is done by looking at significant changes in a number of diagnostic features (i.e., device manufacturer and OS, requested host name, error codes, etc.), which convey information directly linked to the potential origins of the detected anomalies. For the purpose of detecting significant changes in the time-series of diagnostic features, we propose a scheme based on change point detection applied to the entropy of the considered features. The proposed solution is tested using both real and synthetic data from a nationwide mobile ISP, the latter generated from real traffic statistics to resemble the real mobile network traffic. To show the operational value of the proposed framework, we report the results of the diagnosis in two prototypical cases.

Online Social Networks (OSNs) have rapidly become an integral part of our daily lives, and hundreds of millions of people are nowadays remotely connected trough popular OSNs such as Facebook, Google+, Twitter and WhatsApp. While much has been said and studied about the social aspects of OSNs, little is known about the network side of OSNs, specially regarding their network and traffic footprints, as well as their content delivery infrastructures. In this paper we study these networking aspects of OSNs, vivisecting the most popular OSNs in western countries: Facebook and WhatsApp. By analyzing two large-scale traffic traces collected at the cellular network of a major European ISP, we characterize and compare the networking behavior of Facebook and WhatsApp, considering not only the traffic flows but also the network infrastructures hosting them. Our study serves the main purpose of better understanding how major OSNs are provisioned in today's Internet. To the best of our knowledge, this is the first paper providing such an analysis using large-scale measurements in cellular networks.

In this paper we challenge the applicability of entropy-based approaches for detecting and diagnosis network traffic anomalies, and claim that full statistics (i.e., empirical probability distributions) should be applied to improve the change-detection capabilities. We support our claim by detecting and diagnosing large-scale traffic anomalies in a real cellular network, caused by specific OTT (Over The Top) services and smartphone devices. Our results clearly suggest that anomaly detection and diagnosis based on entropy analysis is prone to errors and misses typical characteristics of traffic anomalies, particularly in the studied scenario.

Internet-scale services like YouTube are provisioned by large Content Delivery Networks (CDNs), which push content as close as possible to the end-users to improve their Quality of Experience (QoE) and to pursue their own optimization goals. Adopting space and time variant traffic delivery policies, CDNs serve users' requests from multiple servers/caches at different physical locations and different times. CDNs traffic distribution policies can have a relevant impact on the traffic routed through the Internet Service Provider (ISP), as well as unexpected negative effects on the end-user QoE. In the event of poor QoE due to faulty CDN server selection, a major problem for the ISP is to avoid being blamed by its customers. In this paper we show a real case study in which Google CDN server selection policies negatively impact the QoE of the customers of a major European ISP watching YouTube. We argue that it is extremely important for the ISP to rapidly and automatically detect such events to increase its visibility on the overall operation of the network, as well as to promptly answer possible customer complaints. We therefore present an Anomaly Detection (AD) system for detecting unexpected cache-selection changes in the traffic delivered by CDNs. The proposed algorithm improves over traditional AD approaches by analyzing the complete probability distribution of the monitored features, as well as by self-adapting its functioning to dynamic environments, providing better detection capabilities.

Today's Internet consists of massive scale web services and Content Delivery Networks (CDNs). This paper sheds light on the way major Internet-scale web services content is hosted and delivered. By analyzing a full month of HTTP traffic traces collected at the mobile network of a major European ISP, we characterize the paradigmatic case of Facebook, considering not only the traffic flows but also the main organizations and CDNs providing them. Our study serves the main purpose of better understanding how major web services are provisioned in today's Internet, paying special attention to the temporal dynamics of the service delivery and the interplays between the involved hosting organizations. To the best of our knowledge, this is the first paper providing such an analysis in mobile networks.

YouTube is the most popular service in today's Internet. Google relies on its massive content delivery network (CDN) to push YouTube videos as close as possible to the end-users, both to improve their watching experience as well as to reduce the load on the core of the network, using dynamic server selection strategies. However, we show that such a dynamic approach can actually have negative effects on the end-user quality of experience (QoE). Through the comprehensive analysis of one month of YouTube flow traces collected at the network of a large European ISP, we report a real case study in which YouTube QoE-relevant degradation affecting a large number of users occurs as a result of Google's server selection strategies. We present an iterative and structured process to detect, characterize, and diagnose QoE-relevant anomalies in CDN distributed services such as YouTube. The overall process uses statistical analysis methodologies to unveil the root causes behind automatically detected problems linked to the dynamics of CDNs' server selection strategies.

In this paper we review state-of-the-art botnet detection algorithms that reveal the control traffic of malicious peer-topeer (P2P) networks by targeting topological properties of their interconnectivity graph. This class of detection methods does not rely on the exchanged content and therefore is also applicable to encrypted control traffic. However, in practice, an ISP monitoring customer traffic over an edge router will usually see only a fraction of the overall botnet, thus restricting the available bot connectivity information and limiting the applicability of general community detection approaches. In this paper we critically review graph based detection methods suitable for edge router monitoring using two types of real network traces. We show experimentally that using meta-graphs of mutual contacts proposed by Coskun et al. 2010 has the highest potential on result quality. We improve this approach by presenting a computationally less complex algorithm with similar result quality. Furthermore we explain ways to alleviate the cost of dealing with false positives in the result set.

Collaboration of defensive network components of multiple operators is a promising approach for increasing anomaly detection accuracy. This concept involves sharing of possibly sensitive data, hence privacy preservation has to be taken into account. In this paper, we argue that common approaches for sharing traffic information often impede proper analysis due to privacy-preserving mangling operations, and ignore the opportunity to exploit additional knowledge of the originating network operator for interpreting the monitored data. We propose COMINDIS, a lightweight framework for sharing notions of suspiciousness among network operators, and show how to exploit different detection systems for deriving a better understanding of Internet hosts' activities. We evaluate the system both by using a network emulator and by experimenting with a real traffic trace.

The Weber-Fechner Law (WFL) is an important principle in psychophysics which describes the relationship be- tween the magnitude of a physical stimulus and its perceived intensity. With the sensory system of the human body, in many cases this dependency turns out to be of logarithmic nature. Re- cent quantitative QoE research shows that in several different scenarios a similar logarithmic relationship can be observed be- tween the size of a certain QoS parameter of the communication system and the resulting QoE on the user side as observed during appropriate user trials. In this paper, we discuss this surprising link in more detail. After a brief survey on the background of the WFL, we review its basic implications with respect to related work on QoE assessment for VoIP, most notably the recently published IQX hypothesis, before we present results of our own trials on QoE assessment for mobile broadband scenarios which confirm this dependency also for data services. Finally, we point out some conclusions and directions for further research.

In this work we present a novel scheme for statistical-based anomaly detection in 3G cellular networks. The traffic data collected by a passive monitoring system are reduced to a set of per-mobile user counters, from which time-series of unidimensional feature distributions are derived. An example of feature is the number of TCP SYN packets seen in uplink for each mobile user in fixed-length time bins. We design a change-detection algorithm to identify deviations in each distribution time-series. Our algorithm is designed specifically to cope with the marked non-stationarities, daily/weekly seasonality and longterm trend that characterize the global traffic in a real network. The proposed scheme was applied to the analysis of a large dataset from an operational 3G network. Here we present the algorithm and report on our practical experience with the analysis of real data, highlighting the key lessons learned in the perspective of the possible adoption of our anomaly detection tool on a production basis.

In this work we present a simple toy-model that is able to explain certain empirical observations reported in a set of previous papers by Hohn et al. about the wavelet spectrum of real traffic traces. Therein, the authors found that the wavelet spectrum is substantially invariant to flow scrambling and truncation. Such finding suggested that super-flow structures above the transport layer - i.e., sessions - can be ignored for modeling the packet arrival process. Based on the proposed toy-model, we offer an interpretation framework that goes in the opposite direction, indicating that sessions, not transport-layer flows, should be taken as the main structural entities in simplified on/off models.

In this contribution we address the problem of using cellular network signaling for inferring real-time road traffic information. We survey and categorize the approaches that have been proposed in the literature for a cellular-based road monitoring system and identify advantages and limitations. We outline a unified framework that encompasses UMTS and GPRS data collection in addition to GSM, and prospectively combines passive and active monitoring techniques. We identify the main research challenges that must be faced in designing and implementing such an intelligent road traffic estimation system via third-generation cellular networks.

Modeling TCP is fundamental for understanding Internet behavior. The reason is that TCP is responsible for carrying a huge quota of the Internet traffic. During last decade many analytical models have attempted to capture dynamics and steady-state behavior of standard TCP congestion control algorithms. In particular, models proposed in literature have been mainly focused on finding relationships among the throughput achieved by a TCP flow, the segment loss probability, and the round trip time (RTT) of the connection, which the flow goes through. Recently, Westwood+ TCP algorithm has been proposed to improve the performance of classic New Reno TCP, especially over paths characterized by high bandwidth-delay products. In this paper, we develop an analytic model for the throughput achieved by Westwood+ TCP congestion control algorithm when in the presence of paths with time-varying RTT. The proposed model has been validated by using the ns-2 simulator and Internet-like scenarios. Validation results have shown that this model provides relative prediction errors smaller than 10%. Moreover, it has been shown that a similar accuracy is achieved by analogous models proposed for New Reno TCP.