The exponential growth of data traffic and the increasing complexity of networked applications demand effective solutions capable of passively inspecting and analysing the network traffic for monitoring and security purposes. Implementing network probes in software using general-purpose operating systems have been made possible by advances in packet-capture technologies, such as kernel-bypass frameworks, and by multi-queue adapters designed to distribute the network workload in multi-core processors. Modern SmartNICs, in addition, have introduced stateful mechanisms to associate actions to network flows such as forwarding packets or updating traffic statistics for an individual flow. In this paper, we describe our experience in exploiting those functionalities in a modern network probe and we perform a detailed study of the performance characteristics under different scenarios. Compared to pure CPU-based solutions, SmartNICs with flow-offload technologies provide substantial benefits when implementing forwarding applications. However, the main limitation of having to keep large flow tables in the host memory remains largely unsolved for realistic monitoring and security applications.

Network log events produced by network probes are used by security analyzers to detect traffic anomalies and threats. While it is relatively trivial for a probe to mark specific events as suspicious, it is much more challenging for log analyzers to create a comprehensive picture of the overall network. Machine learning can potentially help in this, however there is no specific solution for analyzing network event logs. This paper covers the experiments and design choices that have been made to create a machine learning-based tool able to analyze network event logs. The tool has been evaluated in the Suspicious Network Event Recognition Cup at IEEE BigData 2019, achieving an AUC (Area Under the Curve) of over 90%, making it accurate enough for deployment in real life scenarios.

Summary Compressed bitmap indexes are used in systems such as Git or Oracle to accelerate queries. They represent sets and often support operations such as unions, intersections, differences, and symmetric differences. Several important systems such as Elasticsearch, Apache Spark, Netflix's Atlas, LinkedIn's Pivot, Metamarkets' Druid, Pilosa, Apache Hive, Apache Tez, Microsoft Visual Studio Team Services, and Apache Kylin rely on a specific type of compressed bitmap index called Roaring. We present an optimized software library written in C implementing Roaring bitmaps: CRoaring. It benefits from several algorithms designed for the single‐instruction–multiple‐data instructions available on commodity processors. In particular, we present vectorized algorithms to compute the intersection, union, difference, and symmetric difference between arrays. We benchmark the library against a wide range of competitive alternatives, identifying weaknesses and strengths in our software. Our work is available under a liberal open‐source license.

Current practices in network security deployment require multiple specialised devices as firewalls, traffic shapers, sensors or Intrusion Detection Systems (IDSs) to handle malicious traffic. This practice not only increases the overall operational costs but also makes network administration complicated. The high cost of Distributed Denial of Service (DDoS) mitigation devices empowers centralised services and network architectures as there is not a cost-effective model to deploy them at the “true edge” of the network. This paper describes the design and implementation of a multi-10 Gbit extensible network traffic analysis and policing system. It is composed of logical detection and enforcement functions built from reusable underlying primitives. As an example of such modular approach, we present an innovative DDoS scrubbing system composed of various attack detection primitives, combined with enforcement primitives that include traffic filtering, rate limiting, and proxying. Based on commodity hardware and open source software, such system is price, space, and power efficient enough to be practically deployable at the edge of the network. Performance measurements carried on 10Gbit networks, show that it can effectively provide both traffic visibility and enforcement of a wide range of network traffic policies.

Network traffic analysis was traditionally limited to packet header, because the transport protocol and application ports were usually sufficient to identify the application protocol. With the advent of port-independent, peer-to-peer, and encrypted protocols, the task of identifying application protocols became increasingly challenging, thus creating a motivation for creating tools and libraries for network protocol classification. This paper covers the design and implementation of nDPI, an open-source library for protocol classification using both packet header and payload. nDPI was extensively validated in various monitoring projects ranging from Linux kernel protocol classification, to analysis of 10 Gbit traffic, reporting both high protocol detection accuracy and efficiency.

Network traffic recorders are devices that record massive volumes of network traffic for security applications, like retrospective forensic investigations. When deployed over very high-speed networks, traffic recorders must process and store millions of packets per second. To enable interactive explorations of such large traffic archives, packet indexing mechanisms are required. Indexing packets at wire rates (10 Gbps and above) on commodity hardware imposes unparalleled requirements for high throughput index creation. Such indexing throughputs are presently untenable with modern indexing technologies and current processor architectures. In this work, we propose to intelligently offload indexing to commodity General Processing Units (GPUs). We introduce algorithms for building compressed bitmap indexes in real time on GPUs and show that we can achieve indexing throughputs of up to 185 millions records per second, which is an improvement by one order of magnitude compared to the state-of-the-art. This shows that indexing network traffic at multi-10-Gbps rates is well within reach.

Capturing packets to disk at line rate and with high precision packet timestamping is required whenever an evidence of network communications has to be provided. Typical applications of long-term network traffic repositories are network troubleshooting, analysis of security violations, and analysis of high-frequency trading communications. Appliances for 10 Gbit packet capture to disk are often based on dedicated network adapters, and therefore very expensive, making them usable only in specific domains. This paper covers the design and implementation of n2disk, a packet capture to disk application, capable of dumping 10 Gbit traffic to disk using commodity hardware and open-source software. In addition to packet capture, n2disk is able to index the traffic at line-rate during capture, enabling users to efficiently search specific packets in network traffic dump files.

Monitoring of large distributed networks requires the deployment of several probes at different network locations where traffic to be analyzed is flowing. Each probe analyzes the traffic and sends the monitoring data toward a centralized management station often using protocols such as NetFlow and IPFIX. As each probe monitors a part of the traffic, the data collector has the responsibility of merging data coming from all the probes and correlate information. This task adds extra load on collectors and prevents traffic information to be available until it has been correlated, thus preventing (near) realtime traffic monitoring. This paper describes how the microcloud architecture can be used to provide real-time traffic monitoring and correlation on large distributed environments where monitoring traffic is analyzed by several probes that collectively concur to the monitoring task. This work has been successfully validated on using this architecture for monitoring the. It DNS ccTLD and a large 3G mobile network with million of users.

The DNS is one of the core protocols on which the Internet is built upon. Hidden behind higher-level protocols such as email and web, it carries valuable information that can be exploited for understanding trends and preferences of the Internet community. In this paper we propose novel methodologies for modelling DNS traffic that allow Internet domains, DNS resolvers and their interactions to be represented effectively by means of graphs. DNS traffic collected at “.it” ccTLD DNS domain servers has been used to validate this work on a large scale. We found highly-skewed, fat-tailed domain and resolver degree frequencies, obeying power laws at least in their tails. These findings shed light on the the scale-free nature of the DNS ecosystem, where a few domains and a few resolvers are responsible for most of the DNS activity.

The domain name system (DNS) is a complex distributed database on which most Internet services rely on. As its monitoring is critical, it is necessary to continuously monitor DNS traffic for identifying anomalies, measuring performance, and generating usage statistics. This paper describes the design and implementation of a distributed realtime DNS monitoring system, that is able to monitor the authoritative name servers of the it country code Top Level Domain (ccTLD). In addition to the production of usage records, it is able to understand trends, characterize economical relationships, and also track suspicious activities.

The demand of highly flexible and easy to deploy network monitoring systems has pushed companies toward software based network monitoring probes implemented with commodity hardware rather than with expensive and highly specialized network devices. Deploying software probes under virtual machines executed on the same physical box is attractive for reducing deployment costs and for simplifying the management of advanced network monitoring architectures built on top of heterogeneous monitoring tools (i.e. Intrusion Detection Systems and Performance Monitoring Systems). Unfortunately, software probes are usually not able to meet the performance requirements when deployed in virtualized environments as virtualization introduces severe performance bottlenecks when performing packet capture, which is the core activity of passive network monitoring systems.

Multi-core systems are the current dominant trend in computer processors. However, kernel network layers often do not fully exploit multi-core architectures. This is due to issues such as legacy code, resource competition of the RX-queues in network interfaces, as well as unnecessary memory copies between the OS layers. The result is that packet capture, the core operation in every network monitoring application, may even experience performance penalties when adapted to multi-core architectures. This work presents common pitfalls of network monitoring applications when used with multi-core systems, and presents solutions to these issues. We describe the design and implementation of a novel multi-core aware packet capture kernel module that enables monitoring applications to scale with the number of cores. We showcase that we can achieve high packet capture performance on modern commodity hardware.

The use of the Internet as a medium for real-time communications has grown significantly over the past few years. However, the best-effort model of this network is not particularly well-suited to the demands of users who are familiar with the reliability, quality and security of the public switched telephone network. If the growth is to continue, monitoring and real time analysis of communication data will be needed in order to ensure good call quality, and should degradation occur, to take corrective action. Writing this type of monitoring application is difficult and time consuming: VoIP traffic not only tends to use dynamic ports, but its real-time nature, along with the fact that its packets tend to be small, impose non-trivial performance requirements. In this paper we present RTC-Mon, the real-time communications monitoring framework, which provides an extensible platform for the quick development of high-speed, real-time monitoring applications. While the focus is on VoIP traffic, the framework is general and is capable of monitoring any type of real-time communications traffic. We present testbed performance results for the various components of RTC-Mon, showing that it can monitor a large number of concurrent flows without losing packets. In addition, we implemented a proof-of-concept application that can not only track statistics about a large number of calls and their users, but that consists of only 800 lines of code, showing that the framework is efficient and that it also significantly reduces development time.

Today's networks present several management challenges due to the variety of network types and the integration of different network media. Network administrators need automated tools to support human effort, gathering information about the status and behaviour of network elements. Network monitoring is probably the most fundamental aspect of automated network management. The goal of this paper is to describe the design and implementation of ntop a simple, open-source, portable traffic measurement and monitoring tool, which supports various management activities, including network optimisation and planning, and detection of network security violations. In addition, it describes some scenarios where ntop can be effectively used for identifying common network problems as well as detecting intruders and security violations.

Desktop-based management is the activity of managing networks and systems using standard desktop tools (for instance a text editor), methods (drag and drop) and paradigms (trashcan). The principles behind desktop-based management are: every piece of management resource/information must be accessible from the desktop; the composition of management information (for instance weekly router usage) should be accessible from the desktop, hence virtually from every application; every management resource should be visible at desktop level (for instance into the file system) and accessible from several applications.