Our analytics task is to identify, characterize, and visualize anomalous subsets of as large of a collection of network connection data as possible. We use a combination of HPC resources, advanced algorithms, and visualization techniques. To effectively and efficiently identify the salient portions of the data, we rely on a multistage workflow that includes data acquisition, summarization (feature extraction), novelty detection, and classification. Once these subsets of interest have been identified and automatically characterized, we use a stateof- the-art high-dimensional query system to extract this data for interactive visualization. Our approach is equally useful for other large-data analysis problems where it is more practical to identify interesting subsets of the data for visualization than it is to render all data elements. By reducing the size of the rendering workload, we enable highly interactive and useful visualizations.

This paper presents a family of bitmap algorithms that address the problem of counting the number of distinct header patterns (flows) seen on a high speed link. Such counting can be used to detect DoS attacks and port scans, and to solve measurement problems. Counting is especially hard when processing must be done within a packet arrival time (8 nsec at OC-768 speeds) and, hence, must require only a small number of accesses to limited, fast memory. A naive solution that maintains a hash table requires several Mbytes because the number of flows can be above a million. By contrast, our new probabilistic algorithms take very little memory and are fast. The reduction in memory is particularly important for applications that run multiple concurrent counting instances. For example, we replaced the port scan detection component of the popular intrusion detection system Snort with one of our new algorithms. This reduced memory usage on a ten minute trace from 50 Mbytes to 5.6 Mbytes while maintaining a 99.77% probability of alarming on a scan within 6 seconds of when the large-memory algorithm would. The best known prior algorithm (probabilistic counting) takes 4 times more memory on port scan detection and 8 times more on a measurement application. Fundamentally, this is because our algorithms can be customized to take advantage of special features of applications such as a large number of instances that have very small counts or prior knowledge of the likely range of the count.

With the advent of computational grids, networking performance over the wide-area network (WAN) has become a critical component in the grid infrastructure. Unfortunately, many high-performance grid applications only use a small fraction of the available bandwidth because operating systems and their associated protocol stacks are still tuned for yesterdays WAN speeds. As a result, network gurus undertake the tedious process of manually, tuning system buffers to allow TCP flow control to scale to today's WAN grid environments. Although recent research has shown how to set the size of these system buffers automatically at connection set-up, the buffer sizes are only appropriate at the beginning of the connection's lifetime. To address these problems, we describe an automated and scalable technique called dynamic right-sizing. We implement this technique in user space (in particular for bulk-data transfer) so that end users do not have to modify the kernel to achieve a significant increase in throughput.

The state of the art in general purpose software systems for large-scale traffic measurement has not progressed much past the venerable libpcap. In this paper we describe a new data analysis system that provides a scalable, flexible system for composing ad-hoc analyses of high-speed, streming data. This agility allows researchers, network security analysts, or network operators to easily compose new analysis functions. A growing tool box of filtering, measurement, and statistical tools allows new approaches to be tested with a minimum of software development. Further, a dynamic type system allows polymorphic analysis modules to operate on arbitrary forms of structured data, thus allowing easy integration of multiple data sources such as packet traces, netflow records, or security logs. In this paper we present this system and demonstrate its capabilities while performing several measurements, such as computing probability density functions, detecting port-scans, and probabilistic counting of traffic traces.