Conducting data analysis tasks rarely occur in isolation. Especially in intelligence analysis scenarios where different experts contribute knowledge to a shared understanding, members must communicate how insights develop to establish common ground among collaborators. The use of provenance to communicate analytic sensemaking carries promise by describing the interactions and summarizing the steps taken to reach insights. Yet, no universal guidelines exist for communicating provenance in different settings. Our work focuses on the presentation of provenance information and the resulting conclusions reached and strategies used by new analysts. In an open-ended, 30-minute, textual exploration scenario, we qualitatively compare how adding different types of provenance information (specifically data coverage and interaction history) affects analysts' confidence in conclusions developed, propensity to repeat work, filtering of data, identification of relevant information, and typical investigation strategies. We see that data coverage (i.e., what was interacted with) provides provenance information without limiting individual investigation freedom. On the other hand, while interaction history (i.e., when something was interacted with) does not significantly encourage more mimicry, it does take more time to comfortably understand, as represented by less confident conclusions and less relevant information-gathering behaviors. Our results contribute empirical data towards understanding how provenance summarizations can influence analysis behaviors.

In many types of dynamic interactive visualizations, it is often desired to interact with moving objects. Stopping moving objects can make selection easier, but pausing animated content can disrupt perception and understanding of the visualization. To address such problems, we explore selection techniques that only pause a subset of all moving targets in the visualization. We present various designs for controlling pause regions based on cursor trajectory or cursor position. We then report a dual-task experiment that evaluates how different techniques affect both target selection performance and contextual awareness of the visualization. Our findings indicate that all pause techniques significantly improved selection performance as compared to the baseline method without pause, but the results also show that pausing the entire visualization can interfere with contextual awareness. However, the problem with reduced contextual awareness was not observed with our new techniques that only pause a limited region of the visualization. Thus, our research provides evidence that region-limited pause techniques can retain the advantages of selection in dynamic visualizations without imposing a negative effect on contextual awareness.

Analysis of streaming data often involves both real-time monitoring of incoming data as well as contextual awareness of data history. A focus-plus-context approach can support both goals, with variable levels of visual aggregation making it possible to provide a high level of detail for incoming and recent data while providing contextual information about recent history. Visual aggregation reduces data resolution in order to show the context of data over large periods of time within a limited display space. With a controlled experiment, we evaluated the effectiveness of different types of aggregation for four types of stream-analysis tasks. Overall, the results show that a focus-plus-context design has little negative impact on the ability to successfully monitor and analyze streaming data, making it possible to show longer periods of time than other approaches. However, visual aggregation can be problematic for trend recognition tasks. This research demonstrates how the effectiveness of the visualization depends on the specifics of the analysis task.

Despite the best efforts of cyber security analysts, networked computing assets are routinely compromised, resulting in the loss of intellectual property, the disclosure of state secrets, and major financial damages. Anomaly detection methods are beneficial for detecting new types of attacks and abnormal network activity, but such algorithms can be difficult to understand and trust. Network operators and cyber analysts need fast and scalable tools to help identify suspicious behavior that bypasses automated security systems, but operators do not want another automated tool with algorithms they do not trust. Experts need tools to augment their own domain expertise and to provide a contextual understanding of suspicious behavior to help them make decisions. In this paper we present Situ, a visual analytics system for discovering suspicious behavior in streaming network data. Situ provides a scalable solution that combines anomaly detection with information visualization. The system's visualizations enable operators to identify and investigate the most anomalous events and IP addresses, and the tool provides context to help operators understand why they are anomalous. Finally, operators need tools that can be integrated into their workflow and with their existing tools. This paper describes the Situ platform and its deployment in an operational network setting. We discuss how operators are currently using the tool in a large organization's security operations center and present the results of expert reviews with professionals.

We address a crucial element of applied information extraction—accurate identification of basic security entities in text-—by evaluating previous methods and presenting new labelers. Our survey reveals that the previous efforts have not been tested on documents similar to the targeted sources (news articles, blogs, tweets, etc.) and that no sufficiently large publicly available annotated corpus of these documents exists. By assembling a representative test corpus, we perform a quantitative evaluation of previous methods in a realistic setting, revealing an overall lack of recall, and giving insight to the models' beneficial and inhibiting elements. In particular, our results show that many previous efforts overfit to the non-representative test corpora in this domain. Informed by this evaluation, we present three novel cyber entity extractors, which seek to leverage the available labeled data but remain worthwhile on the more diverse documents encountered in the wild. Each new model increases the state of the art in recall, with maximal or near maximal F1 score. Our results establish that the state of the art in cyber entity tagging is characterized by F1 = 0.61.

Visual history tools provide visual representations of the workflow during data analysis tasks. While there is an established need for reviewing analytic processes, and many visual history tools provide visualizations to do so, it is not well known how helpful the tools actually are for process recall. Through a controlled experiment, we evaluated how the presence of a visual history aid and varying levels of visual detail affect process memory. Participants conducted an analysis task using a visual text-document analysis tool. We evaluated their memories of the process both immediately after the analysis and then again one week later. Results showed that even visual history views with reduced data-resolution were effective for aiding process memory. Further, even without inclusion of any data in the visual history aids, the visual cues alone from the final workspace were enough to improve memory of the main themes of analyses.

Public disclosure of important security information, such as knowledge of vulnerabilities or exploits, often occurs in blogs, tweets, mailing lists, and other online sources significantly before proper classification into structured databases. In order to facilitate timely discovery of such knowledge, we propose a novel semi-supervised learning algorithm, PACE, for identifying and classifying relevant entities in text sources. The main contribution of this paper is an enhancement of the traditional bootstrapping method for entity extraction by employing a time-memory trade-off that simultaneously circumvents a costly corpus search while strengthening pattern nomination, which should increase accuracy. An implementation in the cyber-security domain is discussed as well as challenges to Natural Language Processing imposed by the security domain.

Network vulnerability is a critical component of network security. Yet vulnerability analysis has received relatively little attention from the security visualization community. This paper describes nv, a web-based Nessus vulnerability visualization. Nv utilizes treemaps and linked histograms to allow security analysts and systems administrators to discover, analyze, and manage vulnerabilities on their networks. In addition to visualizing single Nessus scans, nv supports the analysis of sequential scans by showing which vulnerabilities have been fixed, remain open, or are newly discovered. Nv operates completely in-browser, to avoid sending sensitive data to outside servers. We discuss the design of nv, as well as provide case studies demonstrating vulnerability analysis workflows which include a multiple-node testbed and data from the 2011 VAST Challenge.

To help increase the confidence that software is secure, researchers and vendors have developed different kinds of automated software security analysis tools. These tools analyze software for weaknesses and vulnerabilities, but the individual tools catch different vulnerabilities and produce voluminous data with many false positives. This paper describes a system that brings together the results of disparate software analysis tools into a visual environment to support the triage and exploration of code vulnerabilities. Our system allows software developers to explore vulnerability results to uncover hidden trends, triage the most important code weaknesses, and show who is responsible for introducing software vulnerabilities. By correlating and normalizing multiple software analysis tools' data, the overall vulnerability detection coverage of software is increased. A visual overview and powerful interaction allows the user to focus attention on the most pressing vulnerabilities within huge volumes of data, and streamlines the secure software development workflow through integration with development tools.

User testing is an integral component of user-centered design, but has only rarely been applied to visualization for cyber security applications. This paper describes a comparative evaluation of a visualization application and a traditional interface for analyzing network packet captures, that was conducted as part of the user-centered design process. Structured, well-defined tasks and exploratory, open-ended tasks were completed with both tools. Accuracy and efficiency were measured for the well-defined tasks, number of insights was measured for exploratory tasks and user perceptions were recorded for each tool. The results of this evaluation demonstrated that users performed significantly more accurately in the well-defined tasks, discovered a higher number of insights and demonstrated a clear preference for the visualization tool. The study presented here may be useful for future visualization for network security visualization evaluation designers. Some of the challenges and lessons learned are described.

Intrusion detection, the process of using network data to identify potential attacks, has become an essential component of information security. Human analysts doing intrusion detection work utilize vast amounts of data from disparate sources to make decisions about potential attacks. Yet, there is limited understanding of this critical human component. This research seeks to understand the work practices of these human analysts to inform the design of a task-appropriate information visualization tool to support network intrusion detection analysis tasks. System design will follow a user-centered, spiral methodology. System evaluation will include both a field-based qualitative evaluation, uncommon in information visualization, and a lab-based benchmarking evaluation.

When performing packet-level analysis in intrusion detection, analysts often lose sight of the big picture while examining these low-level details. In order to prevent this loss of context and augment the available tools for intrusion detection analysis tasks, we developed an information visualization tool, the time-based network traffic visualizer (TNV). TNV is grounded in an understanding of the work practices of intrusion detection analysts, particularly foregrounding the overarching importance of context and time in the process of intrusion detection analysis. The main visual component of TNV is a matrix showing network activity of hosts over time, with connections between hosts superimposed on the matrix, complemented by multiple, linked views showing port activity and the details of the raw packets. Providing low-level textual data in the context of a high-level, aggregated graphical display enables analysts to examine packet-level details within the larger context of activity. This combination has the potential to facilitate the intrusion detection analysis tasks and help novice analysts learn what constitutes normal on a particular network.

This paper presents the Intrusion Detection toolkit (IDtk), an information Visualization tool for intrusion detection (ID). IDtk was developed through a user-centered design process, in which we identi- fied design guidelines to support ID users. ID analysts protect their networks by searching for evidence of attacks in ID system output, firewall and system logs, and other complex, textual data sources. Monitoring and analyzing these sources incurs a heavy cognitive load for analysts. The use of information visualization techniques offers a valuable addition to the toolkit of the ID analyst. Several visualization techniques for ID have been developed, but few usability or field studies have been completed to assess the needs of ID analysts and the usability and usefulness of these tools. We intended to fill this gap by applying a user-centered design process in the development and evaluation of IDtk, a 3D, glyph-based visualization tool that gives the user maximum flexibility in setting up how the visualization display represents ID data. The user can also customize whether the display is a simple, high-level overview to support monitoring, or a more complex 3D view allowing for viewing the data from multiple angles and thus supporting analysis and diagnosis. This flexibility was found crucial in our usability evaluation.

The work of intrusion detection (ID) in accomplishing network security is complex, requiring highly sought-after expertise. While limited automation exists, the role of human ID analysts remains crucial. This paper presents the results of an exploratory field study examining the role of expertise and collaboration in ID work. Through an analysis of the common and situated expertise required in ID work, our results counter basic assumptions about its individualistic character, revealing significant distributed collaboration. Current ID support tools provide no support for this collaborative problem solving. The results of this research highlight ID as an engaging CSCW work domain, one rich with organizational insights, design challenges, and practical import.