To ensure accountability and mitigate harm, it is critical that diverse stakeholders can interrogate black-box automated systems and find information that is understandable, relevant, and useful to them. In this paper, we eschew prior expertise- and role-based categorizations of interpretability stakeholders in favor of a more granular framework that decouples stakeholders’ knowledge from their interpretability needs. We characterize stakeholders by their formal, instrumental, and personal knowledge and how it manifests in the contexts of machine learning, the data domain, and the general milieu. We additionally distill a hierarchical typology of stakeholder needs that distinguishes higher-level domain goals from lower-level interpretability tasks. In assessing the descriptive, evaluative, and generative powers of our framework, we find our more nuanced treatment of stakeholders reveals gaps and opportunities in the interpretability literature, adds precision to the design and comparison of user studies, and facilitates a more reflexive approach to conducting this research.

Risk scoring provides a simple and quantifiable metric for decision support in cyber security operations, including prioritizing how to address discovered software vulnerabilities. However, scoring systems are often opaque to operators, which makes scores difficult to interpret in the context of their own networks, each other, or in a broader threat landscape. This interpretability challenge is exacerbated by recent applications of artificial intelligence (AI) and machine learning (ML) for vulnerability assessment, where opaque machine reasoning can hinder domain experts' trust in the decision-support toolkit or the actionability of its outputs. In this paper, we address this challenge through a combination of visualizations and analytics that complement existing techniques for vulnerability assessment. We present a study toward designing more interpretable visual encodings for decision support for vulnerability assessment. In particular, we consider the problem of making datasets of known vulnerabilities more interpretable at multiple scales, inspired by focus and context principles from the information visualization design community. The first scale considers individually scored vulnerabilities by using an explainable AI (XAI) toolkit for an ML risk-scoring model and by developing new visualizations of CVSS score features. The second scale uses an embedding for vulnerability descriptions to cluster potentially similar vulnerabilities. We outline use cases for these tools and discuss opportunities for applying XAI concepts to cyber risk and vulnerability management.

Conventional network access control approaches are static (e.g., user roles in Active Directory), coarse-grained (e.g., 802.1x), or both (e.g., VLANs). Such systems are unable to meaningfully stop or hinder motivated attackers seeking to spread throughout an enterprise network. To address this threat, we present Dynamic Flow Isolation (DFI), a novel architecture for supporting dynamic, fine-grained access control policies enforced in a Software-Defined Network (SDN). These policies can emit and revoke specific access control rules automatically in response to network events like users logging off, letting the network adaptively reduce unnecessary reachability that could be potentially leveraged by attackers. DFI is oblivious to the SDN controller implementation and processes new packets prior to the controller, making DFI's access control resilient to a malicious or faulty controller or its applications. We implemented DFI for OpenFlow networks and demonstrated it on an enterprise SDN testbed with around 100 end hosts and servers. Finally, we evaluated the performance of DFI and how it enables a novel policy, which is otherwise difficult to enforce, that protects against a surrogate of the recent NotPetya malware in an infection scenario. We found that the threat was most limited in its ability to spread using our policy, which automatically restricted network flows over the course of the attack, compared to no access control or a static role-based policy.

Attackers seeking to deceive web users into visiting malicious websites can exploit limitations of the tools intended to help browsers translate domain names containing non-ASCII characters, or internationalized domain names (IDNs). These attacks, called homograph phishing, involve registering Unicode domain names that are visually similar to legitimate ones but direct users to distinct servers. Tools exist to identify when domains use non-ASCII characters, which get translated by the Punycode protocol to work with the Domain Name System (DNS); however, these tools cannot automatically distinguish between benign use cases and ones with malicious intent, leading to high rates of false-positive alerts and increasing the workload of analysts looking for evidence of homograph phishing.To address this problem, we present PunyVis, a visual analytics system for exploring and identifying potential homograph attacks on large network datasets. By targeting instances of Punycode that use easily-confusable ASCII characters to spoof popular websites, PunyVis quickly condenses large datasets into a small number of potentially malicious records. Using the interactive tool, analysts can evaluate potential phishing instances and view supporting information from multiple data sources, as well as gain insight about overall risk and threat regarding homograph attacks. We demonstrate how PunyVis supports analysts in a case study with domain experts, and identified divergent analysis strategies and the need for interactions that support how analysts begin exploration and pivot around hypotheses. Finally, we discuss design implications and opportunities for cyber visual analytics.

Computer security has long been an arms race between attacks and defenses. While new defenses are proposed and built to stop specific vectors of attacks, novel, sophisticated attacks are devised by attackers to bypass them. This rapid cycle of defenses and attacks has made it difficult to strategically reason about the protection offered by each defensive technique, the coverage of a set of defenses, and possible new vectors of attack for which to design future defenses. In this work, we present QUASAR, a framework that systematically analyzes attacks and defenses at the granularity of the capabilities necessary to mount the attacks. We build a model of attacks in the memory corruption domain, and represent various prominent defenses in this domain. We demonstrate that QUASAR can be used to compare defenses at a fundamental level (what they do instead of how they do it), reason about the coverage of a defensive configuration, and hypothesize about possible new attack strategies. We show that of the top five hypothesized new attack strategies, in fact, four have been published in security venues over the past two years. We investigate the fifth hypothesized vector ourselves and demonstrate that it is, in fact, a viable vector of attack.

We present the design and evaluation of a method for estimating gaze locations during the analysis of static visualizations using crowdsourcing. Understanding gaze patterns is helpful for evaluating visualizations and user behaviors, but traditional eye-tracking studies require specialized hardware and local users. To avoid these constraints, we developed a method called Fauxvea, which crowdsources visualization tasks on the Web and estimates gaze fixations through cursor interactions without eye-tracking hardware. We ran experiments to evaluate how gaze estimates from our method compare with eye-tracking data. First, we evaluated crowdsourced estimates for three common types of information visualizations and basic visualization tasks using Amazon Mechanical Turk (MTurk). In another, we reproduced findings from a previous eye-tracking study on tree layouts using our method on MTurk. Results from these experiments show that fixation estimates using Fauxvea are qualitatively and quantitatively similar to eye tracking on the same stimulus-task pairs. These findings suggest that crowdsourcing visual analysis tasks with static information visualizations could be a viable alternative to traditional eye-tracking studies for visualization research and design.

We present results from an experiment aimed at using logs of interactions with a visual analytics application to better understand how interactions lead to insight generation. We performed an insight-based user study of a visual analytics application and ran post hoc quantitative analyses of participants' measured insight metrics and interaction logs. The quantitative analyses identified features of interaction that were correlated with insight characteristics, and we confirmed these findings using a qualitative analysis of video captured during the user study. Results of the experiment include design guidelines for the visual analytics application aimed at supporting insight generation. Furthermore, we demonstrated an analysis method using interaction logs that identified which interaction patterns led to insights, going beyond insight-based evaluations that only quantify insight characteristics. We also discuss choices and pitfalls encountered when applying this analysis method, such as the benefits and costs of applying an abstraction framework to application-specific actions before further analysis. Our method can be applied to evaluations of other visualization tools to inform the design of insight-promoting interactions and to better understand analyst behaviors.

We present TOME, a novel framework that helps developers quantitatively evaluate user interfaces and design iterations by using histories from crowds of end users. TOME collects user-interaction histories via an interface instrumentation library as end users complete tasks; these histories are compiled using the Keystroke-Level Model (KLM) into task completion-time predictions using CogTool. With many histories, TOME can model prevailing strategies for tasks without needing an HCI specialist to describe users' interaction steps. An unimplemented design change can be evaluated by perturbing a TOME task model in CogTool to reflect the change, giving a new performance prediction. We found that predictions for quick (5-60s) query tasks in an instrumented brain-map interface averaged within 10% of measured expert times. Finally, we modified a TOME model to predict closely the speed-up yielded by a proposed interaction before implementing it.

We present an analysis of two user strategies in interactive data analysis, based on an observational study of four researchers in the immunology domain. Screen captures, video records, interviews, and verbal protocols are used to analyze common procedures in this type of visual data analysis, as well as how these procedures differ among these users. Our findings present a case where skilled users can approach a similar problem with diverging analysis strategies. In the group we observed, strategies fell within two broad categories: within-graph analysis, in which a user generates a few graph layouts and interacts heavily within them, and between-graph analysis, in which a user generates a series of graphs and switches between them in sequence. Differences in strategies lead to distinct interaction patterns, and are likely to be best supported by different interface designs. We characterize these observed strategies and discuss their implications for scientific visualization design and evaluation.

We present an ethnographic study of design differences in visual presentations between academic disciplines. Characterizing design conventions between users and data domains is an important step in developing hypotheses, tools, and design guidelines for information visualization. In this paper, disciplines are compared at a coarse scale between four groups of fields: social, natural, and formal sciences; and the humanities. Two commonplace presentation types were analyzed: electronic slideshows and whiteboard "chalk talks". We found design differences in slideshows using two methods - coding and comparing manually-selected features, like charts and diagrams, and an image-based analysis using PCA called eigenslides. In whiteboard talks with controlled topics, we observed design behaviors, including using representations and formalisms from a participant's own discipline, that suggest authors might benefit from novel assistive tools for designing presentations. Based on these findings, we discuss opportunities for visualization ethnography and human-centered authoring tools for visual information.