Existing frameworks in visualization and HCI emphasize iteration, data grounding, and stakeholder needs; however, they have not fully explored how evaluation might persist across phases, adapt to compressed timelines, and aid stakeholder engagement and elicitation. Building on prior frameworks, we introduce Evaluation-first design EvalOps that centers evaluation as a material component in the design process, emphasizing tighter feedback loops, co-evaluation with stakeholders, malleable forms of evaluation, and goals-to-metrics grounding. We illustrate how EvalOps shapes design outcomes through two case studies of data-visualization and LLM-enabled reasoning tools, demonstrating how evaluation-driven design facilitates alignment and trust, uncovers opportunities earlier, and supports cohesiveness under rapidly changing constraints. We contrast EvalOps with current visualization design methodologies and discuss opportunities for expanding evaluation-centered framings to other active areas of design research.

Online user studies of visualizations, visual encodings, and interaction techniques are ubiquitous in visualization research. Yet, designing, conducting, and analyzing studies effectively is still a major burden. Although various packages support such user studies, most solutions address only facets of the experiment life cycle, make reproducibility difficult, or do not cater to nuanced study designs or interactions. We introduce reVISit 2, a software framework that supports visualization researchers at all stages of designing and conducting browser-based user studies. ReVISit supports researchers in the design, debug & pilot, data collection, analysis, and dissemination experiment phases by providing both technical affordances (such as replay of participant interactions) and sociotechnical aids (such as a mindfully maintained community of support). It is a proven system that can be (and has been) used in publication-quality studies-which we demonstrate through a series of experimental replications. We reflect on the design of the system via interviews and an analysis of its technical dimensions. Through this work, we seek to elevate the ease with which studies are conducted, improve the reproducibility of studies within our community, and support the construction of advanced interactive studies.

We contribute an autoethnographic reflection on the complexity of defining and measuring visualization literacy (i.e., the ability to interpret and construct visualizations) to expose our tacit thoughts that often exist in-between polished works and remain unreported in individual research papers. Our work is inspired by the growing number of empirical studies in visualization research that rely on visualization literacy as a basis for developing effective data representations or educational interventions. Researchers have already made various efforts to assess this construct, yet it is often hard to pinpoint either what we want to measure or what we are effectively measuring. In this autoethnography, we gather insights from 14 internal interviews with researchers who are users or designers of visualization literacy tests. We aim to identify what makes visualization literacy assessment a "wicked" problem. We further reflect on the fluidity of visualization literacy and discuss how this property may lead to misalignment between what the construct is and how measurements of it are used or designed. We also examine potential threats to measurement validity from conceptual, operational, and methodological perspectives. Based on our experiences and reflections, we propose several calls to action aimed at tackling the wicked problem of visualization literacy measurement, such as by broadening test scopes and modalities, improving test ecological validity, making it easier to use tests, seeking interdisciplinary collaboration, and drawing from continued dialogue on visualization literacy to expect and be more comfortable with its fluidity.

Advancements in accessibility technologies such as low-cost swell form printers or refreshable tactile displays promise to allow blind or low-vision (BLV) people to analyze data by transforming visual representations directly to tactile representations. However, it is possible that design guidelines derived from experiments on the visual perception system may not be suited for the tactile perception system. We investigate the potential mismatch between familiar visual encodings and tactile perception in an exploratory study into the strategies employed by BLV people to measure common graphical primitives converted to tactile representations. First, we replicate the Cleveland and McGill study on graphical perception using swell form printing with eleven BLV subjects. Then, we present results from a group interview in which we describe the strategies used by our subjects to read four common chart types. While our results suggest that familiar encodings based on visual perception studies can be useful in tactile graphics, our subjects also expressed a desire to use encodings designed explicitly for BLV people. Based on this study, we identify gaps between the perceptual expectations of common charts and the perceptual tools available in tactile perception. Then, we present a set of guidelines for the design of tactile graphics that accounts for these gaps. Supplemental material is available at https://osf.io/3nsfp/?view_only=7b7b8dcbae1d4c9a8bb4325053d13d9f.

Abstract People craft choropleth maps to monitor, analyze, and understand spatially distributed data. Recent visualization work has addressed several known biases in choropleth maps by developing new model‐ and metrics‐ based approaches (e.g. Bayesian surprise). However, effective use of these techniques requires extensive parameter setting and tuning, making them difficult or impossible for users without substantial technical skills. In this paper we describe SurpriseExplora, which addresses this gap through direct manipulation techniques for re‐targeting a Bayesian surprise model's scope and parameters. We present three use cases to illustrate the capabilities of SurpriseExplora, showing for example how models calculated at a national level can obscure key findings that can be revealed through interaction sequences common to map visualizations (e.g. zooming), and how augmenting funnel‐plot visualizations with interactions that adjust underlying models can account for outliers or skews in spatial datasets. We evaluate SurpriseExplora through an expert review with visualization researchers and practitioners. We conclude by discussing how SurpriseExplora uncovers new opportunities for sense‐making within the broader ecosystem of map visualizations, as well as potential empirical studies with non‐expert populations. Code and demo video available at https://osf.io/7m89w/

Abstract Decisions involving algorithmic rankings affect our lives in many ways, from product recommendations, receiving scholarships, to securing jobs. While tools have been developed for interactively constructing fair consensus rankings from a handful of rankings, addressing the more complex real‐world scenario— where diverse opinions are represented by a larger collection of rankings— remains a challenge. In this paper, we address these challenges by reformulating the exploration of rankings as a dimension reduction problem in a system called FairSpace. FairSpace provides new views, including Fair Divergence View and Cluster Views, by juxtaposing fairness metrics of different local and alternative global consensus rankings to aid ranking analysis tasks. We illustrate the effectiveness of FairSpace through a series of use cases, demonstrating via interactive workflows that users are empowered to create local consensuses by grouping rankings similar in their fairness or utility properties, followed by hierarchically aggregating local consensuses into a global consensus through direct manipulation. We discuss how FairSpace opens the possibility for advances in dimension reduction visualization to benefit the research area of supporting fair decision‐making in ranking based decision‐making contexts. Code, datasets and demo video available at: osf.io/d7cwk

Visualization items-factual questions about visualizations that ask viewers to accomplish visualization tasks-are regularly used in the field of information visualization as educational and evaluative materials. For example, researchers of visualization literacy require large, diverse banks of items to conduct studies where the same skill is measured repeatedly on the same participants. Yet, generating a large number of high-quality, diverse items requires significant time and expertise. To address the critical need for a large number of diverse visualization items in education and research, this paper investigates the potential for large language models (LLMS) to automate the generation of multiple-choice visualization items. Through an iterative design process, we develop the VILA (Visualization Items Generated by Large LAnguage Models) pipeline, for efficiently generating visualization items that measure people's ability to accomplish visualization tasks. We use the VILA pipeline to generate 1,404 candidate items across 12 chart types and 13 visualization tasks. In collaboration with 11 visualization experts, we develop an evaluation rulebook which we then use to rate the quality of all candidate items. The result is the VILA bank of ~1, 100 items. From this evaluation, we also identify and classify current limitations of the VILA pipeline, and discuss the role of human oversight in ensuring quality. In addition, we demonstrate an application of our work by creating a visualization literacy test, VILA-VLAT, which measures people's ability to complete a diverse set of tasks on various types of visualizations; comparing it to the existing VLAT, VILA-VLAT shows moderate to high convergent validity (R = 0.70). Lastly, we discuss the application areas of the VILA pipeline and the VILA bank and provide practical recommendations for their use. All supplemental materials are available at https://osf.io/ysrhq/.

Data visualizations are reaching global audiences. As people who use Right-to-left (RTL) scripts constitute over a billion potential data visualization users, a need emerges to investigate how visualizations are communicated to them. Web design guidelines exist to assist designers in adapting different reading directions, yet we lack a similar standard for visualization design. This paper investigates the design patterns of visualizations with RTL scripts. We collected 128 visualizations from data-driven articles published in Arabic news outlets and analyzed their chart composition, textual elements, and sources. Our analysis suggests that designers tend to apply RTL approaches more frequently for categorical data. In other situations, we observed a mix of Left-to-right (LTR) and RTL approaches for chart directions and structures, sometimes inconsistently utilized within the same article. We reflect on this lack of clear guidelines for RTL data visualizations and derive implications for visualization authoring tools and future research directions.

Visualization literacy is an essential skill for accurately interpreting data to inform critical decisions. Consequently, it is vital to understand the evolution of this ability and devise targeted interventions to enhance it, requiring concise and repeatable assessments of visualization literacy for individuals. However, current assessments, such as the Visualization Literacy Assessment Test (VLAT), are time-consuming due to their fixed, lengthy format. To address this limitation, we develop two streamlined computerized adaptive tests (CATs) for visualization literacy, A-VLAT and A-CALVI, which measure the same set of skills as their original versions in half the number of questions. Specifically, we (1) employ item response theory (IRT) and non-psychometric constraints to construct adaptive versions of the assessments, (2) finalize the configurations of adaptation through simulation, (3) refine the composition of test items of A-CALVI via a qualitative study, and (4) demonstrate the test-retest reliability (ICC: 0.98 and 0.98) and convergent validity (correlation: 0.81 and 0.66) of both CATs via four online studies. We discuss practical recommendations for using our CATs and opportunities for further customization to leverage the full potential of adaptive assessments. All supplemental materials are available at https://osf.io/a6258/.

Graphical perception studies typically measure visualization encoding effectiveness using the error of an "average observer", leading to canonical rankings of encodings for numerical attributes: e.g., position area angle volume. Yet different people may vary in their ability to read different visualization types, leading to variance in this ranking across individuals not captured by population-level metrics using "average observer" models. One way we can bridge this gap is by recasting classic visual perception tasks as tools for assessing individual performance, in addition to overall visualization performance. In this article we replicate and extend Cleveland and McGill's graphical comparison experiment using Bayesian multilevel regression, using these models to explore individual differences in visualization skill from multiple perspectives. The results from experiments and modeling indicate that some people show patterns of accuracy that credibly deviate from the canonical rankings of visualization effectiveness. We discuss implications of these findings, such as a need for new ways to communicate visualization effectiveness to designers, how patterns in individuals' responses may show systematic biases and strategies in visualization judgment, and how recasting classic visual perception tasks as tools for assessing individual performance may offer new ways to quantify aspects of visualization literacy. Experiment data, source code, and analysis scripts are available at the following repository: https://osf.io/8ub7t/?view_only=9be4798797404a4397be3c6fc2a68cc0.

How deep neural networks can aid visualization perception research is a wide-open question. This paper provides insights from three perspectives—prediction, generalization, and interpretation—via training and analyzing deep convolutional neural networks on human correlation judgments in scatterplots across three studies. The first study assesses the accuracy of twenty-nine neural network architectures in predicting human judgments, finding that a subset of the architectures (e.g., VGG-19) has comparable accuracy to the best-performing regression analyses in prior research. The second study shows that the resulting models from the first study display better generalizability than prior models on two other judgment datasets for different scatterplot designs. The third study interprets visual features learned by a convolutional neural network model, providing insights about how the model makes predictions, and identifies potential features that could be investigated in human correlation perception studies. Together, this paper suggests that deep neural networks can serve as a tool for visualization perception researchers in devising potential empirical study designs and hypothesizing about perpetual judgments. The preprint, data, code, and training logs are available at https://doi.org/10.17605/osf.io/exa8m.

Implicit gender bias has costly and complex consequences for women in the workplace, with many women reporting gender microaggressions which result in them being overlooked or disrespected. We present an online desktop virtual environment that follows the story of a male or female self-avatar from the first-person perspective, who either experiences a positive or negative workplace scenario. The negative scenario included many examples from the taxonomy of gender microaggressions. Participants who experienced negative workplace experiences with a female self-avatar had significantly decreased levels of implicit gender bias compared to those who had a male self-avatar. There was evidence of empathy and perspective taking in the negative condition for the female self-avatar. Experiences of a positive workplace scenario showed no significant decreases in implicit gender bias regardless of self-avatar gender. We discuss the implications of these findings and make recommendations for virtual environment technologies and scenarios with respect to the reduction of implicit biases.

Visualizations today are used across a wide range of languages and cultures. Yet the extent to which language impacts how we reason about data and visualizations remains unclear. In this paper, we explore the intersection of visualization and language through a cross-language study on estimative probability tasks with icon-array visualizations. Across Arabic, English, French, German, and Mandarin, n=50 participants per language both chose probability expressions - e.g. likely, probable - to describe icon-array visualizations (Vis-to-Expression), and drew icon-array visualizations to match a given expression (Expression-to-Vis). Results suggest that there is no clear one-to-one mapping of probability expressions and associated visual ranges between languages. Several translated expressions fell significantly above or below the range of the corresponding English expressions. Compared to other languages, French and German respondents appear to exhibit high levels of consistency between the visualizations they drew and the words they chose. Participants across languages used similar words when describing scenarios above 80% chance, with more variance in expressions targeting mid-range and lower values. We discuss how these results suggest potential differences in the expressiveness of language as it relates to visualization interpretation and design goals, as well as practical implications for translation efforts and future studies at the intersection of languages, culture, and visualization. Experiment data, source code, and analysis scripts are available at the following repository: https://osf.io/g5d4r/.

Immersive visualization in virtual reality (VR) allows us to exploit visual cues for perception in 3D space, yet few existing studies have measured the effects of visual cues. Across a desktop monitor and a head-mounted display (HMD), we assessed scatterplot designs which vary their use of visual cues-motion, shading, perspective (graphical projection), and dimensionality-on two sets of data. We conducted a user study with a summary task in which 32 participants estimated the classification accuracy of an artificial neural network from the scatterplots. With Bayesian multilevel modeling, we capture the intricate visual effects and find that no cue alone explains all the variance in estimation error. Visual motion cues generally reduce participants' estimation error; besides this motion, using other cues may increase participants' estimation error. Using an HMD, adding visual motion cues, providing a third data dimension, or showing a more complicated dataset leads to longer response times. We speculate that most visual cues may not strongly affect perception in immersive analytics unless they change people's mental model about data. In summary, by studying participants as they interpret the output from a complicated machine learning model, we advance our understanding of how to use the visual cues in immersive analytics.

Advancements in accessibility have led to mobile applications that help blind or low vision people (BLV) access surrounding information independently. Unfortunately, accessibility of visual information such as images remains limited. Previous research demonstrated that spatial interaction can help BLV users build a mental model of the relative locations of image objects. As haptics has recently become a core component of modern smartphones, we extend this prior research by designing three prototypes that use haptic feedback to reveal object location in images. We evaluate these techniques in terms of experience and ability of BLV users to locate multiple objects in images. Evaluation results in a preliminary study with seven BLV users suggest that the proposed haptic feedback prototype with auditory notifications to identify people and auditory caption can provide a more accessible and engaging image experience.

Combining the preferences of many rankers into one single consensus ranking is critical for consequential applications from hiring and admissions to lending. While group fairness has been extensively studied for classification, group fairness in rankings and in particular rank aggregation remains in its infancy. Recent work introduced the concept of fair rank aggregation for combining rankings but restricted to the case when candidates have a single binary protected attribute, i.e., they fall into two groups only. Yet it remains an open problem how to create a consensus ranking that represents the preferences of all rankers while ensuring fair treatment for candidates with multiple protected attributes such as gender, race, and nationality. In this work, we are the first to define and solve this open Multi-attribute Fair Consensus Ranking (MFCR) problem. As a foundation, we design novel group fairness criteria for rankings, called MANI-Rank, ensuring fair treatment of groups defined by individual protected attributes and their intersection. Leveraging the MANI-Rank criteria, we develop a series of algorithms that for the first time tackle the MFCR problem. Our experimental study with a rich variety of consensus scenarios demonstrates our MFCR methodology is the only approach to achieve both intersectional and protected attribute fairness while also representing the preferences expressed through many base rankings. Our real world case study on merit scholarships illustrates the effectiveness of our MFCR methods to mitigate bias across multiple protected attributes and their intersections.

Interactive web-based applications play an important role for both service providers and consumers. However, web applications tend to be complex, produce high-volume data, and are often ripe for attack. Attack analysis and remediation are complicated by adversary obfuscation and the difficulty in assembling and analyzing logs. In this work, we explore the web application analysis task through log file fusion, distillation, and visualization. Our approach consists of visualizing the logs of web and database traffic with detailed function execution traces. We establish causal links between events and their associated behaviors. We evaluate the effectiveness of this process using data volume reduction statistics, user interaction models, and usage scenarios. Across a set of scenarios, we find that our techniques can filter at least 97.5% of log data and reduce analysis time by 93–96%.

Quantifying user performance with metrics such as time and accuracy does not show the whole picture when researchers evaluate complex, interactive visualization tools. In such systems, performance is often influenced by different analysis strategies that statistical analysis methods cannot account for. To remedy this lack of nuance, we propose a novel analysis methodology for evaluating complex interactive visualizations at scale. We implement our analysis methods in reVISit, which enables analysts to explore participant interaction performance metrics and responses in the context of users’ analysis strategies. Replays of participant sessions can aid in identifying usability problems during pilot studies and make individual analysis processes salient. To demonstrate the applicability of reVISit to visualization studies, we analyze participant data from two published crowdsourced studies. Our findings show that reVISit can be used to reveal and describe novel interaction patterns, to analyze performance differences between different analysis strategies, and to validate or challenge design decisions.

Abstract Incident report triage is a common endeavor in many industry sectors, often coupled with serious public safety implications. For example, at the US Food and Drug Administration (FDA), analysts triage an influx of incident reports to identify previously undiscovered drug safety problems. However, these analysts currently conduct this critical yet error‐prone incident report triage using a generic table‐based interface, with no formal support. Visualization design, task‐characterization methodologies, and evaluation models offer several possibilities for better supporting triage workflows, including those dealing with drug safety and beyond. In this work, we aim to elevate the work of triage through a task‐abstraction activity with FDA analysts. Second, we design an alternative gist‐based summary of text documents used in triage (SumRe). Third, we conduct a crowdsourced evaluation of SumRe with medical experts. Results of the crowdsourced study with medical experts (n = 20) suggest that SumRe better supports accuracy in understanding the gist of a given report, and in identifying important reports for followup activities. We discuss implications of these results, including design considerations for triage workflows beyond the drug domain, as well as methodologies for comparing visualization‐enabled text summaries.

Data visualization is powerful in large part because it facilitates visual extraction of values. Yet, existing measures of perceptual precision for data channels (e.g., position, length, orientation, etc.) are based largely on verbal reports of ratio judgments between two values (e.g., [7]). Verbal report conflates multiple sources of error beyond actual visual precision, introducing a ratio computation between these values and a requirement to translate that ratio to a verbal number. Here we observe raw measures of precision by eliminating both ratio computations and verbal reports; we simply ask participants to reproduce marks (a single bar or dot) to match a previously seen one. We manipulated whether the mark was initially presented (and later drawn) alone, paired with a reference (e.g. a second '100%' bar also present at test, or a y-axis for the dot), or integrated with the reference (merging that reference bar into a stacked bar graph, or placing the dot directly on the axis). Reproductions of smaller values were overestimated, and larger values were underestimated, suggesting systematic memory biases. Average reproduction error was around 10% of the actual value, regardless of whether the reproduction was done on a common baseline with the original. In the reference and (especially) the integrated conditions, responses were repulsed from an implicit midpoint of the reference mark, such that values above 50% were overestimated, and values below 50% were underestimated. This reproduction paradigm may serve within a new suite of more fundamental measures of the precision of graphical perception.

Visualizing multivariate networks is challenging because of the trade-offs necessary for effectively encoding network topology and encoding the attributes associated with nodes and edges. A large number of multivariate network visualization techniques exist, yet there is little empirical guidance on their respective strengths and weaknesses. In this paper, we describe a crowdsourced experiment, comparing node-link diagrams with on-node encoding and adjacency matrices with juxtaposed tables. We find that node-link diagrams are best suited for tasks that require close integration between the network topology and a few attributes. Adjacency matrices perform well for tasks related to clusters and when many attributes need to be considered. We also reflect on our method of using validated designs for empirically evaluating complex, interactive visualizations in a crowdsourced setting. We highlight the importance of training, compensation, and provenance tracking.

Rankings distill a large number of factors into simple comparative models to facilitate complex decision making. Yet key questions remain in the design of mixed-initiative systems for ranking, in particular how best to collect users' preferences to produce high-quality rankings that users trust and employ in the real world. To address this challenge we evaluate the relative merits of three preference collection methods for ranking in a crowdsourced study. We find that with a categorical binning technique, users interact with a large amount of data quickly, organizing information using broad strokes. Alternative interaction modes using pairwise comparisons or sub-lists result in smaller, targeted input from users. We consider how well each interaction mode addresses design goals for interactive ranking systems. Our study indicates that the categorical approach provides the best value-added benefit to users, requiring minimal effort to create sufficient training data for the underlying ranking algorithm.

Abstract Adverse reactions caused by drug‐drug interactions are a major public health concern. Currently, adverse reaction signals are detected through a tedious manual process in which drug safety analysts review a large number of reports collected through post‐marketing drug surveillance. While computational techniques in support of this signal analysis are necessary, alone they are not sufficient. In particular, when machine learning techniques are applied to extract candidate signals from reports, the resulting set is (1) too large in size, i.e., exponential to the number of unique drugs and reactions in reports, (2) disconnected from the underlying reports that serve as evidence and context, and (3) ultimately requires human intervention to be validated in the domain context as a true signal warranting action. In this work, we address these challenges though a visual analytics system, DIVA, designed to align with the drug safety analysis workflow by supporting the detection, screening, and verification of candidate drug interaction signals. DTVA's abstractions and encodings are informed by formative interviews with drug safety analysts. DIVA's coordinated visualizations realize a proposed novel augmented interaction data model (AIM) which links signals generated by machine learning techniques with domain‐specific metadata critical for signal analysis. DIVA's alignment with the drug review process allows an analyst to interactively screen for important signals, triage signals for in‐depth investigation, and validate signals by reviewing the underlying reports that serve as evidence. The evaluation of DIVA encompasses case‐studies and interviews by drug analysts at the US Food and Drug Administration ‐ both of which confirm that DIVA indeed is effective in supporting analysts in the critical task of exploring and verifying dangerous drug‐drug interactions.

The diverse and vibrant ecosystem of interactive visualizations on the web presents an opportunity for researchers and practitioners to observe and analyze how everyday people interact with data visualizations. However, existing metrics of visualization interaction behavior used in research do not fully reveal the breadth of peoples' open-ended explorations with visualizations. One possible way to address this challenge is to determine high-level goals for visualization interaction metrics, and infer corresponding features from user interaction data that characterize different aspects of peoples' explorations of visualizations. In this paper, we identify needs for visualization behavior measurement, and develop corresponding candidate features that can be inferred from users' interaction data. We then propose metrics that capture novel aspects of peoples' open-ended explorations, including exploration uniqueness and exploration pacing. We evaluate these metrics along with four other metrics recently proposed in visualization literature by applying them to interaction data from prior visualization studies. The results of these evaluations suggest that these new metrics 1) reveal new characteristics of peoples' use of visualizations, 2) can be used to evaluate statistical differences between visualization designs, and 3) are statistically independent of prior metrics used in visualization research. We discuss implications of these results for future studies, including the potential for applying these metrics in visualization interaction analysis, as well as emerging challenges in developing and selecting metrics depicting visualization explorations.

Recent visualization research efforts have incorporated experimental techniques and perceptual models from the vision science community. Perceptual laws such as Weber's law, for example, have been used to model the perception of correlation in scatterplots. While this thread of research has progressively refined the modeling of the perception of correlation in scatterplots, it remains unclear as to why such perception can be modeled using relatively simple functions, e.g., linear and log-linear. In this paper, we investigate a longstanding hypothesis that people use visual features in a chart as a proxy for statistical measures like correlation. For a given scatterplot, we extract 49 candidate visual features and evaluate which best align with existing models and participant judgments. The results support the hypothesis that people attend to a small number of visual features when discriminating correlation in scatterplots. We discuss how this result may account for prior conflicting findings, and how visual features provide a baseline for future model-based approaches in visualization evaluation and design.

The scatterplot matrix (SPLOM) is a commonly used technique for visualizing multiclass multivariate data. However, multiclass SPLOMs have issues with overdraw (overlapping points), and most existing techniques for alleviating overdraw focus on individual scatterplots with a single class. This paper explores whether animation using flickering points is an effective way to alleviate overdraw in these multiclass SPLOMs. In a user study with 69 participants, we found that users not only performed better at identifying dense regions using animated SPLOMs, but also found them easier to interpret and preferred them to static SPLOMs. These results open up new directions for future work on alleviating overdraw for multiclass SPLOMs, and provide insights for applying animation to alleviate overdraw in other settings.

The widespread use of text-based search in user interfaces has led designers in visualization to occasionally add search functionality to their creations. Yet it remains unclear how search may impact a person's behavior. Given the unstructured context of the web, users may not have explicit information-seeking goals and designers cannot make assumptions about user attention. To bridge this gap, we observed the impact of integrating search with five visualizations across 830 online participants. In an unguided task, we find that (1) the presence of text-based search influences people's information-seeking goals, (2) search can alter the data that people explore and how they engage with it, and (3) the effects of search are amplified in visualizations where people are familiar with the underlying dataset. These results suggest that text-search in web visualizations drives users towards more diverse information seeking goals, and may be valuable in a range of existing visualization designs.

Drug-drug interaction related adverse events (DIAE) signals are a major public health issue. Drug safety analysts must sift through thousands of adverse event reports submitted daily to U.S. Food and Drug Administration (FDA) to discover unexpected DIAE signals, which if addressed can lead to life-saving actions. To facilitate the DIAE discovery from these massive data sets, we design several technological innovations that together are integrated into an interactive visual analytics system called DEVES 1. First, our state- of-the-art DIAE mining algorithm efficiently infers potential DIAE signals from these reports, and then ranks them based on their significance score. For interpretability of these inferred DIAE signals, domain knowledge of adverse events and already known drug interactions is extracted from external authoritative data sources and then seamlessly integrated with the inferred signal set. Guided by this augmented signal model, DEVES supports advanced signal analytics - empowering the analyst to interact with linked visualizations offering complementary perspectives into the signal set and its supporting evidence in the form of reports. Our demonstration showcases the technological innovations of DEVES using real-world FDA datasets, demonstrating that DEVES effectively supports the core regulatory tasks from signal screening, signal prioritization to signal validation.

Manually constructing rankings is a tedious ad-hoc process, requiring extensive user effort to evaluate data attribute importance, and often leading to undesirable results. Meanwhile, sophisticated learning-to-rank algorithms are able to leverage large amounts of data to generate high quality rankings automatically. In this work we present RanKit, a novel technology that brings the power of automatic learning-to-rank to the public. RanKit serves as a personal recommender system for building rankings from partial user knowledge in the form of item preferences. A user-friendly rank building interface provides rich input modes for preference specification. Visual feedback on the quality of the learned ranking model is given in real time, empowering the user to guide the underlying learn-to-rank algorithm. Users are actively involved with every step of the rank generation process, developing trust in the model and producing personalized rankings suitable for real-world decision making. In this demonstration, the audience works directly with the RanKit system on public domain datasets ranging from college rankings and economic indicators to movies and sports.

Adverse drug reactions (ADRs) caused by drug-drug interactions (DDI) are a major cause of morbidity and mortality worldwide. There is a growing need for computing-supported methods that facilitate the automated signaling of DDI related ADRs (DIARs) that otherwise would remain undiscovered in millions of ADR reports. In this demonstration, we showcase our MeDIAR technology - an end-to-end DIAR signal generation, exploration and validation solution for pharmaceutical regulatory agencies to detect true DIAR signals from a drug surveillance database. MeDIAR's innovations include an efficient rule-driven learning algorithm for deriving DIAR signals from ADR reports, an innovative scoring methodology based on the proposed contextual association cluster model to rank the generated signals by their importance. Further, these ranked signals are augmented with meta information such as their significance level and their severity, along with links to their supporting ADR reports. Lastly, MeDIAR features an interactive visual analytics interface to support drug safety evaluators in reviewing and discovering unknown severe DIARs.

Physical and digital objects often leave markers of our use. Website links turn purple after we visit them, for example, showing us information we have yet to explore. These "footprints" of interaction offer substantial benefits in information saturated environments - they enable us to easily revisit old information, systematically explore new information, and quickly resume tasks after interruption. While applying these design principles have been successful in HCI contexts, direct encodings of personal interaction history have received scarce attention in data visualization. One reason is that there is little guidance for integrating history into visualizations where many visual channels are already occupied by data. More importantly, there is not firm evidence that making users aware of their interaction history results in benefits with regards to exploration or insights. Following these observations, we propose HindSight - an umbrella term for the design space of representing interaction history directly in existing data visualizations. In this paper, we examine the value of HindSight principles by augmenting existing visualizations with visual indicators of user interaction history (e.g. How the Recession Shaped the Economy in 255 Charts, NYTimes). In controlled experiments of over 400 participants, we found that HindSight designs generally encouraged people to visit more data and recall different insights after interaction. The results of our experiments suggest that simple additions to visualizations can make users aware of their interaction history, and that these additions significantly impact users' exploration and insights.

Prostate cancer is the most common cancer among men in the US, and yet most cases represent localized cancer for which the optimal treatment is unclear. Accumulating evidence suggests that the available treatment options, including surgery and conservative treatment, result in a similar prognosis for most men with localized prostate cancer. However, approximately 90% of patients choose surgery over conservative treatment, despite the risk of severe side effects like erectile dysfunction and incontinence. Recent medical research suggests that a key reason is the lack of patient-centered tools that can effectively communicate personalized risk information and enable them to make better health decisions. In this paper, we report the iterative design process and results of developing the PROgnosis Assessment for Conservative Treatment (PROACT) tool, a personalized health risk communication tool for localized prostate cancer patients. PROACT utilizes two published clinical prediction models to communicate the patients' personalized risk estimates and compare treatment options. In collaboration with the Maine Medical Center, we conducted two rounds of evaluations with prostate cancer survivors and urologists to identify the design elements and narrative structure that effectively facilitate patient comprehension under emotional distress. Our results indicate that visualization can be an effective means to communicate complex risk information to patients with low numeracy and visual literacy. However, the visualizations need to be carefully chosen to balance readability with ease of comprehension. In addition, due to patients' charged emotional state, an intuitive narrative structure that considers the patients' information need is critical to aid the patients' comprehension of their risk information.

We present Brain Automated Chorales (BACh), an adaptive brain-computer system that dynamically increases the levels of difficulty in a musical learning task based on pianists' cognitive workload measured by functional near-infrared spectroscopy. As users' cognitive workload fell below a certain threshold, suggesting that they had mastered the material and could handle more cognitive information, BACh automatically increased the difficulty of the learning task. We found that learners played with significantly increased accuracy and speed in the brain-based adaptive task compared to our control condition. Participant feedback indicated that they felt they learned better with BACh and they liked the timings of the level changes. The underlying premise of BACh can be applied to learning situations where a task can be broken down into increasing levels of difficulty.

Decades of research have repeatedly shown that people perform poorly at estimating and understanding conditional probabilities that are inherent in Bayesian reasoning problems. Yet in the medical domain, both physicians and patients make daily, life-critical judgments based on conditional probability. Although there have been a number of attempts to develop more effective ways to facilitate Bayesian reasoning, reports of these findings tend to be inconsistent and sometimes even contradictory. For instance, the reported accuracies for individuals being able to correctly estimate conditional probability range from 6% to 62%. In this work, we show that problem representation can significantly affect accuracies. By controlling the amount of information presented to the user, we demonstrate how text and visualization designs can increase overall accuracies to as high as 77%. Additionally, we found that for users with high spatial ability, our designs can further improve their accuracies to as high as 100%. By and large, our findings provide explanations for the inconsistent reports on accuracy in Bayesian reasoning tasks and show a significant improvement over existing methods. We believe that these findings can have immediate impact on risk communication in health-related fields.

Information graphics, or infographics, combine elements of data visualization with design and have become an increasingly popular means for disseminating data. While several studies have suggested that aesthetics in visualization and infographics relate to desirable outcomes like engagement and memorability, it remains unknown how quickly aesthetic impressions are formed, and what it is that makes an infographic appealing. We address these questions by analyzing 1,278 participants' ratings on appeal after seeing infographics for 500ms. Our results establish that: 1) people form a reliable first impression of the appeal of an infographic based on a mere exposure effect, 2) this first impression is largely based on colorfulness and visual complexity, and 3) age, gender, and education level influence the preferred level of colorfulness and complexity. More generally, these findings suggest that outcomes such as engagement and memorability might be determined much earlier than previously thought.

Abstract As data visualization becomes further intertwined with the field of graphic design and information graphics, small graphical alterations are made to many common chart formats. Despite the growing prevalence of these embellishments, their effects on communication of the charts’ data is unknown. From an overview of the design space, we have outlined some of the common embellishments that are made to bar charts. We have studied the effects of these chart embellishments on the communication of the charts’ data through a series of user studies on Amazon's Mechanical Turk platform. The results of these studies lead to a better understanding of how each chart type is perceived, and help provide guiding principles for the graphic design of charts.

The Visualization for Cyber Security research community (VizSec) addresses longstanding challenges in cyber security by adapting and evaluating information visualization techniques with application to the cyber security domain. This research effort has created many tools and techniques that could be applied to improve cyber security, yet the community has not yet established unified standards for evaluating these approaches to predict their operational validity. In this paper, we survey and categorize the evaluation metrics, components, and techniques that have been utilized in the past decade of VizSec research literature. We also discuss existing methodological gaps in evaluating visualization in cyber security, and suggest potential avenues for future research in order to help establish an agenda for advancing the state-of-the-art in evaluating cyber security visualizations.

Despite years of research yielding systems and guidelines to aid visualization design, practitioners still face the challenge of identifying the best visualization for a given dataset and task. One promising approach to circumvent this problem is to leverage perceptual laws to quantitatively evaluate the effectiveness of a visualization design. Following previously established methodologies, we conduct a large scale (n=1687) crowdsourced experiment to investigate whether the perception of correlation in nine commonly used visualizations can be modeled using Weber's law. The results of this experiment contribute to our understanding of information visualization by establishing that: (1) for all tested visualizations, the precision of correlation judgment could be modeled by Weber's law, (2) correlation judgment precision showed striking variation between negatively and positively correlated data, and (3) Weber models provide a concise means to quantify, compare, and rank the perceptual precision afforded by a visualization.

Recent research suggests that individual personality differences can influence performance with visualizations. In addition to stable personality traits, research in psychology has found that temporary changes in affect (emotion) can also significantly impact performance during cognitive tasks. In this paper, we show that affective priming also influences user performance on visual judgment tasks through an experiment that combines affective priming with longstanding graphical perception experiments. Our results suggest that affective priming can influence accuracy in common graphical perception tasks. We discuss possible explanations for these findings, and describe how these findings can be applied to design visualizations that are less (or more) susceptible to error in common visualization contexts.

Collaboration is prevalent for network security teams to protect networking environments, yet few network visualization tools are designed for collaborative analysis. With the increasing complexity and volume of dynamic networks, it is important to adopt strategies of joint decision-making through developing collaborative visualization approaches. In this paper, we present a formal user study to evaluate how paired users collaborate under co-located and distributed collaboration environments to tackle the problems of intrusion detection. Ten paired participants are requested to use network visualization patterns to identify attacks existed in the datasets. We observe participants behaviors and collect their performances from the aspects of coordination and communication, which include prioritizing goals and directions, dividing and balancing workloads, and negotiating analysis decisions while maintaining situational awareness. Based on the results, we conclude several coordination strategies and summarize the values of communication for collaborative detection. We also discuss human-related factors in the process of joint decision-making. Our study provides useful information for future design and development of collaborative visualization systems.

Network vulnerability is a critical component of network security. Yet vulnerability analysis has received relatively little attention from the security visualization community. This paper describes nv, a web-based Nessus vulnerability visualization. Nv utilizes treemaps and linked histograms to allow security analysts and systems administrators to discover, analyze, and manage vulnerabilities on their networks. In addition to visualizing single Nessus scans, nv supports the analysis of sequential scans by showing which vulnerabilities have been fixed, remain open, or are newly discovered. Nv operates completely in-browser, to avoid sending sensitive data to outside servers. We discuss the design of nv, as well as provide case studies demonstrating vulnerability analysis workflows which include a multiple-node testbed and data from the 2011 VAST Challenge.

This paper presents a new approach to intrusion detection that supports the identification and analysis of network anomalies using an interactive coordinated multiple views (CMV) mechanism. A CMV visualization consisting of a node-link diagram, scatterplot, and time histogram is described that allows interactive analysis from different perspectives, as some network anomalies can only be identified through joint features in the provided spaces. Spectral analysis methods are integrated to provide visual cues that allow identification of malicious nodes. An adjacency-based method is developed to generate the time histogram, which allows users to select time ranges in which suspicious activity occurs. Data from Sybil attacks in simulated wireless networks is used as the test bed for the system. The results and discussions demonstrate that intrusion detection can be achieved with a few iterations of CMV exploration. Quantitative results are collected on the accuracy of our approach and comparisons are made to single domain exploration and other high-dimensional projection methods. We believe that this approach can be extended to anomaly detection in general networks, particularly to Internet networks and social networks.

This paper presents the design of the final stage of a new game currently in development, entitled cMotion, which will use virtual humans to teach emotion recognition and programming concepts to children. Having multiple facets, cMotion is designed to teach the intended users how to recognize facial expressions and manipulate an interactive virtual character using a visual drag-and-drop programming interface. By creating a game which contextualizes emotions, we hope to foster learning of both emotions in a cultural context and computer programming concepts in children. The game will be completed in three stages which will each be tested separately: a playable introduction which focuses on social skills and emotion recognition, an interactive interface which focuses on computer programming, and a full game which combines the first two stages into one activity.