As Large Language Models (LLMs) rise in popularity, it is necessary to assess their capability in critically relevant domains. We present a comprehensive evaluation framework, grounded in science communication research, to assess LLM responses to questions about climate change. Our framework emphasizes both presentational and epistemological adequacy, offering a fine-grained analysis of LLM generations spanning 8 dimensions and 30 issues. Our evaluation task is a real-world example of a growing number of challenging problems where AI can complement and lift human performance. We introduce a novel protocol for scalable oversight that relies on AI Assistance and raters with relevant education. We evaluate several recent LLMs on a set of diverse climate questions. Our results point to a significant gap between surface and epistemological qualities of LLMs in the realm of climate communication.

Visualization researchers and practitioners engaged in generating or evaluating designs are faced with the difficult problem of transforming the questions asked and actions taken by target users from domain-specific language and context into more abstract forms. Existing abstract task classifications aim to provide support for this endeavour by providing a carefully delineated suite of actions. Our experience is that this bottom-up approach is part of the challenge: low-level actions are difficult to interpret without a higher-level context of analysis goals and the analysis process. To bridge this gap, we propose a framework based on analysis reports derived from open-coding 20 design study papers published at IEEE InfoVis 2009-2015, to build on the previous work of abstractions that collectively encompass a broad variety of domains. The framework is organized in two axes illustrated by nine analysis goals. It helps situate the analysis goals by placing each goal under axes of specificity (Explore, Describe, Explain, Confirm) and number of data populations (Single, Multiple). The single-population types are Discover Observation, Describe Observation, Identify Main Cause, and Collect Evidence. The multiple-population types are Compare Entities, Explain Differences, and Evaluate Hypothesis. Each analysis goal is scoped by an input and an output and is characterized by analysis steps reported in the design study papers. We provide examples of how we and others have used the framework in a top-down approach to abstracting domain problems: visualization designers or researchers first identify the analysis goals of each unit of analysis in an analysis stream, and then encode the individual steps using existing task classifications with the context of the goal, the level of specificity, and the number of populations involved in the analysis.

Abstract Little is known about how people structure sets of visualizations to support sequential viewing. We contribute findings from several studies examining visualization sequencing and reception. In our first study, people made decisions between various possible structures as they ordered a set of related visualizations (consisting of either bar charts or thematic maps) into what they considered the clearest sequence for showing the data. We find that most people structure visualization sequences hierarchically: they create high level groupings based on shared data properties like time period, measure, level of aggregation, and spatial region, then order the views within these groupings. We also observe a tendency for certain types of similarities between views, like a common spatial region or aggregation level, to be seen as more appropriate categories for organizing views in a sequence than others, like a common time period or measure. In a second study, we find that viewers’ perceptions of the quality and intention of different sequences are largely consistent with the perceptions of the users who created them. The understanding of sequence preferences and perceptions that emerges from our studies has implications for the development of visualization authoring tools and sequence recommendations for guided analysis.

In this paper, we would like to investigate how people make sense of unfamiliar information visualizations. In order to achieve the research goal, we conducted a qualitative study by observing 13 participants when they endeavored to make sense of three unfamiliar visualizations (i.e., a parallel-coordinates plot, a chord diagram, and a treemap) that they encountered for the first time. We collected data including audio/video record of think-aloud sessions and semi-structured interview; and analyzed the data using the grounded theory method. The primary result of this study is a grounded model of NOvice's information Vlsualization Sensemaking (NOVIS model), which consists of the five major cognitive activities: 1 encountering visualization, 2 constructing a frame, 3 exploring visualization, 4 questioning the frame, and 5 floundering on visualization. We introduce the NOVIS model by explaining the five activities with representative quotes from our participants. We also explore the dynamics in the model. Lastly, we compare with other existing models and share further research directions that arose from our observations.

Google Fusion Tables aims to support an ecosystem of structured data on the Web by providing a tool for managing and visualizing data on the one hand, and for searching and exploring for data on the other. This paper describes a few recent developments in our efforts to further the ecosystem.

We take a new, scenario-based look at evaluation in information visualization. Our seven scenarios, evaluating visual data analysis and reasoning, evaluating user performance, evaluating user experience, evaluating environments and work practices, evaluating communication through visualization, evaluating visualization algorithms, and evaluating collaborative data analysis were derived through an extensive literature review of over 800 visualization publications. These scenarios distinguish different study goals and types of research questions and are illustrated through example studies. Through this broad survey and the distillation of these scenarios, we make two contributions. One, we encapsulate the current practices in the information visualization research community and, two, we provide a different approach to reaching decisions about what might be the most effective evaluation of a given information visualization. Scenarios can be used to choose appropriate research questions and goals and the provided examples can be consulted for guidance on how to design one's own study.

Interaction cost is an important but poorly understood factor in visualization design. We propose a framework of interaction costs inspired by Norman's Seven Stages of Action to facilitate study. From 484 papers, we collected 61 interaction-related usability problems reported in 32 user studies and placed them into our framework of seven costs: (1) Decision costs to form goals; (2) System-power costs to form system operations; (3) Multiple input mode costs to form physical sequences; (4) Physical-motion costs to execute sequences; (5) Visual-cluttering costs to perceive state; (6) View-change costs to interpret perception; (7) State-change costs to evaluate interpretation. We also suggested ways to narrow the gulfs of execution (2-4) and evaluation (5-7) based on collected reports. Our framework suggests a need to consider decision costs (1) as the gulf of goal formation.

In interfaces that provide multiple visual information resolutions (VIR), low-VIR overviews typically sacrifice visual details for display capacity, with the assumption that users can select regions of interest to examine at higher VIRs. Designers can create low-VIRs based on multi-level structure inherent in the data, but have little guidance with single-level data. To better guide design tradeoff between display capacity and visual target perceivability, we looked at overview use in two multiple-VIR interfaces with high-VIR displays either embedded within, or separate from, the overviews. We studied two visual requirements for effective overview and found that participants would reliably use the low-VIR overviews only when the visual targets were simple and had small visual spans. Otherwise, at least 20% chose to use the high-VIR view exclusively. Surprisingly, neither of the multiple-VIR interfaces provided performance benefits when compared to using the high-VIR view alone. However, we did observe benefits in providing side-by-side comparisons for target matching. We conjecture that the high cognitive load of multiple-VIR interface interactions, whether real or perceived, is a more considerable barrier to their effective use than was previously considered.

Scientific measurements are often depicted as line graphs. State-of-the-art high throughput systems in life sciences, telemetry and electronics measurement rapidly generate hundreds to thousands of such graphs. Despite the increasing volume and ubiquity of such data, few software systems provide efficient interactive management, navigation and exploratory analysis of large line graph collections. To address these issues, we have developed Line Graph Explorer (LGE). LGE is a novel and visually scalable line graph management system that supports facile navigation and interactive visual analysis. LGE provides a compact overview of the entire collection by encoding the y-dimension of individual line graphs with color instead of space, thus enabling the analyst to see major common features and alignments of the data. Using Focus+Context techniques, LGE provides interactions for viewing selected compressed graphs in detail as standard line graphs without losing a sense of the general pattern and major features of the collection. To further enhance visualization and pattern discovery, LGE provides sorting and clustering of line graphs based on similarity of selected graph features. Sequential sorting by associated line graph metadata is also supported. We illustrate the features and use of LGE with examples from meteorology and biology.

Two studies evaluated linear and hierarchy+elision small‐screen display formats for clinical reasoning tasks. A controlled, quantitative study with 28 medically naive participants using a task abstracted from clinical use of laboratory results found that both display formats supported rapid and accurate decision making. Distribution of the search targets significantly affected speed, with decisions in linear format made 13% faster (4.7 sec) when all targets could be viewed on a single screen than when targets required scrolling between several screens and in hierarchical format 15% faster (5.1 sec) when all the targets were confined within one category. Performance was equivalent regardless of the relative order of the target results and data in the laboratory report. In a qualitative study, 7 physicians used the displays to perform a realistic diagnosis. Physicians were comfortable with both display formats, but preference varied with clinical experience. The 5 less experienced clinicians favored hierarchy+elision, whereas the 2 highly experienced clinicians tended to prefer the linear display.

In order to display web pages designed for desktop-sized monitors, some small-screen web browsers provide single-column or thumbnail views. Both have limitations. Single-column views affect page layouts and require users to scroll significantly more. Thumbnail views tend to reduce contained text beyond readability, so differentiating visually similar areas requires users to zoom. In this paper, we present Summary Thumbnails-thumbnail views enhanced with readable text fragments. Summary Thumbnails help users identify viewed material and distinguish between visually similar areas. In our user study, participants located content in web pages about 41% faster and with 71% lower error rates when using the Summary Thumbnail interface than when using the Single-Column interface, and zoomed 59% less than when using the Thumbnail interface. Nine of the eleven participants preferred Summary Thumbnails over both the Thumbnail and Single-Column interfaces.