The life cycle of a peer-produced dataset follows the phases of growth, maturity, and decline. Paying crowdworkers is a proven method to collect and organize information into structured tables. However, these tabular representations may contain inaccuracies due to errors or data changing over time. Thus, the maturation phase of a dataset can benefit from the additional human examination. One method to improve accuracy is to recruit additional paid crowdworkers to verify and correct errors. An alternative method relies on unpaid contributors, collectively editing the dataset during regular use. We describe two case studies to examine different strategies for human verification and maintenance of in-flux tabular datasets. The first case study examines traditional micro-task verification strategies with paid crowdworkers, while the second examines long-term maintenance strategies with unpaid contributions from non-crowdworkers. Two paid verification strategies that produced more accurate corrections at a lower cost per accurate correction were redundant data collection followed by final verification from a trusted crowdworker and allowing crowdworkers to review any data freely. In the unpaid maintenance strategies, contributors provided more accurate corrections when asked to review data matching their interests. This research identifies considerations and future approaches to collectively improving information accuracy and longevity of tabular information.

This work analyzes sensemaking frameworks and experiments with an iteratively designed visual analysis tool to identify design implications for facilitating research idea generation using visualizations. Our tool, ThoughtFlow, structures and visualizes literature collections using topic models to bridge the information gap between core activities during research ideation. To help users stay focused on a topic while discovering relevant documents, we designed and analyzed usage patterns for two types of embedded visualization that help determine document relevance while minimizing distraction. We analyzed how research ideation outcomes and processes differ when using ThoughtFlow and conventional search engines by augmenting insight-based evaluation with concept-map analysis. Our results suggest that operations afforded by topic models match well with later ideation stages when coherent topics have emerged, but not with early stages when users are still relying heavily on individual keywords to gather background knowledge. We also present qualitative evidence that citation sparklines encourage more exploration of recommended references, and that a preference for paper thumbnails may depend on the consistency between the evidence and the current mental frame.

The goal of this study was to assess the relationship of task difficulty and mental effort with performance during physicians’ interaction with electronic health records (EHRs). A total of 38 physicians were asked to identify abnormal results and take follow-up action to “close the loop” on care delivery. Task difficulty was quantified via task-flow strategies and computer mouse-click patterns. Mental effort was quantified using eye movements based on changes in pupillary dilations (task evoked pupillary response or TEPR) and blink rate. Performance was quantified based on commission errors (error vs. no-error). Results indicated that physicians had different task-flow strategies; however, with improved awareness of the patient status, they exhibited an optimal task-flow strategy. Overall, performance was related to task-flow strategies, computer mouse-click patterns, and blink rate, indicating that physicians had lower task-difficulty and experienced lower mental effort with improved awareness of patient follow-up status. This is an important finding demonstrating that task-flows are a dominant predictor of physician performance when comparing between EHR designs. On the contrary, mouse-click patterns and blink rate are both useful predictors of physician performance when assessment was done within an EHR.

Objective was to assess the relationship between task demand, mental effort, task difficulty, and performance during physicians’ interaction with electronic health records (EHRs). Seventeen physicians performed three EHR-based scenarios with varying task demands. Mental effort was measured using eye tracking measures via task evoked pupillary responses (TEPR), blink frequency, and gaze speed; task difficulty (or user behavior) was measured using frequent mouse click patterns and task flow; user performance was quantified using two types of omission errors: (i) omission errors with no evidence of trying to complete the task and (ii) omission error with evidence of trying but unable to complete the task. The results indicated that task demand significantly increased mental effort, but not task difficulty. Task demand, mental effort, and task difficulty all predicted performance. Specifically, there was a significant relationship between (i) task demand, TEPR and omission errors with no evidence of trying to complete the task, and (ii) blink frequency, repeated search clicks and omission error with evidence of trying but unable to complete the task. In concert, results suggest that physicians’ performance during EHR interaction was negatively affected by task demands and increase in mental effort. This highlight the need for implementation of appropriate quality assurance (QA) measures, in addition to EHR usability improvement, to minimize omission errors and improve physician’s performance. Additionally, the lack of relationship between task demand and task difficulty highlights a need for further methodological and empirical studies to advance our understanding from theory to application during physician–EHR interaction.

We present the design and evaluation of a method for estimating gaze locations during the analysis of static visualizations using crowdsourcing. Understanding gaze patterns is helpful for evaluating visualizations and user behaviors, but traditional eye-tracking studies require specialized hardware and local users. To avoid these constraints, we developed a method called Fauxvea, which crowdsources visualization tasks on the Web and estimates gaze fixations through cursor interactions without eye-tracking hardware. We ran experiments to evaluate how gaze estimates from our method compare with eye-tracking data. First, we evaluated crowdsourced estimates for three common types of information visualizations and basic visualization tasks using Amazon Mechanical Turk (MTurk). In another, we reproduced findings from a previous eye-tracking study on tree layouts using our method on MTurk. Results from these experiments show that fixation estimates using Fauxvea are qualitatively and quantitatively similar to eye tracking on the same stimulus-task pairs. These findings suggest that crowdsourcing visual analysis tasks with static information visualizations could be a viable alternative to traditional eye-tracking studies for visualization research and design.

We present results from an experiment aimed at using logs of interactions with a visual analytics application to better understand how interactions lead to insight generation. We performed an insight-based user study of a visual analytics application and ran post hoc quantitative analyses of participants' measured insight metrics and interaction logs. The quantitative analyses identified features of interaction that were correlated with insight characteristics, and we confirmed these findings using a qualitative analysis of video captured during the user study. Results of the experiment include design guidelines for the visual analytics application aimed at supporting insight generation. Furthermore, we demonstrated an analysis method using interaction logs that identified which interaction patterns led to insights, going beyond insight-based evaluations that only quantify insight characteristics. We also discuss choices and pitfalls encountered when applying this analysis method, such as the benefits and costs of applying an abstraction framework to application-specific actions before further analysis. Our method can be applied to evaluations of other visualization tools to inform the design of insight-promoting interactions and to better understand analyst behaviors.

As crowdsourcing has gained prominence in recent years, an increasing number of people turn to popular crowdsourcing platforms for their many uses. Experienced members of the crowdsourcing community have developed numerous systems both separately and in conjunction with these platforms, along with other tools and design techniques, to gain more specialized functionality and overcome various shortcomings. It is unclear, however, how novice requesters using crowdsourcing platforms for general tasks experience existing platforms and how, if at all, their approaches deviate from the best practices established by the crowdsourcing research community. We conduct an experiment with a class of 19 students to study how novice requesters design crowdsourcing tasks. Each student tried their hand at crowdsourcing a real data collection task with a fixed budget and realistic time constraint. Students used Amazon Mechanical Turk to gather information about the academic careers of over 2,000 professors from 50 top Computer Science departments in the U.S. In addition to curating this dataset, we classify the strategies which emerged, discuss design choices students made on task dimensions, and compare these novice strategies to best practices identified in crowdsourcing literature. Finally, we summarize design pitfalls and effective strategies observed to provide guidelines for novice requesters.

When visualizing data with uncertainty, a common approach is to treat uncertainty as an additional dimension and encode it using a visual variable. The effectiveness of this approach depends on how the visual variables chosen for representing uncertainty and other attributes interact to influence the user's perception of each variable. We report a user study on the perception of graph edge attributes when uncertainty associated with each edge and the main edge attribute are visualized simultaneously using two separate visual variables. The study covers four visual variables that are commonly used for visualizing uncertainty on line graphical primitives: lightness, grain, fuzziness, and transparency. We select width, hue, and saturation for visualizing the main edge attribute and hypothesize that we can observe interference between the visual variable chosen to encode the main edge attribute and that to encode uncertainty, as suggested by the concept of dimensional integrality. Grouping the seven visual variables as color-based, focus-based, or geometry-based, we further hypothesize that the degree of interference is affected by the groups to which the two visual variables belong. We consider two further factors in the study: discriminability level for each visual variable as a factor intrinsic to the visual variables and graph-task type (visual search versus comparison) as a factor extrinsic to the visual variables. Our results show that the effectiveness of a visual variable in depicting uncertainty is strongly mediated by all the factors examined here. Focus-based visual variables (fuzziness, grain, and transparency) are robust to the choice of visual variables for encoding the main edge attribute, though fuzziness has stronger negative impact on the perception of width and transparency has stronger negative impact on the perception of hue than the other uncertainty visual variables. We found that interference between hue and lightness is much greater than that between saturation and lightness, though all three are color-based visual variables. We also found a compound relationship between discriminability level and the degree of dimensional integrality. We discuss the generalizability and limitation of the results and conclude with design considerations for visualizing graph uncertainty derived from these results, including recommended choices of visual variables when the relative importance of data attributes and graph tasks is known.

We present an ethnographic study of design differences in visual presentations between academic disciplines. Characterizing design conventions between users and data domains is an important step in developing hypotheses, tools, and design guidelines for information visualization. In this paper, disciplines are compared at a coarse scale between four groups of fields: social, natural, and formal sciences; and the humanities. Two commonplace presentation types were analyzed: electronic slideshows and whiteboard "chalk talks". We found design differences in slideshows using two methods - coding and comparing manually-selected features, like charts and diagrams, and an image-based analysis using PCA called eigenslides. In whiteboard talks with controlled topics, we observed design behaviors, including using representations and formalisms from a participant's own discipline, that suggest authors might benefit from novel assistive tools for designing presentations. Based on these findings, we discuss opportunities for visualization ethnography and human-centered authoring tools for visual information.