Visualization tools are typically evaluated in controlled studies that observe the short-term usage of these tools by participants on preselected data sets and benchmark tasks. Though such studies provide useful suggestions, they miss the long-term usage of the tools. A longitudinal study of a bioinformatics data set analysis is reported here. The main focus of this work is to capture the entire analysis process that an analyst goes through from a raw data set to the insights sought from the data. The study provides interesting observations about the use of visual representations and interaction mechanisms provided by the tools, and also about the process of insight generation in general. This deepens our understanding of visual analytics, guides visualization developers in creating more effective visualization tools in terms of user requirements, and guides evaluators in designing future studies that are more representative of insights sought by users from their data sets.

High-throughput experiments, such as gene expression microarrays in the life sciences, result in very large data sets. In response, a wide variety of visualization tools have been created to facilitate data analysis. A primary purpose of these tools is to provide biologically relevant insight into the data. Typically, visualizations are evaluated in controlled studies that measure user performance on predetermined tasks or using heuristics and expert reviews. To evaluate and rank bioinformatics visualizations based on real-world data analysis scenarios, we developed a more relevant evaluation method that focuses on data insight. This paper presents several characteristics of insight that enabled us to recognize and quantify it in open-ended user tests. Using these characteristics, we evaluated five microarray visualization tools on the amount and types of insight they provide and the time it takes to acquire it. The results of the study guide biologists in selecting a visualization tool based on the type of their microarray data, visualization designers on the key role of user interaction techniques, and evaluators on a new approach for evaluating the effectiveness of visualizations for providing insight. Though we used the method to analyze bioinformatics visualizations, it can be applied to other domains.

Many algorithm visualizations have been created, but little is known about which features are most important to their success. We believe that pedagogically useful visualizations exhibit certain features that hold across a wide range of visualization styles and content. We began our efforts to identify these features with a review that attempted to identify an initial set of candidates. We then ran two experiments that attempted to identify the effectiveness for a subset of features from the list. We identified a small number of features for algorithm visualizations that seem to have a significant impact on their pedagogical effectiveness, and found that several others appear to have little impact. The single most important feature studied is the ability to directly control the pace of the visualization. An algorithm visualization having a minimum of distracting features, and which focuses on the logical steps of an algorithm, appears to be best for procedural understanding of the algorithm. Providing a good example for the visualization to operate on proved significantly more effective than letting students construct their own data sets. Finally, a pseudocode display, a series of questions to guide exploration of the algorithm, or the ability to back up within the visualization did not show a significant effect on learning.