OBJECTIVE: This article reports results from a systematic literature review related to the evaluation of data visualizations and visual analytics technologies within the health informatics domain. The review aims to (1) characterize the variety of evaluation methods used within the health informatics community and (2) identify best practices. METHODS: A systematic literature review was conducted following PRISMA guidelines. PubMed searches were conducted in February 2017 using search terms representing key concepts of interest: health care settings, visualization, and evaluation. References were also screened for eligibility. Data were extracted from included studies and analyzed using a PICOS framework: Participants, Interventions, Comparators, Outcomes, and Study Design. RESULTS: After screening, 76 publications met the review criteria. Publications varied across all PICOS dimensions. The most common audience was healthcare providers (n = 43), and the most common data gathering methods were direct observation (n = 30) and surveys (n = 27). About half of the publications focused on static, concentrated views of data with visuals (n = 36). Evaluations were heterogeneous regarding setting and measurements used. DISCUSSION: When evaluating data visualizations and visual analytics technologies, a variety of approaches have been used. Usability measures were used most often in early (prototype) implementations, whereas clinical outcomes were most common in evaluations of operationally-deployed systems. These findings suggest opportunities for both (1) expanding evaluation practices, and (2) innovation with respect to evaluation methods for data visualizations and visual analytics technologies across health settings. CONCLUSION: Evaluation approaches are varied. New studies should adopt commonly reported metrics, context-appropriate study designs, and phased evaluation strategies.

Despite the recent popularity of visual analytics focusing on big data, little is known about how to support users that use visualization techniques to explore multi-dimensional datasets and accomplish specific tasks. Our lack of models that can assist end-users during the data exploration process has made it challenging to learn from the user's interactive and analytical process. The ability to model how a user interacts with a specific visualization technique and what difficulties they face are paramount in supporting individuals with discovering new patterns within their complex datasets. This paper introduces the notion of visualization systems understanding and modeling user interactions with the intent of guiding a user through a task thereby enhancing visual data exploration. The challenges faced and the necessary future steps to take are discussed; and to provide a working example, a grammar-based model is presented that can learn from user interactions, determine the common patterns among a number of subjects using a K-Reversible algorithm, build a set of rules, and apply those rules in the form of suggestions to new users with the goal of guiding them along their visual analytic process. A formal evaluation study with 300 subjects was performed showing that our grammar-based model is effective at capturing the interactive process followed by users and that further research in this area has the potential to positively impact how users interact with a visualization system.

Despite the significant amount of longitudinal information that Electronic Health Records (EHRs) include, modeling the progression of a particular disease is still a challenging task. Generally, when the trajectory of a cohort of patients is analyzed some patients take similar paths from/to a certain clinical disease while others take vastly different paths. Most existing approaches to analyze trajectories of a patient have focused on a specific disease with the intention of being able to predict whether a new patient will develop a similar diagnosis or not. This paper presents a novel approach to identifying the trajectory of patients by modeling the clinical path as an automata and using grammar induction to reduce the graph. The resulting reduced model represents the common trajectory for the patients without incorporating events that are specific to a particular patient. The proposed model has been tested with 5,000 patients that have sustained a mild traumatic brain injury (mTBI) and have developed post-traumatic stress disorder (PTSD). The model has produced a trajectory indicating the common clinical path that those patients take from their first mTBI to their first diagnosis of PTSD.

As medical organizations modernize their operations, they are increasingly adopting electronic health records (EHRs) and deploying new health information technology systems that create, gather, and manage their information. As a result, the amount of data available to clinicians, administrators, and researchers in the healthcare system continues to grow at an unprecedented rate.1 However, despite the substantial evidence showing the benefits of EHR adoption, e-prescriptions, and other components of health information exchanges, healthcare providers often report only modest improvements in their ability to make better decisions by using more comprehensive clinical information.2,3 The large volume of clinical data now being captured for each patient poses many challenges to (a) clinicians trying to combine data from different disparate systems and make sense of the patient’s condition within the context of the patient’s medical history, (b) administrators trying to make decisions grounded in data, (c) researchers trying to understand differences in population outcomes, and (d) patients trying to make use of their own medical data. In fact, despite the many hopes that access to more information would lead to more informed decisions, access to comprehensive and large-scale clinical data resources has instead made some analytical processes even more difficult.4 Visual analytics is an emerging discipline that has shown significant promise in addressing many of these information overload challenges. Visual analytics is the science of analytical reasoning facilitated by advanced interactive visual interfaces.5 In order to facilitate reasoning over, and interpretation of, complex data, visual analytics techniques combine concepts from data mining, machine learning, human computing interaction, and human cognition. As the volume of health-related data continues to grow at unprecedented rates and new information systems are deployed to those already overrun with too much data, there is a need for exploring how visual analytics methods can be used to avoid information overload. Information overload is the problem that arises when individuals try to analyze a number of variables that surpass the limits of human cognition.6 Information overload often leads to users ignoring, overlooking, or misinterpreting crucial information. The information overload problem is widespread in the healthcare domain and can result in incorrect interpretations of data, wrong diagnoses, and missed warning signs of impending changes to patient conditions. The multi-modal and heterogeneous properties of EHR data together with the frequency of redundant, irrelevant, and subjective measures pose significant challenges to users trying to synthesize the information and obtain actionable insights. Yet despite these challenges, the promise of big data in healthcare remains.1 There is a critical need to support research and pilot projects to study effective ways of using visual analytics to support the analysis of large amounts of medical data. Currently new interactive interfaces are being developed to unlock the value of large-scale clinical databases for a wide variety of different tasks. For instance, visual analytics could help provide clinicians with more effective ways to combine the longitudinal clinical data with the patient-generated health data to better understand patient progression. Patients could be supported in understanding personalized wellness plans and comparing their health measurements against similar patients. Researchers could use visual analytics tools to help perform population-based analysis and obtain insights from large amounts of clinical data. Hospital administrators could use visual analytics to better understand the productivity of an organization, gaps in care, outcomes measurements, and patient satisfaction. Visual analytics systems—by combining advanced interactive visualization methods with statistical inference and correlation models—have the potential to support intuitive analysis for all of these user populations while masking the underlying complexity of the data. This special focus issue of JAMIA is dedicated to new research, applications, case studies, and approaches that use visual analytics to support the analysis of complex clinical data. The issue provides a broad picture of active work in this area, including four research and applications papers, one review paper, two brief communications, and one case report. First, West et al.7 present a systematic review of innovative visual analytics approaches that have been proposed to illustrate EHR data. The review paper groups over 800 articles in different categories and is a great resource for readers who are interested in understanding what has been done in applying visual analytics in healthcare settings. In the first of four research and application papers, Klimov et al.8 present a data-driven temporal data mining system with an interactive visual analytics interface to support the analysis and exploration of renal-damage risk factors in type II diabetes patients. Second, Huang et al. present a system to visually analyze the polymorbidity associated with chronic kidney disease. The study uses a large-scale database with over 14 000 patients and a Sankey-style visualization tool to help clinicians predict the outcome of complex disease based on comorbidities the patients have developed.9 Third, Hirsch et al.10 present a system called HARVEST, an interactive temporal visualization for longitudinal patient record. The system implements a patient record summarizer by extracting content from patient notes, aggregating and presenting information from multiple care settings, and supporting clinicians by providing a flexible tool for reviewing patients’ information. Finally, Soulakis et al.11 discuss practical methods for employing network analysis to visualize and describe multidisciplinary, collaborative care for hospitalized heart failure patients. The authors used data for over 500 patients and describe the provider’s collaboration network as a subset of collaborations within a complex graph with 1504 nodes and 83 998 edges. In a comprehensive case study, Warner et al.12 present a network visualization-based system to explore, analyze, and display the phenotypic patterns that may have remained occult or hidden in basic statistical and pairwise correlation analysis. The system allows clinicians and researchers to quickly generate hypotheses and gain deeper understanding of subpopulations. Finally, two brief communications describe early results from a pair of visual analysis systems. First, Basole et al.13 present an interactive system that uses visual analytics techniques to explore clinical data from 5784 pediatric asthma emergency department patients. The system makes cohort identification more efficient and possibly even more accurate that regular approaches. Second, Ratwani et al.14 present a dashboard-based system that employs visualization techniques to more effectively explore patient safety event reports, find patterns, and analyze the correlation between different hospital organizations. This special issue provides readers with a sample of the variety of work being performed around the world on the topic of visual analytics in healthcare. Researchers in this discipline are exploring a range of topics and a growing, vibrant research community has emerged.15 Many challenges remain, but we believe that visual analytics techniques will become essential elements in the future of health informatics. As the articles in this issue demonstrate, these methods promise to allow users of all sorts—clinicians, researchers, administrators, patients, and more—to derive actionable, meaningful insights form the vast and complex data resources that the modern health system is now creating. Yet this issue also indirectly highlights some of the areas of informatics that need more attention. These include the visual exploration of unstructured data (e.g., text), the challenge of clinical or operational adoption, and the lack of standardized methods for evaluation, validation, and measurement of efficacy of visual analytics tools. These are all critical areas for future work.

A concussion is a poorly understood mild traumatic brain injury (mTBI) that alters the way the brain functions. Clinical practice guidelines suggest different algorithms that should be followed to evaluate TBI patients [11]. At different steps, the clinical guidelines ask for physical, cognitive, behavioral, imaging, and neuropsychological evaluations which result in dozens of measures that should be analyzed to better establish an understanding of the patient's condition. The complexity, multimodal, and subjective properties of many of these data combined with the overlapping symptoms of comorbid issues raise many challenges for physicians who must integrate these disparate measurements to develop a comprehensive understanding of the patient's condition and for researchers trying to understand the effects of concussions. The complexity of determining the long-term effects of concussions as well as the data challenges faced by clinicians when trying to diagnose mTBI shows that research on the effects of concussion is a big data problem that only when large, comprehensive, and standardized clinical information are analyzed simultaneously, is when new knowledge can be obtained. This paper presents a large-scale informatics database that has been designed to enable research in the understanding of the effects of concussions. The database consists of millions of longitudinal clinical data points ranging from encounters, type of encounters, clinical notes, diagnosis codes, imaging findings, and other clinical information.

Constructing an image classification system using strong, local invariant descriptors is both time consuming and tedious, requiring many experimentations and parameter tunings to obtain an adequately performing model. Furthermore training a system in a given domain and then migrating the model to a separate domain will likely yield poor performance. As the recent Boston Marathon attacks demonstrated, large, unstructured image databases from traffic cameras, security systems, law enforcement officials, and citizens can be quickly amassed for authorities to review, however, reviewing each and every image is a expensive undertaking, in terms of both time and human intervention. Inherently, reviewing crime scene images is a classification task. For example, authorities may want to know if a given image contains a suspect, a suspicious package, or if there are injured people in the photo. Given an emergency situation, these classifications will be needed as quickly and accurately as possible. In this work we present a rapidly deployable image classification system using "feature-views", which each view consists of a set of weak, global features. These weak global descriptors are computationally simple to extract, intuitive to understand, and require substantially less parameter tuning than their local invariant counterparts. We demonstrate that by combining weak features with ensemble methods we are able to outperform current state-of-the-art methods or achieve comparable accuracy with much less effort and domain knowledge. Finally we provide both theoretical and empirical justification for our ensemble framework that can be used to construct rapidly deployable image classification systems called "Ecosembles".

Medication errors are one of the safety problems most frequently seen in hospital organizations. It is estimated that 12.2% of all hospitalized patients are involved in some form of adverse drug event (ADE) [1]. A significant amount of ADEs result from handing the incorrect drug to a patient or prescribing the wrong medication. This paper introduces a simple yet robust classification technique that can be used to automatically identify prescriptions drugs within images. The system uses a modified shape distribution technique to examine the shape, color, and imprint of a pill and create an invariant descriptor that can be used to recognize the same drug under different viewing conditions. The proposed technique has been successfully evaluated with 568 of the most prescribed drugs in the United States and has shown a 91.13% accuracy in automatically identifying the correct medication.

Abstract As collections of 2D/3D images continue to grow, interest in effective ways to visualize and explore the statistical morphological properties of a group of images has surged. Recently, deformation models have emerged as simple methods to capture the variability and statistical properties of a collection of images. Such models have proven to be effective in tasks such as image classification, generation, registration, segmentation, and analysis of modes of variation. A crucial element missing from most statistical models has been an effective way to summarize and visualize the statistical morphological properties of a group of images. This paper evaluates different visualization techniques that can be extended and used to illustrate the information captured by such statistical models. First, four illustration techniques are described as methods to summarize the statistical morphological properties as captured by deformation models. Second, results of a user study conducted to compare the effectiveness of each visualization technique are presented. After comparing the performance of 40 subjects, we found that statistical annotation techniques present significant benefits when analyzing the structural properties of a group of images.

The devastating power of hurricanes was evident during the 2005 hurricane season, the most active season on record. This has prompted increased efforts by researchers to understand the physical processes that underlie the genesis, intensification, and tracks of hurricanes. This research aims at facilitating an improved understanding into the structure of hurricanes with the aid of visualization techniques. Our approach was developed by a mixed team of visualization and domain experts. To better understand these systems, and to explore their representation in NWP models, we use a variety of illustration-inspired techniques to visualize their structure and time evolution. Illustration-inspired techniques aid in the identification of the amount of vertical wind shear in a hurricane, which can help meteorologists predict dissipation. Illustration-style visualization, in combination with standard visualization techniques, helped explore the vortex rollup phenomena and the mesovortices contained within. We evaluated the effectiveness of our visualization with the help of six hurricane experts. The expert evaluation showed that the illustration-inspired techniques were preferred over existing tools. Visualization of the evolution of structural features is a prelude to a deeper visual analysis of the underlying dynamics.

Visualization of volumetric data faces the difficult task of finding effective parameters for the transfer functions. Those parameters can determine the effectiveness and accuracy of the visualization. Frequently, volumetric data includes multiple structures and features that need to be differentiated. However, if those features have the same intensity and gradient values, existing transfer functions are limited at effectively illustrating those similar features with different rendering properties. We introduce texture-based transfer functions for direct volume rendering. In our approach, the voxel's resulting opacity and color are based on local textural properties rather than individual intensity values. For example, if the intensity values of the vessels are similar to those on the boundary of the lungs, our texture-based transfer function will analyze the textural properties in those regions and color them differently even though they have the same intensity values in the volume. The use of texture-based transfer functions has several benefits. First, structures and features with the same intensity and gradient values can be automatically visualized with different rendering properties. Second, segmentation or prior knowledge of the specific features within the volume is not required for classifying these features differently. Third, textural metrics can be combined and/or maximized to capture and better differentiate similar structures. We demonstrate our texture-based transfer function for direct volume rendering with synthetic and real-world medical data to show the strength of our technique.

Analyzing, visualizing, and illustrating changes within time-varying volumetric data is challenging due to the dynamic changes occurring between timesteps. The changes and variations in computational fluid dynamic volumes and atmospheric 3D datasets do not follow any particular transformation. Features within the data move at different speeds and directions making the tracking and visualization of these features a difficult task. We introduce a texture-based feature tracking technique to overcome some of the current limitations found in the illustration and visualization of dynamic changes within time-varying volumetric data. Our texture-based technique tracks various features individually and then uses the tracked objects to better visualize structural changes. We show the effectiveness of our texture-based tracking technique with both synthetic and real world time-varying data. Furthermore, we highlight the specific visualization, annotation, registration, and feature isolation benefits of our technique. For instance, we show how our texture-based tracking can lead to insightful visualizations of time-varying data. Such visualizations, more than traditional visualization techniques, can assist domain scientists to explore and understand dynamic changes.