In this work we study the identification of spatial correlation in distributions of 2D scalar fields, presented across different forms of visual displays. We study simple visual displays that directly show color-mapped scalar fields, namely those drawn from a distribution, and whether humans can identify strongly correlated spatial regions in these displays. In this setting, the recognition of correlation requires making judgments on a set of fields, rather than just one field. Thus, in our experimental design we compare two basic visualization designs: animation-based displays against juxtaposed views of scalar fields, along different choices of color scales. Moreover, we investigate the impacts of the distribution itself, controlling for the level of spatial correlation and discriminability in spatial scales. Our study's results illustrate the impacts of these distribution characteristics, while also highlighting how different visual displays impact the types of judgments made in assessing spatial correlation. Supplemental material is available at https://osf.io/zn4qy/.

Generative models are becoming a transformative technology for the creation and editing of images. However, it remains challenging to harness these models for precise image manipulation. These challenges often manifest as inconsistency in the editing process, where both the type and amount of semantic change, depend on the image being manipulated. Moreover, there exist many methods for computing image manipulations, whose development is hindered by the matter of inconsistency. This paper aims to address these challenges by improving how we evaluate, compare, and explore the space of manipulations offered by a generative model. We present Concept Lens, a visual interface that is designed to aid users in understanding semantic concepts carried in image manipulations, and how these manipulations vary over generated images. Given the large space of possible images produced by a generative model, Concept Lens is designed to support the exploration of both generated images, and their manipulations, at multiple levels of detail. To this end, the layout of Concept Lens is informed by two hierarchies: a hierarchical organization of (1) original images, grouped by their similarities, and (2) image manipulations, where manipulations that induce similar changes are grouped together. This layout allows one to discover the types of images that consistently respond to a group of manipulations, and vice versa, manipulations that consistently respond to a group of codes. We show the benefits of this design across multiple use cases, specifically, studying the quality of manipulations for a single method, and offering a means of comparing different methods.

Transfer function design is crucial in volume rendering, as it directly influences the visual representation and interpretation of volumetric data. However, creating effective transfer functions that align with users’ visual objectives is often challenging due to the complex parameter space and the semantic gap between transfer function values and features of interest within the volume. In this work, we propose a novel approach that leverages recent advancements in language-vision models to bridge this semantic gap. By employing a fully differentiable rendering pipeline and an image-based loss function guided by language descriptions, our method generates transfer functions that yield volume-rendered images closely matching the user’s intent. We demonstrate the effectiveness of our approach in creating meaningful transfer functions from simple descriptions, empowering users to intuitively express their desired visual outcomes with minimal effort. This advancement streamlines the transfer function design process and makes volume rendering more accessible to a wider range of users.

Abstract We present concept‐aligned neurons, or CAN, a visualization design for comparing deep neural networks. The goal of CAN is to support users in understanding the similarities and differences between neural networks, with an emphasis on comparing neuron functionality across different models. To make this comparison intuitive, CAN uses concept‐based representations of neurons to visually align models in an interpretable manner. A key feature of CAN is the hierarchical organization of concepts, which permits users to relate sets of neurons at different levels of detail. CAN's visualization is designed to help compare the semantic coverage of neurons, as well as assess the distinctiveness, redundancy, and multi‐semantic alignment of neurons or groups of neurons, all at different concept granularity. We demonstrate the generality and effectiveness of CAN by comparing models trained on different datasets, neural networks with different architectures, and models trained for different objectives, e.g. adversarial robustness, and robustness to out‐of‐distribution data.

Abstract We present CUPID: a visualization method for the contextual understanding of prompt‐conditioned image distributions. CUPID targets the visual analysis of distributions produced by modern text‐to‐image generative models, wherein a user can specify a scene via natural language, and the model generates a set of images, each intended to satisfy the user's description. CUPID is designed to help understand the resulting distribution, using contextual cues to facilitate analysis: objects mentioned in the prompt, novel, synthesized objects not explicitly mentioned, and their potential relationships. Central to CUPID is a novel method for visualizing high‐dimensional distributions, wherein contextualized embeddings of objects, those found within images, are mapped to a low‐dimensional space via density‐based embeddings. We show how such embeddings allows one to discover salient styles of objects within a distribution, as well as identify anomalous, or rare, object styles. Moreover, we introduce conditional density embeddings, whereby conditioning on a given object allows one to compare object dependencies within the distribution. We employ CUPID for analyzing image distributions produced by large‐scale diffusion models, where our experimental results offer insights on language misunderstanding from such models and biases in object composition, while also providing an interface for discovery of typical, or rare, synthesized scenes.

In recent years, considerable work has been devoted to explaining predictive, deep learning-based models, and in turn how to evaluate explanations. An important class of evaluation methods are ones that are human-centered, which typically require the communication of explanations through visualizations. And while visualization plays a critical role in perceiving and understanding model explanations, how visualization design impacts human perception of explanations remains poorly understood. In this work, we study the graphical perception of model explanations, specifically, saliency-based explanations for visual recognition models. We propose an experimental design to investigate how human perception is influenced by visualization design, wherein we study the task of alignment assessment, or whether a saliency map aligns with an object in an image. Our findings show that factors related to visualization design decisions, the type of alignment, and qualities of the saliency map all play important roles in how humans perceive saliency-based visual explanations.

Abstract Generative adversarial networks (GAN) have witnessed tremendous growth in recent years, demonstrating wide applicability in many domains. However, GANs remain notoriously difficult for people to interpret, particularly for modern GANs capable of generating photo‐realistic imagery. In this work we contribute a visual analytics approach for GAN interpretability, where we focus on the analysis and visualization of GAN disentanglement. Disentanglement is concerned with the ability to control content produced by a GAN along a small number of distinct, yet semantic, factors of variation. The goal of our approach is to shed insight on GAN disentanglement, above and beyond coarse summaries, instead permitting a deeper analysis of the data distribution modeled by a GAN. Our visualization allows one to assess a single factor of variation in terms of groupings and trends in the data distribution, where our analysis seeks to relate the learned representation space of GANs with attribute‐based semantic scoring of images produced by GANs. Through use‐cases, we show that our visualization is effective in assessing disentanglement, allowing one to quickly recognize a factor of variation and its overall quality. In addition, we show how our approach can highlight potential dataset biases learned by GANs.

Abstract In this paper we present a reconstruction technique for the reduction of unsteady flow data based on neural representations of time‐varying vector fields. Our approach is motivated by the large amount of data typically generated in numerical simulations, and in turn the types of data that domain scientists can generate in situ that are compact, yet useful, for post hoc analysis. One type of data commonly acquired during simulation are samples of the flow map, where a single sample is the result of integrating the underlying vector field for a specified time duration. In our work, we treat a collection of flow map samples for a single dataset as a meaningful, compact, and yet incomplete, representation of unsteady flow, and our central objective is to find a representation that enables us to best recover arbitrary flow map samples. To this end, we introduce a technique for learning implicit neural representations of time‐varying vector fields that are specifically optimized to reproduce flow map samples sparsely covering the spatiotemporal domain of the data. We show that, despite aggressive data reduction, our optimization problem — learning a function‐space neural network to reproduce flow map samples under a fixed integration scheme — leads to representations that demonstrate strong generalization, both in the field itself, and using the field to approximate the flow map. Through quantitative and qualitative analysis across different datasets we show that our approach is an improvement across a variety of data reduction methods, and across a variety of measures ranging from improved vector fields, flow maps, and features derived from the flow map.

Conventional delay-and-sum (DAS) beamforming is highly efficient but also suffers from various sources of image degradation. Several adaptive beamformers have been proposed to address this problem, including more recently proposed deep learning methods. With deep learning, adaptive beamforming is typically framed as a regression problem, where clean ground-truth physical information is used for training. Because it is difficult to know ground truth information in vivo, training data are usually simulated. However, deep networks trained on simulations can produce suboptimal in vivo image quality because of a domain shift between simulated and in vivo data. In this work, we propose a novel domain adaptation (DA) scheme to correct for domain shift by incorporating unlabeled in vivo data during training. Unlike classification tasks for which both input domains map to the same target domain, a challenge in our regression-based beamforming scenario is that domain shift exists in both the input and target data. To solve this problem, we leverage cycle-consistent generative adversarial networks to map between simulated and in vivo data in both the input and ground truth target domains. Additionally, to account for separate as well as shared features between simulations and in vivo data, we use augmented feature mapping to train domain-specific beamformers. Using various types of training data, we explore the limitations and underlying functionality of the proposed DA approach. Additionally, we compare our proposed approach to several other adaptive beamformers. Using the DA DNN beamformer, consistent in vivo image quality improvements are achieved compared to established techniques.

Visual exploration of large multi-dimensional datasets has seen tremendous progress in recent years, allowing users to express rich data queries that produce informative visual summaries, all in real time. Techniques based on data cubes are some of the most promising approaches. However, these techniques usually require a large memory footprint for large datasets. To tackle this problem, we present NeuralCubes: neural networks that predict results for aggregate queries, similar to data cubes. NeuralCubes learns a function that takes as input a given query, for instance, a geographic region and temporal interval, and outputs the result of the query. The learned function serves as a real-time, low-memory approximator for aggregation queries. Our models are small enough to be sent to the client side (e.g. the web browser for a web-based application) for evaluation, enabling data exploration of large datasets without database/network connection. We demonstrate the effectiveness of NeuralCubes through extensive experiments on a variety of datasets and discuss how NeuralCubes opens up opportunities for new types of visualization and interaction.

Abstract We present an approach for compressing volumetric scalar fields using implicit neural representations. Our approach represents a scalar field as a learned function, wherein a neural network maps a point in the domain to an output scalar value. By setting the number of weights of the neural network to be smaller than the input size, we achieve compressed representations of scalar fields, thus framing compression as a type of function approximation. Combined with carefully quantizing network weights, we show that this approach yields highly compact representations that outperform state‐of‐the‐art volume compression approaches. The conceptual simplicity of our approach enables a number of benefits, such as support for time‐varying scalar fields, optimizing to preserve spatial gradients, and random‐access field evaluation. We study the impact of network design choices on compression performance, highlighting how simple network architectures are effective for a broad range of volumes.

Advances in language modeling have led to the development of deep attention-based models that are performant across a wide variety of natural language processing (NLP) problems. These language models are typified by a pre-training process on large unlabeled text corpora and subsequently fine-tuned for specific tasks. Although considerable work has been devoted to understanding the attention mechanisms of pre-trained models, it is less understood how a model's attention mechanisms change when trained for a target NLP task. In this paper, we propose a visual analytics approach to understanding fine-tuning in attention-based language models. Our visualization, Attention Flows, is designed to support users in querying, tracing, and comparing attention within layers, across layers, and amongst attention heads in Transformer-based language models. To help users gain insight on how a classification decision is made, our design is centered on depicting classification-based attention at the deepest layer and how attention from prior layers flows throughout words in the input. Attention Flows supports the analysis of a single model, as well as the visual comparison between pre-trained and fine-tuned models via their similarities and differences. We use Attention Flows to study attention mechanisms in various sentence understanding tasks and highlight how attention evolves to address the nuances of solving these tasks.

We introduce a new technique to visualize complex flowing phenomena by using concepts from shape analysis. Our approach uses techniques that examine the intrinsic geometry of manifolds through their heat kernel, to obtain representations of such manifolds that are isometry-invariant and multi-scale. These representations permit us to compute heat kernel signatures of each point on that manifold, and we can use these signatures as features for classification and segmentation that identify points that have similar structural properties. Our approach adapts heat kernel signatures to unsteady flows by formulating a notion of shape where pathlines are observations of a manifold living in a high-dimensional space. We use this space to compute and visualize heat kernel signatures associated with each pathline. Besides being able to capture the structural features of a pathline, heat kernel signatures allow the comparison of pathlines from different flow datasets through a shape matching pipeline. We demonstrate the analytic power of heat kernel signatures by comparing both (1) different timesteps from the same unsteady flow as well as (2) flow datasets taken from ensemble simulations with varying simulation parameters. Our analysis only requires the pathlines themselves, and thus it does not utilize the underlying vector field directly. We make minimal assumptions on the pathlines: while we assume they are sampled from a continuous, unsteady flow, our computations can tolerate pathlines that have varying density and potential unknown boundaries. We evaluate our approach through visualizations of a variety of two-dimensional unsteady flows.

We present a technique to synthesize and analyze volume-rendered images using generative models. We use the Generative Adversarial Network (GAN) framework to compute a model from a large collection of volume renderings, conditioned on (1) viewpoint and (2) transfer functions for opacity and color. Our approach facilitates tasks for volume analysis that are challenging to achieve using existing rendering techniques such as ray casting or texture-based methods. We show how to guide the user in transfer function editing by quantifying expected change in the output image. Additionally, the generative model transforms transfer functions into a view-invariant latent space specifically designed to synthesize volume-rendered images. We use this space directly for rendering, enabling the user to explore the space of volume-rendered images. As our model is independent of the choice of volume rendering process, we show how to analyze volume-rendered images produced by direct and global illumination lighting, for a variety of volume datasets.

We challenge a common assumption in active learning, that a list-based interface populated by informative samples provides for efficient and effective data annotation.We show how a 2D scatterplot populated with diverse and representative samples can yield improved models given the same time budget.We consider this for bootstrapping-based information extraction, in particular named entity classification, where human and machine jointly label data.To enable effective data annotation in a scatterplot, we have developed an embeddingbased bootstrapping model that learns the distributional similarity of entities through the patterns that match them in a large data corpus, while being discriminative with respect to human-labeled and machine-promoted entities.We conducted a user study to assess the effectiveness of these different interfaces, and analyze bootstrapping performance in terms of human labeling accuracy, label quantity, and labeling consensus across multiple users.Our results suggest that supervision acquired from the scatterplot interface, despite being noisier, yields improvements in classification performance compared with the list interface, due to a larger quantity of supervision acquired.

We consider the problem of taking a video that is comprised of multiple periods of repetitive motion, and reordering the frames of the video into a single period, producing a detailed, single cycle video of motion. This problem is challenging, as such videos often contain noise, drift due to camera motion and from cycle to cycle, and irrelevant background motion/occlusions, and these factors can confound the relevant periodic motion we seek in the video. To address these issues in a simple and efficient manner, we introduce a tracking free Eulerian approach for synthesizing a single cycle of motion. Our approach is geometric: we treat each frame as a point in high-dimensional Euclidean space, and analyze the sliding window embedding formed by this sequence of points, which yields samples along a topological loop regardless of the type of periodic motion. We combine tools from topological data analysis and spectral geometric analysis to estimate the phase of each window, and we exploit the sliding window structure to robustly reorder frames. We show quantitative results that highlight the robustness of our technique to camera shake, noise, and occlusions, and qualitative results of single-cycle motion synthesis across a variety of scenarios.

Abstract The area of surface reconstruction has seen substantial progress in the past two decades. The traditional problem addressed by surface reconstruction is to recover the digital representation of a physical shape that has been scanned, where the scanned data contain a wide variety of defects. While much of the earlier work has been focused on reconstructing a piece‐wise smooth representation of the original shape, recent work has taken on more specialized priors to address significantly challenging data imperfections, where the reconstruction can take on different representations—not necessarily the explicit geometry. We survey the field of surface reconstruction, and provide a categorization with respect to priors, data imperfections and reconstruction output. By considering a holistic view of surface reconstruction, we show a detailed characterization of the field, highlight similarities between diverse reconstruction techniques and provide directions for future work in surface reconstruction.

Effectively exploring and browsing document collections is a fundamental problem in visualization. Traditionally, document visualization is based on a data model that represents each document as the set of its comprised words, effectively characterizing what the document is. In this paper we take an alternative perspective: motivated by the manner in which users search documents in the research process, we aim to visualize documents via their usage, or how documents tend to be used. We present a new visualization scheme - cite2vec - that allows the user to dynamically explore and browse documents via how other documents use them, information that we capture through citation contexts in a document collection. Starting from a usage-oriented word-document 2D projection, the user can dynamically steer document projections by prescribing semantic concepts, both in the form of phrase/document compositions and document:phrase analogies, enabling the exploration and comparison of documents by their use. The user interactions are enabled by a joint representation of words and documents in a common high-dimensional embedding space where user-specified concepts correspond to linear operations of word and document vectors. Our case studies, centered around a large document corpus of computer vision research papers, highlight the potential for usage-based document visualization.

Bio-inspired robot swarms encompass a rich space of dynamics and collective behaviors.Given some agent measurements of a swarm at a particular time instance, an important problem is the classification of the swarm behavior.This is challenging in practical scenarios where information from only a small number of agents may be available, resulting in limited agent samples for classification.Another challenge is recognizing emerging behavior: the prediction of swarm behavior prior to convergence of the attracting state.In this paper we address these challenges by modeling a swarm's collective motion as a low-dimensional linear subspace.We illustrate that for both synthetic and real data, these behaviors manifest as low-dimensional subspaces, and that these subspaces are highly discriminative.We also show that these subspaces generalize well to predicting emerging behavior, highlighting that there exists low-dimensional structure in transient agent behavior.In order to learn distinct behavior subspaces, we extend previous work on subspace estimation and identification from missing data to that of compressive measurements, where compressive measurements arise due to agent positions scattered throughout the domain.We demonstrate improvement in performance over prior works with respect to limited agent samples over a wide range of agent models and scenarios.

Learning a kernel matrix from relative comparison human feedback is an important problem with applications in collaborative filtering, object retrieval, and search. For learning a kernel over a large number of objects, existing methods face significant scalability issues inhibiting the application of these methods to settings where a kernel is learned in an online and timely fashion. In this paper we propose a novel framework called Efficient online Relative comparison Kernel LEarning (ERKLE), for efficiently learning the similarity of a large set of objects in an online manner. We learn a kernel from relative comparisons via stochastic gradient descent, one query response at a time, by taking advantage of the sparse and low-rank properties of the gradient to efficiently restrict the kernel to lie in the space of positive semidefinite matrices. In addition, we derive a passive-aggressive online update for minimally satisfying new relative comparisons as to not disrupt the influence of previously obtained comparisons. Experimentally, we demonstrate a considerable improvement in speed while obtaining improved or comparable accuracy compared to current methods in the online learning setting.

Long-term modeling of background motion in videos is an important and challenging problem used in numerous applications such as segmentation and event recognition. A major challenge in modeling the background from point trajectories lies in dealing with the variable length duration of trajectories, which can be due to such factors as trajectories entering and leaving the frame or occlusion from different depth layers. This work proposes an online method for background modeling of dynamic point trajectories via tracking of a linear subspace describing the background motion. To cope with variability in trajectory durations, we cast subspace tracking as an instance of subspace estimation under missing data, using a least-absolute deviations formulation to robustly estimate the background in the presence of arbitrary foreground motion. Relative to previous works, our approach is very fast and scales to arbitrarily long videos as our method processes new frames sequentially as they arrive.

Abstract Structured light scanning is ubiquituous in 3D acquisition. It is capable of capturing high geometric detail at a low cost under a variety of challenging scene conditions. Recent methods have demonstrated robustness in the presence of artifacts due to global illumination, such as inter‐reflections and sub‐surface scattering, as well as imperfections caused by projector defocus. For comparing approaches, however, the quantitative evaluation of structured lighting schemes is hindered by the challenges in obtaining ground truth data, resulting in a poor understanding for these methods across a wide range of shapes, materials, and lighting configurations. In this paper, we present a benchmark to study the performance of structured lighting algorithms in the presence of errors caused due to the above properties of the scene. In order to do this, we construct a synthetic structured lighting scanner that uses advanced physically based rendering techniques to simulate the point cloud acquisition process. We show that, under conditions similar to that of a real scanner, our synthetic scanner replicates the same artifacts found in the output of a real scanner. Using this synthetic scanner, we perform a quantitative evaluation of four different structured lighting techniques – gray‐code patterns, micro‐phase shifting, ensemble codes, and unstructured light scanning. The evaluation, performed on a variety of scenes, demonstrate that no one method is capable of adequately handling all sources of error – each method is appropriate for addressing distinct sources of error.

We present a benchmark for the evaluation and comparison of algorithms which reconstruct a surface from point cloud data. Although a substantial amount of effort has been dedicated to the problem of surface reconstruction, a comprehensive means of evaluating this class of algorithms is noticeably absent. We propose a simple pipeline for measuring surface reconstruction algorithms, consisting of three main phases: surface modeling, sampling, and evaluation. We use implicit surfaces for modeling shapes which are capable of representing details of varying size and sharp features. From these implicit surfaces, we produce point clouds by synthetically generating range scans which resemble realistic scan data produced by an optical triangulation scanner. We validate our synthetic sampling scheme by comparing against scan data produced by a commercial optical laser scanner, where we scan a 3D-printed version of the original surface. Last, we perform evaluation by comparing the output reconstructed surface to a dense uniformly distributed sampling of the implicit surface. We decompose our benchmark into two distinct sets of experiments. The first set of experiments measures reconstruction against point clouds of complex shapes sampled under a wide variety of conditions. Although these experiments are quite useful for comparison, they lack a fine-grain analysis. To complement this, the second set of experiments measures specific properties of surface reconstruction, in terms of sampling characteristics and surface features. Together, these experiments depict a detailed examination of the state of surface reconstruction algorithms.

Abstract We introduce the medial kernel, an association measure which provides for a robust construction of volume‐aware distances defined directly on point clouds. The medial kernel is a similarity measure defined as the likelihood of two points belonging to a common interior medial ball. We use the medial kernel to construct a random walk on the point cloud, where movement in the walk is restricted to regions containing similar medial balls. Our distances are defined as the diffusion distances of this random walk, assigning low distance to points belonging to similar medial regions. These distances allow for a robust means of processing incomplete point clouds, capable of distinguishing nearby yet separate undersampled components, while also associating points which are far in Euclidean distance yet mutually share an interior volume. We leverage these distances for several applications: volumetric part segmentation, the construction of function bases, and reconstruction‐by‐parts – a surface reconstruction method which adheres to the medial kernel.

Abstract We introduce medial diffusion for the matching of undersampled shapes undergoing a nonrigid deformation. We construct a diffusion process with respect to the medial axis of a shape, and use the quantity of heat diffusion as a measure which is both tolerant of missing data and approximately invariant to nonrigid deformations. A notable aspect of our approach is that we do not define the diffusion on the shape's medial axis, or similar medial representation. Instead, we construct the diffusion process directly on the shape. This permits the diffusion process to better capture surface features, such as varying spherical and cylindrical parts, as well as combine with other surface‐based diffusion processes. We show how to use medial diffusion to detect intrinsic symmetries, and for computing correspondences between pairs of shapes, wherein shapes contain substantial missing data.