Visual anomaly detection is advantageous for typical industrial applications due to its minimal training data requirements. However, most visual anomaly detection methods assume a fixed background. This assumption leads to performance degradation on varied backgrounds. One primary cause is the misclassification of untrained backgrounds as anomalies. To mitigate the influence of varied backgrounds, we propose Adaptive Activated Anomaly Detection (AAAD), which distinguishes between the inspection target and backgrounds. AAAD incorporates segmentation with contrastive learning to emphasize subtle differences between the inspection target and backgrounds. To minimize the load of annotation for the additional segmentation task, weakly few-shot labeled images are used. Evaluation on an industrial dataset with varied backgrounds demonstrates that the pixel-level AUROC of AAAD significantly surpass those of conventional methods.

To explain how a convolutional neural network (CNN) processes an image, it is common to generate classification activation maps (CAMs) to reveal image areas that are relevant to output. Nevertheless, it is also known that existing methods could generate unstable visualization, especially for intermediate layers inside CNNs. In this paper, we present a new CAM-generation algorithm for these layers. We first revisit Taylor expansion to estimate how much each input element in an intermediate layer locally contributes to the function, and derive an iterative algorithm to accumulate the global contribution to the output. Because our iterative algorithm essentially integrates the contribution, we call our algorithm Integral-based Classification Activation Maps (IntegralCAM). By combining our contribution estimation and element-wise weighting, experiments show that we could generate stable CAMs for intermediate CNN layers.

Given images of a person, person re- identification (Person ReID) techniques aim to find images of the same person from previously collected images. Because of large data sets of person images and the advance of deep learning, convolutional neural networks (CNNs) successfully boost the accuracy of Person ReID algorithms, but it can be difficult to explain and to troubleshoot issues due to the complexity of CNNs. In this paper, we present a visualization-based approach to understand a CNN-based Person ReID algorithm. As Person ReID algorithms are often designed to map images of the same person into similar feature vectors, given two images, we design an algorithm to estimate how much each element in a CNN layer contributes to the similarity between their feature vectors. Based on the estimation, we build a visualization tool to interactively locate and visualize the activation of highly-contributing elements, other than manually examining all. Our visualization tool also supports various user interaction widgets to explore a Person ReID data set, locate difficult cases, and analyze the reason behind their similarities. We show a use case with our tool to understand and troubleshoot issues in a CNN-based Person ReID algorithm.

While convolution neural networks (CNNs) successfully boost the accuracy of object detection algorithms, it is still challenging to detect far-away objects since they can be tiny in an image. To enhance CNNs’ ability for tiny object detection, this paper presents an algorithm called Saliency-Guided Feature Module (SGFM). Based on low-level image features, we compute the saliency over the image to indicate where foreground objects could be, and use SGFM to enhance the CNN’s feature maps that focus on similar areas. We also present a new dataset named Camera Monitoring System Driving Dataset (CMS-DD) where images were captured from the view angle of side mirrors on driving vehicles so far-away objects can look even tinier. Our experiments show that SGFMs can further improve a recent state-of-the-art object detector in practical driving scenes like CMS-DD.

Deep reinforcement learning (RL), where a policy represented by a deep neural network is trained, has shown some success in playing video games and chess. However, applying RL to real-world tasks like robot control is still challenging. Because generating a massive number of samples to train control policies using RL on real robots is very expensive, hence impractical, it is common to train in simulations, and then transfer to real environments. The trained policy, however, may fail in the real world because of the difference between the training and the real environments, especially the difference in dynamics. To diagnose the problems, it is crucial for experts to understand (1) how the trained policy behaves under different dynamics settings, (2) which part of the policy affects the behaviors the most when the dynamics setting changes, and (3) how to adjust the training procedure to make the policy robust.This paper presents DynamicsExplorer, a visual analytics tool to diagnose the trained policy on robot control tasks under different dynamics settings. DynamicsExplorer allows experts to overview the results of multiple tests with different dynamics-related parameter settings so experts can visually detect failures and analyze the sensitivity of different parameters. Experts can further examine the internal activations of the policy for selected tests and compare the activations between success and failure tests. Such comparisons help experts form hypotheses about the policy and allows them to verify the hypotheses via DynamicsExplorer. Multiple use cases are presented to demonstrate the utility of DynamicsExplorer.

We propose a self-supervised approach to improve the training of Generative Adversarial Networks (GANs) via inducing the discriminator to examine the structural consistency of images. Although natural image samples provide ideal examples of both valid structure and valid texture, learning to reproduce both together remains an open challenge. In our approach, we augment the training set of natural images with modified examples that have degraded structural consistency. These degraded examples are automatically created by randomly exchanging pairs of patches in an image’s convolutional feature map. We call this approach feature exchange. With this setup, we propose a novel GAN formulation, termed Feature eXchange GAN (FX-GAN), in which the discriminator is trained not only to distinguish real versus generated images, but also to perform the auxiliary task of distinguishing between real images and structurally corrupted (feature-exchanged) real images. This auxiliary task causes the discriminator to learn the proper feature structure of natural images, which in turn guides the generator to produce images with more realistic structure. Compared with strong GAN baselines, our proposed self-supervision approach improves generated image quality, diversity, and training stability for both the unconditional and class-conditional settings.

Development of object detectors for video in applications such as autonomous driving requires an iterative training process with data that initially requires human labeling. Later stages of development require tuning a large set of parameters that may not have labeled data available. For each training iteration and parameter selection decision, insight is needed into object detector performance. This work presents a visualization method called Space-Time Slicing to assist a human developer in the development of object detectors for driving applications without requiring labeled data. Space-Time Slicing reveals patterns in the detection data that can suggest the presence of false positives and false negatives. It may be used to set such parameters as image pixel size in data preprocessing and confidence thresholds for object classifiers by comparing performance across different conditions.

The Amazon Picking Challenge and the Amazon Robotics Challenge have shown significant progress in object picking from a cluttered scene, yet object placement remains challenging. It is useful to have pose-aware placement based on human and machine readable pieces on an object. For example, the brandname of an object placed on a shelf should be facing the human customers. The robotic vision challenges in the object placement task: a) the semantics and geometry of the object to be placed need to be analysed jointly; b) and the occlusions among objects in a cluttered scene could make it hard for proper understanding and manipulation. To overcome these challenges, we develop a pose-aware placement approach by spotting the semantic labels (e.g., brandnames) of objects in a cluttered tote and then carrying out a sequence of actions to place the objects on a shelf or on a conveyor with desired poses. Our major contributions include 1) providing an open benchmark dataset of objects and brandnames with multi-view segmentation for training and evaluations; 2) carrying out comprehensive evaluations for our brandname-based fully convolutional network (FCN) that can predict the affordance and grasp to achieve pose-aware placement, whose success rates decrease along with clutters; 3) showing that active manipulation with two cooperative manipulators and grippers can effectively handle the occlusion of brandnames. We analyzed the success rates and discussed the failure cases to provide insights for future applications. All data and benchmarks are available at https://text-pick-n-place.github.io/TextPNP/.

A well-known limitation of classic continuous treemaps is that they generally provide two (or at most a few) visual mappings for data variables apart from the hierarchical relationships. Typically, one variable maps to cell area; another maps to color. However, many data-centric tasks require human users to consider multiple variables simultaneously. The current work introduces the concept of equal-height, variable-width cells in treemaps, which affords the packing of multiple variables into the cell areas of the terminals of the hierarchy. We demonstrate how color and some largely width-invariant graphs can be utilized in the cell areas to add additional visual information in a multi-variate treemap. Examples come from machine learning and from finance applications.

In this paper we propose a novel network module, namely Robust Attentional Pooling (RAP), that potentially can be applied in an arbitrary network for generating single vector representations for classification. By taking a feature matrix for each data sample as the input, our RAP learns data-dependent weights that are used to generate a vector through linear transformations of the feature matrix. We utilize feature selection to control the sparsity in weights for compressing the data matrices as well as enhancing the robustness of attentional pooling. As exemplary applications, we plug RAP into PointNet and ResNet for point cloud and image recognition, respectively. We demonstrate that our RAP significantly improves the recognition performance for both networks whenever sparsity is high. For instance, in extreme cases where only one feature per matrix is selected for recognition, RAP achieves more than 60% improvement over PointNet in terms of accuracy on the ModelNet40 dataset.

We propose VLASE, a framework to use semantic edge features from images to achieve on-road localization. Semantic edge features denote edge contours that separate pairs of distinct objects such as building-sky, road-sidewalk, and building-ground. While prior work has shown promising results by utilizing the boundary between prominent classes such as sky and building using skylines, we generalize this to consider 19 semantic classes. We extract semantic edge features using CASENet architecture and utilize VLAD framework to perform image retrieval. We achieve improvement over state-of-the-art localization algorithms such as SIFT-VLAD and its deep variant NetVLAD. Ablation study shows the importance of different semantic classes, and our unified approach achieves better performance compared to individual prominent features such as skylines. We also introduce SLC Marathon dataset, a challenging dataset covering most of Salt Lake City with sufficient lighting variations.

We present Barcode, a global binary descriptor for images captured from a vehicle-mounted camera with two applications: localization and turn classification. Barcode characterizes an image by encoding the distribution of vertical lines into a binary descriptor: in each vertical stripe of an image, if any vertical line exists the corresponding bit is set to 1, otherwise 0. For localization, our approach uses a database of geolocated images, each having its Barcode precomputed during a preprocessing stage. In the run time, we first generate the binary descriptor for each image and then use the descriptor to find the location in the database via Hamming distance metric. For turn classification, we train a deep neural network that uses a set of Barcodes from consecutive images to classify turns (left, right, straight, and stationary). We show that Barcode extraction can be done at 100-1000~Hz, localization at 10~kHz, and turn classification at 1~kHz. We show compelling experimental results on KITTI dataset and other sequences captured near Harvard and Purdue campuses.

Visualization applications implicitly run queries on the data to retrieve distributions and statistical measures derivable from distributions. Distribution based data summaries can substitute for the raw data to answer statistical queries of different kinds. However, frequent access to the raw data is no longer practical, if possible at all, for answering large number of queries on large-scale data. Our work addresses the issue by accelerating range distribution query, which returns the distribution of an axis-aligned query region. Maintaining the interactivity of such query is a challenging task because the workload and the response time of such queries scale up with the data and the query size. In this paper, we present a framework for answering range distribution queries for any arbitrary region in near constant time, regardless of data and query size. We adapt an integral histogram based data structure to bound the workload which is a combination of computation, I/O and communication cost. We propose two novel transformations of this data structure -- a decomposition and a similarity-driven indexing -- to reduce the huge storage cost associated with it. In addition to studying the performance of range distribution query, we also demonstrate the benefits that our technique offers to visualization applications which directly or indirectly require distributions.

Large scale scientific simulations frequently use streamline based techniques to visualize flow fields. As the shape of a streamline is often related to some underlying property of the field, it is important to identify streamlines (or their parts) with unique geometric features. In this paper, we introduce a metric, called the box counting ratio, which measures the geometric complexity of streamlines by measuring their space-filling capacity at different scales. We propose a novel interactive visualization framework which utilizes this metric to extract, organize and visualize features of varying density and complexity hidden in large numbers of streamlines. The proposed framework extracts complex regions of varying density from the streamlines, and organizes and presents them on an interactive 2D information space, allowing user selection and visualization of streamlines. We also extend this framework to support exploration using an ensemble of measures including box counting ratio. Our framework allows the user to easily visualize and interact with features otherwise hidden in large vector field data. We strengthen our claims with case studies using combustion and climate simulation data sets.

Streamline-based techniques are designed based on the idea that properties of streamlines are indicative of features in the underlying field. In this paper, we show that statistical distributions of measurements along the trajectory of a streamline can be used as a robust and effective descriptor to measure the similarity between streamlines. With the distribution-based approach, we present a framework for interactive exploration of 3D vector fields with streamline query and clustering. Streamline queries allow us to rapidly identify streamlines that share similar geometric features to the target streamline. Streamline clustering allows us to group together streamlines of similar shapes. Based on user's selection, different clusters with different features at different levels of detail can be visualized to highlight features in 3D flow fields. We demonstrate the utility of our framework with simulation data sets of varying nature and size.

Abstract While isosurfaces have been widely used for scalar data visualization, it is often difficult to determine if the selected isosurfaces for visualization are sufficient to represent the entire scalar field. In this paper, we present an information‐theoretic approach to evaluate the representativeness of a given isosurface set. Our basic idea is that given two isosurfaces that enclose a subvolume, if the intermediate isosurfaces in the subvolume can be generated by smoothly morphing from one isosurface to the other, no additional isosurfaces are needed since the geometry of the true isosurfaces within the subvolume can be easily inferred. To realize this idea, given a pair of isosurfaces, to determine if such a smooth condition in the enclosed region is satisfied, we use a level‐set approach to generate the intermediate surfaces. On each intermediate surface, we sample the values from the scalar field and exam the distribution. If the entropy of the distribution is low, this intermediate surface is aligned well with a true isosurface in the scalar field. For the intermediate surfaces generated by the level‐set method from the boundary isosurfaces, the distributions of scalar values from the level‐set surfaces form a 2D distribution, called isosurface information map. This information map can be used as an indicator of the representativeness of the boundary isosurfaces for the data in the subregion, allowing a quantitative measurement of information representable by the input isosurfaces. Based on this information‐theoretic approach, this paper presents an isosurface selection algorithm that can automatically select isosurfaces for more effective visualization of scalar fields.

Histograms computed from local regions are commonly used in many visualization applications, and allowing the user to query histograms interactively in regions of arbitrary locations and sizes plays an important role in feature identification and tracking. Computing histograms in regions with arbitrary location and size, nevertheless, can be time consuming for large data sets since it involves expensive I/O and scan of data elements. To achieve both performance- and storage-efficient query of local histograms, we present a new algorithm called WaveletSAT, which utilizes integral histograms, an extension of the summed area tables (SAT), and discrete wavelet transform (DWT). Similar to SAT, an integral histogram is the histogram computed from the area between each grid point and the grid origin, which can be be pre-computed to support fast query. Nevertheless, because one histogram contains multiple bins, it will be very expensive to store one integral histogram at each grid point. To reduce the storage cost for large integral histograms, WaveletSAT treats the integral histograms of all grid points as multiple SATs, each of which can be converted into a sparse representation via DWT, allowing the reconstruction of axis-aligned region histograms of arbitrary sizes from a limited number of wavelet coefficients. Besides, we present an efficient wavelet transform algorithm for SATs that can operate on each grid point separately in logarithmic time complexity, which can be extended to parallel GPU-based implementation. With theoretical and empirical demonstration, we show that WaveletSAT can achieve fast preprocessing and smaller storage overhead than the conventional integral histogram approach with close query performance.

We present a file layout algorithm for flow fields to improve runtime I/O efficiency for out-of-core streamline computation. Because of the increasing discrepancy between the speed of processors and storage devices, the cost of I/O becomes a major bottleneck for out-of-core computation. To reduce the I/O cost, loading data with better spatial locality has proved to be effective. It is also known that sequential file access is more efficient. To facilitate efficient streamline computation, we propose to reorganize the data blocks in a file following the data access pattern so that more efficient I/O and effective prefetching can be accomplished. To achieve the goal, we divide the domain into small spatial blocks and order the blocks into a linear layout based on the underlying flow directions. The ordering is done using a weighted directed graph model which can be formulated as a linear graph arrangement problem. Our goal is to arrange the file in a way consistent with the data access pattern during streamline computation. This allows us to prefetch a contiguous segment of data at a time from disk and minimize the memory cache miss rate. We use a recursive partitioning method to approximate the optimal layout. Our experimental results show that the resulting file layout reduces I/O cost and hence enables more efficient out-of-core streamline computation.

Flow fields are an important product of scientific simulations. One popular flow visualization technique is particle advection, in which seeds are traced through the flow field. One use of these traces is to compute a powerful analysis tool called the Finite-Time Lyapunov Exponent (FTLE) field, but no existing particle tracing algorithms scale to the particle injection frequency required for high-resolution FTLE analysis. In this paper, a framework to trace the massive number of particles necessary for FTLE computation is presented. A new approach is explored, in which processes are divided into groups, and are responsible for mutually exclusive spans of time. This pipelining over time intervals reduces overall idle time of processes and decreases I/O overhead. Our parallel FTLE framework is capable of advecting hundreds of millions of particles at once, with performance scaling up to tens of thousands of processes.

Visualization of flow fields with geometric primitives is often challenging due to occlusion that is inevitably introduced by 3D streamlines. In this paper, we present a novel view-dependent algorithm that can minimize occlusion and reveal important flow features for three dimensional flow fields. To analyze regions of higher importance, we utilize Shannon's entropy as a measure of vector complexity. An entropy field in the form of a three dimensional volume is extracted from the input vector field. To utilize this view-independent complexity measure for view-dependent calculations, we introduce the notion of a maximal entropy projection (MEP) framebuffer, which stores maximal entropy values as well as the corresponding depth values for a given viewpoint. With this information, we develop a view-dependent streamline selection algorithm that can evaluate and choose streamlines that will cause minimum occlusion to regions of higher importance. Based on a similar concept, we also propose a viewpoint selection algorithm that works hand-in-hand with our streamline selection algorithm to maximize the visibility of high complexity regions in the flow field.

Particle tracing for streamline and path line generation is a common method of visualizing vector fields in scientific data, but it is difficult to parallelize efficiently because of demanding and widely varying computational and communication loads. In this paper we scale parallel particle tracing for visualizing steady and unsteady flow fields well beyond previously published results. We configure the 4D domain decomposition into spatial and temporal blocks that combine in-core and out-of-core execution in a flexible way that favors faster run time or smaller memory. We also compare static and dynamic partitioning approaches. Strong and weak scaling curves are presented for tests conducted on an IBM Blue Gene/P machine at up to 32 K processes using a parallel flow visualization library that we are developing. Datasets are derived from computational fluid dynamics simulations of thermal hydraulics, liquid mixing, and combustion.

Because of the ever increasing size of output data from scientific simulations, supercomputers are increasingly relied upon to generate visualizations. One use of supercomputers is to generate field lines from large scale flow fields. When generating field lines in parallel, the vector field is generally decomposed into blocks, which are then assigned to processors. Since various regions of the vector field can have different flow complexity, processors will require varying amounts of computation time to trace their particles, causing load imbalance, and thus limiting the performance speedup. To achieve load-balanced streamline generation, we propose a workload-aware partitioning algorithm to decompose the vector field into partitions with near equal workloads. Since actual workloads are unknown beforehand, we propose a workload estimation algorithm to predict the workload in the local vector field. A graph-based representation of the vector field is employed to generate these estimates. Once the workloads have been estimated, our partitioning algorithm is hierarchically applied to distribute the workload to all partitions. We examine the performance of our workload estimation and workload-aware partitioning algorithm in several timings studies, which demonstrates that by employing these methods, better scalability can be achieved with little overhead.

A range of well-known treatment methods for destroying tumor and similar harmful growth in human body utilizes the coherence between the inherently periodic movement of the affected body part and periodic respiratory signal of the patient, with the objective of minimizing damage to surrounding normal tissues. Such methods require constant monitoring by an operator who observes the 3D body motion via its 2D projection onto an ultrasound imaging plane and studies the synchronism of this motion with the respiratory signal. Keeping an attentive eye on the respiratory signal as well as the ultrasound video for the entire treatment period is often inconvenient and burdensome. In this paper, we propose a video visualization technique called CycleStack Plot which reduces this cognitive overhead by blending the video and the signal together in a stack-like layout. This visualization reveals the inherent synchronism between the target's movement and the respiratory signal, visually highlights significant phase shifts of either of the two cyclic phenomena, with the hope of arresting the operator's attention. Our proposed visualization also provides a visual overview for the post-treatment analysis which enables educated users to quickly and effectively skim through the excessively long process. This paper demonstrates the utility of CycleStack Plot with a case study using real ultrasound videos. In addition, a user study has been performed to evaluate the merits and limitations of the proposed method with respect to the conventional way of watching a video and a signal side-by-side. Even though the motivation of the proposed visualization is improvement of medical applications that use ultrasound, the core techniques discussed here have potential to be extended to other application domains requiring analysis of cyclic patterns from videos.

The process of visualization can be seen as a visual communication channel where the input to the channel is the raw data, and the output is the result of a visualization algorithm. From this point of view, we can evaluate the effectiveness of visualization by measuring how much information in the original data is being communicated through the visual communication channel. In this paper, we present an information-theoretic framework for flow visualization with a special focus on streamline generation. In our framework, a vector field is modeled as a distribution of directions from which Shannon's entropy is used to measure the information content in the field. The effectiveness of the streamlines displayed in visualization can be measured by first constructing a new distribution of vectors derived from the existing streamlines, and then comparing this distribution with that of the original data set using the conditional entropy. The conditional entropy between these two distributions indicates how much information in the original data remains hidden after the selected streamlines are displayed. The quality of the visualization can be improved by progressively introducing new streamlines until the conditional entropy converges to a small value. We describe the key components of our framework with detailed analysis, and show that the framework can effectively visualize 2D and 3D flow data.

To analyze time-varying data sets, tracking features over time is often necessary to better understand the dynamic nature of the underlying physical process. Tracking 3D time-varying features, however, is non-trivial when the boundaries of the features cannot be easily defined. In this paper, we propose a new framework to visualize time-varying features and their motion without explicit feature segmentation and tracking. In our framework, a time-varying feature is described by a time series or Time Activity Curve (TAC). To compute the distance, or similarity, between a voxel's time series and the feature, we use the Dynamic Time Warping (DTW) distance metric. The purpose of DTW is to compare the shape similarity between two time series with an optimal warping of time so that the phase shift of the feature in time can be accounted for. After applying DTW to compare each voxel's time series with the feature, a time-invariant distance field can be computed. The amount of time warping required for each voxel to match the feature provides an estimate of the time when the feature is most likely to occur. Based on the TAC-based distance field, several visualization methods can be derived to highlight the position and motion of the feature. We present several case studies to demonstrate and compare the effectiveness of our framework.

We present a new algorithm to explore and visualize multivariate time-varying data sets. We identify important trend relationships among the variables based on how the values of the variables change over time and how those changes are related to each other in different spatial regions and time intervals. The trend relationships can be used to describe the correlation and causal effects among the different variables. To identify the temporal trends from a local region, we design a new algorithm called SUBDTW to estimate when a trend appears and vanishes in a given time series. Based on the beginning and ending times of the trends, their temporal relationships can be modeled as a state machine representing the trend sequence. Since a scientific data set usually contains millions of data points, we propose an algorithm to extract important trend relationships in linear time complexity. We design novel user interfaces to explore the trend relationships, to visualize their temporal characteristics, and to display their spatial distributions. We use several scientific data sets to test our algorithm and demonstrate its utilities.