The rise of generative AI has increased attention to voice interfaces. In immersive analytics, we conceptualize this trend as Speech-driven Immersive Analytics. While speech interfaces enable natural interactions, users, especially novices, still face a learning curve in articulating analytic intent and exploring data during the foraging phase. Prior work has primarily addressed these challenges through multimodal interaction or textual disambiguation. We introduce a context-aware Speech-driven Immersive Analytics framework (SIA) as a speech-oriented approach that leverages speech acts to convey actionable intent. This framework (SIA) was designed based on a formative study, a prototype development, three technical studies, and a user study. By extracting speech acts from utterances, SIA infers analytic tasks and embodiment tendencies, then integrates them with spatial, chart, and data context to generate feedforward: previews of potential actions and outcomes. The formative study identified user needs. The technical studies demonstrated that SIA improved the inference quality, enabling context-aware feedforward generation. The user study highlighted that the SIA-based prototype was responsive and intuitive, and feedforward helped users learn during the onboarding phase of data exploration. In particular, the user study identified which feedforward elements participants referenced and how they applied them when expressing intent in immersive analytics. Our key technical findings emphasize that the ensemble model, embedded in the Uncertainty Estimator, improves accuracy and stabilizes task inference. The Projector’s context summary was critical in generating context-aware feedforward. Based on these results, we discuss future research directions for intelligent Speech-driven Immersive Analytics.

Embodiment shapes how users verbally express intent when interacting with data through speech interfaces in immersive analytics. Despite growing interest in Natural Language Interactions (NLIs) for visual analytics in immersive environments, users' speech patterns and their use of embodiment cues in speech remain underexplored. Understanding their interplay is crucial to bridging the gap between users' intent and an immersive analytic system. To address this, we report the results from 15 participants in a user study conducted using the Wizard of Oz method. We performed axial coding on 1,280 speech acts derived from 734 utterances, examining how analysis tasks are carried out with embodiment and linguistic features. Next, we measured Speech Input Uncertainty for each analysis task using the semantic entropy of utterances, estimating how uncertain users' speech inputs appear to an analytic system. Through these analyses, we identified five speech input patterns, showing that users dynamically blend embodied and non-embodied speech acts depending on data analysis tasks, phases, and Embodiment Reliance driven by the counts and types of embodiment cues in each utterance. We then examined how these patterns align with user reflections on factors that challenge speech interaction during the study. Finally, we propose design implications aligned with the five patterns.

We introduce a framework for compactly representing light field content with the novel concept of neural subspaces. While the recently proposed neural light field representation achieves great compression results by encoding a light field into a single neural network, the unified design is not optimized for the composite structures exhibited in light fields. Moreover, encoding every part of the light field into one network is not ideal for applications that require rapid transmission and decoding. We recognize this problem's connection to subspace learning. We present a method that uses several small neural networks, specializing in learning the neural subspace for a particular light field segment. Moreover, we propose an adaptive weight sharing strategy among those small networks, improving parameter efficiency. In effect, this strategy enables a concerted way to track the similarity among nearby neural subspaces by leveraging the layered structure of neural networks. Furthermore, we develop a soft-classification technique to enhance the color prediction accuracy of neural representations. Our experimental results show that our method better reconstructs the light field than previous methods on various light field scenes. We further demonstrate its successful deployment on encoding light fields with irregular viewpoint layout and dynamic scene content.

High-precision virtual environments are increasingly important for various education, simulation, training, performance, and entertainment applications. We present HoloCamera, an innovative volumetric capture instrument to rapidly acquire, process, and create cinematic-quality virtual avatars and scenarios. The HoloCamera consists of a custom-designed free-standing structure with 300 high-resolution RGB cameras mounted with uniform spacing spanning the four sides and the ceiling of a room-sized studio. The light field acquired from these cameras is streamed through a distributed array of GPUs that interleave the processing and transmission of 4K resolution images. The distributed compute infrastructure that powers these RGB cameras consists of 50 Jetson AGX Xavier boards, with each processing unit dedicated to driving and processing imagery from six cameras. A high-speed Gigabit Ethernet network fabric seamlessly interconnects all computing boards. In this systems paper, we provide an in-depth description of the steps involved and lessons learned in constructing such a cutting-edge volumetric capture facility that can be generalized to other such facilities. We delve into the techniques employed to achieve precise frame synchronization and spatial calibration of cameras, careful determination of angled camera mounts, image processing from the camera sensors, and the need for a resilient and robust network infrastructure. To advance the field of volumetric capture, we are releasing a high-fidelity static light-field dataset, which will serve as a benchmark for further research and applications of cinematic-quality volumetric light fields.

This paper presents a compact driver circuit with independent pixel-level temperature regulation for thermo-optic based 2D Nanophotonic phased arrays (NPAs) in light detection and ranging (LIDAR) and Virtual Reality (VR) applications. To minimize the interconnection density, the proposed driver unit uses only a single electrical contact to its corresponding NPA pixel for both heating and temperature measurement functions. The driver was fabricated using TSMC 65 nm technology and each unit is realized in an area of$15\ \mu\mathrm{m}\times 15\ \mu\mathrm{m}$. The design enables scalable 3D heterogeneous integration between any tile-based NPA with pixel pitch below$15\ \mu\mathrm{m}$and its electrical control system. The temperature regulation performance of the proposed circuit was characterized by intentionally introducing a ±20% variation to the load resistance to simulate the temperature deviation in the NPA. The measured phase errors are suppressed by the feedback controller to a maximum of$0.07\pi$and an average of$0.02\pi$within the full$2\pi$phase shift operation range.

WiFi networks are essential to our daily lives, but their signals are not visible to us. Therefore, it is challenging to evaluate the health of a network or make changes to ensure an optimal configuration. Traditional visualization approaches, such as contour lines, are not intuitive and lead to challenges in the analysis and comprehension of networks. In this paper, we introduce two novel visualizations: Wavelines and Stacked Bars. We then compared these visualizations to the state-of-the-art visualization technique of contour lines. We carried out a user study with 32 participants to validate that our novel visualizations can improve user confidence, accuracy, and completion time for the tasks of router localization, ranking of signal strengths, channel interference, and router coverage. We selected these tasks after extensive discussions with domain experts. We believe that our findings will assist network analysts in visually understanding our increasingly rich signal environments.

Neural representations have shown great promise in their ability to represent radiance and light fields while being very compact compared to the image set representation. However, current representations are not well suited for streaming as decoding can only be done at a single level of detail and requires downloading the entire neural network model. Furthermore, high-resolution light field networks can exhibit flickering and aliasing as neural networks are sampled without appropriate filtering. To resolve these issues, we present a progressive multi-scale light field network that encodes a light field with multiple levels of detail. Lower levels of detail are encoded using fewer neural network weights enabling progressive streaming and reducing rendering time. Our progressive multi-scale light field network addresses aliasing by encoding smaller anti-aliased representations at its lower levels of detail. Additionally, per-pixel level of detail enables our representation to support dithered transitions and foveated rendering.

We present VIINTER, a method for view interpolation by interpolating the implicit neural representation (INR) of the captured images. We leverage the learned code vector associated with each image and interpolate between these codes to achieve viewpoint transitions. We propose several techniques that significantly enhance the interpolation quality. VIINTER signifies a new way to achieve view interpolation without constructing 3D structure, estimating camera poses, or computing pixel correspondence. We validate the effectiveness of VIINTER on several multi-view scenes with different types of camera layout and scene composition. As the development of INR of images (as opposed to surface or volume) has centered around tasks like image fitting and super-resolution, with VIINTER, we show its capability for view interpolation and offer a promising outlook on using INR for image manipulation tasks.

The Nanophotonic Phased Array (NPA) is an emerging holographic display technology. With chip-scaled sizes, high refresh rates, and integrated light sources, a large-scale NPA can enable high-resolution real-time dynamic holographic displays. However, one of the critical challenges impeding the development of such large-scale NPAs is the high electrical power consumption required to modulate the amplitude and phase of each of the pixel elements. We argue that the modulation of all the elements on the array is, in fact, not necessary to produce a high-quality image. We propose a simple method that outputs the configuration of a sparse NPA, along with the amplitude and the phase required at each active pixel to generate the desired image at the observation plane. We identify the set of active pixels according to their optimized intensities. We observe that the brighter pixels have a greater influence on the target image, and it is these that we must focus on in image formation. Using as few as 10% of the total pixels from a dense 2D array of light-emitting elements, we show that a perceptually acceptable holographic image can be generated. We compare various sparse sampling methods through computational simulations and show that our proposed method gives superior qualitative and quantitative results. We believe our study will help advance research on sparse NPAs and facilitate the use of large-scale NPAs to display high-resolution 3D holographic images.

We present a novel neural representation for light field content that enables compact storage and easy local reconstruction with high fidelity. We use a fully-connected neural network to learn the mapping function between each light field pixel’s coordinates and its corresponding color values. Since neural networks that simply take in raw coordinates are unable to accurately learn data containing fine details, we present an input transformation strategy based on the Gegenbauer polynomials, which previously showed theoretical advantages over the Fourier basis. We conduct experiments that show our Gegenbauer-based design combined with sinusoidal activation functions leads to a better light field reconstruction quality than a variety of network designs, including those with Fourier-inspired techniques introduced by prior works. Moreover, our SInusoidal Gegenbauer NETwork, or SIGNET, can represent light field scenes more compactly than the state-of-the-art compression methods while maintaining a comparable reconstruction quality. SIGNET also innately allows random access to encoded light field pixels due to its functional design. We further demonstrate that SIGNET’s super-resolution capability without any additional training.

Holographic displays and computer-generated holography offer a unique opportunity in improving optical resolutions and depth characteristics of near-eye displays. The thermally-modulated Nanopho-tonic Phased Array (NPA), a new type of holographic display, affords several advantages, including integrated light source and higher refresh rates, over other holographic display technologies. However, the thermal phase modulation of the NPA makes it susceptible to the thermal proximity effect where heating one pixel affects the temperature of nearby pixels. Proximity effect correction (PEC) methods have been proposed for 2D Fourier holograms in the far field but not for Fresnel holograms at user-specified depths. Here we extend an existing PEC method for the NPA to Fresnel holograms with phase-only hologram optimization and validate it through computational simulations. Our method is not only effective in correcting the proximity effect for the Fresnel holograms of 2D images at desired depths but can also leverage the fast refresh rate of the NPA to display 3D scenes with time-division multiplexing.

With the rapidly increasing resolutions of 360° cameras, head-mounted displays, and live-streaming services, streaming high-resolution panoramic videos over limited-bandwidth networks is becoming a critical challenge. Foveated video streaming can address this rising challenge in the context of eye-tracking-equipped virtual reality head-mounted displays. However, conventional log-polar foveated rendering suffers from a number of visual artifacts such as aliasing and flickering. In this paper, we introduce a new log-rectilinear transformation that incorporates summed-area table filtering and off-the-shelf video codecs to enable foveated streaming of 360° videos suitable for VR headsets with built-in eye-tracking. To validate our approach, we build a client-server system prototype for streaming 360° videos which leverages parallel algorithms over real-time video transcoding. We conduct quantitative experiments on an existing 360° video dataset and observe that the log-rectilinear transformation paired with summed-area table filtering heavily reduces flickering compared to log-polar subsampling while also yielding an additional 10% reduction in bandwidth usage.

Light fields capture both the spatial and angular rays, thus enabling free-viewpoint rendering and custom selection of the focal plane. Scientists can interactively explore pre-recorded microscopic light fields of organs, microbes, and neurons using virtual reality headsets. However, rendering high-resolution light fields at interactive frame rates requires a very high rate of texture sampling, which is challenging as the resolutions of light fields and displays continue to increase. In this article, we present an efficient algorithm to visualize 4D light fields with 3D-kernel foveated rendering (3D-KFR). The 3D-KFR scheme coupled with eye-tracking has the potential to accelerate the rendering of 4D depth-cued light fields dramatically. We have developed a perceptual model for foveated light fields by extending the KFR for the rendering of 3D meshes. On datasets of high-resolution microscopic light fields, we observe 3.47×-7.28× speedup in light field rendering with minimal perceptual loss of detail. We envision that 3D-KFR will reconcile the mutually conflicting goals of visual fidelity and rendering speed for interactive visualization of light fields.

While 360° images are becoming ubiquitous due to popularity of panoramic content, they cannot directly work with most of the existing depth estimation techniques developed for perspective images. In this paper, we present a deep-learning-based framework of estimating depth from 360° images. We present an adaptive depth refinement procedure that refines depth estimates using normal estimates and pixel-wise uncertainty scores. We introduce double quaternion approximation to combine the loss of the joint estimation of depth and surface normal. Furthermore, we use the double quaternion formulation to also measure stereo consistency between the horizontally displaced depth maps, leading to a new loss function for training a depth estimation CNN. Results show that the new double-quaternion-based loss and the adaptive depth refinement procedure lead to better network performance. Our proposed method can be used with monocular as well as stereo images. When evaluated on several datasets, our method surpasses state-of-the-art methods on most metrics.

External ventricular drainage (EVD) is a high-risk medical procedure that involves inserting a catheter inside a patient’s skull, through the brain and into a ventricle, to drain cerebrospinal fluid and thus relieve elevated intracranial pressure. Once the catheter has entered the skull, its tip cannot be seen. The neurosurgeon has to imagine its location inside the cranium and direct it toward the ventricle using only anatomic landmarks. The EVD catheter is thin and thus hard to track using infra-red depth sensors. Traditional optical tracking using fiducial or other markers inevitably changes the shape or weight of the medical instrument. We present an augmented reality system that depicts the catheter for EVD and a new technique to precisely track the catheter inside the skull. Our technique uses a new linear marker detection method that requires minimal changes to the catheter and is well suited for tracking other thin medical devices that require high-precision tracking.

Optimizing rendering performance is critical for a wide variety of virtual reality (VR) applications. Foveated rendering is emerging as an indispensable technique for reconciling interactive frame rates with ever-higher head-mounted display resolutions. Here, we present a simple yet effective technique for further reducing the cost of foveated rendering by leveraging ocular dominance - the tendency of the human visual system to prefer scene perception from one eye over the other. Our new approach, eye-dominance-guided foveated rendering (EFR), renders the scene at a lower foveation level (with higher detail) for the dominant eye than the non-dominant eye. Compared with traditional foveated rendering, EFR can be expected to provide superior rendering performance while preserving the same level of perceived visual quality.

Thermally modulated Nanophotonic Phased Arrays (NPAs) can be used as phase-only holographic displays. Compared to the holographic displays based on Liquid Crystal on Silicon Spatial Light Modulators (LCoS SLMs), NPAs have the advantage of integrated light source and high refresh rate. However, the formation of the desired wavefront requires accurate modulation of the phase which is distorted by the thermal proximity effect. This problem has been largely overlooked and existing approaches to similar problems are either slow or do not provide a good result in the setting of NPAs. We propose two new algorithms based on the iterative phase retrieval algorithm and the proximal algorithm to address this challenge. We have carried out computational simulations to compare and contrast various algorithms in terms of image quality and computational efficiency. This work is going to benefit the research on NPAs and enable the use of large-scale NPAs as holographic displays.

We present Geollery, an interactive mixed reality social media platform for creating, sharing, and exploring geotagged information. Geollery introduces a real-time pipeline to progressively render an interactive mirrored world with three-dimensional (3D) buildings, internal user-generated content, and external geotagged social media. This mirrored world allows users to see, chat, and collaborate with remote participants with the same spatial context in an immersive virtual environment. We describe the system architecture of Geollery, its key interactive capabilities, and our design decisions. Finally, we conduct a user study with 20 participants to qualitatively compare Geollery with another social media system, Social Street View. Based on the participants' responses, we discuss the benefits and drawbacks of each system and derive key insights for designing an interactive mirrored world with geotagged social media. User feedback from our study reveals several use cases for Geollery including travel planning, virtual meetings, and family gathering.

Reconstruction of the physical world in real time has been a grand challenge in computer graphics and 3D vision. In this paper, we introduce an interactive pipeline to reconstruct a mirrored world at two levels of detail. Given a pair of latitude and longitude coordinates, our pipeline streams and caches depth maps, street view panoramas, and building polygons from Google Maps and OpenStreetMap APIs. At a fine level of detail for close-up views, we render textured meshes using adjacent local street views and depth maps. When viewed from afar, we apply projection mappings to 3D geometries extruded from building polygons for a coarse level of detail. In contrast to teleportation, our system allows users to virtually walk through the mirrored world at the street level. We present an application of our approach by incorporating it into a mixed-reality social platform, Geollery, and validate our real-time strategies on various platforms including mobile phones, workstations, and head-mounted displays.

We introduce a novel secure display system, which uses visual cryptography [4] with tolerance for tracking. Our system brings cryptographic privacy from text to virtual worlds [3]. Much like traditional encryption that uses a public key and a private key, our system uses two images that are both necessary for visual decryption of the data. The public image could be widely shared on a printed page, on a traditional display (desktop, tablet, or smartphone), or in a multi-participant virtual world, while the other private image can be exclusively on a user's personal AR or VR display. Only the recipient is able to visually decrypt the data by fusing both images. In contrast to prior art, our system is able to provide tracking tolerance, making it more practically usable in modern VR and AR systems. We model the probability of misalignment caused by head or body jitter as a Gaussian distribution. Our algorithm diffuses the second image using the normalized probabilities, thus enabling the visual cryptography to be tolerant of alignment errors due to tracking.

Deep learning has emerged as a powerful tool for feature-driven labeling of datasets. However, for it to be effective, it requires a large and finely labeled training dataset. Precisely labeling a large training dataset is expensive, time-consuming, and error prone. In this article, we present a visually driven deep-learning approach that starts with a coarsely labeled training dataset and iteratively refines the labeling through intuitive interactions that leverage the latent structures of the dataset. Our approach can be used to (a) alleviate the burden of intensive manual labeling that captures the fine nuances in a high-dimensional dataset by simple visual interactions, (b) replace a complicated (and therefore difficult to design) labeling algorithm by a simpler (but coarse) labeling algorithm supplemented by user interaction to refine the labeling, or (c) use low-dimensional features (such as the RGB colors) for coarse labeling and turn to higher-dimensional latent structures that are progressively revealed by deep learning, for fine labeling. We validate our approach through use cases on three high-dimensional datasets and a user study.

Designing volume visualizations showing various structures of interest is critical to the exploratory analysis of volumetric data. The last few years have witnessed dramatic advances in the use of convolutional neural networks for identification of objects in large image collections. Whereas such machine learning methods have shown superior performance in a number of applications, their direct use in volume visualization has not yet been explored. In this paper, we present a deep-learning-assisted volume visualization to depict complex structures, which are otherwise challenging for conventional approaches. A significant challenge in designing volume visualizations based on the high-dimensional deep features lies in efficiently handling the immense amount of information that deep-learning methods provide. In this paper, we present a new technique that uses spectral methods to facilitate user interactions with high-dimensional features. We also present a new deep-learning-assisted technique for hierarchically exploring a volumetric dataset. We have validated our approach on two electron microscopy volumes and one magnetic resonance imaging dataset.

Far peripheral vision (beyond 60° eccentricity) is beginning to be supported in the latest virtual and augmented reality (VR and AR) headsets. This benefits the VR and AR experiences by allowing a greater amount of information to be conveyed, reducing visual clutter, and enabling subtle visual attention management. However, the visual properties of the far periphery are different from those of the central vision, because of the physiological differences between the areas on the visual cortex responsible for the respective vision types. In this paper, we investigate the perception time in the far peripheral vision, specifically the time it takes for a user to perceive a pattern at a high eccentricity. We have characterized the perception time in the far peripheral vision by conducting a user study on 40 participants in which the participants distinguish between two types of patterns displayed at several sizes and at various eccentricities in their field of view. Our results show that at higher eccentricities, participants take longer to perceive a pattern. Based on user study data, we are able to characterize the desired scaling of patterns at higher eccentricities, so that they can be perceived within a similar amount of time as in the central vision.

The commoditization of virtual and augmented reality devices and the availability of inexpensive consumer depth cameras have catalyzed a resurgence of interest in spatiotemporal performance capture. Recent systems like Fusion4D and Holoportation address several crucial problems in the real-time fusion of multiview depth maps into volumetric and deformable representations. Nonetheless, stitching multiview video textures onto dynamic meshes remains challenging due to imprecise geometries, occlusion seams, and critical time constraints. In this paper, we present a practical solution towards real-time seamless texture montage for dynamic multiview reconstruction. We build on the ideas of dilated depth discontinuities and majority voting from Holoportation to reduce ghosting effects when blending textures. In contrast to their approach, we determine the appropriate blend of textures per vertex using view-dependent rendering techniques, so as to avert fuzziness caused by the ubiquitous normal-weighted blending. By leveraging geodesics-guided diffusion and temporal texture fields, our algorithm mitigates spatial occlusion seams while preserving temporal consistency. Experiments demonstrate significant enhancement in rendering quality, especially in detailed regions such as faces. We envision a wide range of applications for Montage4D, including immersive telepresence for business, training, and live entertainment.

The analysis of vast amounts of network data for monitoring and safeguarding a core pillar of the internet, the root DNS, is an enormous challenge. Understanding the distribution of the queries received by the root DNS, and how those queries change over time, in an intuitive manner is sought. Traditional query analysis is performed packet by packet, lacking global, temporal, and visual coherence, obscuring latent trends and clusters. Our approach leverages the pattern recognition and computational power of deep learning with 2D and 3D rendering techniques for quick and easy interpretation and interaction with vast amount of root DNS network traffic. Working with real-world DNS experts, our visualization reveals several surprising latent clusters of queries, potentially malicious and benign, discovers previously unknown characteristics of a real-world root DNS DDOS attack, and uncovers unforeseen changes in the distribution of queries received over time. These discoveries will provide DNS analysts with a deeper understanding of the nature of the DNS traffic under their charge, which will help them safeguard the root DNS against future attack.

Analyzing live-cell images is particularly challenging because cells simultaneously move and undergo systematic changes. Visually inspecting live-cell images therefore involves simultaneously tracking individual cells and detecting relevant spatio-temporal changes. The high cognitive burden of such a complex task makes this kind of analysis inefficient and error prone. In this paper, we describe a deep-learning-assisted visualization based on automatically derived high-level features to identify target cell changes in live-cell images. Applying a novel user-mediated color assignment scheme that maps abstract features into corresponding colors, we create color-based visual annotations that facilitate visual reasoning and analysis of complex time-varying live-cell image datasets.

Modern imaging technologies enable the study of microstructural features, which require capturing the finest details in high-resolution gigapixel images. Nevertheless, the resolution disparity between gigapixel images and megapixel displays presents a challenge to effective visual analysis because subtle texture differences are hardly perceivable at coarser resolutions. In this paper, we present a hierarchical segmentation technique based on the joint distribution of intensity and noise-resistant local binary patterns to differentiate subtle microstructural textures across various scales. The coarse-to-fine segmentation procedure subdivides each parent segment into texturally-distinct child segments at progressively higher resolutions. The hierarchical structure of segments allows creating intermediate segmentation results interactively. Based on the intermediate results, we highlight regions with texture differences using distinct colors, which provide salient visual hints to users despite the current viewing resolution. Our new technique has been validated on large microscopy images and shows promising results.

Much of the success of convolutional neural networks (CNNs) is due to the enormous collections of labeled data, powerful GPUs, and modern network architectures that facilitate the training and testing of larger and deeper models. The limited size of labeled volumetric microscopy images, however, hinders the proper training of CNNs for volume segmentation. Here, we design various 2D and 3D CNNs with factorized convolutions and online feature-level augmentations to address challenges due to the scarcity of training data. Based on the experimental results, we found that the 3D CNNs consistently outperformed the 2D counterparts. In addition, the 3D CNN that uses both factorized convolutions and online feature-level augmentations achieved the best segmentation performance.

Diffusion kurtosis imaging (DKI) is gaining rapid adoption in the medical imaging community due to its ability to measure the non-Gaussian property of water diffusion in biological tissues. Compared to traditional diffusion tensor imaging (DTI), DKI can provide additional details about the underlying microstructural characteristics of the neural tissues. It has shown promising results in studies on changes in gray matter and mild traumatic brain injury where DTI is often found to be inadequate. The DKI dataset, which has high-fidelity spatio-angular fields, is difficult to visualize. Glyph-based visualization techniques are commonly used for visualization of DTI datasets; however, due to the rapid changes in orientation, lighting, and occlusion, visually analyzing the much more higher fidelity DKI data is a challenge. In this paper, we provide a systematic way to manage, analyze, and visualize high-fidelity spatio-angular fields from DKI datasets, by using spherical harmonics lighting functions to facilitate insights into the brain microstructure.

Visual exploration of volumetric datasets to discover the embedded features and spatial structures is a challenging and tedious task. In this paper we present a semi-automatic approach to this problem that works by visually segmenting the intensity-gradient 2D histogram of a volumetric dataset into an exploration hierarchy. Our approach mimics user exploration behavior by analyzing the histogram with the normalized-cut multilevel segmentation technique. Unlike previous work in this area, our technique segments the histogram into a reasonable set of intuitive components that are mutually exclusive and collectively exhaustive. We use information-theoretic measures of the volumetric data segments to guide the exploration. This provides a data-driven coarse-to-fine hierarchy for a user to interactively navigate the volume in a meaningful manner.

The three papers in this special section were presented at the Symposium on Interactive 3D Graphics and Games (I3D) that was held in San Francisco, CA, 18-20 February 2011.

The field of visualization has addressed navigation of very large datasets, usually meshes and volumes. Significantly less attention has been devoted to the issues surrounding navigation of very large images. In the last few years the explosive growth in the resolution of camera sensors and robotic image acquisition techniques has widened the gap between the display and image resolutions to three orders of magnitude or more. This paper presents the first steps towards navigation of very large images, particularly landscape images, from an interactive visualization perspective. The grand challenge in navigation of very large images is identifying regions of potential interest. In this paper we outline a three-step approach. In the first step we use multi-scale saliency to narrow down the potential areas of interest. In the second step we outline a method based on statistical signatures to further cull out regions of high conformity. In the final step we allow a user to interactively identify the exceptional regions of high interest that merit further attention. We show that our approach of progressive elicitation is fast and allows rapid identification of regions of interest. Unlike previous work in this area, our approach is scalable and computationally reasonable on very large images. We validate the results of our approach by comparing them to user-tagged regions of interest on several very large landscape images from the Internet.

Mesh saliency has been proposed as a computational model of perceptual importance for meshes, and it has been used in graphics for abstraction, simplification, segmentation, illumination, rendering, and illustration. Even though this technique is inspired by models of low-level human vision, it has not yet been validated with respect to human performance. Here, we present a user study that compares the previous mesh saliency approaches with human eye movements. To quantify the correlation between mesh saliency and fixation locations for 3D rendered images, we introduce the normalized chance-adjusted saliency by improving the previous chance-adjusted saliency measure. Our results show that the current computational model of mesh saliency can model human eye movements significantly better than a purely random model or a curvature-based model.

Artists, illustrators, photographers, and cinematographers have long used the principles of contrast and composition to guide visual attention. In this paper we introduce geometry modification as a tool to persuasively direct visual attention. We build upon recent advances in mesh saliency to develop techniques to alter geometry to elicit greater visual attention. Eye-tracking-based user studies show that our approach successfully guides user attention in a statistically significant manner. Our approach operates directly on geometry, and therefore, produces view-independent results that can be used with existing view-dependent techniques of visual persuasion.

BACKGROUND: The recent availability of new, less expensive high-throughput DNA sequencing technologies has yielded a dramatic increase in the volume of sequence data that must be analyzed. These data are being generated for several purposes, including genotyping, genome resequencing, metagenomics, and de novo genome assembly projects. Sequence alignment programs such as MUMmer have proven essential for analysis of these data, but researchers will need ever faster, high-throughput alignment tools running on inexpensive hardware to keep up with new sequence technologies. RESULTS: This paper describes MUMmerGPU, an open-source high-throughput parallel pairwise local sequence alignment program that runs on commodity Graphics Processing Units (GPUs) in common workstations. MUMmerGPU uses the new Compute Unified Device Architecture (CUDA) from nVidia to align multiple query sequences against a single reference sequence stored as a suffix tree. By processing the queries in parallel on the highly parallel graphics card, MUMmerGPU achieves more than a 10-fold speedup over a serial CPU version of the sequence alignment kernel, and outperforms the exact alignment component of MUMmer on a high end CPU by 3.5-fold in total application time when aligning reads from recent sequencing projects using Solexa/Illumina, 454, and Sanger sequencing technologies. CONCLUSION: MUMmerGPU is a low cost, ultra-fast sequence alignment program designed to handle the increasing volume of data produced by new, high-throughput sequencing technologies. MUMmerGPU demonstrates that even memory-intensive applications can run significantly faster on the relatively low-cost GPU than on the CPU.

Recent research in visual saliency has established a computational measure of perceptual importance. In this paper we present a visual-saliency-based operator to enhance selected regions of a volume. We show how we use such an operator on a user-specified saliency field to compute an emphasis field. We further discuss how the emphasis field can be integrated into the visualization pipeline through its modifications of regional luminance and chrominance. Finally, we validate our work using an eye-tracking-based user study and show that our new saliency enhancement operator is more effective at eliciting viewer attention than the traditional Gaussian enhancement operator.

In this paper, we introduce geometry-dependent lighting that allows lighting parameters to be defined independently and possibly discrepantly over an object or scene based on the local geometry. We present and discuss Light Collages, a lighting design system with geometry-dependent lights for effective feature-enhanced visualization. Our algorithm segments the objects into local surface patches and places lights that are locally consistent but globally discrepant to enhance the perception of shape. We use spherical harmonics for efficiently storing and computing light placement and assignment. We also outline a method to find the minimal number of light sources sufficient to illuminate an object well with our globally discrepant lighting approach.

Traditional geometry representations have focused on representing the details of the geometry in a deterministic fashion. In this article we propose a statistical representation of the geometry that leverages local coherence for very large datasets. We show how the statistical analysis of a densely sampled point model can be used to improve the geometry bandwidth bottleneck, both on the system bus and over the network as well as for randomized rendering, without sacrificing visual realism. Our statistical representation is built using a clustering-based hierarchical principal component analysis (PCA) of the point geometry. It gives us a hierarchical partitioning of the geometry into compact local nodes representing attributes such as spatial coordinates, normal, and color. We pack this information into a few bytes using classification and quantization. This allows our representation to directly render from compressed format for efficient remote as well as local rendering. Our representation supports both view-dependent and on-demand rendering. Our approach renders each node using quasi-random sampling utilizing the probability distribution derived from the PCA analysis. We show many benefits of our approach: (1) several-fold improvement in the storage and transmission complexity of point geometry; (2) direct rendering from compressed data; and (3) support for local and remote rendering on a variety of rendering platforms such as CPUs, GPUs, and PDAs.

Research over the last decade has built a solid mathematical foundation for representation and analysis of 3D meshes in graphics and geometric modeling. Much of this work however does not explicitly incorporate models of low-level human visual attention. In this paper we introduce the idea of mesh saliency as a measure of regional importance for graphics meshes. Our notion of saliency is inspired by low-level human visual system cues. We define mesh saliency in a scale-dependent manner using a center-surround operator on Gaussian-weighted mean curvatures. We observe that such a definition of mesh saliency is able to capture what most would classify as visually interesting regions on a mesh. The human-perception-inspired importance measure computed by our mesh saliency operator results in more visually pleasing results in processing and viewing of 3D meshes. compared to using a purely geometric measure of shape. such as curvature. We discuss how mesh saliency can be incorporated in graphics applications such as mesh simplification and viewpoint selection and present examples that show visually appealing results from using mesh saliency.

We introduce Light Collages - a lighting design system for effective visualization based on principles of human perception. Artists and illustrators enhance perception of features with lighting that is locally consistent and globally inconsistent. Inspired by these techniques, we design the placement of light sources to convey a greater sense of realism and better perception of shape with globally inconsistent lighting. Our algorithm segments the objects into local surface patches and uses a number of perceptual heuristics, such as highlights, shadows, and silhouettes, to enhance the perception of shape. We show our results on scientific and sculptured datasets.

Subsurface scattering is important for photo-realistic rendering of translucent materials. We make approximations to the BSSRDF model and propose a simple lighting model to simulate the effects on translucent meshes. Our approximations are based on the observation that subsurface scattering is relatively local due to its exponential falloff.In the preprocessing stage we build subsurface scattering neighborhood information, which includes all the vertices within effective scattering range from each vertex. We then modify the traditional local illumination model into a run-time two-stage process. The first stage involves computation of reflection and transmission of light on surface vertices. The second stage bleeds in scattering effects from a vertex's neighborhood to generate the final result. We then merge the run-time two-stage process into a run-time single-stage process using precomputed integrals, and reduce the complexity of our run-time algorithm to O ( N ), where N is the number of vertices. The selection of the optimum set size for precomputed integrals is guided by a standard imagespace error-metric. Furthermore, we show how to compress the precomputed integrals using spherical harmonics. We compensate for the inadequacy of spherical harmonics for storing high frequency components by a reference points scheme to store high frequency components of the precomputed integrals explicitly. With this approach, we greatly reduce memory usage without loss of visual quality under a high-frequency lighting environment and achieve interactive frame rates for medium-sized scenes. Our model is able to capture the most important features of subsurface scattering: reflection and transmission due to multiple scattering.

We present a novel rendering primitive that combines the modeling brevity of points with the rasterization efficiency of polygons. The surface is represented by a sampled collection of Differential Points (DP), each with embedded curvature information that captures the local differential geometry in the vicinity of that point. This is a more general point representation that, for the cost of a few additional bytes, packs much more information per point than the traditional point-based models. This information is used to efficiently render the surface as a collection of local geometries. To use the hardware acceleration, the DPs are quantized into 256 different types and each sampled point is approximated by the closest quantized DP and is rendered as a normal-mapped rectangle. The advantages to this representation are: 1) The surface can be represented more sparsely compared to other point primitives, 2) it achieves a robust hardware accelerated per-pixel shading - even with no connectivity information, and 3) it offers a novel point-based simplification technique that factors in the complexity of the local geometry. The number of primitives being equal, DPs produce a much better quality of rendering than a pure splat-based approach. Visual appearances being similar, DPs are about two times faster and require about 75 percent less disk space in comparison to splatting primitives.

We propose the idea of using variable-precision geometry transformations and lighting to accelerate 3D graphics rendering. Multiresolution approaches reduce the number of primitives to be rendered; our approach complements the multiresolution techniques as it reduces the precision of each graphics primitive. Our method relates the minimum number of bits of accuracy required in the input data to achieve a desired accuracy in the display output. We achieve speedup by taking advantage of (a) SIMD parallelism for arithmetic operations, now increasingly common on modern processors, and (b) spatial-temporal coherence in frame-to-frame transformations and lighting. We show the results of our method on datasets from several application domains including laser-scanned, procedural, and mechanical CAD datasets.

View-dependent simplification has emerged as a powerful tool for graphics acceleration in visualization of complex environments. However, view-dependent simplification techniques have not been able to take full advantage of the underlying graphics hardware. Specifically, triangle strips are a widely used hardware-supported mechanism to compactly represent and efficiently render static triangle meshes. However, in a view-dependent framework, the triangle mesh connectivity changes at every frame, making it difficult to use triangle strips. We present a novel data structure, Skip Strip, that efficiently maintains triangle strips during such view-dependent changes. A Skip Strip stores the vertex hierarchy nodes in a skip-list-like manner with path compression. We anticipate that Skip Strips will provide a road map to combine rendering acceleration techniques for static datasets, typical of retained-mode graphics applications, with those for dynamic datasets found in immediate-mode applications.

We propose a technique for performing view‐dependent geometry and topology simplifications for level‐of‐detail‐based renderings of large models. The algorithm proceeds by preprocessing the input dataset into a binary tree, the view‐dependence tree of general vertex‐pair collapses. A subset of the Delaunay edges is used to limit the number of vertex pairs considered for topology simplification. Dependencies to avoid mesh foldovers in manifold regions of the input object are stored in the view‐dependence tree in an implicit fashion. We have observed that this not only reduces the space requirements by a factor of two, it also highly localizes the memory accesses at run time. The view‐dependence tree is used at run time to generate the triangles for display. We also propose a cubic‐spline‐based distance metric that can be used to unify the geometry and topology simplifications by considering the vertex positions and normals in an integrated manner.

We present a topology simplifying approach that can be used for genus reductions, removal of protuberances, and repair of cracks in polygonal models in a unified framework. Our work is complementary to the existing work on geometry simplification of polygonal datasets and we demonstrate that using topology and geometry simplifications together yields superior multiresolution hierarchies than is possible by using either of them alone. Our approach can also address the important issue of repair of cracks in polygonal models, as well as for rapid identification and removal of protuberances based on internal accessibility in polygonal models. Our approach is based on identifying holes and cracks by extending the concept of /spl alpha/-shapes to polygonal meshes under the L/sub /spl infin// distance metric. We then generate valid triangulations to fill them using the intuitive notion of sweeping an L/sub /spl infin// cube over the identified regions.

We present a z-buffered image-space-based rendering technique that allows navigation in complex static environments.The rendering speed is relatively insensitive to the complexity of the scene as the rendering is performed a priori, and the scene is converted into a bounded complexity representation in the image space.Real-time performance is attained by using hardware texture mapping to implement the imagespace warping and hardware fine transformations to compute the viewpoint-dependent warping function.Our pro posed method correctly simulates the kinetic depth effect (parallax), occlusion, and can resolve the missing visibility information by combining z-bufTered environment maps from multiple viewpoints.

Genus-reducing simplifications are important in constructing multiresolution hierarchies for level-of-detail-based rendering, especially for datasets that have several relatively small holes, tunnels, and cavities. We present a genus-reducing simplification approach that is complementary to the existing work on genus-preserving simplifications. We propose a simplification framework in which genus-reducing and genus-preserving simplifications alternate to yield much better multiresolution hierarchies than would have been possible by using either one of them. In our approach we first identify the holes and the concavities by extending the concept of /spl alpha/-hulls to polygonal meshes under the L/sub /spl infin// distance metric and then generate valid triangulations to fill them.

We present an algorithm for performing adaptive real-time level-of-detail-based rendering for triangulated polygonal models. The simplifications are dependent on viewing direction, lighting, and visibility and are performed by taking advantage of image-space, object-space, and frame-to-frame coherences. In contrast to the traditional approaches of precomputing a fixed number of level-of-detail representations for a given object, our approach involves statically generating a continuous level-of-detail representation for the object. This representation is then used at run time to guide the selection of appropriate triangles for display. The list of displayed triangles is updated incrementally from one frame to the next. Our approach is more effective than the current level-of-detail-based rendering approaches for most scientific visualization applications, where there are a limited number of highly complex objects that stay relatively close to the viewer. Our approach is applicable for scalar (such as distance from the viewer) as well as vector (such as normal direction) attributes.

We propose the idea of simplification envelopes for generating a hierarchy of level-of-detail approximations for a given polygonal model.Our approach guarantees that all points of an approximation are within a user-specifiable distance from the original model and that all points of the original model are within a distance from the approximation.Simplification envelopes provide a general framework within which a large collection of existing simplification algorithms can run.We demonstrate this technique in conjunction with two algorithms, one local, the other global.The local algorithm provides a fast method for generating approximations to large input meshes (at least hundreds of thousands of triangles).The global algorithm provides the opportunity to avoid local "minima" and possibly achieve better simplifications as a result.Each approximation attempts to minimize the total number of polygons required to satisfy the above constraint.The key advantages of our approach are: General technique providing guaranteed error bounds for genus-preserving simplification Automation of both the simplification process and the selection of appropriate viewing distances Prevention of self-intersection Preservation of sharp features Allows variation of approximation distance across different portions of a model CR Categories and Subject Descriptors: I.3.

Almost all scientific visualization involving surfaces is currently done via triangles. The speed at which such triangulated surfaces can be displayed is crucial to interactive visualization and is bounded by the rate at which triangulated data can be sent to the graphics subsystem for rendering. Partitioning polygonal models into triangle strips can significantly reduce rendering times over transmitting each triangle individually. We present new and efficient algorithms for constructing triangle strips from partially triangulated models, and experimental results showing these strips are on average 15% better than those from previous codes. Further, we study the impact of larger buffer sizes and various queuing disciplines on the effectiveness of triangle strips.

Presents an algorithm for performing view-dependent simplifications of a triangulated polygonal model in real-time. The simplifications are dependent on viewing direction, lighting and visibility, and are performed by taking advantage of image-space, object-space and frame-to-frame coherences. A continuous level-of-detail representation for an object is first constructed off-line. This representation is then used at run-time to guide the selection of appropriate triangles for display. The list of displayed triangles is updated incrementally from one frame to the next. Our approach is more effective than the current level-of-detail-based rendering approaches for most scientific visualization applications where there are a limited number of highly complex objects that stay relatively close to the viewer.

We present a simple, robust, and practical method for object simplification for applications where gradual elimination of high frequency details is desired. This is accomplished by converting an object into multi resolution volume rasters using a controlled filtering and sampling technique. A multiresolution triangle mesh hierarchy can then be generated by applying the Marching Cubes algorithm. We further propose an adaptive surface generation algorithm to reduce the number of triangles generated by the standard Marching Cubes. Our method simplifies the topology of objects in a controlled fashion. In addition, at each level of detail, multilayered meshes can be used for an efficient antialiased rendering.

Presents a simple, robust and practical method for object simplification for applications where gradual elimination of high-frequency details is desired. This is accomplished by sampling and low-pass filtering the object into multi-resolution volume buffers and applying the marching cubes algorithm to generate a multi-resolution triangle-mesh hierarchy. Our method simplifies the genus of objects and can also help existing object simplification algorithms achieve better results. At each level of detail, a multi-layered mesh can be used for an optional and efficient antialiased rendering.

A parallel, analytic approach for defining and computing the inter and intra molecular interfaces in three dimensions is described. The molecular interface surfaces are derived from approximations to the power diagrams over the participating molecular units. For a given molecular interface our approach can generate a family of interface surfaces parametrized by /spl alpha/ and /spl beta/, where /spl alpha/ is the radius of the solvent molecule (also known as the probe radius) and /spl beta/ is the interface radius that defines the size of the molecular interface. Molecular interface surfaces provide biochemists with a powerful tool to study surface complementarity and to efficiently characterize the interactions during a protein substrate docking. The complexity of our algorithm for molecular environments is O(nk log/sup 2/ k), where n is the number of atoms in the participating molecular units and k is the average number of neighboring atoms-a constant, given /spl alpha/ and /spl beta/.

An interactive visualization of weighted three-dimensional alpha-hulls is presented for static and dynamic spheres. The alpha-hull is analytically computed and represented by a triangulated mesh. The entire surface is computed and displayed in real-time at interactive rates. The weighted three-dimensional alpha-hulls are equivalent to smooth molecular surfaces of biochemistry. Biochemistry applications of interactive computation and display of alpha-hulls or smooth molecular surfaces are outlined.

An algorithm for rapid computation of Richards's smooth molecular surface is described. The entire surface is computed analytically, triangulated, and displayed at interactive rates. The faster speeds for our program have been achieved by algorithmic improvements, paralleling the computations, and by taking advantage of the special geometrical properties of such surfaces. Our algorithm is easily parallelable and it has a time complexity of O (k log k) over n processors, where n is the number of atoms of the molecule and k is the average number of neighbors per atom.>

We describe a software system on the Pixel-Planes 5 graphics engine that displays user-defined antialiased procedural textures at rates of about 30 frames per second for use in realtime graphics applications.Our system allows a user to create textures that can modulate both diffuse and specular color, the sharpness of specular highlights, the amount of transparency and the surface normals of an object.We describe a texture editor that allows a user to interactively create and edit procedural textures.Antialiasing is essential for real-time textures, and in this paper we present some techniques for antialiasing procedural textures.Another direction we are exploring is the use of dynamic textures, which are functions of time or orientation.Examples of textures we have generated include a translucent fire texture that waves and flickers and an animated water texture that shows the use of both environment mapping and normal perturbation (bump mapping).