Multiplex imaging is revolutionizing pathology by enabling the simultaneous visualization of multiple biomarkers within tissue samples, providing molecular-level insights that traditional hematoxylin and eosin (H&E) staining cannot provide. However, the complexity and cost of multiplex data acquisition have hindered its widespread adoption. Additionally, most existing large repositories of H&E images lack corresponding multiplex images, limiting opportunities for multi-modal analysis. To address these challenges, we leverage recent advances in latent diffusion models (LDMs), which excel at modeling complex data distributions by utilizing their powerful priors for fine-tuning to a target domain. In this paper, we introduce a novel framework for virtual multiplex staining that utilizes pretrained LDM parameters to generate multiplex images from H&E images using a conditional diffusion model. Our approach enables marker-by-marker generation by conditioning the diffusion model on each marker, while sharing the same architecture across all markers. To tackle the challenge of varying pixel value distributions across different marker stains and to improve inference speed, we fine-tune the model for single-step sampling, enhancing both color contrast fidelity and inference efficiency through pixel-level loss functions. We validate our framework on two publicly available datasets, notably demonstrating its effectiveness in generating up to 18 different marker types with improved accuracy, a substantial increase over the 2-3 marker types achieved in previous approaches. This validation highlights the potential of our framework, pioneering virtual multiplex staining. Finally, this paper bridges the gap between H&E and multiplex imaging, potentially enabling retrospective studies and large-scale analyses of existing H&E image repositories.

Abstract Single‐cell RNA sequencing (scRNA‐seq) has gained prominence as a valuable technique for examining cellular gene expression patterns at the individual cell level. In the analysis of scRNA‐seq datasets, it is common practice to visualise a subset of principal components (PCs), obtained via principal component analysis (PCA), using dimensionality reduction techniques such as t‐stochastic neighbour embedding (t‐SNE). Determining the number of PCs (i.e. dimensionality) is a critical step that influences the outcome of single‐cell analysis, and this process typically requires a labour‐intensive manual assessment involving the inspection of numerous projection plots. To address this challenge, we present a visualisation system that assists analysts in efficiently determining the optimal dimensionality of scRNA‐seq data. The proposed system employs two hull heatmaps, a cell type heatmap and a cluster heatmap, which offer comprehensive representations of target cells of multiple cell types across various dimensionalities through the utilisation of a convex hull‐embedded colour map. The cell type heatmap shows overlaps between cell types, and the cluster heatmap compares cell clustering results. The proposed hull heatmaps effectively alleviate the labourious task of manually evaluating hundreds of projection plots for searching for the optimal dimensionality. Additionally, our system offers interactive visualisation of gene expression levels and an intuitive lasso selection tool, thereby enabling analysts to progressively refine the convex hulls on the hull heatmaps. We validated the usefulness of the proposed system through two quantitative evaluations and three case studies.

Immersive analytics is gaining attention across multiple domains due to its capability to facilitate intuitive data analysis in expansive environments through user interaction with data. However, creating immersive analytics systems for specific tasks is challenging due to the need for programming expertise and significant development effort. Despite the introduction of various immersive visualization authoring toolkits, domain experts still face hurdles in adopting immersive analytics into their workflow, particularly when faced with dynamically changing tasks and data in real time. To lower such technical barriers, we introduce XROps, a web-based authoring system that allows users to create immersive analytics applications through interactive visual programming, without the need for low-level scripting or coding. XROps enables dynamic immersive analytics authoring by allowing users to modify each step of the data visualization process with immediate feedback, enabling them to build visualizations on-the-fly and adapt to changing environments. It also supports the integration and visualization of real-time sensor data from XR devices-a key feature of immersive analytics-facilitating the creation of various analysis scenarios. We evaluated the usability of XROps through a user study and demonstrate its efficacy and usefulness in several example scenarios.

In the rapidly evolving field of deep learning, traditional methodologies for designing models predominantly rely on code-based frameworks. While these approaches provide flexibility, they create a significant barrier to entry for non-experts and obscure the immediate impact of architectural decisions on model performance. In response to this challenge, recent no-code approaches have been developed with the aim of enabling easy model development through graphical interfaces. However, both traditional and no-code methodologies share a common limitation that the inability to predict model outcomes or identify issues without executing the model. To address this limitation, we introduce an intuitive visual feedback-based no-code approach to visualize and analyze deep learning models during the design phase. This approach utilizes dataflow-based visual programming with dynamic visual encoding of model architecture. A user study was conducted with deep learning developers to demonstrate the effectiveness of our approach in enhancing the model design process, improving model understanding, and facilitating a more intuitive development experience. The findings of this study suggest that real-time architectural visualization significantly contributes to more efficient model development and a deeper understanding of model behaviors.

Neurons have a polarized structure, with dendrites and axons, and compartment-specific functions can be affected by the dwelling mitochondria. Recent studies have shown that the morphology of mitochondria is closely related to the functions of neurons and neurodegenerative diseases. However, the conventional mitochondria analysis workflow mainly relies on manual annotations and generic image-processing software. Moreover, even though there have been recent developments in automatic mitochondria analysis using deep learning, the application of existing methods in a daily analysis remains challenging because the performance of a pretrained deep learning model can vary depending on the target data, and there are always errors in inference time, requiring human proofreading. To address these issues, we introduce MitoVis, a novel visualization system for end-to-end data processing and an interactive analysis of the morphology of neuronal mitochondria. MitoVis introduces a novel active learning framework based on recent contrastive learning, which allows accurate fine-tuning of the neural network model. MitoVis also provides novel visual guides for interactive proofreading so that users can quickly identify and correct errors in the result with minimal effort. We demonstrate the usefulness and efficacy of the system via case studies conducted by neuroscientists. The results show that MitoVis achieved up to 13.3× faster total analysis time in the case study compared to the conventional manual analysis workflow.

Single-cell RNA sequencing (scRNA-seq) is becoming popular in studying the gene expression of cells at the single-cell level. ScRNA-seq enables analysts to characterize cell types, thereby providing a better understanding of dynamic biological processes. In scRNA-seq data analysis, principal component analysis (PCA) is commonly used to reduce at least thousands of dimensions in the raw data to a manageable size so that analysts can visualize and cluster cells to identify different cell types. The conventional process to determine the optimal dimensionality includes a laborious manual review of hundreds of different projection plots. To address this problem, we introduce a dimensionality explorer for single-cell analysis, which is a visualization system that helps analysts to effectively determine the optimal dimensionality of scRNA-seq data. It employs a hull heatmap, which provides a holistic view of overlaps among multiple cell types across various dimensionalities using a convex hull-embedded color map. The hull heatmap effectively reduces the burden of manually reviewing hundreds of projection plots to determine the optimal dimensionality. Our system also provides interactive gene expression level visualization and intuitive lasso selection, thereby allowing analysts to progressively refine the convex hulls of the hull heatmap. We demonstrate the usefulness of the proposed system through a user study and three case studies conducted by domain experts.

The accurate prediction of cancer drug sensitivity according to the multiomics profiles of individual patients is crucial for precision cancer medicine. However, the development of prediction models has been challenged by the complex crosstalk of input features and the resistance-dominant drug response information contained in public databases. In this study, we propose a novel multidrug response prediction framework, response-aware multitask prediction (RAMP), via a Bayesian neural network and restrict it by soft-supervised contrastive regularization. To utilize network embedding vectors as representation learning features for heterogeneous networks, we harness response-aware negative sampling, which applies cell line-drug response information to the training of network embeddings. RAMP overcomes the prediction accuracy limitation induced by the imbalance of trained response data based on the comprehensive selection and utilization of drug response features. When trained on the Genomics of Drug Sensitivity in Cancer dataset, RAMP achieved an area under the receiver operating characteristic curve > 89%, an area under the precision-recall curve > 59% and an $\textrm{F}_1$ score > 52% and outperformed previously developed methods on both balanced and imbalanced datasets. Furthermore, RAMP predicted many missing drug responses that were not included in the public databases. Our results showed that RAMP will be suitable for the high-throughput prediction of cancer drug sensitivity and will be useful for guiding cancer drug selection processes. The Python implementation for RAMP is available at https://github.com/hvcl/RAMP.

Microsatellite instability (MSI) refers to alterations in the length of simple repetitive genomic sequences. MSI status serves as a prognostic and predictive factor in colorectal cancer. The MSI-high status is a good prognostic factor in stage II/III cancer, and predicts a lack of benefit to adjuvant fluorouracil chemotherapy in stage II cancer but a good response to immunotherapy in stage IV cancer. Therefore, determining MSI status in patients with colorectal cancer is important for identifying the appropriate treatment protocol. In the Pathology Artificial Intelligence Platform (PAIP) 2020 challenge, artificial intelligence researchers were invited to predict MSI status based on colorectal cancer slide images. Participants were required to perform two tasks. The primary task was to classify a given slide image as belonging to either the MSI-high or the microsatellite-stable group. The second task was tumor area segmentation to avoid ties with the main task. A total of 210 of the 495 participants enrolled in the challenge downloaded the images, and 23 teams submitted their final results. Seven teams from the top 10 participants agreed to disclose their algorithms, most of which were convolutional neural network-based deep learning models, such as EfficientNet and UNet. The top-ranked system achieved the highest F1 score (0.9231). This paper summarizes the various methods used in the PAIP 2020 challenge. This paper supports the effectiveness of digital pathology for identifying the relationship between colorectal cancer and the MSI characteristics.

Dendritic spines are dynamic, submicron-scale protrusions on neuronal dendrites that receive neuronal inputs. Morphological changes in the dendritic spine often reflect alterations in physiological conditions and are indicators of various neuropsychiatric conditions. However, owing to the highly dynamic and heterogeneous nature of spines, accurate measurement and objective analysis of spine morphology are major challenges in neuroscience research. Most conventional approaches for analyzing dendritic spines are based on two-dimensional (2D) images, which barely reflect the actual three-dimensional (3D) shapes. Although some recent studies have attempted to analyze spines with various 3D-based features, it is still difficult to objectively categorize and analyze spines based on 3D morphology. Here, we propose a unified visualization framework for an interactive 3D dendritic spine analysis system, DXplorer, that displays 3D rendering of spines and plots the high-dimensional features extracted from the 3D mesh of spines. With this system, users can perform the clustering of spines interactively and explore and analyze dendritic spines based on high-dimensional features. We propose a series of high-dimensional morphological features extracted from a 3D mesh of dendritic spines. In addition, an interactive machine learning classifier with visual exploration and user feedback using an interactive 3D mesh grid view ensures a more precise classification based on the spine phenotype. A user study and two case studies were conducted to quantitatively verify the performance and usability of the DXplorer. We demonstrate that the system performs the entire analytic process effectively and provides high-quality, accurate, and objective analysis.

This paper presents a mobile AR system for visualizing airflow and temperature change made by virtual air conditioners. Even though there have been efforts to integrate the results of airflow/temperature simulation into the real world via AR, they support neither interactive modeling of the environments nor real-time simulation. This paper presents an AR system, where 3D mapping and air conditioner installation are made interactively, and then airflow/temperature simulation and visualization are made at real time. The proposed system is designed in a client-server architecture, where the server is in charge of simulation and the rest is taken by the client.

Instance segmentation, the task of identifying and separating each individual object of interest in the image, is one of the actively studied research topics in computer vision. Although many feed-forward networks produce high-quality binary segmentation on different types of images, their final result heavily relies on the post-processing step, which separates instances from the binary mask. In comparison, the existing iterative methods extract a single object at a time using discriminative knowledge-based properties (e.g., shapes, boundaries, etc.) without relying on post-processing. However, they do not scale well with a large number of objects. To exploit the advantages of conventional sequential segmentation methods without impairing the scalability, we propose a novel iterative deep reinforcement learning agent that learns how to differentiate multiple objects in parallel. By constructing a relational graph between pixels, we design a reward function that encourages separating pixels of different objects and grouping pixels that belong to the same instance. We demonstrate that the proposed method can efficiently perform instance segmentation of many objects without heavy post-processing.

With the advent of advances in self-supervised learning, paired clean-noisy data are no longer required in deep learning-based image denoising. However, existing blind denoising methods still require the assumption with regard to noise characteristics, such as zero-mean noise distribution and pixel-wise noise-signal independence; this hinders wide adaptation of the method in the medical domain. On the other hand, unpaired learning can overcome limitations related to the assumption on noise characteristics, which makes it more feasible for collecting the training data in real-world scenarios. In this paper, we propose a novel image denoising scheme, Interdependent Self-Cooperative Learning (ISCL), that leverages unpaired learning by combining cyclic adversarial learning with self-supervised residual learning. Unlike the existing unpaired image denoising methods relying on matching data distributions in different domains, the two architectures in ISCL, designed for different tasks, complement each other and boost the learning process. To assess the performance of the proposed method, we conducted extensive experiments in various biomedical image degradation scenarios, such as noise caused by physical characteristics of electron microscopy (EM) devices (film and charging noise), and structural noise found in low-dose computer tomography (CT). We demonstrate that the image quality of our method is superior to conventional and current state-of-the-art deep learning-based unpaired image denoising methods.

With the advent of advances in imaging and computing technologies, large-scale data acquisition and processing have become commonplace in many science and engineering disciplines. Conventional workflows for large-scale data processing usually rely on in-house or commercial software that are designed for domain-specific computing tasks. Recent advances in MapReduce, which was originally developed for batch processing textual data via a simplified programming model of the map and reduce functions, have expanded its applications to more general tasks in big-data processing, such as scientific computing, and biomedical image processing. However, as shown in previous work, volume rendering and visualization using MapReduce is still considered challenging and impractical owing to the disk-based, batch-processing nature of its computing model. In this article, contrary to this common belief, we show that the MapReduce computing model can be effectively used for interactive visualization. Our proposed system is a novel extension of Spark, one of the most popular open-source MapReduce frameworks, which offers GPU-accelerated MapReduce computing. To minimize CPU-GPU communication and overcome slow, disk-based shuffle performance, the proposed system supports GPU in-memory caching and MPI-based direct communication between compute nodes. To allow for GPU-accelerated in-situ visualization using raster graphics in Spark, we leveraged the CUDA-OpenGL interoperability, resulting in faster processing speeds by several orders of magnitude compared to conventional MapReduce systems. We demonstrate the performance of our system via several volume processing and visualization tasks, such as direct volume rendering, iso-surface extraction, and numerical simulations with in-situ visualization.

This paper presents DXR, a toolkit for building immersive data visualizations based on the Unity development platform. Over the past years, immersive data visualizations in augmented and virtual reality (AR, VR) have been emerging as a promising medium for data sense-making beyond the desktop. However, creating immersive visualizations remains challenging, and often require complex low-level programming and tedious manual encoding of data attributes to geometric and visual properties. These can hinder the iterative idea-to-prototype process, especially for developers without experience in 3D graphics, AR, and VR programming. With DXR, developers can efficiently specify visualization designs using a concise declarative visualization grammar inspired by Vega-Lite. DXR further provides a GUI for easy and quick edits and previews of visualization designs in-situ, i.e., while immersed in the virtual world. DXR also provides reusable templates and customizable graphical marks, enabling unique and engaging visualizations. We demonstrate the flexibility of DXR through several examples spanning a wide range of applications.

Compressed sensing magnetic resonance imaging (CS-MRI) has provided theoretical foundations upon which the time-consuming MRI acquisition process can be accelerated. However, it primarily relies on iterative numerical solvers, which still hinders their adaptation in time-critical applications. In addition, recent advances in deep neural networks have shown their potential in computer vision and image processing, but their adaptation to MRI reconstruction is still in an early stage. In this paper, we propose a novel deep learning-based generative adversarial model, RefineGAN, for fast and accurate CS-MRI reconstruction. The proposed model is a variant of fully-residual convolutional autoencoder and generative adversarial networks (GANs), specifically designed for CS-MRI formulation; it employs deeper generator and discriminator networks with cyclic data consistency loss for faithful interpolation in the given under-sampled -space data. In addition, our solution leverages a chained network to further enhance the reconstruction quality. RefineGAN is fast and accurate-the reconstruction process is extremely rapid, as low as tens of milliseconds for reconstruction of a image, because it is one-way deployment on a feed-forward network, and the image quality is superior even for extremely low sampling rate (as low as 10%) due to the data-driven nature of the method. We demonstrate that RefineGAN outperforms the state-of-the-art CS-MRI methods by a large margin in terms of both running time and image quality via evaluation using several open-source MRI databases.

In this paper, we propose a novel machine learning-based voxel classification method for highly-accurate volume rendering. Unlike conventional voxel classification methods that incorporate intensity-based features, the proposed method employs dictionary based features learned directly from the input data using hierarchical multi-scale 3D convolutional sparse coding, a novel extension of the state-of-the-art learning-based sparse feature representation method. The proposed approach automatically generates high-dimensional feature vectors in up to 75 dimensions, which are then fed into an intelligent system built on a random forest classifier for accurately classifying voxels from only a handful of selection scribbles made directly on the input data by the user. We apply the probabilistic transfer function to further customize and refine the rendered result. The proposed method is more intuitive to use and more robust to noise in comparison with conventional intensity-based classification methods. We evaluate the proposed method using several synthetic and real-world volume datasets, and demonstrate the methods usability through a user study.

Due to its simplicity and scalability, MapReduce has become a de facto standard computing model for big data processing. Since the original MapReduce model was only appropriate for embarrassingly parallel batch processing, many follow-up studies have focused on improving the efficiency and performance of the model. Spark follows one of these recent trends by providing in-memory processing capability to reduce slow disk I/O for iterative computing tasks. However, the acceleration of Spark's in-memory processing using graphics processing units (GPUs) is challenging due to its deep memory hierarchy and host-to-GPU communication overhead. In this paper, we introduce a novel GPU-accelerated MapReduce framework that extends Spark's in-memory processing so that iterative computing is performed only in the GPU memory. Having discovered that the main bottleneck in the current Spark system for GPU computing is data communication on a Java virtual machine, we propose a modification of the current Spark implementation to bypass expensive data management for iterative task offloading to GPUs. We also propose a novel GPU in-memory processing and caching framework that minimizes host-to-GPU communication via lazy evaluation and reuses GPU memory over multiple mapper executions. The proposed system employs message-passing interface (MPI)-based data synchronization for inter-worker communication so that more complicated iterative computing tasks, such as iterative numerical solvers, can be efficiently handled. We demonstrate the performance of our system in terms of several iterative computing tasks in big data processing applications, including machine learning and scientific computing. We achieved up to 50 times speed up over conventional Spark and about 10 times speed up over GPU-accelerated Spark.

In the past decade, many numerical algorithms for the Eikonal equation have been proposed. Recently, the research of Eikonal equation solver has focused more on developing efficient parallel algorithms in order to leverage the computing power of parallel systems, such as multi-core CPUs and GPUs (Graphics Processing Units). In this paper, we introduce an efficient parallel algorithm that extends Jeong et al.'s FIM (Fast Iterative Method, [1]), originally developed for the GPU, for multi-core shared memory systems. First, we propose a parallel implementation of FIM using a lock-free local queue approach and provide an in-depth analysis of the parallel performance of the method. Second, we propose a new parallel algorithm, Group-Ordered Fast Iterative Method (GO-FIM), that exploits causality of grid blocks to reduce redundant computations, which was the main drawback of the original FIM. In addition, the proposed GO-FIM method employs clustering of blocks based on the updating order where each cluster can be updated in parallel using multi-core parallel architectures. We discuss the performance of GO-FIM and compare with the state-of-the-art parallel Eikonal equation solvers.

In this paper, we introduce a fast alternating method for reconstructing highly undersampled dynamic MRI data using 3D convolutional sparse coding. The proposed solution leverages Fourier Convolution Theorem to accelerate the process of learning a set of 3D filters and iteratively refine the MRI reconstruction based on the sparse codes found subsequently. In contrast to conventional CS methods which exploit the sparsity by applying universal transforms such as wavelet and total variation, our approach extracts and adapts the temporal information directly from the MRI data using compact shift-invariant 3D filters. We provide a highly parallel algorithm with GPU support for efficient computation, and therefore, the reconstruction outperforms CPU implementation of the state-of-the art dictionary learning-based approaches by up to two orders of magnitude. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Wavelet transform has been widely used in many signal and image processing applications. Due to its wide adoption for time-critical applications, such as streaming and real-time signal processing, many acceleration techniques were developed during the past decade. Recently, the graphics processing unit (GPU) has gained much attention for accelerating computationally-intensive problems and many solutions of GPU-based discrete wavelet transform (DWT) have been introduced, but most of them did not fully leverage the potential of the GPU. In this paper, we present various state-of-the-art GPU optimization strategies in DWT implementation, such as leveraging shared memory, registers, warp shuffling instructions, and thread- and instruction-level parallelism (TLP, ILP), and finally elaborate our hybrid approach to further boost up its performance. In addition, we introduce a novel mixed-band memory layout for Haar DWT, where multi-level transform can be carried out in a single fused kernel launch. As a result, unlike recent GPU DWT methods that focus mainly on maximizing ILP, we show that the optimal GPU DWT performance can be achieved by hybrid parallelism combining both TLP and ILP together in a mixed-band approach. We demonstrate the performance of our proposed method by comparison with other CPU and GPU DWT methods.

Inherently multi-dimensional n-ary indexing structures such as R-trees are not well suited for the GPU because of their irregular memory access patterns and recursive back-tracking function calls. It has been known that traversing hierarchical tree structures in an irregular manner makes it difficult to exploit parallelism and to maximize the utilization of GPU processing units. Moreover, the recursive tree search algorithms often fail with large indexes because of the GPU's tiny runtime stack size. In this paper, we propose a novel parallel tree traversal algorithm-massively parallel restart scanning (MPRS) for multi-dimensional range queries that avoids recursion and irregular memory access. The proposed MPRS algorithm traverses hierarchical tree structures with mostly contiguous memory access patterns without recursion, which offers more chances to optimize the parallel SIMD algorithm. We implemented the proposed MPRS range query processing algorithm on n-ary bounding volume hierarchies including R-trees and evaluated its performance using real scientific datasets on an NVIDIA Tesla M2090 GPU. Our experiments show braided parallel SIMD friendly MPRS range query algorithm achieves at least 80 percent warp execution efficiency while task parallel tree traversal algorithm shows only 9-15 percent efficiency. Moreover, braided parallel MPRS algorithm accesses 7-20 times less amount of global memory than task parallel parent link algorithm by virtue of minimal warp divergence.

Discrete wavelet transform (DWT) has been widely used in many image compression applications, such as JPEG2000 and compressive sensing MRI. Even though a lifting scheme [1] has been widely adopted to accelerate DWT, only a handful of research has been done on its efficient implementation on many-core accelerators, such as graphics processing units (GPUs). Moreover, we observe that rearranging the spatial locations of wavelet coefficients at every level of DWT significantly impairs the performance of memory transaction on the GPU. To address these problems, we propose a mixed-band lifting wavelet transform that reduces uncoalesced global memory access on the GPU and maximizes on-chip memory bandwidth by implementing in-place operations using registers. We assess the performance of the proposed method by comparing with the state-of-the-art DWT libraries, and show its usability in a compressive sensing (CS) MRI application.

As the size of image data from microscopes and telescopes increases, the need for high-throughput processing and visualization of large volumetric data has become more pressing. At the same time, many-core processors and GPU accelerators are commonplace, making high-performance distributed heterogeneous computing systems affordable. However, effectively utilizing GPU clusters is difficult for novice programmers, and even experienced programmers often fail to fully leverage the computing power of new parallel architectures due to their steep learning curve and programming complexity. In this paper, we propose Vivaldi, a new domain-specific language for volume processing and visualization on distributed heterogeneous computing systems. Vivaldi's Python-like grammar and parallel processing abstractions provide flexible programming tools for non-experts to easily write high-performance parallel computing code. Vivaldi provides commonly used functions and numerical operators for customized visualization and high-throughput image processing applications. We demonstrate the performance and usability of Vivaldi on several examples ranging from volume rendering to image segmentation.

Abstract Collaborative slide image viewing systems are becoming increasingly important in pathology applications such as telepathology and E‐learning. Despite rapid advances in computing and imaging technology, current digital pathology systems have limited performance with respect to remote viewing of whole slide images on desktop or mobile computing devices. In this paper we present a novel digital pathology client–server system that supports collaborative viewing of multi‐plane whole slide images over standard networks using multi‐touch‐enabled clients. Our system is built upon a standard HTTP web server and a MySQL database to allow multiple clients to exchange image and metadata concurrently. We introduce a domain‐specific image‐stack compression method that leverages real‐time hardware decoding on mobile devices. It adaptively encodes image stacks in a decorrelated colour space to achieve extremely low bitrates (0.8 bpp) with very low loss of image quality. We evaluate the image quality of our compression method and the performance of our system for diagnosis with an in‐depth user study.

This paper presents the first volume visualization system that scales to petascale volumes imaged as a continuous stream of high-resolution electron microscopy images. Our architecture scales to dense, anisotropic petascale volumes because it: (1) decouples construction of the 3D multi-resolution representation required for visualization from data acquisition, and (2) decouples sample access time during ray-casting from the size of the multi-resolution hierarchy. Our system is designed around a scalable multi-resolution virtual memory architecture that handles missing data naturally, does not pre-compute any 3D multi-resolution representation such as an octree, and can accept a constant stream of 2D image tiles from the microscopes. A novelty of our system design is that it is visualization-driven: we restrict most computations to the visible volume data. Leveraging the virtual memory architecture, missing data are detected during volume ray-casting as cache misses, which are propagated backwards for on-demand out-of-core processing. 3D blocks of volume data are only constructed from 2D microscope image tiles when they have actually been accessed during ray-casting. We extensively evaluate our system design choices with respect to scalability and performance, compare to previous best-of-breed systems, and illustrate the effectiveness of our system for real microscopy data from neuroscience.

We describe a set of image editing and viewing tools that explicitly take into account the resolution of the display on which the image is viewed. Our approach is twofold. First, we design editing tools that process only the visible data, which is useful for images larger than the display. This encompasses cases such as multi-image panoramas and high-resolution medical data. Second, we propose an adaptive way to set viewing parameters such brightness and contrast. Because we deal with very large images, different locations and scales often require different viewing parameters. We let users set these parameters at a few places and interpolate satisfying values everywhere else. We demonstrate the efficiency of our approach on different display and image sizes. Since the computational complexity to render a view depends on the display resolution and not the actual input image resolution, we achieve interactive image editing even on a 16 gigapixel image.

Histology is the study of the structure of biological tissue using microscopy techniques. As digital imaging technology advances, high resolution microscopy of large tissue volumes is becoming feasible; however, new interactive tools are needed to explore and analyze the enormous datasets. In this paper we present a visualization framework that specifically targets interactive examination of arbitrarily large image stacks. Our framework is built upon two core techniques: display-aware processing and GPU-accelerated texture compression. With display-aware processing, only the currently visible image tiles are fetched and aligned on-the-fly, reducing memory bandwidth and minimizing the need for time-consuming global pre-processing. Our novel texture compression scheme for GPUs is tailored for quick browsing of image stacks. We evaluate the usability of our viewer for two histology applications: digital pathology and visualization of neural structure at nanoscale-resolution in serial electron micrographs.

Recent advances in scanning technology provide high resolution EM (Electron Microscopy) datasets that allow neuro-scientists to reconstruct complex neural connections in a nervous system. However, due to the enormous size and complexity of the resulting data, segmentation and visualization of neural processes in EM data is usually a difficult and very time-consuming task. In this paper, we present NeuroTrace, a novel EM volume segmentation and visualization system that consists of two parts: a semi-automatic multiphase level set segmentation with 3D tracking for reconstruction of neural processes, and a specialized volume rendering approach for visualization of EM volumes. It employs view-dependent on-demand filtering and evaluation of a local histogram edge metric, as well as on-the-fly interpolation and ray-casting of implicit surfaces for segmented neural structures. Both methods are implemented on the GPU for interactive performance. NeuroTrace is designed to be scalable to large datasets and data-parallel hardware architectures. A comparison of NeuroTrace with a commonly used manual EM segmentation tool shows that our interactive workflow is faster and easier to use for the reconstruction of complex neural processes.

The distribution of geometric features is anisotropic by its nature. Intrinsic properties of surfaces such as normal curvatures, for example, varies with direction. In this paper this characteristic of a shape is used to create a new anisotropic geodesic (AG) distance map on parametric surfaces. We first define local distance (LD) from a point as a function of both the surface point and a unit direction in its tangent plane and then define a total distance as an integral of that local distance. The AG distance between points on the surface is then defined as their minimum total distance. The path between the points that attains the minimum is called the anisotropic geodesic path. This differs from the usual geodesic in ways that enable it to better reveal geometric features. Minimizing total distances to attain AG distance is performed by associating the LD function with the tensor speed function that controls wave propagation of the convex Hamilton-Jacobi (H-J) equation solver. We present two different, but related metrics for the local distance function, a curvature tensor and a difference curvature tensor. Each creates a different AG distance. Some properties of both new AG distance maps are presented, including parametrization invariance. We then demonstrate the effectiveness of the proposed geodesic map as a shape discriminator in several applications, including surface segmentation and partial shape matching.

In this paper we present a method to compute and visualize volumetric white matter connectivity in diffusion tensor magnetic resonance imaging (DT-MRI) using a Hamilton-Jacobi (H-J) solver on the GPU (Graphics Processing Unit). Paths through the volume are assigned costs that are lower if they are consistent with the preferred diffusion directions. The proposed method finds a set of voxels in the DTI volume that contain paths between two regions whose costs are within a threshold of the optimal path. The result is a volumetric optimal path analysis, which is driven by clinical and scientific questions relating to the connectivity between various known anatomical regions of the brain. To solve the minimal path problem quickly, we introduce a novel numerical algorithm for solving H-J equations, which we call the Fast Iterative Method (FIM). This algorithm is well-adapted to parallel architectures, and we present a GPU-based implementation, which runs roughly 50-100 times faster than traditional CPU-based solvers for anisotropic H-J equations. The proposed system allows users to freely change the endpoints of interesting pathways and to visualize the optimal volumetric path between them at an interactive rate. We demonstrate the proposed method on some synthetic and real DT-MRI datasets and compare the performance with existing methods.

We present a method for the adaptive reconstruction of a surface directly from an unorganized point cloud. The algorithm is based on an incrementally expanding neural network and the statistical analysis of its learning process. In particular, we make use of the simple observation that during the learning process the normal of a vertex near a sharp edge or a high curvature area of the target space, statistically, will vary more than the normal of a vertex near a flat area. We use the information obtained from the study of these normal variations to steer the learning process in an adaptive meshing application, producing meshes with more triangles near the high curvature areas. The same information is used in a feature detection application.

We study the use of neural network algorithms in surface reconstruction from an unorganized point cloud, and meshing of an implicit surface. We found that for such applications, the most suitable type of neural networks is a modified version of the growing cell structure we propose here. The algorithm works by sampling randomly a target space, usually a point cloud or an implicit surface, and adjusting accordingly the neural network. The adjustment includes the connectivity of the network. Doing several experiments we found that the algorithm gives satisfactory results in some challenging situations involving sharp features and concavities. Another attractive feature of the algorithm is that its speed is virtually independent of the size of the input data, making it particularly suitable for the reconstruction of a surface from a very large point set.

A modified quadric error metric (QEM) for simplification of Loop subdivision surfaces is presented The suggested error metric not only measures the geometric difference but also controls the smoothness and well-shapedness of the triangles that result from the decimation process. Minimizing the error with respect to the original limit surface, our method allows for drastic simplification of Loop control meshes with convenient control over the reproduction of sharp features.