Abstract In this paper, we present a new approach for improving particle-based boundary handling in adaptive Smoothed Particle Hydrodynamics (SPH) fluid simulations. Traditional SPH methods often struggle to maintain accurate boundary conditions when particle sizes vary. Our method dynamically adjusts the sizes of boundary particles to align with nearby fluid particles, ensuring consistent interactions across different resolutions. We also introduce an enhanced boundary surface sampling technique that uses density gradients to initialize and optimize boundary particle positions after size adjustments. Through various experiments, our approach demonstrates improved surface detail accuracy and reduced unwanted fluid behavior near boundaries compared to non-adaptive methods, while maintaining computational efficiency. The source code and experiment setups can be accessed under https://doi.org/10.5281/zenodo.15519146 .

Learning editable high-resolution scene representations for dynamic scenes is an open problem with applications across the domains from autonomous driving to creative editing - the most successful approaches today make a trade-off between editability and supporting scene complexity: neural atlases represent dynamic scenes as two deforming image layers, foreground and background, which are editable in 2D, but break down when multiple objects occlude and interact. In contrast, scene graph models make use of annotated data such as masks and bounding boxes from autonomous-driving datasets to capture complex 3D spatial relationships, but their implicit volumetric node representations are challenging to edit view-consistently. We propose Neural Atlas Graphs (NAGs), a hybrid high-resolution scene representation, where every graph node is a view-dependent neural atlas, facilitating both 2D appearance editing and 3D ordering and positioning of scene elements. Fit at test-time, NAGs achieve state-of-the-art quantitative results on the Waymo Open Dataset - by 5 dB PSNR increase compared to existing methods - and make environmental editing possible in high resolution and visual quality - creating counterfactual driving scenarios with new backgrounds and edited vehicle appearance. We find that the method also generalizes beyond driving scenes and compares favorably - by more than 7 dB in PSNR - to recent matting and video editing baselines on the DAVIS video dataset with a diverse set of human and animal-centric scenes. Project Page: https://princeton-computational-imaging.github.io/nag/

High-quality depth and normal maps play a crucial role in various image processing and computer vision applications. However, the noisy data from sensors requires meticulous pre-processing. Denoising depth and normals separately, can lead to inconsistent geometric information, neg. atively impacting the performance of applications relying on this data. While some recent studies have suggested joint denoising approaches that aim to achieve coherent geometric representations, these methods are based on simple geometric assumptions, such as piecewise planar regions, and thus suffer from lower accuracy for non-planar geometry. In this paper, we present a versatile non-planar model specifically designed to handle both planar and non-planar geometry. To achieve this, we formulate the underlying problem as a partitioning of our non-planar model parameters on weighted finite graphs. Solving this problem involves utilizing a modified version of the Cut Pursuit algorithm, which efficiently divides input data into regions of similar geometric models. Finally, we leverage these resulting partitions and their associated model parameters to compute a denoised and infilled depth image, along with its coherent normal map.

Computational imaging concepts based on integrated edge AI and neural sensor concepts solve vision problems in an end-to-end, task-specific manner, by jointly optimizing the algorithmic and hardware parameters to sense data with high information value. They yield energy, data, and privacy efficient solutions, but rely on novel hardware concepts, yet to be scaled up. In this work, we present the first truly end-to-end trained imaging pipeline that optimizes imaging sensor parameters, available in standard CMOS design methods, jointly with the parameters of a given neural network on a specific task. Specifically, we derive an analytic, differentiable approach for the sensor layout parameterization that allows for task-specific, locally varying pixel resolutions. We present two pixel layout parameterization functions: rectangular and curvilinear grid shapes that retain a regular topology. We provide a drop-in module that approximates sensor simulation given existing high-resolution images to directly connect our method with existing deep learning models. We show for two different downstream tasks, classification and semantic segmentation, that network predictions benefit from learnable pixel layouts.

Implicit representations allow to use a parametric function that maps (spatial) coordinates to the value that is traditionally stored in each pixel, e.g. RGB values, instead of a discrete grid. This has recently proven quite advantageous as an internal representation for images or scenes for deep learning models. Yet, its potential to ensure certain properties of the solution has not yet been fully explored. In this work, we demonstrate that implicit representations are a powerful tool for enforcing a variety of different geometric constraints in image segmentation. While convexity, star-shape, path-connectedness, periodicity, or symmetry of the (spatial or space-time) region to be segmented are very challenging to enforce for pixel-wise discretizations, a suitable parametrization of an implicit representation, mapping spatial or spatio-temporal coordinates to the likeliness of a pixel belonging to the fore- or background, allows to **provably** ensure such constraints. Several numerical examples demonstrate that challenging segmentation scenarios can benefit from the inclusion of application-specific constraints, e.g. when occlusions prevent a faithful segmentation with classical approaches.

Abstract Particle-based boundary representations are frequently used in smoothed particle hydrodynamics (SPH) due to their simple integration into fluid solvers. Commonly, incompressible fluid solvers estimate the current density and corresponding forces in case the current density exceeds the rest density to push fluid particles apart. Close to the boundary, the calculation of the fluid particles’ density involves both neighboring fluid and neighboring boundary particles, yielding an overestimation of density, and, subsequently, wrong pressure forces and wrong velocities leading to the disturbed fluid particles’ behavior in the vicinity of the boundary. In this paper, we present a detailed explanation of this disturbed fluid particle behavior, which is mainly due to the combined or coupled handling of the fluid–fluid particle and the fluid–boundary particle interaction. We propose the decoupled handling of both interaction types, leading to two densities for a given fluid particle, i.e., fluid-induced density and boundary-induced density. In our approach, we alternately apply the corresponding fluid-induced and boundary-induced forces during pressure estimation. This separation avoids force overestimation and reduces unintended fluid dynamics near the boundary, as well as a inconsistent fluid–boundary distance across different fluid amounts and different particle-based boundary handling methods. We compare our method with two regular state-of-the-art methods in different experiments and show how our method handles detailed boundary shapes.

Abstract Metaphoric glyphs are intuitively related to the underlying problem domain and enhance the readability and learnability of the resulting visualization. Their construction, however, implies an appropriate modification of the base icon, which is a predominantly manual process. In this paper, we introduce theparametric contour-based modification (PACEMOD)approach that lays the foundations of automated, controllable icon manipulations. Technically, the PACEMOD parametric representation utilizes diffusion curves, enriched with new degrees of freedom in arc-length parameterization, which allows for manipulation of the icon contours’ geometry and the related color attributes. Moreover, we propose an implementation of our generic approach for a specific, automated design of metaphoric glyphs, based on periodic, wave-like contour modifications. Finally, the practicality of such periodic contour modifications is demonstrated by two visualization examples, which comprise uncertainty visualization of a rain forecast and gradient glyphs applied to COVID-19 data. In summary, with the PACEMOD approach we introduce an instrument that facilitates a user-centered design of metaphoric glyphs and provides a generic basis for potential further implementations according to specific applications.

Abstract Typical point-based rendering algorithms cannot directly handle large outliers or high amounts of noise without either a costly pre-processing of the point cloud or using multiple render passes. In this paper, we propose an A-buffer-based approach that directly renders unprocessed point clouds and filters out outliers in real time, without introducing additional render passes. The core concept of our approach uses bins along each pixel ray in which intermediate information is accumulated. To improve storage efficiency, we extend the binned A-buffer approach using per-ray bin hashing. Our method significantly improves visual quality when rendering noisy point clouds with varying noise levels and large outliers, while only requiring little performance and memory overhead compared to traditional point rendering methods.

Abstract In this paper, we propose an accurate and controllable rendering process for implicit surfaces with no or unknown analytic Lipschitz constants. Our process is built upon a ray-casting approach where we construct an adaptive Chebyshev proxy along each ray to perform an accurate intersection test via a robust and multi-stage searching method. By taking into account approximation errors and numerical conditions, our methods comprise several pre-conditioning and post-processing stages to improve the numerical accuracy, which potentially applied recursively. The intersection search is performed by evaluating a QR decomposition on the Chebyshev proxy function, which can be done in a numerically accurate way. Our process achieves comparable accuracy to other techniques that impose more constraints on the surface, e.g., knowledge of Lipschitz constants, and higher accuracy compared to approaches that impose similar constraints as our approach.

Abstract Boundary handling is an important aspect of fluid simulation, and several boundary handling approaches exist in smoothed particle hydrodynamics (SPH), which have individual strengths and weaknesses. However, comparing different boundary handling approaches is challenging as there is no common basis for evaluations, that is, no universal set of experiments with quantitative evaluation across different methods, especially within computer animation where many evaluations rely mainly on visual perception. This article proposes a set of experiments to aid the evaluation of the main categories of fluid‐boundary interactions that are important in computer animation, that is, no motion (resting) fluid, tangential and normal motion of a fluid with respect to the boundary, and a fluid impacting a corner. We propose ten experiments, comprising experimental setup and quantitative evaluation with optional visual inspections, that are arranged in four groups which focus on one of the main category of fluid‐boundary interactions. We use these experiments to evaluate three particle‐based boundary handling methods, that is, pressure mirroring, pressure boundaries, and moving least squares pressure extrapolation, in combination with two incompressible SPH fluid simulation methods, namely IISPH and DFSPH, to establish a quantifiable relation between different combinations of boundary handling with simulation approaches and the main categories of fluid‐boundary interactions. Finally, we summarize all results in a rating table and show how our experiments can be used to determine the promising method for specific requirements regarding a given constellation of fluid‐boundary interaction.

Partitioning of images is a fundamental image processing task, which is closely related to various problems and applications in computer vision. Due to the hard nature of the underlying problem, existing algorithms are very compute intense. In this work we present a stochastic algorithm to efficiently approximate solutions of the image partitioning problem. Our contributions lie in the novel convex reformulation of the underlying graph cut problem, along with the application of a stochastic solver. These changes allow a faster convergence compared to other graph cut based methods, which is confirmed by our experiments.

In this paper, we propose a deep optimization prior approach with application to the estimation of material-related model parameters from terahertz (THz) data that is acquired using a Frequency Modulated Continuous Wave (FMCW) THz scanning system. A stable estimation of the THz model parameters for low SNR and shot noise configurations is essential to achieve acquisition times required for applications in, e.g., quality control. Conceptually, our deep optimization prior approach estimates the desired THz model parameters by optimizing for the weights of a neural network. While such a technique was shown to improve the reconstruction quality for convex objectives in the seminal work of Ulyanov et al., our paper demonstrates that deep priors also allow to find better local optima in the non-convex energy landscape of the nonlinear inverse problem arising from THz imaging. We verify this claim numerically on various THz parameter estimation problems for synthetic and real data under low SNR and shot noise conditions. While the low SNR scenario not even requires regularization, the impact of shot noise is significantly reduced by total variation (TV) regularization. We compare our approach with existing optimization techniques that require sophisticated physically motivated initialization, and with a 1D single-pixel reparametrization method.

Recently deep generative models have achieved impressive progress in modeling the distribution of training data. In this work, we present for the first time a generative model for 4D light field patches using variational autoencoders to capture the data distribution of light field patches. We develop a generative model conditioned on the central view of the light field and incorporate this as a prior in an energy minimization framework to address diverse light field reconstruction tasks. While pure learning-based approaches do achieve excellent results on each instance of such a problem, their applicability is limited to the specific observation model they have been trained on. On the contrary, our trained light field generative model can be incorporated as a prior into any model-based optimization approach and therefore extend to diverse reconstruction tasks including light field view synthesis, spatial-angular super resolution and reconstruction from coded projections. Our proposed method demonstrates good reconstruction, with performance approaching end-to-end trained networks, while outperforming traditional model-based approaches on both synthetic and real scenes. Furthermore, we show that our approach enables reliable light field recovery despite distortions in the input.

Abstract This paper presents a novel technique for progressive online integration of uncalibrated image sequences with substantial geometric and/or photometric discrepancies into a single, geometrically and photometrically consistent image. Our approach can handle large sets of images, acquired from a nearly planar or infinitely distant scene at different resolutions in object domain and under variable local or global illumination conditions. It allows for efficient user guidance as its progressive nature provides a valid and consistent reconstruction at any moment during the online refinement process. Our approach avoids global optimization techniques, as commonly used in the field of image refinement, and progressively incorporates new imagery into a dynamically extendable and memory‐efficient Laplacian pyramid. Our image registration process includes a coarse homography and a local refinement stage using optical flow. Photometric consistency is achieved by retaining the photometric intensities given in a reference image, while it is being refined. Globally blurred imagery and local geometric inconsistencies due to, e.g. motion are detected and removed prior to image fusion. We demonstrate the quality and robustness of our approach using several image and video sequences, including handheld acquisition with mobile phones and zooming sequences with consumer cameras.

Abstract In this paper, we present a novel hash map‐based sparse data structure for Smoothed Particle Hydrodynamics, which allows for efficient neighbourhood queries in spatially adaptive simulations as well as direct ray tracing of fluid surfaces. Neighbourhood queries for adaptive simulations are improved by using multiple independent data structures utilizing the same underlying self‐similar particle ordering, to significantly reduce non‐neighbourhood particle accesses. Direct ray tracing is performed using an auxiliary data structure, with constant memory consumption, which allows for efficient traversal of the hash map‐based data structure as well as efficient intersection tests. Overall, our proposed method significantly improves the performance of spatially adaptive fluid simulations and allows for direct ray tracing of the fluid surface with little memory overhead.

In this paper, we present a novel semi-analytical boundary handling method for spatially adaptive and divergence-free smoothed particle hydrodynamics (SPH) simulations, including two-way coupling. Our method is consistent under varying particle resolutions and allows for the treatment of boundary features below the particle resolution. We achieve this by first introducing an analytic solution to the interaction of SPH particles with planar boundaries, in 2D and 3D, which we extend to arbitrary boundary geometries using signed distance fields (SDF) to construct locally planar boundaries. Using this boundary-integral-based approach, we can directly evaluate boundary contributions, for any quantity, allowing an easy integration into state of the art simulation methods. Overall, our method improves interactions with small boundary features, readily handles spatially adaptive fluids, preserves particle-boundary interactions across varying resolutions, can directly be implemented in existing SPH methods, and, for non-adaptive simulations, provides a reduction in memory consumption as well as an up to 2× speedup relative to current particle-based boundary handling approaches.

Recent advances in photographic sensing technologies have made it possible to achieve light detection in terms of a single photon. Photon counting sensors are being increasingly used in many diverse applications. We address the problem of jointly recovering spatial and temporal scene radiance from very few photon counts. Our ConvNet-based scheme effectively combines spatial and temporal information present in measurements to reduce noise. We demonstrate that using our method one can acquire videos at a high frame rate and still achieve good quality signal-to-noise ratio. Experiments show that the proposed scheme performs quite well in different challenging scenarios while the existing approaches are unable to handle them.

In this paper, we introduce a novel variational approach to estimate the scene flow from RGB-D images. We regularize the ill-conditioned problem of scene flow estimation in a unified framework by enforcing piecewise rigid motion through decomposition into rotational and translational motion parts. Our model crucially regularizes these components by an L0“norm”, thereby facilitating implicit motion segmentation in a joint energy minimization problem. Yet, we also show that this energy can be efficiently minimized by a proximal primal-dual algorithm. By implementing this approximate L0rigid motion regularization, our scene flow estimation approach implicitly segments the observed scene of into regions of nearly constant rigid motion. We evaluate our joint scene flow and segmentation estimation approach on a variety of test scenarios, with and without ground truth data, and demonstrate that we outperform current scene flow techniques.

We propose a novel method for creating high-resolution class activation maps from a given deep convolutional neural network which was trained for image classification. The resulting class activation maps not only provide information about the localization of the main objects and their instances in the image, but are also accurate enough to predict their shapes. Rather than pursuing a weakly supervised learning strategy, the proposed algorithm is a multiscale extension of the classical class activation maps using a principal component analysis of the classification network feature maps, guided filtering, and a conditional random field. Nevertheless, the resulting shape information is competitive with state-of-the-art weakly supervised segmentation methods on datasets on which the latter have been trained, while being significantly better at generalizing to other datasets and unknown classes.

Abstract The advent of affordable consumer grade RGB‐D cameras has brought about a profound advancement of visual scene reconstruction methods. Both computer graphics and computer vision researchers spend significant effort to develop entirely new algorithms to capture comprehensive shape models of static and dynamic scenes with RGB‐D cameras. This led to significant advances of the state of the art along several dimensions. Some methods achieve very high reconstruction detail, despite limited sensor resolution. Others even achieve real‐time performance, yet possibly at lower quality. New concepts were developed to capture scenes at larger spatial and temporal extent. Other recent algorithms flank shape reconstruction with concurrent material and lighting estimation, even in general scenes and unconstrained conditions. In this state‐of‐the‐art report, we analyze these recent developments in RGB‐D scene reconstruction in detail and review essential related work. We explain, compare, and critically analyze the common underlying algorithmic concepts that enabled these recent advancements. Furthermore, we show how algorithms are designed to best exploit the benefits of RGB‐D data while suppressing their often non‐trivial data distortions. In addition, this report identifies and discusses important open research questions and suggests relevant directions for future work.

Abstract 2D Confocal Raman Microscopy (CRM) data consist of high dimensional per‐pixel spectral data of 1000 bands and allows for complex spectral and spatial‐spectral analysis tasks, i.e., in material discrimination, material thickness, and spatial material distributions. Currently, simple integral methods are commonly applied as visual analysis solutions to CRM data which exhibit restricted discrimination power in various regards. In this paper we present a novel approach for the visual analysis of 2D multispectral CRM data using multi‐variate visualization techniques. Due to the large amount of data and the demand of an explorative approach without a‐priori restriction, our system allows for arbitrary interactive (de)selection of varaibles w/o limitation and an unrestricted online definition/construction of new, combined properties. Our approach integrates CRM specific quantitative measures and handles material‐related features for mixed materials in a quantitative manner. Technically, we realize the online definition/construction of new, combined properties as semi‐automatic, cascaded, 1D and 2D multidimensional transfer functions (MD‐TFs). By interactively incorporating new (raw or derived) properties, the dimensionality of the MD‐TF space grows during the exploration procedure and is virtually unlimited. The final visualization is achieved by an enhanced color mixing step which improves saturation and contrast.

Interactive real-time scene acquisition from hand-held depth cameras has recently developed much momentum, enabling applications in ad-hoc object acquisition, augmented reality and other fields. A key challenge to online reconstruction remains error accumulation in the reconstructed camera trajectory, due to drift-inducing instabilities in the range scan alignments of the underlying iterative-closest-point (ICP) algorithm. Various strategies have been proposed to mitigate that drift, including SIFT-based pre-alignment, color-based weighting of ICP pairs, stronger weighting of edge features, and so on. In our work, we focus on surface curvature as a feature that is detectable on range scans alone and hence does not depend on accurate multi-sensor alignment. In contrast to previous work that took curvature into consideration, however, we treat curvature as an independent quantity that we consistently incorporate into every stage of the real-time reconstruction pipeline, including densely curvature-weighted ICP, range image fusion, local surface reconstruction, and rendering. Using multiple benchmark sequences, and in direct comparison to other state-of-the-art online acquisition systems, we show that our approach significantly reduces drift, both when analyzing individual pipeline stages in isolation, as well as seen across the online reconstruction pipeline as a whole.

In this paper we introduce a novel method to adaptive incompressible SPH simulations. Instead of using a scheme with a number of fixed particle sizes or levels, our approach allows continuous particle sizes. This enables us to define optimal particle masses with respect to, e.g., the distance to the fluid's surface. A required change in mass due to the dynamics of the fluid is properly and stably handled by our scheme of mass redistribution. This includes temporally smooth changes in particle masses as well as sudden mass variations in regions of high flow dynamics. Our approach guarantees low spatial variations in particle size, which is a core property in order to achieve large adaptivity ratios for incompressible fluid simulations. Conceptually, our approach allows for infinite continuous adaptivity, practically we achieved adaptivity ratios up to 5 orders of magnitude, while still being mass preserving and numerically stable, yielding unprecedented vivid surface detail at comparably low computational cost and moderate particle counts.

This work proposes a textual and graphical domain-specific language (DSL) designed especially for modeling and writing data and image processing algorithms. Since reusing algorithms and other functionality leads to higher program quality and mostly shorter development time, this approach introduces a novel component-based language design. Special diagrams and structures, such as components, component-diagrams and component-instance-diagrams are introduced. The new language constructs allow an abstract and object-oriented description of data and image processing tasks. Additionally, a compatible graphical design interface is proposed, giving modelers and architects the opportunity to decide which kind of modeling they prefer (graphical or textual, including round-trip engineering).

Continuous-wave time-of-flight (ToF) cameras show great promise as low-cost depth image sensors in mobile applications. However, they also suffer from several challenges, including limited illumination intensity, which mandates the use of large numerical aperture lenses, and thus results in a shallow depth of field, making it difficult to capture scenes with large variations in depth. Another shortcoming is the limited spatial resolution of currently available ToF sensors. In this paper we analyze the image formation model for blurred ToF images. By directly working with raw sensor measurements but regularizing the recovered depth and amplitude images, we are able to simultaneously deblur and super-resolve the output of ToF cameras. Our method outperforms existing methods on both synthetic and real datasets. In the future our algorithm should extend easily to cameras that do not follow the cosine model of continuous-wave sensors, as well as to multi-frequency or multi-phase imaging employed in more recent ToF cameras.

In this paper, we provide a new, real photo dataset with precise ground-truth for intrinsic image research. Prior ground-truth datasets have been restricted to rather simple illumination conditions and scene geometries, or have been enhanced using image synthesis methods. The dataset provided in this paper is based on complex multi-illuminant scenarios under multi-colored illumination conditions and challenging cast shadows. We provide full per-pixel intrinsic ground-truth data for these scenarios, i.e. reflectance, specularity, shading, and illumination for scenes as well as preliminary depth information. Furthermore, we evaluate 3 state-of-the-art intrinsic image recovery methods, using our dataset.

Abstract We propose a framework for unified visualization of advective and diffusive concentration fluxes, which play a key role in many phenomena like, e.g. Marangoni convection and microscopic mixing. The main idea is the decomposition of fluxes into their concentration and velocity parts. Using this flux decomposition, we are able to convey advective‐diffusive concentration transport using integral lines. In order to visualize superimposed flux effects, we introduce a new graphical metaphor, the stream feather, which adds extensions to stream tubes pointing in the directions of deviating fluxes. The resulting unified visualization of macroscopic advection and microscopic diffusion allows for deeper insight into complex flow scenarios that cannot be achieved with current volume and surface rendering techniques alone. Our approach for flux decomposition and visualization of advective‐diffusive flows can be applied to any kind of (simulation) data if velocity and concentration data are available. We demonstrate that our techniques can easily be integrated into Smoothed Particle Hydrodynamics (SPH) based simulations.

Abstract The diagnosis of certain spine pathologies, such as scoliosis, spondylolisthesis and vertebral fractures, is part of the daily clinical routine. Very frequently, magnetic resonance image data are used to diagnose these kinds of pathologies in order to avoid exposing patients to harmful radiation, like X‐ray. We present a method which detects and segments all acquired vertebral bodies, with minimal user intervention. This allows an automatic diagnosis to detect scoliosis, spondylolisthesis and crushed vertebrae. Our approach consists of three major steps. First, vertebral centres are detected using a Viola–Jones like method, and then the vertebrae are segmented in a parallel manner, and finally, geometric diagnostic features are deduced in order to diagnose the three diseases. Our method was evaluated on 26 lumbar datasets containing 234 reference vertebrae. Vertebra detection has 7.1% false negatives and 1.3% false positives. The average Dice coefficient to manual reference is 79.3% and mean distance error is 1.76 mm. No severe case of the three illnesses was missed, and false alarms occurred rarely—0% for scoliosis, 3.9% for spondylolisthesis and 2.6% for vertebral fractures. The main advantages of our method are high speed, robust handling of a large variety of routine clinical images, and simple and minimal user interaction.

Real-time or online 3D reconstruction has wide applicability and receives further interest due to availability of consumer depth cameras. Typical approaches use a moving sensor to accumulate depth measurements into a single model which is continuously refined. Designing such systems is an intricate balance between reconstruction quality, speed, spatial scale, and scene assumptions. Existing online methods either trade scale to achieve higher quality reconstructions of small objects/scenes. Or handle larger scenes by trading real-time performance and/or quality, or by limiting the bounds of the active reconstruction. Additionally, many systems assume a static scene, and cannot robustly handle scene motion or reconstructions that evolve to reflect scene changes. We address these limitations with a new system for real-time dense reconstruction with equivalent quality to existing online methods, but with support for additional spatial scale and robustness in dynamic scenes. Our system is designed around a simple and flat point-Based representation, which directly works with the input acquired from range/depth sensors, without the overhead of converting between representations. The use of points enables speed and memory efficiency, directly leveraging the standard graphics pipeline for all central operations, i.e., camera pose estimation, data association, outlier removal, fusion of depth maps into a single denoised model, and detection and update of dynamic objects. We conclude with qualitative and quantitative results that highlight robust tracking and high quality reconstructions of a diverse set of scenes at varying scales.

Modern imaging optics are highly complex systems consisting of up to two dozen individual optical elements. This complexity is required in order to compensate for the geometric and chromatic aberrations of a single lens, including geometric distortion, field curvature, wavelength-dependent blur, and color fringing. In this article, we propose a set of computational photography techniques that remove these artifacts, and thus allow for postcapture correction of images captured through uncompensated, simple optics which are lighter and significantly less expensive. Specifically, we estimate per-channel, spatially varying point spread functions, and perform nonblind deconvolution with a novel cross-channel term that is designed to specifically eliminate color fringing.

Abstract In this paper, we introduce a fast and consistent smoothed particle hydrodynamics (SPH) technique which is suitable for convection–diffusion simulations of incompressible fluids. We apply our temporal blending technique to reduce the number of particles in the simulation while smoothly changing quantity fields. Our approach greatly reduces the error introduced in the pressure term when changing particle configurations. Compared to other methods, this enables larger integration time‐steps in the transition phase. Our implementation is fully GPU‐based to take advantage of the parallel nature of particle simulations.

The processing power of parallel coprocessors like the Graphics Processing Unit (GPU) is dramatically increasing. However, until now only a few approaches have been presented to utilize this kind of hardware for mesh clustering purposes. In this paper, we introduce a Multilevel clustering technique designed as a parallel algorithm and solely implemented on the GPU. Our formulation uses the spatial coherence present in the cluster optimization and hierarchical cluster merging to significantly reduce the number of comparisons in both parts. Our approach provides a fast, high-quality, and complete clustering analysis. Furthermore, based on the original concept, we present a generalization of the method to data clustering. All advantages of the mesh-based techniques smoothly carry over to the generalized clustering approach. Additionally, this approach solves the problem of the missing topological information inherent to general data clustering and leads to a Local Neighbors k-means algorithm. We evaluate both techniques by applying them to Centroidal Voronoi Diagram (CVD)-based clustering. Compared to classical approaches, our techniques generate results with at least the same clustering quality. Our technique proves to scale very well, currently being limited only by the available amount of graphics memory.

Interactive visualization systems allow extensive adjustments of the visualization process, to give the user full control over the visualization result. However, this flexibility increases the complexity of the user interface, impeding the task of finding a suitable set of visualization parameters for a given problem. Visual assistance tools can guide the user in this process, thereby helping to manage the complexity. In this paper, we show how two of such tools, lenses and detectors, can be applied to interactive visualization of remote sensing data.

Because bin-picking effectively mirrors great challenges in robotics, it has been a relevant robotic showpiece application for several decades. In this paper we describe the computer vision algorithms in combination with the sophisticated control schemes of the robot and demonstrate a reliable and robust solution to the chosen problem. This paper approaches the bin-picking issue by applying the latest state-of-the-art hardware components, namely an impedance controlled lightweight robot and a Time-of-Flight camera. Lightweight robots have gained new capabilities in both sensing and actuation without suffering a decrease in speed and payload. Time-of- Flight cameras are superior to common proximity sensors in the sense that they provide depth and intensity images in video frame rate independent of textures. The bin-picking solution presented in this paper aims at extending the classical bin-picking problem by incorporating an environment model and allowing for the physical human-robot interaction during the entire process. Existing imprecisions in Time-of-Flight camera measurements and environment uncertainties are compensated by the compliant behavior of the robot. The overall process is implemented in a generic state machine that also monitors the entire bin-picking process.

Abstract A growing number of applications depend on accurate and fast 3D scene analysis. Examples are model and lightfield acquisition, collision prevention, mixed reality and gesture recognition. The estimation of a range map by image analysis or laser scan techniques is still a time‐consuming and expensive part of such systems. A lower‐priced, fast and robust alternative for distance measurements are time‐of‐flight (ToF) cameras. Recently, significant advances have been made in producing low‐cost and compact ToF devices, which have the potential to revolutionize many fields of research, including computer graphics, computer vision and human machine interaction (HMI). These technologies are starting to have an impact on research and commercial applications. The upcoming generation of ToF sensors, however, will be even more powerful and will have the potential to become ‘ubiquitous real‐time geometry devices’ for gaming, web‐conferencing, and numerous other applications. This paper gives an account of recent developments in ToF technology and discusses the current state of the integration of this technology into various graphics‐related applications.

Large software systems typically exist in many revisions. In order to analyze such systems, their evolution needs to be analyzed, too. The main challenge in this context is to cope with the large volume of data and to visualize the system and its evolution as a whole. This paper presents an approach for visualizing the evolution of large-scale systems which uses three different, tightly integrated visualizations that are 3-dimensional and/or animated and which support different analysis tasks. According to a first empirical study, all tasks are supported well by at least one visualization.

Interactive visualization of remote sensing data allows the user to explore the full scope of the data sets. Combining and comparing different modalities can give additional insight. In this paper, we present a 3D visualization framework for interactive exploration of remote sensing data. Data from different modalities can be combined into a single view. The visualization can be distributed across multiple graphics processing units and/or hosts, allowing interactive exploration of remote sensing data in virtual reality systems.

We present a new VR installation at the University of Siegen, Germany. It consists of a 180deg cylindrical rear-projection screen and a front-projection floor, allowing both immersive VR applications with user tracking and convincing presentations for a larger audience.

Abstract The quest for the ideal flow visualization reveals two major challenges: interactivity and accuracy. Interactivity stands for explorative capabilities and real‐time control. Accuracy is a prerequisite for every professional visualization in order to provide a reliable base for analysis of a data set. Geometric flow visualization has a long tradition and comes in very different flavors. Among these, stream, path and streak lines are known to be very useful for both 2D and 3D flows. Despite their importance in practice, appropriate algorithms suited for contemporary hardware are rare. In particular, the adaptive construction of the different line types is not sufficiently studied. This study provides a profound representation and discussion of stream, path and streak lines. Two algorithms are proposed for efficiently and accurately generating these lines using modern graphics hardware. Each includes a scheme for adaptive time‐stepping. The adaptivity for stream and path lines is achieved through a new processing idea we call ‘selective transform feedback’. The adaptivity for streak lines combines adaptive time‐stepping and a geometric refinement of the curve itself. Our visualization is applied, among others, to a data set representing a simulated typhoon. The storage as a set of 3D textures requires special attention. Both algorithms explicitly support this storage, as well as the use of precomputed adaptivity information.

Point target analysis is an important tool to analyze the quality of SAR images. To permit interactive visual analysis, visualization applications need to automatically detect point targets in a SAR image and estimate associated quality measurements such as the peak sidelobe ratio (PSLR). This task is computationally expensive. In this paper, we propose methods for automatic point target detection that work on hierarchical data structures and process the image data on the graphics processing unit (GPU) to allow interactive use. For each detected point target in the currently visualized area of the image, the visualization application can then display color-coded quality measurements, thus providing the user with an overview of the point targets in the scene as well as an immediate impression of the SAR image quality. Detailed point target analysis results can be displayed on demand.

In this paper a novel clustering algorithm is proposed, namely Variational Multilevel Mesh Clustering (VMLC). The algorithm incorporates the advantages of both hierarchical and variational (Lloyd) algorithms, i.e. the initial number of seeds is not predefined and on each level the obtained clustering configuration is quasi-optimal. The algorithm performs a complete mesh analysis regarding the underlying energy functional. Thus, an optimized multilevel clustering is built. The first benefit of this approach is that it resolves the inherent problems of variational algorithms, for which the result and the convergence is strictly related to the initial number and selection of seeds. On the other hand, the greedy nature of hierarchical approaches, i.e. the non-optimal shape of the clusters in the hierarchy, is solved. We present an optimized implementation based on an incremental data structure. We demonstrate the generic nature of our approach by applying it for the generation of optimized multilevel Centroidal Voronoi Diagrams and planar mesh approximation.

Abstract Typically, flow volumes are visualized by defining their boundary as iso‐surface of a level set function. Grid‐based level sets offer a good global representation but suffer from numerical diffusion of surface detail, whereas particle‐based methods preserve details more accurately but introduce the problem of unequal global representation. The particle level set (PLS) method combines the advantages of both approaches by interchanging the information between the grid and the particles. Our work demonstrates that the PLS technique can be adapted to volumetric dye advection via streak volumes, and to the visualization by time surfaces and path volumes. We achieve this with a modified and extended PLS, including a model for dye injection. A new algorithmic interpretation of PLS is introduced to exploit the efficiency of the GPU, leading to interactive visualization. Finally, we demonstrate the high quality and usefulness of PLS flow visualization by providing quantitative results on volume preservation and by discussing typical applications of 3D flow visualization.

Abstract The paper describes a technique to generate high‐quality light field representations from volumetric data. We show how light field galleries can be created to give unexperienced audiences access to interactive high‐quality volume renditions. The proposed light field representation is lightweight with respect to storage and bandwidth capacity and is thus ideal as exchange format for visualization results, especially for web galleries. The approach expands an existing sphere‐hemisphere parameterization for the light field with per‐pixel depth. High‐quality paraboloid maps from volumetric data are generated using GPU‐based ray‐casting or slicing approaches. Different layers, such as isosurfaces, but not restricted to, can be generated independently and composited in real time. This allows the user to interactively explore the model and to change visibility parameters at run‐time.

Abstract Image‐based rendering techniques are a powerful alternative to traditional polygon‐based computer graphics. This paper presents a novel light field rendering technique which performs per‐pixel depth correction of rays for high‐quality reconstruction. Our technique stores combined RGB and depth values in a parabolic 2D texture for every light field sample acquired at discrete positions on a uniform spherical setup. Image synthesis is implemented on the GPU as a fragment program which extracts the correct image information from adjacent cameras for each fragment by applying per‐pixel depth correction of rays. We show that the presented image‐based rendering technique provides a significant improvement compared to previous approaches. We explain two different rendering implementations which make use of a uniform parametrisation to minimise disparity problems and ensure full six degrees of freedom for virtual view synthesis. While one rendering algorithm implements an iterative refinement approach for rendering light fields with per pixel depth correction, the other approach employs a raycaster, which provides superior rendering quality at moderate frame rates. GPU based per‐fragment depth correction of rays, used in both implementations, helps reducing ghosting artifacts to a non‐noticeable amount and provides a rendering technique that performs without exhaustive pre‐processing for 3D object reconstruction and without real‐time ray‐object intersection calculations at rendering time.

The visualization of synthetic aperture radar (SAR) data involves the mapping from high dynamic range amplitude values to gray values of a lower dynamic range display device. This dynamic range reduction process controls the amount of information in the displayed result and is therefore an important part of each SAR visualization system. Interactive systems provide the user with immediate feedback on the changes of the reduction method and its parameters. In this letter, we examine different dynamic range reduction techniques known from the tone mapping of optical images. The techniques are analyzed, regarding their applicability to SAR data, and incorporated into our interactive visualization framework based on programmable graphics hardware.

Synthetic aperture radar data presents specific problems for interactive visualization. The high amount of multiplicative speckle noise has to be reduced. The high dynamic range of the amplitude data must be mapped to the lower dynamic range of display devices in a way that makes image features appropriately visible. In addition to interactive navigation in the data, it is desirable to allow interactive selection of despeckling and dynamic range reduction methods and adjustment of their parameters. Graphics processing units (GPUs) can be seen as ubiquitous parallel coprocessors with extreme computational power. In this paper, we propose a GPU-based framework for interactive visualization of SAR data. Data management techniques are used to make full use of the GPU. We reworked well-known despeckling and dynamic range reduction techniques for the GPU programming model and implemented them in our framework. Both navigation in large data sets and adjustment of processing parameters are fully interactive.

Following the goals of our cooperation treaty between FGAN and ZESS (University Siegen), we work closely together on the complex research field of bistatic exploration. Single tasks of the overall topic are for instance experimental missions, processing, image formation, position- and attitude estimation, synchronisation, simulation, parameter estimation, and visualization. This paper presents an overview about the common projects of FGAN, ZESS, and FOMAAS.

Abstract The most important technique to visualize 3D scalar data, as they arise e.g. in medicine from tomographic measurement, is direct volume rendering. A transfer function maps the scalar values to optical properties which are used to solve the integral of light transport in participating media. Many medical data sets, especially MRI data, however, are difficult to visualize due to different tissue types being represented by the same scalar value. The main problem is that interesting structures will be occluded by less important structures because they share the same range of data values. Occlusion, however, is a view‐dependent problem and cannot be solved easily by transfer function design. This paper proposes a new method to display different entities inside the volume data in a single rendering pass. The proposed opacity peeling technique reveals structures in the data set that cannot be visualized directly by one‐or multi‐dimensional transfer functions without explicit segmentation. We also demonstrate real‐time implementations using texture mapping and multiple render targets. Categories and Subject Descriptors (according to ACM CCS): I.3.7 [Computer Graphics]: Three‐Dimensional Graphics and Realism

Several techniques have been developed to approximate Bidirectional Reflectance Distribution Functions (BRDF) with acceptable quality and performance for realtime applications. The recently published Homomorphic Factorization by McCool et al. is a general approximation approach that can be used with various setups and for different quality requirements.In this paper we propose a new technique based on the Homomorphic Factorization. Instead of approximating the BRDF, our technique factorizes the full lighting computation of an isotropic BRDF in a global illumination scenario. With this method materials in complex lighting situations can be simulated with only two textures by using commonly available computation capabilities of current graphics hardware.The new technique can also be considered as a generalized approach to several environment map prefiltering techniques. Existing prefiltering techniques are usually limited to specific BRDFs or require advanced hardware capabilities like 3D texturing. With the factorization only common 2D textures are required.

Abstract An algorithm for surface reconstruction from a polyhedron with arbitrary topology consisting of triangular faces is presented. The first variant of the algorithm constructs a curve network consisting of cubic Bézier curves meeting with tangent plane continuity at the vertices. This curve network is extended to a smooth surface by replacing each of the networks facets with a split patch consisting of three triangular Bézier patches. The remaining degrees of freedom of the curve network and the split patches are determined by minimizing a quadratic functional. This optimization process works either for the curve network and the split patches separately or in one simultaneous step. The second variant of our algorithm is based on the construction of an optimized curve network with higher continuity. Examples demonstrate the quality of the different methods.