3D Gaussian Splatting (3DGS) enables efficient reconstruction and high-fidelity real-time rendering of complex scenes on consumer hardware. However, due to its rasterization-based formulation, 3DGS is constrained to ideal pinhole cameras and lacks support for secondary lighting effects. Recent methods address these limitations by tracing the particles instead, but, this comes at the cost of significantly slower rendering. In this work, we propose 3D Gaussian Unscented Transform (3DGUT), replacing the EWA splatting formulation with the Unscented Transform that approximates the particles through sigma points, which can be projected exactly under any nonlinear projection function. This modification enables trivial support of distorted cameras with time dependent effects such as rolling shutter, while retaining the efficiency of rasterization. Additionally, we align our rendering formulation with that of tracing-based methods, enabling secondary ray tracing required to represent phenomena such as reflections and refraction within the same 3D representation. The source code is available at: https://github.com/nv-tlabs/3dgrut.

We introduce OmniRe, a comprehensive system for efficiently creating high-fidelity digital twins of dynamic real-world scenes from on-device logs. Recent methods using neural fields or Gaussian Splatting primarily focus on vehicles, hindering a holistic framework for all dynamic foregrounds demanded by downstream applications, e.g., the simulation of human behavior. OmniRe extends beyond vehicle modeling to enable accurate, full-length reconstruction of diverse dynamic objects in urban scenes. Our approach builds scene graphs on 3DGS and constructs multiple Gaussian representations in canonical spaces that model various dynamic actors, including vehicles, pedestrians, cyclists, and others. OmniRe allows holistically reconstructing any dynamic object in the scene, enabling advanced simulations (~60 Hz) that include human-participated scenarios, such as pedestrian behavior simulation and human-vehicle interaction. This comprehensive simulation capability is unmatched by existing methods. Extensive evaluations on the Waymo dataset show that our approach outperforms prior state-of-the-art methods quantitatively and qualitatively by a large margin. We further extend our results to 5 additional popular driving datasets to demonstrate its generalizability on common urban scenes. Code and results are available at [omnire](https://ziyc.github.io/omnire/).

Particle-based representations of radiance fields such as 3D Gaussian Splatting have found great success for reconstructing and re-rendering of complex scenes. Most existing methods render particles via rasterization, projecting them to screen space tiles for processing in a sorted order. This work instead considers ray tracing the particles, building a bounding volume hierarchy and casting a ray for each pixel using high-performance GPU ray tracing hardware. To efficiently handle large numbers of semi-transparent particles, we describe a specialized rendering algorithm which encapsulates particles with bounding meshes to leverage fast ray-triangle intersections, and shades batches of intersections in depth-order. The benefits of ray tracing are well-known in computer graphics: processing incoherent rays for secondary lighting effects such as shadows and reflections, rendering from highly-distorted cameras common in robotics, stochastically sampling rays, and more. With our renderer, this flexibility comes at little cost compared to rasterization. Experiments demonstrate the speed and accuracy of our approach, as well as several applications in computer graphics and vision. We further propose related improvements to the basic Gaussian representation, including a simple use of generalized kernel functions which significantly reduces particle hit counts.

Abstract We propose a method that allows users to define flow features in form of patterns represented as sparse sets of stream line segments. Our approach finds similar occurrences in the same or other time steps. Related approaches define patterns using dense, local stencils or support only single segments. Our patterns are defined sparsely and can have a significant extent, i.e., they are integration‐based and not local. This allows for a greater flexibility in defining features of interest. Similarity is measured using intrinsic curve properties only, which enables invariance to location, orientation, and scale. Our method starts with splitting stream lines using globally consistent segmentation criteria. It strives to maintain the visually apparent features of the flow as a collection of stream line segments. Most importantly, it provides similar segmentations for similar flow structures. For user‐defined patterns of curve segments, our algorithm finds similar ones that are invariant to similarity transformations. We showcase the utility of our method using different 2D and 3D flow fields.

Abstract In flow visualization, integral surfaces rapidly tend to expand, fold and produce vast amounts of occlusion. While silhouette enhancements and local transparency mappings proved useful for semi‐transparent depictions, they still introduce visual clutter when surfaces grow more complex. An effective visualization of the flow requires a balance between the presentation of interesting surface parts and the avoidance of occlusions that hinder the view. In this paper, we extend the concept of opacity optimization to surfaces to obtain a global approach to the occlusion problem. Starting with a partition of the surfaces into patches, we compute per‐patch opacity as minimizer of a bounded‐variable least‐squares problem. For the final rendering, opacity is interpolated on the surfaces. The resulting visualization technique is interactive, frame‐coherent, view‐dependent and driven by domain knowledge.

Abstract Stream surfaces are a well‐studied and widely used tool for the visualization of 3D flow fields. Usually, stream surface seeding is carried out manually in time‐consuming trial and error procedures. Only recently automatic selection methods were proposed. Local methods support the selection of a set of stream surfaces, but, contrary to global selection methods, they evaluate only thequalityof the seeding lines but not the quality of the whole stream surfaces. Global methods, on the other hand, only support the selection of asingleoptimal stream surface until now. However, for certain flow fields a single stream surface is not sufficient to represent all flow features. In our work, we overcome this limitation by introducing a global selection technique for asetof stream surfaces. All selected surfaces optimize global stream surface quality measures and are guaranteed to be mutually distant, such that they can convey different flow features. Our approach is an efficient extension of the most recent global selection method for single stream surfaces. We illustrate its effectiveness on a number of analytical and simulated flow fields and analyze the quality of the results in a user study.

In this article, we study the regularization of quadratic energies that are integrated over discrete domains. This is a fairly general setting, often found in, but not limited to, geometry processing. The standard Tikhonov regularization is widely used such that, for instance, a low-pass filter enforces smoothness of the solution. This approach, however, is independent of the energy and the concrete problem, which leads to artifacts in various applications. Instead, we propose a regularization that enforces a low variation of the energy and is problem specific by construction. Essentially, this approach corresponds to minimization with respect to a different norm. Our construction is generic and can be plugged into any quadratic energy minimization, is simple to implement, and has no significant runtime overhead. We demonstrate this for a number of typical problems and discuss the expected benefits.

This paper applies Poisson-based methods to assist in interactive exploration of steady flow fields. Using data-driven deformations we obtain flow-orthogonal and flow-tangential surfaces by a flux-based optimization. Surfaces are positioned interactively and deformed in real-time according to local flow. The deformed surfaces are particularly useful for defining seed structures. We show how the same gradient-based computational framework can be applied to obtain parametrizations of flow-aligned surfaces. This way it is easy to define nontrivial seed structures for integration-based flow visualization methods. Additionally, the flow-aligned parametrizations are employed for view-independent surface-based LIC visualizations. We apply our method to a number of data sets to show the effectiveness of our deformations and parametrization-based seed extraction methods for interactive flow exploration.

Abstract Stream surfaces are well‐known and widely‐used structures for 3D flow visualization. A single surface can be sufficient to represent important global flow characteristics. Unfortunately, due to the huge space of possible stream surfaces, finding the globally most representative stream surface turns out to be a hard task that is usually performed by time‐consuming manual trial and error exploration using slight modifications of seed geometries. To assist users we propose a new stream surface selection method that acts as an automatic preprocessing step before data analysis. We measure stream surface relevance by a novel surface‐based quality measure that prefers surfaces where the flow is aligned with principal curvature directions. The problem of seed structure selection can then be reduced to the computation of simple minimal paths in a weighted graph spanning the domain. We apply a simulated annealing‐based optimization method to find smooth seed curves of globally near‐optimal stream surfaces. We illustrate the effectiveness of our method on a series of synthetic and real‐world data sets.