ABSTRACT Physically based modal sound synthesis generates realistic audio by modeling structural vibration and acoustic radiation from object geometry and material properties. However, traditional approaches rely on computationally expensive eigenvalue analysis and acoustic transfer simulations, making them impractical for real‐time virtual reality (VR), especially for dynamically generated or fractured objects. We present NeuroSonic, a unified neural framework for real‐time physically based sound synthesis directly from point cloud geometry. Our key idea is to amortize modal analysis and acoustic transfer using neural networks, replacing costly numerical solvers with efficient feed‐forward inference. The framework predicts modal eigenvalues, eigenvectors, and acoustic transfer functions, forming a physically grounded representation for sound synthesis. By operating on point clouds, our method avoids voxelization artifacts and handles geometrically complex and thin structures. Once trained, NeuroSonic computes complete modal and acoustic information in under 0.02 s, enabling real‐time performance. Experiments show that our approach achieves accuracy comparable to physically based solvers while providing orders‐of‐magnitude speedup, making it a practical solution for interactive VR environments.

Prior Arbitrary-Scale Image Super-Resolution (ASISR) methods often experience a significant performance decline when the upsampling factor exceeds the range covered by the training data, introducing substantial blurring. To address this issue, we propose a unified model, Stroke-based Cyclic Amplifier (SbCA), for ultra-large upsampling tasks. The key of SbCA is the stroke vector amplifier, which decomposes the image into a series of strokes represented as vector graphics for magnification. Then, the detail completion module also restores missing details, ensuring high-fidelity image reconstruction. Our cyclic strategy achieves ultra-large upsampling by iteratively refining details with this unified SbCA model, trained only once for all, while keeping sub-scales within the training range. Our approach effectively addresses the distribution drift issue and eliminates artifacts, noise and blurring, producing high-quality, high-resolution super-resolved images. Experimental validations on both synthetic and real-world datasets demonstrate that our approach significantly outperforms existing methods in ultra-large upsampling tasks (e.g. $\times 100$ ), delivering visual quality far superior to state-of-the-art techniques.

Simulating heterogeneous multibody systems with both deformable and stiff components remains a challenge for GPU solvers. While Newton-Krylov methods are popular for their matrix-free nature and good GPU compatibility, they often suffer from severe ill-conditioning and slow convergence when handling stiff contacts and material disparities in deformable multibody systems. In this paper, we present Heterogeneous Subspace Corrections (HSC), a novel Newton-CG variant to efficiently simulate such complex systems. HSC decouples the system into two Krylov iterations: a Newton-CG procedure for deformable bodies and a Neumann-based iteration for the affine subsystem. We introduce a GPU-based Adaptive Cross Approximation (ACA) algorithm to exploit the low-rank nature of the coupling matrices, which manages to reduce the overhead of sparse matrix-vector multiplications substantially on the GPU. For the affine subsystem, we propose a dedicated data structure for fast assembling contact Hessians in parallel. A variety of experimental results demonstrate that our framework consistently outperforms existing GPU simulators, achieving convergence rates comparable to direct solvers even in scenes with high-resolution models and extensive stiff contacts.

We present a real-time pipeline to approximate global illumination in interactive or dynamic scenes, including both 3D Gaussian models and conventional meshes. Building on a formulated surface light transport model for 3D Gaussians, we address key performance challenges through a fast compound stochastic ray-tracing algorithm and a hardware 3D Gaussian rasterizer, and we implement multiple RTGI (real-time global illumination) techniques for 3D Gaussians. Our pipeline enables real-time rendering of interactive scenes featuring editable materials, lights, meshes, and 3D Gaussian models, effectively capturing multi-bounce diffuse and one-bounce glossy light transport. Our approach covers a wide range of dynamic light sources, including area lights, directional lights, and environmental lighting. Comprehensive experimental results demonstrate that our approach can efficiently render global illumination for both 3DGS models and hybrid models combining meshes and 3DGS, all in real time. Our pipeline highlights the potential of 3D Gaussians in real-time global illumination, and offers insights into performance optimization.

We propose a novel framework for Russian Roulette and Splitting (RRS) tailored to wavefront path tracing, a highly parallel rendering architecture that processes path states in batched, stage-wise execution for efficient GPU utilization. Traditional RRS methods, with unpredictable path counts, are fundamentally incompatible with wavefront's preallocated memory and scheduling requirements. To resolve this, we introduce a normalized RRS formulation with a bounded path count, enabling stable and memory-efficient execution. Furthermore, we pioneer the use of neural networks to learn RRS factors, presenting two models: NRRS and AID-NRRS. At a high level, both feature a carefully designed RRSNet that explicitly incorporates RRS normalization, with only subtle differences in their implementation. To balance computational cost and inference accuracy, we introduce Mix-Depth, a path-depth-aware mechanism that adaptively regulates neural evaluation, further improving efficiency. Extensive experiments demonstrate that our method outperforms traditional heuristics and recent RRS techniques in both rendering quality and performance across a variety of complex scenes.

With the rise of digital fashion, reusing high-quality garment assets to assemble new outfits has become increasingly important for improving design efficiency and reducing production costs. However, combining multiple garments often introduces complex inter-garment intersections that are difficult to resolve. In this paper, we propose a novel framework that introduces a midsurface representation to simplify multilayered garments for intersection-free outfit assembly. Each garment is approximated by a watertight tetrahedral enclosure, enabling efficient resolution of inter-garment collisions on the midsurface level. To assemble an outfit, our method progressively untangles pairs of single-layer midsurfaces and incrementally constructs a merged midsurface. To recover the intersection-free full geometry from these deformed midsurfaces and enable instantaneous transfer across different poses, we uses embedded anchors to drive inversion-free deformation of enclosing tetrahedral cages. Through various examples, we demonstrate that our method provides a scalable and automated solution for virtual outfit coordination, enabling the direct reuse of garment assets in high-fidelity, collision-free digital fashion workflows.

Recent research on learnable neural representations has been widely adopted in the field of 3D scene reconstruction and neural rendering applications. However, traditional feature grid representations often suffer from a substantial memory footprint, posing a significant bottleneck for modern parallel computing hardware. In this paper, we present neural vertex features, a generalized formulation of learnable representation for neural rendering tasks involving explicit mesh surfaces. Instead of uniformly distributing neural features throughout 3D space, our method stores learnable features directly at mesh vertices, leveraging the underlying geometry as a compact and structured representation for neural processing. This not only optimizes memory efficiency, but also improves feature representation by aligning compactly with the surface using task-specific geometric priors. Additionally, neural vertex features offer improved feature representation by compactly aligning with the surface using task-specific geometric priors. We validate our neural representation across diverse neural rendering tasks, with a specific emphasis on neural radiosity. Experimental results demonstrate that our method reduces memory consumption to only one-fifth (or even less) of grid-based representations, while maintaining comparable rendering quality and lowering inference overhead.

To bridge the gap between artists and non-specialists, we present a unified framework, Neural-Polyptych, to facilitate the creation of expansive, high-resolution paintings by seamlessly incorporating interactive hand-drawn sketches with fragments from original paintings. We have designed a multi-scale GAN-based architecture to decompose the generation process into two parts, each responsible for identifying global and local features. To enhance the fidelity of semantic details generated from users' sketched outlines, we introduce a Correspondence Attention module utilizing our Reference Bank strategy. This ensures the creation of high-quality, intricately detailed elements within the artwork. The final result is achieved by carefully blending these local elements while preserving coherent global consistency. Consequently, this methodology enables the production of digital paintings at megapixel scale, accommodating diverse artistic expressions and enabling users to recreate content in a controlled manner. We validate our approach to diverse genres of both Eastern and Western paintings. Applications such as large painting extension, texture shuffling, genre switching, mural art restoration, and recomposition can be successfully based on our framework.

To relieve perceived pain in patients undergoing colonoscopy, we developed an endoscopist-patient co-participative VR tool (EPC-VR) based on A Neurocognitive Model of Attention to Pain. It allows the patient to play a VR game actively and supports the endoscopist in triggering a distraction mechanism to divert the patient's attention away from the medical procedure. We performed a comparative clinical study with 40 patients. Patients' perception of pain and affective responses were evaluated, and the results support the effectiveness of EPC-VR: active VR playing with endoscopists' participation can help relieve the perceived pain and scare of patients undergoing colonoscopy. Finally, 87.5% of patients opt to use the VR application in the next colonoscopy.

Meshless simulation of elasticity is important for deformable simulation in computer graphics. While shape matching is a popular meshless solution, it is limited to a subset of elastic constitutive models, challenging the simulation of generic elastic constitutive models using meshless integration. In contrast, peridynamics offers a more versatile capacity and can describe various material behavior through non-local interactions between vertices. However, the size of each stencil Hessian matrix varies with the number of nearby integration points, leading to inefficiency and accuracy loss. To address these challenges, we present an efficient and robust solver for generic elastic models based on peridynamics. We propose an efficient first-order Hessian proxy derived from the positive-negative decomposition of the stress tensor. The proposed symmetric positive definite proxies ensure convergence within a reasonable number of iterations while also being easy to parallelize on GPU. To further enhance stability, particularly for hyperelastic models, we propose enforcing strain limiting between peridynamics bonds to prevent tensile instability in meshless integration. Our algorithm includes two iteration loops of strain limiting and elastic Jacobis, and the pipeline is well-suited for GPU implementation. We evaluated the performance of our approach with a wide range of elastic constitutive models in diverse testing scenarios against the alternative numerical solvers. Our method features superior efficiency and faster convergence compared to existing numerical solvers. These compelling results underscore the practicality and effectiveness of our method for simulating elasticity via meshless integration.

Previous acoustic transfer methods rely on extensive precomputation and storage of data to enable real-time interaction and auditory feedback. However, these methods struggle with complex scenes, especially when dynamic changes in object position, material, and size significantly alter sound effects. These continuous variations lead to fluctuating acoustic transfer distributions, making it challenging to represent with basic data structures and render efficiently in real time. To address this challenge, we present Neural Acoustic Transfer, a novel approach that leverages implicit neural representations to encode acoustic transfer functions and their variations. This enables real-time prediction of dynamically evolving sound fields and their interactions with the environment under varying conditions. To efficiently generate high-quality training data for the neural acoustic field while avoiding reliance on mesh quality of a model, we develop a fast and efficient Monte-Carlo-based boundary element method (BEM) approximation, suitable for general scenarios with smooth Neumann boundary conditions. In addition, we devise strategies to mitigate potential singularities during the synthesis of training data, thereby enhancing its reliability. Together, these methods provide robust and accurate data that empower the neural network to effectively model complex sound radiation space. We demonstrate our method's numerical accuracy and runtime efficiency (within several milliseconds for 30 s audio) through comprehensive validation and comparisons in diverse acoustic transfer scenarios. Our approach allows for efficient and accurate modeling of sound behavior in dynamically changing environments, which can benefit a wide range of interactive applications such as virtual reality, augmented reality, and advanced audio production.

Prior foveated rendering methods often suffer from a limitation where the shading load escalates with increasing display resolution, leading to decreased efficiency, particularly when dealing with retinal-level resolutions. To tackle this challenge, we begin with the essence of the human visual system (HVS) perception and present visual acuity-consistent foveated rendering (VaFR), aiming to achieve exceptional rendering performance at retinal-level resolutions. Specifically, we propose a method with a novel log-polar mapping function derived from the human visual acuity model, which accommodates the natural bandwidth of the visual system. This mapping function and its associated shading rate guarantee a consistent output of rendering information, regardless of variations in the display resolution of the VR HMD. Consequently, our VaFR outperforms alternative methods, improving rendering speed while preserving perceptual visual quality, particularly when operating at retinal resolutions. We validate our approach using both the rasterization and ray-casting rendering pipelines. We also validate our approach using different binocular rendering strategies for HMD devices. In diverse testing scenarios, our approach delivers better perceptual visual quality than prior foveated rendering while achieving an impressive speedup of 6.5×-9.29× for deferred rendering of 3D scenarios and an even more powerful speedup of 10.4×-16.4× for ray-casting at retinal resolution. Additionally, our approach significantly enhances the rendering performance of binocular 8 K path tracing, achieving smooth frame rates.

We propose a GPU-based iterative method for accelerated elastodynamic simulation with the log-barrier-based contact model. While Newton's method is a conventional choice for solving the interior-point system, the presence of ill-conditioned log barriers often necessitates a direct solution at each linearized substep and costs substantial storage and computational overhead. Moreover, constraint sets that vary in each iteration present additional challenges in algorithm convergence. Our method employs a novel barrier-augmented Lagrangian method to improve system conditioning and solver efficiency by adaptively updating an augmentation constraint sets. This enables the utilization of a scalable, inexact Newton-PCG solver with sparse GPU storage, eliminating the need for direct factorization. We further enhance PCG convergence speed with a domain-decomposed warm start strategy based on an eigenvalue spectrum approximated through our in-time assembly. Demonstrating significant scalability improvements, our method makes simulations previously impractical on 128 GB of CPU memory feasible with only 8 GB of GPU memory and orders-of-magnitude faster. Additionally, our method adeptly handles stiff problems, surpassing the capabilities of existing GPU-based interior-point methods. Our results, validated across various complex collision scenarios involving intricate geometries and large deformations, highlight the exceptional performance of our approach.

Robust light transport algorithms, particularly bidirectional path tracing (BDPT), face significant challenges when dealing with specular or highly glossy involved paths. BDPT constructs the full path by connecting sub-paths traced individually from the light source and camera. However, it remains difficult to sample by connecting vertices on specular and glossy surfaces with narrow-lobed BSDF, as it poses severe constraints on sampling in the feasible direction. To address this issue, we propose a novel approach, called proxy sampling , that enables efficient sub-path connection of these challenging paths. When a low-contribution specular/glossy connection occurs, we drop out the problematic neighboring vertex next to this specular/glossy vertex from the original path, then retrace an alternative sub-path as a proxy to complement this incomplete path. This newly constructed complete path ensures that the connection adheres to the constraint of the narrow lobe within the BSDF of the specular/glossy surface. Unbiased reciprocal estimation is the key to our method to obtain a probability density function (PDF) reciprocal to ensure unbiased rendering. We derive the reciprocal estimation method and provide an efficiency-optimized setting for efficient sampling and connection. Our method provides a robust tool for substituting problematic paths with favorable alternatives while ensuring unbiasedness. We validate this approach in the probabilistic connections BDPT for addressing specular-involved difficult paths. Experimental results have proved the effectiveness and efficiency of our approach, showcasing high-performance rendering capabilities across diverse settings.

Identifying an appropriate radius for unbiased kernel estimation is crucial for the efficiency of radiance estimation. However, determining both the radius and unbiasedness still faces big challenges. In this paper, we first propose a statistical model of photon samples and associated contributions for progressive kernel estimation, under which the kernel estimation is unbiased if the null hypothesis of this statistical model stands. Then, we present a method to decide whether to reject the null hypothesis about the statistical population (i.e., photon samples) by the F-test in the Analysis of Variance. Hereby, we implement a progressive photon mapping (PPM) algorithm, wherein the kernel radius is determined by this hypothesis test for unbiased radiance estimation. Second, we propose VCM+, a reinforcement of Vertex Connection and Merging (VCM), and derive its theoretically unbiased formulation. VCM+ combines hypothesis testing-based PPM with bidirectional path tracing (BDPT) via multiple importance sampling (MIS), wherein our kernel radius can leverage the contributions from PPM and BDPT. We test our new algorithms, improved PPM and VCM+, on diverse scenarios with different lighting settings. The experimental results demonstrate that our method can alleviate light leaks and visual blur artifacts of prior radiance estimate algorithms. We also evaluate the asymptotic performance of our approach and observe an overall improvement over the baseline in all testing scenarios.

Children diagnosed with Autism Spectrum Disorder (ASD) often exhibit motor disorders. Dance Movement Therapy (DMT) has shown great potential for improving the motor control ability of children with ASD. However, traditional DMT methods often lack vividness and are difficult to implement effectively. To address this issue, we propose a Mixed Reality DMT approach, utilizing interactive virtual agents. This approach offers immersive training content and multi-sensory feedback. To improve the training performance of children with ASD, we introduce a novel training paradigm featuring a self-guided mode. This paradigm enables the rapid creation of a virtual twin agent of the child with ASD using a single photo to embody oneself, which can then guide oneself during training. We conducted an experiment with the participation of 24 children diagnosed with ASD (or ASD propensity), recording their training performance under various experimental conditions. Through expert rating, behavior coding of training sessions, and statistical analysis, our findings revealed that the use of the twin agent for self-guidance resulted in noticeable improvements in the training performance of children with ASD. These improvements were particularly evident in terms of enhancing movement quality and refining overall target-related responses. Our study holds clinical potential in the field of medical treatment and rehabilitation for children with ASD.

Role-playing is widely used in many areas, such as psychotherapy and behavior change. However, few studies have explored the possible effects of playing multiple roles in a single role-playing process. We propose a new role-playing paradigm, called role-exchange playing, in which a user plays two opposite roles successively in the same simulated event for better cognitive enhancement. We designed an experiment with this novel role-exchange playing strategy in the immersive virtual environments; and school bullying was chosen as a scenario in this case. A total of 234 middle/high school students were enrolled in the mixed-design experiment. From the user study, we found that through role-exchange, students developed more morally correct opinions about bullying, as well as increased empathy and willingness to engage in supportive behavior. They also showed increased commitment to stopping bullying others. Our role-exchange paradigm could achieve a better effect than traditional role-playing methods in situations where participants have no prior experience associated with the roles they play. Therefore, using role-exchange playing in the immersive virtual environments to educate minors can help prevent them from bullying others in the real world. Our study indicates a positive significance in moral education of teenagers. Our role-exchange playing may have the potential to be extended to such applications as counseling, therapy, and crime prevention.

Understanding human visual attention in immersive virtual reality (VR) is crucial for many important applications, including gaze prediction, gaze guidance, and gaze-contingent rendering. However, previous works on visual attention analysis typically only explored one specific VR task and paid less attention to the differences between different tasks. Moreover, existing task recognition methods typically focused on 2D viewing conditions and only explored the effectiveness of human eye movements. We first collect eye and head movements of 30 participants performing four tasks, i.e., Free viewing, Visual search, Saliency, and Track, in 15 360-degree VR videos. Using this dataset, we analyze the patterns of human eye and head movements and reveal significant differences across different tasks in terms of fixation duration, saccade amplitude, head rotation velocity, and eye-head coordination. We then propose EHTask - a novel learning-based method that employs eye and head movements to recognize user tasks in VR. We show that our method significantly outperforms the state-of-the-art methods derived from 2D viewing conditions both on our dataset (accuracy of 84.4% versus 62.8%) and on a real-world dataset ( 61.9% versus 44.1%). As such, our work provides meaningful insights into human visual attention under different VR tasks and guides future work on recognizing user tasks in VR.

We present an efficient locomotion technique that can reduce cybersickness through aligning the visual and vestibular induced self-motion illusion. Our locomotion technique stimulates proprioception consistent with the visual sense by intentional head motion, which includes both the head's translational movement and yaw rotation. A locomotion event is triggered by the hand-held controller together with an intended physical head motion simultaneously. Based on our method, we further explore the connections between the level of cybersickness and the velocity of self motion through a series of experiments. We first conduct Experiment 1 to investigate the cybersickness induced by different translation velocities using our method and then conduct Experiment 2 to investigate the cybersickness induced by different angular velocities. Our user studies from these two experiments reveal a new finding on the correlation between translation/angular velocities and the level of cybersickness. The cybersickness is greatest at the lowest velocity using our method, and the statistical analysis also indicates a possible U-shaped relation between the translation/angular velocity and cybersickness degree. Finally, we conduct Experiment 3 to evaluate the performances of our method and other commonly-used locomotion approaches, i.e., joystick-based steering and teleportation. The results show that our method can significantly reduce cybersickness compared with the joystick-based steering and obtain a higher presence compared with the teleportation. These advantages demonstrate that our method can be an optional locomotion solution for immersive VR applications using commercially available HMD suites only.

Abstract We present a simulation method of pursuit‐evasion behaviors between different herds with runtime situational awareness. We model the collective movement of predators and prey in the hunting process based on an improved Boid model and the finite state machine involving multiple perception factors about the situations. Taking wolves hunting caribous as an example, our method not only simulates the collective behavior, but also realizes individual intelligent behaviors consistent with the observations. We conducted intensive experiments, and the results show that our method can simulate more realistic pursuit‐evasion behavior under different testing scenarios with a variety of parameters. Our method can be easily transferred to the theoretical study of collective behaviors of other herds.

We present a novel learning-based modal sound synthesis approach that includes a mixed vibration solver for modal analysis and a radiation network for acoustic transfer. Our mixed vibration solver consists of a 3D sparse convolution network and a Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) module for iterative optimization. Moreover, we highlight the correlation between a standard numerical vibration solver and our network architecture. Our radiation network predicts the Far-Field Acoustic Transfer maps (FFAT Maps) from the surface vibration of the object. The overall running time of our learning-based approach for most new objects is less than one second on a RTX 3080 Ti GPU while maintaining a high sound quality close to the ground truth solved by standard numerical methods. We also evaluate the numerical and perceptual accuracy of our approach on different objects with various shapes and materials.

Bidirectional path tracing (BDPT) can be accelerated by selecting appropriate light sub-paths for connection. However, existing algorithms need to perform frequent distribution reconstruction and have expensive overhead. We present a novel approach, SPCBPT, for probabilistic connections that constructs the light selection distribution in sub-path space. Our approach bins the sub-paths into multiple subspaces and keeps the sub-paths in the same subspace of low discrepancy, wherein the light sub-paths can be selected by a subspace-based two-stage sampling method, i.e., first sampling the light subspace and then resampling the light sub-paths within this subspace. The subspace-based distribution is free of reconstruction and provides efficient light selection at a very low cost. We also propose a method that considers the Multiple Importance Sampling (MIS) term in the light selection and thus obtain an MIS-aware distribution that can minimize the upper bound of variance of the combined estimator. Prior methods typically omit this MIS weights term. We evaluate our algorithm using various benchmarks, and the results show that our approach has superior performance and can significantly reduce the noise compared with the state-of-the-art method.

We present the design and results of an experiment investigating the occurrence of self-illusion and its contribution to realistic behavior consistent with a virtual role in virtual environments. Self-illusion is a generalized illusion about one's self in cognition, eliciting a sense of being associated with a role in a virtual world, despite sure knowledge that this role is not the actual self in the real world. We validate and measure self-illusion through an experiment where each participant occupies a non-human perspective and plays a non-human role using this role's behavior patterns. 77 participants were enrolled for the user study according to the priori power analysis. In the mixed-design experiment with different levels of manipulations, we asked the participants to play a cat (a non-human role) within an immersive VE and captured their different kinds of responses, finding that the participants with higher self-illusion can connect themselves to the virtual role more easily. Based on statistical analysis of questionnaires and behavior data, there is some evidence that self-illusion can be considered a novel psychological component of presence because it is dissociated from sense of embodiment (SoE), plausibility illusion (Psi), and place illusion (PI). Moreover, self-illusion has the potential to be an effective evaluation metric for user experience in a virtual reality system for certain applications.

Human visual attention in immersive virtual reality (VR) is key for many important applications, such as content design, gaze-contingent rendering, or gaze-based interaction. However, prior works typically focused on free-viewing conditions that have limited relevance for practical applications. We first collect eye tracking data of 27 participants performing a visual search task in four immersive VR environments. Based on this dataset, we provide a comprehensive analysis of the collected data and reveal correlations between users' eye fixations and other factors, i.e. users' historical gaze positions, task-related objects, saliency information of the VR content, and users' head rotation velocities. Based on this analysis, we propose FixationNet - a novel learning-based model to forecast users' eye fixations in the near future in VR. We evaluate the performance of our model for free-viewing and task-oriented settings and show that it outperforms the state of the art by a large margin of 19.8% (from a mean error of 2.93° to 2.35°) in free-viewing and of 15.1% (from 2.05° to 1.74°) in task-oriented situations. As such, our work provides new insights into task-oriented attention in virtual environments and guides future work on this important topic in VR research.

Granular media is the second-most-manipulated substance on Earth, second only to water. However, simulation of granular media is still challenging due to the complexity of granular materials and the large number of discrete solid particles. As we know, dry granular materials could form a hybrid state between a fluid and a solid, therefore we propose a two-layer model and divide the simulation domain into a dilute layer, where granules can move freely as a fluid, and a dense layer, where granules act more like a solid. Motivated by the shallow water equations, we derive a set of shallow sand equations for modeling dry granular flows by depth-integrating three-dimensional governing equations along its vertical direction. Unlike previous methods for simulating a 2D granular media, our model does not restrict the depth of the granular media to be shallow anymore. To allow efficient fluid-solid interactions, we also present a ray casting algorithm for one-way solid-fluid coupling. Finally, we introduce a particle-tracking method to improve the visual representation. Our method can be efficiently implemented based on a height field and is fully compatible with modern GPUs, therefore allows us to simulate large-scale dry granular flows in real time.

Fast simulation techniques are strongly favored in computer graphics, especially for the nonlinear inhomogeneous elastic materials. The homogenization theory is a perfect match to simulate inhomogeneous deformable objects with its coarse discretization, as it reveals how to extract information at a fine scale and to perform efficient computation with much less DOF. The existing homogenization method is not applicable for ubiquitous nonlinear materials with the limited input deformation displacements. In this paper, we have proposed a homogenization method for the efficient simulation of nonlinear inhomogeneous elastic materials. Our approach allows for a faithful approximation of fine, heterogeneous nonlinear materials with very coarse discretization. Modal analysis provides the basis of a linear deformation space and modal derivatives extend the space to a nonlinear regime; based on this, we exploited modal derivatives as the input characteristic deformations for homogenization. We also present a simple elastic material model that is nonlinear and anisotropic to represent the homogenized materials. The nonlinearity of material deformations can be represented properly with this model. The material properties for the coarsened model were solved via a constrained optimization that minimizes the weighted sum of the strain energy deviations for all input deformation modes. An arbitrary number of bases can be used as inputs for homogenization, and greater weights are placed on the more important low-frequency modes. Based on the experimental results, this study illustrates that the homogenized material properties obtained from our method approximate the original nonlinear material behavior much better than the existing homogenization method with linear displacements, and saves orders of magnitude of computational time. The proposed homogenization method for nonlinear inhomogeneous elastic materials is capable of capturing the nonlinear dynamics of the original dynamical system well.

3D vehicle detection based on point cloud is a challenging task in real-world applications such as autonomous driving. Despite significant progress has been made, we observe two aspects to be further improved. First, the semantic context information in LiDAR is seldom explored in previous works, which may help identify ambiguous vehicles. Second, the distribution of point cloud on vehicles varies continuously with increasing depths, which may not be well modeled by a single model. In this work, we propose a unified model SegVoxelNet to address the above two problems. A semantic context encoder is proposed to leverage the free-of-charge semantic segmentation masks in the bird's eye view. Suspicious regions could be highlighted while noisy regions are suppressed by this module. To better deal with vehicles at different depths, a novel depth-aware head is designed to explicitly model the distribution differences and each part of the depth-aware head is made to focus on its own target detection range. Extensive experiments on the KITTI dataset show that the proposed method outperforms the state-of-the-art alternatives in both accuracy and efficiency with point cloud as input only.

Model sound synthesis is a physically-based sound synthesis method used to generate audio content in games and virtual worlds. We present a novel learning-based impact sound synthesis algorithm called Deep-Modal. Our approach can handle sound synthesis for common arbitrary objects, especially dynamic generated objects, in real-time. We present a new compact strategy to represent the mode data, corresponding to frequency and amplitude, as fixed-length vectors. This is combined with a new network architecture that can convert shape features of 3D objects into mode data. Our network is based on an encoder-decoder architecture with the contact positions of objects and external forces embedded. Our method can synthesize interactive sounds related to objects of various shapes at any contact position, as well as objects of different materials and sizes. The synthesis process only takes ~0.01s on a GTX 1080 Ti GPU. We show the effectiveness of Deep-Modal through extensive evaluation using different metrics, including recall and precision of prediction, sound spectrogram, and a user study.

Abstract The treatment of solid boundary conditions remains one of the most challenging parts in the SPH method. We present a semi‐analytical approach to handle complex solid boundaries of arbitrary shape. Instead of calculating a renormalizing factor for the particle near the boundary, we propose to calculate the volume integral inside the solid boundary under the local spherical frame of a particle. By converting the volume integral into a surface integral, a computer aided design (CAD) mesh file representing the boundary can be naturally integrated for particle simulations. To accelerate the search for a particle's neighboring triangles, a uniform grid is applied to store indices of intersecting triangles. The new semi‐analytical solid boundary handling approach is integrated into a position‐based method [MM13] as well as a projection‐based [HWW*20] to demonstrate its effectiveness in handling complex boundaries. Experiments show that our method is able to achieve comparable results with those simulated using ghost particles. In addition, since our method requires no boundary particles for deforming surfaces, our method is flexible enough to handle complex solid boundaries, including sharp corners and shells.

We present a novel chi-squared progressive photon mapping algorithm (CPPM) that constructs an estimator by controlling the bandwidth to obtain superior image quality. Our estimator has parametric statistical advantages over prior nonparametric methods. First, we show that when a probability density function of the photon distribution is subject to uniform distribution, the radiance estimation is unbiased under certain assumptions. Next, the local photon distribution is evaluated via a chi-squared test to determine whether the photons follow the hypothesized distribution (uniform distribution) or not. If the statistical test deems that the photons inside the bandwidth are uniformly distributed, bandwidth reduction should be suspended. Finally, we present a pipeline with a bandwidth retention and conditional reduction scheme according to the test results. This pipeline not only accumulates sufficient photons for a reliable chi-squared test, but also guarantees that the estimate converges to the correct solution under our assumptions. We evaluate our method on various benchmarks and observe significant improvement in the running time and rendering quality in terms of mean squared error over prior progressive photon mapping methods.

We conduct novel analyses of users' gaze behaviors in dynamic virtual scenes and, based on our analyses, we present a novel CNN-based model called DGaze for gaze prediction in HMD-based applications. We first collect 43 users' eye tracking data in 5 dynamic scenes under free-viewing conditions. Next, we perform statistical analysis of our data and observe that dynamic object positions, head rotation velocities, and salient regions are correlated with users' gaze positions. Based on our analysis, we present a CNN-based model (DGaze) that combines object position sequence, head velocity sequence, and saliency features to predict users' gaze positions. Our model can be applied to predict not only realtime gaze positions but also gaze positions in the near future and can achieve better performance than prior method. In terms of realtime prediction, DGaze achieves a 22.0% improvement over prior method in dynamic scenes and obtains an improvement of 9.5% in static scenes, based on using the angular distance as the evaluation metric. We also propose a variant of our model called DGaze_ET that can be used to predict future gaze positions with higher precision by combining accurate past gaze data gathered using an eye tracker. We further analyze our CNN architecture and verify the effectiveness of each component in our model. We apply DGaze to gaze-contingent rendering and a game, and also present the evaluation results from a user study.

Background Eye tracking technology is receiving increased attention in the field of virtual reality. Specifically, future gaze prediction is crucial in pre-computation for many applications such as gaze-contingent rendering, advertisement placement, and content-based design. To explore future gaze prediction, it is necessary to analyze the temporal continuity of visual attention in immersive virtual reality. Methods In this paper, the concept of temporal continuity of visual attention is presented. Subsequently, an autocorrelation function method is proposed to evaluate the temporal continuity. Thereafter, the temporal continuity is analyzed in both free-viewing and task-oriented conditions. Results Specifically, in free-viewing conditions, the analysis of a free-viewing gaze dataset indicates that the temporal continuity performs well only within a short time interval. A task-oriented game scene condition was created and conducted to collect users' gaze data. An analysis of the collected gaze data finds the temporal continuity has a similar performance with that of the free-viewing conditions. Temporal continuity can be applied to future gaze prediction and if it is good, users' current gaze positions can be directly utilized to predict their gaze positions in the future. Conclusions The current gaze's future prediction performances are further evaluated in both free-viewing and task-oriented conditions and discover that the current gaze can be efficiently applied to the task of short-term future gaze prediction. The task of long-term gaze prediction still remains to be explored.

We propose to address the face reconstruction in the wild by using a multi-metric regression network, MMFace, to align a 3D face morphable model (3DMM) to an input image. The key idea is to utilize a volumetric sub-network to estimate an intermediate geometry representation, and a parametric sub-network to regress the 3DMM parameters. Our parametric sub-network consists of identity loss, expression loss, and pose loss which greatly improves the aligned geometry details by incorporating high level loss functions directly defined in the 3DMM parametric spaces. Our high-quality reconstruction is robust under large variations of expressions, poses, illumination conditions, and even with large partial occlusions. We evaluate our method by comparing the performance with state-of-the-art approaches on latest 3D face dataset LS3D-W and Florence. We achieve significant improvements both quantitatively and qualitatively. Due to our high-quality reconstruction, our method can be easily extended to generate high-quality geometry sequences for video inputs.

We present a novel, data-driven eye-head coordination model that can be used for realtime gaze prediction for immersive HMD-based applications without any external hardware or eye tracker. Our model (SGaze) is computed by generating a large dataset that corresponds to different users navigating in virtual worlds with different lighting conditions. We perform statistical analysis on the recorded data and observe a linear correlation between gaze positions and head rotation angular velocities. We also find that there exists a latency between eye movements and head movements. SGaze can work as a software-based realtime gaze predictor and we formulate a time related function between head movement and eye movement and use that for realtime gaze position prediction. We demonstrate the benefits of SGaze for gaze-contingent rendering and evaluate the results with a user study.

Abstract In this paper, we exploit the use of peridynamics theory for graphical animation of material deformation and fracture. We present a new meshless framework for elastoplastic constitutive modelling that contrasts with previous approaches in graphics. Our peridynamics‐based elastoplasticity model represents deformation behaviours of materials with high realism. We validate the model by varying the material properties and performing comparisons with finite element method (FEM) simulations. The integral‐based nature of peridynamics makes it trivial to model material discontinuities, which outweighs differential‐based methods in both accuracy and ease of implementation. We propose a simple strategy to model fracture in the setting of peridynamics discretization. We demonstrate that the fracture criterion combined with our elastoplasticity model could realistically produce ductile fracture as well as brittle fracture. Our work is the first application of peridynamics in graphics that could create a wide range of material phenomena including elasticity, plasticity, and fracture. The complete framework provides an attractive alternative to existing methods for producing modern visual effects.

Abstract Peridynamics is a formulation of the classical elastic theory that is targeted at simulating deformable objects with discontinuities, especially fractures. Till now, there are few studies that have been focused on how to model general hyperelastic materials with peridynamics. In this paper, we target at proposing a general strain energy function of hyperelastic materials for peridynamics. To get an intuitive model that can be easily controlled, we formulate the strain energy density function as a function parameterized by the dilatation and bond stretches, which can be decomposed into multiple one‐dimensional functions independently. To account for nonlinear material behaviors, we also propose a set of nonlinear basis functions to help design a nonlinear strain energy function more easily. For an anisotropic material, we additionally introduce an anisotropic kernel to control the elastic behavior for each bond independently. Experiments show that our model is flexible enough to approximately regenerate various hyperelastic materials in classical elastic theory, including St. Venant‐Kirchhoff and Neo‐Hookean materials.

Abstract As light travels fast in the physical world, it is generally hard to capture the propagation of light with a real camera. However, with the development of photon mapping, it is possible to simulate and visualize such fantastic effect. In this paper, we propose a new algorithm based on progressive photon mapping to render the time‐varying light motion. By incorporating our algorithm with participating media, we synthesis the animation of time‐varying light beams expanding in space with slow motion. Interesting phenomena can be observed in our experiments through designating either a constant light source or a bullet light that only emits instantaneously. Our algorithm is efficient in the sense that all frames of the animation can be rendered with only one convergence of the progressive photon mapping, showing the advantage of revealing the light propagation process by our algorithm. The experiments demonstrate the effectiveness and efficiency of our approach.

We present an interactive and stable multi-contact dynamic simulation algorithm for rigid bodies. Our approach is based on fast frictional dynamics (FFD) [14], which is designed for large sets of non-convex rigid bodies. We use a new friction model that performs velocity-level multi-contact simulation using impulse decomposition. Moreover, we accurately handle friction at each contact point using contact distribution and frictional impulse solvers, which also account for relative motion. We evaluate our algorithm's performance on many complex multi-body benchmarks with thousands of contacts. In practice, our dynamics simulation algorithm takes a few milliseconds per timestep and exhibits more stable behaviors.

Abstract We extend the material point method (MPM) for robust simulation of extremely large elastic deformation. This facilitates the application of MPM towards a unified solver since its versatility has been demonstrated lately with simulation of varied materials. Extending MPM for invertible elasticity requires accounting for several of its inherent limitations. MPM as a meshless method exhibits numerical fracture in large tensile deformations. We eliminate it by augmenting particles with connected material domains. Besides, constant redefinition of the interpolating functions between particles and grid introduces accumulated error which behaves like artificial plasticity. We address this problem by utilizing the Lagrangian particle domains as enriched degrees of freedom for simulation. The enrichment is applied dynamically during simulation via an error metric based on local deformation of particles. Lastly, we novelly reformulate the computation in reference configuration and investigate inversion handling techniques to ensure the robustness of our method in regime of degenerated configurations. The power and robustness of our method are demonstrated with various simulations that involve extreme deformations.

Simultaneous multi-impact simulation is a challenging problem that frequently arises in physically-based modeling of rigid bodies. There are several physical criteria that should be satisfied for rigid body collision handling, but existing methods generally fail to meet one or more of them. In order to capture the inner process of potential energy variation, which is the physical foundation of collisions in a multi-impact system, we present a novel quadratic contact energy model for rigid body simulation. By constructing quadratic energy functions with respect to the impulses, post-impact reactions of rigid bodies can be computed efficiently. Our model can satisfy the physical criteria and can simulate various natural phenomena including the wave effect. Also, our model can be easily combined with Linear Complementary Problem (LCP) and can provide feasible results with any restitution coefficient. In practice, our model can solve the simultaneous multi-impact problem efficiently and robustly, and we highlight its performance on different benchmarks.

Abstract We present a novel method for flexible and efficient simulation of example‐based elastic deformation. The geometry of all input shapes is projected into a common shape space spanned by the Laplace–Beltrami eigenfunctions. The eigenfunctions are coupled to be compatible across shapes. Shape representation in the common shape space is scale‐invariant and topology‐independent. The limitation of previous example‐based approaches is circumvented that all examples must have identical topology with the simulated object. Additionally, our method allows examples that are arbitrary in size, similar but not identical in shape with the object. We interpolate the examples via a weighted‐energy minimization to find the target configuration that guides the object to desired deformation. Large deformation between examples is handled by a physically plausible energy metric. This optimization is efficient as the eigenfunctions are pre‐computed and the problem dimension is small. We demonstrate the benefits of our approach with animation results and performance analysis.

Abstract Enforcing fluid incompressibility is one of the time‐consuming aspects in SPH. In this paper, we present a local Poisson SPH (LPSPH) method to solve incompressibility for particle based fluid simulation. Considering the pressure Poisson equation, we first convert it into an integral form, and then apply a discretization to convert the continuous integral equation to a discretized summation over all the particles in the local pressure integration domain determined by the local geometry. To control the approximation error, we further integrate our local pressure solver into the predictive‐corrective framework to avoid the computational cost of solving a pressure Poisson equation globally. Our method can effectively eliminate the large density deviations mainly caused by the solid boundary treatment and free surface topological change, and show advantage of a higher convergence rate over the predictive‐corrective incompressible SPH (PCISPH).

Simulating solid-fluid coupling with the classical meshless methods is an difficult issue due to the lack of the Kronecker delta property of the shape functions when enforcing the essential boundary conditions. In this work, we present a novel staggered meshless method to overcome this problem. We create a set of staggered particles from the original particles in each time step by mapping the mass and momentum onto these staggered particles, aiming to stagger the velocity field from the pressure field. Based on this arrangement, an new approximate projection method is proposed to enforce divergence-free on the fluid velocity with compatible boundary conditions. In the simulations, the method handles the fluid and solid in a unified meshless manner and generalizes the formulations for computing the viscous and pressure forces. To enhance the robustness of the algorithm, we further propose a new framework to handle the degeneration case in the solid-fluid coupling, which guarantees stability of the simulation. The proposed method offers the benefit that various slip boundary conditions can be easily implemented. Besides, explicit collision handling for the fluid and solid is avoided. The method is easy to implement and can be extended from the standard SPH algorithm in a straightforward manner. The paper also illustrates both one-way and two-way couplings of the fluids and rigid bodies using several test cases in two and three dimensions.

We propose a mesh less method to simulate elastic solids. Explicit integration methods are widely used in fluid/solid simulators for their efficiency, but these methods are not unconditionally stable: without sufficient small time steps, simulated particles may move beyond range of each other, resulting in simulation breakdown or other unexpected errors. This problem which usually appears under large deformations is called numerical fracture. We use anisotropic kernels to reduce numerical fracture without resampling procedure. During each time step, we update the anisotropic kernels from the analysis of the strain tensor to capture the directions of the deformation. Results illustrate that our method improves the stability of the simulation with minimum computation cost.

In this paper, we present a novel fast Continuous Collision Detection (CCD) method using SIMD capacity of CPU and idea of dimension reduction. We apply a parallel linear filter culling performed in one-dimensional subspace followed by a parallel planar filter culling performed in two-dimensional subspace before each elementary test, which simultaneously and conservatively tests the relative motion of each primitive pairs in various selected subspace. CPU's SIMD capacity is utilized for parallelizing the projection and filtering process in each subspace. Parallel filter culling in subspace removes a large amount of redundant elementary tests with low cost, and improves the overall performance of collision query. We demonstrate the advantages of our approach when comparing with previous alternatives in various dynamic scenes as benchmarks. In experiments, we observe up to 99% removal of false positives, and a huge magnitude of speed improvement on elementary tests (over 3x). Since our method only correlates the elementary test, it is scalable and can be easily integrated with various available single or multicore CPU based CCD algorithm. In addition, the performance of our method is less sensitive to varying step time.

Abstract We propose a meshless method to simulate brittle fracture. For brittle solids, stress computation can be difficult because brittle materials generally require small timesteps which bring about heavy computational burden. Furthermore, treating the brittle objects as deformable bodies will cause inevitable visual artifact. We treat the brittle objects as fully rigid bodies and solve the brittle stress distribution with Meshless Local Petrov‐Galerkin as a quasistatic problem, so visual artifact disppears and no timestep restriction exists. As a meshless framework, our method has the advantage of easy‐resampling around high stress areas to improve computation accuracy. To generate fractured pieces, unlike previous methods which explicitly track the crack propagation, we also present a novel damage based model. Our model supports user‐control of the fracture pattern which is especially useful when simulating anisotropic materials such as glass or wood. Results show that our meshless framework is physically feasible and user controllable. Copyright © 2011 John Wiley & Sons, Ltd.

Streamline based method is one of the most important vector field visualization methods. In the past streamline placements algorithms, little physical related feature was considered, which is very important in our opinion. A novel streamline placement algorithm for 3D vector field is introduced in this paper. We measure the difference between the inflow and the outflow to evaluate the local spatial-varying feature at a specified field point. A Difference of Inflow and Outflow Matrix (DIOM) is then calculated to describe the global appearance of the field. We draw streamlines by choosing the local extreme points in DIOM as seeds. DIOM is somewhat like flow divergence and is physics-related thus re-flects intrinsic characteristics of the vector field. The strategy performs well in revealing features of the vector field even with relatively few streamlines both in 3D vector field

We propose a novel efficient deforming filter culling method for continuous collision detection (CCD) problem performed by dimension reduction in subspace. We present a fast linear filter (1D reduced filter) considering relative motion between primitives. We also provide a conservative and fast planar filter test (2D reduced filter) for self-collision feature pairs considering relative motion between vertex and edge. Filter test in subspace removes large amount of false positives and elementary tests with low cost, and improve the overall performance of collision query. We demonstrate our approach and compare it with previous alternatives in kinds of dynamic scenes. Combined with our linear and planar reduced filter, we observe a magnitude of speed improvement on elementary tests (over 2x) compared against previous ones. Our method keeps stable performance for simulations with large step time.

Abstract: A new classification algorithm based on multivariate analysis is proposed to discover and simulate the grading policy on school transcript data sets. The framework comprises three major steps. First, factor analysis is adopted to separate the scores of several different subjects into grading‐related ones and grading‐unrelated ones. Second, multidimensional scaling is employed for dimensionality reduction to facilitate subsequent data visualization and interpretation. Finally, a support vector machine is trained to classify the filtered data into different grades. This work provides an attractive framework for intelligent data analysis and decision making. It also exhibits the advantages of high classification accuracy and supports intuitive data interpretation.

Abstract Stylized wireframe rendering of 3D model is widely used in animation software in order to depict the configuration of deformable model in comprehensible ways. However, since some inherent flaws in traditional depth test based rendering technology, shape of lines can not been preserved as continuous movement or deformation of models. There often exists severe aliasing like flickering artifact when objects rendered in line form animate, especially rendered with thick or dashed line. To cover this artifact, unlike traditional approach, we propose a novel fast line drawing method with high visual fidelity for wireframe depiction which only depends on intrinsic topology of primitives without any preprocessing step or extra adjacent information pre‐stored. In contrast to previous widely‐used solutions, our method is advantageous in highly accurate visibility, clear and stable line appearance without flickering even for thick and dashed lines with uniform width and steady configuration as model moves or animates, so that it is strongly suitable for animation system. In addition, our approach can be easily implemented and controlled without any additional preestimate parameters supplied by users. Copyright © 2010 John Wiley & Sons, Ltd.

It is a challenge work to render natural lighting phenomena in real-time. A major reason is due to high computational expense to simulate the physical model of atmosphere scattering. Another is due to the lack of power and programmability in the graphic hardware. In this paper, we propose unified volumes representation for light shaft and shadow, which is an efficient method of simulating natural light shafts and shadows with atmospheric scattering effect. We give the analytic formula of light shaft without numerical integration and then make use of the current graphic hardware to implement the integral computation on each volume surface for scattering. Our approach can not only simulate the lighting effect with single light source but also multiple parallel light sources according to the physical model of skylight and sunlight. With acceleration of the GPU, we can generate realistic appearance with high frame rate satisfying real time application. It can possibly be used in current commercial game or other virtual reality systems.

In this paper we propose a GPU based interactive geometric modeling approach to designing fine level features on subdivision surfaces. Displacement mapping is a technique for adding fine geometric detail to surfaces by using two-dimensional height map to produce photo-realistic surfaces. Due to space inefficiency and time consuming to render displacement map, this technique is generally limited in offline cinematic content creation packages. We propose a new approach to designing fine level features on subdivision surfaces via displacement mapping interactively on the latest GPU. Our method can reduce the bandwidth of the graphics channel by generating complex geometric detail on GPU, without feeding a large number of vertices to the AGP or PCI-E. Moreover, we introduce feature modification tools to flexibly control and adjust the created features. Designers can preview the features at the rendering stage, saving the time to generate the satisfying features on surfaces. The proposed approach is efficient and robust, and can be applied in many interactive graphics applications such as computer gaming, geometric modeling and computer animation.

This paper presents a novel approach to implementing dynamic LOD on GPU. For our purpose, a quadtree structure is created based on seamless geometry image atlas, which is a 3D surface representation in parameter space by combining the features of geometry images and poly-cube maps. All the nodes in the quadtree are packed into the atlas textures. There are two rendering passes in our approach. In the first pass, the LOD selection is performed in the fragment shaders. The resultant buffer is taken as the input texture to the second rendering pass by vertex texturing, and thus the node culling and triangulation can be performed in the vertex shaders. Our LOD algorithm can generate adaptive meshes dynamically, and can be fully implemented on GPU. It improves the efficiency of LOD selection, and alleviates the computing load on CPU.

Point and polygon representations have their respective merits in rendering objects. In this paper, we propose a novel hybrid multi-resolution approach, PQuadtrees, to efficiently render highly detailed objects. PQuadtrees are constructed from geometry images. Both point and polygon are tightly integrated into a uniform structure. While traversing the P-Quadtrees in rendering, the part of surface that face the viewer can be rendered by coarser quad mesh to reduce the numbers of rendering primitives. The shading details can be enhanced by hardware accelerated normal mapping. The view dependent LOD selects the finer hierarchy on silhouette, which is rendered by points. The rendering of large-scale model is greatly accelerated while the visual effect both at the surfaces and the silhouette is guaranteed.

View-dependent level-of-detail (LOD) and visibility culling are two powerful tools for accelerating the rendering of very large models in a real-time visualization, especially in walkthrough of a large-scale terrain environment. In this paper, we propose a visibility-driven Continuous LOD (CLOD) framework for terrain, which takes advantage of both techniques. The visibility determination is based on the well-known occlusion horizon algorithm. By making use of the features of the terrain extracted in pre-processing stage, a new cascading occlusion culling (COC) algorithm is proposed to cull those regions classified as invisible to current viewpoint in real time. The time consumption and storage overheads that we spend on visibility preprocessing are quite small. Visibility-driven CLOD enhances culling efficiency and improves the frame rates significantly for walkthrough of a terrain environment.