ABSTRACT Physically based modal sound synthesis generates realistic audio by modeling structural vibration and acoustic radiation from object geometry and material properties. However, traditional approaches rely on computationally expensive eigenvalue analysis and acoustic transfer simulations, making them impractical for real‐time virtual reality (VR), especially for dynamically generated or fractured objects. We present NeuroSonic, a unified neural framework for real‐time physically based sound synthesis directly from point cloud geometry. Our key idea is to amortize modal analysis and acoustic transfer using neural networks, replacing costly numerical solvers with efficient feed‐forward inference. The framework predicts modal eigenvalues, eigenvectors, and acoustic transfer functions, forming a physically grounded representation for sound synthesis. By operating on point clouds, our method avoids voxelization artifacts and handles geometrically complex and thin structures. Once trained, NeuroSonic computes complete modal and acoustic information in under 0.02 s, enabling real‐time performance. Experiments show that our approach achieves accuracy comparable to physically based solvers while providing orders‐of‐magnitude speedup, making it a practical solution for interactive VR environments.

We present a real-time pipeline to approximate global illumination in interactive or dynamic scenes, including both 3D Gaussian models and conventional meshes. Building on a formulated surface light transport model for 3D Gaussians, we address key performance challenges through a fast compound stochastic ray-tracing algorithm and a hardware 3D Gaussian rasterizer, and we implement multiple RTGI (real-time global illumination) techniques for 3D Gaussians. Our pipeline enables real-time rendering of interactive scenes featuring editable materials, lights, meshes, and 3D Gaussian models, effectively capturing multi-bounce diffuse and one-bounce glossy light transport. Our approach covers a wide range of dynamic light sources, including area lights, directional lights, and environmental lighting. Comprehensive experimental results demonstrate that our approach can efficiently render global illumination for both 3DGS models and hybrid models combining meshes and 3DGS, all in real time. Our pipeline highlights the potential of 3D Gaussians in real-time global illumination, and offers insights into performance optimization.

Prior foveated rendering methods often suffer from a limitation where the shading load escalates with increasing display resolution, leading to decreased efficiency, particularly when dealing with retinal-level resolutions. To tackle this challenge, we begin with the essence of the human visual system (HVS) perception and present visual acuity-consistent foveated rendering (VaFR), aiming to achieve exceptional rendering performance at retinal-level resolutions. Specifically, we propose a method with a novel log-polar mapping function derived from the human visual acuity model, which accommodates the natural bandwidth of the visual system. This mapping function and its associated shading rate guarantee a consistent output of rendering information, regardless of variations in the display resolution of the VR HMD. Consequently, our VaFR outperforms alternative methods, improving rendering speed while preserving perceptual visual quality, particularly when operating at retinal resolutions. We validate our approach using both the rasterization and ray-casting rendering pipelines. We also validate our approach using different binocular rendering strategies for HMD devices. In diverse testing scenarios, our approach delivers better perceptual visual quality than prior foveated rendering while achieving an impressive speedup of 6.5×-9.29× for deferred rendering of 3D scenarios and an even more powerful speedup of 10.4×-16.4× for ray-casting at retinal resolution. Additionally, our approach significantly enhances the rendering performance of binocular 8 K path tracing, achieving smooth frame rates.

Background Eye tracking technology is receiving increased attention in the field of virtual reality. Specifically, future gaze prediction is crucial in pre-computation for many applications such as gaze-contingent rendering, advertisement placement, and content-based design. To explore future gaze prediction, it is necessary to analyze the temporal continuity of visual attention in immersive virtual reality. Methods In this paper, the concept of temporal continuity of visual attention is presented. Subsequently, an autocorrelation function method is proposed to evaluate the temporal continuity. Thereafter, the temporal continuity is analyzed in both free-viewing and task-oriented conditions. Results Specifically, in free-viewing conditions, the analysis of a free-viewing gaze dataset indicates that the temporal continuity performs well only within a short time interval. A task-oriented game scene condition was created and conducted to collect users' gaze data. An analysis of the collected gaze data finds the temporal continuity has a similar performance with that of the free-viewing conditions. Temporal continuity can be applied to future gaze prediction and if it is good, users' current gaze positions can be directly utilized to predict their gaze positions in the future. Conclusions The current gaze's future prediction performances are further evaluated in both free-viewing and task-oriented conditions and discover that the current gaze can be efficiently applied to the task of short-term future gaze prediction. The task of long-term gaze prediction still remains to be explored.