Image-guided percutaneous needle puncture is a critical yet challenging clinical procedure, constrained by the high cognitive demand of mental model construction and manipulation of 3D anatomy via scrolling through 2D cross-sectional images. While virtual reality (VR) simulators provide a risk-free training platform, many focus on simulation fidelity but lack structured, self-directed learning frameworks. In this paper, we present PunctVR, a VR system that incorporates the instructional principles of scaffolding. PunctVR features a training mode employing a phased subgoal workflow and instructional guidance scaffolding, and an assessment-only test mode where both the workflow and enhanced 3D visualization are removed. We conducted a between-subject experiment with 16 physicians, comparing training using a baseline multiplanar reconstruction (MPR) view with a combined MPR + 3D visualization across two difficulty levels. Our test mode results indicate that all trainees significantly improved their performance after training. Furthermore, those who trained with the integrated 3D visualization achieved a greater reduction in puncture time in both easy and hard cases. These findings suggest that PunctVR effectively enhances procedural efficiency in simulated needle puncture training and provides important insights into how learning scaffolding can accelerate skill acquisition and retention for image-guided interventions.

Recent research developments on rendering dynamic humans using neural radiance fields are remarkable. These methods often utilize learning implicit geometry and image appearance rendering for digital humans. However, keeping the complex and fast motions in detail, such as fingers, clothes, and faces, remains a challenge. Inspired by temporal information from human motion, we propose an architecture among adjacent frames by constructing a model on global and local levels. For the global level, we propose a hidden Markov model (HMM)based method to capture the global similarity among adjacent frames. At the local level, we introduce a module composed of a multi-head attention mechanism on a triplet canonical space structure for patch-level local temporal information. Experiments on two public datasets of dynamic human rendering (ZJU-MoCap and the People-Snapshot dataset) demonstrate that the proposed method outperforms advanced methods quantitatively and qualitatively.

Abstract As virtual reality display technologies advance, resolutions and refresh rates continue to approach human perceptual limits, presenting a challenge for real‐time rendering algorithms. Neural super‐resolution is promising in reducing the computation cost and boosting the visual experience by scaling up low‐resolution renderings. However, the added workload of running neural networks cannot be neglected. In this article, we try to alleviate the burden by exploiting the foveated nature of the human visual system, in a way that we upscale the coarse input in a heterogeneous manner instead of uniform super‐resolution according to the visual acuity decreasing rapidly from the focal point to the periphery. With the help of dynamic and geometric information (i.e., pixel‐wise motion vectors, depth, and camera transformation) available inherently in the real‐time rendering content, we propose a neural accumulator to effectively aggregate the amortizedly rendered low‐resolution visual information from frame to frame recurrently. By leveraging a partition‐assemble scheme, we use a neural super‐resolution module to upsample the low‐resolution image tiles to different qualities according to their perceptual importance and reconstruct the final output adaptively. Perceptually high‐fidelity foveated high‐resolution frames are generated in real‐time, surpassing the quality of other foveated super‐resolution methods.

With the speedy increase of display resolution and the demand for interactive frame rate, rendering acceleration is becoming more critical for a wide range of virtual reality applications. Foveated rendering addresses this challenge by rendering with a non-uniform resolution for the display. Motivated by the non-linear optical lens equation, we present rectangular mapping-based foveated rendering (RMFR), a simple yet effective implementation of foveated rendering framework. RMFR supports varying level of foveation according to the eccentricity and the scene complexity. Compared with traditional foveated rendering methods, rectangular mapping-based foveated rendering provides a superior level of perceived visual quality while consuming minimal rendering cost.

Virtual Reality (VR) is becoming ubiquitous with the rise of consumer displays and commercial VR platforms. Such displays require low latency and high quality rendering of synthetic imagery with reduced compute overheads. Recent advances in neural rendering showed promise of unlocking new possibilities in 3D computer graphics via image-based representations of virtual or physical environments. Specifically, the neural radiance fields (NeRF) demonstrated that photo-realistic quality and continuous view changes of 3D scenes can be achieved without loss of view-dependent effects. While NeRF can significantly benefit rendering for VR applications, it faces unique challenges posed by high field-of-view, high resolution, and stereoscopic/egocentric viewing, typically causing low quality and high latency of the rendered images. In VR, this not only harms the interaction experience but may also cause sickness. To tackle these problems toward six-degrees-of-freedom, egocentric, and stereo NeRF in VR, we present the first gaze-contingent 3D neural representation and view synthesis method. We incorporate the human psychophysics of visual- and stereo-acuity into an egocentric neural representation of 3D scenery. We then jointly optimize the latency/performance and visual quality while mutually bridging human perception and neural scene synthesis to achieve perceptually high-quality immersive interaction. We conducted both objective analysis and subjective studies to evaluate the effectiveness of our approach. We find that our method significantly reduces latency (up to 99% time reduction compared with NeRF) without loss of high-fidelity rendering (perceptually identical to full-resolution ground truth). The presented approach may serve as the first step toward future VR/AR systems that capture, teleport, and visualize remote environments in real-time.

Calibration is a key step for on-vehicle AR-HUD systems to ensure the augmented information to be correctly viewed by the driver. State-of-art calibration methods require setting up of spatial tracking devices or attaching markers on vehicles, which is time-consuming and error-prone. In this paper, we present a novel multi-viewpoints calibration method for AR-HUD using only a mixed reality glasses such as HoloLens. The full calibration process can be done in one minute and provides high precise calibration result, while no markers need to be attached on vehicle.