Redirected Walking (RDW) enables continuous locomotion in virtual environments (VEs) within limited physical spaces. However, classic RDW methods rely on static settings with tight boundaries, which can reduce their applicability in large shared physical spaces where boundary constraints are not dominant, especially under dynamic and multi-user conditions. To overcome this, we introduce a new task: Redirected Walking in Open Physical Spaces (OPSRDW), which allows users to navigate expansive VEs safely and naturally despite dynamic obstacles and without requiring a fixed boundary model. We further propose Dynamic Control and Redirection with Safety Constraints (DyCoRe), which formulates OPS-RDW as a constrained optimization problem. DyCoRe uses Dynamic Control Barrier Functions to model real-time collision avoidance constraints and solves an online Quadratic Programming problem to compute optimal velocities that minimize path deviation while ensuring safety. These velocities are mapped into real-time redirection gains, guiding users along natural paths while reducing collision risk. A relaxation mechanism is incorporated to handle infeasible scenarios. Extensive simulations suggest consistent improvements in safety and obstacle clearance over state-of-theart methods. User studies demonstrate that DyCoRe significantly improves navigation continuity, reduces the number of resets, and tends to reduce perceived discomfort compared to classic RDW strategies. DyCoRe provides an efficient and learning-free solution for safe and natural VR locomotion in open physical spaces with dynamic obstacles and multiple users.

Cybersickness is a major challenge in virtual reality (VR), adversely affecting user comfort and usability. While prior research has examined visual and system factors, the impact of different VR tasks under varied rendering conditions remains underexplored. To address this, we investigated three interaction tasks—navigation, selection, and manipulation—together with two rendering parameters: center area radius (CAR) and peripheral resolution (PR). Cybersickness severity and frequency were assessed using Simulator Sickness Questionnaire (SSQ) scores and electroencephalography (EEG) data. Results show that task type significantly affects cybersickness severity, with CAR playing a critical role. Moreover, α-band power spectral density (PSD) strongly correlates with SSQ scores, suggesting its potential as a biomarker for cybersickness. Neural responses also exhibited temporal delays compared to subjective reports, offering insights into the mechanisms of cybersickness. These findings advance understanding of task- and rendering-related influences, informing more effective prediction and mitigation strategies in VR system design.

Intuitive motion control is essential for virtual reality, allowing users to manipulate objects naturally while receiving realistic and responsive visual feedback. 3D Gaussian splatting provides real-time, photorealistic scene rendering, making it promising for virtual reality applications. Still, it falls short in accurate motion control of dynamic objects due to its unstructured global motion representation and redundant motion learning. To address these problems, we propose a motion hierarchical Gaussian based dynamic control method. First, a motion hierarchical Gaussian representation is introduced and initialized with semantic and deformation information. Then a motion hierarchical decomposition method is proposed to optimize the local motion in the representation. The representation is next optimized by a local motion analysis based refinement method. We also design a set of motion control operations for the motion hierarchical Gaussian. Experimental results show that our method achieves high-precision motion reconstruction, accurate motion decomposition, real-time, intuitively and immersive VR motion control.

Transferring existing Human-Object Interaction (HOI) motion to novel objects is essential for robotics, virtual reality. Traditional approaches only model spatial surface correspondences between humans and source objects or between source and target objects, ignoring the internal topological structures of humans, the internal topology of objects, the non-surface spatial topological relationships, and temporal motion relations. In this paper, we propose a spatial-temporal relation guided motion transfer framework. Firstly, we define a spatial-temporal relation interaction graph representation(STRIG) to model the human internal topology, object internal topology and human-object global topology together with the temporal motion relation. We propose a STRIGs-guided motion transfer diffusion model for generating spatially, semantically and temporally consistent HOI motions that are adapted to novel objects. To tackle the absence of ground-truth motions after transfer, we introduce a spatial-temporal relation optimization strategy. Extensive experiments demonstrate that our method consistently outperforms other approaches in terms of motion transfer quality, performance, and sequence stability, with particularly robustness under large variations in target object topology.

Effective Augmented Reality (AR) guidance for complex assembly faces a dual challenge: the inability of conventional liaison graphs to represent procedural logic, and the cognitive burden imposed by visual instructions. We argue that the solution requires a more expressive structure to overcome these representational deficits and a narration approach to mediate instruction complexity. Our method first employs an assembly hypergraph to capture the task's hierarchical information, from which an A* search algorithm generates an optimal assembly path. Then a Large Language Model (LLM)-mediated narration workflow is designed to address the ergonomic deficiencies of the machine-centric path. It employs an optimizer to improve fluency, followed by a narrator that crafts the steps into an intuitive instruction narration. A within-subjects user study (N = 24) revealed a progressive enhancement from our method's components. The transition from a liaison-graph baseline to the hypergraph alone improved objective outcomes by reducing task time and errors and improving subjective ratings (SUS, NASA-TLX, TAM, and ARI). Subsequently, augmenting the LLM-mediated narration maintained these gains while lowering cognitive load and elevating user experience and usability. Our findings indicate the value of our AR assembly design and discuss the opportunities of using LLM as a mediation layer for better user interaction.

3D multi-object arrangement is a fundamental task in VR that relies on accurate and natural initial selection alongside rapid and convenient subsequent manipulation to ensure high efficiency. However, existing methods fail to support efficient multi-object arrangement in highly occluded scenes with densely packed candidate objects through controller-free natural interactions. In this article, we propose an efficient, scene-aware multi-object arrangement method (MOA) designed for fast, precise, and convenient object arrangement. First, MOA introduces an importance-driven multi-object initial selection algorithm that assigns higher spatiotemporally correlated object importance (IMP) to target objects, establishing a natural multi-object initial selection mode that enables quick and accurate selection of high-IMP objects. Subsequently, it presents an auxiliary-structure-guided multi-object manipulation algorithm that constructs an auxiliary manipulation structure to assist subsequent multi-object manipulation, alongside a multi-modal interaction mode that facilitates swift and natural manipulation. Compared to state-of-the-art controller-free and controller-based methods, MOA significantly improves task performance, reduces task load, and enhances convenience in complex multi-object arrangement scenes involving hundreds of highly occluded objects need to be arranged.

Group navigation is a virtual reality (VR) technology designed to replace personal navigation, enhancing efficiency and aiding guided tours in path planning for other users. Group navigation techniques require users to have a clear understanding of navigation and travel modalities. Tour guides must ensure that users do not intersect with the environment or other users' models and must provide reasonable tour formations. To address these needs, we propose a scene-aware automatic formation generation method for fast and easy-to-use guided tours. First, we generate a limited number of candidate points for the exhibition and initialize the visiting group formation. Next, we optimize the formation to maximize view quality using our proposed viewpoint observation score. Finally, we match the visitors to the optimized formation to ensure a minimal view deflection angle. Moreover, we conducted a user study to evaluate the performance of our approach. Compared to the current method, our approach significantly increased navigation efficiency and view quality. Additionally, it substantially decreased the task load and improved the system usability for both tour guides and visitors.

Audio-driven talking head generation necessitates seamless integration of audio and visual data amidst the challenges posed by diverse input portraits and intricate correlations between audio and facial motions. In response, we propose a robust framework GoHD designed to produce highly realistic, expressive, and controllable portrait videos from any reference identity with any motion. GoHD innovates with three key modules: Firstly, an animation module utilizing latent navigation is introduced to improve the generalization ability across unseen input styles. This module achieves high disentanglement of motion and identity, and it also incorporates gaze orientation to rectify unnatural eye movements that were previously overlooked. Secondly, a conformer-structured conditional diffusion model is designed to guarantee head poses that are aware of prosody. Thirdly, to estimate lip-synchronized and realistic expressions from the input audio within limited training data, a two-stage training strategy is devised to decouple frequent and frame-wise lip motion distillation from the generation of other more temporally dependent but less audio-related motions, e.g., blinks and frowns. Extensive experiments validate GoHD's advanced generalization capabilities, demonstrating its effectiveness in generating realistic talking face results on arbitrary subjects.

Many VR/AR applications require the photorealistic rendering of hand-object interactions. Virtual hands are driven by users' hand poses captured via motion tracking to interact with virtual objects. The driven pose can be very noisy due to the constraints of tracking hardware and computation accuracy. This noise may lead to distorted hand poses and penetration artifacts during rendering. In this paper, we introduce the Gaussian Hand-Object Interaction Denoiser, the Gaussian splatting-based hand-object interaction denoising method, which effectively denoises the input twisted and penetrated hand poses to produce photorealistic results. We first propose the innovative joint-to-Gaussian surface representation, which accurately models the spatial relationships between hand skeleton joints and object Gaussians while highlighting hand-object penetrations and generalizing well to new hand poses and objects. Then, we propose a geometry-aware de-penetration algorithm that eliminates penetrations by detecting intersections between skeleton bones and object Gaussians and reposing any penetrated fingers onto the estimated underlying surface of the object. Experiments demonstrate that our method not only effectively reduces hand-object penetration depth but also produces more realistic rendering quality compared to the state-of-the-art methods MANUS+GEARS, MANUS+GeneOH, and$2 \text{DGS}+\text{Gene} \text{OH}$. The user study results show that our method significantly improves the users' visual perceptual experience regarding penetration and stability metrics. Project page: https://github.com/ZhaoLizz/GSHOIDenoiser

Object grouping task is an important research direction for fast manipulation of a large number of objects. It can help users to improve the efficiency of multi-object manipulation. However, the current research on this aspect is still immature. For this task, in this paper, based on the brush metaphor, we propose a method for grouping objects based on bare hands in virtual reality scenes. We design a number of interactions to facilitate the user's grouping of objects in the three-dimensional virtual space. Object grouping in virtual reality could encompass two subtasks: group generation and group modification. The emphasis of these tasks varies, with the former focusing on creating groups from ungrouped objects and the latter focusing on modifying group members once they are generated. The results of the empirical study show that our method has better performance in accomplishing both sub-tasks compared to the Ray method, Screen method and Cone method.

This article describes a mixed reality visual instruction approach for remote collaboration between a trainee and an expert. The expert authors the visual instructions through a virtual reality interface. The instructions are shown to the trainee overlaid onto the workspace using an augmented reality interface. The approach achieves effectiveness and efficiency by addressing three challenges. First, the expert-authored visual instructions are shown to the trainee by taking into account occlusions with the 3D workspace; Second, in addition to abstract visual instructions implemented by arrows, the expert can also author highly suggestive instructions by depicting the target state of the workspace realistically by selecting, copying, pasting, and repositioning workspace objects; Third, multiple instructions can be concatenated in sequences that the trainee executes on their own, without any additional guidance from the expert; The approach has been evaluated in a controlled user study with three experiments. The experiment verification confirms that compared to the conventional instruction, this approach achieves significantly lower error rates, shorter task completion times, and lower rotation angular errors. Moreover, the approach allows the trainee to execute the entire sequence robustly, without real-time instruction from the expert.

Rendering quality and performance greatly affect the user's immersion in VR experiences. 3D Gaussian Splatting-based methods can achieve photo-realistic rendering with speeds of over 100 fps in static scenes, but the speed drops below 10 fps in monocular dynamic scenes. Foveated rendering provides a possible solution to accelerate rendering without compromising visual perceptual quality. However, 3DGS and foveated rendering are not compatible. In this paper, we propose Fov-GS, a foveated 3D Gaussian splatting method for rendering dynamic scenes in real time. We introduce a 3D Gaussian forest representation that represents the scene as a forest. To construct the 3D Gaussian forest, we propose a 3D Gaussian forest initialization method based on dynamic-static separation. Subsequently, we propose a 3D Gaussian forest optimization method based on deformation field and Gaussian decomposition to optimize the forest and deformation field. To achieve real-time dynamic scene rendering, we present a 3D Gaussian forest rendering method based on HVS models. Experiments demonstrate that our method not only achieves higher rendering quality in the foveal and salient regions compared to the SOTA methods but also dramatically improves rendering performance, achieving up to 11.33X speedup. We also conducted a user study, and the results prove that the perceptual quality of our method has a high visual similarity with the ground truth.

In virtual reality applications, in addition to visual feedback, real objects can be used as props for virtual objects to provide passive haptic feedback, which greatly enhances user immersion. Usually, real object props are not one-to-one correspondence with virtual objects. Haptic retargeting technique is proposed to establish the virtual-real correspondence by introducing an offset between the virtual hand and the real hand. Sometimes, the offset is too large to cause user discomfort, and it is necessary to introduce a reset between two haptic retargeting operations to force the virtual hand and the real hand to coincide in order to eliminate the offset. However, too many resets can interfere with this immersion. To address this problem, we propose a haptic retargeting method based on proxy importance calculation using multiple props in virtual reality. The concept of proxy importance for props is introduced first, and then a proxy importance based prop selection and placement method for moving virtual objects are proposed. We also improve the performance of our method by using the props' weighted proxy importance strategy for multi-user collaboration. Compared to the state-of-the-art methods, our method significantly reduces the number of resets, the task completion time, hand movement distances, and task load without the cost of cybersickness in the single-user task. In the multi-user collaborative task, our method also achieves significant improvement using the strategy that weights the proxy importance of the props.

3D scene graph generation is essential for spatial computing in Extended Reality (XR), providing structured semantics for task planning and intelligent perception. However, unlike instance-segmentation-driven setups, generating semantic scene graphs still suffer from limited accuracy due to coarse and noisy point cloud data typically acquired in practice, and from the lack of interactive strategies to incorporate users' spatialized and intuitive guidance. We identify three key challenges: designing controllable interaction forms, involving guidance in inference, and generalizing from local corrections. To address these, we propose SGSG, a Stroke-Guided Scene Graph generation method that enables users to interactively refine 3D semantic relationships and improve predictions in real time. We propose three types of strokes and a lightweight SGstrokes dataset tailored for this modality. Our model integrates stroke guidance representation and injection for spatio-temporal feature learning and reasoning correction, along with intervention losses that combine consistency-repulsive and geometry-sensitive constraints to enhance accuracy and generalization. Experiments and the user study show that SGSG outperforms state-of-the-art methods 3DSSG and SGFN in overall accuracy and precision, surpasses JointSSG in predicate-level metrics, and reduces task load across all control conditions, establishing SGSG as a new benchmark for interactive 3D scene graph generation and semantic understanding in XR. Implementation resources are available at: https://github.com/Sycamore-Ma/SGSG-runtime.

With the increasing complexity of geometry and rendering effects in virtual reality (VR) scenes, existing foveated rendering methods for VR head-mounted displays (HMDs) struggle to meet users' demands for VR scene rendering with high frame rates ($\geq 60fps$≥60fps for rendering binocular foveated images in VR scenes containing over 50 m triangles). Current research validates that auditory content affects the perception of the human visual system (HVS). However, existing foveated rendering methods primarily model the HVS's eccentricity-dependent visual perception ability on the visual content in VR while ignoring the impact of auditory content on the HVS's visual perception. In this article, we introduce an auditory-content-based perceived rendering quality analysis to quantify the impact of visual perception under different auditory conditions in foveated rendering. Based on the analysis results, we propose an audio-visual aware foveated rendering method (AvFR). AvFR first constructs an audio-visual feature-driven perception model that predicts the eccentricity-based visual perception in real time by combining the scene's audio-visual content, and then proposes a foveated rendering cost optimization algorithm to adaptively control the shading rate of different regions with the guidance of the perception model. In complex scenes with visual and auditory content containing over 1.17 m triangles, AvFR renders high-quality binocular foveated images at an average frame rate of 116$fps$fps. The results of the main user study and performance evaluation validate that AvFR achieves significant performance improvement (up to 1.4× speedup) without lowering the perceived visual quality compared with the state-of-the-art VR-HMD foveated rendering method.

Foveated rendering provides an idea for improving the image synthesis performance of neural radiance fields (NeRF) methods. In this article, we propose a scene-aware foveated neural radiance fields method to synthesize high-quality foveated images in complex VR scenes at high frame rates. First, we construct a multi-ellipsoidal neural representation to enhance the neural radiance field's representation capability in salient regions of complex VR scenes based on the scene content. Then, we introduce a uniform sampling based foveated neural radiance field framework to improve the foveated image synthesis performance with one-pass color inference, and improve the synthesis quality by leveraging the foveated scene-aware objective function. Our method synthesizes high-quality binocular foveated images at the average frame rate of 66 frames per second ($FPS$FPS) in complex scenes with high occlusion, intricate textures, and sophisticated geometries. Compared with the state-of-the-art foveated NeRF method, our method achieves significantly higher synthesis quality in both the foveal and peripheral regions with 1.41-1.46× speedup. We also conduct a user study to prove that the perceived quality of our method has a high visual similarity with the ground truth.

Passive haptics utilize the shape of a physical object to convey feedback to the user and enhance immersion in virtual reality. Haptic retargeting is a passive haptic interaction method. Its mapping of physical objects to virtual objects solves the matching problem between virtual and physical objects in the passive haptic method. However, most existing haptic retargeting methods improve efficiency without considering the important factor of user comfort. In this article, we propose an efficient and comfortable haptic retargeting method based on reset point optimization. First, we construct two maps indicating user interaction comfort: the RULA score map and the dominant hand gain map. Subsequently, we propose a reset point optimization algorithm based on these two maps. Moreover, we also optimize the selection of the physical proxy and the placement location when the reset occurs. The user study results show a significant improvement in the efficiency and comfort of our method compared to state-of-the-art methods.

In extended reality (XR) applications, enhancing user perception often necessitates head-mounted display (HMD) removal. However, existing methods suffer from low time performance and suboptimal reconstruction quality. In this paper, we propose a half face mapping 3D Gaussian splatting avatar based HMD removal method (HFM-GS), which can perform real-time and high-fidelity online restoration of the complete face in HMD-occluded videos for XR applications after a short un-occluded face registration. We establish a mapping field between the upper and lower face Gaussians to enhance the adaptability to deformation. Then, we introduce correlation weight-based sampling to improve time performance and handle variations in the number of Gaussians. At last, we ensure model robustness through Gaussian Segregation Strategy. Compared to two state-of-the-art methods, our method achieves better quality and time performance. The results of the user study show that fidelity is significantly improved with our method.

Cybersickness significantly impairs user comfort and immersion in virtual reality (VR). Effective identification of cybersickness leveraging physiological, visual, and motion data is a critical prerequisite for its mitigation. However, current methods primarily employ direct feature fusion across modalities, which often leads to limited accuracy due to inadequate modeling of inter-modal relationships. In this paper, we propose a multimodal contrastive learning method for cybersickness recognition. First, we introduce Brain Connectivity Graph Representation (BCGR), an innovative graph-based representation that captures cybersickness-related connectivity patterns across modalities. We further develop three BCGR instances: E-BCGR, constructed based on EEG signals; MV-BCGR, constructed based on video and motion data; and S-BCGR, obtained through our proposed standardized decomposition algorithm. Then, we propose a connectivity-constrained contrastive fusion module, which aligns E-BCGR and MV-BCGR into a shared latent space via graph contrastive learning while utilizing S-BCGR as a connectivity constraint to enhance representation quality. Moreover, we construct a multimodal cybersickness dataset comprising synchronized EEG, video, and motion data collected in VR environments to promote further research in this domain. Experimental results demonstrate that our method outperforms existing state-of-the-art methods across four critical evaluation metrics: accuracy, sensitivity, specificity, and the area under the curve. Source code: https://github.com/PEKEW/cybersickness-bcgr.

Group selection in virtual reality is an important means of multi-object selection, which allows users to quickly group multiple objects and can significantly improve the operation efficiency of multiple types of objects. In this paper, we propose a group selection method based on multiple rounds of probability permutation, in which the efficiency of group selection is substantially improved by making the object layout of the next round easier to be batch-selected through interactive selection, object grouping probability computation, and position rearrangement in each round of the selection process. We conducted ablation experiments to determine the algorithm coefficients and validate the effectiveness of the algorithm. In addition, an empirical user study was conducted to evaluate the ability of our method to significantly improve the efficiency of the group selection task in an immersive virtual reality environment. The reduced operations also indirectly reduce the user task load and improve usability.

Rendering animatable and realistic hand avatars is pivotal for enhancing user experiences in human-centered AR/VR applications. While recent initiatives have utilized neural radiance fields to forge hand avatars with lifelike appearances, these methods are often hindered by high computational demands and the necessity for extensive training views. In this paper, we introduce GaussianHand, the first Gaussian-based real-time 3D rendering approach that enables efficient free-view and free-pose hand avatar animation from sparse view images. Our approach encompasses two key innovations. We first propose Hand Gaussian Blend Shapes that effectively models hand surface geometry while ensuring consistent appearance across various poses. Second, we introduce the Neural Residual Skeleton, equipped with Residual Skinning Weights, designed to rectify inaccuracies involved in Linear Blend Skinning deformations due to geometry offsets. Experiments demonstrate that our method not only achieves far more realistic rendering quality with as few as 5 or 20 training views, compared to the 139 views required by existing methods, but also excels in efficiency, achieving up to 125 frames per second for real-time rendering and remarkably surpassing recent methods.

Text entry poses a significant challenge in the realm of virtual reality (VR). This paper introduces FanPad, a novel solution designed to facilitate dual-hand text input within head-mounted displays (HMDs). FanPad accomplishes this by ingeniously mapping and curving the 26 typing keys (T26) QWERTY keyboard onto the touchpads of both controllers. The curved key layout of FanPad is derived from the natural movement of the thumb when interacting with the touchpad, resembling an arc with a thumb-length fixed radius.To optimize the experience, we introduce a customization process for the FanPad curve to better cope with individual hand shapes and thumb movements. We also provide a version with more overlap area named FanPad-Ov for different users with different typing habits.Our first user study examined the effects of curving and different overlap areas by comparing four potential layouts. The results clearly favor the FanPad and FanPad-Ov layout compared to the nocurving version, SKPad(-Ov). Subsequently, the second user study was conducted to assess long-term performance and improvement on customized FanPads. Notably, novices achieved a typing speed of 19.73 words per minute (WPM), demonstrating a remarkable increase of 58.47% after a 60-phrase training in six days. The highest typing speed reached an impressive 24.19 WPM.

Abstract Sub‐surface scattering is always an important feature in translucent material rendering. When light travels through optically thick media, its transport within the medium can be approximated using diffusion theory, and is appropriately described by the bidirectional scattering‐surface reflectance distribution function (BSSRDF). BSSRDF methods rely on assumptions about object geometry and light distribution in the medium, which limits their applicability to general participating media problems. However, despite the high computational cost of path tracing, BSSRDF methods are often favoured due to their suitability for real‐time applications. We review these methods and discuss the most recent breakthroughs in this field. We begin by summarizing various BSSRDF models and then implement most of them in a 2D searchlight problem to demonstrate their differences. We focus on acceleration methods using BSSRDF, which we categorize into two primary groups: pre‐computation and texture methods. Then we go through some related topics, including applications and advanced areas where BSSRDF is used, as well as problems that are sometimes important yet are ignored in sub‐surface scattering estimation. In the end of this survey, we point out remaining constraints and challenges, which may motivate future work to facilitate sub‐surface scattering.

Hierarchical multi-granularity image classification is a challenging task that aims to tag each given image with multiple granularity labels simultaneously. Existing methods tend to overlook that different image regions contribute differently to label prediction at different granularities, and also insufficiently consider relationships between the hierarchical multi-granularity labels. We introduce a sequence-to-sequence mechanism to overcome these two problems and propose a multi-granularity sequence generation (MGSG) approach for the hierarchical multi-granularity image classification task. Specifically, we introduce a transformer architecture to encode the image into visual representation sequences. Next, we traverse the taxonomic tree and organize the multi-granularity labels into sequences, and vectorize them and add positional information. The proposed multi-granularity sequence generation method builds a decoder that takes visual representation sequences and semantic label embedding as inputs, and outputs the predicted multi-granularity label sequence. The decoder models dependencies and correlations between multi-granularity labels through a masked multi-head self-attention mechanism, and relates visual information to the semantic label information through a cross-modality attention mechanism. In this way, the proposed method preserves the relationships between labels at different granularity levels and takes into account the influence of different image regions on labels with different granularities. Evaluations on six public benchmarks qualitatively and quantitatively demonstrate the advantages of the proposed method. Our project is available at https://github.com/liuxindazz/mgsg .

Text entry is a recurring task in mixed reality (MR) applications, and the ability of eyes-free text entry methods to allow users to enter text without focusing on the input device is ideal and compelling. However, existing eyes-free text entry methods leave much to be desired regarding efficiency and accuracy. In this paper, we propose a new light-occlusion text entry method in MR environment that uses dual thumb typing on a touchscreen. We design a partially visible keyboard as visual feedback to improve user performance. In addition, we optimize the underlying keyboard by collecting eyes-free typing data through a user study. The results show that our method has high typing speed, low error rate, and is very novice-friendly. After a short training period, the average typing speed of the novice group can reach 26.23 WPM (words per minute), while the average typing speed of the potential expert group can reach 30.62 WPM.

Manipulating with virtual objects is a fundamental requirement in virtual environments. A good manipulation method needs to consider efficiency, accuracy, and comfort. This paper proposes VVIR-OM, an object manipulation method in virtual reality (VR) based on a variable virtual interaction region (VVIR). A hand interaction hemisphere region (HIHR) is introduced and constructed in real space, where the user is more comfortable manipulating the objects. Then a VVIR is introduced, and an interaction heat volume (IHV) based method is proposed to update VVIR during the process of the manipulation. At last, a mapping algorithm is proposed to map the user hand position in HIHR to the position in VVIR. Two user studies are designed to evaluate the performance of VVIR-OM. Compared to the state-of-the-art methods, VVIR-OM achieves significant improvements in task completion time, manipulation precision, and a significant reduction in fatigue. Moreover, VVIR-OM outperforms other methods in terms of task load and usability without the cost of cybersickness.

We propose a new scene-aware foveated rendering method, which incorporates the scene awareness and characteristics of the human visual system into the mapping-based foveated rendering framework. First, we generate the conservative visual importance map that encodes the visual features of the scene, visual acuity, and gaze motion. Second, we construct the pixel size control map using a convolution kernel method. Third, we utilize the pixel size control map to guide the foveated rendering. At last, a temporal coherent refinement strategy is used to maintain the smooth foveated rendering for the adjacent frames. Compared to the state-of-the-art mapping-based foveated rendering methods using the same compression ratio, our method achieves smaller MSE, higher PSNR, and SSIM in the fovea, periphery, salient regions, and the whole image. We also conducted user studies, and the results proved that the perceptual quality of our method has a high visual similarity with the around truth rendered with the full resolution.

Neural radiance fields (NeRF) has achieved revolutionary breakthrough in the novel view synthesis task for complex 3D scenes. However, this new paradigm struggles to meet the requirements for real-time rendering and high perceptual quality in virtual reality. In this paper, we propose VPRF, a novel visual perceptual based radiance fields representation method, which for the first time integrates the visual acuity and contrast sensitivity models of human visual system (HVS) into the radiance field rendering framework. Initially, we encode both the appearance and visual sensitivity information of the scene into our radiance field representation. Then, we propose a visual perceptual sampling strategy, allocating computational resources according to the HVS sensitivity of different regions. Finally, we propose a sampling weight-constrained training scheme to ensure the effectiveness of our sampling strategy and improve the representation of the radiance field based on the scene content. Experimental results demonstrate that our method renders more efficiently, with higher PSNR and SSIM in the foveal and salient regions compared to the state-of-the-art FoV-NeRF. The results of the user study confirm that our rendering results exhibit high-fidelity visual perception.

The rapid advances in technologies have brought new interaction paradigms of smart objects (e.g., digital devices) beyond digital device screens. By utilizing spatial properties, configurations, and movements of smart objects, designing spatial interaction, which is one of the emerging interaction paradigms, efficiently promotes engagement with digital content and physical facility. However, as an important phase of design, prototyping such interactions still remains challenging, since there is no ad-hoc approach for this emerging paradigm. Designers usually rely on methods that require fixed hardware setup and advanced coding skills to script and validate early-stage concepts. These requirements restrict the design process to a limited group of users in indoor scenes. To facilitate the prototyping to general usages, we aim to figure out the design difficulties and underlying needs of current design processes for spatially-aware object interactions by empirical studies. Besides, we explore the design space of the spatial interaction for smart objects and discuss the design space in an input-output spatial interaction model. Based on these findings, we present ProObjAR, an all-in-one novel prototyping system with an Augmented Reality Head Mounted Display (AR-HMD). Our system allows designers to easily obtain the spatial data of smart objects being prototyped, specify spatially-aware interactive behaviors from an input-output event triggering workflow, and test the prototyping results in situ. From the user study, we find that ProObjAR simplifies the design procedure and increases design efficiency to a large extent and thus advancing the development of spatially-aware applications in smart ecosystems.

We present "Double Doodles'' to make full use of two sequential inputs of a VR controller with 9 DOFs in total, 3 DOFs of the first input sequence for the generation of motion paths and 6 DOFs of the second input sequence for motion gestures. While engineering our system, we take ergonomics into consideration and design a set of user-defined motion gestures to describe character motions. We employ a real-time deep learning-based approach for highly accurate motion gesture classification. We then integrate our approach into a prototype system, and it allows users to directly create character animations in VR environments using motion gestures with a VR controller, followed by animation preview and animation interactive editing. Finally, we evaluate the feasibility and effectiveness of our system through a user study, demonstrating the usefulness of our system for visual storytelling dedicated to amateurs, as well as for providing fast drafting tools for artists.

Most previous approaches for analyzing food images have relied on extensively annotated datasets, resulting in significant human labeling expenses due to the varied and intricate nature of such images. Inspired by the effectiveness of contrastive self-supervised methods in utilizing unlabelled data, weiqing explore leveraging these techniques on unlabelled food images. In contrastive self-supervised methods, two views are randomly generated from an image by data augmentations. However, regarding food images, the two views tend to contain similar informative contents, causing large mutual information, which impedes the efficacy of contrastive self-supervised learning. To address this problem, we propose Feature Suppressed Contrast (FeaSC) to reduce mutual information between views. As the similar contents of the two views are salient or highly responsive in the feature map, the proposed FeaSC uses a response-aware scheme to localize salient features in an unsupervised manner. By suppressing some salient features in one view while leaving another contrast view unchanged, the mutual information between the two views is reduced, thereby enhancing the effectiveness of contrast learning for self-supervised food pre-training. As a plug-and-play module, the proposed method consistently improves BYOL and SimSiam by 1.70% ~ 6.69% classification accuracy on four publicly available food recognition datasets. Superior results have also been achieved on downstream segmentation tasks, demonstrating the effectiveness of the proposed method.

We are pleased to present the proceedings of the 30th IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2023), held March 25–29, 2023, in Shanghai, China, in hybrid format. These proceedings contain 69 of the 130 papers presented at IEEE VR 2023, with the other 61 papers being published in the IEEE VR 2023 special issue of the Transactions on Visualization and Computer Graphics. IEEE VR 2023 had 612 submissions for an acceptance rate of 21%.

Optimizing rendering performance improves the user's immersion in virtual scene exploration. Foveated rendering uses the features of the human visual system (HVS) to improve rendering performance without sacrificing perceptual visual quality. We collect and analyze the viewing motion of different locomotion methods, and describe the effects of these viewing motions on HVS's sensitivity, as well as the advantages of these effects that may bring to foveated rendering. Then we propose the locomotion-aware foveated rendering method (LaFR) to further accelerate foveated rendering by leveraging the advantages. In LaFR, we first introduce the framework of LaFR. Secondly, we propose an eccentricity-based shading rate controller that provides the shading rate control of the given region in foveated rendering. Thirdly, we propose a locomotion-aware log-polar mapping method, which controls the foveal average shading rate, the peripheral shading rate decrease speed, and the overall shading quantity with the locomotion-aware coefficients based on the eccentricity-based shading rate controller. LaFR achieves similar perceptual visual quality as the conventional foveated rendering while achieving up to 1.6× speedup. Compared with the full resolution rendering, LaFR achieves up to 3.8× speedup.

Recently, virtual reality (VR) technology has been widely used in medical, military, manufacturing, entertainment, and other fields. These applications must simulate different complex material surfaces, various dynamic objects, and complex physical phenomena, increasing the complexity of VR scenes. Current computing devices cannot efficiently render these complex scenes in real time, and delayed rendering makes the content observed by the user inconsistent with the user’s interaction, causing discomfort. Foveated rendering is a promising technique that can accelerate rendering. It takes advantage of human eyes’ inherent features and renders different regions with different qualities without sacrificing perceived visual quality. Foveated rendering research has a history of 31 years and is mainly focused on solving the following three problems. The first is to apply perceptual models of the human visual system into foveated rendering. The second is to render the image with different qualities according to foveation principles. The third is to integrate foveated rendering into existing rendering paradigms to improve rendering performance. In this survey, we review foveated rendering research from 1990 to 2021. We first revisit the visual perceptual models related to foveated rendering. Subsequently, we propose a new foveated rendering taxonomy and then classify and review the research on this basis. Finally, we discuss potential opportunities and open questions in the foveated rendering field. We anticipate that this survey will provide new researchers with a high-level overview of the state-of-the-art in this field, furnish experts with up-to-date information, and offer ideas alongside a framework to VR display software and hardware designers and engineers.

When searching for a dynamic target in an unknown real world scene, search efficiency is greatly reduced if users lack information about the spatial structure of the scene. Most target search studies, especially in robotics, focus on determining either the shortest path when the target’s position is known, or a strategy to find the target as quickly as possible when the target’s position is unknown. However, the target’s position is often known intermittently in the real world, e.g., in the case of using surveillance cameras. Our goal is to help user find a dynamic target efficiently in the real world when the target’s position is intermittently known. In order to achieve this purpose, we have designed an AR guidance assistance system to provide optimal current directional guidance to users, based on searching a prediction graph. We assume that a certain number of depth cameras are fixed in a real scene to obtain dynamic target’s position. The system automatically analyzes all possible meetings between the user and the target, and generates optimal directional guidance to help the user catch up with the target. A user study was used to evaluate our method, and its results showed that compared to free search and a top-view method, our method significantly improves target search efficiency.

In this special issue of IEEE Transactions on Visualization and Computer Graphics (TVCG), we are pleased to present the top papers from the 30th IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2023), held March 25–29, 2023, in Shanghai, China, in hybrid format.

Object Manipulation is a fundamental interaction in virtual reality (VR). The efficiency and accuracy of object manipulation are important to provide immersion to users. We propose a manipulation method with adaptive gains to improve the efficiency and accuracy of object manipulation in VR applications. First, we introduce manipulation gains. We then design an experiment to collect user behavior during manipulation to determine fitting functions for calculating manipulation gain. At last, we design a user study to evaluate the performance of our distant object manipulation method with adaptive gains. The results show that, compared with the state of the art methods, our method has a significant improvement in the completion time, and the manipulation accuracy of the tasks. Moreover, our method significantly increases usability and reduces task load.

Object locating in virtual reality (VR) has been widely used in many VR applications, such as virtual assembly, virtual repair, virtual remote coaching. However, when there are a large number of objects in the virtual environment(VE), the user cannot locate the target object efficiently and comfortably. In this paper, we propose a label guidance based object locating method for locating the target object efficiently in VR. Firstly, we introduce the label guidance based object locating pipeline to improve the efficiency of the object locating. It arranges the labels of all objects on the same screen, lets the user select the target labels first, and then uses the flying labels to guide the user to the target object. Then we summarize five principles for constructing the label layout for object locating and propose a two-level hierarchical sorted and orientated label layout based on the five principles for the user to select the candidate labels efficiently and comfortably. After that, we propose the view and gaze based label guidance method for guiding the user to locate the target object based on the selected candidate labels. It generates specific flying trajectories for candidate labels, updates the flying speed of candidate labels, keeps valid candidate labels, and removes the invalid candidate labels in real time during object locating with the guidance of the candidate labels. Compared with the traditional method, the user study results show that our method significantly improves efficiency and reduces task load for object locating.

Currently, ray tracing and image-based lighting (IBL) have shortcomings when rendering the metallic virtual object displayed in the holographic pyramid in mixed reality. Ray tracing can hardly achieve the interactive frame rates, and IBL cannot accurately render the reflection result of the foreground near the virtual object. In this paper, we propose a mixed reality rendering method to render glossy and specular reflection effects on metallic virtual objects displayed in the holographic pyramid based on the surrounding real environment at interactive frame rates. First, we acquire the real environment data with four RGBD cameras and a panoramic camera; then, we introduce a foreground point cloud generation method to extract a temporally stable foreground point cloud from RGBD videos captured in real time; after this, we propose an efficient ray tracing method to render the dynamic glossy and specular reflections on the virtual objects that are displayed in the holographic pyramid. We test our method on several real and synthetic scenes. Compared with IBL and screen-space ray tracing, our method can generate the rendering results closer to the ground truth at the same time cost. Compared with Monte Carlo path tracing, our method is 2.5-4.5× faster in generating rendering results of the comparable quality.

In this special issue of IEEE Transactions on Visualization and Computer Graphics (TVCG), we are pleased to present the journal papers from the 21st IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2022), which will be held as a hybrid conference between October 17 and 21, 2022 in Singapore. ISMAR continues the over twenty year long tradition of IWAR, ISMR, and ISAR, and is the premier conference for Mixed and Augmented Reality in the world.

Text entry is a frequently used task in virtual reality (VR) applications, and controller is the most common interactive device in current VR systems. However, in terms of typing speed, there is still a gap between the existing controller-based text entry techniques and using a physical keyboard in reality, so it is important to improve the efficiency of the controller-based text entry. In this paper, we introduce Flower Text Entry, a single-controller text entry method based on a newly designed flower-shaped keyboard using hand 3D translation interaction for letters selection. We conduct user studies to optimize the keyboard design and the mapping between the interaction and selection, so as to evaluate our method. The results show that our method has high typing speed, lower error rate, and is very friendly to novices compared with the state-of-the-art controller-based text entry methods. After a short training, the novice group can type at 17.65 words per minute (WPM), and the potential expert group can type at 22.97 WPM. The highest typing speed is up to 30.80 WPM achieved by a potential expert participant.

In this special issue ofIEEE Transactions on Visualization and Computer Graphics(TVCG), we are pleased to present a subset of papers from the 29th IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2022), held virtually March 12-16, 2022, in Christchurch, New Zealand.

Foveated rendering provides an idea for accelerating rendering algorithms without sacrificing the perceived rendering quality in virtual reality applications. In this paper, we propose a foveated stochastic lightcuts method to render high-quality many-lights illumination effects in high perception-sensitive regions. First, we introduce a spatiotemporal-luminance based lightcuts generation method to generate lightcuts with different accuracy for different visual perception-sensitive regions. Then we propose a multi-resolution light samples selection method to select the light sample for each node in the lightcuts more efficiently. Our method supports full-dynamic scenes containing over 250k dynamic light sources and dynamic diffuse/specular/glossy objects. It provides frame rates up to 110fps for high-quality many-lights illumination effects in high perception-sensitive regions of the HVS in VR HMDs. Compared with the state-of-the-art stochastic lightcuts method using the same rendering time, our method achieves smaller mean squared errors in the fovea and periphery. We also conduct user studies to prove that the perceived quality of our method has a high visual similarity with the results of the ground truth rendered by using the stochastic lightcuts with 2048 light samples per pixel.

This article presents an occlusion management approach that handles fine-grain occlusions, and that quantifies and localizes occlusions as a user explores a virtual environment (VE). Fine-grain occlusions are handled by finding the VE region where they occur, and by constructing a multiperspective visualization that lets the user explore the region from the current location, with intuitive head motions, without first having to walk to the region. VE geometry close to the user is rendered conventionally, from the user's viewpoint, to anchor the user, avoiding disorientation and simulator sickness. Given a viewpoint, residual occlusions are quantified and localized as VE voxels that cannot be seen from the given viewpoint but that can be seen from nearby viewpoints. This residual occlusion quantification and localization helps the user ascertain that a VE region has been explored exhaustively. The occlusion management approach was tested in three controlled studies, which confirmed the exploration efficiency benefit of the approach, and in perceptual experiments, which confirmed that exploration efficiency does not come at the cost of reducing spatial awareness and sense of presence, or of increasing simulator sickness.

We introduce a collaborative manipulation method to improve the efficiency and accuracy of object manipulation in virtual reality applications with multiple users. When multiple users manipulate an object in collaboration, a certain user may have a better perspective than other users at a certain moment, and can clearly observe the object to be manipulated and the target position, and it is more efficient and accurate for him to manipulate the object. We construct a viewpoint quality function and evaluate the viewpoints of multiple users by calculating its three components: the visibility of the object need to be manipulated, the visibility of target, the depth and distance combined of the target. By comparing the viewpoint quality of multiple users, the user with the highest viewpoint quality is determined as the dominant manipulator, who can manipulate the object at the moment. A temporal filter is proposed to filter the dominant sequence generated by the previous frames and the current frame, which reduces the dominant manipulator jumping back and forth between multiple users in a short time slice, making the determination of the dominant manipulator more stable. We have designed a user study and tested our method with three multi-user collaborative manipulation tasks. Compared to the previous methods, our method showed significant improvement in task completion time, rotation accuracy, user participation and task load.

With the popularity of using computer vision technology in monitoring system, there is an increasing societal concern on intruding people's privacy as the captured images/videos may contain identity-related information e.g. people's face. Existing methods on protecting such privacy focus on removing the identity-related information from faces. However, this would weaken the utility of current monitoring system. In this paper, we develop a face anonymization framework that could obfuscate visual appearance while preserving the identity discriminability. The framework is composed of two parts: an identity-aware region discovery module and an identity-aware face confusion module. The former adaptively locates the identity-independent attributes on human faces, and the latter generates the privacy-preserving faces using original faces and discovered facial attributes. To optimize the face generator, we employ a multi-task based loss function, which consists of discriminator loss, identify preserving loss, and reconstruction loss functions. Our model can achieve a balance between recognition utility and appearance anonymizing by modifying different numbers of facial attributes according to pratical demands, and provide a variety of results. Extensive experiments conducted on two public benchmarks Celeb-A and VGG-Face2 demonstrate the effectiveness of our model under distinct face recognition scenarios.

We introduce the disocclusion headlight, a method for VR selection assistance based on alleviating occlusions at the center of the user's field of view. The user's visualization of the VE is modified to reduce overlap between objects. This way, selection candidate objects have larger image footprints, which facilitates selection. The modification is confined to the center of the frame, with continuity to the periphery of the frame which is rendered conventionally. The selection assistance is provided automatically, without any interaction from the user. Furthermore, our method disoccludes without destroying the local spatial relationships between selection candidates, which allows solving complex selection queries based on the relative position of objects. We have tested our method on three selection tasks, where we compared it to two state-of-the-art VR selection techniques, i.e., the alpha cursor and the flower cone. Our method showed significant advantages in terms of shorter task completion times, and of fewer selection errors.

In this paper, we provide a bidirectional shadow rendering method to render shadows between real and virtual objects in the 360° videos in real time. We construct a 3D scene approximation from the current output viewpoint to approximate the real scene geometry nearby in the video. Then, we propose a ray casting based algorithm to determine the shadow regions on the virtual objects cast by the real objects. After that, we introduce an object-aware shadow mapping method to cast shadows from virtual objects to real objects. Finally, we use a shadow intensity estimation algorithm to determine the shadow intensity of virtual objects and real objects to obtain shadows consistent with the input 360° video. The experiment results prove the effectiveness of our bidirectional shadow rendering method for mixed 360° videos. Our method can generate visually realistic shadows for virtual objects and real objects in 360° video in realtime, and make virtual objects more natural to integrate with real scenes in 360° videos of the mixed reality applications.

Plant disease diagnosis is very critical for agriculture due to its importance for increasing crop production. Recent advances in image processing offer us a new way to solve this issue via visual plant disease analysis. However, there are few works in this area, not to mention systematic researches. In this paper, we systematically investigate the problem of visual plant disease recognition for plant disease diagnosis. Compared with other types of images, plant disease images generally exhibit randomly distributed lesions, diverse symptoms and complex backgrounds, and thus are hard to capture discriminative information. To facilitate the plant disease recognition research, we construct a new large-scale plant disease dataset with 271 plant disease categories and 220,592 images. Based on this dataset, we tackle plant disease recognition via reweighting both visual regions and loss to emphasize diseased parts. We first compute the weights of all the divided patches from each image based on the cluster distribution of these patches to indicate the discriminative level of each patch. Then we allocate the weight to each loss for each patch-label pair during weakly-supervised training to enable discriminative disease part learning. We finally extract patch features from the network trained with loss reweighting, and utilize the LSTM network to encode the weighed patch feature sequence into a comprehensive feature representation. Extensive evaluations on this dataset and another public dataset demonstrate the advantage of the proposed method. We expect this research will further the agenda of plant disease recognition in the community of image processing.

Virtual reality (VR) applications require high-performance rendering algorithms to efficiently render 3D scenes on the VR head-mounted display, to provide users with an immersive and interactive virtual environment. Foveated rendering provides a solution to improve the performance of rendering algorithms by allocating computing resources to different regions based on the human visual acuity, and renders images of different qualities in different regions. Rasterization-based methods and ray tracing methods can be directly applied to foveated rendering, but rasterization-based methods are difficult to estimate global illumination (GI), and ray tracing methods are inefficient for rendering scenes that contain paths with low probability. Photon mapping is an efficient GI rendering method for scenes with different materials. However, since photon mapping cannot dynamically adjust the rendering quality of GI according to the human acuity, it cannot be directly applied to foveated rendering. In this paper, we propose a foveated photon mapping method to render realistic GI effects in the foveal region. We use the foveated photon tracing method to generate photons with high density in the foveal region, and these photons are used to render high-quality images in the foveal region. We further propose a temporal photon management to select and update the valid foveated photons of the previous frame for improving our method's performance. Our method can render diffuse, specular, glossy and transparent materials to achieve effects specifically related to GI, such as color bleeding, specular reflection, glossy reflection and caustics. Our method supports dynamic scenes and renders high-quality GI in the foveal region at interactive rates.

A common approach for Augmented Reality labeling is to display the label text on a flag planted into the real world element at a 3D anchor point. When there are more than just a few labels, the efficiency of the interface decreases as the user has to search for a given label sequentially. The search can be accelerated by sorting the labels alphabetically, but sorting all labels results in long and intersecting leader lines from the anchor points to the labels. This paper proposes a partially-sorted concentric label layout that leverages the search efficiency of sorting while avoiding the label display problems of long or intersecting leader lines. The labels are partitioned into a small number of sorted sequences displayed on circles of increasing radii. Since the labels on a circle are sorted, the user can quickly search each circle. A tight upper bound derived from circular permutation theory limits the number of circles and thereby the complexity of the label layout. For example, 12 labels require at most three circles. When the application allows it, the labels are presorted to further reduce the number of circles in the layout. The layout was tested in a user study where it significantly reduced the label searching time compared to a conventional single-circle layout.

The redirected jumping (RDJ) technique is a new locomotion method that saves physical tracking area and enhances the body movement experience of users in virtual reality. In a previous study, the range of imperceptible manipulation gains in RDJ was discussed in an empty virtual environment (VE). In this study, we conducted three tasks to investigate the influence of alley width on the detection threshold of jump redirection in a VE. The results demonstrated that the imperceptible distance gain range in RDJ was not associated with the width of the alleys. The imperceptible height and rotation gain ranges in RDJ are related to the width of the alleys. We preliminarily summarized the relationship between the occlusion distance and manipulation range of the three gains in a complex environment. Simultaneously, the guiding principle for choosing three gains in RDJ according to the occlusion distance in a complex environment is provided.

When a user walks freely in an unknown virtual scene and searches for multiple dynamic targets, the lack of a comprehensive understanding of the environment may have a negative impact on the execution of virtual reality tasks. Previous studies can help users with auxiliary tools, such as top view maps or trails, and exploration guidance, for example, automatically generated paths according to the user location and important static spots in virtual scenes. However, in some virtual reality applications, when the scene has complex occlusions, and the user cannot obtain any real-time position information of the dynamic target, the above assistance cannot help the user complete the task more effectively. We design a virtual camera priority-based assistance to help the user search dynamic targets efficiently. Instead of forcing users to go to destinations, we provide an optimized instant path to guide them to places where they are more likely to find dynamic targets when they ask for help. We assume that a certain number of virtual cameras are fixed in virtual scenes to obtain extra depth maps, which capture the depth information of the scene and the locations of the dynamic targets. Our methodautomatically analyzes the priority of these virtual cameras, chooses the destination, and generates an instant path to assist the user in finding the dynamic targets. Our method is suitable for various virtual reality applications that do not require manual supervision or input. A user study is designed to evaluate the proposed method. The results indicate that compared with three conventional navigation methods, such as the top-view method, our method can help users find dynamic targets more efficiently. The advantages include reducing the task completion time, reducing the number of resets, increasing the average distance between resets, and reducing user task load. We presented a method for improving dynamic target searching efficiency in virtual scenes by virtual camera priority-based path guidance. Compared with three conventional navigation methods, such as the top-view method, this method can help users find dynamic targets more effectively.

Foveated rendering distributes computational resources based on visual acuity, more in the foveal regions of our eyes and less in the periphery. The traditional rasterization method can be adapted into the foveated rendering framework in a quite straightforward way, but it's difficult for estimating global illumination. Instant Radiosity is an efficient global illumination method. It generates Virtual Point Lights (VPLs) on the surface of the virtual scenes from light sources and uses these VPLs to simulate light bounces. However, instant radiosity can not be adapted into the foveated rendering pipeline directly, and is too slow for virtual reality experience. What's more, instant radiosity does not consider temporal coherence, therefore it lacks temporal stability for dynamic scenes. In this paper, we propose a foveated rendering method for instant radiosity with more accurate global illumination effects in the foveal region and less accurate global illumination in the peripheral region. We define a foveated importance for each VPL, and use it to smartly distribute the VPLs to guarantee the rendering precision of the foveal region. Meanwhile, we propose a novel VPL reuse scheme, which updates only a small fraction of VPLs over frames, which ensures temporal coherence and improves time efficiency. Our method supports dynamic scenes and achieves high quality in the foveal regions at interactive frame rates.

In a co-located multi-user collaborative virtual reality (VR) application a collaborator should be able to indicate a workspace location to the user, such that they can refer to it simultaneously as they work together. Due to their different viewpoints, the collaborator sees some parts of the virtual environment (VE) that the user does not, and communication breaks down when the collaborator's reference is not visible to the user. The conventional solutions of asking the user to move around to gain line of sight to the collaborator's reference, or of asking the user to toggle back and forth between their view and that of the collaborator can be inefficient and ineffective. This paper proposes a method for improving collaboration in VR by alleviating the disparity between the user and the collaborator views of the VE. The user is shown a multiperspective visualization of the VE that transitions smoothly from the user to the collaborator's perspective. The multiperpsective visualization is based on the switch camera, a novel camera model with curved rays that splice together the user and collaborator views. The multiperspective visualization is computed first by warping the VE geometry, through projection with the switch camera followed by unprojection with a conventional camera, and then by rendering the warped VE conventionally, for each user eye. A controlled user study with three tasks shows that VR collaboration using the switch camera multiperpsective visualization is faster, more reliable, and less taxing on the user than the conventional approaches of viewpoint translation or view toggling.

Path redirection for virtual reality (VR) navigation allows the user to explore a large virtual environment (VE) while the VR application is hosted in a limited physical space. Static mapping redirection methods deform the virtual scene to fit the physical space. The challenge is to deform the virtual scene in a reasonable way, making the distortions friendly to the user’s visual perception. In this paper we propose a feature-guided path redirection method that finds and takes into account the visual features of 3D virtual scenes. In a first offline step, a collection of view-independent and view-dependent visual features of the VE are extracted and stored in a visual feature map. Then, in a second offline step, the navigation path is deformed to fit in the confines of the available physical space through a mass-spring system optimization, according to distortion sensitive factors derived from the visual feature map. Finally, a novel detail preserving rendering algorithm is employed to preserve the original visual detail as the user navigates the VE on the redirected path. We tested our method on several scenes, where our method showed a reduced VE 3D mesh distortion, when compared to the path redirection methods without feature guidance.

VR applications rely on the user's ability to explore the virtual scene efficiently. In complex scenes, occlusions limit what the user can see from a given location, and the user has to navigate the viewpoint around occluders to gain line of sight to the hidden parts of the scene. When the disoccluded regions prove to be of no interest, the user has to retrace their path, making scene exploration inefficient. Furthermore, the user might not be able to assume a viewpoint that would reveal the occluded regions due to physical limitations, such as obstacles in the real world hosting the VR application, viewpoints beyond the tracked area, or viewpoints above the user's head that cannot be reached by walking. Several occlusion management methods have been proposed in visualization research, such as top view, X-ray, and multiperspective visualization, which help the user see more from the current position, having the potential to improve the exploration efficiency of complex scenes. This paper reports on a study that investigates the potential of these three occlusion management methods in the context of VR applications, compared to conventional navigation. Participants were required to explore two virtual scenes to purchase five items in a virtual Supermarket, and to find three people in a virtual parking garage. The task performance metrics were task completion time, total distance traveled, and total head rotation. The study also measured user spatial awareness, depth perception, and simulator sickness. The results indicate that users benefit from top view visualization which helps them learn the scene layout and helps them understand their position within the scene, but the top view does not let the user find targets easily due to occlusions in the vertical direction, and due to the small image footprint of the targets. The X-ray visualization method worked better in the garage scene, a scene with a few big occluders and a low occlusion depth complexity' and less well in the Supermarket scene, a scene with many small occluders that create high occlusion depth complexity. The multi-perspective visualization method achieves better performance than the top view method and the X-ray method, in both scenes. There are no significant differences between the three methods and the conventional method in terms of spatial awareness, depth perception, and simulator sickness.

We present a method for the fast computation of the intersection between a ray and the geometry of a scene. The scene geometry is simplified with a 2D array of voxelizations computed from different directions, sampling the space of all possible directions. The 2D array of voxelizations is compressed using a vector quantization approach. The ray-scene intersection is approximated using the voxelization whose rows are most closely aligned with the ray. The voxelization row that contains the ray is looked up, the row is truncated to the extent of the ray using bit operations, and a truncated row with non-zero bits indicates that the ray intersects the scene. We support dynamic scenes with rigidly moving objects by building a separate 2D array of voxelizations for each type of object, and by using the same 2D array of voxelizations for all instances of an object type. We support complex dynamic scenes and scenes with deforming geometry by computing and rotating a single voxelization on the fly. We demonstrate the benefits of our method in the context of interactive rendering of scenes with thousands of moving lights, where we compare our method to ray tracing, to conventional shadow mapping, and to imperfect shadow maps.

Virtual Reality (VR) applications allow a user to explore a scene intuitively through a tracked head-mounted display (HMD). However, in complex scenes, occlusions make scene exploration inefficient, as the user has to navigate around occluders to gain line of sight to potential regions of interest. When a scene region proves to be of no interest, the user has to retrace their path, and such a sequential scene exploration implies significant amounts of wasted navigation. Furthermore, as the virtual world is typically much larger than the tracked physical space hosting the VR application, the intuitive one-to-one mapping between the virtual and real space has to be temporarily suspended for the user to teleport or redirect in order to conform to the physical space constraints. In this paper we introduce a method for improving VR exploration efficiency by automatically constructing a multiperspective visualization that removes occlusions. For each frame, the scene is first rendered conventionally, the z-buffer is analyzed to detect horizontal and vertical depth discontinuities, the discontinuities are used to define disocclusion portals which are 3D scene rectangles for routing rays around occluders, and the disocclusion portals are used to render a multiperpsective image that alleviates occlusions. The user controls the multiperspective disocclusion effect, deploying and retracting it with small head translations. We have quantified the VR exploration efficiency brought by our occlusion removal method in a study where participants searched for a stationary target, and chased a dynamic target. Our method showed an advantage over conventional VR exploration in terms of reducing the navigation distance, the view direction rotation, the number of redirections, and the task completion time. These advantages did not come at the cost of a reduction in depth perception or situational awareness, or of an increase in simulator sickness.

The papers in this special section were presented at the 2015 ACM Symposium on Virtual Reality Software and Technology (VRST’15).

We present a novel direct lighting rendering method for static scenes with meso-structured surfaces and multiple area light sources. We use coarse geometry models with textures bound on them as inputs. The textures contain not only color maps, but also height field maps to represent the small concave and convex details for meso-structured surfaces. We explore traditional radiosity process as the main frame of our algorithm, and extend radiosity to directional radiosity for approximating light transportation on meso-structured surfaces. In pre-processing step, first, the form factors between the triangles of coarse models and the patches of area light sources are pre-computed. Then the directional radiosity of each triangle is computed by gathering irradiance according to directions sampled uniformly on the hemi-sphere of them. At last, directional radiosity of a vertex is estimated according to the directional radiosity of triangles passing through this vertex. For each frame, we combine relief texture mapping with directional radiosity to shade meso-structured surface. The experiment results show that our method has very similar quality and better time performance for rendering the scenes with meso-structured surfaces and multiple area light sources compared with SM+RTM method.

Abstract We present a soft shadow rendering algorithm that is general, efficient and accurate. The algorithm supports fully dynamic scenes, with moving and deforming blockers and receivers, and with changing area light source parameters. For each output image pixel, the algorithm computes a tight but conservative approximation of the set of triangles that block the light source as seen from the pixel sample. The set of potentially blocking triangles allows estimating visibility between light points and pixel samples accurately and efficiently. As the light source size decreases to a point, our algorithm converges to rendering pixel accurate hard shadows.

ABSTRACT This paper advocates a method for real‐time physical deformation and arbitrary cutting simulation of heterogeneous objects with multi‐material distribution, whose originality centers on the tight coupling of domain‐specific finite element method (FEM) and material distance‐aware meshless approach in a CUDA‐centric parallel simulation framework. We employ hierarchical hexahedron serving as basic building blocks for accurate material‐aware FEM simulation. Meanwhile, local meshless systems are designed to support cross‐FEM‐domain coupling and material‐sensitive propagation while respecting the regularity of finite elements. Directly benefiting from the structural regularity and uniformity of finite elements, our hybrid solution enables the local stiffness matrix pre‐computation and dynamic assembling, adaptive topological updating and precise cutting reconstruction. Moreover, our mathematically‐rigorous solver guarantees unconditional stableness. Experiments demonstrate the superiorities of our system. Copyright © 2014 John Wiley & Sons, Ltd.

It is hard to efficiently model the light transport in scenes with translucent objects for interactive applications. The inter-reflection between objects and their environments and the subsurface scattering through the materials intertwine to produce visual effects like color bleeding, light glows, and soft shading. Monte-Carlo based approaches have demonstrated impressive results but are computationally expensive, and faster approaches model either only inter-reflection or only subsurface scattering. In this paper, we present a simple analytic model that combines diffuse inter-reflection and isotropic subsurface scattering. Our approach extends the classical work in radiosity by including a subsurface scattering matrix that operates in conjunction with the traditional form factor matrix. This subsurface scattering matrix can be constructed using analytic, measurement-based or simulation-based models and can capture both homogeneous and heterogeneous translucencies. Using a fast iterative solution to radiosity, we demonstrate scene relighting and dynamically varying object translucencies at near interactive rates.

The feed-forward pipeline based on projection followed by rasterization handles the rays that leave the eye efficiently: these first-order rays are modeled with a simple camera that projects geometry to screen. Second-order rays however, as, for example, those resulting from specular reflections, are challenging for the feed-forward approach. We propose an extension of the feed-forward pipeline to handle second-order rays resulting from specular and glossy reflections. The coherence of second-order rays is leveraged through clustering, the geometry reflected by a cluster is approximated with a depth image, and the color samples captured by the second-order rays of a cluster are computed by intersection with the depth image. We achieve quality specular and glossy reflections at interactive rates in fully dynamic scenes.

Compression of 3D dynamic datasets in remote visualization still remains two challenges. One is low time performance due to the grown data and complex computation of compression algorithm. Another is small compression factor because of dynamic scenes without known equations of their motions. In this paper, we propose a fast and compact compression for 3D dynamic datasets. It accelerates compression with KD-tree construction and node-grid mapping for the dynamic data, which allow parallel rigid body decomposition and merging with disjoint union method. To increase the compression factor, composite rigid body is introduced with consideration of temporary motion consistency among rigid bodies. The results of the experiments show that our algorithm can compress dynamic datasets quickly and obtain high compression factor to reduce limitation of bandwidth.

This paper proposes a parallel and direct semantic feature curve extraction method from 3D volume image for vectorization and illustration. Our approach is motivated by reconstructing 3D geometric information from multiple rendered images under multi-view in computer vision. The 2D rendered images are rich in the visual sense by color and opacity that convey the structure of volume data, so it is significant for the user to understand the structure of 3D volume data better if we can recover feature curves from those 2D images. Compared with conventional line extraction methods, which mainly focus on extracting feature curves from iso-surfaces in object space, we extract feature curves directly from volume images. Most of the computation can be computed in parallel on GPU with CUDA acceleration.

Light propagation in scenes with translucent objects is hard to model efficiently for interactive applications. The inter-reflections between objects and their environments and the subsurface scattering through the materials intertwine to produce visual effects like color bleeding, light glows and soft shading. Monte-Carlo based approaches have demonstrated impressive results but are computationally expensive, and faster approaches model either only inter-reflections or only subsurface scattering. In this paper, we present a simple analytic model that combines diffuse inter-reflections and isotropic subsurface scattering. Our approach extends the classical work in radiosity by including a subsurface scattering matrix that operates in conjunction with the traditional form-factor matrix. This subsurface scattering matrix can be constructed using analytic, measurement-based or simulation-based models and can capture both homogeneous and heterogeneous translucencies. Using a fast iterative solution to radiosity, we demonstrate scene relighting and dynamically varying object translucencies at near interactive rates.

Abstract Texture bombing is a texture synthesis approach that saves memory by stopping short of assembling the output texture from the arrangement of input texture patches; instead, the arrangement is used directly at run time to texture surfaces. However, several problems remain in need of better solutions. One problem is improving texture diversification. A second problem is that mipmapping cannot be used because texel data is not stored explicitly. The lack of an appropriate level‐of‐detail (LoD) scheme results in severe minification artefacts. We present a just‐in‐time texturing method that addresses these two problems. Texture diversification is achieved by modelling a texture patch as an umbrella, a versatile hybrid 3‐D geometry and texture structure with parameterized appearance. The LoD is adapted continuously with a hierarchical algorithm that acts directly on the arrangement map. Results show that our method can model and render the diversity present in nature with only small texture memory requirements.

Troubleshooting unresponsive sensor nodes is a significant challenge in remote sensor network deployments. This paper introduces the tele-diagnostic powertracer, an in-situ troubleshooting tool that uses external power measurements to determine the internal health condition of an unresponsive host and the most likely cause of its failure. We developed our own low-cost power meter with low-bandwidth radio to report power measurements and findings, hence allowing remote (i.e., tele-) diagnosis. The tool was deployed and tested in a remote solar-powered sensing network for acoustic and visual environmental monitoring. It was shown to successfully distinguish between several categories of failures that cause unresponsive behavior including energy depletion, antenna damage, radio disconnection, system crashes, and anomalous reboots. It was also able to determine the internal health conditions of an unresponsive node, such as the presence or absence of sensing and data storage activities (for each of multiple sensors). The paper explores the feasibility of building such a remote diagnostic tool from the standpoint of economy, scale and diagnostic accuracy. To the authors' knowledge, this is the first paper that presents a remote diagnostic tool that uses power measurements to diagnose sensor system failures.

In this paper, we present a reliable storage service, called SolarStore, that adaptively trades-off storage reliability versus energy consumption in solar-powered sensor networks. SolarStore adopts a predominantly disconnected network model, where long-running data-collection experiments are conducted in the absence of a continuous connection to the outside world. SolarStore (i) replicates data in the network until the next upload opportunity, and (ii) adapts the degree of data replication dynamically depending on solar energy and storage availability. The goal is to maximize the amount of data that can eventually be retrieved from the network subject to energy and storage constraints. Maximization of retrievable data implies minimizing sensing blackouts due to energy depletion as well as minimizing loss due to node damage in harsh environmental conditions. We have deployed an outdoor solar-powered sensor network, on which SolarStore is implemented and tested. An indoor testbed is also set up for performance evaluation under environmental conditions not attained locally. Experiments show that SolarStore is successful in dynamically responding to variations in the environment in a manner that increases retrievable data under different node failure scenarios.

In this paper, we present AdaptSens: a reliable data collection and storage system for solar-powered sensor networks. Unlike battery-operated devices, solar-powered systems have a less predictable energy supply and their ability to harvest energy depends on past spending, thereby creating incentives for adaptive matching of energy supply and demand. Our storage system is novel in its layered architecture and its incremental layer activation mechanism. AdaptSens provides a set of functions, in separate layers, such as sensory data collection, replication (to prevent failure-induced data loss), and storage balancing (to prevent depletion-induced data loss). The mechanism utilizes surplus energy when available by activating more layers, and resorts to progressively more energy-efficient (partial hibernation) modes when energy is scarce. Best reliability is achieved when all layers are active but meaningful intermediate modes allow different degrees of energy conservation. The efficacy of AdaptSens in trading off reliability for energy is tested on both an outdoor system and an indoor testbed. Evaluation results show that AdaptSens minimizes the sum of all data losses when combining the energy, storage and node failure factors.

This article presents the design, implementation, and evaluation of EnviroMic , a low-cost experimental prototype of a novel distributed acoustic monitoring, storage, and trace retrieval system designed for disconnected operation. Our intended use of acoustic monitoring is to study animal populations in the wild. Since a permanent connection to the outside world is not assumed and due to the relatively large size of audio traces, the system must optimally exploit available resources such as energy and network storage capacity. Towards that end, we design, prototype, and evaluate distributed algorithms for coordinating acoustic recording tasks, reducing redundancy of data stored by nearby sensors, filtering out silence, and balancing storage utilization in the network. For experimentation purposes, we implement EnviroMic on a TinyOS-based platform and systematically evaluate its performance through both indoor testbed experiments and an outdoor deployment. Results demonstrate up to a four-fold improvement in effective storage capacity of the network compared to uncoordinated recording.

This is a new method to avoid crack in terrain rendering. It does not use the traditional technique to deal with crack in geometry space. It draws terrain to a texture using FBO (frame buffer object) instead of to screen directly without operation to dispose seam. Then, it uses "inpainting", a technique in image processing, to treat the texture in image space, so that the seamless image rendering can be realized. As for rendering terrain, it uses a quad tree to manage terrain data, and preserves the values of height in a texture. Then it can fetch them from video memory. In rendering a node, it uses geomorphing operation to avoid popping.