Intrinsic image decomposition decomposes an image into reflectance and shading. It has been applied in image editing, augmented reality, and geometry estimation. However, the complete decoupling between reflectance and shading, as well as the consistency of the reconstructed image with the original image, have become the main challenges in the application of intrinsic image decomposition. To improve the performance of the intrinsic image decomposition algorithm for these two challenges, we propose a novel deep learning framework that works separately to learn features unique to different intrinsic images. Based on this framework, we developed more effective loss functions to strengthen the decoupling of reflectance and shading and to maintain the decomposition without losing as much information of the original image as possible. We trained the network on a mixture of synthetic and real datasets and evaluated the results of the experiments on real datasets. The results show that our proposed method not only outperformed existing state-of-the-art methods in qualitative and quantitative comparisons in terms of reflectance but was also competitive in terms of reconstructed consistency and shading. Finally, we implemented several realistic image-editing applications, and the results were visually superior to other results.

In educational Virtual Reality (VR) environments, objects irrelevant to learning can lead to students' inattention, which adversely affects learning. However, removing these objects from virtual scenes is not feasible, as they are crucial for creating a realistic and immersive experience. Balancing the need to maintain students' attention while preserving the integrity of scenarios is a challenging task. In this paper, we introduce a gaze-contingent rendering (GCR) technique to address such an issue, which is independent of specific elements or configurations in virtual scenes and adaptable across various contexts. Specifically, we utilize gaze-aware rendering adjustments to adaptively reduce the visibility of objects irrelevant to learning while highlighting relevant ones. We develop three GCR strategies (i.e., blur, pixelation, and underexposure) and investigate how these strategies affect students' visual attention, academic achievement, and perceptions of the learning activity across different scenarios. Our findings indicate that the proposed rendering strategies effectively achieve the goals of sustaining visual attention and improving academic achievement without significantly impacting immersion or engagement. As an initial exploration of GCR for maintaining attention within educational VR, this study may inspire new directions in future research on GCR and visual attention maintenance in immersive VR.

Current virtual reality (VR) system takes gesture interaction based on camera, handle and touch screen as one of the mainstream interaction methods, which can provide accurate gesture input for it. However, limited by application forms and the volume of devices, these methods cannot extend the interaction area to such surfaces as walls and tables. To address the above challenge, we propose AudioGest, a portable, plug-and-play system that detects the audio signal generated by finger tapping and sliding on the surface through a set of microphone devices without extensive calibration. First, an audio synthesis-recognition pipeline based on micro-contact dynamics simulation is constructed to generate modal audio synthesis from different materials and physical properties. Then the accuracy and effectiveness of the synthetic audio are verified by mixing the synthetic audio with real audio proportionally as the training sets. Finally, a series of desktop office applications are developed to demonstrate the application potential of AudioGest's scalability and versatility in VR scenarios.

Intrinsic decomposition separates an image into reflectance and shading, which contributes to image editing, augmented reality, etc. Despite recent efforts dedicated to this field, effectively separating colored illumination from reflectance and correctly restoring it into shading remains an challenge. We propose a deep intrinsic decomposition method to address this issue. Specifically, by transforming intrinsic decomposition process in RGB image domains into the combination of intensity and chromaticity domains, we propose a novel macro intrinsic decomposition network framework. This framework enables the generation of finer intrinsic components through more relevant features propagation and more detailed sub-constraints guidance. In order to expand the macro network, we integrate multiple attention mechanism modules in key positions of encoders, which enhances the extraction of distinct features. We also propose a skip connection module based on specific deep features guidance, which can filter out features that are physically irrelevant to each intrinsic component. Our method not only outperforms state-of-the-art methods across multiple datasets, but also robustly separates illumination from reflectance and restores it into shading in various types of images. By leveraging our intrinsic images, we achieve visually superior image editing effects compared to other methods, while also being able to manipulate the inherent lighting of the original scene.

Intrinsic decomposition, the process of decomposing an image into reflectance and shading, is widely used in virtual and augmented reality tasks. Reflectance and shading often exhibit large gradients at the object edges, and the intrinsic properties on the same object tend to be similar. This spatial coherence is closely related to semantic consistency because objects within the same semantic category often exhibit similar intrinsic properties. Therefore, incorporating semantic segmentation into a deep intrinsic decomposition framework helps the network distinguish between different object instances and understand high-level scene structures. To this end, we design an intrinsic decomposition network jointly trained with a dedicated semantic segmentation module, allowing semantic cues to enhance the decomposition of reflectance and shading. The semantic module provides guidance during training but is removed during inference, improving performance without increasing the inference cost. Additionally, to capture the global contextual dependencies critical for intrinsic decomposition, we adopt a Transformer-based backbone. The proposed backbone enables the model to associate distant regions with similar material properties, thereby maintaining consistency in reflectance and learning smooth illumination patterns across a scene. A convolutional decoder is also designed to output predictions with improved details. Experiments demonstrate that our approach achieves state-of-the-art performance in the quantitative evaluations on the Intrinsic Images in the Wild (IIW) and Shading Annotations in the wild (SAW) datasets.