Images are often obstructed by various obstacles due to capture limitations, hindering the observation of objects of interest. Most existing methods address occlusions from specific elements like fences or raindrops, but are constrained by the wide range of real-world obstructions, making comprehensive data collection impractical. To overcome these challenges, we propose Instruct2See, a novel zero-shot framework capable of handling both seen and unseen obstacles. The core idea of our approach is to unify obstruction removal by treating it as a soft-hard mask restoration problem, where any obstruction can be represented using multi-modal prompts, such as visual semantics and textual instructions, processed through a cross-attention unit to enhance contextual understanding and improve mode control. Additionally, a tunable mask adapter allows for dynamic soft masking, enabling real-time adjustment of inaccurate masks. Extensive experiments on both in-distribution and out-of-distribution obstacles show that Instruct2See consistently achieves strong performance and generalization in obstruction removal, regardless of whether the obstacles were present during the training phase. Code and dataset are available at https://jhscut.github.io/Instruct2See.

Monocular depth prediction has received significant attention in recent years. However, the impact of illumination variations, which can shift scenes to unseen domains, has often been overlooked. To address this, we introduce the first indoor scene dataset featuring RGB-D images captured under multiple illumination conditions, allowing for a comprehensive exploration of indoor depth prediction. Additionally, we propose a novel method, MI-Transformer, which leverages global illumination understanding through large receptive fields to capture depth-attention contexts. This enables our network to overcome local window limitations and effectively mitigate the influence of changing illumination conditions. To evaluate the performance and robustness, we conduct extensive qualitative and quantitative analyses on both the proposed dataset and existing benchmarks, comparing our method with state-of-the-art approaches. The experimental results demonstrate the superiority of our method across various metrics, making it the first solution to achieve robust monocular depth estimation under diverse illumination conditions. We provide the codes, pre-trained models, and dataset openly accessible athttps://github.com/ViktorLiang/midepth.

In this study, we present a new and innovative framework for acquiring high-quality SVBRDF maps. Our approach addresses the limitations of the current methods and proposes a new solution. The core of our method is a simple hardware setup consisting of a consumer-level camera, LED lights, and a carefully designed network that can accurately obtain the high-quality SVBRDF properties of a nearly planar object. By capturing a flexible number of images of an object, our network uses different subnetworks to train different property maps and employs appropriate loss functions for each of them. To further enhance the quality of the maps, we improved the network structure by adding a novel skip connection that connects the encoder and decoder with global features. Through extensive experimentation using both synthetic and real-world materials, our results demonstrate that our method outperforms previous methods and produces superior results. Furthermore, our proposed setup can also be used to acquire physically based rendering maps of special materials.

Abstract The traditional microfacet rendering models usually only consider the straight propagation of light and do not take into account the diffraction effect when calculating the radiance of outgoing light. However, ignoring the energy generated by diffraction can lead to darker rendering results when the object's surface has many small details. To address this issue, we introduce a diffraction energy term in the microfacet model to compensate for the energy loss caused by diffraction. Starting from the Fresnel‐Kirchhoff diffraction theorem, we combine it with the Cook‐Torrance model. By incorporating the computed diffraction radiance into the outgoing radiance of the microfacet, we obtain a diffraction‐compensated BRDF (Bidirectional Reflectance Distribution Function) model. Experimental results demonstrate that our proposed method has a significant effect in compensating for outgoing light and produces more realistic rendering results.

Reduced-order simulation is an emerging method for accelerating physical simulations with high DOFs, and recently developed neural-network-based methods with nonlinear subspaces have been proven effective in diverse applications as more concise subspaces can be detected. However, the complexity and landscape of simulation objectives within the subspace have not been optimized, which leaves room for enhancement of the convergence speed. This work focuses on this point by proposing a general method for finding optimized subspace mappings, enabling further acceleration of neural reduced-order simulations while capturing comprehensive representations of the configuration manifolds. We achieve this by optimizing the Lipschitz energy of the elasticity term in the simulation objective, and incorporating the cubature approximation into the training process to manage the high memory and time demands associated with optimizing the newly introduced energy. Our method is versatile and applicable to both supervised and unsupervised settings for optimizing the parameterizations of the configuration manifolds. We demonstrate the effectiveness of our approach through general cases in both quasi-static and dynamics simulations. Our method achieves acceleration factors of up to 6.83 while consistently preserving comparable simulation accuracy in various cases, including large twisting, bending, and rotational deformations with collision handling. This novel approach offers significant potential for accelerating physical simulations, and can be a good add-on to existing neural-network-based solutions in modeling complex deformable objects.

Abstract In this article, we propose a novel cloth simulation method based on the material point method (abbr. MPM). The response to the cloth strain is characterized by three separated models: the continuous in‐plane hyperelasticity, the discrete bending elastoplasticity, and the continuous frictional contact elastoplasticity. We make use of the moving least squares MPM (abbr. MLS‐MPM) transfer scheme and a discrete bending model to reduce computation overhead. We further extend the hyperelastic bending model to be elastoplastic, which can be further used for the simulation of a wide range of thin shell materials, such as thin metal sheets. A fast and approximate signed distance field generation method for humanoid animation is also proposed. Experimental results demonstrate that our method can efficiently and robustly simulate high‐resolution challenging scenes with hundreds of thousands of triangles.

Using the raw data from consumer-level RGB-D cameras as input, we propose a deep-learning-based approach to efficiently generate RGB-D images with completed information in high resolution. To process the input images in low resolution with missing regions, new operators for adaptive convolution are introduced in our deep-learning network that consists of three cascaded modules—the completion module, the refinement module, and the super-resolution module. The completion module is based on an architecture of encoder–decoder, where the features of input raw RGB-D will be automatically extracted by the encoding layers of a deep neural network. The decoding layers are applied to reconstruct the completed depth map, which is followed by a refinement module to sharpen the boundary of different regions. For the super-resolution module, we generate RGB-D images in high resolution by multiple layers for feature extraction and a layer for upsampling. Benefited from the adaptive convolution operators proposed in this article, our results outperform the existing deep-learning-based approaches for RGB-D image complete and super-resolution. As an end-to-end approach, high-fidelity RGB-D images can be generated efficiently at the rate of 22 frames/s.Note to Practitioners—With the development of consumer-level RGB-D cameras, industries have started to employ these low-cost sensors in many robotic and automation applications. However, images generated by consumer-level RGB-D cameras are generally in low resolution. Moreover, the depth images often have incomplete regions when the surface of an object is transparent, highly reflective, or beyond the distance of sensing. With the help of our method, engineers are able to “repair” the images captured by consumer-level RGB-D cameras in high efficiency. As the typical deep-learning networks are employed in this approach, the proposed approach fits well with the GPU-based hardware architecture of deep-learning computation—therefore, it potentially can be integrated into the hardware of RGB-D cameras.

Abstract Hair modeling plays an important role in computer animation, virtual reality, and other applications. This paper proposes an encoder‐decoder network, named HAO‐CNN, to recover 3D hair strand models from a single image. Specifically, HAO‐CNN generates a volumetric vector field (VVF) from the oriented map of hairstyles. However, instead of directly working on the full resolution VVFs, we introduce the adapted O‐CNN to predict the adaptive representation of VVFs in order to greatly reduce the memory cost. In addition, we fuse the features from different layers of the encoding stage for both capturing the global structure and being aware of hair filaments. Considering the difficulty of acquiring true three‐dimensional (3D) hair models, we augment the dataset with 340 3D hair models by 1,800 hair models via interactive editing using the software and render their oriented maps as training data. Then given a hair photo associated with human head, we segment out the hair region, compute its two‐dimensional oriented map using Gabor filter, and feed it into the network to produce a hair volumetric vector field which is then converted into hairline models using an improved VVF‐to‐strands algorithm. This greatly decreases the time cost of approaches based on volumetric vector fields.

With the rapid development of the display technologies, 3D shape data is becoming another important media kind. However, most of the existing 3D shape acquisition methods are either expensive or expertise-dependent. In this paper, we present an image representation for the 3D faces to bridge the gap between the feature-lacking 3D shapes and the powerful deep neural network learning tools. To achieve this, with the training set, we first extract the radial curves for each 3D face, and reform the curves into an image matrix, which enable to apply the classical Generative Adversarial Network model for the image synthesis. Finally, we propose a refining process to transform the output images into 3D synthetic faces. Our experimental results demonstrate the capability of our method which can correctly reflect the affinities among the different facial expressions and can generate the 3D faces.

Abstract Cage‐based deformation has become a popular method for shape deformation in computer graphics and animation. To edit a shape first requires a cage to be built to envelop the target model which is a tedious work by manual approaches. In this paper, we develop an automatic method to generate the cage for a model using voxelization based decomposition. We first voxelize the input model, and then use the seed filling algorithm to group the inner voxels. By dilating the inner voxel groups, we decompose the model into broad regions and narrow regions. Then we construct partial cages using different strategies and unite them to get a cage. Experiment results demonstrate that our method is effective, efficient as well as robust to model transformation. Copyright © 2014 John Wiley & Sons, Ltd.

In this paper, we present an approach to semantic cage generating and semantic cage based finite element mesh editing to allow users to conduct the required editing on complex finite element mesh effectively and efficiently through cage-based deformation during product design. In the approach, design semantics involved in a finite element mesh are incorporated with its cage to enable the cage based and semantic based editing of the finite element mesh. Firstly, the design semantics including semantic features and semantic relations are extracted from the original mesh, and a simplified mesh with the semantics preserved is constructed. Then, the semantic cage without cage-model intersection is automatically created by offsetting the simplified mesh and mapping the semantics from the original mesh to the cage. Finally, the editing manipulation is directly imposed on the semantic features rather than single vertex or face of the cage to achieve the required modification on the mesh. The experimental results show the effectiveness and potentials of the proposed approach.

Volumetric mesh models are widely used nowadays in areas like product quality evaluation, physically-based deformation, numerical simulation, and so on. In the application of Computer Aided Design (CAD) and Computer Aided Engineering (CAE) integration, providing a method to directly manipulate mesh models can reduce much effort in the simulation process. However, this integration is limited or only provided from CAD to CAE at current time. In this paper, we propose a framework for volumetric mesh editing based on feature dimensions. In our framework, the volumetric mesh is first decomposed into volumetric features based on corresponding surface features. Then, random-walk based interpolation algorithm is applied to manipulation of local features. Our framework allows users to change the dimensions of the recognized features on the volumetric mesh directly. Then optimization, which is limited to local features, is employed to refine the tetrahedrons of the volumetric features once the quality of the elements does not meet the requirements. Experimental results show that the features hold precisely after the editing operations. Additionally, the quality of the elements keeps well after the tetrahedral mesh has been edited.

Mesh editing can provide various models for FEA-simulation in industry. This paper proposes a framework for FEA-mesh editing with feature constrained. In the framework, cage-based technique is first used to edit the base-decomposition model. Vertices of the constrained feature are transformed into a local form. Parameters are analyzed before editing operation, and our method permits the user to add constraints on the parameters of the feature. This framework can also keep consistence for the disconnected assembly mesh model. Experimental results show that constrained features are kept precisely after mesh editing. Additionally, experimental data indicate our method is efficient and achieves real-time response.

The coarse bounding cage of a dense mesh plays important roles in computer graphics, computer vision, and geometric design. Specifically, in volume-based deformation, a coarse bounding cage is required to manipulate the dense mesh model it enclosed; in subdivision surface fitting, the fitting starts from a coarse cage bounding the fitted dense mesh or point set; and so on. However, the generation of a coarse bounding cage is mainly by interactive ways, which are very tedious and time-consuming. In this paper, we develop a fully automatic method to generate a coarse cage bounding a dense mesh model. The automatically generated coarse bounding cage can keep the topological structure and major geometric features of the original mesh model, which is validated by theoretical analysis and experimental data presented in this paper. Further more, we employ the automatically generated coarse bounding cage in some applications, such as deformation, and subdivision fitting, producing good results.

Medial axis (MA) is widely used in many fields nowadays. However, the efficiency of generating MA of complicated models with current methods is still not satisfactory. A novel approach to constructively generate MA is proposed in this paper. With this method, the MA of the resultant model set up with two primitives and a Boolean operation upon them is generated by combining the MAs of the two primitives in a certain way instead of regenerating from scratch. First, the properties of MA are investigated. During Boolean operation, the boundaries that will vanish are found and the region of the model on the basis of which some new MA segments(MASs) need to be generated is determined, and the new MASs are generated based on the region using tracing method. The MA of the resultant model is constructed by combining the new generated MASs with the reserved MASs of the two primitives at last.