3D reconstruction faces challenges such as geometric warping and structural ambiguity, particularly in intricate topologies, heavy occlusions and complex backgrounds. These problems are partly attributed to excessive feature entanglement, which induces semantic confusion and spatial ambiguity. To address these limitations, we propose an occlusion sensitivity-guided semantic-disentangled Mamba-CNN network that enables human-controllable disentanglement of multi-attribute information within a bias-free semi-supervised framework. Specifically, we create various occlusion conditions and assign pseudo-labels to the augmented data within a semi-supervised framework, which enables the exploration of occlusion sensitivity of different semantic attributes for human-controllable semantic disentanglement. To reduce the bias between the augmented samples and their assigned pseudo-labels, we use linear PIoU and nonlinear MS-SSIM algorithms to calculate the confidence of pseudo-labels, which minimizes the error propagation caused by the bias. Then, we develop a disentangled multi-depth Mamba-CNN block that combines CNN’s local feature extraction capability with Mamba’s ability to capture long-range dependencies. This allows our model to effectively capture disentangled multi-attribute spatial features and semantic representations. However, critical cross-level semantic attribute connections could be lost in the disentanglement process. To tackle this, we propose a multi-attribute semantic query block to dynamically re-establish these connections and minimize cross-attribute information loss. Extensive quantitative and qualitative evaluations on object and face reconstruction demonstrate that our method outperforms existing state-of-the-art approaches. Codes and all data are publicly available at https://github.com/Ray-tju/sensitivity-guided-semantic-disentangled-Mamba-CNN.

Single-view three-dimensional (3D) object reconstruction has always been a long-term challenging task. Objects with complex topologies are hard to accurately reconstruct, which makes existing methods suffer from blurring of shape boundaries between multiple components in the object. Moreover, most of them cannot balance learning between global geometric structure information and local detail information. In this article, we propose a multi-scale edge-guided learning network (MEGLN) to utilize the global edge information guiding the network to better capture and recover local details. The goal is to exploit the multi-scale learning strategy to learn global edge information and local details, thus achieving robust 3D object reconstruction. We first design a multi-scale Gaussian difference block (MGDB) to extract global edge geometry features for input images of different scales and adopt the attention mechanism to aggregate the extracted global edge geometry features of different scales. Second, we design a multi-scale feature interaction block (MFIB) to learn local details, which utilizes the multi-scale feature interaction to capture the features of multiple objects or components at multiple scales. The MFIB can learn and capture better as much local detail information as possible under the guidance of global edge information. Finally, we dynamically fuse the predicted probabilities of the MGDB and MFIB to obtain the final predicted result, which makes our MEGLN able to recover 3D shapes with global complex topological structures and rich local details via the multi-scale learning strategy. Extensive qualitative and quantitative experimental results on the ShapeNet dataset demonstrate that our approach achieves competitive performance compared with state-of-the-art methods. Code is available at https://github.com/Ray-tju/MEGLN .

In this work, we propose Geometry Normal Consistency Loss (GNC) for 3D face reconstruction and dense alignment. The existing methods based on the strong constraints of the 3DMM parameter regression only consider reducing the error between 68 landmarks, while they rarely consider the geometric contour structure relation of the face. Instead, we take into account the discrete 68 landmarks as loss constrain by introducing geometry area and normal consistency loss, which naturally defines the holistic and local geometric contour structure of the face. In detail, we select the inverted triangle formed by the leftmost and rightmost landmarks on the cheek and the lowest point of the chin to globally constrain the entire facial feature. In addition, we triangulate the facial landmarks to construct triangular patches, and calculate the normals of the patches, which aims to make use of normal consistency between the local patch and the corresponding ground truth to reconstruct rich local details. Extensive experimental results on AFLW2000-3D and AFLW datasets demonstrate that our GNC achieves compelling performance compared to the state of the arts.

3D face reconstruction from a single-view image in the wild is a long-standing challenging problem. Traditional 3DMM-based methods directly regressed parameters, which probably caused that the network learned the discriminative informative features in the face insufficiently. In this paper, we propose a replay attention and data augmentation network (RADAN) for 3D dense alignment and face reconstruction. Instead of the traditional attention mechanism, our replay attention module aims to increase the sensitivity of the network to informative features by adaptively recalibrating the weight response in the attention mechanism, which typically reinforces the distinguishability of the learned feature representation. In this way, the network is able to further improve the accuracy of face reconstruction and dense alignment in an unconstrained environment. Moreover, to improve the generalization performance of the model and the ability of the network to capture local details, we present a data augmentation strategy to preprocess the sample data, which generates the images that contain more local details and occluded face in a cropping and pasting manner. Extensive qualitative and quantitative experimental results on widely-evaluated benchmarking datasets demonstrate that our approach achieves competitive performance compared to state-of-the-art methods. Code is available at https://github.com/zhouzhiyuanl/RADANet.

In this paper, we propose a multi-granularity feature interaction and relation reasoning network (MFIRRN) which can recover a detail-rich 3D face and perform more accurate dense alignment in an unconstrained environment. Traditional 3DMM-based methods directly regress parameters, resulting in the lack of fine-grained details in the reconstruction 3D face. To this end, we use different branches to capture discriminative features at different granularities, especially local features at medium and fine granularities. Meanwhile, the finer-grained branch network shares its information with the adjacent coarser-grained branch network to achieve feature interaction. Our model performs cross-granular information integration and inter-granular relationship reasoning to obtain prediction results. Extensive experiments on AFLW2000-3D and AFLW datasets demonstrate the validity of our method. The code is publicly available at https://github.com/leilimaster/MFIRRN.

Multi-view stereo based on deep learning is mostly dedicated to improving the accuracy of point clouds, whlile complex scenes, occlusion, and other factors limit their reconstruction completeness, especially in the area with drastic changes in depth direction. In this paper, we propose a multi-view stereo network based on depth edge flow (DEF-MVSNet), using the reference image as a guide to dynamically infer the edge coordinates to improve reconstruction completeness. First, we ignore the boundaries in the depth prediction stage to generate better initial depth inference results. Then, we use a EdgeDetect module to extract the obviously features of the reference image and predict the pixel offset of the depth map. Finally, the EdgeFlow module modifies the initial depth map coordinates according to the offset and uses multiple iterations to dynamically update the depth map. The experimental results prove that our method has a great improvement in the completeness of reconstruction compared with MVSNet and R-MVSNet without increasing the memory and time overhead. Code and models are publicly available at https://github.com/linkuizzZ/EF-MVSNet.

3D face reconstruction based on a single image is a longstanding challenging problem in computer vision. Existing end-to-end methods are difficult to reconstruct rich 3D face details. To solve this problem, we propose a two-stream convolutional neural network combined with a face super-resolution method, which can effectively restore the image’s 3D position information. Our method combines an attention fusion mechanism, which can learn the individual attention mapping of each feature subspace, and effectively learn cross-channel information while learning multi-scale and multi-frequency features. Meanwhile, our module obtains the most discriminative features in different local areas, and enhances the consistency and correlation between the attention areas. Experimental results show that our SRCNet has made significant improvements in the 3D face reconstruction and face alignment of the AFLW2000-3D and AFLW datasets.

3D object reconstruction from a single-view image is a long-standing challenging problem. Previous work was difficult to accurately reconstruct 3D shapes with a complex topology which has rich details at the edges and corners. Moreover, previous works used synthetic data to train their network, but domain adaptation problems occurred when tested on real data. In this paper, we propose a Dynamic Multi-branch Information Fusion Network (DmifNet) which can recover a high-fidelity 3D shape of arbitrary topology from a 2D image. Specifically, we design several side branches from the intermediate layers to make the network produce more diverse representations to improve the generalization ability of network. In addition, we utilize DoG (Difference of Gaussians) to extract edge geometry and corners information from input images. Then, we use a separate side branch network to process the extracted data to better capture edge geometry and corners feature information. Finally, we dynamically fuse the information of all branches to gain final predicted probability. Extensive qualitative and quantitative experiments on a large-scale publicly available dataset demonstrate the validity and efficiency of our method. Code and models are publicly available at https://github.com/leilimaster/DmifNet.