In this paper we study the task of a single-view image-guided point cloud completion. Existing methods have got promising results by fusing the information of image into point cloud explicitly or implicitly. However, given that the image has global shape information and the partial point cloud has rich local details, We believe that both modalities need to be given equal attention when performing modality fusion. To this end, we propose a novel dual-channel modality fusion network for image-guided point cloud completion(named DMF-Net), in a coarse-to-fine manner. In the first stage, DMF-Net takes a partial point cloud and corresponding image as input to recover a coarse point cloud. In the second stage, the coarse point cloud will be upsampled twice with shape-aware upsampling transformer to get the dense and complete point cloud. Extensive quantitative and qualitative experimental results show that DMF-Net outperforms the state-of-the-art unimodal and multimodal point cloud completion works on ShapeNet-ViPC dataset.

Computed tomography (CT) image processing plays a vital role in both clinical practice and scientific research, encompassing techniques such as low-dose CT enhancement, artifact removal, CT-MRI modality conversion, and more. Previous deep learning-based methods often require training specific networks from scratch for each task, which is resource-intensive in both the training and inference stages. In this context, we introduce CT-MIE, which offers a new paradigm for multi-task processing of CT images. We integrate task-specific textual prompts into the U-Net via a Text-Pixel Attention Pyramid decoder, which leverages BiomedCLIP to generate Text-Pixel Attention Maps and progressively refines resolution using a pyramid network for improved multi-task medical image processing. Additionally, the encoder incorporates a Hybrid Multilayer Perceptrons Window Self-Attention module, enhancing receptive field interactions and the model’s ability to tackle complex tasks. CT-MIE achieves excellent performance during extensive experimental evaluation in various image enhancement tasks. We have released the code at https://github.com/zyc-123/CT-MIE.git.

Visual question answering (VQA) tasks have witnessed significant advancements in recent years. So far, enhancing the robustness of models on diverse datasets and improving their performance in3D environmentsremains a challenging research direction. In this paper, we propose a high-performance framework called Bias3D-VQA for 3D-VQA based on generative adversarial networks via bias learning, addressing the inherent biases that arise from the model’s dependency on dataset-specific patterns or tendencies during training. Such biases often lead the model to focus on more frequently occurring but incorrect answers. Our framework comprises a target model, a bias model, and a generative adversarial component. In each training iteration, we employ an alternating training approach for the target and bias models. When training the bias model, fake point cloud data is generated from random noise, and then we accumulate biases present in language modality and various modules through adversarial training. When training the target model, both the question and the 3D point cloud are inputted into the bias model simultaneously, and the output of the bias model is utilized to correct the loss of the target model. Our approach(Bias3D-VQA) is the first to focus on enhancing model robustness by addressing diverse biases in the 3D-VQA domain. Our target model demonstrates superior performance compared to state-of-the-art models, showing significant improvements in classification accuracy and text generation quality. Notably, in the metrics such as EM@1 and CIDEr, our model even surpasses some pre-trained models with large additional datasets. The source code is available athttps://github.com/coderr727/bias_3DQA

Point clouds frequently contain noise and outliers, presenting obstacles for downstream applications. In this work, we introduce a novel denoising method for point clouds. By leveraging the latent space, we explicitly un-cover noise components, allowing for the extraction of a clean latent code. This, in turn, facilitates the restoration of clean points via inverse transformation. A key component in our network is a new multi-level graph convolution network for capturing rich geometric structural features at various scales from local to global. These features are then integrated into the invertible neural network which bijectively maps the latent space, to guide the noise disentanglement process. Additionally, we employ an invertible mono-tone operator to model the transformation process, effectively enhancing the representation of integrated geometric features. This enhancement allows our network to pre-cisely differentiate between noise factors and the intrinsic clean points in the latent code by projecting them onto separate channels. Both qualitative and quantitative evaluations demonstrate that our method outperforms state-of-the-art methods at various noise levels. The source code is available at https://github.com/yanbiaol/PD-LTS.

3D dense captioning aims to semantically describe each object detected in a 3D scene, which plays a significant role in 3D scene understanding. Previous works lack a complete definition of 3D spatial relationships and the directly integrate visual and language modalities, thus ignoring the discrepancies between the two modalities. To address these issues, we propose a novel complete 3D relationship extraction modality alignment network, which consists of three steps: 3D object detection, complete 3D relationships extraction, and modality alignment caption. To comprehensively capture the 3D spatial relationship features, we define a complete set of 3D spatial relationships, including the local spatial relationship between objects and the global spatial relationship between each object and the entire scene. To this end, we propose a complete 3D relationships extraction module based on message passing and self-attention to mine multi-scale spatial relationship features and inspect the transformation to obtain features in different views. In addition, we propose the modality alignment caption module to fuse multi-scale relationship features and generate descriptions to bridge the semantic gap from the visual space to the language space with the prior information in the word embedding, and help generate improved descriptions for the 3D scene. Extensive experiments demonstrate that the proposed model outperforms the state-of-the-art methods on the ScanRefer and Nr3D datasets.

AI-aided fashion design has attracted growing interest because it eliminates tedious manual operations. However, existing methods are costly because they require abundant labeled data or paired images for training. In addition, they have low flexibility in attribute editing. To overcome these limitations, we propose UFS-Net, a new unsupervised network for fashion style editing and generation. Specifically, we initially design a coarse-to-fine embedding process to embed the user-defined sketch and the real clothing into the latent space of StyleGAN. Subsequently, we propose a feature fusion scheme to generate clothing with attributes provided by the sketch. In this way, our network requires neither labels nor sketches during the training but can perform flexible attribute editing and conditional generation. Extensive experiments reveal that our method significantly outperforms state-of-the-art approaches. In addition, we introduce a new dataset, Fashion-Top, to address the limitations in the existing fashion datasets.

Point clouds obtained by LiDAR and other sensors are usually sparse and irregular. Low-quality point clouds have serious influence on the final performance of downstream tasks. Recently, a point cloud upsampling network with normalizing flows has been proposed to address this problem. However, the network heavily relies on designing specialized architectures to achieve invertibility. In this paper, we propose a novel invertible residual neural network for point cloud upsampling, called PU-INN, which allows unconstrained architectures to learn more expressive feature transformations. Then, we propose a conditional injector to improve nonlinear transformation ability of the neural network while guaranteeing invertibility. Furthermore, a lightweight interpolator is proposed based on semantic similarity distance in the latent space, which can intuitively reflect the interpolation changes in Euclidean space. Qualitative and quantitative results show that our method outperforms the state-of-the-art works in terms of distribution uniformity, proximity-to-surface accuracy, 3D reconstruction quality, and computation efficiency.

Transformer architectures have recently been introduced into the field of visual question answering (VQA), due to their powerful capabilities of information extraction and fusion. However, existing Transformer-like models, including models using a single Transformer structure and large-scale pre-training generic visual-linguistic models, do not fully utilize both positional information of words in questions and positional information of objects in images, which are shown in this paper to be crucial in VQA tasks. To address this challenge, we propose a novel positional attention guided Transformer-like architecture, which can adaptively extracts positional information within and across the visual and language modalities, and use this information to guide high-level interactions in inter- and intra-modality information flows. In particular, we design and assemble three positional attention modules into a single Transformer-like model MCAN. We show that the positional information introduced in intra-modality interaction can adaptively modulate inter-modality interaction according to different inputs, which plays an important role for visual reasoning. Experimental results demonstrate that our model outperforms the state-of-the-art models and is particularly good at handling object counting questions. Overall, our model achieves the accuracy of 70.10%, 71.27%, and 71.52% on the datasets of COCO-QA, VQA v1.0 test-std and VQA v2.0 test-std, respectively.

Point cloud upsampling aims to generate dense point clouds from given sparse ones, which is a challenging task due to the irregular and unordered nature of point sets. To address this issue, we present a novel deep learning-based model, called PU-Flow, which incorporates normalizing flows and weight prediction techniques to produce dense points uniformly distributed on the underlying surface. Specifically, we exploit the invertible characteristics of normalizing flows to transform points between euclidean and latent spaces and formulate the upsampling process as ensemble of neighbouring points in a latent space, where the ensemble weights are adaptively learned from local geometric context. Extensive experiments show that our method is competitive and, in most test cases, it outperforms state-of-the-art methods in terms of reconstruction quality, proximity-to-surface accuracy, and computation efficiency. The source code will be publicly available at https://github.com/unknownue/puflow.

3D reconstruction from single-view images is a long-standing research problem. There have been various methods based on point clouds and volumetric representations. In spite of success in 3D models generation, it is quite challenging for these approaches to deal with models with complex topology and fine geometric details. Thanks to the recent advance of deep shape representations, learning the structure and detail representation using deep neural networks is a promising direction. In this article, we propose a novel approach named STD-Net to reconstruct 3D models utilizing mesh representation that is well suited for characterizing complex structures and geometry details. Our method consists of (1) an auto-encoder network for recovering the structure of an object with bounding box representation from a single-view image; (2) a topology-adaptive GCN for updating vertex position for meshes of complex topology; and (3) a unified mesh deformation block that deforms the structural boxes into structure-aware meshes. Evaluation on ShapeNet and PartNet shows that STD-Net has better performance than state-of-the-art methods in reconstructing complex structures and fine geometric details.

Simulating liquid-textile interaction has received great attention in computer graphics recently. Most existing methods take textiles as particles or parameterized meshes. Although these methods can generate visually pleasing results, they cannot simulate water content at a microscopic level due to the lack of geometrically modeling of textile's anisotropic structure. In this paper, we develop a method for yarn-level simulation of hygroscopicity of textiles and evaluate it using various quantitative metrics. We model textiles in a fiber-yarn-fabric multi-scale manner and consider the dynamic coupled physical mechanisms of liquid spreading, including wetting, wicking, moisture sorption/desorption, and transient moisture-heat transfer in textiles. Our method can accurately simulate liquid spreading on textiles with different fiber materials and geometrical structures with consideration of air temperatures and humidity conditions. It visualizes the hygroscopicity of textiles to demonstrate their moisture management ability. We conduct qualitative and quantitative experiments to validate our method and explore various factors to analyze their influence on liquid spreading and hygroscopicity of textiles.

Video-based violence recognition has become a crucial topic owing to the development of surveillance cameras. However, with the extra temporal dimension and no precision range of violent video data, violence recognition is a challenging problem. In this study, we propose a dual temporal excitation network (DTE-Net) consisting of a shift temporal adaptive module (STAM) and a sparse object interaction transformer (SOI-Tr) module. The STAM extracts coarse-grained local and global temporal information by fusing shift module with temporal adaptive modeling module. The SOI-Tr module utilizes important object attention to excite fine-grained global temporal representation reasoning. In addition, we create a multi-class violence (MCV) dataset of video clips extracted from real-world scenes to address the limitation of poorly diversified categories in most existing violence datasets. Finally, we also conduct extensive experiments on five violence datasets, including the MCV, and the results show that our network outperforms state-of-the-art performance.

Text-based visual question answering (T-VQA) aims to answer questions about images by comprehending both detected objects and OCR(optical character recognition) tokens. Most existing methods fail to eliminate the noisy and redundant detected objects, and ignore the modality-specific information. To address these concerns, we propose multimodal global enhanced network (MGEN) for T-VQA. In MGEN the multi-modal global enhanced OCR graph focus on modeling the spatial relationships between OCR tokens rather than objects with noise and redundancy. Then, we introduce the multi-modal global enhanced transformer module, which is formed using the proposed attention mechanism, to reflect the specificity in the various modalities. The preceding two modules can leverage global features, implying that not only will the model's attention be directed to critical parts, but the noise can also be further reduced. Extensive experiments demonstrate the effectiveness and superiority of the proposed MGEN against the state-of-the-art methods.

Superficial linguistic correlations is a critical issue for Visual Question Answering (VQA), where models can achieve high performance by exploiting the connection between question and answer, but fail to obtain better generalization ability for out-of-domain data. To ease such issue, VQA-CP v2.0 greedily re-partitions the distribution of VQA v2.0's training and test divides, it suppresses the performance improvement acquired by superficial linguistic correlations. However, some opportunistic methods (such as inverse supervision) can take advantage of the dataset's distribution characteristics to obtain high performance, which is incompatible with academic efforts to increase the model's visual reasoning and modal fusion abilities. To address this problem, we propose a more reasonable dataset in which we attempt to make the training split conform to the long-tailed distribution and the test split more balanced, so that inverse supervision does not result in performance gains and superficial linguistic correlations still can not assist the model in achieving high accuracy. Besides, we propose a decoupled training schema which can obtain better representation and visual reasoning modules to compensate for the shortcomings of ensemble-based methods that selectively learn some samples. Without any further annotations, such schema achieves state-of-the-art performance. In VQA-CP v2.0, it outperforms the simple baseline model UpDn by 15.54%. And its accuracy on VQA v2.0 has almost no drop compared to UpDn. Code is available at https://github.com/asklvd/new-benchmark-for-robust-VQA.

Due to the challenges of densely compacted body parts, nonrigid clothing items, and severe overlap in crowd scenes, human parsing needs to focus more on multilevel feature representations compared to general scene parsing tasks. Based on this observation, we propose to introduce the auxiliary task of human mask and edge detection to facilitate human parsing. Different from human parsing, which exploits the discriminative features of each category, human mask and edge detection emphasizes the boundaries of semantic parsing regions and the difference between foreground humans and background clutter, which benefits the parsing predictions of crowd scenes and small human parts. Specifically, we extract human mask and edge labels from the human parsing annotations and train a shared encoder with three independent decoders for the three mutually beneficial tasks. Furthermore, the decoder feature maps of the human mask prediction branch are further exploited as attention maps, indicating human regions to facilitate the decoding process of human parsing and human edge detection. In addition to these auxiliary tasks, we further alleviate the problem of deformed clothing items under various human poses by tracking the deformation patterns with the deformable convolution. Extensive experiments show that the proposed method can achieve superior performance against state-of-the-art methods on both single and multiple human parsing datasets. Codes and trained models are available https://github.com/ViktorLiang/MGDAN .

Human poses play a critical role in human-centric product design. Despite considerable researches on pose synthesis and pose-driven product design, most of them adopt the simple stick figure model that captures only skeletons rather than real body geometries and do not link human poses to the environment (e.g., chairs for sitting). This paper focuses on user-tailored ergonomic design and rating of chairs using scanned human geometries. Fully utilizing the anthropometric information of the human models, our method considers more ergonomic guidelines of chair design (such as pressure distribution and support intensity) and links the geometry of 3D chair models and human-to-chair interactions into the pose deformation constraints of the human avatars. The core of our method is a pose generation algorithm which rigs the user's successive poses through coarse- and fine-level pose deformations. We define a non-linear energy function with contact, collision, and joint limit terms, and solve it using a hill-climbing algorithm. The fitting results allow us to quantitatively evaluate the chair model in terms of various ergonomic criteria. Our method is flexible and effective and can be applied to users with varying body shapes and a wide range of chairs. Moreover, the proposed technique can be easily extended to other furniture, such as desk, bed, and cabinet. Extensive evaluations and a user study demonstrate the efficiency and advantages of the proposed virtual fitting method. Given that our method avoids tedious on-site trying, facilitates the exploration/evaluation of various chair products, and provides valuable feedback for the designers and manufacturers to deliver customized products, it is ideal for online shopping of chairs.

Abstract Rig‐space physics a finite element method(FEM) based simulation technique that aims at adding secondary motion on a character while maintaining seamless cooperation with traditional animation pipelines. We enhance the rig‐space physics by introducing several techniques, including general field interaction, proportional‐derivative control, and improved material control. This allows an animator to perform various interferences to the simulation process and create more abundant animation effects. Moreover, we also improve the numerical stability of the simulation algorithm by prepending a conjugate gradient procedure. Copyright © 2016 John Wiley & Sons, Ltd.

Different from the 4-points interpolation subdivision curve scheme, the proposed subdivision curve scheme in this paper is based on cube Bezier curve. More importantly, the normal vectors are used to generate a circle from a triangle with the advantages of only 3 vertices and 3 edges. Furthermore, we make new contributions in the subdivision surface scheme compared to the existing methods. It has excellent smoothness and also can be fast implemented.

This paper investigates the convergence of an algorithm geometrically interpolating a given polygon using uniform quadratic and cubic B-splines respectively. The geometric interpolation method views the polygon itself as the initial guess of the control polygon of the B-spline and reduces the approximate error by iteratively updating the control points with the deviation from the interpolated vertices to their nearest foot points on the current B-spline curve. We demonstrate that the algorithm usually does not converge if the nearest points are searched on the whole curve and present a sufficient condition under which the algorithm is convergent, for quadratic and cubic B-splines respectively. Furthermore, we introduce a new strategy to update the control points incrementally. Experiments show that though our condition constrains the search range of the nearest points, it can still produce interpolation curves with the same high quality as the original method.