The accurate segmentation of brain tumors plays an important role in clinical diagnosis and treatment. Multimodal magnetic resonance imaging (MRI) can provide rich and complementary information for accurate brain tumor segmentation. However, the common problems of incomplete modalities and small samples in clinical practice seriously affect the performance of multimodal segmentation. In this work, we design a new framework, named M$^{2}$SegMamba, using Mamba and Masked Autoencoder networks for both supervised and self-supervised learning, aimed at handling small sample brain tumor segmentation under various incomplete multimodality settings. We construct a masking strategy suitable for multimodal brain tumors to precisely extract image features, which serves as the foundation for image segmentation. By fully leveraging the capabilities of the Mamba network, we design a multi-traversal method to facilitate the interaction between inter-modal and cross-modal image features. Meanwhile, the introduction of TSmamba in skipping connections efficiently integrates multimodal features. Auxiliary regularizers are introduced in both the encoder and decoder to further enhance the model's robustness to incomplete modalities. We conducted experiments on the BraTS 2018 and BraTS 2020 datasets, and the results demonstrate that our method outperforms state-of-the-art brain tumor segmentation methods on most subsets of missing modalities.

This paper presents MonoRelief V2, an end-to-end model designed for directly recovering 2.5D reliefs from single images under complex material and illumination variations. In contrast to its predecessor, MonoRelief V1 (Gao et al. 2025), which was solely trained on synthetic data, MonoRelief V2 incorporates real data to achieve improved robustness, accuracy and efficiency. To overcome the challenge of acquiring large-scale real-world dataset, we generate approximately 15,000 pseudo-real images using a text-to-image generative model, and derive corresponding depth pseudo-labels through fusion of depth and normal predictions. Furthermore, we construct a small-scale real-world dataset (800 samples) via multi-view reconstruction and detail refinement. MonoRelief V2 is then progressively trained on the pseudo-real and real-world datasets. Comprehensive experiments demonstrate its state-of-the-art performance both in depth and normal predictions, highlighting its strong potential for a range of downstream applications.

This study focuses on multi-human relief modeling using a single photograph. Although previous studies successfully modeled 3D humans from single photographs, they were limited to reconstructing 3D individuals and could not be applied to multi-human scenes with complex inter-body and outer-body occlusions. In this study, we introduce MMRelief, a novel solution that takes a significant step toward high-quality and generalized multi-human relief modeling. MMRelief uses a three-step approach to achieve its objectives. First, it predicts an occlusion-aware depth map based on ZoeDepth [12]. Subsequently, it predicts a detailed normal map using a photo-to-normal network. Finally, MMRelief combines the strengths of both maps and constructs human relief using depth-constrained normal integration. Experimental results demonstrate that MMRelief has achieved state-of-the-art performance in normal human estimation. It can handle different styles of human photos with varying poses and dresses while producing reliefs with accurate body occlusions, reasonable depth ordering, and faithful geometrical details. The project page is at https://github.com/yanqingliu3856/MMRelief.

Unsupervised domain adaptation (UDA) aims to adapt models learned from a well-annotated source domain to a target domain, where only unlabeled samples are available. To this end, adversarial training is widely used in conventional UDA methods to reduce the discrepancy between source and target domains. Recently, prompt tuning has emerged as an efficient way to adapt large pre-trained vision-language models like CLIP to a variety of downstream tasks. In this paper, we present a novel method named Adversarial DuAl Prompt Tuning (ADAPT) for UDA, which employs text prompts and visual prompts to guide CLIP simultaneously. Rather than simply performing a joint optimization of text prompts and visual prompts, we integrate text prompt tuning and visual prompt tuning into a collaborative framework where they engage in an adversarial game: text prompt tuning focuses on distinguishing between source and target images, whereas visual prompt tuning seeks to align source and target domains. Unlike most existing adversarial training-based UDA approaches, ADAPT does not require explicit domain discriminators for domain alignment. Instead, the objective is effectively achieved at both global and category levels through modeling the joint probability distribution of images on domains and categories. Extensive experiments on four benchmark datasets demonstrate the effectiveness of our ADAPT method for UDA. We have released our code at https://github.com/Liuziyi1999/ADAPT.

Incomplete multi-view clustering has gained considerable attention in recent years due to the prevalence of incomplete multi-view data in real-world applications. However, existing methods often struggle to effectively deal with large-scale datasets, particularly those with a significant number of missing instances. To address these issues, we propose a novel method called Hubness-Enabled Clustering and Recovery for Large-Scale Incomplete Multi-View Data (HENRI). HENRI utilizes the consensus hubs of all views to identify informative anchors to handle large-scale incomplete datasets. Furthermore, it incorporates a novel sample-level fusion strategy that effectively integrates information from all views, leading to remarkable outcomes in both cluster formation and missing data reconstruction. HENRI demonstrates exceptional capability in capturing the underlying structures of the data and recovering missing information, even when faced with a significant number of instances with incomplete data in partial views. To validate its effectiveness, we conducted experiments on 6 complete datasets and 31 incomplete datasets, comparing against 11 baseline methods. The results are impressive, demonstrating the superior performance of HENRI over the state-of-the-art methods.

Deep multi-view subspace clustering aims to reveal a common subspace structure by exploiting rich multi-view information. Despite promising progress, current methods focus only on multi-view consistency and complementarity, often overlooking the adverse influence of entangled superfluous information in features. Moreover, most existing works lack scalability and are inefficient for large-scale scenarios. To this end, we innovatively propose a deep subspace clustering method via Multi-view Feature Decoupling (MvFD). First, MvFD incorporates well-designed multi-type auto-encoders with self-supervised learning, explicitly decoupling consistent, complementary, and superfluous features for every view. The disentangled and interpretable feature space can then better serve unified representation learning. By integrating these three types of information within a unified framework, we employ information theory to obtain a minimal and sufficient representation with high discriminability. Besides, we introduce a deep metric network to model self-expression correlation more efficiently, where network parameters remain unaffected by changes in sample numbers. Extensive experiments show that MvFD yields State-of-the-Art performance in various types of multi-view datasets.

In this article, we introduce MonoRelief, a novel method that combines the strengths of a depth map and a normal map to achieve high-quality relief recovery from a single image. By constructing a large-scale relief dataset that encompasses a diverse range of relief shapes, materials, and lighting conditions, we enable the training of a robust normal estimation network capable of handling various types of relief images. Furthermore, we leverage the state-of-the-art method, DepthAnything v2 (Yang et al. 2024), to generate depth maps from the input images. By integrating the strengths of both maps, MonoRelief recovers 2.5D reliefs with reasonable depth structures and intricate geometrical details. We validate the effectiveness and robustness of MonoRelief through comprehensive experiments, and showcase its potential in a variety of downstream applications, including Image-to-Relief, Text-to-Relief, Lines-to-Relief and relief reproduction.

Recently, self-supervised denoising methods have attracted significant attention due to the considerable challenge posed by constructing a large-scale real noise dataset for supervised training. The most representative self-supervised denoisers are based on blind-spot networks (BSNs), which exclude the central pixel of receptive field. However, excluding any input pixel potentially leads to the loss of vital information required for accurate predictions, especially when the excluded pixel corresponds to the output position. In addition, a standard BSN has struggled to effectively reduce real-world noise due to the spatial correlation of noise, though it makes the significant results with independently distributed synthetic noise. In this paper, we propose a novel self-supervised real-world image denoising framework called Complementary-BSN based on two reciprocal branches (Mask-Map branch and Enhanced-PD-BSN branch) with an efficient loss function to employ the pixels information ignored by masked convolution and provide additional optimization target for self-supervised output. Specifically, we exploit a block-wise random-placing (BRP) scheme for further weaken the noisy correlation to avoid the illusion of image structure recovery due to existing complex noise and make Complementary-BSN more suitable for real noise. Additionally, we develop an efficient strategy (multi-stride PD (MPD)) to fuse multiple PD strides for inference, narrowing the restoration gap between textural and flat regions. Extensive experiments on real-world datasets demonstrate that our method achieves superior performance to other state-of-the-art (SOTA) self-supervised denoising methods. The code is available athttps://github.com/cuijin7382/Complementary-BSN.

Image sequence interpolation is a critical research area in computer vision with broad applications in video frame interpolation and medical image interlayer interpolation. Traditional deep learning-based methods in this domain predominantly rely on deep convolutional neural networks (CNNs), which, despite their effectiveness, are limited by the inherent constraints of CNN architecture, impacting their interpolation accuracy. To address these limitations, we introduce the Pre-ISIformer, a parallel multi-channel adaptive image sequence interpolation network founded on pre-trained transformers. This innovative network is composed of three integral modules: 1) Global feature extraction module is designed to extract primary features from the input images using a pre-trained Swin-transformer model, ensuring comprehensive global feature coverage. 2) Feature sequence construction module adaptively decomposes the object’s motion path across different frames, facilitating a detailed analysis of motion dynamics. And 3) Intermediate image reconstruction module is responsible for accurately capturing target displacements. Furthermore, we incorporate distinct metrics for pixel loss and gradient loss to meticulously reconstruct the texture and contours of the intermediate images. Our network has been rigorously tested on various datasets for two primary applications: video frame interpolation and interlayer interpolation in medical imaging. The results from these experiments showcase the superior performance and effectiveness of the Pre-ISIformer, establishing it as a significant advancement in the field of image sequence interpolation.

Multiview spectral clustering, which stands out with its remarkable clustering performance, has drawn increasing research attention. Its core is properly weighing the contribution of different views and comprehensively utilizing multiview information in the clustering process. Although existing methods, like exponential decay and root loss, have achieved significant progress, they still have limitations in their weighting scheme, application generalization, and model efficiency. To handle these limitations, we propose a novel Similarity-Induced Weighted Consensus Laplacian matrix learning method for multiview clustering (MC), named SIWCL. This method has two distinctive features: 1) instead of conventional Laplacian matrix learning, SIWCL resorts to consensus Laplacian matrix learning as the MC framework for effectively exploiting complementary information from multiple views and 2) we argue that there could be outlier views that exhibit an uneven similarity distribution with other views, and equally treating them with other views can hurt model performance. Therefore, beyond consensus Laplacian matrix learning, SIWCL introduces a novel weighting strategy that adaptively assigns weights to views according to their consistency with other views based on the multiview similarity matrices. Notably, this novel method has only one hyperparameter and closed-form solutions, greatly improving the efficiency and generalization. Experiments show that the weights obtained by the proposed weighting strategy are correlated to the quality of the clustering structure. The comparisons between the proposed method and other state-of-the-art baseline methods over eight datasets demonstrate the robustness and superior clustering performance of SIWCL.

This paper aims at extending the method of Zhang et al. (2023) to produce not only portrait bas-reliefs from single photographs, but also high-depth reliefs with reasonable depth ordering. We cast this task as a problem of style-aware photo-to-depth translation, where the input is a photograph conditioned by a style vector and the output is a portrait relief with desired depth style. To construct ground-truth data for network training, we first propose an optimization-based method to synthesize high-depth reliefs from 3D portraits. Then, we train a normal-to-depth network to learn the mapping from normal maps to relief depths. After that, we use the trained network to generate high-depth relief samples using the provided normal maps from Zhang et al. (2023). As each normal map has pixel-wise photograph, we are able to establish correspondences between photographs and high-depth reliefs. By taking the bas-reliefs of Zhang et al. (2023), the new high-depth reliefs and their mixtures as target ground-truths, we finally train a encoder-to-decoder network to achieve style-aware relief modeling. Specially, the network is based on a U-shaped architecture, consisting of Swin Transformer blocks to process hierarchical deep features. Extensive experiments have demonstrated the effectiveness of the proposed method. Comparisons with previous works have verified its flexibility and state-of-the-art performance.

Multi-modal medical image fusion is a long-standing important research topic that can obtain informative medical images and assist doctors diagnose and treat diseases more efficiently. However, most fusion methods extract and fuse features by subjectively defining constraints, which easily distorts the unique information of source images. In this work, we present a novel end-to-end unsupervised network to fuse multi-modal medical images. It is composed of a generator and two symmetrical discriminators. The former aims to generate a ”real-like” fused image based on a specifically designed content and structure loss, while the latter are devoted to distinguishing the differences between the fused image and the source ones. They are trained alternately until discriminators cannot distinguish the fused image from the source ones. In addition, the symmetrical discriminator scheme is conducive to maintaining the feature consistency among different modalities. More importantly, to enhance the retention degree of texture details, U-Net is adopted as the generator heuristically, where the up-sampling method is modified to bilinear interpolation for avoiding checkerboard artifacts. As for the optimization, we define the content loss function, which preserves the gradient information and pixel activity of source images. Both visual analysis and quantitative evaluation of experimental results show the superiority of our method as compared to the cutting-edge baselines.

In this paper, we present an end-to-end neural solution to model portrait bas-relief from a single photograph, which is cast as a problem of image-to-depth translation. The main challenge is the lack of bas-relief data for network training. To solve this problem, we propose a semi-automatic pipeline to synthesize bas-relief samples. The main idea is to first construct normal maps from photos, and then generate bas-relief samples by reconstructing pixel-wise depths. In total, our synthetic dataset contains 23 k pixel-wise photo/bas-relief pairs. Since the process of bas-relief synthesis requires a certain amount of user interactions, we propose end-to-end solutions with various network architectures, and train them on the synthetic data. We select the one that gave the best results through qualitative and quantitative comparisons. Experiments on numerous portrait photos, comparisons with state-of-the-art methods and evaluations by artists have proven the effectiveness and efficiency of the selected network.

To reproduce the latent high-quality image from its observed image, traditional image restoration approaches based on low-rank prior usually solve the low-rank matrix approximation problem by minimizing the nuclear norm. However, most of approaches work on the desired singular values individually, and ignore the underlying statistical property of singular values. In this work, we propose a probability-inducing nuclear norm minimization (PINNM) algorithm, where a probability-inducing singular value estimator is presented to estimate the desired singular values. For further filling-in the image details lost in handling the singular values, a simple yet efficient residual cascade scheme is designed to refine the image quality by using the intermediate recovered image. Then, the proposed PINNM algorithm is applied on two classic image restoration tasks: image super-resolution and denoising. Experimental results demonstrate that, the proposed PINNM algorithm outperforms many state-of-the-art approaches both quantitatively and qualitatively for the aforementioned tasks. The source code is available at https://github.com/cvzh/PINNM.

Time series research in academic and industrial fields has attracted wide attention. However, the frequency information contained in time series still lacks effective modeling. The studies found that time series forecasting relies on different frequency patterns: short-term series forecasting relies more on high-frequency components, while long-term forecasting focuses more on low-frequency data. To better describe the multifrequency mode, a dual-stage multifeature adaptive frequency domain prediction model (DMAFD) is proposed in this paper. DMAFD contains two stages. First, it adopts the XGBoost algorithm to obtain a feature vector by analyzing the feature importance. Second, the frequency feature extraction of time series and the frequency aware modeling of the target sequence is integrated, for building an end-to-end prediction network based on the dependence of time series on frequency mode. The innovation is reflected in the fact that the prediction network can automatically focus on multifrequency components according to the dynamic evolution of the input sequence. Extensive experiments on four real data sets from different fields show that DMAFD obtains higher accuracy and smaller lags in time step analysis compared with state-of-the-art algorithms.

Abstract Different from other types of bas‐reliefs, a flower bas‐relief contains a large number of depth‐discontinuity edges. Most existing line‐based methods reconstruct free‐form surfaces by ignoring the depth‐discontinuities, thus are less efficient in modeling flower bas‐reliefs. This paper presents a neural‐based solution which benefits from the recent advances in CNN. Specially, we use line gradients to encode the depth orderings at leaf edges. Given a line drawing, a heuristic method is first proposed to compute 2D gradients at lines. Line gradients and dense curvatures interpolated from sparse user inputs are then fed into a neural network, which outputs depths and normals of the final bas‐relief. In addition, we introduce an object‐based method to generate flower bas‐reliefs and line drawings for network training. Extensive experiments show that our method is effective in modelling bas‐reliefs with depth‐discontinuity edges. User evaluation also shows that our method is intuitive and accessible to common users.

Multiview spectral clustering aims to separate data into different clusters efficiently by the use of multiview information. Many studies learn the affinity matrix from the original high-dimensional data, whose noise goes against the clustering results. Besides, some methods based on self-representation subspace clustering have a high time complexity. In this paper, we propose a simple, yet effective, and efficient method named Kernel-based Low-rank Tensorized Multiview Spectral Clustering (KLTMSC) to address these issues. Instead of using the original data to get the affinity matrix, KLTMSC learns the affinity matrix from kernel representation of the high-dimensional data to reduce the noisy information. Furthermore, to be robust to noise, the low-rank tensor is learned in the process of exploring the high-order correlations between data. Experiments on real-world data sets show that our method not only yields better results but also is quite time-saving compared with other state-of-the-art models.

BACKGROUND: Chromatin 3D conformation plays important roles in regulating gene or protein functions. High-throughout chromosome conformation capture (3C)-based technologies, such as Hi-C, have been exploited to acquire the contact frequencies among genomic loci at genome-scale. Various computational tools have been proposed to recover the underlying chromatin 3D structures from in situ Hi-C contact map data. As connected residuals in a polymer, neighboring genomic loci have intrinsic mutual dependencies in building a 3D conformation. However, current methods seldom take this feature into account. RESULTS: We present a method called ShNeigh, which combines the classical MDS technique with local dependence of neighboring loci modeled by a Gaussian formula, to infer the best 3D structure from noisy and incomplete contact frequency matrices. We validated ShNeigh by comparing it to two typical distance-based algorithms, ShRec3D and ChromSDE. The comparison results on simulated Hi-C dataset showed that, while keeping the high-speed nature of classical MDS, ShNeigh can recover the true structure better than ShRec3D and ChromSDE. Meanwhile, ShNeigh is more robust to data noise. On the publicly available human GM06990 Hi-C data, we demonstrated that the structures reconstructed by ShNeigh are more reproducible between different restriction enzymes than by ShRec3D and ChromSDE, especially at high resolutions manifested by sparse contact maps, which means ShNeigh is more robust to signal coverage. CONCLUSIONS: Our method can recover stable structures in high noise and sparse signal settings. It can also reconstruct similar structures from Hi-C data obtained using different restriction enzymes. Therefore, our method provides a new direction for enhancing the reconstruction quality of chromatin 3D structures.

Abstract In contrast to 3D model that can be freely observed, p ortrait bas‐relief projects slightly from the background and is limited by fixed viewpoint. In this paper, we propose a novel method to reconstruct the underlying 3D shape from a single 2.5D bas‐relief, providing observers wider viewing perspectives. Our target is to make the reconstructed portrait has natural depth ordering and similar appearance to the input. To achieve this, we first use a 3D template face to fit the portrait. Then, we optimize the face shape by normal transfer and Poisson surface reconstruction. The hair and body regions are finally reconstructed and combined with the 3D face. From the resulting 3D shape, one can generate new reliefs with varying poses and thickness, freeing the input one from fixed view. A number of experimental results verify the effectiveness of our method.

As we all know, video frame rate determines the quality of the video. The higher the frame rate, the smoother the movements in the picture, the clearer the information expressed, and the better the viewing experience for people. Video interpolation aims to increase the video frame rate by generating a new frame image using the relevant information between two consecutive frames, which is essential in the field of computer vision. The traditional motion compensation interpolation method will cause holes and overlaps in the reconstructed frame, and is easily affected by the quality of optical flow. Therefore, this paper proposes a video frame interpolation method via optical flow estimation with image inpainting. First, the optical flow between the input frames is estimated via combined local and global-total variation (CLG-TV) optical flow estimation model. Then, the intermediate frames are synthesized under the guidance of the optical flow. Finally, the nonlocal self-similarity between the video frames is used to solve the optimization problem, to fix the pixel loss area in the interpolated frame. Quantitative and qualitative experimental results show that this method can effectively improve the quality of optical flow estimation, generate realistic and smooth video frames, and effectively increase the video frame rate.

Automatic and accurate semantic segmentation of pathological structures in medical images is challenging because of noisy disturbance, deformable shapes of pathology, and low contrast between soft tissues. Classical superpixel-based classification algorithms suffer from edge leakage due to complexity and heterogeneity inherent in medical images. Therefore, we propose a deep U-Net with superpixel region merging processing incorporated for edge enhancement to facilitate and optimize segmentation. Our approach combines three innovations: (1) different from deep learning--based image segmentation, the segmentation evolved from superpixel region merging via U-Net training getting rich semantic information, in addition to gray similarity; (2) a bilateral filtering module was adopted at the beginning of the network to eliminate external noise and enhance soft tissue contrast at edges of pathogy; and (3) a normalization layer was inserted after the convolutional layer at each feature scale, to prevent overfitting and increase the sensitivity to model parameters. This model was validated on lung CT, brain MR, and coronary CT datasets, respectively. Different superpixel methods and cross validation show the effectiveness of this architecture. The hyperparameter settings were empirically explored to achieve a good trade-off between the performance and efficiency, where a four-layer network achieves the best result in precision, recall, F-measure, and running speed. It was demonstrated that our method outperformed state-of-the-art networks, including FCN-16s, SegNet, PSPNet, DeepLabv3, and traditional U-Net, both quantitatively and qualitatively. Source code for the complete method is available at https://github.com/Leahnawho/Superpixel-network.

We present a novel solution to enable portrait relief modeling from a single image. The main challenges are geometry reconstruction, facial details recovery and depth structure preservation. Previous image-based methods are developed for portrait bas-relief modeling in 2.5D form, but not adequate for 3D-like high relief modeling with undercut features. In this paper, we propose a template-based framework to generate portrait reliefs of various forms. Our method benefits from Shape-from-Shading (SFS). Specifically, we use bi-Laplacian mesh deformation to guide the relief modeling. Given a portrait image, we first use a template face to fit the portrait. We then apply bi-Laplacian mesh deformation to align the facial features. Afterwards, SFS-based reconstruction with a few user interactions is used to optimize the face depth, and create a relief with similar appearance to the input. Both depth structures and geometric details can be well constructed in the final relief. Experiments and comparisons to other methods demonstrate the effectiveness of the proposed method.

Nonlocal self-similarity of images has attracted considerable interest in the field of image processing and has led to several state-of-the-art image denoising algorithms, such as block matching and 3-D, principal component analysis with local pixel grouping, patch-based locally optimal wiener, and spatially adaptive iterative singular-value thresholding. In this paper, we propose a computationally simple denoising algorithm using the nonlocal self-similarity and the low-rank approximation (LRA). The proposed method consists of three basic steps. First, our method classifies similar image patches by the block-matching technique to form the similar patch groups, which results in the similar patch groups to be low rank. Next, each group of similar patches is factorized by singular value decomposition (SVD) and estimated by taking only a few largest singular values and corresponding singular vectors. Finally, an initial denoised image is generated by aggregating all processed patches. For low-rank matrices, SVD can provide the optimal energy compaction in the least square sense. The proposed method exploits the optimal energy compaction property of SVD to lead an LRA of similar patch groups. Unlike other SVD-based methods, the LRA in SVD domain avoids learning the local basis for representing image patches, which usually is computationally expensive. The experimental results demonstrate that the proposed method can effectively reduce noise and be competitive with the current state-of-the-art denoising algorithms in terms of both quantitative metrics and subjective visual quality.

In this paper, we introduce a novel approach to bas-relief generation and shape editing that uses gradient-based mesh deformation as the theoretical foundation. Our approach differs from image-based methods in that it operates directly on the triangular mesh, and ensures that the mesh topology remains unchanged during geometric processing. By implicitly deforming the input mesh through gradient manipulation, our approach is applicable to both plane surface bas-relief generation and curved surface bas-relief generation. We propose a series of gradient-based algorithms, such as height field deformation, high slope optimization, fine detail preservation, curved surface flattening and relief mapping. Additionally, we present two types of shape editing tools that allow the user to interactively modify the bas-relief to exhibit a desired shape. Experimental results indicate that the proposed approach is effective in producing plausible and impressive bas-reliefs.