Noisy supervision refers to supervising network learning with targets corrupted by noise, encompassing both weakly supervised learning with noisy targets and fully unsupervised denoising using unpaired noisy images. It alleviates the data collection burden and enhances the practical applicability of deep learning techniques. Existing methods face two main limitations: they are ineffective at handling noise with long-range correlations, commonly found in real-world scenarios such as low-light imaging and remote sensing, and rely on pixel-wise loss functions that offer limited supervision for image deblurring and super-resolution. This work addresses these challenges by leveraging the Fourier domain, where spatially correlated noise exhibits sparsity and independence, and Fourier coefficients capture global information that enables stronger supervision. We prove that Fourier coefficients of a wide range of noise converge in distribution to the Gaussian distribution and establish a statistical equivalence between learning with clean and noisy targets in the Fourier domain. Based on these insights, we develop a weakly supervised framework for image restoration learning with noisy targets, and construct a fully unsupervised denoising method tailored to stripe-wise noise. Extensive experiments show that our approaches achieve superior performance in both quantitative metrics and perceptual quality.

Fully supervised ultrasound segmentation requires extensive labeled data, which limits its clinical use. While existing semi-supervised methods leverage unlabeled data, they typically treat all regions equally-overlooking critical variations in learning difficulty. We find that boundary and low-contrast regions show persistently low confidence, indicating higher learning difficulty. To address this, we propose a novel framework featuring Bidirectional Replacement and Dynamic Reweighting. Our method constructs easy and hard samples by exchanging regions between images, then applies curriculum-inspired reweighting to adaptively balance their contributions during training. Extensive experiments on three ultrasound datasets demonstrate consistent superiority over state-of-the-art methods under varying supervision levels, confirming effectiveness and robustness.

Supervised deep denoising methods have significantly advanced low-dose computed tomography (CT) denoising but require paired clean-noisy samples that are often unavailable in clinical applications. Current most self-supervised methods are motivated by the Noise2Noise framework and achieve successful low-dose CT denoising based on massive adjacent low-dose CT slices. However, these methods are limited by slice-similarity necessity with small slice spacing. Large slice spacing with spatial misalignment brings inaccurate predictions of denoising results, necessitating more general method. To this end, we propose Global Patch Matching-Diffusion Refinement method that constructs global volume similar images for self-supervised optimization, followed by the perceptual refinement from diffusion models, termed GPM-DR. Our approach extracts global similar patches from CT sequences via mask similarity and$\mathrm{L}^{2}$norm-based sliding window matching, which enriches the training data diversity. Subsequently, the denoised CT images are incorporated into diffusion models conditioned on constructed noisy CT images to further refine the perceptual image quality. The proposed GPM-DR effectively mitigates low-dose CT image noise while addressing discrepancies between slices with powerful generative ability. The results demonstrate that our method outperforms representative self-supervised method. Extensive experiments demonstrate its exceptional denoising performance with few original low-dose CT slices.

Foundation models have demonstrated significant potential in medical AI due to their strong representation and generalization capabilities. Most existing foundation models are trained on 2D data for slice-based diagnosis. However, accurate medical diagnosis often requires comprehensive 3D spatial context. Applying 2D foundation models to individual slices from 3D volumes fails to utilize rich spatial information in volumetric data, limiting diagnostic performance. To bridge this 2D-3D gap, we propose a method to effectively adapt pretrained 2D foundation models for Parkinson's disease diagnosis using 3D brain magnetic resonance images. Specifically, we introduce a multi-scale fusion module into representative 2D foundation models (e.g., BioMedCLIP) to generate scale-aware semantic features that capture lesion characteristics across spatial resolutions. Inspired by radiologists' diagnostic practices, we design a multi-plane slice selector for foundation models to identify the most informative slices from axial, sagittal, and coronal planes. This enhances computational efficiency and diagnostic accuracy by leveraging diverse anatomical perspectives. Furthermore, we propose a 3D adapter module to capture intra-slice features and inter-slice dependencies, and meanwhile adapt foundation models to learn long-range spatial relationships. Experimental results demonstrate that our method outperforms state-of-theart methods while using fewer learnable parameters.

Image denoising is a fundamental task in computer vision. While prevailing deep learning-based supervised and self-supervised methods have excelled in eliminating in-distribution noise, their susceptibility to out-of-distribution (OOD) noise remains a significant challenge. The recent emergence of contrastive language-image pretraining (CLIP) model has showcased exceptional capabilities in open-world image recognition and segmentation. Yet, the potential for leveraging CLIP to enhance the robustness of low-level tasks remains largely unexplored. This paper un-covers that certain dense features extracted from the frozen ResNet image encoder of CLIP exhibit distortion-invariant and content-related properties, which are highly desirable for generalizable denoising. Leveraging these properties, we devise an asymmetrical encoder-decoder denoising network, which incorporates dense features including the noisy image and its multi-scale features from the frozen ResNet encoder of CLIP into a learnable image decoder to achieve generalizable denoising. The progressive feature augmentation strategy is further proposed to mitigate feature over-fitting and improve the robustness of the learnable decoder. Extensive experiments and comparisons conducted across diverse OOD noises, including synthetic noise, real-world sRGB noise, and low-dose CT image noise, demonstrate the superior generalization ability of our method.

Real-world noise removal is crucial in low-level computer vision. Due to the remarkable generation capabilities of diffusion models, recent attention has shifted towards leveraging diffusion priors for image restoration tasks. However, existing diffusion priors-based methods either consider simple noise types or rely on approximate posterior estimation, limiting their effectiveness in addressing structured and signal-dependent noise commonly found in real-world images. In this paper, we build upon diffusion priors and propose adaptive likelihood estimation and MAP inference during the reverse diffusion process to tackle real-world noise. We introduce an independent, non-identically distributed likelihood combined with the noise precision (inverse variance) prior and dynamically infer the precision posterior using variational Bayes during the generation process. Meanwhile, we rectify the estimated noise variance through local Gaussian convolution. The final denoised image is obtained by propagating intermediate MAP solutions that balance the updated likelihood and diffusion prior. Additionally, we explore the local diffusion prior inherent in low-resolution diffusion models, enabling direct handling of high-resolution noisy images. Extensive experiments and analyses on diverse real-world datasets demonstrate the effectiveness of our method. Code is available at https://github.com/HUST-Tan/DiffusionVI.

Despite the success of deep learning methods in multi-modality segmentation tasks, they typically produce a deterministic output, neglecting the underlying uncertainty. The absence of uncertainty could lead to over-confident predictions with catastrophic consequences, particularly in safety-critical clinical applications. Recently, uncertainty estimation has attracted increasing attention, offering a measure of confidence associated with machine decisions. Nonetheless, existing uncertainty estimation approaches primarily focus on single-modality networks, leaving the uncertainty of multi-modality networks a largely under-explored domain. In this study, we present the first exploration of multi-modality uncertainties in the context of tumor segmentation on PET/CT. Concretely, we assessed four well-established uncertainty estimation approaches across various dimensions, including segmentation performance, uncertainty quality, comparison to single-modality uncertainties, and correlation to the contradictory information between modalities. Through qualitative and quantitative analyses, we gained valuable insights into what benefits multi-modality uncertainties derive, what information multi-modality uncertainties capture, and how multi-modality uncertainties correlate to information from single modalities. Drawing from these insights, we introduced a novel uncertainty-driven loss, which incentivized the network to effectively utilize the complementary information between modalities. The proposed approach outperformed the backbone network by 4.53 and 2.92 Dices in percentages on two PET/CT datasets while achieving lower uncertainties. This study not only advanced the comprehension of multi-modality uncertainties but also revealed the potential benefit of incorporating them into the segmentation network.

During the process of computed tomography (CT), metallic implants often cause disruptive artifacts in the reconstructed images, impeding accurate diagnosis. Many supervised deep learning-based approaches have been proposed for metal artifact reduction (MAR). However, these methods heavily rely on training with paired simulated data, which are challenging to acquire. This limitation can lead to decreased performance when applying these methods in clinical practice. Existing unsupervised MAR methods, whether based on learning or not, typically work within a single domain, either in the image domain or the sinogram domain. In this paper, we propose an unsupervised MAR method based on the diffusion model, a generative model with a high capacity to represent data distributions. Specifically, we first train a diffusion model using CT images without metal artifacts. Subsequently, we iteratively introduce the diffusion priors in both the sinogram domain and image domain to restore the degraded portions caused by metal artifacts. Besides, we design temporally dynamic weight masks for the image-domian fusion. The dual-domain processing empowers our approach to outperform existing unsupervised MAR methods, including another MAR method based on diffusion model. The effectiveness has been qualitatively and quantitatively validated on synthetic datasets. Moreover, our method demonstrates superior visual results among both supervised and unsupervised methods on clinical datasets. Codes are available in github.com/DeepXuan/DuDoDp-MAR.

Recently deep learning techniques have significantly advanced image super-resolution (SR). Due to the black-box nature, quantifying reconstruction uncertainty is crucial when employing these deep SR networks. Previous approaches for SR uncertainty estimation mostly focus on capturing pixel-wise uncertainty in the spatial domain. SR uncertainty in the frequency domain which is highly related to image SR is seldom explored. In this paper, we propose to quantify spectral Bayesian uncertainty in image SR. To achieve this, a Dual-Domain Learning (DDL) framework is first proposed. Combined with Bayesian approaches, the DDL model is able to estimate spectral uncertainty accurately, enabling a reliability assessment for high frequencies reasoning from the frequency domain perspective. Extensive experiments under non-ideal premises are conducted and demonstrate the effectiveness of the proposed spectral uncertainty. Furthermore, we propose a novel Spectral Uncertainty based Decoupled Frequency (SUDF) training scheme for perceptual SR. Experimental results show the proposed SUDF can evidently boost perceptual quality of SR results without sacrificing much pixel accuracy.

Real-world single image denoising is crucial and practical in computer vision. Bayesian inversions combined with score priors now have proven effective for single image denoising but are limited to white Gaussian noise. Moreover, applying existing score-based methods for real-world denoising requires not only the explicit train of score priors on the target domain but also the careful design of sampling procedures for posterior inference, which is complicated and impractical. To address these limitations, we propose a score priors-guided deep variational inference, namely ScoreDVI, for practical real-world denoising. By considering the deep variational image posterior with a Gaussian form, score priors are extracted based on easily accessible minimum MSE Non-i.i.d Gaussian denoisers and variational samples, which in turn facilitate optimizing the variational image posterior. Such a procedure adaptively applies cheap score priors to denoising. Additionally, we exploit a Non-i.i.d Gaussian mixture model and variational noise posterior to model the real-world noise. This scheme also enables the pixel-wise fusion of multiple image priors and variational image posteriors. Besides, we develop a noise-aware prior assignment strategy that dynamically adjusts the weight of image priors in the optimization. Our method outperforms other single image-based real-world denoising methods and achieves comparable performance to dataset-based unsupervised methods.

Accurate segmentation of abdominal organs on MRI is crucial for computer-aided surgery and computer-aided diagnosis. Most state-of-the-art methods for MRI segmentation employ an encoder-decoder structure, with skip connections concatenating shallow features from the encoder and deep features from the decoder. In this work, we noticed that simply concatenating shallow and deep features was insufficient for segmentation due to the feature gap between shallow features and deep features. To mitigate this problem, we quantified the feature gap from spatial and semantic aspects and proposed a spatial loss and a semantic loss to bridge the feature gap. The spatial loss enhanced spatial details in deep features, and the semantic loss introduced semantic information into shallow features. The proposed method successfully aggregated the complementary information between shallow and deep features by formulating and bridging the feature gap. Experiments on two abdominal MRI datasets demonstrated the effectiveness of the proposed method, which improved the segmentation performance over a baseline with nearly zero additional parameters. Particularly, the proposed method has advantages for segmenting organs with blurred boundaries or in a small scale, achieving superior performance than state-of-the-art methods.

Online Action Detection (OAD) has attracted more and more attention in recent years. A network for OAD generally consists of three parts: a frame-level feature extractor, a temporal modeling module, and an action classifier. Most recent OAD networks use a single-channel Recurrent Neural Network (RNN) to capture long-term history information, with spatial and temporal features concatenated as network input. In OAD, spatial features describe object appearance and scene configuration within each frame while temporal features capture motion cues over time. It is crucial to effectively fuse both spatial and temporal features. In this paper, we propose a new framework named TwinLSTM based on two-channel Long Short-Term Memory (LSTM) network for OAD, in which each channel is used to extract and handle either spatial features or temporal features. To more effectively fuse both spatial and temporal features, we design a prediction fusion module (PFM) to utilize hidden states of both channels to obtain more action content, including information interaction and future context prediction. We evaluate TwinLSTM on two challenging datasets: THUMOS14 and HDD. Experiments show that TwinLSTM outperforms existing single-channel models by a significant margin. We also show the effectiveness of PFM through comprehensive ablation studies.

Despite end-to-end deep learning methods have recently advanced state-of-the-art for dynamic scene deblurring, they are often biased towards learning low-frequency (LF) information, thus missing sufficient high-frequency (HF) details. In this paper, we experimentally verify that different image frequencies affect the final deblurring quality in different manners. Considering this, we point out that the LF learning bias problem arises from the existing training scheme with frequencies coupled, to some extent. Concretely, current training scheme fails to distinguish different frequencies but optimize them as a whole towards one common objective, thereby resulting in sub-optimal results. To ameliorate this problem, we propose an alternative training strategy, namely Decoupled Frequency Learning (DFL). Specifically, DFL treats deblurring task as two separate sub-tasks, which correspond to image LF and HF components, respectively. Different losses are tailored-designed for different frequencies to better guide their learning towards appropriate objectives. The proposed DFL scheme is simple yet effective, and compatible to any existing deep models. Extensive experiments on public benchmarks demonstrate its clear benefits to the state-of-the-art in terms of both quantitative measures and perceptual quality.

Prior learning is a fundamental problem in the field of image processing. In this paper, we conduct a detailed study on (1) how to model and learn the prior of the image patch group, which consists of a group of non-local similar image patches, and (2) how to apply the learned prior to the whole image denoising task. To tackle the first problem, we propose a new prior model named Group Sparsity Mixture Model (GSMM). With the bilateral matrix multiplication, the GSMM can model both the local feature of a single patch and the relation among non-local similar patches, and thus it is very suitable for patch group based prior learning. This is supported by the parameter analysis which demonstrates that the learned GSMM successfully captures the inherent strong sparsity embodied in the image patch group. Besides, as a mixture model, GSMM can be used for patch group classification. This makes the image denoising method based on GSMM capable of processing patch groups flexibly. To tackle the second problem, we propose an efficient and effective patch group based image denoising framework, which is plug-and-play and compatible with any patch group prior model. Using this framework, we construct two versions of GSMM based image denoising methods, both of which outperform the competing methods based on other prior models, e.g., Field of Experts (FoE) and Gaussian Mixture Model (GMM). Also, the better version is competitive with the state-of-the-art model based method WNNM with about ×8 faster average running speed.

The lack of clean images undermines the practicability of supervised image prior learning methods, of which the training schemes require a large number of clean images. To free image prior learning from the image collection burden, a novel Self-Supervised learning method for Gaussian Mixture Model (SS-GMM) is proposed in this paper. It can simultaneously achieve the noise level estimation and the image prior learning directly from only a single noisy image. This work is derived from our study on eigenvalues of the GMM’s covariance matrix. Through statistical experiments and theoretical analysis, we conclude that (1) covariance eigenvalues for clean images hold the sparsity; and that (2) those for noisy images contain sufficient information for noise estimation. The first conclusion inspires us to impose a sparsity constraint on covariance eigenvalues during the learning process to suppress the influence of noise. The second conclusion leads to a self-contained noise estimation module of high accuracy in our proposed method. This module serves to estimate the noise level and automatically determine the specific level of the sparsity constraint. Our final derived method requires only minor modifications to the standard expectation-maximization algorithm. This makes it easy to implement. Very interestingly, the GMM learned via our proposed self-supervised learning method can even achieve better image denoising performance than its supervised counterpart, i.e., the EPLL. Also, it is on par with the state-of-the-art self-supervised deep learning method, i.e., the Self2Self. Code is available at https://github.com/HUST-Tan/SS-GMM.

Many analysis-based regularizations proposed so far employ a common prior information, i.e., edges in an image are sparse. However, in local edge regions and texture regions, this prior may not hold. As a result, the performance of regularizations based on the edge sparsity may be unsatisfactory in such regions for image-related inverse problems. These regularizations tend to smooth out the edges while eliminating the noise. In other words, these regularizations' abilities of preserving edges are limited. In this paper, a new prior that the corner points in a natural image are sparse was proposed to construct regularizations. Intuitively, even in local edge regions and texture regions, the sparsity of corner points may still exist, and hence, the regularizations based on it can achieve better performance than those based on the edge sparsity. As an example, by utilizing the sparsity of corner points, we proposed a new regularization based on Noble's corner measure function. Our experiments demonstrated the excellent performance of the proposed regularization for both image denoising and deblurring problems, especially in local edge regions and texture regions.

Cone-beam computed tomography (CBCT) plays an important role in radiation therapy. Statistical iterative reconstruction (SIR) algorithms with specially designed penalty terms provide good performance for low-dose CBCT imaging. Among others, the total variation (TV) penalty is the current state-of-the-art in removing noises and preserving edges, but one of its well-known limitations is its staircase effect. Recently, various penalty terms with higher order differential operators were proposed to replace the TV penalty to avoid the staircase effect, at the cost of slightly blurring object edges. We developed a novel SIR algorithm using a neural network for CBCT reconstruction. We used a data-driven method to learn the "potential regularization term" rather than design a penalty term manually. This approach converts the problem of designing a penalty term in the traditional statistical iterative framework to designing and training a suitable neural network for CBCT reconstruction. We proposed using transfer learning to overcome the data deficiency problem and an iterative deblurring approach specially designed for the CBCT iterative reconstruction process during which the noise level and resolution of the reconstructed images may change. Through experiments conducted on two physical phantoms, two simulation digital phantoms, and patient data, we demonstrated the excellent performance of the proposed network-based SIR for CBCT reconstruction, both visually and quantitatively. Our proposed method can overcome the staircase effect, preserve both edges and regions with smooth intensity transition, and provide reconstruction results at high resolution and low noise level.

Cone-beam computed tomography (CBCT) has been widely used in radiation therapy. For accurate patient setup and treatment target localization, it is important to obtain high-quality reconstruction images. The total variation (TV) penalty has shown the state-of-the-art performance in suppressing noise and preserving edges for statistical iterative image reconstruction, but it sometimes leads to the so-called staircase effect. In this paper, we proposed to use a new family of penalties-the Hessian Schatten (HS) penalties-for the CBCT reconstruction. Consisting of the second-order derivatives, the HS penalties are able to reflect the smooth intensity transitions of the underlying image without introducing the staircase effect. We discussed and compared the behaviors of several convex HS penalties with orders 1, 2, and for CBCT reconstruction. We used the majorization-minimization approach with a primal-dual formulation for the corresponding optimization problem. Experiments on two digital phantoms and two physical phantoms demonstrated the proposed penalty family's outstanding performance over TV in suppressing the staircase effect, and the HS penalty with order 1 had the best performance among the HS penalties tested.

Level sets and graph cuts are two state-of-the-art image segmentation methods in use today. The two methods are apparently different from each other not only because they originate from different theory foundations but also because they employ image information in different ways — level sets typically use image information in a point-wise way, whereas graph cuts use image information in a pairwise way. In this paper, we derive an equivalence relationship between the two methods through kernel technology. In particular, we show that the kernelization of the Chan-Vese (CV) functional — a functional widely used in the level set community — is exactly the energy optimized in the average association — a well-known graph cut criterion. We refer to the level sets method using the kernelized version of the CV functional as kernel active contour. The kernel active contour has computational complexity O(n2) due to the involved kernel technology. We propose a fast implementation for kernel active contour with computational complexity only O(n) using random projection. The kernel active contour is evaluated on synthetic and real images and compared with several existing level set and graph cut methods for image segmentation.

In this paper, we present a novel frame-based denoising algorithm for photon-limited 3-D images. We first construct a new 3-D nonseparable filterbank by adding elements to an existing frame in a structurally stable way. In contrast with the traditional 3-D separable wavelet system, the new filterbank is capable of using edge information in multiple directions. We then propose a data-adaptive hysteresis thresholding algorithm based on this new 3-D nonseparable filterbank. In addition, we develop a new validation strategy for denoising of photon-limited images containing sparse structures, such as neurons (the structure of interest is less than 5% of total volume). The validation method, based on tubular neighborhoods around the structure, is used to determine the optimal threshold of the proposed denoising algorithm. We compare our method with other state-of-the-art methods and report very encouraging results on applications utilizing both synthetic and real data.

Prior models play an important role in the wavelet-based Bayesian image estimation problem. Although it is well known that a residual dependency structure always remains among natural image wavelet coefficients, only few multivariate prior models with a closed parametric form are available in the literature. In this paper, we develop new multivariate prior models that not only match well with the observed statistics of the wavelet coefficients of natural images, but also have a simple parametric form. These prior models are very effective for Bayesian image estimation and lead to an improved estimation performance over related earlier techniques.

A new system, called dual ridgelet frame, is introduced. The construction of the dual ridgelet frame starts with a dual frame constructed using a biorthogonal wavelet basis in the Radon domain, and, then, the image of the resulting dual frame under an isometric map from the Radon domain to the L/sup 2/(R/sup 2/) spatial domain is a dual frame again, and we call it a dual ridgelet frame. The dual ridgelet frame can be thought of as an extension of the notion of orthonormal ridgelet. It provides a more flexible and effective tool for image analysis and processing applications. The high performance of the dual ridgelet frame for image denoising is demonstrated experimentally.

In this paper, we aim at analyzing the characteristic of neuronal population responses to instantaneous or time-dependent inputs and the role of synapses in neural information processing. We have derived an evolution equation of the membrane potential density function with synaptic depression, and obtain the formulas for analytic computing the response of instantaneous re rate. Through a technical analysis, we arrive at several signi cant conclusions: The background inputs play an important role in information processing and act as a switch betwee temporal integration and coincidence detection. the role of synapses can be regarded as a spatio-temporal lter; it is important in neural information processing for the spatial distribution of synapses and the spatial and temporal relation of inputs. The instantaneous input frequency can affect the response amplitude and phase delay.