Single-frame infrared small target (SIRST) detection is crucial for both military and civilian applications, but remains challenging due to low resolution and small target sizes. Most existing methods model the detection task as a semantic segmentation task, which requires high-resolution feature maps and incurs significant computational costs. Moreover, manual annotations often struggle to achieve pixel-level precision, and the inherent ambiguity in the annotations can affect the training outcomes. This paper treats the SIRST detection task as a bounding box regression problem and proposes a novel target detection network architecture, named adaptive fusion bounding box regression network (ABRNet). Specifically, to address the challenges posed by complex and changeable backgrounds, we design an adaptive receptive field module. This module utilizes spatial selection masks to choose feature maps with varying receptive field sizes, thereby leveraging the unique prior knowledge inherent in different scenarios. In addition, we introduce a cross-scale feature encoding fusion structure to alleviate the network’s low tolerance to bounding box perturbations. The module fuses multi-scale local and global features to capture the fine details of small targets. By combining high-dimensional features with detailed features, it facilitates accurate bounding box regression, thereby improving detection performance. Additionally, we employ linear interval mapping to achieve dynamic balancing of hard samples. Experimental results on public datasets demonstrate ABRNet’s superiority over state-of-the-art methods.

Accurate segmentation of cardiac structures in echocardiography videos is vital for diagnosing heart disease. However, challenges such as speckle noise, low spatial resolution, and incomplete video annotations hinder the accuracy and efficiency of segmentation tasks. Existing video-based segmentation methods mainly utilize optical flow estimation and cross-frame attention to establish pixel-level correlations between frames, which are usually sensitive to noise and have high computational costs. In this paper, we present an innovative echocardiography video segmentation framework that exploits the inherent spatio-temporal correlation of echocardiography video feature tensors. Specifically, we perform adaptive tensor singular value decomposition (t-SVD) on the video semantic feature tensor within a learnable 3D transform domain. By utilizing learnable thresholds, we preserve the principal singular values to reduce redundancy in the high-dimensional spatio-temporal feature tensor and enforce its potential low-rank property. Through this process, we can capture the temporal evolution of the target tissue by effectively utilizing information from limited labeled frames, thus overcoming the constraints of sparse annotations. Furthermore, we introduce a memory flow method that propagates relevant information between adjacent frames based on the multi-scale affinities to precisely resolve frame-to-frame variations of dynamic tissues, thereby improving the accuracy and continuity of segmentation results. Extensive experiments conducted on both public and private datasets validate the superiority of our proposed method over state-of-the-art methods, demonstrating improved performance in echocardiography video segmentation.

Highly undersampled schemes in magnetic resonance fingerprinting (MRF) typically lead to aliasing artifacts in reconstructed images, thereby reducing quantitative imaging accuracy. Existing studies mainly focus on improving the reconstruction quality by incorporating temporal or spatial data priors. However, these methods seldom exploit the underlying MRF data structure driven by imaging physics and usually suffer from high computational complexity due to the high-dimensional nature of MRF data. In addition, data priors constructed in a pixel-wise manner struggle to incorporate non-local and non-linear correlations. To address these issues, we introduce a novel MRF reconstruction framework based on the graph embedding framework, exploiting non-linear and non-local redundancies in MRF data. Our work remodels MRF data and parameter maps as graph nodes, redefining the MRF reconstruction problem as a structure-preserved graph embedding problem. Furthermore, we propose a novel scheme for accurately estimating the underlying graph structure, demonstrating that the parameter nodes inherently form a low-dimensional representation of the high-dimensional MRF data nodes. The reconstruction framework is then built by preserving the intrinsic graph structure between MRF data nodes and parameter nodes and extended to exploiting the globality of graph structure. Our approach integrates the MRF data recovery and parameter map estimation into a single optimization problem, facilitating reconstructions geared toward quantitative accuracy. Moreover, by introducing graph representation, our methods substantially reduce the computational complexity, with the computational cost showing a minimal increase as the data acquisition length grows. Experiments show that the proposed method can reconstruct high-quality MRF data and multiple parameter maps within reduced computational time.

Deep unrolling networks that utilize sparsity priors have achieved great success in dynamic magnetic resonance (MR) imaging. The convolutional neural network (CNN) is usually utilized to extract the transformed domain, and then the soft thresholding (ST) operator is applied to the CNN-transformed data to enforce the sparsity priors. However, the ST operator is usually constrained to be the same across all channels of the CNN-transformed data. In this paper, we propose a novel operator, called soft thresholding with channel attention (AST), that learns the threshold for each channel. In particular, we put forward a novel deep unrolling shrinkage network (DUS-Net) by unrolling the alternating direction method of multipliers (ADMM) for optimizing the transformed l1norm dynamic MR reconstruction model. Experimental results on an open-access dynamic cine MR dataset demonstrate that the proposed DUS-Net outperforms the state-of-the-art methods. The source code is available at https://github.com/yhao-z/DUS-Net.

We proposed a learned masked robust principal component analysis (LMRPCA) algorithm for single-frame infrared small target detection. Firstly, the original images are constructed into patch images, which are separated into low-rank and sparse components corresponding to the backgrounds and foreground masks. The optimization function is solved by alternating directions of multipliers method (ADMM), which is mapped to trainable convolutional layers. We use elements of convolutional sparse coding to improve representation learning for foreground masks and side information in the auxiliary transform domain. By doing so, we assign learnable weights to different feature maps by using a reweighted−l1− l1minimization. Numerical experiments show that our proposed LMRPCA can segment and locate the targets precisely.

Clouds often appear in remote sensing images, which seriously affect the application of remote sensing images. Therefore, cloud removal is an important preprocessing process in remote sensing image applications. In this paper, we propose a generative adversarial network-based cloud removal method for optical remote sensing images with the assistance of synthetic aperture radar (SAR) images. Our model is an end-to-end model, which consists of a translation module, an attention module, a generator, and a discriminator . We introduce the attention mechanism to accurately locate the cloud regions. With the obtained attention maps as the prior information, the proposed method can remove the clouds while preserving the cloud-free regions. In addition, we include the structural similarity index (SSIM) and the attention penalty in the loss function to improve the performance of the proposed method. Numerical experiments show that the proposed model provides improved cloud removal performance compared with the state-of-the-art methods.

Detection of infrared dim and small targets with diverse and cluttered background plays a significant role in many applications. In this paper, we propose a deep low-rank and sparse patch-image network, termed as Deep-LSP-Net, to effectively detect small targets in a single infrared image. Specifically, by using the local patch construction scheme, we first transform the original infrared image into a patch-image, which can be decomposed as a superposition of the low-rank background component and the sparse target component. The target detection is thus formulated as an optimization problem with low-rank and sparse regularizations, which can be solved by the alternating direction method of multipliers (ADMM). We unroll the iterative algorithm into deep neural networks, where a generalized sparsifying transform and a singular value thresholding operator are learned by the convolutional neural networks (CNNs) to avoid tedious parameter tuning and improve the interpretability of the neural networks. We conduct comprehensive experiments on two public datasets. Both qualitative and quantitative experimental results demonstrate that the proposed algorithm can obtain improved performance in small infrared target detection compared with state-of-the-art algorithms.

Long-wave infrared (LWIR) bands in multispectral datasets are extremely useful in many applications. However, the LWIR bands usually suffer from undesirable stripe noise, which impedes their further application. Compared with emission-dominated LWIR, the mid-wave infrared (MWIR) bands containing both emitted and reflected radiation usually exhibit higher image quality. In this article, we propose a novel two-stage MWIR energy separation and image guidance (MES-IG) algorithm to destripe the LWIR images with the assistance of the MWIR bands. In the first stage, we decompose the MWIR image into the emitted and reflected components by solving a constrained optimization problem. Specifically, we impose the low-rank penalty to enforce the similarities between MWIR and LWIR, and we use the total variation (TV) regularization to exploit the similarities between MWIR and visible and near-infrared (VNIR) images. In the second stage, the obtained emitted component of MWIR is considered as the guidance image to remove the stripes in the LWIR images by adopting the 1-D guided filter algorithm. Numerical experiments on the Chinese Gaofen-5 satellite and the Moderate Resolution Imaging Spectroradiometer (MODIS) data demonstrate the utility of the proposed method in providing improved LWIR image destriping performance over the state-of-the-art algorithms.

Magnetic resonance fingerprinting (MRF) can rapidly perform simultaneous imaging of multiple tissue parameters. However, the rapid acquisition schemes used in MRF inevitably introduce aliasing artifacts in the recovered tissue fingerprints, reducing the accuracy of the predicted parameter maps. Current regularized reconstruction methods are based on iterative procedures which are usually time-consuming. In addition, most of the current deep learning-based methods for MRF often lack interpretability owing to the black-box nature, and most deep learning-based methods are not applicable for non-Cartesian scenarios, which limits the practical applications. In this paper, we propose a joint reconstruction model incorporating MRF-physics prior and the data correlation constraint for non-Cartesian MRF reconstruction. To avoid time-consuming iterative procedures, we unroll the reconstruction model into a deep neural network. Specifically, we propose a learned CANDECOMP/PARAFAC (CP) decomposition module to exploit the tensor low-rank priors of high-dimensional MRF data, which avoids computationally burdensome singular value decomposition. Inspired by the MRF-physics, we also propose a Bloch response manifold module to learn the mapping between reconstructed MRF data and the multiple parameter maps. Numerical experiments show that the proposed network can reconstruct high-quality MRF data and multiple parameter maps within significantly reduced computational time.

In recent years, deep neural networks have been widely used for object detection in optical remote sensing images. Most deep learning models focus on local feature extraction. However, they usually have the limitation of lack of explicability and are not capable of utilizing the rich contextual relationships over different objects. In this paper, we introduce a novel deep learning network for arbitrary-oriented object detection of optical remote sensing images. The proposed network incorporates the self-attention mechanism and relationship reasoning module to improve the detection performance. Specifically, we use a position attention module and a channel attention module on top of feature proposal network (FPN) to get a refined feature map. Moreover, we introduce a semantic relationship attention module, which employs self-attention mechanism in the semantic space to get an improved classifier. In addition, in order to improve the interpretability of the proposed network, we introduce a relationship reasoning module to exploit the prior knowledge of different objects. Experimental results on the public dataset of DOTA show that the proposed method is able to achieve improved performance compared with the state-of-the-art methods on aerial objects detection.

Small infrared target detection is a significant technique in both civil and military applications. Regularized optimization methods that exploit both the sparsity and the low-rank prop-erties of the infrared image have achieved good performance. In this paper, we propose to unroll the sparse and low-rank regularized model to a deep neural network to effectively sep-arate the infrared target and the background. Specifically, we adopt the infrared patch-image (IPI) model to transform the original infrared image into a patch-image using local patch construction. A deep network flow graph is proposed by si-multaneously exploiting a learned low-rank prior and a spar-sity prior to promote the target detection performance. Exper-imental results demonstrate that the proposed IPI-net is able to provide improved performance in small infrared target de-tection compared with the state-of-the-art algorithms.

The degradation of spaceborne hyperspectral images (HSIs) usually results from various types of noise. In this letter, we propose a 3D hybrid higher degree total variation regularized nonconvex local low-rank tensor recovery (H2DTV-NLRTR) model to restore the HSIs. Inspired by the good performance of the higher degree total variation penalty in image denoising, we first develop a 3D hybrid higher degree total variation penalty term, which is able to capture the fine image details and edges along the spatial dimensions and spectral dimension. The tensor multi-Schatten-pnorm is chosen as the relaxation of the low-rank tensor constraint, which can not only separate the low-rank clean HSI patches from noisy images effectively but also improve the computational efficiency. The proposed H2DTV-NLRTR model can simultaneously characterize the spectral correlation and the spatial structure of the HSI dataset by incorporating the H2DTV penalty in the nonconvex local low-rank tensor recovery problem. In addition, we adopt a fast iterative majorize-minimize algorithm to efficiently solve the corresponding optimization problem. The numerical experiments on both simulated and real HSI datasets demonstrate that the proposed algorithm provides consistently improved restoration results compared with the state-of-the-art algorithms.

Due to the capability of fast multiparametric quantitative imaging, magnetic resonance fingerprinting (MRF) is becoming a promising quantitative magnetic resonance imaging approach. However, the artifacts caused by the highly undersampled data acquisition lead to inaccurate estimation of the tissue parameter maps. Based on the assumption that the 3-D MRF data can be modeled as a piecewise smooth signal, with the discontinuities localized to the zero sets of a bandlimited function, we exploit the low-rank property of the structured Toeplitz matrix constructed from the Fourier measurements. In addition, we adopt the subspace projection scheme to improve the accuracy of parameter estimation. In order to efficiently solve the regularized problem, we propose an iterative two-stage algorithm, which alternately updates the k -space data and projects the space-time matrix into the dictionary space. Numerical experiments demonstrate that the proposed algorithm shows significant improvement in MRF time-series images reconstruction and can provide more accurate parameter maps over the state-of-the-art algorithms.

Long-wave infrared (LWIR) images have important applications in retrieving land surface temperature. However, LWIR images are often inevitably suffered from stripe noise, a special type of spatial domain fixed pattern noise. This paper proposes a novel spectral separation algorithm for LWIR image destriping. Specifically, since the mid-wave infrared (MWIR) bands contain both radiant and reflective energy, we use MWIR as reference images to remove the stripe noise in the LWIR bands. We formulate the spectral separation problem as a convex optimization problem, where the difference between LWIR and the radiant component of MWIR, and the difference between the visible and near infrared (VNIR) image and the reflective component of MWIR are regularized to exploit the similarities between the corresponding bands. The obtained radiant component is then utilized to recover the LWIR band. Experimental results using Gaofen-5 datasets demonstrate that the proposed algorithm has good performance in removing the stripe noise.

Four-dimensional magnetic resonance imaging (4D-MRI) can provide 3D tissue properties and the temporal profiles at the same time. However, further applications of 4D-MRI is limited by the long acquisition time and motion artifacts. We introduce a regularized image reconstruction method to recover 4D MR images from their undersampled Fourier coefficients, named HDTV-LLR. We adopt the three-dimensional higher degree total variation and the local low-rank penalties to simultaneously exploit the spatial and temporal correlations of the dataset. In order to solve the resulting optimization problem efficiently, we propose a fast alternating minimization algorithm. The performance of the proposed method is demonstrated in the context of 4D cardiac MR images reconstruction with undersampling factors of 12 and 16. The proposed method is compared with iGRASP, and schemes using either low-rank or sparsity constraint alone. Numerical results show that the proposed method enables accelerated 4D-MRI with improved image quality and reduced artifacts.

Due to the sensor malfunction or poor observation conditions, optical remote sensing images often suffer from information loss such as dead pixels or cloud contamination. We propose a remote sensing image inpainting method based on structured low-rank matrix approximation. The hybrid regularizations are applied to recover the piecewise constant and the piecewise linear components of the image separately by exploiting the low-rank properties of the structured Toeplitz matrices of the two image components. The corresponding optimization problem can be solved using the half-circulant approximation of the Toeplitz matrix. Experimental results demonstrate the efficacy of the proposed method.

Hyperspectral image (HSI) restoration is an important preprocessing step in HSI data analysis to improve the image quality for subsequent applications of HSI. In this article, we introduce a spatial-spectral patch-based nonconvex sparsity and low-rank regularization method for HSI restoration. In contrast to traditional approaches based on convex penalties or nonconvex spectral penalty alone, we consider the sparsity of HSI in the spatial-spectral domain and combine the nonconvex low-rank penalty and the nonconvex 3-D total variation (TV)-like sparsity regularization to fully exploit the correlations in both spatial-spectral dimensions of the HSI data set. In addition, we propose a fast iterative variable splitting-based algorithm to effectively solve the corresponding optimization problem. Numerical experiments on both simulated and real HSI data sets demonstrate that the proposed nonconvex low-rank and TV (NonLRTV) method significantly improves the recovered image quality compared with the state-of-the-art algorithms.

Hyperspectral image (HSI) restoration is an essential pre-processing step in order to obtain more useful images for subsequent applications. However, traditional methods based on convex regularization or nonconvex spectral penalty alone are not able to fully exploit the spatial-spectral properties of the HSI datasets. In this paper, by utilizing the nonconvex spectral penalty and the nonconvex spatial penalty, we propose a novel nonconvex hybrid regularization (NHR) model, which can preserve the image features and remove the mixed noise, including Gaussian noise, stripes, deadlines, and etc. The corresponding optimization problem can be efficiently solved using an iterative algorithm based on the Augmented Lagrangian Multipliers (ALM) method. Experimental results on both simulated and real HSI images prove that the proposed NHR method significantly improves the image quality.

Recent theory of mapping an image into a structured low-rank Toeplitz or Hankel matrix has become an effective method to restore images. In this paper, we introduce a generalized structured low-rank algorithm to recover images from their undersampled Fourier coefficients using infimal convolution regularizations. The image is modeled as the superposition of a piecewise constant component and a piecewise linear component. The Fourier coefficients of each component satisfy an annihilation relation, which results in a structured Toeplitz matrix. We exploit the low-rank property of the matrices to formulate a combined regularized optimization problem. In order to solve the problem efficiently and to avoid the high-memory demand resulting from the large-scale Toeplitz matrices, we introduce a fast and a memory-efficient algorithm based on the half-circulant approximation of the Toeplitz matrix. We demonstrate our algorithm in the context of single and multi-channel MR images recovery. Numerical experiments indicate that the proposed algorithm provides improved recovery performance over the state-of-the-art approaches.

In High Efficiency Video Coding (HEVC), a quad-tree based Coding Unit (CU) partitioning scheme is employed, achieving a substantial improvement in coding efficiency compared with previous standards. The superior coding efficiency of HEVC is achieved at the expense of greatly increased complexity. A fast intra coding scheme consisting of fast CU depth decision and fast prediction mode decision is proposed to reduce the computational requirement. Classification of the homogeneity of video content using an adaptive double thresholds scheme is employed to reduce the number of Rate Distortion (RD) evaluations. The partition information of spatially neighbouring CUs is utilised to further narrow the depth range. The construction of the initial candidate list is improved for each Prediction Unit (PU). Then the prediction mode correlation between neighbouring quad-tree coding levels is considered to predict the most likely coding mode. The Hadamard cost of prediction modes is examined to further reduce the candidate modes. The computational complexity of HEVC intra coding is therefore reduced. Simulation results show that the proposed algorithm reduces encoding time by up to 71% compared with the HM 13.0 implementation, while having a negligible impact on rate distortion, with increases in bit-rate of 1.82%.

We introduce a novel image regularization termed as multiple degree total variation (MDTV). This type of regularization combines the first and second degree directional derivatives, thus providing a good balance between preservation of edges and region smoothness. In order to solve the resulting optimization problem, we proposed a fast majorize minimize algorithm. We demonstrate the utility of the MDTV regularization in the context of image denoising and compressed sensing. We compare the proposed method with standard TV, and the state of the art higher degree methods, including higher degree total variation (HDTV) and total generalized variation (TGV) based schemes. Numerical results indicate that MDTV penalty provides improved image recovery performance.

HEVC employs a quad-tree based Coding Unit (CU) structure to achieve a significant improvement in coding efficiency compared with previous standards. However, the computational complexity is greatly increased. We proposed a fast mode decision algorithm to reduce intra coding complexity. Firstly, an initial candidate list of intra modes is constructed for each Prediction Unit (PU). The prediction mode correlation between adjacent quad-tree coding levels and between temporal neighbouring frames is used to predict the most likely coding mode. The number of prediction mode that need to be evaluated in residual quad-tree (RQT) process is further reduced by taking the Hadamard cost of prediction mode into consideration. Simulation results show that the proposed algorithm saves encoding time by up to 51% compared with the HM 13.0 implementation, while having a negligible impact on rate distortion.

High Efficiency Video Coding (HEVC) incorporates a flexible quad-tree block partitioning scheme and up to 35 prediction modes for intra coding. This enables a significant improvement in coding efficiency compared with previous standards. The superior coding efficiency of HEVC is achieved at the expense of greatly increased complexity. A fast Coding Unit (CU) depth decision algorithm is proposed to reduce the computational requirement for intra coding. An adaptive double thresholds scheme is employed to classify the homogeneity of video content. The classification is used to reduce the number of Rate Distortion (RD) evaluations in the CU depth decision process. The partition information of the temporally co-located CU and the spatially neighbouring CUs is jointly utilised to further narrow the depth range that needs to be evaluated. The computational complexity of HEVC intra coding is therefore reduced. Simulation results show that the proposed algorithm reduces encoding time by up to 57% compared with the HM 13.0 implementation, while having a negligible impact on rate distortion, with PSNR losses of 0.01dB and increases in bit-rate of 0.31%.

We introduce a family of novel image regularization penalties called generalized higher degree total variation (HDTV). These penalties further extend our previously introduced HDTV penalties, which generalize the popular total variation (TV) penalty to incorporate higher degree image derivatives. We show that many of the proposed second degree extensions of TV are special cases or are closely approximated by a generalized HDTV penalty. Additionally, we propose a novel fast alternating minimization algorithm for solving image recovery problems with HDTV and generalized HDTV regularization. The new algorithm enjoys a tenfold speed up compared with the iteratively reweighted majorize minimize algorithm proposed in a previous paper. Numerical experiments on 3D magnetic resonance images and 3D microscopy images show that HDTV and generalized HDTV improve the image quality significantly compared with TV.

We introduce novel image regularization penalties to overcome the practical problems associated with the classical total variation (TV) scheme. Motivated by novel reinterpretations of the classical TV regularizer, we derive two families of functionals involving higher degree partial image derivatives; we term these families as isotropic and anisotropic higher degree TV (HDTV) penalties, respectively. The isotropic penalty is the L(1) - L(2) mixed norm of the directional image derivatives, while the anisotropic penalty is the separable L(1) norm of directional derivatives. These functionals inherit the desirable properties of standard TV schemes such as invariance to rotations and translations, preservation of discontinuities, and convexity. The use of mixed norms in isotropic penalties encourages the joint sparsity of the directional derivatives at each pixel, thus encouraging isotropic smoothing. In contrast, the fully separable norm in the anisotropic penalty ensures the preservation of discontinuities, while continuing to smooth along the linelike features; this scheme thus enhances the linelike image characteristics analogous to standard TV. We also introduce efficient majorize-minimize algorithms to solve the resulting optimization problems. The numerical comparison of the proposed scheme with classical TV penalty, current second-degree methods, and wavelet algorithms clearly demonstrate the performance improvement. Specifically, the proposed algorithms minimize the staircase and ringing artifacts that are common with TV and wavelet schemes, while better preserving the singularities. We also observe that anisotropic HDTV penalty provides consistently improved reconstructions compared with the isotropic HDTV penalty.

We introduce a novel algorithm to recover sparse and low-rank matrices from noisy and undersampled measurements. We pose the reconstruction as an optimization problem, where we minimize a linear combination of data consistency error, nonconvex spectral penalty, and nonconvex sparsity penalty. We majorize the nondifferentiable spectral and sparsity penalties in the criterion by quadratic expressions to realize an iterative three-step alternating minimization scheme. Since each of these steps can be evaluated either analytically or using fast schemes, we obtain a computationally efficient algorithm. We demonstrate the utility of the algorithm in the context of dynamic magnetic resonance imaging (MRI) reconstruction from sub-Nyquist sampled measurements. The results show a significant improvement in signal-to-noise ratio and image quality compared with classical dynamic imaging algorithms. We expect the proposed scheme to be useful in a range of applications including video restoration and multidimensional MRI.

We introduce a novel algorithm to reconstruct dynamic magnetic resonance imaging (MRI) data from under-sampled k-t space data. In contrast to classical model based cine MRI schemes that rely on the sparsity or banded structure in Fourier space, we use the compact representation of the data in the Karhunen Louve transform (KLT) domain to exploit the correlations in the dataset. The use of the data-dependent KL transform makes our approach ideally suited to a range of dynamic imaging problems, even when the motion is not periodic. In comparison to current KLT-based methods that rely on a two-step approach to first estimate the basis functions and then use it for reconstruction, we pose the problem as a spectrally regularized matrix recovery problem. By simultaneously determining the temporal basis functions and its spatial weights from the entire measured data, the proposed scheme is capable of providing high quality reconstructions at a range of accelerations. In addition to using the compact representation in the KLT domain, we also exploit the sparsity of the data to further improve the recovery rate. Validations using numerical phantoms and in vivo cardiac perfusion MRI data demonstrate the significant improvement in performance offered by the proposed scheme over existing methods.