Previous deblurring methods mostly tackle global motion blur due to camera shake but struggle with local motion blur from object movement, facing challenges like the random motion blur locations, data imbalance, directional ambiguity, and positional uncertainty. To fill the vacancy of real-world local motion deblurring, we establish ReLoBlur, the first real-world local motion deblurring dataset. ReLoBlur is captured by a synchronized beam-splitting photographing system and annotated via our developed Local Blur Foreground Mask Generator (LBFMG). To bridge the gap between local and global motion deblurring, we propose a Local Blur-Aware Gated network (LBAG) with gate blocks to focus deblurring on blurred regions, and a Blur-Aware Patch Cropping Strategy (BAPC) to address the data imbalance problem. Acknowledging directional ambiguity and positional uncertainty from shooting errors and non-uniform object motion, we enhance LBAG with LBAGp, guided by center-related distance, and optimized by a symmetric minimization loss. Extensive experiments prove the reliability of the ReLoBlur dataset, and demonstrate that LBAG and LBAGp achieve better local motion deblurring performance compared to state-of-the-art (SOTA) CNN-based deblurring methods.

Lunar exploration has attracted considerable attention, with the lunar poles emerging as the next exploration hot spot for the cold trapping of volatiles in the permanently shadowed regions (PSRs) at these poles. Remote sensing via the satellite’s optical load is one of the most important ways to get the scientific data of PSRs. However, the illumination conditions at the lunar poles are quite different from the low latitude areas and how to get appropriate optical signal remains challenging. Thus, simulation of the optical remote sensing process, which provides reference for the choice of satellites’ imaging parameters to ensure the implementation of lunar exploration project, is of great value. In this article, an optical imaging chain modeling for the PSRs at the lunar south pole, which includes lunar 3-D topography, observing satellite’s orbit, instrument’s parameters, and other environmental parameters, has been built. To demonstrate the physical accuracy, some PSRs’ observations acquired by narrow angle cameras (NACs) equipped on the lunar reconnaissance orbiter (LRO) are compared with the corresponding images simulated by the proposed imaging chain model. The digital value’s difference between the simulated images and real captured images is generally less than 50 for 12-bit images ranging from 0 to 4095, indicating a good fit considering the uncertainty of soil’s absolute reflectance and the noise in the real captured images. In addition, the impact of the imaging chain’s parameters is revealed with the proposed algorithm. The simulation method will provide reference and assist future optical imaging of PSRs.

Push-broom imaging systems, including linear array (LA) and time delay integration (TDI) sensors, are extensively used in remote sensing for high-resolution image acquisition with continuous spatial coverage. However, platform-induced jitter introduces significant distortions and blurring, especially in TDI systems with multiple integration stages, where the cumulative effects of jitter are more pronounced. Traditional jitter models often struggle to accurately simulate these effects in the presence of measurement noise, hindering effective image restoration. In this article, we propose a novel jitter-aware restoration framework that addresses these challenges in both LA and TDI push-broom imaging systems. Central to our approach is the introduction of an equivalent jitter model (EJM) that is robust to measurement noise. By averaging time-shifted jitter curves across multiple integration stages, the EJM effectively smooths out noise-induced fluctuations, providing a reliable characterization of jitter effects. Leveraging this model, we develop a jitter-aware restoration network (JARNet), a two-stage restoration network that combines optical flow correction (OFC) with spatial-frequency residual learning to mitigate geometric distortions and motion blur. We also design a custom data synthesis pipeline to generate realistic jitter-degraded datasets, facilitating effective training of the network. Experimental results on both synthetic LA and TDI datasets demonstrate that JARNet outperforms state-of-the-art methods in terms of peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and gradient magnitude similarity deviation (GMSD) metrics. Our framework offers a robust solution for restoring high-quality remote sensing images degraded by jitter, significantly advancing the state-of-the-art in this domain. The source code will be made publicly available upon publication athttps://github.com/naturezhanghn/EJM.

Linear Array Pushbroom (LAP) imaging technology is widely used in the realm of remote sensing. However, images acquired through LAP always suffer from distortion and blur because of camera jitter. Traditional methods for restoring LAP images, such as algorithms estimating the point spread function (PSF), exhibit limited performance. To tackle this issue, we propose a Jitter-Aware Restoration Network (JARNet), to remove the distortion and blur in two stages. In the first stage, we formulate an Optical Flow Correction (OFC) block to refine the optical flow of the degraded LAP images, resulting in pre-corrected images where most of the distortions are alleviated. In the second stage, for further enhancement of the pre-corrected images, we integrate two jitter-aware techniques within the Spatial and Frequency Residual (SFRes) block: 1) introducing Coordinate Attention (CoA) to the SFRes block in order to capture the jitter state in orthogonal direction; 2) manipulating image features in both spatial and frequency domains to leverage local and global priors. Additionally, we develop a data synthesis pipeline, which applies Continue Dynamic Shooting Model (CDSM) to simulate realistic degradation in LAP images. Both the proposed JARNet and LAP image synthesis pipeline establish a foundation for addressing this intricate challenge. Extensive experiments demonstrate that the proposed two-stage method outperforms state-of-the-art image restoration models. Code is available at https://github.com/JHW2000/JARNet.

Most existing deblurring methods focus on removing global blur caused by camera shake, while they cannot well handle local blur caused by object movements. To fill the vacancy of local deblurring in real scenes, we establish the first real local motion blur dataset (ReLoBlur), which is captured by a synchronized beam-splitting photographing system and corrected by a post-progressing pipeline. Based on ReLoBlur, we propose a Local Blur-Aware Gated network (LBAG) and several local blur-aware techniques to bridge the gap between global and local deblurring: 1) a blur detection approach based on background subtraction to localize blurred regions; 2) a gate mechanism to guide our network to focus on blurred regions; and 3) a blur-aware patch cropping strategy to address data imbalance problem. Extensive experiments prove the reliability of ReLoBlur dataset, and demonstrate that LBAG achieves better performance than state-of-the-art global deblurring methods and our proposed local blur-aware techniques are effective.

Correcting the optical aberrations and the manufacturing deviations of cameras is a challenging task. Due to the limitation on volume and the demand for mass production, existing mobile terminals cannot rectify optical degradation. In this work, we systematically construct the perturbed lens system model to illustrate the relationship between the deviated system parameters and the spatial frequency response (SFR) measured from photographs. To further address this issue, an optimization framework is proposed based on this model to build proxy cameras from the machining samples' SFRs. Engaging with the proxy cameras, we synthetic data pairs, which encode the optical aberrations and the random manufacturing biases, for training the learning-based algorithms. In correcting aberration, although promising results have been shown recently with convolutional neural networks, they are hard to generalize to stochastic machining biases. Therefore, we propose a dilated Omni-dimensional dynamic convolution (DOConv) and implement it in post-processing to account for the manufacturing degradation. Extensive experiments which evaluate multiple samples of two representative devices demonstrate that the proposed optimization framework accurately constructs the proxy camera. And the dynamic processing model is well-adapted to manufacturing deviations of different cameras, realizing perfect computational photography. The evaluation shows that the proposed method bridges the gap between optical design, system machining, and post-processing pipeline, shedding light on the joint of image signal reception (lens and sensor) and image signal processing (ISP).

Time delay and integration cameras (TDI cameras) are widely used in remote sensing areas because they capture high-resolution, high signal-to-noise ratio (SNR) images and to image in low-light environments. However, the image quality captured by TDI cameras may be affected by many degradation factors, including jitter, charge transfer time mismatch and drift angle. Moreover, compared with the single-line push-broom cameras and area gaze cameras used in remote sensing, the degraded effect of the TDI camera may accumulate during the charge accumulation process. In this paper, we present a fast imaging simulation method for remote sensing TDI cameras based on image resampling that can accurately simulate the degraded image quality affected by different degradation factors. The simulated image pairs can provide a sufficient dataset for modern supervised-learning image restoration methods. In addition, we present a novel network, containing a row-attention block and row-encoder block to help resolve the row-variant blur to resolve the degraded images. We test our image restoration method on the simulated degraded image datasets and real images; the results show that the proposed method can effectively restore degraded images. Our restoration method does not rely on auxiliary information detected by high-frequency sensors or multispectral bands, and it achieves better results than other blind restoration methods.

Low-light imaging with handheld mobile devices is a challenging issue. Limited by the existing models and training data, most existing methods cannot be effectively applied in real scenarios. In this paper, we propose a new low-light image restoration method by using the complementary information of short- and long-exposure images. We first propose a novel data generation method to synthesize realistic short- and long-exposure raw images by simulating the imaging pipeline in low-light environment. Then, we design a new long-short-exposure fusion network (LSFNet) to deal with the problems of low-light image fusion, including high noise, motion blur, color distortion and misalignment. The proposed LSFNet takes pairs of short- and long-exposure raw images as input, and outputs a clear RGB image. Using our data generation method and the proposed LSFNet, we can recover the details and color of the original scene, and improve the low-light image quality effectively. Experiments demonstrate that our method can outperform the state-of-the-art methods.

To meet the space limitation of optical elements, free-form surfaces or high-order aspherical lenses are adopted in mobile cameras to compress volume. However, the application of free-form surfaces also introduces the problem of image quality mutation. Existing model-based deconvolution methods are inefficient in dealing with the degradation that shows a wide range of spatial variants over regions. And the deep learning techniques in low-level and physics-based vision suffer from a lack of accurate data. To address this issue, we develop a degradation framework to estimate the spatially variant point spread functions (PSFs) of mobile cameras. When input extreme-quality digital images, the proposed framework generates degraded images sharing a common domain with real-world photographs. Supplied with the synthetic image pairs, we design a Field-Of-View shared kernel prediction network (FOV-KPN) to perform spatial-adaptive reconstruction on real degraded photos. Extensive experiments demonstrate that the proposed approach achieves extreme-quality computational imaging and outperforms the state-of-the-art methods. Furthermore, we illustrate that our technique can be integrated into existing postprocessing systems, resulting in significantly improved visual quality.

Abstract This paper presents a generative adversarial network (GAN) with patch match algorithm to realize a high‐quality digital zooming using two camera modules with different focal lengths. In dual camera system, shorter focal length module produces the wide‐view image with the low resolution. On the other hand, the longer focal length module produces the tele‐view image via optical zooming. The long‐focal image contains more details than short‐focal image and can be used to guide short‐focal image to reconstruct high frequency part. Firstly, a feature extraction block (FEB) is advanced to extract feature of long‐focal image and short focal‐image to reconstruct a wide‐view image with different resolutions. Next, a patch match algorithm is integrated into convolution neural networks (CNN) to fuse information of long‐focal with short‐focal image and generate a new fused image. Finally, the fused image and short‐focal image are merged with a feature fusion block (FFB) to predict high‐resolution images. In addition, generative adversarial networks are used for filtering information integrated by previous network and output the zoomed image. Extensive experiments on benchmark datasets show that our algorithm achieves favorable performance against state‐of‐the‐art methods.

As the popularity of mobile photography continues to grow, considerable effort is being invested in the reconstruction of degraded images. Due to the spatial variation in optical aberrations, which cannot be avoided during the lens design process, recent commercial cameras have shifted some of these correction tasks from optical design to postprocessing systems. However, without engaging with the optical parameters, these systems only achieve limited correction for aberrations. In this work, we propose a practical method for recovering the degradation caused by optical aberrations. Specifically, we establish an imaging simulation system based on our proposed optical point spread function model. Given the optical parameters of the camera, it generates the imaging results of these specific devices. To perform the restoration, we design a spatial-adaptive network model on synthetic data pairs generated by the imaging simulation system, eliminating the overhead of capturing training data by a large amount of shooting and registration. Moreover, we comprehensively evaluate the proposed method in simulations and experimentally with a customized digital-single-lens-reflex camera lens and HUAWEI HONOR 20, respectively. The experiments demonstrate that our solution successfully removes spatially variant blur and color dispersion. When compared with the state-of-the-art deblur methods, the proposed approach achieves better results with a lower computational overhead. Moreover, the reconstruction technique does not introduce artificial texture and is convenient to transfer to current commercial cameras. Project Page: https://github.com/TanGeeGo/ImagingSimulation .

Image blur caused by camera movement is common in long-exposure photography. A recent approach to address image blur is to record camera motion via inertial sensors in imaging equipment such as smartphones and single-lens reflex (SLR) cameras. However, because of device performance limitations, directly estimating a blur kernel from sensor data is infeasible. Previous works that have attempted to correct blurry image content via sensor data have also been susceptible to theoretical defects. Here, we propose a novel method of deblurring images that uses inertial sensors and a short-long-short (SLS) exposure strategy. Assisted short-exposure images captured before and after the formal long-exposure image are employed to correct the sensor data. A half-blind deconvolution algorithm is proposed to refine the estimated kernel. An extra smoothing filter is integrated into the framework to address the coarse initial kernel. Hence, we propose a fast solution for optimization that uses the iteratively reweighted least squares (IRLS) method in the frequency domain. We evaluate these methods via several blind deconvolutions. Quantitative indicators and the visual performance of the image deblurring results show that our method performs better than previous methods in terms of image quality restoration and computational time cost. This method will increase the feasibility of applying deblurring to imaging devices.

Low‐light enhancement methods suffer from the over‐enhancement problem which could induce the loss of the important texture and make images look unnatural. Moreover, some low‐light images contain strong light areas that must be weakened to improve the visual effect. In this study, an enhancement method with strong light weakening and bright halo suppressing is presented. Firstly, the bright channel prior is applied to the inverted image to weaken the strong light both in and around the strong light areas. Then, a dehazing‐type of algorithm with the dark channel prior is employed via superpixel segmentation to enhance the low‐light image. Finally, a revised non‐local denoising method is proposed to further refine the enhanced image. Experimental results showed that the proposed method achieved better visual effects compared with other state‐of‐the‐art methods. Besides, the quantitative evaluation showed that the authors’ method outperforms the other methods both in the aspect of enhancement and denoising.

Recently, the convolutional neural network has brought impressive improvements for object detection. However, detecting tiny objects in large-scale remote sensing images still remains challenging. First, the extreme large input size makes the existing object detection solutions too slow for practical use. Second, the massive and complex backgrounds cause serious false alarms. Moreover, the ultratiny objects increase the difficulty of accurate detection. To tackle these problems, we propose a unified and self-reinforced network called remote sensing region-based convolutional neural network ($\mathcal {R}^{2}$-CNN), composing of backbone Tiny-Net, intermediate global attention block, and final classifier and detector. Tiny-Net is a lightweight residual structure, which enables fast and powerful features extraction from inputs. Global attention block is built upon Tiny-Net to inhibit false positives. Classifier is then used to predict the existence of target in each patch, and detector is followed to locate them accurately if available. The classifier and detector are mutually reinforced with end-to-end training, which further speed up the process and avoid false alarms. Effectiveness of$\mathcal {R}^{2}$-CNN is validated on hundreds of GF-1 images and GF-2 images that are$18\,000 \times 18\,192$pixels, 2.0-m resolution, and$27\,620 \times 29\,200$pixels, 0.8-m resolution, respectively. Specifically, we can process a GF-1 image in 29.4 s on Titian X just with single thread. According to our knowledge, no previous solution can detect the tiny object on such huge remote sensing images gracefully. We believe that it is a significant step toward practical real-time remote sensing systems.

We propose an image stabilization method based on support vector machine (SVM). Since SVM is very effective in solving nonlinear regression problems, an SVM model was constructed and trained to simulate the vibration characteristic. Then this model was used to predict and compensate for the vibration. A simulation system was built and four assessment metrics including the signal-to-noise ratio (SNR), gray mean gradient (GMG), Laplacian (LAP), and modulation transfer function (MTF) were used to verify our approach. Experimental results showed that this new method allows the image plane to locate stably on the CCD, and high quality images can be obtained.

Selecting the optimal parameters for support vector machine (SVM) has long been a hot research topic. Aiming for support vector classification/regression (SVC/SVR) with the radial basis function (RBF) kernel, we summarize the rough line rule of the penalty parameter and kernel width, and propose a novel linear search method to obtain these two optimal parameters. We use a direct-setting method with thresholds to set the epsilon parameter of SVR. The proposed method directly locates the right search field, which greatly saves computing time and achieves a stable, high accuracy. The method is more competitive for both SVC and SVR. It is easy to use and feasible for a new data set without any adjustments, since it requires no parameters to set.

In an image restoration process, to obtain good results is challenging because of the unavoidable existence of noise even if the blurring information is already known. To suppress the deterioration caused by noise during the image deblurring process, we propose a new deblurring method with a known kernel. First, the noise in the measurement process is assumed to meet the Gaussian distribution to fit the natural noise distribution. Second, the first and second orders of derivatives are supposed to satisfy the independent Gaussian distribution to control the non-uniform noise. Experimental results show that our method is obviously superior to the Wiener filter, regularized filter, and Richardson-Lucy (RL) algorithm. Moreover, owing to processing in the frequency domain, it runs faster than the other algorithms, in particular about six times faster than the RL algorithm.