Masked autoencoder (MAE) shows that severe augmentation during training produces robust representations for high-level tasks. This paper brings the MAE-like framework to nighttime image enhancement, demonstrating that severe augmentation during training produces strong network priors that are resilient to real-world night haze degradations. We propose a novel nighttime image dehazing method with self-prior learning. Our main novelty lies in the design of severe augmentation, which allows our model to learn robust priors. Unlike MAE that uses masking, we leverage two key challenging factors of nighttime images as augmentation: light effects and noise. During training, we intentionally degrade clear images by blending them with light effects as well as by adding noise, and subsequently restore the clear images. This enables our model to learn clear background priors. By increasing the noise values to approach as high as the pixel intensity values of the glow and light effect blended images, our augmentation becomes severe, resulting in stronger priors. While our self-prior learning is considerably effective in suppressing glow and revealing details of background scenes, in some cases, there are still some undesired artifacts that remain, particularly in the forms of over-suppression. To address these artifacts, we propose a self-refinement module based on the semi-supervised teacher-student framework. Our NightHaze, especially our MAE-like self-prior learning, shows that models trained with severe augmentation effectively improve the visibility of input haze images, approaching the clarity of clear nighttime images. Extensive experiments demonstrate that our NightHaze achieves state-of-the-art performance, outperforming existing nighttime image dehazing methods by a substantial margin of 15.5% for MUSIQ and 23.5% for ClipIQA.

Removing soft and self shadows that lack clear boundaries from a single image is still challenging. Self shadows are shadows that are cast on the object itself. Most existing methods rely on binary shadow masks, without considering the ambiguous boundaries of soft and self shadows. In this paper, we present DeS3, a method that removes hard, soft and self shadows based on adaptive attention and ViT similarity. Our novel ViT similarity loss utilizes features extracted from a pre-trained Vision Transformer. This loss helps guide the reverse sampling towards recovering scene structures. Our adaptive attention is able to differentiate shadow regions from the underlying objects, as well as shadow regions from the object casting the shadow. This capability enables DeS3 to better recover the structures of objects even when they are partially occluded by shadows. Different from existing methods that rely on constraints during the training phase, we incorporate the ViT similarity during the sampling stage. Our method outperforms state-of-the-art methods on the SRD, AISTD, LRSS, USR and UIUC datasets, removing hard, soft, and self shadows robustly. Specifically, our method outperforms the SOTA method by 16% of the RMSE of the whole image on the LRSS dataset.

Existing deep-learning-based methods for nighttime video deraining rely on synthetic data due to the absence of real-world paired data. However, the intricacies of the real world, particularly with the presence of light effects and low-light regions affected by noise, create significant domain gaps, hampering synthetic-trained models in removing rain streaks properly and leading to over-saturation and color shifts. Motivated by this, we introduce NightRain, a novel nighttime video deraining method with adaptive-rain-removal and adaptive-correction. Our adaptive-rain-removal uses unlabeled rain videos to enable our model to derain real-world rain videos, particularly in regions affected by complex light effects. The idea is to allow our model to obtain rain-free regions based on the confidence scores. Once rain-free regions and the corresponding regions from our input are obtained, we can have region-based paired real data. These paired data are used to train our model using a teacher-student framework, allowing the model to iteratively learn from less challenging regions to more challenging regions. Our adaptive-correction aims to rectify errors in our model's predictions, such as over-saturation and color shifts. The idea is to learn from clear night input training videos based on the differences or distance between those input videos and their corresponding predictions. Our model learns from these differences, compelling our model to correct the errors. From extensive experiments, our method demonstrates state-of-the-art performance. It achieves a PSNR of 26.73dB, surpassing existing nighttime video deraining methods by a substantial margin of 13.7%.

Visibility in hazy nighttime scenes is frequently reduced by multiple factors, including low light, intense glow, light scattering, and the presence of multicolored light sources. Existing nighttime dehazing methods often struggle with handling glow or low-light conditions, resulting in either excessively dark visuals or unsuppressed glow outputs. In this paper, we enhance the visibility from a single nighttime haze image by suppressing glow and enhancing low-light regions. To handle glow effects, our framework learns from the rendered glow pairs. Specifically, a light source aware network is proposed to detect light sources of night images, followed by the APSF (Angular Point Spread Function)-guided glow rendering. Our framework is then trained on the rendered images, resulting in glow suppression. Moreover, we utilize gradient-adaptive convolution, to capture edges and textures in hazy scenes. By leveraging extracted edges and textures, we enhance the contrast of the scene without losing important structural details. To boost low-light intensity, our network learns an attention map, then adjusted by gamma correction. This attention has high values on low-light regions and low values on haze and glow regions. Extensive evaluation on real nighttime haze images, demonstrates the effectiveness of our method. Our experiments demonstrate that our method achieves a PSNR of 30.38dB, outperforming state-of-the-art methods by 13% on GTA5 nighttime haze dataset. Our data and code is available at: https://github.com/jinyeying/nighttime_dehaze.

In this paper, a highly-adaptive unsharp masking (UM) method is proposed and called the blurriness-guided UM, or BUM, in short. The proposed BUM exploits the estimated local blurriness as the guidance information to perform pixel-wise enhancement. The consideration of local blurriness is motivated by the fact that enhancing a highly-sharp or a highly-blurred image region is undesirable, since this could easily yield unpleasant image artifacts due to over-enhancement or noise enhancement, respectively. Our proposed BUM algorithm has two powerful adaptations as follows. First, the enhancement strength is adjusted for each pixel on the input image according to the degree of local blurriness measured at the local region of this pixel's location. All such measurements collectively form the blurriness map, from which the scaling matrix can be obtained using our proposed mapping process. Second, we also consider the type of layer-decomposition filter exploited for generating the base layer and the detail layer, since this consideration would effectively help to prevent over-enhancement artifacts. In this paper, the layer-decomposition filter is considered from the viewpoint of edge-preserving type versus non-edge-preserving type. Extensive simulations experimented on various test images have clearly demonstrated that our proposed BUM is able to consistently yield superior enhanced images with better perceptual quality to that of using a fixed enhancement strength or other state-of-the-art adaptive UM methods.

In this paper, we highlight a new filtering concept and methodology, called the semantic image content filtering (SICF), which aims to remove insignificant small details from the image while preserving its main structure. Such image content separation is not possible to achieve by using any conventional linear filter as it is essentially designed to perform frequency separation. To realize an effective SICF, a novel image filtering algorithm, called the edge-preserving scale-aware filter (ESF), is proposed in this paper. Our proposed ESF yields a significant improvement over a recently-developed scale-aware filter, called the rolling guidance filter (RGF). The key success of our ESF lies in the developed adaptive relative total variation filter (ARTVF), which replaces the RGF's Gaussian filter for generating a much improved initial guidance image. Extensive simulation results obtained from various test images have clearly demonstrated that the proposed ESF outperforms other state-of-the-art methods on conducting SICF task. That is, the semantically-important large-scale image structure has been better preserved, while the insignificant small details have been removed more effectively.

It has been observed that enhancing a highly-blurred image region could often lead to unpleasant noise amplification. Motivated by this, an adaptive unsharp masking (UM) method is proposed in this paper, which incorporates the estimated local blurriness information into the enhancement process to adaptively determine the scaling factor for each pixel on the detail layer. To achieve this goal, a pixel-wise local blurriness estimation method is developed for generating a pixel-wise blurriness map, followed by individually converting each blurriness measurement on the map to a scaling factor via a mapping process. The proposed method not only avoids noise amplification in blurred regions but also addresses `high-level' considerations, such as photographer's original intention on making background more blurred for creating special aesthetic effect. Extensive simulations conducted on various test images have demonstrated that our approach is able to deliver much superior perceptual quality of enhanced images compared to other state-of-the-art UM methods.

A recently introduced image interpolation method, called the contrast-guided interpolation (CGI), has shown superior performance on producing high-quality interpolated image. However, its iterative edge diffusion (IED) process for diffusing continuous-valued directional variation (DV) fields inevitably incurs high computational complexity due to its iterative optimization process. The key objective of this letter lies in how to greatly reduce the computation of this diffusion process while maintaining CGI's superior performance on its interpolated image. The novelty of this letter started with a critical observation as follows. Since each diffused DV field needs to be thresholded for generating a binary contrast-guided decision map (CDM) in the subsequent step, such binarization operation will definitely destroy the fidelity that was preserved previously through the data term of the IED's energy functional. Therefore, the data term is lifted in our approach to yield a new energy functional. It turns out that the diffusion equation derived from this simplified functional is, in fact, the well-known heat equation, from which a highly attractive property of the heat equation can be exploited for conducting diffusion. That is, given a desired amount of diffusion to yield, it can be realized by simply convolving the DV field with a Gaussian kernel once, rather than gradually updating the DV field through iterations. Note that the variance of the Gaussian kernel corresponds to the amount of diffusion desired. As a result, the total computation time is significantly reduced. Extensive simulation results have shown that the proposed CED can generate nearly identical CDMs as those produced by the IED, while only requiring about 1/10 of its computation time. By replacing the IED with the proposed CED in the CGI framework, the total run time of our fast CGI is only 1/4 of the original CGI's on average.

A recently developed demosaicing methodology, called residual interpolation (RI), has demonstrated superior performance over the conventional color-component difference interpolation. However, it has been observed that the existing RI-based methods fail to fully exploit the potential of RI strategy on the reconstruction of the most important G channel, as only the R and B channels are restored through the RI strategy. Since any reconstruction error introduced in the G channel will be carried over into the demosaicing process of the other two channels, this makes the restoration of the G channel highly instrumental to the quality of the final demosaiced image. In this paper, a novel iterative RI (IRI) process is developed for reconstructing a highly accurate G channel first; in essence, it can be viewed as an iterative refinement process for the estimation of those missing pixel values on the G channel. The key novelty of the proposed IRI process is that all the three channels will mutually guide each other until a stopping criterion is met. Based on the restored G channel, the mosaiced R and B channels will be, respectively, reconstructed by exploiting the existing RI method without iteration. Extensive simulations conducted on two commonly-used test datasets for demosaicing algorithms have demonstrated that our algorithm has achieved the best performance in most cases, compared with the existing state-of-the-art demosaicing methods on both objective and subjective performance evaluations.

A new demosaicing approach has been introduced recently, which is based on conducting interpolation on the generated residual fields rather than on the color-component difference fields as commonly practiced in most demosaicing methods. In view of its attractive performance delivered by such residual interpolation (RI) strategy, a new RI-based demosaicing method is proposed in this paper that has shown much improved performance. The key success of our approach lies in that the RI process is iteratively deployed to all the three channels for generating a more accurately reconstructed G channel, from which the R channel and the B channel can be better reconstructed as well. Extensive simulations conducted on two commonly-used test datasets have clearly demonstrated that our algorithm is superior to the existing state-of-the-art demosaicing methods, both on objective performance evaluation and on subjective perceptual quality.