Abstract Generative Adversarial Networks (GANs) have gained prominence in computer vision, with applications that extend to image fusion. Existing fusion methods often require extensive labeled data and task-specific training, limiting their generalizability. To address these limitations, this paper presents the Reconstruction-Guided Generative Adversarial Network (R2GAN), a generic GAN-based approach designed for generic image fusion, including visible-infrared, medical, and multi-focus image fusion. The proposed R2GAN architecture consists of a primary generator to improve fusion capabilities and auxiliary generators to ensure accurate reconstruction of source image features. To optimize the model, we propose a reconstruction-guided loss function to preserve the feature distribution of the source images and improve the consistency between the fused and source images. Additionally, we introduce a semantic segmentation-guided approach to generate a comprehensive and realistic Paired Multi-Focus image dataset (PMF) to train the R2GAN model. Experimental results in multiple fusion tasks demonstrate that R2GAN delivers superior performance, outperforming state-of-the-art image fusion methods. The R2GAN framework source code is available for access on GitHub at https://github.com/CHAHI24680/R2GAN .

Haze is one of the atmospheric image degradations that causes severe distortions to outdoor images such as low contrast, color shift, and structure damage. Due to the unique physical characteristics of haze, the quality of hazy images is not accurately assessed using general-purpose image quality assessment (IQA) approaches. Therefore, several haze-aware IQA approaches have been proposed to provide more efficient dehazing quality evaluation. These approaches extract several haze-aware features to be either combined to form a single IQA metric or fed to a regression model that predicts the dehazing quality. However, these haze-relevant features are extracted using pixel intensity , in which luminance and structure information are inseparable, leading to less correlation between such features and the type of degradation they are supposed to represent. To address this issue, we propose a singular value decomposition (SVD) based IQA metric that can effectively separate the luminance component of an image from structure. This separation offers the ability to accurately evaluate the degradation at two different levels i.e. luminance and structure. The experimental results show that our proposed SVD-based dehazing quality evaluator (SDQE) outperforms the existing state-of-the-art non-reference IQA metrics in terms of accuracy and processing time.

Generative adversarial networks (GANs) represent a significant advance in the field of deep learning for image generation problems. With their ability to generate highly realistic and diverse images, GANs are quickly becoming the preferred technique for a wide range of applications. However, GANs come with several limitations and challenges, chief among which are their collapse mode and instability during training. In this paper, we propose a novel generator GAN model that incorporates the singular value decomposition (SVD) process into the decoder module. The SVD integration allows the generator to recognize the underlying structures in the feature space, resulting in more robust and effective image generation. This is achieved by adjusting the singular values during the training process, effectively optimizing the SVD-based generator to produce images that match the ground truth. Another advantage of SVD integration is its ability to perform discriminative spatial decomposition that clearly reflects the differences between the generated features and the target features. By including SVD in the generator model, the resulting loss values are higher and less prone to gradient vanishing than conventional GANs, such as Pix2Pix. Our proposed SVD-based generator model can be integrated into any auto-encoder architecture, making it as a generic and versatile solution for various image-generation tasks. The effectiveness of the proposed SVD-based GAN in image generation has been validated in three challenging benchmarks for image restoration and visible-to-infrared image translation. Extensive experiments demonstrate the significant quantitative and qualitative improvements achieved by our SVD-based GAN compared to baseline GAN architectures. The overall system outperforms the current state of the art in all benchmarks tested.

Eye blink detection is a challenging problem that many researchers are working on because it has the potential to solve many facial analysis tasks, such as face anti-spoofing, driver drowsiness detection, and some health disorders. There have been few attempts to detect blinking in the wild scenario, while most of the work has been done under controlled conditions. Moreover, current learning approaches are designed to process sequences that contain only a single blink ignoring the case of the presence of multiple eye blinks. In this work, we propose a fast framework for eye blink detection and eye blink verification that can effectively extract multiple blinks from image sequences considering several challenges such as lighting changes, variety of poses, and change in appearance. The proposed framework employs fast landmarks detector to extract multiple facial key points including the ones that identify the eye regions. Then, an SVD-based method is proposed to extract the potential eye blinks in a moving time window that is updated with new images every second. Finally, the detected blink candidates are verified using a 2D Pyramidal Bottleneck Block Network (PBBN). We also propose an alternative approach that uses a sequence of frames instead of an image as input and employs a continuous 3D PBBN that follows most of the state-of-the-art approaches schemes. Experimental results show the better performance of the proposed approach compared to the state-of-the-art approaches.

The complexity of a scene in addition to the need for real-time processing are the main challenges that face any background/foreground separation approach for maritime environment. Recent studies onLow-rank and Sparse Separation(LSS) achieved good performance when compared to traditional background subtraction techniques in segregating the foreground from a complex background. However, the issue of maintaining this type of a separation via an updating mechanism is not well addressed by the majority of LSS approaches. The study presents a tensor based singular value decomposition approach for background/foreground separation. The approach is uniquely designed to deal with most challenges related to a maritime environment such as sea dynamics, boat wakes, variety of foreground objects, and camera jitter. Furthermore, the proposed approach operates incrementally via updating the separation components as opposed to reperforming the decomposition on the entire video sequence when a set of frames arrives. Additionally, a forgetting mechanism is employed in the proposed approach to efficiently handle challenges such asStationary Foreground Objects(SFOs) and ghost effects. The performance of the proposed method with several state-of-the-art LSS and Non-LSS techniques on videos with complex maritime scenarios are evaluated. The results exhibit better performance over most of the tested challenges and also demonstrate the capability of the proposed method to perform the separation in less computational time.

The background Initialization (BI) problem has attracted the attention of researchers in different image/video processing fields. Recently, a tensor-based technique called spatiotemporal slice-based singular value decomposition (SS-SVD) has been proposed for background initialization. SS-SVD applies the SVD on the tensor slices and estimates the background from low-rank information. Despite its efficiency in background initialization, the performance of SS-SVD requires further improvement in the case of complex sequences with challenges such as stationary foreground objects (SFOs), illumination changes, low frame-rate, and clutter. In this paper, a self-motion-assisted tensor completion method is proposed to overcome the limitations of SS-SVD in complex video sequences and enhance the visual appearance of the initialized background. With the proposed method, the motion information, extracted from the sparse portion of the tensor slices, is incorporated with the low-rank information of SS-SVD to eliminate existing artifacts in the initiated background. Efficient blending schemes between the low-rank (background) and sparse (foreground) information of the tensor slices is developed for scenarios such as SFO removal, lighting variation processing, low frame-rate processing, crowdedness estimation, and best frame selection. The performance of the proposed method on video sequences with complex scenarios is compared with the top-ranked state-of-the-art techniques in the field of background initialization. The results not only validate the improved performance over the majority of the tested challenges but also demonstrate the capability of the proposed method to initialize the background in less computational time.

In tasks such as abandoned luggage detection and stopped car detection, stationary foreground objects (SFOs) need to be detected and properly classified in real time. Different methods have been proposed to detect SFOs, but they are mainly focused on certain types of objects. In this paper, an incremental singular value decomposition-based method is presented to detect all types of SFOs such as abandoned objects and removed objects. The proposed method decomposes the video tensor spatiotemporally and divides it into background and foreground components. An appropriate analysis is applied to the foreground tensor to define a pixel time series of each stationary foreground category. Such analysis leads to the fact that SFOs can be detected easily owing to their continuous persistence in the decomposed foreground tensor. Furthermore, the unique structure of the pixel time series of each category allows identifying the category of the detected objects, whether they are abandoned or removed, and detecting the exact time of the start and end of each event. The results demonstrate that the proposed method achieves a superior performance in detecting SFOs at both object and pixel levels. In addition, the proposed method is computationally simple, and its complexity is lower compared to other approaches; hence, it can adequately satisfy real-time requirements.

Extracting the background from a video in the presence of various moving patterns is the focus of several background-initialization approaches. To model the scene background using rank-one matrices, this paper proposes a background-initialization technique that relies on the singular-value decomposition (SVD) of spatiotemporally extracted slices from the video tensor. The proposed method is referred to as spatiotemporal slice-based SVD (SS-SVD). To determine the SVD components that best model the background, a depth analysis of the computation of the left/right singular vectors and singular values is performed, and the relationship with tensor-tube fibers is determined. The analysis proves that a rank-1 matrix extracted from the first left and right singular vectors and singular value represents an efficient model of the scene background. The performance of the proposed SS-SVD method is evaluated using 93 complex video sequences of different challenges, and the method is compared with state-of-the-art tensor/matrix completion-based methods, statistical-based methods, search-based methods, and labeling-based methods. The results not only show better performance over most of the tested challenges, but also demonstrate the capability of the proposed technique to solve the background-initialization problem in a less computational time and with fewer frames.