Abstract Generative Adversarial Networks (GANs) have gained prominence in computer vision, with applications that extend to image fusion. Existing fusion methods often require extensive labeled data and task-specific training, limiting their generalizability. To address these limitations, this paper presents the Reconstruction-Guided Generative Adversarial Network (R2GAN), a generic GAN-based approach designed for generic image fusion, including visible-infrared, medical, and multi-focus image fusion. The proposed R2GAN architecture consists of a primary generator to improve fusion capabilities and auxiliary generators to ensure accurate reconstruction of source image features. To optimize the model, we propose a reconstruction-guided loss function to preserve the feature distribution of the source images and improve the consistency between the fused and source images. Additionally, we introduce a semantic segmentation-guided approach to generate a comprehensive and realistic Paired Multi-Focus image dataset (PMF) to train the R2GAN model. Experimental results in multiple fusion tasks demonstrate that R2GAN delivers superior performance, outperforming state-of-the-art image fusion methods. The R2GAN framework source code is available for access on GitHub at https://github.com/CHAHI24680/R2GAN .

Semantic segmentation is a well-studied topic and one of the most challenging tasks in computer vision applications, such as autonomous driving. Deep learning approaches based on convolutional neural networks (CNN) have demonstrated exceptional success on this task in recent years. Despite this success, existing approaches are plagued by higher-order inconsistencies between the ground truth images and the ones predicted by the segmentation model. This paper proposes a novel post-processing scheme based on adversarial learning to counter these inconsistencies. Such a scheme can be combined with a variety of existing CNN-based semantic segmentation networks to improve their segmentation performances. The proposed scheme is a Two-Stream Conditional Generative Adversarial Network (TScGAN), with one stream having initial semantic segmentation masks predicted by an existing CNN, while the other stream utilizes scene images to retain high-level information under a supervised residual network structure. In addition, TScGAN incorporates a novel dynamic weighting mechanism, which leads to significant and consistent gains in segmentation performance. Several comparative tests on public benchmark driving databases, including Cityscapes, Mapillary, and Berkeley DeepDrive100K, demonstrate the effectiveness of the proposed method when used with state-of-the-art CNN-based semantic segmentation models. Furthermore, the ablation experiment proved the structural rationality of our two-stream structure. The code for TScGAN could be found at https://github.com/epan-utbm/TScGAN-for-Improving-Semantic-Predictions

Haze is one of the atmospheric image degradations that causes severe distortions to outdoor images such as low contrast, color shift, and structure damage. Due to the unique physical characteristics of haze, the quality of hazy images is not accurately assessed using general-purpose image quality assessment (IQA) approaches. Therefore, several haze-aware IQA approaches have been proposed to provide more efficient dehazing quality evaluation. These approaches extract several haze-aware features to be either combined to form a single IQA metric or fed to a regression model that predicts the dehazing quality. However, these haze-relevant features are extracted using pixel intensity , in which luminance and structure information are inseparable, leading to less correlation between such features and the type of degradation they are supposed to represent. To address this issue, we propose a singular value decomposition (SVD) based IQA metric that can effectively separate the luminance component of an image from structure. This separation offers the ability to accurately evaluate the degradation at two different levels i.e. luminance and structure. The experimental results show that our proposed SVD-based dehazing quality evaluator (SDQE) outperforms the existing state-of-the-art non-reference IQA metrics in terms of accuracy and processing time.

Generative adversarial networks (GANs) represent a significant advance in the field of deep learning for image generation problems. With their ability to generate highly realistic and diverse images, GANs are quickly becoming the preferred technique for a wide range of applications. However, GANs come with several limitations and challenges, chief among which are their collapse mode and instability during training. In this paper, we propose a novel generator GAN model that incorporates the singular value decomposition (SVD) process into the decoder module. The SVD integration allows the generator to recognize the underlying structures in the feature space, resulting in more robust and effective image generation. This is achieved by adjusting the singular values during the training process, effectively optimizing the SVD-based generator to produce images that match the ground truth. Another advantage of SVD integration is its ability to perform discriminative spatial decomposition that clearly reflects the differences between the generated features and the target features. By including SVD in the generator model, the resulting loss values are higher and less prone to gradient vanishing than conventional GANs, such as Pix2Pix. Our proposed SVD-based generator model can be integrated into any auto-encoder architecture, making it as a generic and versatile solution for various image-generation tasks. The effectiveness of the proposed SVD-based GAN in image generation has been validated in three challenging benchmarks for image restoration and visible-to-infrared image translation. Extensive experiments demonstrate the significant quantitative and qualitative improvements achieved by our SVD-based GAN compared to baseline GAN architectures. The overall system outperforms the current state of the art in all benchmarks tested.

The ability to sense and understanding the driving environment is a key technology for ADAS and autonomous driving. Human drivers have to pay more visual attention to important or target elements and ignore unnecessary ones present in their field of sight. A model that computes this visual attention of targets in a specific driving environment is essential and useful in supporting autonomous driving, object-specific tracking & detection, driving training, car collision warning, traffic sign detection, etc. In this paper, we propose a new framework of visual attention that can predict important objects in the driving scene using a conditional generative adversarial network. A large scale Visual Attention Driving Database (VADD) of saliency heat-maps is built from existing driving datasets using a saliency mechanism. The proposed framework model takes its strength from these saliency heat-maps as conditioning label variables. The results show that the proposed approach makes us able to predict heat-maps of most important objects in a driving environment.

Detecting moving objects from a moving vehicle is a challenging problem and crucial for autonomous driving, especially in urban scenarios. The current literature has focused on this task as many approaches have been dedicated to moving object detection. These approaches consist of multistage pipelines, including semantic segmentation and optical flow estimation, and require multiple sources of information from active and passive sensors. However, they fail to accurately segment moving objects due to the large ego- camera motion, in addition to the high processing time. In this work, we propose a novel approach to moving object detection by processing information only from a camera. Our approach is based on integrating an encoder-decoder network (EDNet) with a semantic segmentation model (Mask R-CNN), where Mask R-CNN detects the objects of interest and the EDNet classifies their motion (moving/static) over two consecutive frames. We compare the results of our proposed model with existing MOD models on three SOTA benchmarks. We achieved SOTA performance in terms of visual quality and accuracy with competitive speed.