A single-modal infrared or visible image offers limited representation in scenes with lighting degradation or extreme weather. We propose a multi-modal fusion framework, named SDSFusion, for all-day and all-weather infrared and visible image fusion. SDSFusion exploits the commonality in image processing to achieve enhancement, fusion, and semantic task interaction in a unified framework guided by semantic awareness and multi-scale features and losses. To address the disparity between infrared and visible images in degraded scenes, we differentiate modal features in a unified fusion model. Unlike existing joint fusion methods, we propose an adversarial generative network that refines the reconstruction of low-light images by embedding fused features. It provides feature-level brightness supplementation and image reconstruction to refine brightness and contrast. Extensive experiments in degraded scenes confirm that our approach is superior to state-of-the-art approaches in visual quality and performance, demonstrating the effectiveness of interaction improvement. The code will be posted at: https://github.com/Liling-yang/SDSFusion.

Seeking feature correspondences among two or more images is an important problem in computer vision and image processing. The putative matches constructed by the similarity of feature descriptors are often contaminated by many false matches. Typically, the local neighborhood points of a true match point have a rank order, which will be maintained in the corresponding image, and we call it rank consistency. In this paper, we design a number of sorting plans to obtain the neighborhood rank lists by taking full advantage of the local neighborhood geometry structure. In order to measure the differences between rank lists, we adopt the statistically famous Kendall rank correlation coefficient and generalize its definition for matching problem. We design a neighborhood common element guidance strategy and a multi-neighborhood strategy to improve the universality and robustness of our method. Our method has linear complexity and it has superiority over state-of-the-art methods on several challenging data sets. It also performs well in image registration and loop-closure detection tasks. The source code of our method is publicly available athttps://github.com/MnYangs/mGKRCC.

In this letter, we propose a strategy for unmanned aerial vehicle (UAV) image stitching to generate natural-looking panoramas. Traditional methods using homography to perform alignment cannot account for images with parallax, so they require that the input images should be taken from the same viewpoint or the scene should be near the planar. However, remote sensing images obtained by UAVs usually do not satisfy such an ideal situation, and the stitching results always suffer from artifacts. To overcome these challenges and obtain natural-looking panoramas, a global alignment strategy is proposed to better align the input images. Combined with a shape-preserving warp, the stitching results can achieve better alignment accuracy while maintaining the shape. Meanwhile, locality preserving matching (LPM) is used to eliminate mismatches during feature detection and matching for accurate alignment. In addition, to make the stitching results more natural-looking, we also use multiband blending to eliminate artifacts that may exist in the results due to unmodeled effects. Experiments show that our stitching strategy can effectively improve alignment accuracy and obtain natural-looking results compared to other state-of-the-art methods.

Building footprint extraction plays an important role in many remote-sensing (RS) applications such as urban planning and disaster monitoring. Mainly, the exploitation of contextual information in a fixed receptive field is the focus of previous research, which makes it difficult to generically extract buildings that vary greatly in size and shape, especially when isolated large buildings are surrounded by dense small buildings. To improve this problem, we attempt to teach the network to adjust the receptive field and enhance useful feature information adaptively. In this article, we propose a novel adaptive screening feature network (ASF-Net), which can independently screen and enhance effective feature information from two aspects. On the one hand, we propose a deepened space up-sampling block to screen useful information and help establish boundaries. On the other hand, we propose an Adaptive Information Utilization Block (AIUB) to enlarge the receptive field of feature maps and refine the incomplete building footprint. As a result, the more accurate multiscale building footprint is inferred from the enhanced features. Experimental results on the popular aerial image segmentation datasets show that ASF-Net obtains competitive results [80.2% intersection over union (IoU) on the Inria aerial image labeling dataset and 74.2% IoU on the Massachusetts buildings dataset] in comparison with several state-of-the-art models. The TensorFlow implementation is available athttps://github.com/jyx0516/ASF-Net.

Feature matching is the foundation and key task of remote sensing image registration, which is to establish a reliable point corresponding relationship between the feature points of two images. In this article, a simple and effective local consensus method for rigid and nonrigid feature matching is proposed and applied to solve the problem of high outliers ratio caused by nonrigid transformation, nonlinear radiation difference, and speckle noise in the remote sensing image registration task. We first establish the putative feature correspondences according to the similarity between local descriptors and then use local consensus constraints (including neighborhood consensus and motion vector consensus) to remove outliers. The specific steps are given as follows. First, we use the neighborhood consensus constraint of feature points to carry out preliminary filtering to remove outliers with obvious errors and retain a large number of inliers, so as to obtain a clean reliable set. Then, the reliable set space is grided into several nonoverlapping cells, and the estimated motion vector is calculated for each cell. By taking the comprehensive deviation between the ordinary motion vectors and estimated motion vectors, we transform the matching problem into a mathematical optimization model and derive a closed-form solution with linear time and linear space complexities. In this way, our method can also significantly increase the speed of operation without sacrificing accuracy. A large number of feature matching experiments on remote sensing prove that our method is superior to existing methods and also has good results in the general scene.

In rural areas where disasters occur frequently, the calculation of building areas is crucial in property assessment. In the segmentation algorithm, Mask R-CNN can distinguish the adjacent objects and extract the outline of an object. Based on this observation, we propose a novel method to calculate the building areas based on Mask R-CNN and adopt the concept of transfer learning to train our model, which can achieve good results with a small number of drone aerial images as training samples. The proposed method involves three main steps: 1) pretraining using open-source satellite remote sensing images; 2) fine-tuning with a small number of drone aerial images; and 3) testing with new images and area calculation based on the number of building pixels. The experiments show that the proposed method can achieve good results in terms of F1 score and intersection over union.

This article proposes a strategy for drone image stitching using local mesh-based bundle adjustment and shape-preserving transform, which aims to effectively stitch multiple overlapping drone images into a natural panoramic image. Existing traditional methods using a simple homography cannot handle the situation that the input drone images have parallax effect, and the image mosaic result always suffers from artifacts. In order to achieve natural-looking stitching results without the above limitation, we divide the proposed method into the following steps. Starting from initial feature sets obtained by off-the-shelf feature extraction methods, we incorporate the parallax errors into an energy minimum framework and construct a robust alignment energy. This energy can be minimized efficiently based on local bundle adjustment and robust$3\sigma $principle, which could eliminate parallax effects and achieve accurate alignment. Then the seamless panoramic image is obtained by warping the target image and the source images onto the mesh plane directly. An image patch can be transformed by projective transformation (e.g., homography), which provides good alignment but may cause distortions. Consequently, combined with mesh-based shape-preserving transform, our proposed strategy can improve the naturalness of the results flexibly. Experiments show that our stitching strategy can eliminate parallax effects more effectively and achieve natural-looking results compared to other state-of-the-art methods.

The goal of unmanned aerial vehicle (UAV) image mosaicing is to create natural-looking mosaics free of artifacts due to the parallax of the image and relative camera motion. UAV remote sensing is a low-altitude technology and the UAV imaged scene is not effectively planar, yielding parallax on images. In this paper, we apply local homography to match UAV images, which can reduce misalignment artifacts or “ghosting” in the results compared with 2D projective transforms or global homography. In addition, when an object in three dimensions is mapped to an image plane, different surfaces have different projections. These projections vary with the viewpoint in a sequence of UAV images, which still causes artifacts near some tall buildings if we only use local homography. We propose a novel stitching method based multi-region local projection deformation, that divides the overlapping regions of input images into several regions, then meshes image to calculate local projections by partitioned regions. Specifically, we use a strategy where multiple regions have different weights for calculating local projections, which can significantly reduce ghosting due to these projections vary with the viewpoint and parallax. The benefits of the proposed approach are demonstrated using a variety of challenging cases.

Focusing on the characteristics of remote sensing images, this study proposes a new algorithm for feature matching of remote sensing images to eliminate mismatch. The algorithm utilizes feature descriptors, such as scale-invariant feature transform, for rough correspondence and the thin-plate spline for non-rigid transformation. Under the Bayesian framework, correspondence and transformation are alternately optimized by the expectation-maximization algorithm to automatically eliminate mismatched points. We also introduce a local geometrical constraint to maintain the internal structure of adjacent feature points. We apply this method to a large number of remote sensing images, and the experimental results reveal the method's superiority over the state-of-the-art.