Heuristic information for consensus set sampling is essential for correspondence-based point cloud registration, but existing approaches typically rely on supervised learning or expert-driven parameter tuning. In this work, we propose HeMoRa, a new unsupervised framework that trains a Heuristic information Generator (HeGen) to estimate sampling probabilities for correspondences using a Multi-order Reward Aggregator (MoRa) loss. The core of MoRa is to train HeGen through extensive trials and feedback, enabling unsupervised learning. While this process can be implemented using policy optimization, directly applying the policy gradient to optimize HeGen presents challenges such as sensitivity to noise and low reward efficiency. To address these issues, we propose a Maximal Reward Propagation (MRP) mechanism that enhances the training process by prioritizing noise-free signals and improving reward utilization. Experimental results show that equipped with HeMoRa, the consensus set sampler achieves improvements in both robustness and accuracy. For example, on the 3DMatch dataset with FCGF feature, the registration recall of our unsupervised methods (Ours+SM and Ours+SC2) even outperforms the state-of-the-art supervised method VBreg. Our code is available at HeMoRa.

Robust estimation is essential in correspondence-based Point Cloud Registration (PCR). Existing methods using maximal clique search in compatibility graphs achieve high recall but suffer from exponential time complexity, limiting their use in time-sensitive applications. To address this challenge, we propose a fast and robust estimator, TurboReg, built upon a novel lightweight clique, TurboClique, and a highly parallelizable Pivot-Guided Search (PGS) algorithm. First, we define the TurboClique as a 3-clique within a highly-constrained compatibility graph. The lightweight nature of the 3-clique allows for efficient parallel searching, and the highly-constrained compatibility graph ensures robust spatial consistency for stable transformation estimation. Next, PGS selects matching pairs with high SC$^2$ scores as pivots, effectively guiding the search toward TurboCliques with higher inlier ratios. Moreover, the PGS algorithm has linear time complexity and is significantly more efficient than the maximal clique search with exponential time complexity. Extensive experiments show that TurboReg achieves state-of-the-art performance across multiple real-world datasets, with substantial speed improvements. For example, on the 3DMatch+FCGF dataset, TurboReg (1K) operates $208.22\times$ faster than 3DMAC while also achieving higher recall. Our code is accessible at \href{https://github.com/Laka-3DV/TurboReg}{\texttt{TurboReg}}.

Accurate automatic interpretation of crops is crucial for agricultural monitoring and food security assessment. In recent years, there has been a rise in platforms generating multimodal and multitemporal remote sensing images at an unprecedented speed. These images provide rich temporal, spatial, and spectral information, enabling more comprehensive land cover classification. Therefore, single-modal crop classification can benefit from complementary modalities. However, given the distinct characteristics of different modality sensors, enhancing the performance of deep networks by integrating diverse modality data remains a significant challenge. Unlike previous methods, this article proposes a two-stage fusion network named CMINet for crop classification from satellite image time series (SITS). Specifically, we adopt a decoupled framework to encode the spatial and temporal features of SITS. At each level of the spatial encoder, we design a cross-modal transfer module (CMTM) to complement bimodality branch features by transferring knowledge from one modality to rectify the features of another modality. After rectified feature pairs are encoded by a temporal encoder, we develop a cross-attention fusion module (CAFM) to conduct adequate context exchange before merging. The seamless combination of these novel designs forms a robust multimodal representation, outperforming the state-of-the-art methods on two public multimodal crop classification datasets. Compared to existing methods, our CMINet improves at least 1.1% OA, 1.7% mF1, and 2.0% mIoU on the PASITS-R dataset and 2.6% OA, 1.1% mF1, and 2.8% mIoU on the South Sudan dataset.

Due to the complementary nature of optical and SAR images, their alignment is of increasing interest. However, due to the significant radiometric differences between them, precise matching becomes a very challenging problem. Although current advanced structural features and deep learning-based methods have proposed feasible solutions, there is still much potential for improvement. In this paper, we propose a hybrid matching method using attention-enhanced structural features (namely AESF), which combines the advantages of both handcrafted-based and learning-based methods to improve the accuracy of optical and SAR image matching. It mainly consists of two modules: a novel effective multi-branch global attention (MBGA) module and a joint multi-cropping image matching loss function (MCTM) module. The MBGA module is designed to focus on shared information in structural feature descriptors of heterogeneous images across space and channel dimensions, significantly improving the expressive capacity of the classical structural features and generating more refined and robust image features. The MCTM module is constructed to fully exploit the association between global and local information of the input image, which can optimize the triple loss discriminator to discriminate positive and negative samples. To validate the effectiveness of the proposed method, it is compared with five state-of-the-art matching methods by using various optical and SAR datasets. The experimental results show that the matching accuracy at the 1-pixel threshold is improved by about 1.8%-8.7% compared with the most advanced deep learning method (OSMNet) and 6.5%-23% compared with the handcrafted description method (CFOG).

Weakly supervised building extraction methods, utilizing image-level labels, offer a cost-effective solution by significantly reducing the need for pixel-level annotation in high-resolution (HR) remote sensing (RS) images. These methods often focus on class activation map (CAM) optimization based on features extracted from individual images, missing out on the benefits of associating building features from multiple RS images (i.e., n images) to improve CAMs. This limitation leaves room for improvement in both CAM optimization and pseudo-mask generation. To address this, we propose the building-aware clustering and activation refinement network (BAC-AR-Net), a novel weakly supervised network to enhance weakly supervised building extraction performance. The building-aware clustering (BAC) module aggregates and clusters feature maps from multiple building samples to obtain common features of buildings. The common features are subsequently used to extract regions with similar building semantics, thereby enhancing the accuracy and completeness of building coverage in CAMs. Additionally, the activation refinement module is designed to generate pseudo-masks with clear boundaries and an effective separation of buildings and background. Experiments were conducted on the ISPRS Potsdam and Vaihingen datasets as well as a self-built building dataset to verify the effectiveness of our proposed method. The results show the proposed method outperforms both the weakly supervised semantic segmentation and weakly supervised building extraction methods that use image-level labels, achieving IoU accuracies of 0.8556, 0.8163, and 0.7797 on the respective datasets. This study introduces a novel weakly supervised learning framework to the RS application, with a particular focus on building extraction and semantic segmentation tasks.

Maximum correntropy criterion (MCC) is a robust and powerful technique to handle heavy-tailed nonGaussian noise, which has many applications in the fields of vision, signal processing, machine learning, etc. In this article, we introduce several contributions to the MCC and propose an augmented MCC (AMCC), which raises the robustness of classic MCC variants for robust fitting to an unprecedented level. Our first contribution is to present an accurate bandwidth estimation algorithm based on the probability density function (PDF) matching, which solves the instability problem of the Silverman's rule. Our second contribution is to introduce the idea of graduated nonconvexity (GNC) and a worst-rejection strategy into MCC, which compensates for the sensitivity of MCC to high outlier ratios. Our third contribution is to provide a definition of local distribution measure to evaluate the quality of inliers, which makes the MCC no longer limited to random outliers but is generally suitable for both random and clustered outliers. Our fourth contribution is to show the generalizability of the proposed AMCC by providing eight application examples in geometry perception and performing comprehensive evaluations on five of them. Our experiments demonstrate that 1) AMCC is empirically robust to 80%$-$90% of random outliers across applications, which is much better than Cauchy M-estimation, MCC, and GNC-GM; 2) AMCC achieves excellent performance in clustered outliers, whose success rate is 60%$-$70% percentage points higher than the second-ranked method at 80% of outliers; 3) AMCC can run in real-time, which is 10$-$100 times faster than RANSAC-type methods in low-dimensional estimation problems with high outlier ratios. This gap will increase exponentially with the model dimension.

With the development of 3D matching technology, correspondence-based point cloud registration gains more attention. Unfortunately, 3D keypoint techniques inevitably produce a large number of outliers, i.e., outlier rate is often larger than 95%. Guaranteed outlier removal (GORE) Bustos and Chin has shown very good robustness to extreme outliers. However, the high computational cost (exponential in the worst case) largely limits its usages in practice. In this paper, we propose the first$O(N^{2})$time GORE method, called quadratic-time GORE (QGORE), which preserves the globally optimal solution while largely increases the efficiency. QGORE leverages a simple but effective voting idea via geometric consistency for upper bound estimation, which achieves almost the same tightness as the one in GORE. We also present a one-point RANSAC by exploring “rotation correspondence” for lower bound estimation, which largely reduces the number of iterations of traditional 3-point RANSAC. Further, we propose al$_{p}$p-like adaptive estimator for optimization. Extensive experiments show that QGORE achieves the same robustness and optimality as GORE while being 1$\sim$2 orders faster. The source code will be made publicly available.

Point cloud registration (PCR) is an important and fundamental problem in 3D computer vision, whose goal is to seek an optimal rigid model to register a point cloud pair. Correspondence-based PCR techniques do not require initial guesses and gain more attentions. However, 3D keypoint techniques are much more difficult than their 2D counterparts, which results in extremely high outlier rates. Current robust techniques suffer from very high computational cost. In this paper, we propose a polynomial time ($O(N^2)$) outlier removal method. Its basic idea is to reduce the input set into a smaller one with a lower outlier rate based on bound principle. To seek tight lower and upper bounds, we originally define two concepts, i.e., correspondence matrix (CM) and augmented correspondence matrix (ACM). We propose a cost function to minimize the determinant of CM or ACM, where the cost of CM rises to a tight lower bound and the cost of ACM leads to a tight upper bound. Then, we propose a scale-adaptive Cauchy estimator (SA-Cauchy) for further optimization. Extensive experiments on simulated and real PCR datasets demonstrate that the proposed method is robust at outlier rates above 99% and 1~2 orders faster than its competitors.

Severe nonlinear radiation distortion (NRD) is the bottleneck problem of multimodal image matching. Although many efforts have been made in the past few years, such as the radiation-variation insensitive feature transform (RIFT) and the histogram of orientated phase congruency (HOPC), almost all these methods are based on frequency-domain information that suffers from high computational overhead and memory footprint. In this article, we propose a simple but very effective multimodal feature matching algorithm in the spatial domain, called locally normalized image feature transform (LNIFT). We first propose a local normalization filter to convert original images into normalized images for feature detection and description, which largely reduces the NRD between multimodal images. We demonstrate that normalized matching pairs have a much larger correlation coefficient than the original ones. We then detect oriented FAST and rotated brief (ORB) keypoints on the normalized images and use an adaptive nonmaximal suppression (ANMS) strategy to improve the distribution of keypoints. We also describe keypoints on the normalized images based on a histogram of oriented gradient (HOG), such as a descriptor. Our LNIFT achieves rotation invariance the same as ORB without any additional computational overhead. Thus, LNIFT can be performed in near real-time on images with 1024$\times 1024$pixels (only costs 0.32 s with 2500 keypoints). Four multimodal image datasets with a total of 4000 matching pairs are used for comprehensive evaluations, including synthetic aperture radar (SAR)–optical, infrared–optical, and depth–optical datasets. Experimental results show that LNIFT is far superior to RIFT in terms of efficiency (0.49 s versus 47.8 s on a$1024 \times 1024$image), success rate (99.9% versus 79.85%), and number of correct matches (309 versus 119). The source code and datasets will be publicly available athttps://ljy-rs.github.io/web.

Traditional image feature matching methods cannot obtain satisfactory results for multi-modal remote sensing images (MRSIs) in most cases because different imaging mechanisms bring significant nonlinear radiation distortion differences (NRD) and complicated geometric distortion. The key to MRSI matching is trying to weakening or eliminating the NRD and extract more edge features. This paper introduces a new robust MRSI matching method based on co-occurrence filter (CoF) space matching (CoFSM). Our algorithm has three steps: (1) a new co-occurrence scale space based on CoF is constructed, and the feature points in the new scale space are extracted by the optimized image gradient; (2) the gradient location and orientation histogram algorithm is used to construct a 152-dimensional log-polar descriptor, which makes the multi-modal image description more robust; and (3) a position-optimized Euclidean distance function is established, which is used to calculate the displacement error of the feature points in the horizontal and vertical directions to optimize the matching distance function. The optimization results then are rematched, and the outliers are eliminated using a fast sample consensus algorithm. We performed comparison experiments on our CoFSM method with the scale-invariant feature transform (SIFT), upright-SIFT, PSO-SIFT, and radiation-variation insensitive feature transform (RIFT) methods using a multi-modal image dataset. The algorithms of each method were comprehensively evaluated both qualitatively and quantitatively. Our experimental results show that our proposed CoFSM method can obtain satisfactory results both in the number of corresponding points and the accuracy of its root mean square error. The average number of obtained matches is namely 489.52 of CoFSM, and 412.52 of RIFT. As mentioned earlier, the matching effect of the proposed method was significantly greater than the three state-of-art methods. Our proposed CoFSM method achieved good effectiveness and robustness. Executable programs of CoFSM and MRSI datasets are published: https://skyearth.org/publication/project/CoFSM/.

Point cloud registration (PCR) is an important task in photogrammetry and remote sensing, whose goal is to seek a seven-parameter similarity transformation to register a pair of point clouds. Traditional iterative closest point (ICP) variants highly rely on the initial parameters, and most of them cannot deal with cross-source (multisource) point clouds with scale changes. In this article, we propose a flexible correspondence-based PCR method, which is initial-guess free, fast, and robust. We first decompose the full seven-parameter registration problem into three subproblems, i.e., scale, rotation, and translation estimations, based on line vectors. Then, we propose a one-point random sample consensus (RANSAC) algorithm to estimate the scale and translation parameters. For the rotation estimation, we introduce a graduated optimization strategy into Tukey’s biweight function and propose a scale-annealing biweight estimator. We evaluate the proposed method on both same-source and cross-source data. Results show that the proposed method is robust against over 99% outliers and is one to two orders of magnitude faster than its competitors. The source code of our method will be made public.

Deep learning has achieved great success in recent years, among which the convolutional neural network (CNN) method is outstanding in image segmentation and image recognition. It is also widely used in satellite imagery road extraction and, generally, can obtain accurate and extraction results. However, at present, the extraction of roads based on CNN still requires a lot of manual preparation work, and a large number of samples can be marked to achieve extraction, which has to take long drawing cycle and high production cost. In this article, a new CNN sample set production method is proposed, which uses the GPS trajectories of floating car as training set (GPSTasST), for the multilevel urban roads extraction from high-resolution remote sensing imagery. This method rasterizes the GPS trajectories of floating car into a raster map and uses the processed raster map to label the satellite image to obtain a road extraction sample set. CNN can extract roads from remote sensing imagery by learning the training set. The results show that the method achieves a harmonic mean of precision and recall higher than road extraction method from single data source while eliminating the manual labeling work, which shows the effectiveness of this work.

Robust estimation, which aims to recover the geometric transformation from outlier contaminated observations, is essential for many remote sensing and photogrammetry applications. This article presents a novel robust geometric model estimation method based on scaled Welsch q-norm (lq-norm, 0qLS) problem and a weighted least-squares (WLS) problem] by using alternating direction method of multipliers (ADMM) method. For the WLS problem, we introduce a coarse-to-fine strategy into the iterative reweighted least-squares (IRLS) method. We change the weight function by decreasing its scale parameter. This strategy can largely avoid that the solver gets stuck in local minimums. We adapt the proposed cost into classical remote sensing tasks and develop new robust feature matching (RFM), robust exterior orientation (REO), and robust absolute orientation (RAO) algorithms. Both synthetic and real experiments demonstrate that the proposed method significantly outperforms the other compared state-of-the-art methods. Our method is still robust even if the outlier rate is up to 90%.

Traditional feature matching methods, such as scale-invariant feature transform (SIFT), usually use image intensity or gradient information to detect and describe feature points; however, both intensity and gradient are sensitive to nonlinear radiation distortions (NRD). To solve this problem, this paper proposes a novel feature matching algorithm that is robust to large NRD. The proposed method is called radiation-variation insensitive feature transform (RIFT). There are three main contributions in RIFT. First, RIFT uses phase congruency (PC) instead of image intensity for feature point detection. RIFT considers both the number and repeatability of feature points and detects both corner points and edge points on the PC map. Second, RIFT originally proposes a maximum index map (MIM) for feature description. The MIM is constructed from the log-Gabor convolution sequence and is much more robust to NRD than traditional gradient map. Thus, RIFT not only largely improves the stability of feature detection but also overcomes the limitation of gradient information for feature description. Third, RIFT analyses the inherent influence of rotations on the values of the MIM and realises rotation invariance. We use six different types of multi-modal image datasets to evaluate RIFT, including optical-optical, infrared-optical, synthetic aperture radar (SAR)-optical, depth-optical, map-optical, and day-night datasets. Experimental results show that RIFT is superior to SIFT and SAR-SIFT on multi-modal images. To the best of our knowledge, RIFT is the first feature matching algorithm that can achieve good performance on all the abovementioned types of multi-modal images. The source code of RIFT and the multi-modal image datasets are publicly available1.

Haze and cloud seriously degrade the quality of optical remote sensing images, which largely decrease their interpretability and intelligibility. In this letter, we propose a two-stage haze and thin cloud removal method based on homomorphic filtering (HF) and sphere model improved dark channel prior (DCP). Compared with current dehazing methods, the most advantage of the proposed method is that our method can deal with uneven haze, thick haze, and thin cloud. We observe that haze and cloud are highly related to the illumination component and mainly located in the low frequency of an image. Thus, we adapt HF to enhance the haze image, which makes the distribution of haze more even. In the second stage, we analyze the drawback of DCP, i.e., the transmission estimated by DCP is very sensitive to noise. To draw this issue, we propose a novel sphere model to estimate a more accurate transmission map. The sphere model improved DCP is more suitable for thick haze images than the traditional DCP. Extensive experimental results show that the proposed method significantly outperforms the compared state-of-the-art methods. The source code and data sets used in the letter are made public.

Illumination and color consistency are very important for optical remote-sensing image mosaicking. In this letter, we propose a simple but effective technique that simultaneously performs image illumination and color correction for multiview images. In this framework, we first present an uneven illumination removal algorithm based on bright channel prior, which guarantees the illumination consistency inside a single image. We then adapt a pairwise color-correction method to coarsely align the color tone between source and reference images. In this stage, we give a new single-image quality metric which combines brightness deviation, color cast, and entropy together for automatic reference-image selection. Finally, we perform a least-squares adjustment (LSA) procedure to obtain optimal illumination and color consistency among multiview images. In detail, we first perform a pairwise image matching by using SIFT algorithm; once sparse local patch correspondences obtained, the illumination and color relationship between images can be established based on a global gamma correction model; the illumination and color errors can then be minimized by LSA. Extensive experiments on both challenging synthetic and real optical remote-sensing image data sets show that it significantly outperforms the compared state-of-the-art approaches. All the source code and data sets used in this letter are made public.

This letter proposes a robust feature matching algorithm for remote sensing images based on lq-estimator. We start with a set of initial matches provided by a feature matching method such as scale-invariant feature transform and then focus on global transformation estimation from contaminated observations and outliers elimination as well. We use an affine model to describe the global transformation and minimize a new cost function based on lq-norm. We apply an augmented Lagrangian function and an alternating direction method of multipliers to solve such a nonconvex and nonsmooth optimization problem. Extensive experiments on real remote sensing data demonstrate that the proposed method is effective, efficient, and robust. Our method outperforms state-of-the-art methods and can easily handle situations with up to 90% outliers. In addition, the proposed method is much faster than RANSAC.