Existing cloud detection methods often rely on deep neural networks, leading to excessive computational overhead. To address this, we propose a shallow convolutional neural network (CNN)-Transformer hybrid architecture that limits the maximum downsampling rate to 8×. This design preserves local details while effectively capturing global context through a lightweight Transformer branch. To enhance adaptability across diverse cloud scenes, we introduce two novel statistics-driven modules: statistics-adaptive convolution (SAC) and statistical mixing augmentation (SMA). SAC dynamically generates convolutional kernels based on input feature statistics, enabling adaptive feature extraction for varying cloud patterns. SMA improves model generalization by interpolating channel-wise statistics across training samples, increasing feature diversity. Experiments on four datasets show that the proposed method achieves state-of-the-art performance with 732K parameters and 1G multiply-accumulate operations. Code will be available at https://weix-liu.github.io/ for further research.

Domain adaptive object detection methods for remote sensing images typically rely on large-scale target domain data and multi-epoch offline adaption training. However, the wide variation in remote sensing conditions makes it difficult to gather sufficient data for every potential target domain, especially for unexpected domains. To address this challenge, we propose Learnable Online Style Adaptation (LOSA), a method that enables the source model to adapt effectively to new target domain styles with low test-time latency. Specifically, LOSA captures target domain styles using shallow feature channel statistics and predicts style shifts based on channel dependencies to recalibrate target features. Through a coarse-to-fine alignment loss between online target features and pre-computed source domain statistics, LOSA autonomously learns domain-specific style adaptation strategies. By adopting a dynamically optimized high learning rate, LOSA only requires a small number of samples for test-time training, making it suitable for real-time applications. Experimental results across various scenarios, including normal-to-corrupted, cross-band, and sim-to-real adaptation, demonstrate that the proposed method significantly improves cross-domain object detection performance. Moreover, our method can be well generalized to cross-domain image classification tasks.

In the field of simultaneous localization and mapping (SLAM), map-based localization has been widely used in autonomous driving, particularly for all-speed and all-road adaptive cruise, automatic parking, and other high-level functions. As a result, LiDAR sensors are frequently used in visual-based SLAM to improve the overall accuracy of ego-motion estimation and environment reconstruction. In this article, a novel tightly coupled monocular hybrid visual LiDAR SLAM (HVL-SLAM), which utilizes both visual and LiDAR measurements in tracking and mapping. First, the proposed method reduces the 3-D uncertainty of features by employing object segmentation and Delaunay triangulation. The motion between adjacent frames is then estimated using a hybrid tracking module that minimizes photometric and reprojection error. Finally, a joint optimization method for refining the pose is proposed, which incorporates visual and LiDAR measurements into optimization with dynamic weights, resulting in higher positioning accuracy and robustness. The experiments on the public KITTI odometry benchmark and real-world outdoor datasets demonstrate that HVL-SLAM outperforms state-of-the-art approaches in terms of pose estimation and mapping performance. The code is released to the community. Code available athttps://github.com/kinggreat24/hvl_slam.

Vehicle detection methods based on deep learning have achieved remarkable results on remote sensing images. However, the performance of the detector degrades when the test images are distinct from the training images. Domain adaptive vehicle detection is a promising approach to bridging the domain gap. Existing methods usually adopt fully shared networks, but ignore the problem that features from different domains may be incompatible within a single model. In this letter, we present a novel domain adaptive vehicle detection method based on patch-wise domain-specific channel recalibration (PDSCR). The PDSCR module routes the feature to the corresponding network branch and extracts the channel dependence using separate parameters. In this way, our method can explicitly capture domain-specific information for each domain. Furthermore, we propose a dynamic weighted prototype alignment (DWPA) to avoid the negative effects of false pseudo-labels, especially in the early stage of training. Experimental results of adaptation from our synthetic dataset to three real vehicle detection datasets demonstrate the effectiveness of our method. Code and our synthetic data will be available at https://weix-liu.github.io/.

Optical and synthetic aperture radar (SAR) image keypoint detection is an important foundation for multimodal remote sensing image matching. The influence of nonlinear radiometric differences and geometric deformation between optical and SAR images leads to low repeatability of existing keypoint detection methods. To address the problem that existing keypoint detection methods cannot provide the required homonymous points for heterogenous image matching, we propose a keypoint detection method (SKD-Net) for optical and SAR images, and improve it in terms of both network structure and network optimization. First, we propose a multilevel attention Siamese network, which is composed of multiple convolutional modules and transformer modules with shared weights to extract common features at different levels for keypoint detection. We introduce a transformer module in the keypoint detection pipeline and fuse shallow and deep features to obtain more spatial and rich semantic information to facilitate heterogeneous image keypoint detection. Then, to ensure that the detected keypoints have more homonymous points and localization accuracy, we propose a position consistent loss. Unlike previous loss functions, our designed position-consistent loss function takes the differences between heterogeneous image score maps into account, and it autonomously selects the optimized correct point pairs to enable the network to perform correct learning. Finally, extensive experiments show that our detection method outperforms the current state-of-the-art keypoint detection methods in terms of repeatability, localization accuracy, and matching performance. Our source code is available at https://github.com/zhangschen/ SKD-Net.

Three-dimensional (3-D) measurement of metal surfaces is one of the fundamental tasks for product life-cycle management of complex equipment, which is meaningful but challenging due to its optical characteristics of high reflectivity. To reliably reconstruct 3-D metal surfaces, the commonly used techniques heavily rely on multiple exposures for optimal fusion but do not fit to high-efficiency monitoring. To alleviate this reliance, we propose a novel single-exposure method called deep sinusoidal prior (DSP) for damaged phase recovery of highly reflective surfaces. Specifically, the sinusoidal hypothesis is instilled into an untrained deep neural network (DNN) as two-stream information, in order to bypass the problem of brightness enhancement. Utilizing elaborately designed loss functions, this approach enables us to restore the accurate phase encoding by fitting the DNN to two-stream sinusoidal priors. Experimental results demonstrate that the proposed DSP method has superior performances on damaged phase recovery requiring no training samples. For instance, measuring a standard workpiece, absolute errors of the DSP method have been decreased substantially (81.69% and 59.49%) compared with the direct measurement and achieved similar accuracy (0.0754 versus 0.0744 mm) compared with the reference. Most strikingly, the proposed method, for the first time, demonstrates a new perspective of recovering the reliable phase from a degraded one itself, contributing to the superior generalization capability insensitive to fringe frequencies, imaging settings, and variant scenes.

Deep learning approaches require enough training samples to perform well, but it is a challenge to collect enough real training data and label them manually. In this letter, we propose a practical framework for automatically generating content-rich synthetic images with ground-truth annotations. By rendering 3-D CAD models, we generate two synthetic aircraft image data sets with wide distribution (Syn N and Syn U). For improving the quality of synthetic images, we propose a multiscale attention module which enhances the Cycle-Consistent Adversarial Network (CycleGAN) in spatial and channel dimensions. Then, we compare the synthetic images before and after translation qualitatively and quantitatively. Experiments on Northwestern Polytechnical University (NWPU) very high resolution (VHR)-10, University of Chinese Academy of Sciences, orientation robust object detection in aerial images (UCAS-AOD), and benchmark for object DetectIon in Optical Remote sensing images (DIOR) data sets demonstrate that synthetic data augmentation can improve the performance of aircraft detection in remote sensing images, especially when real data are insufficient. Synthetic data are available at:https://weix-liu.github.io/.

Thermal infrared (TIR) remote-sensing imagery can allow objects to be imaged clearly at night through the long-wave infrared, so that the fusion of thermal infrared and visible (VIS) imagery is a way to improve the remote-sensing interpretation ability. However, due to the large radiation difference between the two kinds of images, it is very difficult to match them. One of the most important issues is the lack of comprehensive consideration of the modality-specific information and modality-shared information, which makes it difficult for the existing methods to obtain a modality-invariant feature representation. In this article, a cross-modality image matching network, which we refer to as CMM-Net, is proposed to realize thermal infrared and visible image matching by learning a modality-invariant feature representation. First, in order to extract the modality-specific features of the imagery, the framework constructs a shallow two-branch network to make full use of the modality-specific information, without sharing parameters. Second, in order to extract the high-level semantic information between the different modalities, modality-shared layers are embedded into the deep layers of the network. In addition, three novel loss functions are designed and combined to learn the modality-invariant feature representation, that is, the discriminative loss of the non-corresponding features in the same modality, the cross-modality loss of the corresponding features between different modalities, and the cross-modality triplet (CMT) loss. The multimodal matching experiments conducted with ground- and airborne-based thermal infrared images and visible images showed that the proposed method outperforms the existing image matching methods by about 2% and 6% for the ground and airborne images, respectively.

The preference analysis approach is widely used in solving multimodel fitting problems, but it does not combine the potential spatial correlation in the data. In this paper, we propose a novel method for introducing spatial information to optimize the results of preference analysis through energy minimization. The method consists of two steps: 1) T-linkage is performed based on localized window sampling to obtain the initial clustering results; and 2) the alternate sampling and clustering framework is applied until the energy is stable. The main contribution is that we introduce the spatial information of the data points in the preference analysis by minimizing an energy function, in order to make the inlier preferences more distinguishable and improve the clustering. The experimental results confirm that the proposed method compares favorably with the current state of the art.

Due to the significant nonlinear intensity changes of multispectral images, automatic image feature point matching is a challenging task. This letter addresses the problem and proposes a novel descriptor combining the structure and texture information to solve the nonlinear intensity variations of multispectral images. We first propose directional maps, i.e., the directional response maps (DMs) and the directional response binary maps (DBMs), which can capture the common structure and texture properties of multispectral images, respectively. We then use the spatial pooling pattern of the histogram of oriented gradients to separately describe the local region of each point of interest based on the DMs and DBMs. In order to speed up the calculation, we apply Gaussian filters to the DMs and average filters to the DBMs to construct the per-pixel histogram bins. Finally, we conjoin the normalized feature vectors corresponding to the structure description and texture description of each point of interest to obtain the histograms of directional maps (HoDMs). The proposed HoDM descriptor was evaluated using three data sets composed of images obtained in both visible light and infrared spectra. The experimental results confirm that the proposed HoDM descriptor is robust to the nonlinear intensity changes of multispectral images and has a superior matching performance as well as a much higher computational efficiency.

In this letter, permutation preference is used to represent the data points for the linkage clustering to segment the tracking points belonging to different motions. In order to exclude the impact of outliers, an alternate sampling and clustering strategy is performed, that iteratively alternates between sampling the hypotheses within the clusters and clustering the points with the permutation preference. As a result, points with similar permutation preferences are sampled as the hypotheses, and outliers are effectively excluded, thus, making the preferences more distinguishable and improving the clustering. The iterative interaction between sampling and clustering results in a good convergent result. The proposed method obtains robust segmentation results with the Hopkins 155 dataset, which are better than the results obtained by the state-of-the-art methods.

Feature extraction is at the core of satellite scene classification task. In this paper, we propose a fast binary coding (FBC) method to effectively generate the global discriminative feature representation of image scenes. Equipped with unsupervised feature learning technique, we first learn a set of optimal “filters” from large quantities of randomly sampled image patches, and then we obtain feature maps by convolving image scene with the learned filter bank. After binarizing the feature maps, a simple skillful conversion of binary-valued feature map to integer-valued feature map is performed. The final statistical histograms, which are considered as the global feature representations of scenes, are computed on the integer-valued feature map similar to the conventional BOW model. Experiments on two datasets demonstrate that the proposed FBC achieve satisfying classification performance as well as has much faster computational speed compared with traditional scene classification methods.

In this paper, we design a Content-based map retrieval (CBM-R) system for land cover/land use (LCLU). The main contributions of our work are listed below. First, our system could allow the user to select region of interest as reference scene with variable shape and size. Whereas in traditional CBIR/CBMR systems, the region of interest is usually with fixed size of which is equal to the size of the analyse window for extracting features. In addition, the user could acquire various retrieval results by specifying corresponding parameters. In the end, by using combination of the feature library, the user could acquire the retrieval result faster.

Morphological profiles are widely used for the classification of high-resolution remote sensing images. Images are degraded by morphological operators with different sizes. Profiles, such as the variation of intensities of the pixels, are extracted during the degradation. In this paper, two issues with morphological profiles are addressed. First, we propose using the topographic map of the image, which is autodual and does not require any structural element, for the degradation of the image. Second, we propose extracting other profiles that are more robust than the intensity variations. Classification experiments are carried out on two IKONOS remote sensing images with a 1-m resolution and one Pléiades 1-A image with a 2-m resolution. The efficiencies of the new profiles are validated by the experimental results.

In this letter, an eigenvalue-based empirical method is proposed in order to estimate the number of endmembers in hyperspectral data. This method is based on the distribution of the differences of the eigenvalues from the correlation and the covariance matrices, respectively. The eigenvalues corresponding to the noise are identical in the covariance and the correlation matrices, while the eigenvalues corresponding to the signal (the endmembers) are larger in the correlation matrix than in the covariance matrix. The proposed method is totally parameter free and very fast. It is validated by experiments carried on both synthetic and real data sets.

Pansharpening may be defined as the process of synthesizing multispectral images at a higher spatial resolution. A wide range of pansharpening methods are available, each producing images with different characteristics. To compare the performances and characteristics of different methods, a contest was held in 2006 by the IEEE Data Fusion Technical Committee. In this contest, À trous wavelet transform-based pansharpening (AWLP) and Laplacian pyramid-based context adaptive (CBD) pansharpening methods were declared as joint winners. While assessing the quantitative quality of the pansharpened images, we observed that the two methods outperform each other depending upon the local content of the scene. Hence, it is interesting to design a method taking advantage of both methods by locally selecting the best one. This adaptive decision fusion is performed based on the local scale of the structure. The interest of the proposed method is verified using both visual and quantitative analyses for different Pléiades data sets.

With increased use of precision agriculture techniques, information concerning within-field crop yield variability is becoming increasingly important for effective crop management. Despite the commercial availability of yield monitors, many crop harvesters are not equipped with them. Moreover, yield monitor data can only be collected at harvest and used for after-season management. On the other hand, remote sensing imagery obtained during the growing season can be used to generate yield maps for both within-season and after-season management. This paper gives an overview on the use of airborne multispectral and hyperspectral imagery and high-resolution satellite imagery for assessing crop growth and yield variability. The methodologies for image acquisition and processing and for the integration and analysis of image and yield data are discussed. Five application examples are provided to illustrate how airborne multispectral and hyperspectral imagery and high-resolution satellite imagery have been used for mapping crop yield variability. Image processing techniques including vegetation indices, unsupervised classification, correlation and regression analysis, principal component analysis, and supervised and unsupervised linear spectral unmixing are used in these examples. Some of the advantages and limitations on the use of different types of remote sensing imagery and analysis techniques for yield mapping are also discussed.

Hyperspectral imagery, which contains hundreds of spectral bands, has the potential to better describe the biological and chemical attributes on the plants than multispectral imagery and has been evaluated in this paper for the purpose of crop yield estimation. The spectrum of each pixel in a hyperspectral image is considered as a linear combinations of the spectra of the vegetation and the bare soil. Recently developed linear unmixing approaches are evaluated in this paper, which automatically extracts the spectra of the vegetation and bare soil from the images. The vegetation abundances are then computed based on the extracted spectra. In order to reduce the influences of this uncertainty and obtain a robust estimation results, the vegetation abundances extracted on two different dates on the same fields are then combined. The experiments are carried on the multidate hyperspectral images taken from two grain sorghum fields. The results show that the correlation coefficients between the vegetation abundances obtained by unsupervised linear unmixing approaches are as good as the results obtained by supervised methods, where the spectra of the vegetation and bare soil are measured in the laboratory. In addition, the combination of vegetation abundances extracted on different dates can improve the correlations (from 0.6 to 0.7).

In this paper, the ability of unsupervised linear unmixing has been evaluated for the hyperspectral image taken on the south pole of the planet Mars by OMEGA instrument aborad MEX. State-of-art methods of the three steps are tested: i) estimation of the number of endmembers; ii) extraction of endmembers; ii) estimation of the abundances. According to the results, it has been found that the combination ELM-VCA-NCLS has the best performance for obtaining the spectral signatures and the abundances of the three principle chemical species (CO2ice, water ice and dust) on the south pole of the Mars.

The first contribution of this paper is to evaluate unsupervised linear unmixing approaches for hyperspectral images for crop yield estimation. In this paper we will use a very efficient endmember extraction approach the Vertex Component Analysis (VCA). The second contribution of this paper is to use the hyperspectral images of the same fields taken on two different dates for yield estimation. Even though the images are often taken in good conditions (sunny and calm weather), the light and weather can still have influence on the observation. In addition, calibration errors can randomly occur in an image. The relations between the yield data and different observations will vary. The fusion of different independent observations on the same scene can reduce the influences of this uncertainty and obtain a robust result. In this paper, we propose to combine, for each field, the unmixing results obtained on two different dates in order to improve the accuracy of the estimation.

As the volume of hyperspectral data for planetary exploration increases, efficient yet accurate algorithms are decisive for their analysis. In this paper, the capability of spectral unmixing for analyzing hyperspectral images from Mars is investigated. For that purpose, we consider the Russell megadune observed by the Compact Reconnaissance Imaging Spectrometer for Mars (CRISM) and the High-Resolution Imaging Science Experiment (HiRISE) instruments. In late winter, this area of Mars is appropriate for testing linear unmixing techniques because of the geographical coexistence of seasonal CO2ice and defrosting dusty features that is not resolved by CRISM. Linear unmixing is carried out on a selected CRISM image by seven state-of-the-art approaches based on different principles. Three physically coherent sources with an increasing fingerprint of dust are recognized by the majority of the methods. Processing of HiRISE imagery allows the construction of a ground truth in the form of a reference abundance map related to the defrosting features. Validation of abundances estimated by spectral unmixing is carried out in an independent and quantitative manner by comparison with the ground truth. The quality of the results is estimated through the correlation coefficient and average error between the reconstructed and reference abundance maps. Intercomparison of the selected linear unmixing approaches is performed. Global and local comparisons show that misregistration inaccuracies between the HiRISE and CRISM images represent the major source of error. We also conclude that abundance maps provided by three methods out of seven are generally accurate, i.e., sufficient for a planetary interpretation.

In order to extract geometrical features from a multispectral image and derive a classification, an approach based on the topographic map of the image is proposed. For each pixel, the most significant structure containing it is extracted. The classification of this pixel is based on its spectral information and the geometrical features of the corresponding structure (its area and perimeter). The results obtained on multispectral remote sensing images taken by two different sensors show the efficiency of the extracted geometrical features for separating some classes with very similar spectral attributes but of different semantic meanings.

Pan sharpening may be defined as the process of synthesizing multispectral images at a higher spatial resolution. Different pan sharpening methods produce images with different characteristics. In the 2006 IEEE Data Fusion Contest, À-trous Wavelet Transform based pansharpening (AWLP) and Context Adaptive (CBD) pansharpening methods were declared as joint winners. While assessing the quantitative quality of the pansharpened images, it was observed that the two methods outperform each other depending upon the local content of the scene. Hence, it is interesting to develop a method which could produce results locally approximately similar to the best method, among the two pansharpening methods. In this paper we propose a method which selects either of the two methods for performing pansharpening on local regions, based upon the size of the objects. The results obtained demonstrate that the proposed method produces images with quantitative results approximately similar to the method which is better among the AWLP and CBD pansharpening methods.

Multispectral and hyperspectral imagery are often used for estimating crop yield. This paper describes an unsupervised unmixing scheme of hyperspectral images on field in order to estimate the crop yield. From the hyperspectral images, the endmembers and their abundance maps are computed by unsupervised unmixing. The abundance maps are then compared with the crop yield data. The results show the capability for estimating crop yield of the unmixing scheme, thanks to the high correlations between the crop yield data and the abundance maps of the endmembers corresponding to crop, even though the scheme is totally unsupervised.

In this paper, we present an unsupervised classification algorithm for hyperspectral images. For reducing the dimension of hyperspectral data, we use a linear unmixing algorithm to extract the endmembers and their abundance maps. Compared to the components obtained by traditional PCA-based method, the abundance maps have physical meanings (such as the abundance of vegetation). For determining the number of endmembers contained in an image, we propose an eigenvalue based approach. The validation of this approach on synthetic data shows that this approach provides a robust estimation of the actual number of endmembers. Using the estimated abundance maps of the endmembers, we perform a preliminary segmentation and use the mean values of the segmented regions as feature for the classification. We then perform K-means classifications on the segmented abundance maps with the number of clusters determined by the Krzanowski and Lai's method.

In this paper, we try to identify and quantify the chemical species present on the surface of planet Mars with the help of hyperspectral images provided by the instrument OMEGA. For this purpose, we suppose that the spectrum of each pixel is a linear mixture of the spectra of different endmembers. From this linear mixture hypothesis, our work is divided into two steps. Firstly, we propose a new unsupervised method for estimating the number of endmembers based on the eigenvalues of covariance and correlation matrix of the hyperspectral data. This method is then validated on synthetic data. With the help of the number estimated by the precedent step, we use the vertex component analysis (VCA) to extract the spectra and the abundances of the end members. The results on hyperspectral image taken by the instrument OMEGA are shown.

Space agencies are rapidly building up massive image databases. A particularity of these databases is that they are made of images with different, but known, resolutions. In this paper, we introduce a new scheme allowing us to compare and index images with different resolutions. This scheme relies on a simplified acquisition model of satellite images and uses continuous wavelet decompositions. We establish a correspondence between scales which permits us to compare wavelet decompositions of images having different resolutions. We validate the approach through several matching and classification experiments, and we show that taking the acquisition process into account yields better results than just using scaling properties of wavelet features.

We study the problem of finding the characteristic scale of a given satellite image. This feature is defined so that it does not depend on the spatial resolution of the image. This is a different problem than achieving scale invariance, as often studied in the literature. Our approach is based on the use of a linear scale space and the total variation (TV). The critical scale is defined as the one at which the normalized TV reaches its maximum. It is shown experimentally, both on synthetic and real data, that the computed characteristic scale is resolution independent.

We study the problem of finding the characteristic scale of a given satellite image. We want to define this feature so that it does not depend on the spatial resolution of the image. Our approach is based on the use of a linear scale space and the total variation. The critical scale is defined as the one at which the normalized total variation is maximum.

In this paper, we propose a new scheme to extrapolate wavelet features with respect to the resolution. By explicitly taking into account the acquisition process of satellite images, we compute how wavelet features behave when the resolution changes. This approach is validated by classifying satellite images with different resolutions.