Recently, while significant progress has been made in remote sensing image change captioning, existing methods fail to filter out areas unrelated to actual changes, making models susceptible to irrelevant features. In this article, we propose a novel multimodal model for remote sensing image change captioning, guided by Key Change Features and Instruction-tuned (KCFI). This model aims to fully leverage the intrinsic knowledge of large language models through visual instructions and enhance the effectiveness and accuracy of change features using pixel-level change detection tasks. Specifically, KCFI includes a ViTs encoder for extracting bi-temporal remote sensing image features, a key feature perceiver for identifying critical change areas, a pixel-level change detection decoder to constrain key change features, and an instruction-tuned decoder based on a large language model. Moreover, to ensure that change captioning and change detection tasks are jointly optimized, we employ a dynamic weight-averaging strategy to balance the losses between the two tasks. We also explore various feature combinations for visual fine-tuning instructions and demonstrate that using only key change features to guide the large language model is the optimal choice. To validate the effectiveness of our approach, we compare it against several state-of-the-art change captioning methods on the LEVIR-CC dataset, achieving the best performance. Our code will be available at https://github.com/yangcong356/KCFI.git.

Visual grounding for remote sensing images (RSVG) aims to localize the referred objects in the remote sensing (RS) images according to a language expression. Existing methods tend to align visual and text features followed by concatenation and then employ a fusion Transformer to learn a token representation for final target localization. However, simple fusion Transformer structure fails to sufficiently learn the location representation of referred object from the multi-modal features. Inspired by the detection Transformer, in this paper, we propose a novel language query based Transformer framework for RSVG termed LQVG. Specifically, we adopt the extracted sentence-level text features as the queries, called language queries, to retrieve and aggregate representation information of the referred object from the multi-scale visual features in the Transformer decoder. The language queries are then converted into object embeddings for final coordinate prediction of referred object. Besides, a multi-scale cross-modal alignment module is devised before the multimodal Transformer to enhance the semantic correlation between the visual and text features, thus facilitating the cross-modal decoding process to generate more precise object representation. Moreover, a new RSVG dataset named RSVG-HR is built to evaluate the performance of the RSVG approaches on very high-resolution remote sensing images with inconspicuous objects. Experimental results on two benchmark datasets demonstrate that our proposed method significantly surpasses the comparison methods and achieves state-of-the-art performance. The dataset and code are available at https://github.com/LANMNG/LQVG.

Deep neural networks have demonstrated impressive capabilities in hyperspectral image (HSI) classification tasks. However, they are highly vulnerable to adversarial attacks, raising significant security concerns, especially in the remote sensing community. Even subtle adversarial perturbations that are imperceptible to human observers can mislead deep learning (DL) models and result in incorrect predictions. Therefore, ensuring the robustness of DL models has become a critical focus in addressing security-related remote sensing tasks. Considerable progress has been made in defending against adversarial attacks in HSI classification. Nevertheless, existing methods primarily concentrate on spatial relationships between pixels while overlooking the valuable spectral information present in HSI. Besides, these methods are usually limited to a specific scale and cannot accommodate the precise classification demands for ground objects with various scales. To address these limitations, we propose a spatial–spectral self-attention learning network (S3ANet) for defending against adversarial attacks in HSI classification. Our S3ANet incorporates a pyramid spatial attention learning module to effectively capture spatial dependency at multiple scales. In addition, it utilizes a global spectral transformer to establish correlations between pixels in the spectral dimension. By employing the defense method of spatial–spectral fusion, our model can effectively address adversarial attacks from a comprehensive perspective, seamlessly integrating both spatial and spectral information. Extensive experiments conducted on four benchmark HSI datasets illustrate that the proposed S3ANet achieves competitive performance compared to state-of-the-art methods when faced with adversarial attacks. The code is available online athttps://github.com/YichuXu/S3ANet.

Road damage presents a significant risk to traffic safety, including pavement distress and disaster-induced damage. Thanks to their high efficiency, computer vision-based methods for pavement distress detection have been widely developed. In disaster scenarios, the automatic extraction of road damage information from extensive social media images plays a critical role in rescue efforts. However, few existing studies have focused on detecting object-level disaster-induced road damage. To fill the gap, this paper presents a Social media image dataset of Object detection for Disaster-induced Road damage (SODR), including 1,552 images and two categories (i.e., collapses and blockages). Additionally, this paper proposes an ensemble learning approach with attention mechanisms based on YOLOv5 (You Only Look Once) network. Initially, attention modules are employed to create two distinct detectors for ensemble learning. Subsequently, one standard YOLOv5 and two variant networks are trained with consistent settings, and test time augmentation is applied during the inference phase. The proposed method has been implemented across five scales of YOLOv5, offering alternatives for balancing accuracy and computational cost. To demonstrate the validity, comprehensive experiments were conducted on two datasets. Compared with some mainstream detectors and ensemble learning methods, our approach achieved competitive results with a fewer number of parameters and a simpler training and testing process. The SODR dataset and source code are available at (https://github.com/nonondayo/yolov5_SODRv1).

Road damage detection is significant for road maintenance. Traditional manual visual inspection methods consume lots of time and labor. Developments in the field of computer vision create opportunities for automated and efficient image-based road damage detection. Through deep convolution neural networks, road damage localization and classification can be achieved simultaneously. This paper proposes an ensemble model with test time augmentation based on the You Only Look Once (YOLOv5) network and attention modules. The approach utilizes a state-of-the-art object detector known as YOLOv5. To focus more on the road in images, five improved YOLOv5 models with attention modules are proposed. Moreover, ensemble learning and test time augmentation are adopted to improve model generalization and detection performance. The proposed method was evaluated through the IEEE Big Data Crowdsensing-based Road Damage Detection Challenge 2022. Different ensemble models achieved an average F1-score of 0.65177 on the five test datasets.

Mobile phone data can provide insightful location-based information on the interactions between individuals and the urban environment, e.g. urban land-use types. Using mobile phone data, this study aims to recognize citywide land use classes in a highly accurate and timely manner, because land use classes are important for urban planning and urban management. How to extract key and discriminative features from mobile-phone-call time series is an essential problem for urban land-use classification. Considering the high dimensional characteristic of mobile-phone-call time series data, an improved land-use classification framework with deep convolutional autoencoder (DCAE) is proposed to automatically recognize land-use types in urban areas, which are geographical regions clustered based on the similarity of features in waveform and magnitude of mobile-phone-call time series. The method is validated using mobile phone data collected in Wuhan city, China. The experimental result shows that the key and discriminative features in waveform and magnitude can be extracted without human prior knowledge and these features extracted by DCAE can improve the performance to distinguish functionally similar land-use types, such as business and industrial/commercial types.

Remote sensing image clustering is a challenging task. Recently, by combining the spectral and spatial information of remote sensing data, the clustering accuracy can be dramatically enhanced. However, it has always been difficult to determine the weight parameter for balancing the spectral and spatial terms of the clustering objective function. In this paper, spectral-spatial clustering with a local weight parameter determination method for remote sensing imagery is proposed, i.e. LSSC. In LSSC, considering the large scale of remote sensing images, the weight parameter is determined locally in a patch image instead of the whole image. The local weight parameter is then used in constructing the objective function of LSSC. Thus, the remote sensing image clustering problem is transformed into an optimization problem. Finally, in order to achieve a better optimization performance, a variant of differential evolution (i.e. jDE) is used as the optimizer due to its powerful optimization capability. Experimental results confirm that the proposed LSSC can acquire a higher clustering accuracy than other spectral-spatial clustering methods.

In this letter, a multifeature probabilistic ensemble conditional random field (MFPECRF) model is proposed to perform the task of change detection for high spatial resolution (HSR) remote sensing imagery. MFPECRF not only considers the spectral feature of single pixels but also the interaction between neighborhood pixels and the structural property of the ground objects in HSR imagery to give a higher detection accuracy than the traditional random field methods, which only utilize spectral and label information. In the unary potential, the spectral and morphological features of the difference image are combined using a probabilistic ensemble strategy, and the pairwise potential considers the contextual information of the observed field. The parameters of MFPECRF are estimated using a piecewise strategy, and the final result is obtained by the use of the loopy belief propagation algorithm. The experimental results of two groups of HSR multispectral images confirm the potential of the proposed method in improving the detection accuracy for HSR imagery.

Hyperspectral remote sensing images, which are characterized by their high dimensionality, provide us with the capability to accurately identify objects on the ground. They can also be used to identify subclasses of objects. However, these subclasses are usually embedded in different subspaces due to the complex distribution of pixels in the feature space. In the literature, few hyperspectral image classification methods can take both the subclass and subspace into consideration at the same time. Motivated by the fact that natural DNA can distinguish biological subspecies (subclasses in hyperspectral images) using critical DNA fragments (subspaces in hyperspectral images), a semisupervised subspace-based DNA encoding and matching classifier for hyperspectral remote sensing imagery (SSDNA) is proposed in this paper. First, in the process of DNA encoding, the hyperspectral remote sensing image is transformed into a DNA cube, in which the first-order spectral curve of the hyperspectral remote sensing image is utilized in order to take the gradient information of the spectral curve into consideration. Second, in the process of DNA optimization, evolutionary algorithms are used to obtain the best DNA library of the typical objects, which includes the following: 1) A multicenter individual representation is designed in order to consider the existence of subclasses in the hyperspectral remote sensing image; 2) the unlabeled samples are utilized in the process of population initialization and fitness calculation to enhance the diversity of the population and the generalization of the classification performance; and 3) the different classes are embedded in different subspaces. A semisupervised technique is used to extract the subspaces, including the global subspace for all the classes and the local subspace for each class. Three hyperspectral data sets were tested and confirm that SSDNA performs better than the other supervised or semisupervised classifiers.

Hyperspectral remote sensing image classification is one of the most challenging tasks. In our previous work, motivated by the similarity between the structures of DNA and hyperspectral remote sensing images, a DNA matching mechanism was used to transform the hyperspectral remote sensing image into a DNA cube for classification. However, the above DNA encoding strategy lacks the process of encoding accurate spectral and spatial feature into the DNA cube, resulting in unsatisfying classification performance. In this paper, a spectral-spatial DNA encoding strategy for encoding accurate spectral and spatial feature of hyperspectral remote sensing image is proposed. In the spectral dimension, the first-order spectral curve is encoded into the DNA cube, while in the spatial dimension, the principal components or their corresponding texture feature (GLCM) are encoded into the DNA cube. Finally, different with the previous DNA encoding classifier using genetic algorithm (GA), the paper combines the discriminative classifier (i.e. SVM) with spectral-spatial DNA encoding to improve classification performance for hyperspectral remote sensing imagery. The experimental results confirmed the effectiveness of the newly devised DNA encoding strategy and the discriminative classifier in classifying the DNA cube.

Hyperspectral remote sensing image clustering, with the large volume, high dimensions, and temporal-spatial spectral diversity, is a challenging task due to finding interesting clusters in the sparse feature space. In this paper, a novel hyperspectral clustering algorithm, namely, an unsupervised spectral matching classifier based on artificial DNA computing (UADSM), is proposed to perform the task of clustering different ground objects in specific spectral DNA feature encoding subspaces. UADSM builds up the clustering framework with the spectral encoding, optimizing, and matching mechanism by introducing the basic notions and operators of artificial DNA computing. By discretized spectral DNA feature encoding processing, the spectral shape, amplitude, and slope features of the hyperspectral data are extracted. Furthermore, the optimal clustering centers in the form of DNA strands can be found by recombining the DNA strands in the spectral DNA encoding subspace. Finally, a reasonable category for each spectral signature is automatically identified by the normalized spectral DNA similarity norm. The traditional clustering methods of k-means, ISODATA, fuzzy c-means classifier, and FCM and MoDEFC after principal component analysis transformation are provided to compare with the UADSM classifier, using Hyperspectral Digital Imagery Collection Experiment and Reflective Optics System Imaging Spectrometer hyperspectral images. The experimental results show that the UADSM classifier can achieve the best classification accuracy; hence, it is considered that the UADSM classifier is an effective clustering method for hyperspectral remote sensing imagery.

In this paper, a novel hyperspectral subspace clustering algorithm based on decision fusion strategy (SCDFS) is proposed. Because the different clusters are contained in different subspace of the same hyper-dimensional data, the clustering processing in different subspace is conducted by genetic K-means algorithm (KGA). The clustering results from different subspace can be combined into decision string. The proposed subspace clustering based on decision fusion strategy is conducted on decision string. Considering the selection of subspace, the decision results may be inaccurate. So by the majority voting processing for different subspace, the steady subspace combination can be determined. Finally, the weighted strategy is introduced into SCDFS algorithm to evaluate the distance of different decision string, and determine the fusion clustering result.

In this paper, a spectral encoding and matching algorithm inspired by biological deoxyribonucleic acid (DNA) computing is proposed to perform the task of spectral signature classification for hyperspectral remote sensing data. As a novel branch of computational intelligence, DNA computing has the strong computing and matching capability to discriminate the tiny differences in DNA strands by DNA encoding and matching in the molecule layer. Similar to DNA discrimination, a hyperspectral remote sensing data matching approach is used to recognize the land cover material from a spectral library or image, according to the rich spectral information. However, it is difficult to apply DNA computing to hyperspectral remote sensing data processing because traditional DNA computing often relies on biochemical reactions of DNA molecules and may result in incorrect or undesirable computations. To utilize the advantages and avoid the problems of biological DNA computing, an artificial DNA computing approach is proposed for spectral encoding and matching for hyperspectral remote sensing data. A DNA computing-based spectral matching approach is used to first transform spectral signatures into DNA codewords by capturing the key spectral features with a spectral feature encoding operation. After DNA encoding, the typical DNA database for interesting classes is constructed and saved by DNA evolutionary operating mechanisms such as crossover, mutation, and structured mutation. During the course of spectral matching, each pixel of the hyperspectral image, or each signature measured in the field, is input to the constructed DNA database. By computing the distance between an unclassified spectrum and the typical DNA codewords from the database, the class property of each pixel is set as the minimum distance class. Experiments using different hyperspectral data sets were performed to evaluate the performance of the proposed artificial DNA computing-based spectral matching algorithm by comparing it with other traditional hyperspectral classifiers, including spectral matching classifiers (binary coding, spectral angle mapper and spectral derivative feature coding (SDFC) matching methods) and a novel statistical method of machine learning termed support vector machine (SVM). Experimental results demonstrate that the proposed algorithm is distinctly superior to the three traditional hyperspectral data classification algorithms. It presents excellent processing efficiency, compared to SVM, with high-dimensional data captured by the Hyperspectral Digital Imagery Collection Experiment sensor, and hence provides an effective option for spectral matching classification of hyperspectral remote sensing data.