Effective feature representation is pivotal in numerous remote sensing image (RSI) interpretation tasks. Notably, a distinct attribute of RSIs is their inclination toward multiscale feature dependence. Previous research predominantly focuses on designing intricate and complex networks or modules to encapsulate rich multiscale features. However, these approaches compromise on either the model’s compactness or its representational efficacy, thereby constraining the practical deployment of remote sensing technologies, particularly in limited-capacity environments like small-scale devices or on-orbit satellites. In this study, we explore the problem of how to augment the diversity of encoded features while avoiding heavy parameter scale growth in deep convolutional neural networks (CNNs). We proposed an adaptive multiscale framework RISV which presents two key features: 1) rich scale information: during training, RISV decomposes each convolutional layer into various-sized convolutions, extracting multiscale characteristics; and 2) small model volume: RISV incorporates a differentiable elect layer after each convolutional layer, adaptively calculating and polarizing channel importance during learning. After training, the added convolution kernel and the significant channels selected by the elect layer will be linearly equivalent merged, minimizing the impact of pruning on the model’s feature extraction capability. Different from traditional model slimming, it focused on a slimmed-down network while enhancing the representation of multiscale features in RSIs. Versatile and adaptable across various model frameworks like VGG and ResNet. Experimental results demonstrate that our methodology not only preserves accuracy across standard skeletal frameworks but also attains a compression ratio exceeding 80%, surpassing the baseline by an average of 40%. Furthermore, the application of GradCAM on the NWPU dataset reveals our method’s proficiency in acquiring detailed and accurate subject information from RSIs. The source code can be available athttps://github.com/GeoX-Lab/RISV.

Learning from a sequence of tasks for a lifetime is essential for an agent toward artificial general intelligence. Despite the explosion of this research field in recent years, most work focuses on the well-known catastrophic forgetting issue. In contrast, this work aims to explore knowledge-transferable lifelong learning without storing historical data and significant additional computational overhead. We demonstrate that existing data-free frameworks, including regularization-based single-network and structure-based multinetwork frameworks, face a fundamental issue of lifelong learning, named anterograde forgetting, i.e., preserving and transferring memory may inhibit the learning of new knowledge. We attribute it to the fact that the learning network capacity decreases while memorizing historical knowledge and conceptual confusion between the irrelevant old knowledge and the current task. Inspired by the complementary learning theory in neuroscience, we endow artificial neural networks with the ability to continuously learn without forgetting while recalling historical knowledge to facilitate learning new knowledge. Specifically, this work proposes a general framework named cycle memory networks (CMNs). The CMN consists of two individual memory networks to store short- and long-term memories separately to avoid capacity shrinkage and a transfer cell between them. It enables knowledge transfer from the long-term to the short-term memory network to mitigate conceptual confusion. In addition, the memory consolidation mechanism integrates short-term knowledge into the long-term memory network for knowledge accumulation. We demonstrate that the CMN can effectively address the anterograde forgetting on several task-related, task-conflict, class-incremental, and cross-domain benchmarks. Furthermore, we provide extensive ablation studies to verify each framework component. The source codes are available at: https://github.com/GeoX-Lab/CMN.

Overcoming catastrophic forgetting is a key difficulty for remote sensing image (RSI) classification in open world applications. The core of this problem lies in the ability of RSI scene classification models to adapt to the changing environment and maintain the learned knowledge while continually learning new knowledge. Mainstream replay-based approaches overcome catastrophic forgetting by reenacting and retracing past experiences in the process of learning new data. However, such approaches rely heavily on the storage of historical data, and the recent rise of new paradigms based on prompt learning offers a new perspective of using only task-related “instructions” (i.e., prompts) to guide the model’s continual learning and reasoning. Therein, the task knowledge encoded by the prompt improves the model’s ability to overcome forgetting while reducing the amount of data and model parameters required by traditional data-driven approaches. Therefore, we propose a continual learning method based on prompt learning for RSI classification. We systematically analyze and reveal the potential of prompt learning for continual learning of RSI classification. Experiments on three publicly available remote sensing datasets show that prompt learning significantly outperforms two comparable methods on 3, 6, and 9 tasks, with an average accuracy (ACC) improvement of approximately 43%. Performance improvements of 4% to 6% were achieved when compared to advanced prototype network methods. We found that prompt-generation strategies and prompt-related components significantly affect performance: (1) prompt-generation strategies are strongly correlated with the model’s performance in overcoming catastrophic forgetting; (2) prompt-related components are correlated with remote sensing images of different scales. The new paradigm of prompt learning potentially provides a new idea for the continual learning problem of RSI classification.

Lifelong learning is important in remote sensing image understanding, especially in an open world where streaming of data are incrementally available. Current related research mainly focuses on preserving learned knowledge (i.e., avoiding catastrophic forgetting) while devoting less attention to the exploitation of historical knowledge to facilitate the learning of new knowledge. Here, we propose a new framework to bridge this gap. It consists of two sub-networks that memorize old and learn new knowledge separately, and exploit the synergy of transfer cells and triple distillation to take advantage of the valuable knowledge from previous learned tasks to facilitate learning new tasks while avoiding forgetting old tasks. Furthermore, it uses an asymmetric structure considering feature generality on historical tasks and scale- and channel-feature dependence on remote sensing images for specific tasks. Experimental results obtained in scene classification on several open benchmarks demonstrate the effectiveness of the framework.

Convolutional neural networks (CNNs) have achieved remarkable results in semantic segmentation of high-resolution remote sensing images (HRRSIs). However, the scales and textures of HRRSIs are diverse, which makes it difficult for a fixed-layer CNN to obtain rich features. In this regard, we propose an end-to-end ensemble fully convolutional network (EFCNet), which mainly includes two modules: the adaptive fusion module (AFM) and the separable convolutional module (SCM). The AFM can fuse features of different scales based on ensemble learning, whereas the SCM can reduce the complexity of the model under multifeature fusion. In the experiment, we use UNet and PSPNet to verify the framework on the ISPRS Vaihingen and Potsdam datasets. The experimental results show that the EFCNet can effectively improve the final segmentation performance and reduce the complexity of the ensemble model.

Generative adversarial networks (GANs) are subject to catastrophic forgetting when learning stream of data. Inspired by the knowledge of neuroscience, this article develops a memory formation system (MFS) to establish memory for GANs. MFS is composed of three modules, including the identifier, weights upgrade (WU), and weights reactivate (WR). These modules simulate the function of encoding, consolidating, and retrieving in memory formation of human. Identifier creates indexes to label continuous tasks and these indexes are used as a cue when the corresponding tasks are recalled. WU and WR work as synaptic consolidation and system consolidation, respectively. In WU, a novel method, weight saliency measure (WSM) is proposed to measure the saliency of weight. Valuable weights are protected and invaluable weights are released for update when GAN is trained for the new task. In WR, traditional data-replay methods are improved by selecting${k}$representatives in each class and their feature vectors are restored. When pseudo data of this class are regenerated, only the samples with distance less than a threshold can be used in future training. Experimental results based on the testing of continual image generation and continual 3-D reconstruction show that MFS can establish a memory system to handle the catastrophic forgetting problem effectively.

Convolutional neural networks (CNNs) have recently been widely used in remote-sensing scene classification. Additionally, it is becoming very popular to automatically learn specific CNN architectures for specific data sets. The rich contextual information in high-resolution remote-sensing images (RSIs) is critical to remote-sensing intelligent understanding tasks. However, architecture learning approaches tend to simplify the original data (i.e., resizing images to smaller resolution) for efficiency, yet result in contextual information loss of RSIs. In this article, we proposed a contextual information-preserved architecture learning (CIPAL) framework for remote-sensing scene classification to utilize the contextual information in RSIs as much as possible during the architecture learning process. We introduce channel compression into CIPAL, which can reduce the memory and time consumption of architecture learning and make it possible to construct a larger architecture space. We add potential operators that are rarely used for scene classification tasks (i.e., atrous convolution) into the architecture space to explore unknown architectures that are more suitable for remote-sensing scenes. The experimental results on four remote-sensing scene classification benchmarks indicate that CIPAL learns architectures with less time consumption than similar works, and the newly found architectures outperform popular hand-designed architectures for better use of contextual information in RSIs. Different architectures are good at learning different representations, and our proposed architecture learning method potentially helps us understand which types of representations are crucial for RSI intelligent understanding.

To infer unknown remote sensing scenarios, most existing technologies use a supervised learning paradigm to train deep neural network (DNN) models on closed datasets. This paradigm faces challenges such as highly spatiotemporal variants and ever-changing scale-heterogeneous remote sensing scenarios. Additionally, DNN models cannot scale to new scenarios. Lifelong learning is an effective solution to these problems. Current lifelong learning approaches focus on overcoming thecatastrophic forgettingissue (i.e., a successive increase in heterogeneous remote sensing scenes causes models to forget historical scenes) while ignoring theknowledge recallissue (i.e., how to facilitate the learning of new scenes by recalling historical experiences), which is a significant problem. This paper proposes a lifelong learning framework called asymmetric collaborative network (SCN) for lifelong remote sensing image classification. This framework consists of two structurally distinct networks: a preserving network (Pres-Net) and a transient network (Trans-Net), which imitates the long- and short-term memory processes in the brain, respectively. Moreover, this framework is based on two synergistic knowledge transfer mechanisms: triple distillation and prior feature fusion. The triple distillation mechanism enables knowledge persistence from Trans-Net to Pres-Net to achieve better memorization; the prior feature fusion mechanism enables knowledge transfer from Pres-Net to Trans-Net to achieve better recall. Experiments on three open datasets demonstrate the effectiveness of SCN for 3-, 6-, and 9-task-length learning. The idea of asymmetric separation networks and the synergistic strategy proposed in this paper are expected to provide new solutions to the translatability of the classification of remote sensing images in real world scenarios.

Enabling a neural network to sequentially learn multiple tasks is of great significance for expanding the applicability of neural networks in real-world applications. However, artificial neural networks face the well-known problem of catastrophic forgetting. What is worse, the degradation of previously learned skills becomes more severe as the task sequence increases, known as the long-term catastrophic forgetting. It is due to two facts: first, as the model learns more tasks, the intersection of the low-error parameter subspace satisfying for these tasks becomes smaller or even does not exist; second, when the model learns a new task, the cumulative error keeps increasing as the model tries to protect the parameter configuration of previous tasks from interference. Inspired by the memory consolidation mechanism in mammalian brains with synaptic plasticity, we propose a confrontation mechanism in which Adversarial Neural Pruning and synaptic Consolidation (ANPyC) is used to overcome the long-term catastrophic forgetting issue. The neural pruning acts as long-term depression to prune task-irrelevant parameters, while the novel synaptic consolidation acts as long-term potentiation to strengthen task-relevant parameters. During the training, this confrontation achieves a balance in that only crucial parameters remain, and non-significant parameters are freed to learn subsequent tasks. ANPyC avoids forgetting important information and makes the model efficient to learn a large number of tasks. Specifically, the neural pruning iteratively relaxes the current task's parameter conditions to expand the common parameter subspace of the task; the synaptic consolidation strategy, which consists of a structure-aware parameter-importance measurement and an element-wise parameter updating strategy, decreases the cumulative error when learning new tasks. Our approach encourages the synapse to be sparse and polarized, which enables long-term learning and memory. ANPyC exhibits effectiveness and generalization on both image classification and generation tasks with multiple layer perceptron, convolutional neural networks, and generative adversarial networks, and variational autoencoder. The full source code is available at https://github.com/GeoX-Lab/ANPyC.

Traditional online map tiles, which are widely used on the Internet, such as by Google Maps and Baidu Maps, are rendered from vector data. The timely updating of online map tiles from vector data, for which generation is time-consuming, is a difficult mission. Generating map tiles over time from remote sensing images is relatively simple and can be performed quickly without vector data. However, this approach used to be challenging or even impossible. Inspired by image-to-image translation (img2img) techniques based on generative adversarial networks (GANs), we proposed a semisupervised generation of styled map tiles based on the GANs (SMAPGAN) model to generate styled map tiles directly from remote sensing images. In this model, we designed a semisupervised learning strategy to pretrain SMAPGAN on rich unpaired samples and fine-tune it on limited paired samples in reality. We also designed the image gradient L1 loss and the image gradient structure loss to generate a styled map tile with global topological relationships and detailed edge curves for objects, which are important in cartography. Moreover, we proposed the edge structural similarity index (ESSI) as a metric to evaluate the quality of the topological consistency between the generated map tiles and ground truth. The experimental results show that SMAPGAN outperforms state-of-the-art (SOTA) works according to the mean squared error, the structural similarity index, and the ESSI. Also, SMAPGAN gained higher approval than SOTA in a human perceptual test on the visual realism of cartography. Our work shows that SMAPGAN is a new tool with excellent potential for producing styled map tiles. Our implementation of SMAPGAN is available at https://github.com/imcsq/SMAPGAN.

Deep neural networks (DNNs), which learn a hierarchical representation of features, have shown remarkable performance in big data analytics of remote sensing. However, previous research indicates that DNNs are easily spoofed by adversarial examples, which are crafted images with artificial perturbations that fool DNN models toward wrong predictions. To comprehensively evaluate the impact of adversarial examples on the remote sensing image (RSI) scene classification, this study tests eight state-of-the-art classification DNNs on six RSI benchmarks. These data sets include both optical and synthetic-aperture radar (SAR) images of different spectral and spatial resolutions. In the experiment, we create 48 classification scenarios and use four cutting-edge attack algorithms to investigate the influence of the adversarial example on the classification of RSIs. The experimental result shows that the fooling rates of the attacks are all over 98% across the 48 scenarios. We also find that, for the optical data, the seriousness of the adversarial problem has a negative relationship with the richness of the feature information. Besides, adversarial examples generated from SAR images are used easily for fooling the models with an average fooling rate of 76.01%. By analyzing the class distribution of these adversarial examples, we find that the distribution of the misclassifications is not affected by the types of models and attack algorithms-adversarial examples of RSIs of the same class cluster on fixed several classes. The analysis of classes of adversarial examples not only helps us explore the relationships between data set classes but also provides insights for further designing defensive algorithms.