Panoptic segmentation unifies the traditional tasks of instance and semantic segmentation. It plays a crucial role in the field of remote sensing; however, it encounters challenges in recognizing small objects and in the model’s ability to generalize across complex scenes. In this paper, we introduce the MFIAE framework to address two specific challenges: a multi-scale interactive attention fusion (MSIAF) module and an adaptive disturbance sparse mixture-of-experts (ADSMoE) module based on Transformer. The MSIAF module is designed to fully utilize the rich contextual information captured by low-resolution features while simultaneously utilizing the advantages of high-resolution features to enhance small object segmentation. In the ADSMoE module, adaptive noise is introduced to disturb the expert selection in order to enhance the randomness and exploration of the model when it comes to selecting experts, thereby improving its capacity for generalization and robustness. Additionally, it also reduces the computational overhead and the model complexity. The experimental results demonstrate that our approach achieves state-of-the-art performance on the BSB Aerial dataset.

Effectively reconstructing 3D hyperspectral images (HSIs) from 2D measurements presents a significant challenge in Coded Aperture Snapshot Spectral Imaging (CASSI) systems. While recent transformers exhibit potential in HSI reconstruction, they often suffer from inadequate exploration of multi-scale spatial-spectral self-similarity, leading to mean effects and information loss. Additionally, these methods struggle with insufficient modeling of the degradation inherent in the compressive imaging process. To address these issues, we propose a novel Mask-guided Multi-scale Spatial-Spectral Transformer (MMSST). Specifically, we introduce a Degradation Aware Mask Attention (DAMA) module to incorporate degradation information of the compressive imaging process. Furthermore, MMSST leverages Local-Regional SpAtial attention (LRSA) and Global-Regional SpEctral attention (GRSE) to effectively exploit multi-scale self-similarity across spatial and spectral dimensions. Extensive experimental results demonstrate the effectiveness of our MMSST.

Recently auxiliary cross-scene information has been widely utilized to improve the hypersperctral image classification performance by knowledge transfer. However, recognition of different objects with the same semantic category is difficult when the object types in similar scenes are different or only limited similarity knowledge is provided. In this paper, a multi-modal feature interactive learning (MMFI) method is proposed based on both hyperspectral image modality and textual modality to distinguish similar objects, which enhances the transfer capability by utilizing the semantic prior from the textual modality. First, the adversarial domain mapping (ADM) module is designed to realize cross-domain knowledge transfer across different scenes in an adversarial learning manner. In particular, the noise is simulated as data distribution in different domains through domain mapping and aggregated with source and target domain data, which is then reconstructed and optimized to learn discriminative and conducive information for transfer. Then, the adaptive interactive learning (AIL) module acts on the latent features of the encoder to mine latent associations among the aggregated features and facilitate the expression of consistent features. In addition, few-shot learning with textual embedding enables more powerful semantic priors for few-shot prototypes, making up for insufficient recognition capability in the presence of hyperspectral image modality only. Experimental results on three datasets demonstrate the superiority of our method.

Remote sensing change detection (CD) aims to accurately reveal surface changes by comparing two temporally separated images of the same area. However, in complex environments, insufficient edge detail recognition and limited feature extraction often affect the accuracy of CD. For this purpose, we propose a novel method named the edge-object co-driven learning network (EOCLNet), which employs a combination of the Pyramid Vision Transformer (PVT) and the Fast Segment Anything Model (FastSAM) as parallel feature extractors to capture rich multilevel features. Specifically, it includes three key components which are the edge extraction module (EEM), the object revelation module (ORM), and the edge-object learning (EOL). EEM explicitly captures edge details by combining low-level spatial features with high-level semantic features, providing essential edge knowledge. ORM reveals changed objects by aggregating the highest two levels of semantic features, providing initial change guidance. EOL is designed to implicitly mine edge clues by establishing relationships between edges and changed objects across multiple levels, receiving outputs from both EEM and ORM. Furthermore, during the training process, the uncertainty from the previous level’s change map is utilized to guide the learning at the next level, thereby achieving a transition from uncertainty to certainty. The effectiveness of EOCLNet is validated on three public datasets, where it outperforms several state-of-the-art CD methods.

Remote sensing change detection (CD) is a highly significant subtask within the field of Earth observation. Recently, weakly supervised CD (WSCD) methods based on image-level annotations have attracted interest, it is challenging to generate a clear margin between changed and unchanged regions with a lack of detailed annotation. In this article, based on class activation maps (CAMs), we propose a novel WSCD network based on self-supervised learning and change mask guidance (SSCMNet). First, we design a multiscale self-supervised constraint (MSC) module to narrow the gap between weak supervision and full supervision and compensate for the inherent shortcomings of CAMs. Second, a change mask guidance (CMG) module is proposed to further guide the network to keep the integrity of changed objects according to the consistency within unchanged regions and inconsistency within changed regions. Finally, to address the challenge of transferring commonly used post-processing methods in semantic segmentation to CD, an adaptive post-processing (APP) module is designed to adaptively select one of the input images for post-processing. We conduct experiments on three publicly available remote sensing CD datasets. Quantitative metrics and visualized results demonstrate the outstanding performance of the proposed method.

Change detection in remote sensing images aims at revealing interesting changes about the earth surface and has been one of the most important issues in earth observation. In recent years, lots of fully-supervised change detection methods have achieved good performance with the help of deep learning architectures, which rely on large amounts of pixel-level labels. However, obtaining high-quality pixel-level labels is laborious and expensive. To alleviate this problem, we propose a novel weakly-supervised change detection way via consistency instance segmentation called Box2Change, which requires only box-level labels and achieves competitive results to fully-supervised change detection method. Compared with pixel-level label, it is much more efficient to get box-level label, which locates the potential changed area by a rectangle box. There are two key components in the proposed method, the Changed Instance Segmentation (CIS) and the Self-Supervised Consistency Learning (SSCL) in affine space. The former generates multi-scale changed instances, which learns positional information from box-level labels and segments the instance boundaries within a given bounded region. The latter introduces affine transform and employs consistency constraints in a self-supervised manner to increases the robustness to pseudo-change situations caused by light or noise. In experiments, three popular public change detection datasets are tested and both visual and numerical assessment are discussed, where the proposed method exhibits competitive performance to fully-supervised methods and achieves the state-of-the-art results compared with the other weakly-supervised change detection methods.

Change detection (CD) in remote sensing aims to reveal meaningful surface changes and has been flourishing in recent years. Compared with fully-supervised methods based on pixel-level labels, image-level labels are easy to acquire, which reduces manual labor to a large extent. However, image-level labels lack spatial-and-shape information while containing the least semantic information, which poses a great challenge to the weakly-supervised CD task. Motivated by the prior that bi-temporal images have background semantic consistency, we propose Background-Driven and Foreground-Refined (BDFR-Net) to ameliorate the above problem. Specifically, there are two key components in the proposed method: the Background-Driven Reconstruction (BDR) with image-level supervision and the Foreground-Refined Learning (FRL) with affinity learning. The former generates changed regions of foreground and background separation, which activates the foreground from image-level supervision and constrains the foreground by maintaining spatial and semantic consistency in background regions. The latter introduces Complementary Fusion and Label Adaption (CFLA) strategies to further refine the foreground, which can mine complementary information from foreground sequences and suppress false activations. In addition, affinity learning is proposed to stabilize and supervise the above process. Complementary relationships between foreground and background are fully utilized. Tested on two popular CD datasets, the results demonstrate that our proposed BDFR-Net produces completely changed regions with clear boundaries and outperforms state-of-the-art weakly-supervised methods.

Change detection (CD) in remote sensing aims at revealing land cover changes according to the category of the ground objects. However, the category information is always missing in current popular vision-based CD methods. Considering that language analysis is really good at identifying different categories, a vision-language joint learning method is proposed in this paper, which consists of two vision-language joint representation (VLJR) modules and a changed instance segmentation (CIS) module. The former combines image features and language features with the help of text encoder and Transformer. The latter generates the final pixel-level CD result with only box-level labeled samples by level-set evolution and box matching supervision, which reduces manual-labor to a large extent. Tested on representative WHU datasets, the proposed method achieves comparable results to fully-supervised CD methods and is ahead of the other weakly-supervised methods.

The task of remote sensing image change detection involves identifying differences between images captured in the same geographical area but at different times. When dealing with dual-time-series images, lighting and seasonal variations often make recognition challenging. To address the challenges, based on Unet++, we innovatively introduce the Spectral-Spatial Attention Module (SSAM) to better focus on fine-grained details. SSAM uses different frequency components to allocate differential weights to channels, allowing the network to pay more attention to the features relevant to the current task. Moreover, to better capture the change details, a multi-level deep supervision strategy is introduced to enhance the discriminative ability and robustness of early features. Our proposed method is named as SSUNet and has been validated on the CDD and LEVIRE-CD datasets, demonstrating significant advantages in detail recognition.

Change detection is a concerned task in the interpretation of remote sensing images. The mining of the relationship on change features is usually implicit in the deep learning networks that contain single-branch or two-branch encoders. However, due to the lack of artificial prior design for the relationship on change features, these networks cannot learn enough semantic information on change features and lead to the poor performance. So, we propose a new network architecture explicit change-relation network (ECRNet) for the explicit mining of change-relation features. In our study of the literature, our suggestion is that the change features for change detection should be divided into prechanged image features, postchanged image features, and change-relation features. In order to fully mining these three kinds of change features, we propose the triple branch network combining the transformer and convolutional neural network (CNN) to extract and fuse these change features from two perspectives of global information and local information, respectively. In addition, we design the continuous change-relation (CCR) branch to further obtain the continuous and detailed change-relation features to improve the change discrimination capability of the model. The experimental results show that our network performs better than those of the existing advanced networks by the F1 score improvements of 0.66/0.37/0.70/1.09 on the very high-resolution (VHR) remote sensing datasets of the LEVIR-CD/SVCD/WHU-CD/SYSU-CD. Our source code is available athttps://github.com/DalongZ/ECRNet.

Semantic segmentation of remote sensing images is essential in various fields, such as earth resource census, environmental pollution monitoring, and land use planning. The segmentation performance has been significantly improved recently with the development of deep learning. However, there are still some challenges in dealing with remote sensing images. One of the main issues is that features within the same category in remote sensing images could vary significantly, while features between different categories could be more similar, leading to confusion in segmentation. Moreover, the presence of large shadow areas narrows the feature differences between categories, making segmentation even more difficult. To address these challenges, one way is to leverage contextual and multi-scale information for accurate segmentation. As a consequence, in this paper, we propose a stair fusion network with context refined attention (SFCRNet). A context-based attention embedding module is proposed to enhance the representation of the processed features by utilizing the context to maximize information retention in the channel and spatial dimensions. It can retain the information on the original channel and the association between it and other channels. Furthermore, we present a stair fusion network where a stair shaped architecture and corresponding fusion module are designed to ensure that rich semantic information from high-level features is continuously transmitted to low-level layers. The experimental results on three datasets demonstrate the effectiveness of our proposed method.

Remote sensing (RS) image change detection (CD) based on deep learning (DL), has received increasing attention recently. However, the general independent learning of bi-temporal images ignores the relationship between them, falling short in learning of the change information. In this paper, a Triple-Double (TD) framework with ability of conjoint cross-attention modeling and joint feature calibrating is proposed for CD. Specifically, the TD framework composed of Triple-branch encoder and Double-branch decoder is constructed to extract diverse features and acquire changed maps with the guidance of original edge cues. To enhance the perception of the connection between the bi-temporal features, the multi-scale difference guidance (MDG) module and conjoint cross-attention (CCA) module are designed for the dual-branch encoder, wherein the CCA introduces a novel and efficient rule for modeling the affinity in spatial and channel dimension simultaneously. Furthermore, a joint feature calibration (JFC) module is introduced to enhance the expression of feature diversity in the joint features within the single-branch encoder. Experimental results on three public datasets demonstrate the superiority of the proposed method compared to the state-of-the-art (SOTA) methods.

Change detection (CD) in remote sensing images aims at revealing earth surface changes between co-registered bitemporal images. A common way to reveal changed areas is to directly mix bitemporal features and generate CD results through supervised learning. However, a certain change usually corresponds to a real object in either of the two images, which exhibits coarse/fine shape in different scales. Therefore, a coarser-to-finer method called candidate-aware and change-guided network (CACG-Net) is proposed to effectively detect changes, where candidate objects are revealed and associated with interesting changes. Specifically, there are three key components. They are multistage change decoder (MCD), candidate-aware learning (CAL) and change guidance module (CGM). MCD reveals the most important changed objects in the coarse shape from the basic features extracted by the backbone (ResNet-18). To capture changes of interest, CAL is designed to select candidate objects in each temporal image, where a segmenter is utilized with variant change-losses. CGM intends to enrich the change details step-by-step through combining coarser change results and finer features, so that changed objects are gradually revealed in a coarser-to-finer way. Furthermore, deep supervision is employed throughout the layers of CACG-Net in the training procedure, which mitigates the learning difficulty in both deep and shallow layers. Test results on four popular datasets indicate that the proposed method outperforms several state-of-the-art CD algorithms in terms of accuracy and efficiency.

Change detection (CD) in remote sensing aims at identifying changes of specific categories from multitemporal images acquired at different moments of a given scene. Due to seasonal alteration and light variation, there are always pseudo-changes hard to be recognized. To this end, we propose a difference guidance learning way to mitigate the effects of pseudo-change, which benefits capturing more discriminative information and identifying real changes. Specifically, it combines difference information with fused features in a guidance way and generates discriminative features in multiple scales. Besides that, feature alignment is conducted in the highest stage to learn feature correlations between bitemporal images, which benefits identifying semantic changes by information exchange. Therefore, the proposed method is named feature alignment and difference guidance network (FADG-Net). Furthermore, a set of convolutional layers with different receptive field sizes is also utilized to capture spatial information across different scales and enhance texture features accordingly. Tested on three public CD datasets, the effectiveness of the proposed FADG-Net is verified, where pseudo-change problem is mitigated and our method is superior to other comparison methods.

With the flourishing of remote sensing (RS) platform techniques, very high-resolution (VHR) images have become more and more popular in recent years, which benefit the task of semantic segmentation but bring new challenges as well. Small objects, such as cars and trees, only occupy a few pixels in VHR images and are usually hard to segment. Moreover, the overlap problem about similar ground objects, such as low vegetation and trees, always results in underperformance. In this article, a content-guided and class-oriented network (CGCO-Net) for VHR image semantic segmentation is proposed to tackle this problem. Specifically, an adaptive content-guided fusion (ACGF) module with deformable convolution is introduced to capture long-distance dependencies and spatial aggregation effectively. With the guidance of the high-level features, the semantic content knowledge is gradually aggregated into low-level features and the details of the original features could be preserved. In addition, a multiscale channel alignment module is introduced into the encoder–decoder structure to further extract the long-range context information and reduce the calculation consumption. In order to improve the ability of pixel-level classification, a class-oriented representation learning (CORL) way is designed with transformer blocks by class embedding and deep supervision, which gradually enhance the discrimination and benefit the final segmentation. Furthermore, a weighted loss function and a threshold optimization strategy are employed to alleviate the sample imbalance problem. Tested on three public datasets and compared with several state-of-the-art methods, the proposed CGCO-net achieves good performance in both qualitative and quantitative analysis.

Recovering the latent hyperspectral image (HSI) from RGB or multispectral image (MSI), which is dubbed spectral super-resolution (SSR), has demonstrated outstanding performance owing to the advancements in convolutional neural networks (CNNs). However, most of the current algorithms concentrate on the pursuit of networks with more expensive or complex structures, while ignoring the significant role of physical degradation models in SSR. In addition, the inherent defects of CNN make these networks focus more on the local correlation, while their ability to model the long-range correlations in the spectral and spatial domains still has room to improve. To overcome this shortcoming, we propose a physical degradation-guided deep prior learning network (PGDL-Net) for SSR via unfolding the optimization process of the blind SSR model, in which the priors of unknown spectral response function (SRF) and latent HSI are learned explicitly and represented by proximal operators. To jointly extract the local and non-local information, we design a hybrid graph Transformer as the proximal operator to solve the latent HSI. Furthermore, to ensure efficient learning of SRF and HSI, we also propose a novel loss function constraining the reconstruction error, degradation consistency, and observation fidelity for the learned SRF and HSI. Experimental results on multiple datasets illustrate the improved performance and stability of our method in SSR.

Existing mainstream MOT methods can be categorised into two frameworks including two-stage and one-stage ones. Two-stage ones divide MOT task into object detection and association tasks which usually achieve high accuracy. One-stage ones train a joint model to achieve both detection and tracking. So their advantage usually lies in the high tracking efficiency. In this paper, we inherit the advantages of the two types frameworks and propose the object knowledge distilled joint detection and tracking framework (OKD-JDT) to achieve accurate as well as efficient tracking. Firstly, the performance of two-stage methods largely depends on the highly performed detection network. So, we treat the detection network as the teacher network to guide the discriminative object feature learning in one-stage methods by using knowledge distillation. Then, in distillation learning, we design the adaptive attention learning to learn the discriminative features from teacher network to student network. In addition, with the similar appearance and uniform moving behaviour of objects in satellite videos, we propose to use joint center point distance and intersection-over-onion (IOU) to generate tracklets. Experiments on JiLin-1 satellite videos with different objects demonstrate the effectiveness and the state-of-the-art performance of the proposed method.

Remote sensing change detection (RSCD), with a view to probing surface changes between bi-temporal images, makes a spurt of progress with the continuous innovation of deep learning. However, the extraction of multi-scale features and the detection of small domain of variation as well as the detail information in RSCD task still has large development space. Besides, current existing methods mostly focus on learning regional information but pay less regard to boundary identification, which leads to inaccurate detection results. Therefore, a spatial-preserving and edge-orienting high-resolution network is proposed to address the problems. In the overall architecture, a dual-branch encoder consists of a pyramid feature extracted branch and an enhanced HR network branch is designed to extract muti-scale bi-temporal features and small change objectives, while two edge-orienting modules (EOM) are embedded in order to utilize edge prior knowledge for further improving the accuracy of change detection. Moreover, spatial-preserving module (SPM) based on the self-attention calculation in spatial dimension is applied in the pyramid part to alleviate the poor location information of the high-level features. The experimental results demonstrate that the proposed network outperforms the cited state-of-the-art methods on LEVIR change detection datasets (LEVIR-CD).

There is a small sample problem in hyperspectral image (HSI) classification task due to the difficulty of labeling samples. It is generally solved using a combination of few-shot learning and cross-domain method. In the paper, we propose a domain-specific and domain-common feature enhancement method for cross-domain few-shot HSI classification. It consists of a domain adaptation module and a feature enhancement module. The former is used to learn domain-specific features of both domains from the beginning of the network, and the latter is used to reduce domain differences by learning domain-common features through feature enhancement. The experimental results indicate that our proposed method performs better than the advanced classification methods.

Semantic segmentation of very high spatial resolution remote sensing images plays a vital role in many fields, such as land resource management, urban planning, and biosphere monitoring. Due to the scare variance between different types of regions, it is important to fully utilize multi-scale features. Moreover, with the complexity of some ground objects, global semantic information should be specially considered. As a consequence, in this paper, we propose a stair fusion network to further refine and fuse low-level and high-level features. In addition, we propose a global information enhancement module (GIEM) to extract global semantic information from the high-level features and reduce the length of delivery chain from them to the final results via a skip connection. Experimental results demonstrate the effectiveness of our model.

The domain shift between the source and target domains limits the performance of traditional convolutional neural networks (CNNs) for feature extraction in remote sensing tasks. We propose an image translation network that uses generative adversarial networks (GANs) to transfer spectral distributions from training to test data, enhancing cross-domain semantic segmentation. Our approach fine-tunes the DeepLab-V3 framework on synthetic training data generated by the proposed network. Experimental results show improved performance in cross-domain semantic segmentation tasks for remote sensing images.

Remote sensing change detection (CD) is of great importance to Earth observation. Recently, Deep Learning (DL) has been increasingly used to extract useful features and make accurate decisions in a large number of remote sensing images, due to its ability to automatically learn semantic features. However, insufficient fusion of bitemporal images and the lack of prior knowledge of edge structures in current DL methods will result in inaccurate CD results, especially for building boundaries. To alleviate these problems, an edge-guided feature-densely-fused network (EGFDFN) is proposed in this paper. In contrast to conventional Siamese networks, EGFDFN extracts bitemporal features from an extra dual decoder instead of a dual encoder to obtain more accurate change features. In addition, an attention and dense fusion module (ADFM) and an edge guidance module (EGM) are used to enhance features and make full use of edge information. Experimental results demonstrate that the proposed method outperforms on LEVIR-CD dataset among other representative methods.

Hyperspectral image change detection (HSI-CD) is one of the main research topics in remote sensing. Theoretically, the ground objects can be considered to have changed when their spectral features behave differently. However, in practical scenarios, this presupposition does not hold as the collected features are affected by many factors, such as illumination conditions, atmospheric effects, and topographic changes. Inspired by the intrinsic image decomposition, we propose a novel unsupervised deep learning framework based on two-stream coupled autoencoder (TSCA) to cope with bi-temporal co-registered HSI-CD. The network consists of two symmetric encoders and a decoder, which can jointly decompose bi-temporal images into abundance coefficients corresponding to the same set of spectral bases. As our network separates the component of spectral variation from multiple images, the extracted abundance features with inherent properties of materials can provide better performance for change detection. Moreover, to enforce alignment of the feature space, a reasonable consistency loss is devised to constrain the solution space, by cross-reconstruction in both branches. Experimental results demonstrate its superiority over the recently developed state of the arts.

During target tracking process, the state of the target is usually unpredictable. In theory, it is often beneficial to automatically assign suitable features to describe the specific target in each frame. Inspired by this, in this paper, we propose a novel dynamic feature-adaptive tracking framework (DFAT) which automatically assigns appropriate features to the consecutive frames during tracking process to boost the tracking performance. To implement DFAT, a large pool consisting of trackers/experts based on correlation filtering (CF) is constructed which is called candidate pool (CandPool). The diversity of the experts lies in their feature configurations and we call them candidate experts (CandExp). In this way, different features can be assigned for continuously changed scenarios and the target. Then to assign suitable experts, for each frame, we design the dynamic tracking process as the following three steps: (1) Several experts which are called executive experts (ExeExp) are selected from the CandPool according to CandExps’ past performance. (2) The ExeExps generate the tracking results and the performance of them are evaluated via a novel evaluation mechanism. (3) The selection rate of each CandExp in the CandPool is updated according to the performance evaluation and the final tracking result is selected. To better evaluate the CandExp, we propose two novel criteria: (1) content similarity weighted intra-evaluation, and (2) response confidence based self-evaluation. Compared with traditional post-event ensemble trackers that use fixed experts, the proposed method learns to dynamically assign appropriate ExeExps selected from a large CandPool which leads to adaption to different cases. Moreover, overfitting caused by fixed experts can also be mitigated via dynamic tracking. Experiments on both public available general and satellite videos based data sets demonstrate the superiority of the proposed method.

Hyperspectral image (HSI) classification is significant in remote sensing applications. However, most methods focus on the spectral and spatial correlation information in the neighborhood while ignoring the feature difference among global different pixels. In this article, we propose a spatial–spectral adaptive learning with pixelwise filtering (SSALPF) method to fully consider the discriminative information of pixels in different spatial locations, which mainly consists of a parallel spatial–spectral adaptive learning (SSAL) module and a pixelwise filtering (PF) module. Specifically, the former aims to obtain joint spatial–spectral discriminative features of each pixel point in a parallel manner and is used as a guide for adaptive selection of filter kernel. The latter uses the adaptive filter kernel to implement pixel-level filtering on HSI, in order to learn the discriminative features contained in different pixel points for classification. The adaptive filter kernel is generated by a linear combination of a predefined dictionary containing multiple filter bases. Experiments demonstrate that the proposed method is superior to other methods on popular hyperspectral datasets.

Heterogeneous bi-temporal images have different visual appearances and inconsistent data distribution for the same scene, making it challenging to detect changes, which need to align the shared information and reduce various unwanted sensor-related noises for comparability. Mainstream methods usually adopt two types of techniques: feature transformation and image translation. The former relies on handcrafted priors while the latter lacks constraints on unwanted backgrounds, leading to limitations such as a lack of robustness to non-intrinsic changes (e.g., seasonal and atmospheric changes, and sensor-related noise) and unsatisfactory detection performance. To overcome these drawbacks, we propose a novel unsupervised predict-refining-involved bidirectional contrastive difference network (PRBCD-Net) composed of a coarse prediction module and iterative refining modules. Each refining module utilizes feature extractors with a cross-reconstruction constraint and bidirectional contrastive constraint to extract discriminative features, and then generate a refined change map by change map optimizers. Two advantages of the proposed PRBCD-Net are: 1) the cross-reconstruction constraint is used to promote the feature distribution consistency of the bi-temporal images by using the forward and backward transformations; 2) the bidirectional contrastive constraint is used to improve the discriminability of features by narrowing the gap between non-intrinsic changes while widening intrinsic changes under the guidance of a coarse change map. Thus, the refining module can generate a finer change map than the coarse one, and the performance can be further improved through multiple iterations. Experimental results demonstrate the effectiveness and robustness of the proposed method compared with state-of-the-art methods.

Self-supervised contrastive learning (CL) can learn high-quality feature representations that are beneficial to downstream tasks without labeled data. However, most CL methods are for image-level tasks. For the fine-grained change detection (FCD) tasks, such as change or change trend detection of some specific ground objects, it is usually necessary to perform pixel-level discriminative analysis. Therefore, feature representations learned by image-level CL may have limited effects on FCD. To address this problem, we propose a self-supervised global–local contrastive learning (GLCL) framework, which extends the instance discrimination task to the pixel level. GLCL follows the current mainstream CL paradigm and consists of four parts, including data augmentation to generate different views of the input, an encoder network for feature extraction, a global CL head, and a local CL head to perform image-level and pixel-level instance discrimination tasks, respectively. Through GLCL, features belonging to different perspectives of the same instance will be pulled closer, while features of different instances will be alienated, which can enhance the discriminativeness of feature representations from both global and local perspectives, thereby facilitating downstream FCD tasks. In addition, GLCL makes a targeted structural adaptation to FCD, i.e., the encoder network is undertaken by the common backbone networks of FCD, which can accelerate the deployment on downstream FCD tasks. Experimental results on several real datasets show that compared with other parameter initialization methods, the FCD models pretrained by GLCL can obtain better detection performance.

Deep learning has significantly advanced the change detection in remote sensing image with its excellent performance. For change detection tasks, there are two critical issues. First, with scale variance of different objects in remote sensing images, effectively aggregating multi-scale features helps to generate fine-grained change objects. Second, it is critical but challenging to fully exploit the variance information between bi-temporal images to avoid pseudo-variation and region blurring. To alleviate the above issues, this paper proposes an efficient multi-dimensional attention-aggregation network (MANet), which keeps better feature aggregation while maintaining excellent differential attention ability. This paper carries three main contributions. First, we propose a multiscale asymmetric convolutional attention (MACA) module. Due to the asymmetric convolution’s ability to focus on feature contours effectively, the MACA can not only aggregate multi-scale features effectively, but also refine the edge information of features. Second, we propose a dual-dimensional attention (DDA) module for adaptively fusing shallow and deep features, which is used to generate rich feature representations. Third, the difference guidance (DG) module is exploited for enhancing the attention of changed regions to mitigate the influence of uncorrelated changes on the change detection result. Experiments on four popular change detection datasets show that our network can accomplish higher detection accuracy than the state-of-the-art networks.

Recently, deep learning-based hyperspectral image (HSI) classification techniques are flourishing and exhibit good performance, where cross domain information is usually utilized to reduce the dependency on large labeled samples. However, the gap between source domain and target domain makes it difficult to carry out knowledge transfer directly. In this paper, an adversarial domain alignment with contrastive learning method is designed for the HSI classification task to achieve feature consistency that benefits transferring knowledge. In details, spectral alignment and semantic alignment are conducted in local and global levels respectively in an adversarial learning way, and the adversarial loss acts on both source and target domains. In order to learn specific features for objects with different spatial scales, a multi-scale selection module is constructed in semantic alignment to select channel features adaptively. Moreover, contrastive learning is employed to increase both robustness and sensitiveness, where augmented data from the same/different samples are forced to be similar/dissimilar with each other. The training process is conducted in a few-shot learning way then the few-shot classification loss, the adversarial loss and the contrastive loss is optimized together. Tested on one source dataset and four target datasets, the experimental results show that the proposed method outperforms the other comparisons.

Few-shot learning (FSL) is an effective method to solve the problem of hyperspectral image (HSI) classification with few labeled samples. It learns transferable knowledge from sufficient labeled auxiliary data to classify unseen classes with limited labeled samples for training. However, the distribution difference between auxiliary data and unseen classes results in the learned transferable knowledge not being well applied to the new task. Therefore, a class-wise attentive cross-domain FSL (CA-CFSL) framework is proposed in this article, in which a feature extractor is learned to extract data features with discriminability and domain invariance. The class-wise attention metric module (CAMM) introduces class-wise attention on the FSL framework to learn more discriminative features, which improves the interclass decision boundaries. Furthermore, an asymmetric domain adversarial module (ADAM) is designed to enhance the ability of extracting domain-invariant representations, which combines asymmetric adversarial training with embedded domain-specific information. Experimental results on four public HSI datasets demonstrate that the proposed method outperforms the existing methods.

Change detection based on heterogeneous images is challenging because of the distribution variance caused by imaging properties of different types of sensor. Most methods deal with this problem by transforming features into a common space. However, the lack of available labeled data limits the training of complex models and representation of heterogeneous distributions. In this paper, we propose to train a network via abundant unlabeled data by adopting cyclic adversarial pre-training in order to learn the relationship between heterogeneous distributions. After pre-training, for change detection, we introduce a constraint to maintain consistency of image content, to avoid the participation of changed pixels in training. Experiments on heterogeneous optical and SAR images prove the effectiveness of our proposed method.

Change detection (CD) of remote sensing images is to detect the change region by analyzing the difference between two bitemporal images. It is extensively used in land resource planning, natural hazards monitoring and other fields. In our study, we propose a novel Siamese neural network for change detection task, namely Dual-UNet. In contrast to previous individually encoded the bitemporal images, we design an encoder differential-attention module to focus on the spatial difference relationships of pixels. In order to improve the generalization of networks, it computes the attention weights between any pixels between bitemporal images and uses them to engender more discriminating features. In order to improve the feature fusion and avoid gradient vanishing, multi-scale weighted variance map fusion strategy is proposed in the decoding stage. Experiments demonstrate that the proposed approach consistently outperforms the most advanced methods on popular seasonal change detection datasets.

Deep learning based semantic segmentation is one of the popular methods in remote sensing image segmentation. In this paper, a network based on the widely used encoder-decoder architecture is proposed to accomplish the synthetic aperture radar (SAR) images segmentation. With the better representation capability of optical images, we propose to enrich SAR images with generated optical images via the generative adversative network (GAN) trained by numerous SAR and optical images. These optical images can be used as expansions of original SAR images, thus ensuring robust result of segmentation. Then the optical images generated by the GAN are stitched together with the corresponding real images. An attention module following the stitched data is used to strengthen the representation of the objects. Experiments indicate that our method is efficient compared to other commonly used methods.

Change detection (CD) plays an increasingly important role in earth observation and reveals surface changes according to multi-temporal images. Although deep learning-based CD methods work well for their excellent modeling ability, objects in different size and shape are generally processed by the same filter kernels in feature extraction, which leads to spatial blurring and degrades the CD performance. In this paper, a spatial adaptive and feature enhanced (SAFE) siamese network is proposed to tackle this problem, where the SAFE consists of a spatial-adaptive (SA) part and a feature-enhanced (FE) part. Specifically, pixel belonging to different objects possesses its own spatial knowledge, which is captured by a soft fusion of multi-scale difference images (DIs) called SA part. Changed and unchanged areas are strengthened or weakened by the FE, which combines object features with each DI accordingly. Moreover, since there are more unchanged pixels than changed pixels, a weight-pair is introduced to balance changed and unchanged objects in the training process. The experimental results verify that compared with four representative CD algorithms, our proposed method performs best on the Change Detection Dataset (CDD).

Deep learning for change detection can provide effective guidance in many applications, such as agricultural development, urban planning, disaster avoidance, etc. In this study, a Siamese deep learning network based on High-Resolution Network (HRNet) is proposed to generate accurate results. HRNet can integrate multi-dimensional features and output high-resolution results which have attracted attention due to its reliable feature extraction ability. In this paper, we extract the feature pairs of several different dimensions, including the two features behind the down-sampling in the stem stage which is an important part of HRNet. Moreover, feature ex-traction and intensive up-sampling tasks are completed by using a variety of feature fusion sub-networks, which are used to enhance the learning ability. Experiments show the superiority of the proposed Siamese HRNet on a widely used change detection dataset.

Recently, hyperspectral image (HSI) classification by deep learning is flourishing. However, only a few labeled samples are available in practice since it is time-and-labor-consuming to label pixels in HSI (called target domain). This paper proposes a domain-adaptive few-shot learning (DAFSL) method to tackle this problem. Specifically, some other HSIs (called source domain) with large labeled samples are fully used as complementary information and a generative architecture is employed to adapt embedded features in source domain to that of target domain. We first perform domain adaptation with unsupervised learning. In details, the embedded features are generated by the encoder of an autoencoder, where both source and target samples could be well recovered and the reconstruction loss is used to measure the gap between source domain and target domain. At the same time, the embedded features are put into a metric space for classification in source domain and the encoder parameter is fine-tuned together with the classifier in target domain with few labels, so that both general and discriminative features are well captured. The experiment results show that DAFSL outperforms the other mainstream methods with limited labeled samples.

Deep learning-based image inpainting has been widely studied, leading to great success. However, many methods adopt convolution and normalization operations, which will bring up some issues to affect the performance. The vanilla normalization cannot distinguish the pixels in corrupted regions from the other valid pixels, resulting in the mean and variance shifts. In addition, the limited receptive field of convolution makes it unable to capture long-range valid information directly. In order to tackle these challenges, we propose a novel deep generative model for image inpainting with two key modules, namely, the channel and spatially adaptive batch normalization (CSA-BN) module, and the selective latent-space-mapping-based contextual attention (SLSM-CA) layer. We replace the vanilla normalization with the CSA-BN module. By channel and spatially adaptive denormalization, the CSA-BN module can mitigate the spatial mean and variance shifts in each channel in a targeted way. In addition, we also integrate the SLSM-CA layer into our model to capture the long-range correlations explicitly. By introducing dual-branch attention and a feature selection module, the SLSM-CA layer can selectively utilize the multi-scale background information to improve prediction quality. What’s more, it introduces the latent spaces to achieve the low-rank approximations of attention matrices and to reduce computational costs. Extensive quantitative and qualitative evaluations demonstrate the superiority of the proposed method compared with state-of-the-art methods.

Artificial neural networks inspired from the learning mechanism of the brain have achieved great successes in machine learning, especially those with deep layers. The commonly used neural networks follow the hierarchical multilayer architecture with no connections between nodes in the same layer. In this article, we propose a new group architectures for neural-network learning. In the new architecture, the neurons are assigned irregularly in a group and a neuron may connect to any neurons in the group. The connections are assigned automatically by optimizing a novel connecting structure learning probabilistic model which is established based on the principle that more relevant input and output nodes deserve a denser connection between them. In order to efficiently evolve the connections, we propose to directly model the architecture without involving weights and biases which significantly reduce the computational complexity of the objective function. The model is optimized via an improved particle swarm optimization algorithm. After the architecture is optimized, the connecting weights and biases are then determined and we find the architecture is robust to corruptions. From experiments, the proposed architecture significantly outperforms existing popular architectures on noise-corrupted images when trained only by pure images.

Detecting changes in complicated remote sensing images have been gaining much attention. One of the main challenges lies in how to detect intrinsic changes robustly while avoiding the false alarms caused by various challenging factors, such as spatial illumination variations, small viewpoint differences, noises and outliers between multi-temporal remote sensing images. To reduce the influence of these factors, in this paper, we propose an unsupervised joint learning model based on a total variation regularization and bipartite deep convolutional neural network, called total variation regularized bipartite network (TVRBN). In this model, parametric feature differences are initialized by a bipartite autoencoder. An objective function is defined as a parametric feature difference term integrated with a change pre-detection constraint term and a total variation regularization to the change probability map with intrinsic changes. Then the unified objective function is optimized jointly to learn the bipartite network parameters and detect an intrinsic change map. Due to the intrinsic difference constraints in feature space and total variation regularization, the proposed TVRBN method can compute higher-quality and smoother change maps, suppress non-intrinsic changes, and overcome small viewpoint differences in comparable images. Extensive experiments on both homogeneous and heterogeneous images demonstrate the robustness of the proposed method by comparing it with state-of-the-art methods.

Remote sensing (RS) image change detection (CD) is an earth observation technique for detecting surface changes in the same area during a period. With the rapid development of deep learning, various deep neural networks especially Siamese ones have been widely used in the field of CD. However, they have the deficiency of insufficient contextual information aggregation, resulting in false and missed detections, and it is difficult to refine the detection of change edges. To alleviate these problems and obtain more accurate results, we propose an efficient self-weighted spatial-temporal attention network (SSANet). In contrast to the Siamese structure, our network is a novel joint learning framework composed of fusion sub-network, difference sub-network, and decoder. Fusion sub-network is used to extract multiscale object features where we propose a multi-core channel-aligning attention (MCA) module to capture the long-range semantic information for multi-scale context aggregation. Difference sub-network is used to extract the difference variation features, where we propose a feature differential reconfiguration (FDR) module to learn the temporal change information. FDR can effectively filter change information and reconstruct features to improve the perception of changed regions. To better balance the MCA and FDR modules, an asymmetric weighting (AW) module is proposed in the decoder to self-weight the multi-scale features and generate the change map. Experiments demonstrate the efficiency of proposed sub-networks and modules, and the state-of-the-art performance of SSANet.

Polarimetric synthetic aperture radar (PolSAR) image classification is one of the hottest issues in remote sensing, where studies on pixel-level information and relationship are of great significance. In this article, graph convolutional network (GCN) is employed to accomplish this pixel-level task benefiting from its excellent capability in structure exploration and information propagation between different pixels. To reduce the communication burden between various PolSAR pixels and high computational cost for the whole PolSAR image, an adaptive GCN (AdapGCN) consisting of pixel-centered subgraphs is proposed in this article. In the AdapGCN, a data-adaptive kernel and a spatial-adaptive kernel are introduced to, respectively, model data structure and spatial structure for PolSAR image. Moreover, a multiscale learning structure is integrated to further explore complicated relations between pixels. Extensive comparative evaluations validate the superiority of our new AdapGCN model for PolSAR image classification over a wide range of state-of-the-art methods on three challenging benchmarks.

This article presents a probabilistic model based on a bipartite convolutional architecture for unsupervised change detection. We aim to develop a robust change detection method that can adapt to different types of data and scenarios for multitemporal coregistered remote sensing images of the same spatial resolution. On the premise of coregistration, unsupervised change detection usually suffers from the distinct appearances (different intensities or data structures) of the same object in multitemporal images, such as images obtained in different climatic conditions (season, illumination, and so on), and by different and even heterogeneous sensors. Since change detection in heterogeneous images can also adapt to other scenarios, many methods have been proposed recently focusing on such data, but most of them are limited by the need for labeled data or by specific assumptions. With the excellent and flexible feature learning capability of neural networks, we model the change detection into a Gibbs probabilistic model based on a bipartite neural network. The model is driven by an energy function defined as the squared feature distance, which is the core of change detection. Via optimizing the model, the difference degree of each pixel is automatically obtained for further identification. The probabilistic model learns to capture the distribution in an unsupervised way. Therefore, the proposed method can adapt to various scenarios without being trained by labeled data. Experiments on different types of data and scenarios demonstrate the superiority of the proposed method.

Recently, many state-of-the-art results for remote sensing image scene classification have been achieved by convolutional neural networks (CNNs) due to their large learning capability. However, in the forward process of CNNs, the high-frequency/texture features are gradually blurred with hierarchical down-sampling and convolution operations. High-frequency features are important to capture the diversity within a class and the similarity between classes. For example, the line features are crucial to distinguish a tennis court from a basketball court. For tennis court in different scenes, the highlight of line features can effectively avoid the influence of diverse background. As a consequence, we propose a Laplacian high-frequency convolutional block (LHCB) based on CNN to extract useful high-frequency features by trainable Laplacian operator. To propagate high-frequency features, we embed LHCB into the existing CNN structures and obtain LHNet. In LHNet, there are two pathways. The original CNN architecture can be taken as the low-frequency pathway and we propose a high-frequency pathway based on LHCB that propagates the residual high-frequency features blurred in each low-frequency layer. Considering that the high-frequency features usually show large variance between images of the same class, we propose a new objective for high-frequency pathway to enhance the intra-class similarity of high-frequency features. The final objective function is obtained by combining the new objective and the baseline classification objective. Numerous experiments on three public available remote sensing image scene classification data sets NWPU-RESISC45, AID and UC Mercerd demonstrate the superior performance of the proposed method.

Recently, satellite videos provide a new way to dynamically monitor the Earth’s surface. The interpretation of satellite videos has attracted more and more attentions. In this article, we focus on the problem of the vehicle tracking in satellite videos. Satellite videos usually own a lower resolution, which leads to the following phenomena: 1) the size of a vehicle target usually includes a few pixels and 2) vehicles are usually with similar appearance which easily results in the wrong tracking within the observing region. General popular tracking methods usually focus on the representation of the target and recognize it from background which are limited in this problem. As a consequence, in this article, we propose to learn motion and background of the target in order to help the trackers recognize the target with higher accuracy. A prediction network is proposed to predict the location probability of the target in each pixel in next frame based on fully convolutional network (FCN) which is learned from previous results. In addition, a segmentation method is introduced to generate the feasible region for target in each frame and assign high probability for such a region. For quantitative comparison, we manually annotate 20 representative vehicle targets from nine satellite videos taken by JiLin-1. In addition, we also selected two public satellite video datasets for experiments. Numerous experimental results demonstrate the superior of the proposed method.

In this article, we propose a sparse feature clustering network (SFCNet) for change detection in synthetic aperture radar (SAR) images. One of the principal problems in dealing with SAR images is to reduce the impact of speckle noise. Therefore, based on a neural network framework for change detection, we introduce the multiobjective sparse feature learning (MO-SFL) model where the sparsity of representation is adaptively learned in order to increase the robustness to different levels of noise. For learning the semantic information of changed and unchanged pixels, the network is fine-tuned by the correctly labeled samples selected from coarse results. The selection criterion influences the change detection result a lot. Therefore, we construct a novel cross-entropy clustering loss (CEC) by introducing a clustering regularization term to learn the discriminative representations. Experiments on simulate and real SAR images demonstrate the superiority of the proposed method over compared methods.

In this paper, a novel unsupervised change detection method called adaptive Contourlet fusion clustering based on adaptive Contourlet fusion and fast non-local clustering is proposed for multi-temporal synthetic aperture radar (SAR) images. A binary image indicating changed regions is generated by a novel fuzzy clustering algorithm from a Contourlet fused difference image. Contourlet fusion uses complementary information from different types of difference images. For unchanged regions, the details should be restrained while highlighted for changed regions. Different fusion rules are designed for low frequency band and high frequency directional bands of Contourlet coefficients. Then a fast non-local clustering algorithm (FNLC) is proposed to classify the fused image to generate changed and unchanged regions. In order to reduce the impact of noise while preserve details of changed regions, not only local but also non-local information are incorporated into the FNLC in a fuzzy way. Experiments on both small and large scale datasets demonstrate the state-of-the-art performance of the proposed method in real applications.

Face is one of the most attractive sensitive information in visual shared data. It is an urgent task to design an effective face deidentification method to achieve a balance between facial privacy protection and data utilities when sharing data. Most of the previous methods for face deidentification rely on attribute supervision to preserve a certain kind of identity-independent utility but lose the other identity-independent data utilities. In this article, we mainly propose a novel disentangled representation learning architecture for multiple attributes preserving face deidentification called replacing and restoring variational autoencoders (R2VAEs). The R2VAEs disentangle the identity-related factors and the identity-independent factors so that the identity-related information can be obfuscated, while they do not change the identity-independent attribute information. Moreover, to improve the details of the facial region and make the deidentified face blends into the image scene seamlessly, the image inpainting network is employed to fill in the original facial region by using the deidentified face asa priori. Experimental results demonstrate that the proposed method effectively deidentifies face while maximizing the preservation of the identity-independent information, which ensures the semantic integrity and visual quality of shared images.

A new method based on the Looking-Around-and-Into (LAaI) mode is proposed for the task of change detection in large-scope Polarimetric Synthetic Aperture Radar (PolSAR) image. Specifically, the LAaI mode consists of two processes named Look-Around and Look-Into, which are accomplished by attention proposal network (APN) and recurrent convolutional neural network (CNN) (Recurrent CNN), respectively. The former provides certain subregions efficiently, and the latter detects changes in subregions accurately. In Look-Around, difference image (DI) of whole PolSAR images is calculated first to get global information; then, APN is established to locate the position of interested subregions intentionally by paying special attention to; next interested subregions that contain changed area in high probability are picked out as candidate-regions. Moreover, candidate-regions are sorted in importance descending order so that highly interested regions have priority to be detected. In Look-Into, candidate-regions of different scales are selected at first; then, Recurrent CNN is constructed and employed to deal with multiscale PolSAR subimages so that clearer and finer change detection results are generated. The process is repeated until all candidate-regions are detected. As a whole, the proposed algorithm based on the LAaI mode looks around whole images first to find out the possible position of changes (candidate-regions generation in Look-Around) and then reveal the exact shape of changes in different scales (multiscale change detection in Look-Into). The effect of APN and Recurrent CNN is verified in experiments, and it shows that the proposed method performs well in the task of change detection in the large-scope PolSAR image.

-norm based sparse response regularization term to restrain unexpected crests in response for CF framework. CF trackers learn online to regress the region of interest into a Gaussian response. However, due to the uncertain transformations of tracked object, there are many unexpected crests in the response map. When the response of tracked object is corrupted by other crests, the tracker will lost the object. Therefore, the sparse response is used to increase the robustness to transformations of tracked object. Since the novel term is directly incorporated into the objective function of the CF framework, it can be used to improve the performance of many methods which are based on this framework. Moreover, from the solutions we derive, the new method will not increase the computational complexity. Through the experiments on benchmarks of OTB-100, TempleColor, VOT2016 and VOT2017, the proposed regularization term can improve the tracking performance of various CF trackers, including those based on standard discriminative CF framework and those based on context-aware CF framework. We also embed the sparse response regularization term in the state-of-the-art integrated tracker MCCT to test its generalization performance. Although MCCT is an expert integrated tracker and owns an exquisite algorithm for selecting experts, the experimental results show that our method can still improve its long-term tracking performance without increasing computational complexity.

Deep neural networks (DNNs), characterized by sophisticated architectures capable of learning a hierarchy of feature representations, have achieved remarkable successes in various applications. Learning DNN's parameters is a crucial but challenging task that is commonly resolved by using gradient-based backpropagation (BP) methods. However, BP-based methods suffer from severe initialization sensitivity and proneness to getting trapped into inferior local optima. To address these issues, we propose a DNN learning framework that hybridizes CC-based optimization with BP-based gradient descent, called BPCC, and implement it by devising a computationally efficient CC-based optimization technique dedicated to DNN parameter learning. In BPCC, BP will intermittently execute for multiple training epochs. Whenever the execution of BP in a training epoch cannot sufficiently decrease the training objective function value, CC will kick in to execute by using the parameter values derived by BP as the starting point. The best parameter values obtained by CC will act as the starting point of BP in its next training epoch. In CC-based optimization, the overall parameter learning task is decomposed into many subtasks of learning a small portion of parameters. These subtasks are individually addressed in a cooperative manner. In this article, we treat neurons as basic decomposition units. Furthermore, to reduce the computational cost, we devise a maturity-based subtask selection strategy to selectively solve some subtasks of higher priority. Experimental results demonstrate the superiority of the proposed method over common-practice DNN parameter learning techniques.

This paper presents a novel bipartite unsupervised residual networks (ResNet) for change detection based on two heterogeneous images acquired by optical and radars sensors on different dates. Most previous change detection methods in the unsupervised field use the shallow network and detect image changes at the pixel level. The proposed method detects image changes at the feature level via a bipartite deep residual networks. The network compares the difference information of the two images by fully considering the joint information of multi-temporal images. The generated image features can well represent the different information between the two images. ResNet is used to learn features from images. A new loss function is proposed for training the ResNet. Experimental results of homogeneous and heterogeneous images show that our method has promising performance compared to existing change detection methods for optical and radar images.

In this paper, inspired from the nuclei in brain, we propose a nucleus neural network (NNN) and corresponding connecting architecture learning method. In a nucleus, the neurons are not assigned as regular layers, i.e., a neuron may connect to any neurons in the nucleus and the connections are self-organized according to data distribution. This type of architecture gets rid of layer limitation and makes full use of processing capability of each neuron. It is crucial to assign connections between all the neuron pairs. To address the time demanding of objectives in traditional architecture learning methods, we propose an efficient architecture learning model for the nucleus based on the principle that more relevant input and output neuron pair deserves higher connecting density. The new objective measures the information flow through the network architecture without involvement of weights and biases which greatly reduces the computational complexity. We find that this novel architecture is robust to irrelevant components in test data. So we reconstruct a new dataset based on the MNIST dataset where the types of digital backgrounds in training and test sets are different. The new dataset is a great challenge for learners because training data and test data are not only independent but also follow different distributions. Experiments demonstrate that NNN achieves significant improvement over architectures with regular layers on the reconstructed dataset.

Image change detection detects the regions of change in multiple images of the same scene taken at different times, which plays a crucial role in many applications. The two most popular image change detection techniques are as follows: pixel-based methods heavily rely on accurate image coregistration while object-based approaches can tolerate coregistration errors to some extent but are sensitive to image segmentation or classification errors. To address these issues, we propose an unsupervised image change detection approach based on a novel bipartite differential neural network (BDNN). The BDNN is a deep neural network with two input ends, which can extract the holistic features from the unchanged regions in the two input images, where two learnable change disguise maps (CDMs) are used to disguise the changed regions in the two input images, respectively, and thus demarcate the unchanged regions therein. The network parameters and CDMs will be learned by optimizing an objective function, which combines a loss function defined as the likelihood of the given input image pair over all possible input image pairs and two constraints imposed on CDMs. Compared with the pixel-based and object-based techniques, the BDNN is less sensitive to inaccurate image coregistration and does not involve image segmentation or classification. In fact, it can even skip over coregistration if the degree of transformation (due to the different view angles and/or positions of the camera) between the two input images is not that large. We compare the proposed approach with several state-of-the-art image change detection methods on various homogeneous and heterogeneous image pairs with and without coregistration. The results demonstrate the superiority of the proposed approach.

In this paper, we propose a multi-scale feature fusion network (MS-FF Net) based on convolutional neural network (CNN) to deal with object detection in VHR images. In CNN, the low-level layers contain rich detail information and the high-level layers contain rich semantic information. Inspired by the idea of feature fusion, we propose an additional multi-scale feature fusion layer (MFL) to fuse the information between detail and semantic features. Then both large and small objects are considered by this network. Moreover, the network architecture and training strategies are designed to improve performance. Experiments on NWPU VHR-10 dataset demonstrate that the method with MFLs achieves significant improvement and outperforms compared methods in terms of mean average precision. Specially, the detection precision of airplane, baseball diamond, basketball court, ground track field and harbor categories exceeds 90% which is much higher than that of compared methods.

Self-paced learning (SPL) is a recently proposed paradigm to imitate the learning process of humans/animals. SPL involves easier samples into training at first and then gradually takes more complex ones into consideration. Current SPL regimes incorporate a self-paced (SP) regularizer into the learning objective with a gradually increasing pace parameter. Therefore, it is difficult to obtain the solution path of the SPL regime and determine where to optimally stop this increasing process. In this paper, a multiobjective SPL method is proposed to optimize the loss function and the SP regularizer simultaneously. A decomposition-based multiobjective particle swarm optimization algorithm is used to simultaneously optimize the two objectives for obtaining the solutions. In the proposed method, a polynomial soft weighting regularizer is proposed to penalize the loss. Theoretical studies are conducted to show that the previous regularizers are roughly particular cases of the proposed polynomial soft weighting regularizer family. Then an implicit decomposition method is proposed to search the solutions with respect to the sample number involved into training. A set of solutions can be obtained by the proposed method and naturally constitute the solution path of the SPL regime. Then a satisfactory solution can be naturally obtained from these solutions by utilizing some effective tools in evolutionary multiobjective optimization. Experiments on matrix factorization and classification problems demonstrate the effectiveness of the proposed technique.

With the rapid increase of remote sensing images in temporal, spectral, and spatial resolutions, it is urgent to develop effective techniques for joint interpretation of spatial-temporal images. Multitype change detection (CD) is a significant research topic in multitemporal remote sensing image analysis, and its core is to effectively measure the difference degree and represent the difference among the multitemporal images. In this paper, we propose a novel difference representation learning (DRL) network and present an unsupervised learning framework for multitype CD task. Deep neural networks work well in representation learning but rely too much on labeled data, while clustering is a widely used classification technique free from supervision. However, the distribution of real remote sensing data is often not very friendly for clustering. To better highlight the changes and distinguish different types of changes, we combine difference measurement, DRL, and unsupervised clustering into a unified model, which can be driven to learn Gaussian-distributed and discriminative difference representations for nonchange and different types of changes. Furthermore, the proposed model is extended into an iterative framework to imitate the bottom-up aggregative clustering procedure, in which similar change types are gradually merged into the same classes. At the same time, the training samples are updated and reused to ensure that it converges to a stable solution. The experimental studies on four pairs of multispectral data sets demonstrate the effectiveness and superiority of the proposed model on multitype CD.

This paper focuses on the connecting structure of deep neural networks and proposes a layerwise structure learning method based on multiobjective optimization. A model with better generalization can be obtained by reducing the connecting parameters in deep networks. The aim is to find the optimal structure with high representation ability and better generalization for each layer. Then, the visible data are modeled with respect to structure based on the products of experts. In order to mitigate the difficulty of estimating the denominator in PoE, the denominator is simplified and taken as another objective, i.e., the connecting sparsity. Moreover, for the consideration of the contradictory nature between the representation ability and the network connecting sparsity, the multiobjective model is established. An improved multiobjective evolutionary algorithm is used to solve this model. Two tricks are designed to decrease the computational cost according to the properties of input data. The experiments on single-layer level, hierarchical level, and application level demonstrate the effectiveness of the proposed algorithm, and the learned structures can improve the performance of deep neural networks.

We propose an unsupervised deep convolutional coupling network for change detection based on two heterogeneous images acquired by optical sensors and radars on different dates. Most existing change detection methods are based on homogeneous images. Due to the complementary properties of optical and radar sensors, there is an increasing interest in change detection based on heterogeneous images. The proposed network is symmetric with each side consisting of one convolutional layer and several coupling layers. The two input images connected with the two sides of the network, respectively, are transformed into a feature space where their feature representations become more consistent. In this feature space, the different map is calculated, which then leads to the ultimate detection map by applying a thresholding algorithm. The network parameters are learned by optimizing a coupling function. The learning process is unsupervised, which is different from most existing change detection methods based on heterogeneous images. Experimental results on both homogenous and heterogeneous images demonstrate the promising performance of the proposed network compared with several existing approaches.

This paper presents a novel neuron learning machine (NLM) which can extract hierarchical features from data. We focus on the single-layer neural network architecture and propose to model the network based on the Hebbian learning rule. Hebbian learning rule describes how synaptic weight changes with the activations of presynaptic and postsynaptic neurons. We model the learning rule as the objective function by considering the simplicity of the network and stability of solutions. We make a hypothesis and introduce a correlation based constraint according to the hypothesis. We find that this biologically inspired model has the ability of learning useful features from the perspectives of retaining abstract information. NLM can also be stacked to learn hierarchical features and reformulated into convolutional version to extract features from 2-dimensional data.

Endmember extraction is a critical step of spectral unmixing. In this paper, a novel endmember extraction algorithm based on evolutionary multi-objective optimization is proposed for hyperspectral remote sensing images. In the proposed method, endmember extraction is modeled as a multi-objective optimization problem. Then the root mean square error between the original image and its remixed image and the number of endmembers are chosen as two conflicting objective functions, which are simultaneously optimized by particle swarm optimization algorithm to find the best tradeoff solutions. In order to promote diversity and speed up the convergence of the algorithm, a new particle status updating strategy and a novel method for selecting leaders are designed. The experimental results on both simulated and real hyperspectral remote sensing images confirm the performance of the proposed approach over some existing methods.

Band selection is a crucial preprocessing step for hyperspectral image classification, which is a classic feature selection method. Feature selection is designed to select feature subsets to represent the whole feature space. For feature selection, two crucial issues need to be handled: preserving information and redundancy reducing. In this paper, a novel feature selection method for hyperspectral image classification is proposed, which is based on a newly designed memetic algorithm. In the proposed method, a suitable objective function is designed, which can measure the contained crucial information and redundancy information in the selected feature subsets. To optimize this objective function efficiently, a novel memetic algorithm is designed. The genetic operator and local search strategy are newly designed according to the characteristic of hyperspectral images. Experiments are implemented on three real data sets compared with some state of arts. The experimental results show that the proposed method can obtain stable and superior feature subsets for classification.

This paper presents to model the Hebb learning rule and proposes a neuron learning machine (NLM). Hebb learning rule describes the plasticity of the connection between presynaptic and postsynaptic neurons and it is unsupervised itself. It formulates the updating gradient of the connecting weight in artificial neural networks. In this paper, we construct an objective function via modeling the Hebb rule. We make a hypothesis to simplify the model and introduce a correlation based constraint according to the hypothesis and stability of solutions. By analysis from the perspectives of maintaining abstract information and increasing the energy based probability of observed data, we find that this biologically inspired model has the capability of learning useful features. NLM can also be stacked to learn hierarchical features and reformulated into convolutional version to extract features from 2-dimensional data. Experiments on single-layer and deep networks demonstrate the effectiveness of NLM in unsupervised feature learning.

Change detection and analysis (CDA) is an important research topic in the joint interpretation of spatial-temporal remote sensing images. The core of CDA is to effectively represent the difference and measure the difference degree between bi-temporal images. In this paper, we propose a novel difference representation learning network (DRLnet) and an effective optimization framework without any supervision. Difference measurement, difference representation learning and unsupervised clustering are combined as a single model, i.e., DRLnet, which is driven to learn clustering-friendly and discriminative difference representations (DRs) for different types of changes. Further, DRLnet is extended into a recurrent learning framework to update and reuse limited training samples and prevent the semantic gaps caused by the saltation in the number of change types from over-clustering stage to the desired one. Experimental results identify the effectiveness of the proposed framework.

Sparse unmixing of hyperspectral data is an important technique aiming at estimating the fractional abundances of the end members. Traditional sparse unmixing is faced with the l0-norm problem which is an NP-hard problem. Sparse unmixing is inherently a multiobjective optimization problem. Most of the recent works combine cost functions into single one to construct an aggregate objective function, which involves weighted parameters that are sensitive to different data sets and difficult to tune. In this paper, a novel multiobjective cooperative coevolutionary algorithm is proposed to optimize the reconstruction term, the sparsity term and the total variation regularization term simultaneously. A problem-dependent cooperative coevolutionary strategy is designed because sparse unmixing encounters a large scale optimization problem. The proposed approach optimizes the nonconvex l0-norm problem directly and can find a better compromise between two or more competing cost function terms automatically. Experimental results on simulated and real hyperspectral data sets demonstrate the effectiveness of the proposed method.

With the application requirement, the technique for change detection based on heterogeneous remote sensing images is paid more attention. However, detecting changes between two heterogeneous images is challenging as they cannot be compared in low-dimensional space. In this paper, we construct an approximately symmetric deep neural network with two sides containing the same number of coupled layers to transform the two images into the same feature space. The two images are connected with the two sides and transformed into the same feature space, in which their features are more discriminative and the difference image can be generated by comparing paired features pixel by pixel. The network is first built by stacked restricted Boltzmann machines, and then, the parameters are updated in a special way based on clustering. The special way, motivated by that two heterogeneous images share the same reality in unchanged areas and retain respective properties in changed areas, shrinks the distance between paired features transformed from unchanged positions, and enlarges the distance between paired features extracted from changed positions. It is achieved through introducing two types of labels and updating parameters by adaptively changed learning rate. This is different from the existing methods based on deep learning that just do operations on positions predicted to be unchanged and extract only one type of labels. The whole process is completely unsupervised without any priori knowledge. Besides, the method can also be applied to homogeneous images. We test our method on heterogeneous images and homogeneous images. The proposed method achieves quite high accuracy.

Multifactorial evolutionary algorithm is used to deal with multifactorial optimization problem which simultaneously optimizes multiple tasks. In this paper, we introduce particle swarm optimization operation into the multifactorial evolutionary algorithm, and propose a hybrid algorithm for multifactorial optimization. The major aim is to utilize particle swarm optimization operation to accelerate the convergence and improve the accuracy of solutions. Experimental comparisons between the proposed hybrid algorithm and the original multi-factorial evolutionary algorithm show that the particle swarm update operators can effectively accelerate the convergence on some benchmark problems.

With the increase of multisource data available from remote sensing platforms, it is demanding to develop unsupervised techniques for change detection from multisource data. The difference in imaging mechanism makes it difficult to carry out a direct comparison between multisource data in original observation spaces. Different sensors provide different descriptions on the same truth in low-dimension observation spaces, but the same truth indicates the comparability of multisource data in some high-dimensional feature spaces. Inspired by this, we try to solve this problem by transforming multisource data into a common high-dimension feature space. In this paper, an iterative coupled dictionary learning (CDL) model is proposed for multisource image change detection. This model aims to establish a pair of coupled dictionaries, one of which is responsible for the data from one sensor, whereas the other is responsible for the data from another sensor. The atoms from these two coupled dictionaries have a one-to-one correspondence at the same location. Such a property guarantees the transferability of the reconstruction coefficients between bitemporal patch pairs and provides us a desired mechanism to bridge multisource data and highlight changes. The contributions can be summarized as follows: CDL is designed to explore the intrinsic difference of multisource data for change detection in a high-dimension feature space, and an iterative scheme for unsupervised sample selection is proposed to keep the purity of training samples and gradually optimize the current coupled dictionaries. The experimental results have demonstrated the feasibility, effectiveness, and robustness of the proposed framework.

This paper presents a novel change detection approach for synthetic aperture radar images based on deep learning. The approach accomplishes the detection of the changed and unchanged areas by designing a deep neural network. The main guideline is to produce a change detection map directly from two images with the trained deep neural network. The method can omit the process of generating a difference image (DI) that shows difference degrees between multitemporal synthetic aperture radar images. Thus, it can avoid the effect of the DI on the change detection results. The learning algorithm for deep architectures includes unsupervised feature learning and supervised fine-tuning to complete classification. The unsupervised feature learning aims at learning the representation of the relationships between the two images. In addition, the supervised fine-tuning aims at learning the concepts of the changed and unchanged pixels. Experiments on real data sets and theoretical analysis indicate the advantages, feasibility, and potential of the proposed method. Moreover, based on the results achieved by various traditional algorithms, respectively, deep learning can further improve the detection performance.

In this letter, a statistical active contour model exploiting the local information is proposed for classifying changed and unchanged regions in the difference image. It incorporates the local information and fuzzy logic for the purpose of enhancing the changed information and of reducing the effect of speckle noise. The fuzzy length term and the fuzzy penalty term are added into the fuzzy energy function. In particular, in order to avoid deriving the complex fuzzy membership updating function, we solve the Euler-Lagrange equation to minimize the fuzzy energy function instead of calculating the fuzzy energy alterations directly. The experimental results on three synthetic aperture radar images confirm the performance of the proposed model over some existing methods.

Hierarchical deep neural networks are currently popular learning models for imitating the hierarchical architecture of human brain. Single-layer feature extractors are the bricks to build deep networks. Sparse feature learning models are popular models that can learn useful representations. But most of those models need a user-defined constant to control the sparsity of representations. In this paper, we propose a multiobjective sparse feature learning model based on the autoencoder. The parameters of the model are learnt by optimizing two objectives, reconstruction error and the sparsity of hidden units simultaneously to find a reasonable compromise between them automatically. We design a multiobjective induced learning procedure for this model based on a multiobjective evolutionary algorithm. In the experiments, we demonstrate that the learning procedure is effective, and the proposed multiobjective model can learn useful sparse features.

In this paper, we present a new Markov Random Field based FCM image segmentation algorithm. A new energy function is proposed to utilize the spatial and contextual information simultaneously. In the proposed energy function, we use a weighted distance to reflect the different effects of neighborhood pixels. By using the new energy function, the new algorithm has a better performance in noise-corrupted images. Experimental results on real and synthetic images show our method is effective.

Image change detection is a process to analyze multi-temproal images of the same scene for identifying the changes that have occurred. In this paper, we propose a novel difference image analysis approach based on deep neural networks for image change detection problems. The deep neural network learning algorithm for classification includes unsupervised feature learning and supervised fine-tuning. Some samples with the labels of high accuracy obtained by a pre-classification are used for fine-tuning. Since a deep neural network can learn complicated functions that can represent high-level abstractions, it can obtain satisfactory results. Theoretical analysis and experiment results on real datasets show that the proposed method outperforms some other methods.