Multivariate long-term time series forecasting is a challenging task, that analyzes historical data across multiple variables to predict future data. To solve this problem, we proposed a new ChannelMixer model that combines channel dependency and channel independency learning strategies. In channel dependency learning, we proposed an Adaptive Variable and Short-Term Temporal Feature Extraction block, which uses the convolutional neural network and Couple-Slice Transform to capture complex short-term interactions. Concurrently, a channel-independent Transformer learning is used to capture long-term periodic information across the intra-subsequences dimension of time series data. Additionally, a linear head with channel independency is included to directly predict long-term trends, ensuring a balance between short-term dynamics and long-term forecasting. Finally, we make the predictions by fusing local-term dynamics features and global trends. Extensive experiments show that ChannelMixer outperforms state-of-the-art models in predictive accuracy.

Change detection in synthetic aperture radar (SAR) images is a hot yet highly challenging task in remote sensing. Existing unsupervised SAR change detection methods often struggle with inherent speckle noise and insufficiently utilize pseudo-labels, particularly neglecting uncertain areas. In this paper, we propose a multilevel difference-enhanced denoise dual-branch network (MDDNet), comprising representation learning and change detection branches. First, fuzzy c-means clustering is employed to generate pseudo-labels, categorizing the image areas as changed, nochanged, and uncertain. Second, we design a denoise representation loss function in the representation learning branch to maximize the use of pseudo-labels, while mitigating speckle noise. Furthermore, a multilevel difference computation module is proposed to focus on changes in ground objects and capture more comprehensive change information. Experimental results on three public SAR datasets show that the proposed method outperforms six state-of-the-art methods, achieving the best performance with an average overall accuracy of 98.86% and an average Kappa coefficient of 89.36%.

Accurate power load forecasting is crucial for the stable operation and optimized management of power systems. Power load data are influenced by various external factors, such as weather conditions, holidays, and policy changes, resulting in significant nonlinearity and temporal dependencies. This complexity complicates prediction efforts. In this paper, we develop a new Dual-Branch Efficient Attention Gate (DBEG) model. This model consists of two branches: the Cross-Variable Correlation Extraction branch for extracting cross-variable correlations and the Temporal Correlation Extraction branch for modeling long-term dependencies along temporal dimension. The DBEG model proposes an efficient attention mechanism that comprehensively extracts information from both variate and temporal dimensions while reducing the computational complexity. Additionally, a gating mechanism is implemented to optimize the integration of outputs from the two branches, thereby enhancing prediction accuracy. Experimental results on two real-world power load datasets show that DBEG significantly outperforms existing methods of power load forecasting, validating its effectiveness and advancement.

Total anomalous pulmonary venous connections (TAPVC) is a serious congenital heart disease, and surgery is the main treatment for TAPVC patients. Surgical repair of TAPVC is challenging, and recently, 3D visualization including 3D printing and virtual reality has been adopted in clinical practice to mitigate this problem. However, the whole heart and great vessels for 3D printing is manually segmented by experts, which is time-consuming, subject, and costly. Though whole heart and great vessel segmentation have been a hot topic in the community for decades, the existing works focus on general diseases, and There is no specific attention on TAPVC. In this paper, we collect the first whole heart and great vessel segmentation dataset for surgical planning of TAPVC. The dataset contains 92 samples, which is annotated by experienced radiologist with about 1 hour. Unlike existing works which doesn’t consider pulmonary veins and left atrium separately, 10 anatomies including left ventricle, right ventricle, left atrium, right atrium, aorta, pulmonary artery, myocardium, superior vena cava, inferior vena cava, and pulmonary veins are annotated. Then, we proposed attention ensemble based network, AE-Network for whole heart and great vessel segmentation dataset for surgical planning of TAPVC. AE-Network adopts a three-branch structure, where each branch network is based on the U-shaped network with attention mechanisms. For each of the three branch, spatial attention, channel attention, and pixel attention are used to fully exploit the context. Dynamic segmentation result fusion (DSRF) is then used to combine all the three branches to achieve accurate and robust segmentation. Experimental results show that AE-Network achieved an average Dice coefficient of 80.60%, significantly outperforming existing works. Our dataset and code will be open-sourced once the paper is accepted.

Heterogeneous change detection (HCD) is a process to determine the change information by analyzing heterogeneous images of the same geographic location taken at different times, which plays an important role in remote sensing applications such as disaster response and environmental monitoring. However, the different imaging mechanisms result in different visual appearances in heterogeneous images, making it difficult to accurately detect changes through direct comparison. To address this problem, we propose a inter-modal and cross-modal self-supervised dual branch learning framework (ICSF) for HCD that incorporates inter-modal and cross-modal learning. First, in the inter-modal branch, we perform contrastive learning on heterogeneous images within their respective modalities to learn the robust and discriminative features, rather than relying on the raw spectral or spatial information from these images. Second, in the cross-modal branch, we perform cross-modal reconstruction to ensure the obtained features exhibit consistent comparability, thereby facilitating the extraction of rich information on the real changes within the images. Next, the difference images (DIs) computed from both branches are further refined using a superpixel segmentation strategy to preserve the consistency of differences within the same ground object. Experimental results on five public datasets with different modality combinations and change events demonstrate the effectiveness of the proposed approach in comparison to ten state-of-the-art (SOTA) methods, achieving the best performance with an average overall accuracy (OA) of 95.88% and an average Kappa coefficient (KC) of 74.20%.

Breast ultrasound (BUS) is an effective imaging modality for breast cancer diagnosis. The structural characteristics of breast lesions play an important role in computer-aided diagnosis. In this paper, a novel structure-aware triplet path network (SATPN) was designed to integrate classification and image reconstruction tasks to achieve accurate diagnosis on BUS images. Specifically, we enhanced clinically-approved structure characteristics of breast lesion by converting original BUS images to BI-RADS-oriented feature maps (BFMs) with a distance-transformation coupled Gaussian filter. Then, the converted BFMs were used as the inputs of the SATPN, which performed a supervised lesion classification task and two separate unsupervised stacked convolutional auto-encoder tasks for benign and malignant image reconstruction. We trained the SATPN with an alternative learning strategy by balancing image reconstruction error and classification label prediction error. The lesion label was determined by weighted voting of reconstruction error and label prediction error. We compared the performance of the SATPN with five deep learning methods using the original images and BFMs as inputs. Experimental results on two BUS datasets showed that SATPN performed the best among the six networks, with classification accuracy around 96%. These findings indicate that SATPN is promising for effective ultrasound computer-aided diagnosis of breast lesions.

Classification of hyperspectral images (HSIs) is a crucial topic in the domain of remote sensing. However, existing HSI classification methods often fail to adequately consider the connection between mid-level and high-level spectral-spatial features. Consequently, we propose a novel method named generative adversarial network-based spectral–spatial learning (GAN-SSL) for HSI classification. The method leverages the high spatial resolution of panchromatic (PAN) images and combines PAN images with HSIs to generate higher-quality HSIs, resulting in improved classification accuracy. Firstly, the dual-stream GAN is employed to generate higher-quality HSI from both HSI and PAN images. Secondly, the shallow-deep feature extraction classification network is utilized to capture both mid-level and high-level spectral–spatial information from the obtained features and a classifier is employed to categorize the extracted information. The experimental results obtained from two datasets indicate that the proposed approach surpasses several state-of-the-art techniques in terms of classification performance.

With the popularization of intelligent agricultural facilities, the demand for electricity in modern agricultural systems has also increased. To meet the continuous demand for electricity in agricultural production, including crop growth, storage, and processing, fine-grained electricity load forecasting becomes crucial, which can provide crucial decision support for the power supply, allocation, and management of agricultural facilities. However, the electricity load data in agricultural facilities is a non-stationary time series, which presents significant challenges for achieving accurate and effective forecasting. Thus, we focus on investigating the electricity load data in agricultural facilities and incorporate covariates, such as temperature, humidity, wind speed, and rainfall, into our analysis. Specifically, we propose a deep learning model based on empirical mode decomposition called EMD-BiLSTM-DLSTM. This model initially decomposes the electricity load time series into a sequence of relatively stationary components using empirical mode decomposition. It then employs a bidirectional long short-term memory network to predict each component, obtaining preliminary prediction results. Finally, a deep long short-term memory network is applied to refine the prediction results by incorporating covariates, resulting in more accurate prediction results. Experimental results show that compared with other time series forecasting methods, the proposed model has significant advantages in prediction accuracy and correlation.

Change detection in synthetic aperture radar (SAR) images is a vital application in remote sensing image processing. Existing unsupervised SAR change detection methods often rely on pre-classification to generate pseudo-labels for classifying the image regions into three classes: nochanged, changed, and uncertain. However, these methods do not fully exploit the pseudo-labels by focusing only on changed and nochanged regions. In this letter, we propose a wavelet transform-based multi-scale self-supervised network (WS2Net), which maximizes the utilization of pseudo-labels and incorporates discriminative feature learning. First, we employ clustering as pre-classification to obtain the aforementioned pseudo-labels. Second, we propose a self-supervised triple loss inspired by contrastive and representation learning. This loss comprises the nochanged and changed losses in the feature domain along with the uncertain loss in the source domain. Furthermore, to extract valuable information from SAR images and to improve the noise robustness of the network, we design a wavelet transform-based multi-scale feature extraction module. Finally, a difference image is generated by comparing the features output from the network, which can be further analyzed through segmentation to obtain the final change map. Comparative experiments are conducted with five state-of-the-art methods on three public SAR data sets, showing that the proposed WS2Net achieves the best performance with an average percent correct classification of 97.89% and an average kappa coefficient of 90.24%.

With the success of ViT (Vision Transformer), Transformer is being increasingly used for hyperspectral image (HSI) classification given its ability to extract global context dependencies. However, existing methods based on transformers tend to classify HSI in the traditional patch-wise manner. Thus, these methods cannot obtain true global features because the inputs of the model are local patches. To solve these problems, a hybrid convolution and ViT network (HCVN) is proposed for HSI classification. HCVN realizes the classification task from the perspective of semantic segmentation, and its input is the entire HSI, which makes it possible to obtain truly meaningful global features. By improving the original ViT, an HCV module is proposed, which enhances the ability of local structure characterization while extracting global features. The HCVN hybrid convolution layer and HCV module realize the extraction and fusion of local and global features. Finally, the dual branch network architecture is used to integrate the spatial and spectral features. Extensive experiments on two datasets verify the effectiveness of the proposed method.

Deep learning (DL) have recently achieved outstanding performance in change detection of multitemporal images. However, most existing DL-based change detection methods still suffer from the problem of insufficient labeled training samples. To overcome this limitation, an unsupervised superpixel-guided self-supervised learning network (S3Net) is proposed for detecting changes occurred on the land surface. By performing principal component analysis on two input images, a triple-channel pseudo-color image containing the main information of both images is first generated, which is used for superpixel segmentation to produce homogeneous image objects. Then, a siamese network composing of two identical subnetworks with shared weight based on transfer learning is trained for pretext task in a self-supervised learning way, aiming to obtain multiscale object-level spatial feature difference images. On this basis, a high-quality difference image is generated by incorporating the pixel-level and object-level difference information using a simple weighted fusion strategy, which can be analyzed by thresholding to produce the final binary change map. The experimental results on four real-world datasets from different sensors show that the proposed approach can obtain superior performance in comparison with several state-of-the-art change detection methods, which further demonstrates its effectiveness and practicability. We make our data and code publicly available (https://github.com/OMEGA-RS/S3Net_CD).

Transfer learning is an effective way to alleviate the problem of insufficient samples in a hyperspectral image (HSI) classification. However, the present transfer learning-based methods usually transfer knowledge from a single source domain, such as the natural image domain. Therefore, these methods cannot simultaneously transfer spectral and spatial knowledge to the target domain in HSIs. Generally, the natural image has rich spatial structure and texture information, while the HSI has abundant spectral information. To better utilize the knowledge learned from natural image datasets and HSI datasets, we proposed a multimodal transfer feature fusion network (MTFFN) for HSI classification. In MTFFN, a dual-branch network structure is designed to transfer the two-modal knowledge from the natural image domain and the source HSI domain to the target domain in two branches, respectively. A multitask learning strategy is adopted to achieve feature fusion. The fused features are used to generate the final classification result. Moreover, a local attention mechanism is designed to extract more meaningful spectral features. Experiments on two public datasets show that the proposed method is effective (https://github.com/HuaipYan/MTFFN).

To address the limitation of the high-dimensionality features and single spatial scale in the spectral–spatial classification of hyperspectral image (HSI), we propose a random multiscale convolutional network (RMCNet) that combines a multiscale dimensionality reduction module (MDRM) and the RMCNet for improving classification accuracy. The MDRM is based on multiscale superpixel segmentations, which implements dimensionality reduction leading to relieve the Hughes problem and reduce the computation burden in deep learning. Then, the multiscale spectral–spatial features are extracted by the RMCNet to adaptive various complex scenes in HSI. Finally, the multiscale spectral–spatial features act as inputs of support vector machine for classification. In the experiments, three benchmark HSIs are used to evaluate the performance of the proposed method. The experimental results demonstrate that the RMCNet can yield a competitive performance compared with the state-of-the-art methods.

Pan sharpening is used to fuse a low-resolution multispectral (MS) image and a high-resolution panchromatic (PAN) image to obtain a high-resolution MS image. This article proposes PSMD-Net, an end-to-end pan-sharpening method based on a multi-scale dense network. A shallow feature extraction layer (SFEL) extracts the shallow features from the original images, and these are used as an input to a global dense feature fusion (GDFF) network to learn the global features for image reconstruction. A multiscale dense block (MDB) is designed to fully extract the spatial and spectral information from the shallow features in the GDFF network. In the proposed network, multiple MDBs are stacked to extract rich, multi-scale dense hierarchical features, and a global dense connection (GDC) is designed to allow direct connections from the state of the current MDB to all subsequent MDBs to extract more advanced features. The extracted hierarchical features are sent to the global feature fusion layer (GFFL) to adaptively learn the global features for image reconstruction. Finally, global residual learning (GRL) is adopted to force the network to pay more attention to the changing part of the image. We perform experiments on simulated and real data from WorldView-2 and WorldView-3 satellites. Visual and quantitative assessment results demonstrate that PSMD-Net yields higher-resolution fusion images than the state-of-the-art methods.

Although the graph-based machine learning has received considerable attention in the remote sensing area and it has been widely used for terrain classification, the construction of graph in most existing algorithms still takes large memory and plenty of computational time especially for large Polarimetric Synthetic Aperture Radar (PolSAR) data. Addressing these issues, we propose a fast semi-supervised classification method based on parallel auction graph in this paper. The spatial relation between pixels is firstly preprocessed using the superpixel segmentation. Then we divide the PolSAR data into multiple groups, and each of them is used to construct a sparse auction graph. The semi-supervised classification is performed parallel on those graphs. Experimental results on simulated and real PolSAR data demonstrate its efficiency and effectiveness compared with existing methods.

Generalized bilinear model (GBM) has been widely used for nonlinear hyperspectral image unmixing. However, it does not take the sparse information of abundance into account, which is a significant characteristic resulting from the correlation of hyperspectral data. This paper aims to extend the GBM by incorporating the sparsity constraint of abundance matrix with the semi-nonnegative matrix factorization, by dividing GBM into the linear part and the second-order part, which are optimized using an alternating optimization algorithm respectively. L1/2-norm is used to explore the sparse characteristic, and the L1/2-constrained semi-nonnegative matrix factorization (L1/2-semi-NMF) algorithm is presented, which leads to better results on both synthetic and real data.

Since hyperspectral images (HSIs) usually have complex content and chaotic background, multiple kinds of features would be helpful for the classification task. Recently, representation-based methods with multifeature combination learning have been proposed. However, multifeature learning and the extended contextual information require much more computational burden, particularly for a large-scale dictionary case. In this letter, we propose a fast joint sparse representation classification method with multifeature combination learning for hyperspectral imagery. Once getting several complementary features (spectral, shape, and texture), the proposed model simultaneously acquires a representation vector for each kind of feature and imposes the joint sparsity ℓrow,0-norm regularization on the representation coefficients. The regularization can enforce the coefficients to share a common sparsity pattern, which preserves the crossfeature information. A new version of the simultaneous orthogonal matching pursuit is presented to solve the aforementioned problem because of its optimization with strong convergence guarantee and efficiency. Moreover, to further improve the classification performance, we incorporate contextual neighborhood information of the image into each kind of feature. Compared with state-of-the-art algorithms, it has been proved that the proposed algorithm with much less memory requirements performs tens to hundreds of times faster than those on real HSIs, while providing the same (or even better) accuracy.

Nonlocal means (NLM) algorithm has been proven to be an effective context-sensitive denoising approach, where many similar patches spatially far from a given patch could provide nonlocal constraint to the local structure. For hyperspectral image, however, the conventional NLM algorithm becomes inapplicable for the high number of spectral bands. In this letter, we incorporate the image nonlocal self-similarity into the maximum a posteriori estimation for hyperspectral classification. The main novelty lies in the following two aspects: The NLM algorithm is exploited to combine similar local structures and nonlocal averaging; a new class-relativity measurement is proposed to describe the self-similarity in the context of the hyperspectral classification. Several experiments on simulated and real hyperspectral data sets are provided to demonstrate the effectiveness of the proposed algorithm.

In order to improve the classification performance for hyperspectral image (HSI), a sparse representation classifier based on spectral information divergence (SID) is proposed. SID measures the discrepancy of probabilistic behaviors between the spectral signatures of two pixels from the aspect of information theory, which can be more effective in preserving spectral properties. Thus, the new method measures the similarity between the reconstructed pixel and the true pixel by SID instead of by the L2 norm used in traditional sparse model. Moreover, the spatial coherency across neighboring pixels sharing a common sparsity pattern is taken into account during the construction of SID-based joint sparse representation model. We propose a new version of the orthogonal matching pursuit method to solve SID-based recovery problems. The proposed SID-based algorithms are applied to real HSI for classification. Experimental results show that our algorithms outperform the classical sparse representation based classification algorithms in most cases.

In this paper, a novel hyperspectral image classification method is proposed, based on group sparse coding. The method is based on this acknowledgement that larger spatial variation exists in high spatial resolution hyperspectral image, which degrades the separability of hyperspectral image. In order to obtain a smooth representation, each pixel and its spatial neighbors are coded together by group sparse coding. Although nothing about class information is included, the neighbor pixels in a small spatial window are inclined to belong to the same class. Thus, that will reduce the within-class scatter and be favorable to the classification task. Then, the obtained sparse representation vectors are used for hyperspectral image classification with SVM. Experimental results show that our method exceeds the classical classification algorithms in accuracy and regional consistency.

A new method of feature extraction based on immune clonal selection algorithm is proposed, in which the immune clonal selection algorithm is used to optimize the projection vector. Some orthogonal bases are randomly selected as the initial basis vector sets from the original feature space, and the direction of the basis vectors is optimized to generate the optimal projection vector using the immune clonal selection algorithm. This method provides a new scheme of applying the immune clonal algorithm to feature extraction. Experimental results on benchmark datasets and MSTAR dataset for SAR target recognition verify the effectiveness of the proposed method.