Seizure prediction through scalp electroencephalogram (EEG) holds considerable practical potential. The primary challenge faced by existing algorithms lies in the individual heterogeneity, which hinders the generalizability of models to new patients. Additionally, inconsistencies in channel settings across various epilepsy centers further limit the applicability of models to diverse datasets. To address these challenges, we incorporate large language models (LLMs) into EEG analysis and propose a novel seizure prediction method based on LLMs (SPLLM), significantly enhancing both model generalizability and applicability. Specifically, this approach reprograms LLMs by transforming EEG signals into textual representations compatible with LLMs via a single-channel pre-training strategy. The method integrates cross-domain knowledge from both text and EEG data through a cross-attention mechanism, utilizing autoregressive pretrained LLMs to capture the temporal dependencies inherent in EEG signals. Moreover, the cross-domain generalization ability of LLMs alleviates patient heterogeneity, while the single-channel pre-training strategy enables the model to adapt to diverse channel settings. On two public datasets and one private dataset, SPLLM increases the average AUC by 8.2%, and the average balanced accuracy by 8.4% compared to existing methods. Experimental results demonstrate that the proposed method not only enhances cross-patient prediction accuracy but also adapts to data from different datasets, offering a scalable solution for the clinical application of seizure prediction.

The inherent ambiguity in distinguishing boundaries between lesions and adjacent tissues makes accurate segmentation in ultrasound images challenging. Although deep learning has improved segmentation accuracy, boundary segmentation quality and its relationship with anatomical structures remain underexplored. To address this, we propose DBF-Net, a dual-branch deep neural network that captures supervised relationships between anatomical structures and boundaries. Additionally, we introduce a feature fusion module to enhance the integration of body and boundary information. We evaluate DBF-Net on three public ultrasound image datasets and demonstrate its superiority over existing methods, achieving Dice Similarity Coefficients of 81.05±10.44% for breast cancer, 76.41±5.52% for brachial plexus nerves, and 87.75±4.18% for infantile hemangiomas on the BUSI, UNS, and UHES datasets, respectively. Our approach outperforms current methods, showcasing its effectiveness in ultrasound image segmentation. Code available at: https://github.com/apple1986/DBF-Net.

In convolutional neural networks (CNNs), downsampling operations are crucial to model performance. Although traditional downsampling methods (such as max pooling and strided convolution) perform well in feature aggregation, receptive field expansion, and computational reduction, they may lead to the loss of key spatial information in semantic segmentation tasks, thereby affecting the pixel-by-pixel prediction accuracy. To this end, this study proposes a downsampling method based on information complementarity - Hybrid Pooling Downsampling (HPD). The core is to replace the traditional method with MinMaxPooling, and effectively retain the light and dark contrast and detail features of the image by extracting the maximum value information of the local area. Experiment on various CNN and Transformer architectures using the ACDC and Synapse datasets demonstrate that HPD outperforms traditional segmentation methods. Specifically HPD improves the mean Dice similarity coefficient by 0.5%. The results show that the HPD module provides an efficient solution for semantic segmentation tasks. Code is available at https://github.com/apple1986/HPD.

Deep learning frameworks have been increasingly applied in the field of sleep stage classification. However, most advanced frameworks rely on huge datasets. In practical applications, data are generally continuously collected in batches. This requires continuous retraining of the model. It is expensive to store data and train models on them. Meanwhile, extensive access to EEG data may infringe upon the privacy of patients. To solve these problems, we propose a data distillation algorithm for sleep stage classification. This method compresses large EEG datasets into a small information synthetic dataset that can be used to train the network from scratch. To our knowledge, this is the first application of data distillation in this field. In the process of data distillation, we use gradient matching to optimize the synthetic dataset, which can avoid falling into the local minimum optimization. At the same time, we introduce data augmentation in the process to enhance the generalization ability of synthetic dataset. Finally, in order to improve the stability of data distillation, we use K-medoids clustering to initialize the synthesized dataset. We validated six sleep stage classification frameworks on three publicly available datasets and proved the superiority of our method. We also explored the application of our method in the field of neural architecture search, achieving robust results.

While patient-specific seizure prediction deep learning (DL) models can deliver remarkable performance tailored to individual patients, the development of patient-independent models that offer satisfactory cross-subject performance holds greater significance and practicality. However, these patient-independent models, which leverage electroencephalogram (EEG) data from multiple patients, may give rise to privacy concerns. This is because EEG data contains sensitive information regarding individuals’ health and mental states. Consequently, from a privacy-preserving perspective, patients may desire the removal of their data information from trained models. Yet, accommodating such forgetting requests presents a formidable challenge: how to enable DL models to forget the data information of specific patients without compromising the performance for others. Although retraining a model from scratch without the data of a specific patient can somewhat address this issue, it becomes computationally prohibitive, especially with large datasets. To tackle this, we introduce an efficient machine unlearning approach called feature unlearning (FU) for seizure prediction. This method modifies the feature projection distribution of specific patients’ data within trained models to match that of models retrained from scratch. Our proposed FU method comprises two primary components: 1) feature shifting, which alters the original distribution of feature projection of specific patients in trained models and feature retaining, which mitigates the adverse effects of feature shifting on other patients, preserving their overall performance through knowledge distillation. We assess our FU method using the CHB-MIT dataset. The results demonstrate that our FU approach can effectively remove the data information of specific patients from trained DL models while maintaining the performance for other patients.

Infrared and visible image fusion aims to extract complementary features to synthesize a single fused image. In our method, we covert the regular image format into the graph space and conduct graph convolutional networks (GCNs) to extract NLss for the reliable infrared and visible image fusion. More specifically, GCNs are first performed on each intra-modal set to aggregate the features and propagate the inherent information, thereby extracting independent intra-modal NLss. Then, such intra-modal non-local self-similarity (NLss) features of infrared and visible images are concatenated to explore cross-domain NLss inter-modally and reconstruct the fused images. Extensive experiments show the superior performance of our method with the qualitative and quantitative analysis on the TNO, RoadScene and M3FD datasets, respectively, outperforming many state-of-the-art (SOTA) methods for the robust and effective infrared and visible image fusion.

Electroencephalogram (EEG) signals are vital in understanding brain activity, but their weak amplitude makes them susceptible to various artifacts. Accurate denoising of EEG data is crucial as a preprocessing step to ensure precise analysis and interpretation. In recent years, the diffusion model has garnered significant attention as a promising approach in generative modeling. This model effectively addresses the issue of over-smoothing in existing deep learning methods and thus has the potential to generate more refined denoised EEG signals. However, the generation process of the standard diffusion model is highly random, limiting its direct application to EEG denoising tasks. To address this limitation, we propose a conditional diffusion model specifically designed for EEG denoising. In this model, the standard diffusion model's denoising network is replaced by a novel dual-branch network, where noisy EEG information is used as a condition to guide the generation of corresponding clean EEG signals. This dual-branch structure leverages the complementary strengths of convolutional neural network (CNN) and Transformer architectures, integrating multi-scale features to comprehensively extract information from the signal. Extensive experiments demonstrate the remarkable performance of EEGDfus in EEG denoising. We tested it on two public datasets. Testing on two public datasets, EEGdenoiseNet and SSED, demonstrated that after denoising, the average correlation coefficient increased to 0.983 and 0.992 for EOG artifact removal, respectively. The proposed model outperforms commonly used baseline models, setting a new state-of-the-art benchmark in the field of EEG denoising.

Electroencephalography (EEG) signals are often contaminated with various physiological artifacts, seriously affecting the quality of subsequent analysis. Therefore, removing artifacts is an essential step in practice. As of now, deep learning-based EEG denoising methods have exhibited unique advantages over traditional methods. However, they still suffer from the following limitations. The existing structure designs have not fully taken into account the temporal characteristics of artifacts. Meanwhile, the existing training strategies usually ignore the holistic consistency between denoised EEG signals and authentic clean ones. To address these issues, we propose a GAN guided parallel CNN and transformer network, named GCTNet. The generator contains parallel CNN blocks and transformer blocks to respectively capture local and global temporal dependencies. Then, a discriminator is employed to detect and correct the holistic inconsistencies between clean and denoised EEG signals. We evaluate the proposed network on both semi-simulated and real data. Extensive experimental results demonstrate that GCTNet significantly outperforms state-of-the-art networks in various artifact removal tasks, as evidenced by its superior objective evaluation metrics. For example, in the task of removing electromyography artifacts, GCTNet achieves 11.15% reduction in RRMSE and 9.81% improvement in SNR over other methods, highlighting the potential of the proposed method as a promising solution for EEG signals in practical applications.

Infrared and visible image fusion (IVF) plays an important role in intelligent transportation system (ITS). The early works predominantly focus on boosting visual appeal of the fused result, although several recent approaches have tried to combine high-level vision task with IVF, they prioritize the design of cascaded structure to seek unified suitable features and fit different tasks. Thus, they tend to bias toward reconstructing raw pixels without considering the significance of semantic features. Therefore, we propose a novel prior semantic guided image fusion method based on the dual-modality strategy, improving the performance of IVF in ITS. Specifically, to explore the independent significant semantic of each modality, we first design two parallel semantic segmentation branches with a refined feature adaptive-modulation (RFaM) mechanism. RFaM can perceive the features that are semantically distinct enough in each semantic segmentation branch. Then, two pilot experiments based on the two branches are conducted to capture the significant prior semantic of source images, which is then applied to guide the fusion task in the integration of semantic segmentation branches and fusion branch. In addition, to aggregate both high-level semantics and impressive visual effects, we further investigate the frequency response of the prior semantics, and propose a multi-level representation-adaptive fusion (MRaF) module to explicitly integrate low-frequency prior semantic with high-frequency details. Extensive experiments on two public datasets demonstrate the superiority of our method over state-of-the-art fusion approaches. Our method has better performance on four quantitative metrics in fusion task and achieves the highest mIoU in semantic segmentation task.

Privacy protection has become increasingly crucial in the field of epilepsy prediction. Some latest studies introduced the source free domain adaptation (SFDA), which only utilizes a pre-trained source model for protecting the source data privacy. However, the existing SFDA methods exist two shortcomings. (1) the offline setting, which is not suitable for real-world online scenarios (2) the poor performance, which is attributed to the absence of labeled calibration data during the adaptation phase. To this end, we proposed a online seizure prediction framework based on fine-tuning and test-time adaptation (FT3A). Specifically, FT3A employs one seizure event target data to fine-tune and continuously adapt pre-trained source model to unlabeled target data stream. In addition, the adaption and prediction is performed simultaneously. On the one hand, we design the task model as a multi-head structure to increase the confidence of the model and reduce error accumulation. On the other hand, a memory bank is introduced to store a small amount of historical EEG data, which helps handle the catastrophic forgetting concern of the model during online adaptation. Extensive experiments on public CHB-MIT dataset and the private freiburg hospital dataset indicate the superiority and generality of the proposed method.

Domain adaptation (DA) techniques are frequently utilized to enhance seizure prediction accuracy by leveraging the labeled electroencephalogram data of existing patients on new patients. Traditional DA methods, however, require access to the source domain while training the adaptation model, which poses a threat to sensitive patient information and privacy. To address this issue, in this article, we propose a novel Gaussian mixture modeling (GMM)-based source-free domain adaptation (GSFDA). Our method leverages the GMM joint source model and target data structure for clustering, employs uncertainty learning to minimize DA uncertainty, and uses the mixup technique to increase model robustness while reducing the impact of noisy pseudolabels. Notably, GSFDA only requires access to the source model parameters, and not the source domain, effectively safeguarding the privacy of patient information. This has substantial clinical implications for seizure prediction.

Convolutional Neural Networks (CNNs) have been widely used in medical image segmentation to efficiently develop computer-aided diagnosis systems. Due to the locality of convolutional operations, they can be used to extract fine-grained features, but with limitations in building global context and long-range spatial relationships. Recently, shifted window-based multi-layer perceptron (Swin-MLP) methods have demonstrated the ability to learn coarse-grained spatial features in a fixed-size window, but they are not well suited to dense-prediction tasks, such as medical image segmentation. To harmonize the strengths and mitigate the weaknesses of CNNs and SwinMLP in extracting features of varying granularity, we proposed two novel granularity fusion modules that use coarse-grained features to guide the fusion of fine-grained features based on the attention mechanism. Specifically, the first fusion module, named as BGFuse (Block Granularity Fuse), could fuse various scale block-grained features from Swin-MLP. The second fusion module, termed as LGFuse (Local Granularity Fuse), could fuse semantic coarse granularity information into fine granularity features. Equipped with these two fusion modules, we present a new attention-guided encoder-decoder network architecture (termed MGFuseSeg) for medical image segmentation. Without bells and whistles, the proposed MGFuseSeg significantly boosts the performance on three challenging segmentation benchmarks including Synapse, ACDC, and ISIC. Codes are available at https://github.com/apple1986/MGFuseSeg

With the flourishing development of deep learning (DL) and the convolution neural network (CNN), electroencephalogram-based (EEG) emotion recognition is occupying an increasingly crucial part in the field of brain-computer interface (BCI). However, currently employed architectures have mostly been designed manually by human experts, which is a time-consuming and labor-intensive process. In this paper, we proposed a novel neural architecture search (NAS) framework based on reinforcement learning (RL) for EEG-based emotion recognition, which can automatically design network architectures. The proposed NAS mainly contains three parts: search strategy, search space, and evaluation strategy. During the search process, a recurrent network (RNN) controller is used to select the optimal network structure in the search space. We trained the controller with RL to maximize the expected reward of the generated models on a validation set and force parameter sharing among the models. We evaluated the performance of NAS on the DEAP and DREAMER dataset. On the DEAP dataset, the average accuracies reached 97.94%, 97.74%, and 97.82% on arousal, valence, and dominance respectively. On the DREAMER dataset, average accuracies reached 96.62%, 96.29% and 96.61% on arousal, valence, and dominance, respectively. The experimental results demonstrated that the proposed NAS outperforms the state-of-the-art CNN-based methods.

Emotion recognition based on electroencephalography (EEG) is a significant task in the brain-computer interface field. Recently, many deep learning-based emotion recognition methods are demonstrated to outperform traditional methods. However, it remains challenging to extract discriminative features for EEG emotion recognition, and most methods ignore useful information in channel and time. This article proposes an attention-based convolutional recurrent neural network (ACRNN) to extract more discriminative features from EEG signals and improve the accuracy of emotion recognition. First, the proposed ACRNN adopts a channel-wise attention mechanism to adaptively assign the weights of different channels, and a CNN is employed to extract the spatial information of encoded EEG signals. Then, to explore the temporal information of EEG signals, extended self-attention is integrated into an RNN to recode the importance based on intrinsic similarity in EEG signals. We conducted extensive experiments on the DEAP and DREAMER databases. The experimental results demonstrate that the proposed ACRNN outperforms state-of-the-art methods.

Emotion recognition based on electroencephalogram (EEG) plays an increasingly important role in the field of brain–computer interfaces. Recently, deep learning has been widely applied to EEG decoding owning to its excellent capabilities in automatic feature extraction. Transformer holds great superiority in processing time-series signals due to its long-term dependencies extraction ability. However, most existing transformer architectures are designed manually by human experts, which is a time-consuming and resource-intensive process. In this article, we propose an automatic transformer neural architectures search (TNAS) framework based on multiobjective evolution algorithm (MOEA) for the EEG-based emotion recognition. The proposed TNAS conducts the MOEA strategy that considers both accuracy and model size to discover the optimal model from well-trained supernet for the emotion recognition. We conducted extensive experiments to evaluate the performance of the proposed TNAS on the DEAP and DREAMER datasets. The experimental results showed that the proposed TNAS outperforms the state-of-the-art methods.

In recent years, deep learning has gained widespread attention in electroencephalogram (EEG)-based emotion recognition. However, deep learning methods are usually time-consuming with a large amount of memory usage, which obstructs their practical usage on resource-constrained devices. In this paper, we propose a binary capsule network (Bi-CapsNet) for EEG emotion recognition with low computational cost and memory usage. The Bi-CapsNet binarizes 32-bit weights and activations to 1 b, and replaces floating-point operations with efficient bitwise operations. To address the issue of function discontinuity in backward propagation, we use a continuous function to approximate the binarization process. Two popular EEG emotion databases, namely, DEAP and DREAMER, are used for performance evaluation. In comparison to its full-precision counterpart, the Bi-CapsNet achieves a $>\!25\times$reduction on the computational cost and a $>\!5\times$ reduction on the memory usage, while with only a $< $1% drop on the recognition accuracy. Compared to some state-of-the-art EEG emotion recognition methods, the proposed method obtains more competitive performance. In addition, the Bi-CapsNet is implemented on a mobile phone via an open-source binary inference framework named Bolt, and it achieves an $\sim\! 5\times$ inference acceleration in comparison to its full-precision counterpart.

A sharp prediction for ultrasound image segmentation is critical to assist doctors in the accurate diagnosis and treatment. Although the existing methods could achieve impressive performance on biomedical image segmentation, they still have difficulty in segmenting pixels near the boundaries of lesions faithfully. To this end, we propose a body and boundary supervision network aiming to improve ultrasound image segmentation performance, especially on the boundary of the segmented objects. In contrast to existing approaches that take lesions as a whole, a novel dual-branch supervision block was developed to explore explicit modeling of the body and boundary of the object at feature level. Furthermore, we presented an adaptive feature fusion strategy to combine feature maps from dual-branch to get more representative features for final segmentation. Experimental results on three public ultrasound datasets including BUSI, HC18 and Hemangioma demonstrate that the proposed network can achieve leading segmentation performance for breast cancer, fetal head circumference and hemangioma on ultrasound images compared with the existing state-of-the-art models.

Sparse unmixing (SU) of hyperspectral image (HSI), as a semisupervised approach, aims to find the optimal subset of the spectral library known in advance to represent each pixel in HSI. However, most of the existing SU methods cannot take full advantage of spatial information and mixed noise in HSI. To this end, we propose a superpixel-based noise-robust SU method (SNRSU) in the presence of mixed noise. First, we perform superpixel segmentation (SS) on the first principal component of HSI to extract the homogeneous regions. Then, we unmix each superpixel based on sparse representation (SR) and low-rank representation (LRR) in the maximuma posterioriframework, which can make full use of the spatial–spectral information in HSI under complex mixed noise. A number of experiments on simulated and real HSI datasets confirm the superior performance of the proposed SNRSU both qualitatively and quantitatively.

In this paper, an unrolling algorithm of the iterative subspace-based optimization method (SOM) is proposed for solving full-wave inverse scattering problems (ISPs). The unrolling network, named SOM-Net, inherently embeds the Lippmann-Schwinger physical model into the design of network structures. The SOM-Net takes the deterministic induced current and the raw permittivity image obtained from back-propagation (BP) as the input. It then updates the induced current and the permittivity successively in sub-network blocks of the SOM-Net by imitating iterations of the SOM. The final output of the SOM-Net is the full predicted induced current, from which the scattered field and the permittivity image can also be deduced analytically. The parameters of the SOM-Net are optimized in a supervised manner with the total loss to simultaneously ensure the consistency of the induced current, the scattered field, and the permittivity in the governing equations. Numerical tests on both synthetic and experimental data verify the superior performance of the proposed SOM-Net over typical ones. The results on challenging examples like scatterers with tough profiles or high permittivity demonstrate the good generalization ability of the SOM-Net. With the use of deep unrolling technology, this work builds a bridge between traditional iterative methods and deep learning methods for solving ISPs.

Band selection is an effective means to alleviate the curse of dimensionality in hyperspectral data. Many methods select a compact and low redundant band subset, which is inadequate as it may degrade the classification performance. Instead, more emphasis shall be put on selecting representative bands. In this article, we propose a robust unsupervised band selection method to address this issue. Our method reveals bandwise representativeness based on the comprehensive interband neighborhood structure. It incorporates an interband neighborhood graph into a sparse self-contained regression model in order to provide a reasonable measure for bandwise representativeness. The derived coefficient matrix not only uncovers bandwise importance values but also is coherent to the generalized interband local neighborhood structure. For constructing the interband neighboring structural graph, an integrated multigraph model is employed to achieve better generalization performance. It combines the benefit of multiple graphs but is insusceptible to the defects of a single one. To enhance the reliability of this model, a joint trace minimum and nonnegative constraint is imposed on the coefficient matrix. Accordingly, a multigraph integrated embedding and robust self-contained regression model (MGRSR) is formulated. In addition, an iterative update algorithm is developed to solve the problem. Comparative experiments on three hyperspectral data sets illustrate that MGRSR is robust to various data and has superior performance compared with several state-of-the-art methods.

Recently, deep neural networks (DNNs) have been applied to emotion recognition tasks based on electroencephalography (EEG), and have achieved better performance than traditional algorithms. However, DNNs still have the disadvantages of too many hyperparameters and lots of training data. To overcome these shortcomings, in this article, we propose a method for multi-channel EEG-based emotion recognition using deep forest. First, we consider the effect of baseline signal to preprocess the raw artifact-eliminated EEG signal with baseline removal. Secondly, we construct 2 D frame sequences by taking the spatial position relationship across channels into account. Finally, 2 D frame sequences are input into the classification model constructed by deep forest that can mine the spatial and temporal information of EEG signals to classify EEG emotions. The proposed method can eliminate the need for feature extraction in traditional methods and the classification model is insensitive to hyperparameter settings, which greatly reduce the complexity of emotion recognition. To verify the feasibility of the proposed model, experiments were conducted on two public DEAP and DREAMER databases. On the DEAP database, the average accuracies reach to 97.69% and 97.53% for valence and arousal, respectively; on the DREAMER database, the average accuracies reach to 89.03%, 90.41%, and 89.89% for valence, arousal and dominance, respectively. These results show that the proposed method exhibits higher accuracy than the state-of-art methods.

Remote photoplethysmography (rPPG) is a non-contact technique for measuring cardiac signals from facial videos. High-quality rPPG pulse signals are urgently demanded in many fields, such as health monitoring and emotion recognition. However, most of the existing rPPG methods can only be used to get average heart rate (HR) values due to the limitation of inaccurate pulse signals. In this paper, a new framework based on generative adversarial network, called PulseGAN, is introduced to generate realistic rPPG pulse signals through denoising the chrominance (CHROM) signals. Considering that the cardiac signal is quasi-periodic and has apparent time-frequency characteristics, the error losses defined in time and spectrum domains are both employed with the adversarial loss to enforce the model generating accurate pulse waveforms as its reference. The proposed framework is tested on three public databases. The results show that the PulseGAN framework can effectively improve the waveform quality, thereby enhancing the accuracy of HR, the interbeat interval (IBI) and the related heart rate variability (HRV) features. The proposed method significantly improves the quality of waveforms compared to the input CHROM signals, with the mean absolute error of AVNN (the average of all normal-to-normal intervals) reduced by 41.19%, 40.45%, 41.63%, and the mean absolute error of SDNN (the standard deviation of all NN intervals) reduced by 37.53%, 44.29%, 58.41%, in the cross-database test on the UBFC-RPPG, PURE, and MAHNOB-HCI databases, respectively. This framework can be easily integrated with other existing rPPG methods to further improve the quality of waveforms, thereby obtaining more reliable IBI features and extending the application scope of rPPG techniques.

Infrared and visible image fusion aims to describe the same scene from different aspects by combining complementary information of multi-modality images. The existing Generative adversarial networks (GAN) based infrared and visible image fusion methods cannot perceive the most discriminative regions, and hence fail to highlight the typical parts existing in infrared and visible images. To this end, we integrate multi-scale attention mechanism into both generator and discriminator of GAN to fuse infrared and visible images (AttentionFGAN). The multi-scale attention mechanism aims to not only capture comprehensive spatial information to help generator focus on the foreground target information of infrared image and background detail information of visible image, but also constrain the discriminators focus more on the attention regions rather than the whole input image. The generator of AttentionFGAN consists of two multi-scale attention networks and an image fusion network. Two multi-scale attention networks capture the attention maps of infrared and visible images respectively, so that the fusion network can reconstruct the fused image by paying more attention to the typical regions of source images. Besides, two discriminators are adopted to force the fused result keep more intensity and texture information from infrared and visible image respectively. Moreover, to keep more information of attention region from source images, an attention loss function is designed. Finally, the ablation experiments illustrate the effectiveness of the key parts of our method, and extensive qualitative and quantitative experiments on three public datasets demonstrate the advantages and effectiveness of AttentionFGAN compared with the other state-of-the-art methods.

Hyperspectral band selection is of great value to alleviate the curse of dimensionality. For many band selection methods, however, the neglect of bandwise usefulness tends to result in the loss of valuable bands, but the retention of useless ones; consequently, this causes deterioration of the classification performance. In this sense, bandwise significance should be emphasized. To address this issue, this article proposes a manifold-preserving and weakly redundant (MPWR) unsupervised band selection method. In the method, a manifold-preserving band-importance metric is put forward to measure the bandwise essentiality. This ensures the retention of bands involving abundant intrinsic structures conductive to classification. Specifically, aimed at obtaining the presented band-importance metric, an attainment algorithm is presented, which mainly relies on the embedding learning and linear regression, followed by the introduction of multi-normalization combination. In addition, concerning the massive redundancy caused by the highly correlated bands, MPWR further establishes a constrained band-weight optimization model. Then, both bandwise manifold-preserving capability and intraband correlation are fully integrated into the band selection process. To solve the problem, a corresponding algorithm within the framework of the alternating direction method of multipliers (ADMM) is also developed. Regarding evaluating the effectiveness of the proposed method, comparative experiments with the state-of-the-art methods are conducted on three public hyperspectral data sets. Experimental results demonstrate the superiority and robustness of MPWR.

Generative adversarial network (GAN) has been widely applied to infrared and visible image fusion. However, the existing GAN-based image fusion methods only establish one discriminator in the network to make the fused image capture gradient information from the visible image, which may result in the loss of some infrared intensity information and texture information on the fused images. To solve this problem and improve the performance of GAN, we extend GAN to multiple discriminators and propose an end-to-end multi-discriminators Wasserstein generative adversarial network (MD-WGAN). In this framework, the fused image can preserve major infrared intensity and detail information from the first discriminator, and keep more texture information that existing in visible image from the second discriminator. We also design a texture loss function via local binary patterns to preserve more texture from visible image. The extensive qualitative and quantitative experiments show the advantages of our method compared with other state-of-the-art fusion methods.

Recently, collaborative representation has received much attention in the hyperspectral image (HSI) classification due to its simplicity and effectiveness. However, the existing collaborative representation-based HSI classification methods ignore the correlation among different classes. To overcome this problem, we propose a discriminative kernel collaborative representation and Tikhonov regularization method (DKCRT) for HSI classification, which can make the kernel collaborative representation of different classes to be more discriminative. Specifically, the kernel trick is adopted to map the original HSI into a high space to improve the class separability. Besides, distance-weighted kernel Tikhonov regularization is adopted to enforce these training samples to have large representation coefficients, which are similar to the test sample in the high-dimensional feature space. Moreover, we add a discriminative regularization term to further enhance the separability of different classes, which can take the correlation among different classes into consideration. Furthermore, to take the spatial information of HSI into consideration, we extend the DKCRT to a joint version named JDKCRT. Experiments on real HSIs demonstrate the efficiency of the proposed DKCRT and JDKCRT.

Several bandwise total variation (TV) regularized low-rank (LR)-based models have been proposed to remove mixed noise in hyperspectral images (HSIs). These methods convert high-dimensional HSI data into 2-D data based on LR matrix factorization. This strategy introduces the loss of useful multiway structure information. Moreover, these bandwise TV-based methods exploit the spatial information in a separate manner. To cope with these problems, we propose a spatial–spectral TV regularized LR tensor factorization (SSTV-LRTF) method to remove mixed noise in HSIs. From one aspect, the hyperspectral data are assumed to lie in an LR tensor, which can exploit the inherent tensorial structure of hyperspectral data. The LRTF-based method can effectively separate the LR clean image from sparse noise. From another aspect, HSIs are assumed to be piecewisely smooth in the spatial domain. The TV regularization is effective in preserving the spatial piecewise smoothness and removing Gaussian noise. These facts inspire the integration of the LRTF with TV regularization. To address the limitations of bandwise TV, we use the SSTV regularization to simultaneously consider local spatial structure and spectral correlation of neighboring bands. Both simulated and real data experiments demonstrate that the proposed SSTV-LRTF method achieves superior performance for HSI mixed-noise removal, as compared to the state-of-the-art TV regularized and LR-based methods.

Sparse unmixing (SU) of hyperspectral data have recently received particular attention for analyzing remote sensing images, which aims at finding the optimal subset of signatures to best model the mixed pixel in the scene. However, most SU methods are based on the commonly admitted linear mixing model, which ignores the possible nonlinear effects (i.e., nonlinearity), and the nonlinearity is merely treated as outlier. Besides, the traditional SU algorithms often adopt the$\ell _{2}$norm loss function, which makes them sensitive to noises and outliers. In this paper, we propose a robust SU (RSU) method with$\ell _{2,1}$norm loss function, which is robust for noises and outliers. Then, the RSU can be solved by the alternative direction method of multipliers. Finally, the experiments on both synthetic data sets and real hyperspectral images demonstrate that the proposed RSU is efficient for solving the hyperspectral SU problem compared with the state-of-the-art algorithms.

Recently, low-rank representation (LRR) based methods have been used for hyperspectral image (HSI) denoising, which can simultaneously remove different types of noise: Gaussian noise, impulse noise, dead lines, and so on. However, the LRR based method does not make full use of the spatial information in HSI. In this paper, we integrate the superpixel segmentation (SS) into the LRR, and propose a novel denoising method named SS-LRR. We first use the principle component analysis (PCA) to obtain the first principle component of HSI. Then the superpixel segmentation is adopted to the first principle component of HSI to get homogeneous regions. Finally, we employ the LRR to each homogeneous region of HSI, which enable us to simultaneously remove all the above mentioned mixed noise. Extensive experiments on both simulated and real hyperspectral images demonstrate that the proposed SS-LRR is efficient for HSI denoising.

Sparse unmixing (SU) of hyperspectral data has recently received particular attention for analyzing remote sensing images, which aims at finding the optimal subset of signatures to best model the mixed pixel in the scene. However, most SU methods are based on the commonly admitted linear mixing model (LMM), which ignores the possible nonlinear effects (i.e. nonlinearity), and the nonlinearity is merely treated as outlier. Besides, the traditional SU algorithms often adopt the ℒ2norm loss function, which makes them sensitive to noises and outliers. In this paper, we propose a robust sparse unmixing (RSU) method with ℒ2,1norm loss function, which is robust for noises and outliers. Then, the RSU can be solved by the alternative direction method of multipliers (ADMM). Finally, experiments on synthetic datasets demonstrate that the proposed RSU is efficient for solving the hyperspectral SU problem compared with state-of-the-art algorithms.

Recently, sparse unmixing (SU) of hyperspectral data has received particular attention for analyzing remote sensing images. However, most of SU methods are based on the commonly admitted linear mixing model (LMM), which ignores the possible nonlinear effects (i.e. nonlinearity). In this paper, we proposed a new method named robust collaborative sparse regression (RCSR) for hyperspectral unmixing, which is based on the robust LMM (rLMM). The rLMM takes the nonlinearity into consideration, and the nonlinearity is merely treated as outlier, which has the underlying sparse property. The RCSR takes the collaborative sparse property of the abundance and sparsely distributed additive property of the outlier into consideration, which can be formed as a robust joint sparse regression problem. Experiments on synthetic datasets demonstrate that the proposed RCSR is efficient for solving the hyperspectral SU problem compared with other five state-of-the-art algorithms.

In this paper, we propose a method for multimodal retinal image registration based on feature guided Gaussian mixture model (GMM) and edge map. We extract two sets of feature points from the edge maps of two images, and formulate image registration as the estimation of a feature guided mixture of densities: a GMM is fitted to one point set, such that both the centers and local features of the Gaussian densities are constrained to coincide with the other point set. The problem is solved under a maximum-likelihood framework together with an iterative EM algorithm initialized by confident feature matches, where the image transformation is modeled by an affine function. Extensive experiments on various retinal images show the robustness of our method, which consistently outperforms other state-of-the-arts, especially when the data is badly degraded.

Recently, low-rank representation (LRR) based hyperspectral image (HSI) denoising method has been proven to be a powerful tool for removing different kinds of noise simultaneously, such as Gaussian, dead pixels and impulse noise. However, the LRR based method cannot make full use of the spatial information in HSI. In this paper, we integrate the graph based segmentation (GS) into the LRR, and propose a novel denoising method named GS-LRR. We first use the principle component analysis (PCA) to obtain the first principle component of HSI. Then the graph based segmentation is adopted to the first principle component of HSI to get homogeneous regions. Finally, we employ the LRR to each homogeneous region of HSI, which enable us to simultaneously remove all the above mentioned mixed noise. Extensive experiments on both simulated and real HSIs demonstrate the efficiency of the proposed GS-LRR.

The generalized bilinear model (GBM) has been widely used for the nonlinear unmixing of hyperspectral images, and traditional GBM solvers include the Bayesian algorithm, the gradient descent algorithm, the semi-nonnegative-matrix-factorization algorithm, etc. However, they suffer from one of the following problems: high computational cost, sensitive to initialization, and the pixelwise algorithm hinders us from applying to large hyperspectral images. In this letter, we apply Nesterov's optimal gradient method to solve the least-square problem under the bound constraint, which is named as the bound projected optimal gradient method (BPOGM). The BPOGM can achieve the optimal convergence rate of$O(1/k^{2})$, with$k$denoting the number of iterations in BPOGM. We further apply the BPOGM to solve the GBM-based unmixing problem. Experiments on both synthetic data sets and real hyperspectral images demonstrate that the BPOGM is efficient for solving the GBM-based unmixing problem.

Recently, the sparse representation-based classification (SRC) methods have been successfully used for the classification of hyperspectral imagery, which relies on the underlying assumption that a hyperspectral pixel can be sparsely represented by a linear combination of a few training samples among the whole training dictionary. However, the SRC-based methods ignore the sparse representation residuals (i.e., outliers), which may make the SRC not robust for outliers in practice. To overcome this problem, we propose a robust SRC (RSRC) method which can handle outliers. Moreover, we extend the RSRC to the joint robust sparsity model named JRSRC, where pixels in a small neighborhood around the test pixel are simultaneously represented by linear combinations of a few training samples and outliers. The JRSRC can also deal with outliers in hyperspectral classification. Experiments on real hyperspectral images demonstrate that the proposed RSC and JRSRC have better performances than the orthogonal matching pursuit (OMP) and simultaneous OMP, respectively. Moreover, the JRSRC outperforms some other popular classifiers.