As an important research element in the field of hyperspectral remote sensing, the purpose of hyperspectral unmixing is to decompose the mixed pixels in a hyperspectral image into endmembers and abundance. With the development of deep learning, it also has good prospects for application in the field of hyperspectral unmixing. Existing hyperspectral unmixing networks tend to focus only on the spectral information in the image which neglecting the spatial information. Therefore, a network based on dual-stream convolutional feature fusion for hyperspectral unmixing is proposed to make full use of the correlation between neighboring pixels in the paper, which includes three parts: dual-stream feature extraction, feature fusion and abundance estimation. The unmixing performance of the network is verified on two real datasets and exhibits better performance compared with other state-of-the-art methods.

Image captioning aims to generate descriptions of images, which requires capturing complex interactions between local regions and global context within image. However, effective global context modeling from image remains a challenging research topic. Existing approaches incorporate global-level information into the initialized input mainly based on transformer architecture. Unlike previous methods that may not be able to capture rich global contextual information, we propose a novel method named Context-Sensitive Transformer (CSTNet), which can discover the inherent global context and further empower the global-to-local interactions. Experimental results on the MSCOCO dataset show that the proposed model can significantly improve the performance of image captioning.

Fusion of hyperspectral and multispectral data to obtain hyperspectral data with high-spatial-resolution has been an important topic in recent years. The fusion methods that from model-driven to data-driven have been proposed constantly. Data driven models, especially the deep neural networks, are widely used owing to their excellent performance. And the design of its structure is highly related to the task. It is noteworthy that the hyperspectral and multispectral spectrum contains abundant information. However, most neural net-work based approaches mainly focus on spatial features and ignore the spectral information, which is also very important for fusion task. In this paper, we propose a ID-convolutional neural network, it can extract hyperspectral and multispectral features, simultaneously, and also can capture the spectral correlation and cross-correlation between hyperspectral and multispectral spectrum. Groups of simulation experiments demonstrate that compared with the state-of-the-art methods, paying more attention to the spectrum can obtain better performance.

Image captioning is a challenging task which aims to generate descriptions of images. Most existing approaches adopt the encoder-decoder architecture, where encoder takes the image as input and decoder predicts corresponding word sequence. However, a common defect of these methods is that the abundant semantic relationships between relevant regions are ignored, leading the decoder to give a misled caption. To alleviate this issue, we propose a novel model, which utilizes sufficient semantic relationships provided by scene graph to guide the word generation process. To some extent, the scene graph narrows the semantic gap between images and descriptions, and hence improves the quality of generated sentences. Extensive experimental results demonstrate that our model achieves superior performance on various quantitative metrics.

We present a spatial-spectral autoencoder (SSAE) for hyperspectral unmixing, including a net for endmember extraction (EENet) and a net for abundance estimation (AENet). The EENet exploits the spatial information in hyperspectral image by a “many to one” strategy, i.e., the abundance of a pixel is combined by the abundances of its adjacent pixels. The idea is based on the assumption: once an endmember is mixed in a pixel, it is mixed in the surrounding pixels with high probability. The strategy promotes a continuous and smooth spatial distribution of abundances, and it is more effective than the other methods for endmember extraction. Besides, to make full use of the rich spectral information and obtain more accurate abundances, we design an AENet, which applies the deep convolutional neural network to estimate the abundances with the endmembers acquired from the EENet. The experiments are conducted on two real datasets, which show the SSAE outperforms the state-of-the-art methods.

Recently, the development of deep learning brings new opportunities for hyperspectral unmixing. In this paper, we propose a new architecture based on Recurrent Neural Networks and Bidirectional-LSTM (BiLSTM) that consists of two blocks: the feature extraction stage based on BiLSTM and the abundance estimation stage via Chain Classifier. BiLSTM can capture the long-distance relation among spectral bands better and Chain Classifier is more suitable to deal with multi-label task that exists in abundance estimation process. We evaluate the proposed method on two real HSI data sets including Jasper Ridge and Urban compared with three state-of-the-art approaches, and the results show significant improvements in accuracy of unmixing.

In this paper, a global descriptor for a LiDAR point cloud, called LiDAR Iris, is proposed for fast and accurate loop-closure detection. A binary signature image can be obtained for each point cloud after several LoG-Gabor filtering and thresholding operations on the LiDAR-Iris image representation. Given two point clouds, their similarities can be calculated as the Hamming distance of two corresponding binary signature images extracted from the two point clouds, respectively. Our LiDAR-Iris method can achieve a pose-invariant loop-closure detection at a descriptor level with the Fourier transform of the LiDAR-Iris representation if assuming a 3D (x,y,yaw) pose space, although our method can generally be applied to a 6D pose space by re-aligning point clouds with an additional IMU sensor. Experimental results on five road-scene sequences demonstrate its excellent performance in loop-closure detection.

Spectral unmixing is an important task in hyperspectral image (HSI) analysis and processing. Sparse representation has become a promising semisupervised method for remotely sensed hyperspectral unmixing and incorporating the spectral or spatial information to improve the spectral unmixing results under a weighted sparse unmixing framework is a recent trend. While most methods focus on analyzing HSI by exploring the spatial information, it is known that hyperspectral data are characterized by its large contiguous set of wavelengths. This information can be naturally used to improve the representation of pixels in HSI. In order to take the advantage of the hyper spectral information as well as the spatial information for hyperspectral unmixing, in this article, we explore and introduce a multiview data processing approach through spectral partitioning to benefit from the abundant spectral information in HSI. Some important findings on the application of multiview data set in sparse unmixing are discussed. Meanwhile, we develop a new spectral–spatial-weighted multiview collaborative sparse unmixing (MCSU) model to tackle such a multiview data set. The MCSU uses a weighted sparse regularizer, which includes both multiview spectral and spatial weighting factors to further impose sparsity on the fractional abundances. The weights are adaptively updated associated with the abundances, and the proposed MCSU can be solved by the alternating direction method of multipliers efficiently. The experimental results on both the simulated and real hyperspectral data sets demonstrate the effectiveness of the proposed MCSU, which can significantly improve the abundance estimation results.

High-resolution (HR) magnetic resonance images (MRI) provide more detailed information for clinical application. However, HR MRI is less available because of the longer scan time and lower signal-to-noise ratio. Spatial resolution is one of the key parameters of MRI. The image post-processing technique super-resolution (SR) is an alternative approach to improve the spatial resolution of MR images. Inspired by advanced deep learning based SR methods, we propose an MRI SR model named progressive sub-band residual learning SR network (PSR-SRN). The proposed model contains two parallel progressive learning streams, where one stream learns on missed high-frequency residuals by sub-band residual learning unit (ISRL) and the other focuses on reconstructing refined MR image. These two streams complement each other and enable to learn complex mappings between "Low-" and "High-" resolution MR images. Besides, we introduce brain-like mechanisms (in-depth supervision and local feedback mechanism) and progressive sub-band learning strategy to emphasize variant textures of MRI. Compared with traditional and deep learning MRI SR methods, our PSR-SRN model shows superior performance.

In this work, we propose a novel paradigm to design a hybrid thermal/EO (Electro-Optical or visible-light) camera, whose thermal and RGB frames are pixel-wisely aligned and temporally synchronized. Compared with the existing schemes, we innovate in three ways in order to make it more compact in dimension, and thus more practical and extendable for real-world applications. The first is a redesign of the structure layout of the thermal and EO cameras. The second is on obtaining a pixel-wise spatial registration of the thermal and RGB frames by a coarse mechanical adjustment and a fine alignment through a constant homography warping. The third innovation is on extending one single hybrid camera to a hybrid camera array, through which we can obtain wide-view spatially aligned thermal, RGB and disparity images simultaneously. The experimental results show that the average error of spatial-alignment of two image modalities can be less than one pixel.

Hyperspectral imagery enhancing is very important to remote sensing interpretation. Recently, fusing hyperspectral data with its corresponding high spatial resolution multispectral data has been an important technology to obtain high-resolution imagery. Deep neural networks are widely used because of their significant performance, while they are still suffered from the problem of been insensitive to non-local information and the problem of overfitting. In this paper, a novel fusion method based on the non-local compressive network is proposed. This network can extract non-local information and also reduce overfitting when dealing with fusion tasks. Because of the introduction of non-local structure, the global relation information is also effectively enhanced according to the self-attention mechanism of features. Experiments on two datasets have shown that the proposed method can obtain better performance than the state-of-art methods.

Hyperspectral image (HSI) is characterized by its huge contiguous set of wavelengths. It is possible and needed to benefit from the “hyper” spectral information as well as the spatial information. For this purpose, we propose a new multiview data generation approach that takes full advantage of the rich spectral and spatial information in HSI, by dividing the original HSI into several spatially homogeneous regions with different band margins. Then, a new sparse unmixing algorithm, called region-based multiview sparse unmixing (RMSU), is presented to tackle such a multiview data model in this letter. The RMSU algorithm combines the multiview learning anda prioriinformation to improve the performance of sparse unmixing by incorporating the multiview information and spectral library into the dictionary learning framework. We also show that RMSU can serve as a dictionary pruning algorithm, which provides a possibility that unmixing algorithms could have higher accuracy and efficiency. Experimental results on both simulated and real hyperspectral data demonstrate the effectiveness of the proposed RMSU algorithm both visually and quantitatively.

This paper presents a novel Markov random field (MRF) method integrating adaptive interclass-pair penalty (aICP2) and spectral similarity information (SSI) for hyper-spectral image (HSI) classification. aICP2structurally combines$K(K - 1)/2$(K is the number of classes) classical “Potts model” with$K(K - 1)/2$interaction coefficients. aICP2tries a new way to solve the key problems, insufficient correction within homogeneous regions, and over-smoothness at class boundaries, in MRF-based HSI classification. It is assumed that different class pairs should be assigned with various degrees of penalties in MRF smoothness process, according to pairwise class separability and spatial class confusion in raw classification map. The Fisher ratio is modified to measure pairwise class separability with a training set. And, gray level co-occurrence matrix is used to measure spatial class confusion degree. Then, aICP2is constructed by combining Fisher ratio and GCLM. aICP2applies larger penalty on class pairs that confuse with each other seriously to provide sufficient smoothness, and vice versa. In addition, to protect class edges and details, SSI is introduced to make the penalty of related neighboring pixels small. aICP2ssi denotes the integration of aICP2and SSI. The further improved method is both interclass-pair and interpixel adaptive in spatial term. A graph-cut-based$\alpha - \beta $-swap method is introduced to optimize the proposed energy function. The experimental results on real HSI data indicate that the proposed method outperforms compared MRF-based and other spectral–spatial approaches in terms of classification accuracies and region uniformity.

Different spectral, spatial, and temporal features have been widely used in the remote-sensing image analysis. The further development of multiple sensor remote-sensing technologies has made it necessary to explore new methods of remote-sensing image fusion using different optical image data sets which provide complementary image properties and a tradeoff among spatial, spectral, and temporal resolutions. However, due to problems in assessing correlations between different types of satellite data with different resolutions, a few efforts have been made to explore spatio-spectral–temporal features. For this purpose, we propose a novel sparse representation model to generate synthesized frequent high-spectral and high-spatial resolution data by blending multiple types: spatio-temporal data fusion, spectral–temporal data fusion, spatio-spectral data fusion, and spatio-spectral–temporal data fusion. The proposed method exploits high-spectral correlation across spectral domains and high self-similarity across spatial domains to learn the spatio-spectral fusion basis. Then, it associates temporal changes using a local constraint sparse representation. The integrated spatio-spectral–temporal sparse representation model based on the learned spectral–spatial and temporal change features strengthens the model’s ability to provide high-resolution data needed to address demanding work in real-world applications. Finally, the proposed method is not restricted to a certain type of data, but it can associate any type of remote-sensing data and be applied to dynamic changes in heterogeneous landscapes. The experimental results illustrate the effectiveness and efficiency of the proposed method.

Linearly derived features have been widely used in hyperspectral image classification to find linear separability of certain classes in recent years. Moreover, nonlinearly transformed features are more effective for class discrimination in real analysis scenarios. However, few efforts have attempted to combine both linear and nonlinear features in the same framework even if they can demonstrate some complementary properties. Moreover, conventional multiple-feature learning-based approaches deal with different features equally, which is not reasonable. This paper proposes an efficient multiple-feature learning-based model with adaptive weights for effectively classifying complex hyperspectral images with limited training samples. A new diversity kernel function is proposed first to simulate the vision perception and analysis procedure of human beings. It could simultaneously evaluate the contrast differences of global features and spatial coherence. Since existing multiple-kernel feature models are always time-consuming, we then design a new adaptive weighted multiple kernel learning method. It employs kernel projection, which could lower the dimensionalities and also learn kernel weights to further discriminate the classification boundaries. For combining both linear and nonlinear features, this paper also proposes a novel decision fusion strategy. The method combines linear and multiple kernel features to balance the classification results of different classifiers. The proposed scheme is tested on several hyperspectral data sets and extended to multisource feature classification environment. The experimental results show that the proposed classification method outperforms most of the existing ones and significantly reduces the computational complexity.

Breast cancer has caused more and more attention in recent years since the mortality rate is increasing and age of onset is trend to be younger than before. Using computer vision technology for automatic classifying benign and masses malignant ones could assist doctors in diagnosing condition. However, the margins and shapes of masses are various and which are very similar with surrounding tissues, there are still a lot of problems in classification for which the extracted features couldn't express the original image very well. Hence, in this paper, a new mass feature extraction method is proposed for enhancing the performance of classification. First, the concentric circle bag-of-words (BOW) of the mass is captured spatially from the mammographic images. Then, the Dirichlet fisher kernel is conducted to enhance discrimination ability of the feature. Finally, the SVM classifier is employed for mass classification. Experiments are conducted on DDSM dataset and achieve good classification accuracy.

Mammographic mass detection is an important task for the early diagnosis of breast cancer. However, it is difficult to distinguish masses from normal regions because of their abundant morphological characteristics and ambiguous margins. To improve the mass detection performance, it is essential to effectively preprocess mammogram to preserve both the intensity distribution and morphological characteristics of regions. In this paper, morphological component analysis is first introduced to decompose a mammogram into a piecewise-smooth component and a texture component. The former is utilized in our detection scheme as it effectively suppresses both structural noises and effects of blood vessels. Then, we propose two novel concentric layer criteria to detect different types of suspicious regions in a mammogram. The combination is evaluated based on the Digital Database for Screening Mammography, where 100 malignant cases and 50 benign cases are utilized. The sensitivity of the proposed scheme is 99% in malignant, 88% in benign, and 95.3% in all types of cases. The results show that the proposed detection scheme achieves satisfactory detection performance and preferable compromises between sensitivity and false positive rates.

A new approach to mass detection in mammography is presented. The main obstacle of building a mass detection system is the similar appearance between masses and density tissues in breast. Hence, the various features of the extracted regions of interest (ROIs) are analyzed by synthesis. Then the support vector machine (SVM), which is designed later to distinguish masses from normal areas, is employed to classify these ROIs exactly. To further improve the performance of SVM, the relevance feedback (RF) is introduced to filter out the false positives. The experimental results illustrate that SVM classifier can effectively detect the mass areas, and the RF-SVM scheme can be efficiently incorporated into this learning framework to further improve detection performance.