Object detection in aerial imagery faces significant challenges caused by extreme scale variation, dense object layouts, and the prevalence of tiny instances. StandardIntersection over Union(IoU) and its variants are sensitive to small localization errors: a minor coordinate deviation may cause a large IoU penalty for tiny objects, and non-overlapping boxes yield no gradients, which limits localization accuracy. To address these issues, we propose ThermalIoU, a heat-diffusion-inspired bounding-box regression loss that represents rigid boxes as continuous diffusive fields. A box is treated as a flat-top initial temperature distribution and diffused by the Gaussian heat kernel, preserving its rectangular geometry while providing smooth boundary support. We further introduce a scale-decoupled and epoch-annealed diffusion parameter, so that small objects obtain broader early-stage gradients whereas larger objects retain sharper localization fields. Experiments on AI-TOD and VisDrone, together with ablations, efficiency measurements, and qualitative visualizations, show that ThermalIoU is a practical training loss with overall gains over CIoU and competitive performance against Inner-IoU and MPDIoU.

Recently, sharpness-aware minimization (SAM) has attracted much attention because of its surprising effectiveness in improving generalization performance. However, compared to stochastic gradient descent (SGD), it is more prone to getting stuck at the saddle points, which as a result may lead to performance degradation. To address this issue, we propose a simple renormalization strategy, dubbed Stable SAM (SSAM), so that the gradient norm of the descent step maintains the same as that of the ascent step. Our strategy is easy to implement and flexible enough to integrate with SAM and its variants, almost at no computational cost. With elementary tools from convex optimization and learning theory, we also conduct a theoretical analysis of sharpness-aware training, revealing that compared to SGD, the effectiveness of SAM is only assured in a limited regime of learning rate. In contrast, we show how SSAM extends this regime of learning rate and then it can consistently perform better than SAM with the minor modification. Finally, we demonstrate the improved performance of SSAM on several representative data sets and tasks.

Cross-domain classification of hyperspectral remote sensing images is one of the hotspots of research in recent years, and its main problem is insufficient training samples. To address this issue, few-shot learning (FSL) has emerged as a promising paradigm in cross-domain classification tasks. However, a notable limitation of most existing FSL methods is that they focus only on local information and less on the critical role of global information. Based on this, this paper proposes a new feature processing method with adaptive band selection, which takes into account the global nature of image features. Firstly, adaptive band analysis is performed in the target domain, and threshold analysis is used to determine the number of selected bands. Secondly, a band selection method is employed to select representative bands from the spectral bands of the high-dimensional data according to the determined band count. Finally, the weights of the selected bands are analyzed, fully considering the importance of pixel weight, and then the results are used as inputs for the classification model. The experimental results on various datasets show that this method can effectively improve the classification accuracy and generalization ability. Meanwhile, the results of the objective accuracy index of the proposed method in different databases improved by 3.9%, 4.7% and 5.4%.

Unsupervised domain adaptation (UDA) is crucial for semantic segmentation of remote sensing images (RS-SS), particularly when data distributions differ between source and target domains. Existing prototype-based UDA methods struggle with complex land cover class distributions and spatial information capture. To address these limitations, the Neighborhood-Assisted Multi-Prototype Group (NAPG) model is proposed. This model enhances cross-domain adaptability and spatial context richness by dynamically determining the number of prototype features and integrating neighborhood similarity gradients. Specifically, NAPG employs the Cross-Domain Representation of Multi-Prototype Group (CDR-MPG) module to generate multi-prototype group, capturing land cover complexity more effectively. Additionally, the Gradient Neighborhood Consistency Estimation (GNCE) module improves spatial representation by reducing intra-class variance and alleviating feature inconsistency. Experiments demonstrate that the proposed NAPG model outperforms state-of-the-art UDA methods across multiple datasets, achieving an mean intersection over union (mIoU) improvement of 3%. The source code is publicly available at https://github.com/Fanrongbo/NAPG-UDA-RS-SS.

Change detection (CD) aims to identify pixels with semantic changes between images. However, annotating massive numbers of pixel-level images is labor-intensive and costly, especially for multitemporal images, which require pixel-wise comparisons by human experts. Considering the excellent performance of visual-language models (VLMs) for zero-shot, OV, etc., with prompt-based reasoning, it is promising to utilize VLMs to make better CD under limited labeled data. In this article, we propose a VLM guidance-based semi-supervised CD method, namely SemiCD-VL. The insight of SemiCD-VL is to synthesize free change labels using VLMs to provide additional supervision signals for unlabeled data. However, almost all current VLMs are designed for single-temporal images and cannot be directly applied to bi- or multitemporal images. Motivated by this, we first propose a VLM-based mixed change event generation (CEG) strategy to yield pseudo-labels for unlabeled CD data. Since the additional supervised signals provided by these VLM-driven pseudo-labels may conflict with the original pseudo-labels from the consistency regularization paradigm (e.g., FixMatch), we propose the dual projection head for de-entangling different signal sources. Further, we explicitly decouple the bitemporal images semantic representation through two auxiliary segmentation decoders, which are also guided by VLM. Finally, to make the model more adequately capture change representations, we introduce contrastive consistency regularization (CCR) by constructing feature-level contrastive loss in auxiliary branches. Extensive experiments show the advantage of SemiCD-VL. For instance, SemiCD-VL improves the FixMatch baseline by$+ 5.3~\text {IoU}^{c}$on WHU-CD and by$+ 2.4~\text {IoU}^{c}$on LEVIR-CD with 5% labels, and SemiCD-VL requires only 5%–10% of the labels to achieve performance similar to the supervised methods. In addition, our CEG strategy, in an unsupervised manner, can achieve performance far superior to state-of-the-art (SOTA) unsupervised CD methods (e.g., IoU improved from 18.8% to 46.3% on LEVIR-CD dataset). The code is available athttps://github.com/likyoo/SemiCD-VL.

For infrared small target detection tasks, deep unfolding techniques have demonstrated effectiveness and practical value. However, existing methods generally emphasize the low-rankness of background and the sparsity of targets within the robust principal component analysis (RPCA) framework, which may overlook the intrinsic gradient prior information existed in background. To address the challenges of complex background estimation and accurate small target detection, we propose a gradient prior-guided deep unfolding network, termed the correlated total variation network (CTVNet). First, we introduce a correlated total variation regularization to simultaneously characterize the low-rankness and local smoothness of the background, and transform it into the estimation of gradient maps. Subsequently, we employ a multi-scale feature fusion network to thoroughly extract gradient priors, replacing the complex and limited analytical computation of gradient correlations. Finally, we unfold the designed iterative algorithm using alternating direction method of multipliers (ADMM) into a learnable network, where each module corresponds to a specific operator within the iterative process, and all parameters are learnable. By training the network end-to-end, the learnable modules can be automatically optimized to better separate the background and the target. Extensive experimental results demonstrate that our proposed method achieves competitive performance compared to several state-of-the-art algorithms while exhibiting superior performance and generalization capabilities on both in-distribution and out-of-distribution data. Our code is available at https://github.com/AuroraPei/CTVNet.

Pansharpening is a crucial task in remote sensing image processing, aiming to generate high-resolution multispectral (HRMS) images by fusing low-resolution multispectral (LRMS) images with high-resolution panchromatic (PAN) images. However, most current deep learning methods for pansharpening rarely consider the frequency differences in PAN and MS images effectively, resulting in harmful mixing of frequency information and inefficient learning of features. Furthermore, frequency separation-based methods continue to face challenges such as insufficient consideration of the relationship between frequency and spatial information, amplification of noise due to separation, and inadequate learning of frequency information. To address these problems, we propose a novel frequency separation and guidance Transformer, named FSGformer, which focuses on the differences and interactions between high- and low-frequency components. Specifically, we design an adaptive frequency separator tailored for pansharpening to effectively differentiate between distinct frequencies. Subsequently, we develop a carefully designed guidance module that enables the fusion process to benefit from the interaction of frequency information. In addition, we introduce a novel Transformer module that features a joint spatial and spectral attention mechanism and integrate it into a meticulously crafted network architecture to support the effective representation of different frequency information, thereby generating high-quality fused results. Moreover, we incorporate a hybrid frequency separation (HFS) loss to enhance overall performance. Extensive experimental evaluations have confirmed the superiority and generality of our FSGformer. The code is available athttps://github.com/lqrscode/FSGformer.

TheRemote Sensing Image Change Captioning(RSICC) technique is designed to enhance geospatial analysis by generating semantic descriptions of differences observed in bi-temporal remote sensing imagery. Although Transformer-based methods have achieved significant advancements in this field, their standard global attention mechanism allows pixels to pay equal attention to all spatial positions, which lacks explicit prior knowledge of spatial proximity correlation, making the model unable to strengthen local associations and weaken long-distance connections. Furthermore, most existing methods rely on global average pooling when reinforcing channel-wise representations, which often dilutes the features of key changed regions by the largely unchanged background, resulting in the inability of feature compression to focus on the critical positions. To address the aforementioned challenges, this paper proposes aDynamic Gaussian Attenuate Transformer(DGAT), which innovatively introduces aDynamic Gaussian Attenuation(DGA) mechanism to model an attenuation law between visual tokens from two perspectives based on Euclidean distance. At the spatial level, DGA constrains visual attention through distance-related Gaussian attenuation, allowing it to prioritize attention on adjacent regions, thereby enhancing the detection of changes in local continuity; at the channel level, DGA identifies the core area based on the visual attention kernel, and uses the joint optimization of dynamic Gaussian weighted pooling and channel modulation to focus on features at key positions, effectively enhancing the expression of important channels. Experimental results on three benchmark RSICC datasets demonstrate that our proposed DGAT achieves significantly superior results, verifying the effectiveness of the DGA mechanism. Our code and model weights are available at https://github.com/Pengfei1005/DGAT.

Lane detection is a critical and challenging task in autonomous driving, particularly in real-world scenarios where traffic lanes can be slender, lengthy, and often obscured by other vehicles, complicating detection efforts. Existing anchor-based methods typically rely on prior lane anchors to extract features and subsequently refine the location and shape of lanes. While these methods achieve high performance, manually setting prior anchors is cumbersome, and ensuring sufficient coverage across diverse datasets often requires a large amount of dense anchors. Furthermore, the use ofNon-Maximum Suppression(NMS) to eliminate redundant predictions complicates real-world deployment and may under-perform in complex scenarios. In this paper, we proposePolar R-CNN, an end-to-end anchor-based method for lane detection. By incorporating both local and global polar coordinate systems, Polar R-CNN facilitates flexible anchor proposals and significantly reduces the number of anchors required without compromising performance. Additionally, we introduce a triplet head with heuristic structure that supports NMS-free paradigm, enhancing deployment efficiency and performance in scenarios with dense lanes. Our method achieves competitive results on five popular lane detection benchmarks—Tusimple,CULane,LLAMAS,CurveLanes, andDL-Rail—while maintaining a lightweight design and straightforward structure. Our source code is available athttps://github.com/ShqWW/PolarRCNN

In data mining, subspace clustering is a crucial technique which determines the union of the underlying subspace to cluster data points in an unsupervised manner. Although deep-learning-based subspace clustering, typically referred to asdeep subspace clustering(DSC), has significantly improved clustering accuracy, existing DSC models still struggle to capture a comprehensive and compact latent representation as they generally explore the spatial domain to extract useful information and face difficulty in balancing the high mutual and low redundant information between the original input space and latent subspace. This leads to the performance of the model being dependent on initialization, resulting in a lack of stability. In this study, a novel network is proposed to extract features in both the frequency domain and spatial domain. We introduce three types of ResBlocks in thediscrete Fourier transform(DFT),discrete cosine transform(DCT), ordiscrete wavelet transform(DWT) frequency domains separately to learn both the low-frequency and high-frequency information in the proposed networks. Additionally, to extract concise and rich latent representations, IB loss is employed by deriving a variational lower bound on the IB objective. Extensive experiments on several benchmark datasets verify the effectiveness of our networks compared to state-of-the-art models. In addition, detailed ablation studies are performed to demonstrate the advantages of the two introduced components.

Recently, view-based approaches, which recognize a 3D object through its projected 2-D images, have been extensively studied and have achieved considerable success in 3D object recognition. Nevertheless, most of them use a pooling operation to aggregate viewwise features, which usually leads to the visual information loss. To tackle this problem, we propose a novel layer called capsule attention layer (CAL) by using attention mechanism to fuse the features expressed by capsules. In detail, instead of dynamic routing algorithm, we use an attention module to transmit information from the lower level capsules to higher level capsules, which obviously improves the speed of capsule networks. In particular, the view pooling layer of multiview convolutional neural network (MVCNN) becomes a special case of our CAL when the trainable weights are chosen on some certain values. Furthermore, based on CAL, we propose a capsule attention convolutional neural network (CACNN) for 3D object recognition. Extensive experimental results on three benchmark datasets demonstrate the efficiency of our CACNN and show that it outperforms many state-of-the-art methods.

This article introduces, to the best of our knowledge, a novel recommendation framework called Cold-start Recommendation based on Knowledge Graph and Meta-learning (CRKM), aimed at enhancing cold-start recommendation performance by addressing the issue of limited user interaction data through the fusion of positive and negative samples. In contrast to other cold-start frameworks, CRKM is divided into three distinct components: the negative sampler, the knowledge graph-based model architecture, and the meta-learner. The negative sampler designed in this article leverages knowledge graphs and popularity information to sample negative labels from items without prior user interaction, thereby mitigating the sparsity of cold-start training data. However, the knowledge graph-based model architecture is responsible for incorporating the nodes and relationships of the knowledge graph into positive and negative samples, using a graph neural network to more effectively learn user and item fusion representations and enhance predictive performance. Finally, the meta-learner performs efficient model initialization parameter updates. We conducted extensive experiments on real-world datasets for cold-start user and item recommendations. CRKM demonstrated notable performance advantages in terms of recall and NDCG when compared to the state-of-the-art methods, thereby validating the rationality and effectiveness of the proposed approach. The source code listing is publicly available at https://gitee.com/kyle-liao/crkm .

Lookahead is a popular stochastic optimizer that can accelerate the training process of deep neural networks. However, the solutions found by Lookahead often generalize worse than those found by its base optimizers, such as SGD and Adam. To address this issue, we propose Sharpness-Aware Lookahead (SALA), a novel optimizer that aims to identify flat minima that generalize well. SALA divides the training process into two stages. In the first stage, the direction towards flat regions is determined by leveraging a quadratic approximation of the optimization trajectory, without incurring any extra computational overhead. In the second stage, however, it is determined by Sharpness-Aware Minimization (SAM), which is particularly effective in improving generalization at the terminal phase of training. In contrast to Lookahead, SALA retains the benefits of accelerated convergence while also enjoying superior generalization performance compared to the base optimizer. Theoretical analysis of the expected excess risk, as well as empirical results on canonical neural network architectures and datasets, demonstrate the advantages of SALA over Lookahead. It is noteworthy that with approximately 25% more computational overhead than the base optimizer, SALA can achieve the same generalization performance as SAM which requires twice the training budget of the base optimizer.

The emergence of generative models has revolutionized the field of remote sensing (RS) image generation. Despite generating high-quality images, existing methods are limited in relying mainly on text control conditions, and thus do not always generate images accurately and stably. In this article, we propose CRS-Diff, a new RS generative framework specifically tailored for RS image generation, leveraging the inherent advantages of diffusion models while integrating more advanced control mechanisms. Specifically, CRS-Diff can simultaneously support text-condition, metadata-condition, and image-condition control inputs, thus enabling more precise control to refine the generation process. To effectively integrate multiple condition control information, we introduce a new conditional control mechanism to achieve multiscale feature fusion (FF), thus enhancing the guiding effect of control conditions. To the best of our knowledge, CRS-Diff is the first multiple-condition controllable RS generative model. Experimental results in single-condition and multiple-condition cases have demonstrated the superior ability of our CRS-Diff to generate RS images both quantitatively and qualitatively compared with previous methods. Additionally, our CRS-Diff can serve as a data engine that generates high-quality training data for downstream tasks, e.g., road extraction. The code is available athttps://github.com/Sonettoo/CRS-Diff.

Since the weight of a self-attention model is not affected by the sequence interval, it can more accurately and completely describe the user interests, so it is widely used in processing sequential recommendation. However, the mainstream self-attention model focuses on the similarity between items when calculating the attention weight of user behavioral patterns but fails to reflect the impact of user sudden drift decisions on the model in time. In this article, we introduce a bias strategy in the self-attention module, referred to as Learning Self-Attention Bias (LSAB) to more accurately learn the fast-changing user behavioral patterns. The introduction of LSAB allows for the adjustment of bias resulting from self-attention weights, leading to enhanced prediction performance in sequential recommendation. In addition, this article designs four attention-weight bias types catering to diverse user behavior preferences. After testing on the benchmark datasets, each bias strategy in LSAB is useful for state-of-the-art and can improve the performance of the models by nearly 5% on average. The source code listing is publicly available at https://gitee.com/kyle-liao/lsab .

Stochastic gradient descent (SGD) is of fundamental importance in deep learning. Despite its simplicity, elucidating its efficacy remains challenging. Conventionally, the success of SGD is ascribed to the stochastic gradient noise (SGN) incurred in the training process. Based on this consensus, SGD is frequently treated and analyzed as the Euler-Maruyama discretization of stochastic differential equations (SDEs) driven by either Brownian or Lévy stable motion. In this study, we argue that SGN is neither Gaussian nor Lévy stable. Instead, inspired by the short-range correlation emerging in the SGN series, we propose that SGD can be viewed as a discretization of an SDE driven by fractional Brownian motion (FBM). Accordingly, the different convergence behavior of SGD dynamics is well-grounded. Moreover, the first passage time of an SDE driven by FBM is approximately derived. The result suggests a lower escaping rate for a larger Hurst parameter, and thus, SGD stays longer in flat minima. This happens to coincide with the well-known phenomenon that SGD favors flat minima that generalize well. Extensive experiments are conducted to validate our conjecture, and it is demonstrated that short-range memory effects persist across various model architectures, datasets, and training strategies. Our study opens up a new perspective and may contribute to a better understanding of SGD.

Deep learning currently rules edge detection. However, the impressive progress heavily relies on high-quality manually annotated labels which require a significant amount of labor and time. In this study, we propose a novel unsupervised learning framework for deep edge detection. It adopts a gradient-based method to generate scale-dependent pseudo edge maps, which match with the hierarchical structure of deep networks. It leverages both the representation learning capability of deep learning, and the simplicity of traditional methods. Experiments on three popular data sets show that the proposed method can suppress non-object edges and reduce the gap with its supervised counterpart due to the introduction of information of various scales and smoothing strategy.

Pansharpening aims at integrating a high-spatial-resolution panchromatic (PAN) image with a low-spatial-resolution multispectral (MS) image to generate a high-resolution MS (HRMS) image. It is a fundamental and significant task in the field of remotely sensed images. Classic and convolutional neural network (CNN)-based algorithms have been developed, over the last decades, for pansharpening based on the spatial detail injection model. However, these algorithms have difficulties in extracting sufficient details or lack interpretability. In this letter, we present an algorithm unfolding pansharpening (AUP) for this task. In the proposed AUP, a two-step optimization model is first designed based on the spatial detail decomposition model. Then, the iteration processes induced by an optimization model are mapped to several detailed convolution (dc) blocks to solve the detail injection by a trainable neural network. Finally, the desired MS details are obtained in end-to-end manners through a decoder. The superiority of the proposed AUP is demonstrated by extensive experiments on datasets acquired by two different kinds of satellites. Each module of the AUP is interpretable, and its fused results are with fewer spectral and spatial distortions.

Infrared and visible image fusion (IVIF) expects to obtain images that retain thermal radiation information from infrared images and texture details from visible images. In this paper, a model-based convolutional neural network (CNN) model, referred to as Algorithm Unrolling Image Fusion (AUIF), is proposed to overcome the shortcomings of traditional CNN-based IVIF models. The proposed AUIF model starts with the iterative formulas of two traditional optimization models, which are established to accomplish two-scale decomposition, i.e., separating low-frequency base information and high-frequency detail information from source images. Then the algorithm unrolling is implemented where each iteration is mapped to a CNN layer and each optimization model is transformed into a trainable neural network. Compared with the general network architectures, the proposed framework combines the model-based prior information and is designed more reasonably. After the unrolling operation, our model contains two decomposers (encoders) and an additional reconstructor (decoder). In the training phase, this network is trained to reconstruct the input image. While in the test phase, the base (or detail) decomposed feature maps of infrared/visible images are merged respectively by an extra fusion layer, and then the decoder outputs the fusion image. Qualitative and quantitative comparisons demonstrate the superiority of our model, which can robustly generate fusion images containing highlight targets and legible details, exceeding the state-of-the-art methods. Furthermore, our network has fewer weights and faster speed.

Object detection is a fundamental and important task in the analysis ofremote sensing images(RSIs), and existing deep learning-based object detection models in this literature strongly rely on predefined anchor boxes and encounter redesigned difficulties related to anchors. In addition, they often ignore the scene-contextual information that objects are usually closely related to their surrounding scene. To deal with these problems, we propose an anchor-free network, referred to asscene-relevant anchor-free network(SRAF-Net), for object detection in RSIs. The SRAF-Net first captures the scene-contextual features of objects by using a designedscene-enhanced feature pyramid network(SE-FPN) and then performs more accurate detection by implementing ascene auxiliary detection head(SADH), which can predict the existence of the objects with the help of the scene-contextual features extracted from the SE-FPN. To deal with insufficient scene diversity in the training stage, a simple yet effective data augmentation module, termedbalanced mixup data augment(BMDA), is introduced by linearly expanding the training dataset to improve the generalization of SRAF-Net. Comprehensive experiments on three publicly available challenging remote sensing datasets demonstrate the effectiveness of the proposed method. The codes will be made publicly available athttps://github.com/Complicateddd/SRAF-Net.

Pansharpening is a special image fusion task of reconstructing a high-resolution multispectral (HRMS) image by integrating a panchromatic (PAN) image of high spatial resolution and a low-resolution multispectral (LRMS) image. To handle such an ill-posed multi-modal fusion task, in this paper, we propose a novel pansharpening method, referred to as model-driven and data-driven network (MD3Net), which combines model-driven and data-driven approaches. The architecture design of MD3Net is inspired from the traditional model constructed based on domain knowledge and thus making its network topology explainable and its input/output predictable. In order to further explore the powerful learning ability of deep learning based approaches, we introduce the deep prior into the MD3Net as its implicit regularization, thus improving its data adaptability and representation capability. Comprehensive experiments conducted on both reduced and full resolution of several acknowledged datasets have qualitatively and quantitatively verified the superiority of our network compared to a benchmark consisting of several state-of-the-art approaches. The code can be downloaded from https://github.com/YinsongYan/M3DNet..

Pan-sharpening is an important technique for remote sensing imaging systems to obtain high resolution multi-spectral images. Recently, deep learning has become the most popular tool for pan-sharpening. This paper develops a model-based deep pan-sharpening approach. Specifically, two optimization problems regularized by the deep prior are formulated, and they are separately responsible for the generative models for panchromatic images and low resolution multispectral images. Then, the two problems are solved by a gradient projection algorithm, and the iterative steps are generalized into two network blocks. By alternatively stacking the two blocks, a novel network, called gradient projection based pan-sharpening neural network, is constructed. The experimental results on different kinds of satellite datasets demonstrate that the new network out-performs state-of-the-art methods both visually and quantitatively. The codes are available at https://github.com/xsxjtu/GPPNN.

Pansharpening is a widely used image enhancement technique for remote sensing. Its principle is to fuse the input high-resolution single-channel panchromatic (PAN) image and low-resolution multi-spectral image and to obtain a high-resolution multi-spectral (HRMS) image. The existing deep learning pansharpening method has two shortcomings. First, features of two input images need to be concatenated along the channel dimension to reconstruct the HRMS image, which makes the importance of PAN images not prominent, and also leads to high computational cost. Second, the implicit information of features is difficult to extract through the manually designed loss function. To this end, we propose a generative adversarial network via the fast guided filter (FGF) for pansharpening. In generator, traditional channel concatenation is replaced by FGF to better retain the spatial information while reducing the number of parameters. Meanwhile, the fusion objects can be highlighted by the spatial attention module. In addition, the latent information of features can be preserved effectively through adversarial training. Numerous experiments illustrate that our network generates high-quality HRMS images that can surpass existing methods, and with fewer parameters.

Pansharpening is a fundamental issue in remote sensing field. This paper proposes a side information partially guided convolutional sparse coding (SCSC) model for pansharpening. The key idea is to split the low resolution multispectral image into a panchromatic image related feature map and a panchromatic image irrelated feature map, where the former one is regularized by the side information from panchromatic images. With the principle of algorithm unrolling techniques, the proposed model is generalized as a deep neural network, called as SCSC pansharpening neural network (SCSC-PNN). Compared with 13 classic and state-of-the-art methods on three satellites, the numerical experiments show that SCSC-PNN is superior to others. The codes are available at https://github.com/xsxjtu/SCSC-PNN.

At present, most of the research on recommender system (RS) based on artificial intelligence focus on the algorithm, but the evaluation metrics being an important process to evaluate the performance of RS are usually ignored. Specifically, independent evaluation metrics cannot effectively reflect the differences between algorithms, so how to effectively couple these evaluation metrics needs further improvement. In order to reflect the difference in RS performance, this paper proposes a rational evaluation framework for RS performance, named AND, which can reflect metrics of accuracy, novelty and diversity. Through comparative experiments, it is verified that the proposed framework, on the basis of the hypothetical model and the state-of-the-art algorithms, can effectively reflect the difference between the recommendation performance with similar seemingly accuracy between different algorithms.

The deep learning method, especially convolution neural networks (CNNs), has recently made ground-breaking advances on object detection in very-high-resolution (VHR) optical remote sensing images. However, as CNN is originally designed for the classification of natural images, these methods are not very suitable for object detection of remote sensing images. First, current CNN-based approaches have difficulty to deal with objects that have large rotation variation, which is widely existed in optical remote sensing images. Second, the detectors based on CNN only have limited receptive fields and, thus, can hardly utilize the global contextual information that is essential for accurate detection of small targets. To address these two issues, this article proposes a novel deep learning-based object detection framework, including a geometric transform module (GTM) and a global contextual feature fusion module (GCFM). Especially, the GTM combines rotation and flip transformation to deal with the multiangle characteristics of objects. The GCFM uses a spatial attention mechanism to adaptively involve global contextual information in feature maps to improve the recognition and location accuracy of targets. We introduce the two modules into the YOLOv3 framework to achieve end-to-end detection with high performance and efficiency. Comprehensive evaluations on three publicly available object detection data sets demonstrate the excellent performance of the proposed methods.

3D object recognition is one of the most important tasks in 3D data processing, and has been extensively studied recently. Researchers have proposed various 3D recognition methods based on deep learning, among which a class of view-based approaches is a typical one. However, in the view-based methods, the commonly used view pooling layer to fuse multi-view features causes a loss of visual information. To alleviate this problem, in this paper, we construct a novel layer called Dynamic Routing Layer (DRL) by modifying the dynamic routing algorithm of capsule network, to more effectively fuse the features of each view. Concretely, in DRL, we use rearrangement and affine transformation to convert features, then leverage the modified dynamic routing algorithm to adaptively choose the converted features, instead of ignoring all but the most active feature in view pooling layer. We also illustrate that the view pooling layer is a special case of our DRL. In addition, based on DRL, we further present a Dynamic Routing Convolutional Neural Network (DRCNN) for multi-view 3D object recognition. Our experiments on three 3D benchmark datasets show that our proposed DRCNN outperforms many state-of-the-arts, which demonstrates the efficacy of our method.

Infrared and visible image fusion, a hot topic in the field of image processing, aims at obtaining fused images keeping the advantages of source images. This paper proposes a novel auto-encoder (AE) based fusion network. The core idea is that the encoder decomposes an image into background and detail feature maps with low- and high-frequency information, respectively, and that the decoder recovers the original image. To this end, the loss function makes the background/detail feature maps of source images similar/dissimilar. In the test phase, background and detail feature maps are respectively merged via a fusion module, and the fused image is recovered by the decoder. Qualitative and quantitative results illustrate that our method can generate fusion images containing highlighted targets and abundant detail texture information with strong reproducibility and meanwhile surpass state-of-the-art (SOTA) approaches.

The fusion of hyperspectral image (HSI) and multispectral image (MSI) is one of the most significant topics in remote sensing image processing. Recently, deep learning (DL) has emerged as an important tool for this task. However, existing DL-based methods have two drawbacks, that is, limited ability for feature extraction and suffering from spectral distortion. To address these issues, this article presents a novel neural network, where sophisticated techniques are employed, including network-in-network convolutional unit, batch normalization, and skip connection. To make full use of the MSI, the proposed model fuses HSI and MSI at different scales. Besides, this article presents a new loss function, called RMSE, angle and Laplacian (RAP) loss (the combination of the relative mean squared error, angle loss, and Laplacian loss), to deal with both spatial and spectral distortions. Experiments conducted on four data sets have verified the rationality of network structure and the proposed loss function and demonstrated that the proposed novel model outperforms state-of-the-art counterparts.

In the literature of object detection in optical remote sensing images, a popular pipeline is first modifying an off-the-shelf deep neural network, then initializing the modified network by pretrained weights on a source data set, and finally fine-tuning the network on a target data set. The procedure works well in practice but might not make full use of underlying knowledge implied by pretrained weights. In this article, we propose a novel method, referred to as Fisher regularization, for efficient knowledge transferring. Based on Bayes' theorem, the method stores underlying knowledge into a Fisher information matrix and fine-tunes parameters based on the knowledge. The proposed method would not introduce extra parameters and is less sensitive to hyperparameters than classical weight decay. Experiments on NWPUVHR-10 and DOTA data sets show that the proposed method is effective and works well with different object detectors.

Endmember is defined as the spectral signature of pure material present in hyperspectral imagery. Estimation of the number of endmembers (NOE) present in a scene is an important preprocessing step and plays a crucial role in hyperspectral image processing, since over- or under-estimation of the NOE will lead to heavily incorrect results. In this article, we develop a thresholding ridge ratio (TRR) criterion based on eigendecomposition for NOE determination. Different from the widely used eigenvalue difference analysis methods, the TRR seeks an adaptive thresholding operation to the ridge ratio of eigenvalue differences, and ridge ratio combined with adaptive thresholding can theoretically guarantee a consistent estimate even when there are several local minima. Based on the TRR criterion, an algorithm is introduced to perform the estimation of NOE. Experimental results on both the simulated and real hyperspectral data sets have demonstrated that the proposed TRR-based algorithm has comparable and even better performances to several benchmark algorithms in the estimation accuracy of the NOE.

In this letter, we address the pansharpening problem, which focuses on constructing a high-resolution (HR) multispectral (MS) image from a low-resolution (LR) MS and an HR panchromatic (Pan) image. The accuracy of pansharpening method based on sparse representation (SR) mainly depends on the construction of dictionary and the learning of sparse coefficients, while the details injection (DI)-based pansharpening method sharpens the MS bands by adding the proper spatial details from Pan. The combination of SR and DI has been put forward as the pansharpening method based on SR of injected details (SR-D). However, limited to the patch-based manner, pansharpening with traditional SR model faces two disadvantages, i.e., limited ability in detail preservation and high sensitivity to misregistration. In this letter, we replace the traditional SR model with convolutional SR (CSR) as a global SR model in the SR-D method and propose a new pansharpening method called CSR of injected details (CSR-D) to overcome the above-mentioned two drawbacks. Experimental results on the IKONOS and WorldView2 data sets show that the proposed method can achieve remarkable spectral and spatial quality on both reduced scale and full scale.

As a biomimetic model of visual information processing, predictive coding (PC) has become increasingly popular for explaining a range of neural responses and many aspects of brain organization. While the development of PC model is encouraging in the neurobiology community, its practical applications in machine learning (e.g., image classification) have not been fully explored yet. In this paper, a novel image processing model called fast inference PC (FIPC) is presented for image representation and classification. Compared with the basic PC model, a regression procedure and a classification layer have been added to the proposed FIPC model. The regression procedure is used to learn regression mappings that achieve fast inference at test time, while the classification layer can instruct the model to extract more discriminative features. In addition, effective learning and fine-tuning algorithms are developed for the proposed model. Experimental results obtained on four image benchmark data sets show that our model is able to directly and fast infer representations and, simultaneously, produce lower error rates on image classification tasks.

The recently developed theory of compressive sensing (CS) exhibits enormous potentials in signal recovery. In this paper, we investigate its application on spectral unmixing, which appears in hyperspectral data analysis and is usually based on a linear mixture model (LMM) that assumes that a mixed pixel is a linear combination of a set of pure spectral signatures (called endmembers) weighted by their corresponding abundances. Unlike the classical LMM that is a compact representation, we first extend it to a sparse representation (SR) by using a redundant and known endmember set instead of the complete one. Then, the SR model is multiplied by a random Gaussian measurement matrix, so spectral unmixing is casted in the framework of CS. Finally, the ℓ1-minimization algorithms are used to recover the nonnegative abundances by solving the SR model and our proposed model named CS+SR, respectively. Experimental results on both simulated and real hyperspectral data demonstrate that the CS+SR model, formed by multiplying a random Gaussian matrix on the SR model, can improve, at least in the sense of probability, the ability of the ℓ1-minimization algorithms for recovering the nonnegative sparse abundances.

Recently, low-rank representation (LRR) method has achieved great success in subspace clustering (SC), which aims to cluster the data points that lie in a union of low-dimensional subspace. Given a set of data points, LRR seeks the lowest rank representation among the many possible linear combinations of the bases in a given dictionary or in terms of the data itself. However, LRR only considers the global Euclidean structure, while the local manifold structure, which is often important for many real applications, is ignored. In this paper, to exploit the local manifold structure of the data, a manifold regularization characterized by a Laplacian graph has been incorporated into LRR, leading to our proposed Laplacian regularized LRR (LapLRR). An efficient optimization procedure, which is based on alternating direction method of multipliers (ADMM), is developed for LapLRR. Experimental results on synthetic and real data sets are presented to demonstrate that the performance of LRR has been enhanced by using the manifold regularization.

Endmember extraction is very important in hyperspectral image analysis. The accurate identification of endmembers enables target detection and classification and efficient spectral unmixing. Although a number of endmember extraction algorithms have been proposed, such as two state-of-the-art algorithms—vertex component analysis (VCA) and simplex growing algorithm (SGA)—it is still a rather challenging task. In this paper, a new maximum simplex volume method based on Householder transformation (HT), referred to as maximum volume by HT (MVHT), is presented for endmember extraction. The proposed algorithm provides consistent results with low computational complexity, which overcomes the disadvantage of the inconsistent result of VCA and the shortcoming of the high computational cost of SGA resulted from calculating the simplex volume. A comparative study and analysis are conducted among the three endmember extraction algorithms, VCA, SGA, and MVHT, on both simulated and real hyperspectral data. The obtained experimental results demonstrate that the proposed MVHT algorithm generally provides a competitive or even better performance over VCA and SGA.

UNLABELLED: Computational methods in molecular biology will increasingly depend on standards-based annotations that describe biological experiments in an unambiguous manner. Annotare is a software tool that enables biologists to easily annotate their high-throughput experiments, biomaterials and data in a standards-compliant way that facilitates meaningful search and analysis. AVAILABILITY AND IMPLEMENTATION: Annotare is available from http://code.google.com/p/annotare/ under the terms of the open-source MIT License (http://www.opensource.org/licenses/mit-license.php). It has been tested on both Mac and Windows.

BACKGROUND: Sharing of microarray data within the research community has been greatly facilitated by the development of the disclosure and communication standards MIAME and MAGE-ML by the MGED Society. However, the complexity of the MAGE-ML format has made its use impractical for laboratories lacking dedicated bioinformatics support. RESULTS: We propose a simple tab-delimited, spreadsheet-based format, MAGE-TAB, which will become a part of the MAGE microarray data standard and can be used for annotating and communicating microarray data in a MIAME compliant fashion. CONCLUSION: MAGE-TAB will enable laboratories without bioinformatics experience or support to manage, exchange and submit well-annotated microarray data in a standard format using a spreadsheet. The MAGE-TAB format is self-contained, and does not require an understanding of MAGE-ML or XML.

SUMMARY: Searching for differentially expressed genes is one of the most common applications for microarrays, yet statistically there are difficult hurdles to achieving adequate rigor and practicality. False discovery rate (FDR) approaches have become relatively standard; however, how to define and control the FDR has been hotly debated. Permutation estimation approaches such as SAM and PaGE can be effective; however, they leave much room for improvement. We pursue the permutation estimation method and describe a convenient definition for the FDR that can be estimated in a straightforward manner. We then discuss issues regarding the choice of statistic and data transformation. It is impossible to optimize the power of any statistic for thousands of genes simultaneously, and we look at the practical consequences of this. For example, the log transform can both help and hurt at the same time, depending on the gene. We examine issues surrounding the SAM 'fudge factor' parameter, and how to handle these issues by optimizing with respect to power.

MOTIVATION: Gene expression array technology has become increasingly widespread among researchers who recognize its numerous promises. At the same time, bench biologists and bioinformaticians have come to appreciate increasingly the importance of establishing a collaborative dialog from the onset of a study and of collecting and exchanging detailed information on the many experimental and computational procedures using a structured mechanism. This is crucial for adequate analyses of this kind of data. RESULTS: The RNA Abundance Database (RAD; http://www.cbil.upenn.edu/RAD) provides a comprehensive MIAME-supportive infrastructure for gene expression data management and makes extensive use of ontologies. Specific details on protocols, biomaterials, study designs, etc. are collected through a user-friendly suite of web annotation forms. Software has been developed to generate MAGE-ML documents to enable easy export of studies stored in RAD to any other database accepting data in this format (e.g. ArrayExpress). RAD is part of a more general Genomics Unified Schema (http://www.gusdb.org), which includes a richly annotated gene index (http://www.allgenes.org), thus providing a platform that integrates genomic and transcriptomic data from multiple organisms. This infrastructure enables a large variety of queries that incorporate visualization and analysis tools and have been tailored to serve the specific needs of projects focusing on particular organisms or biological systems.