Nucleus detection and classification (NDC) in histopathology analysis is a fundamental task that underpins a wide range of high-level pathology applications. However, existing methods heavily rely on labor-intensive nucleus-level annotations and struggle to fully exploit large-scale unlabeled data for learning discriminative nucleus representations. In this work, we propose MUSE (MUlti-scale denSE self-distillation), a novel self-supervised learning method tailored for NDC. At its core is NuLo (Nucleus-based Local self-distillation), a coordinate-guided mechanism that enables flexible local self-distillation based on predicted nucleus positions. By removing the need for strict spatial alignment between augmented views, NuLo allows critical cross-scale alignment, thus unlocking the capacity of models for fine-grained nucleus-level representation. To support MUSE, we design a simple yet effective encoder-decoder architecture and a large field-of-view semi-supervised fine-tuning strategy that together maximize the value of unlabeled pathology images. Extensive experiments on three widely used benchmarks demonstrate that MUSE effectively addresses the core challenges of histopathological NDC. The resulting models not only surpass state-of-the-art supervised baselines but also outperform generic pathology foundation models.

Prompting is fundamental to unlocking the full potential of large language models. To automate and enhance this process, automatic prompt optimization (APO) has been developed, demonstrating effectiveness primarily in text-only input scenarios. However, extending existing APO methods to multimodal tasks—such as video-language generation—introduces two core challenges: (i) visual token inflation, where long visual-token sequences restrict context capacity and result in insufficient feedback signals; (ii) a lack of process-level supervision, as existing methods focus on outcome-level supervision and overlook intermediate supervision, limiting prompt optimization. We present UniAPO: Unified Multimodal Automated Prompt Optimization, the first framework tailored for multimodal APO. UniAPO adopts an EM-inspired optimization process that decouples feedback modeling and prompt refinement, making the optimization more stable and goal-driven. To further address the aforementioned challenges, we introduce a short-long term memory mechanism: historical feedback mitigates context limitations, while historical prompts provide directional guidance for effective prompt optimization. UniAPO achieves consistent gains across text, image, and video benchmarks, establishing a unified framework for efficient and transferable prompt optimization.

Low-dose computed tomography (CT) denoising is crucial for reduced radiation exposure while ensuring diagnostically acceptable image quality. Despite significant advancements driven by deep learning (DL) in recent years, existing DL-based methods, typically trained on a specific dose level and anatomical region, struggle to handle diverse noise characteristics and anatomical heterogeneity during varied scanning conditions, limiting their generalizability and robustness in clinical scenarios. In this paper, we propose FoundDiff, a foundational diffusion model for unified and generalizable LDCT denoising across various dose levels and anatomical regions. FoundDiff employs a two-stage strategy: (i) dose-anatomy perception and (ii) adaptive denoising. First, we develop a dose- and anatomy-aware contrastive language-image pre-training model (DA-CLIP) to achieve robust dose and anatomy perception by leveraging specialized contrastive learning strategies to learn continuous representations that quantify ordinal dose variations and identify salient anatomical regions. Second, we design a dose- and anatomy-aware diffusion model (DA-Diff) to perform adaptive and generalizable denoising by synergistically integrating the learned dose and anatomy embeddings from DA-CLIP into diffusion process via a novel dose and anatomy conditional block (DACB) based on Mamba. Extensive experiments on a large simulated multi-dose CT dataset spanning three anatomical regions, together with cross-dataset evaluations on Mayo-2016, CQ500, and piglet datasets, demonstrate superior denoising performance and strong generalization to unseen dose levels and anatomical regions. The codes and models are available at https://github.com/hao1635/FoundDiff.

Depression is a common mental disorder that affects millions of people worldwide. Although promising, current multimodal methods hinge on aligned or aggregated multi-modal fusion, suffering two significant limitations: (i) inefficient long-range temporal modeling, and (ii) sub-optimal multimodal fusion between intermodal fusion and intramodal processing. In this paper, we propose an audio-visual progressive fusion Mamba for multimodal depression detection, termed DepMamba. DepMamba features two core designs: hierarchical contextual modeling and progressive multimodal fusion. On the one hand, hierarchical modeling introduces convolution neural networks and Mamba to extract the local-to-global features within long-range sequences. On the other hand, the progressive fusion first presents a multimodal collaborative State Space Model (SSM) extracting intermodal and intramodal information for each modality, and then utilizes a multimodal enhanced SSM for modality cohesion. Extensive experimental results on two large-scale depression datasets demonstrate the superior performance of our DepMamba over existing state-of-the-art methods. Code is available at https://github.com/Jiaxin-Ye/DepMamba.

While contrastive multi-view clustering has achieved remarkable success, it implicitly assumes balanced class distribution. However, real-world multi-view data primarily exhibits class imbalance distribution. Consequently, existing methods suffer performance degradation due to their inability to perceive and model such imbalance. To address this challenge, we present the first systematic study of imbalanced multi-view clustering, focusing on two fundamental problems: i. perceiving class imbalance distribution, and ii. mitigating representation degradation of minority samples. We propose PROTOCOL, a novel PaRtial Optimal TranspOrt-enhanced COntrastive Learning framework for imbalanced multi-view clustering. First, for class imbalance perception, we map multi-view features into a consensus space and reformulate the imbalanced clustering as a partial optimal transport (POT) problem, augmented with progressive mass constraints and weighted KL divergence for class distributions. Second, we develop a POT-enhanced class-rebalanced contrastive learning at both feature and class levels, incorporating logit adjustment and class-sensitive learning to enhance minority sample representations. Extensive experiments demonstrate that PROTOCOL significantly improves clustering performance on imbalanced multi-view data, filling a critical research gap in this field.

Accurate localization is essential for the safe and effective navigation of autonomous vehicles, and Simultaneous Localization and Mapping (SLAM) is a cornerstone technology in this context. However, The performance of the SLAM system can deteriorate under challenging conditions such as low light, adverse weather, or obstructions due to sensor degradation. We present A2DO, a novel end-to-end multi-sensor fusion odometry system that enhances robustness in these scenarios through deep neural networks. A2DO integrates LiDAR and visual data, employing a multilayer, multi-scale feature encoding module augmented by an attention mechanism to mitigate sensor degradation dynamically. The system is pretrained extensively on simulated datasets covering a broad range of degradation scenarios and fine-tuned on a curated set of real-world data, ensuring robust adaptation to complex scenarios. Our experiments demonstrate that A2DO maintains superior localization accuracy and robustness across various degradation conditions, showcasing its potential for practical implementation in autonomous vehicle systems.

Diffusion models have demonstrated remarkable capabilities in generating high-quality images. Recent advancements in Layout-to-Image (L2I) generation have leveraged positional conditions and textual descriptions to facilitate precise and controllable image synthesis. Despite overall progress, current L2I methods still exhibit suboptimal performance. Therefore, we propose InstanceAssemble, a novel architecture that incorporates layout conditions via instance-assembling attention, enabling position control with bounding boxes (bbox) and multimodal content control including texts and additional visual content. Our method achieves flexible adaption to existing DiT-based T2I models through light-weighted LoRA modules. Additionally, we propose a Layout-to-Image benchmark, Denselayout, a comprehensive benchmark for layout-to-image generation, containing 5k images with 90k instances in total. We further introduce Layout Grounding Score (LGS), an interpretable evaluation metric to more precisely assess the accuracy of L2I generation. Experiments demonstrate that our InstanceAssemble method achieves state-of-the-art performance under complex layout conditions, while exhibiting strong compatibility with diverse style LoRA modules. The code and pretrained models are publicly available at \url{https://github.com/FireRedTeam/InstanceAssemble}.

Graph Neural Networks (GNNs) are pivotal in graph classification but often struggle with generalization and overfitting. We introduce a unified and efficient Graph Multi-View (GMV) learning framework that integrates multi-view learning into GNNs to enhance robustness and efficiency. Leveraging the lottery ticket hypothesis, GMV activates diverse sub-networks within a single GNN through a novel training pipeline, which includes mixed-view generation, and multi-view decomposition and learning. This approach simultaneously broadens "views" from the data, model, and optimization perspectives during training to enhance the generalization capabilities of GNNs. During inference, GMV only incorporates additional prediction heads into standard GNNs, thereby achieving multi-view learning at minimal cost. Our experiments demonstrate that GMV surpasses other augmentation and ensemble techniques for GNNs and Graph Transformers across various graph classification scenarios.

Blind hyperspectral unmixing (HU) has advanced significantly with the emergence of deep learning-based methods. However, the localized operations of convolutional neural networks (CNNs) and the high computational demands of Transformers present challenges for blind HU. This necessitates the development of image-level unmixing methods capable of capturing long-range spatial-spectral dependencies with low computational demands. This letter proposes a cascaded spatial-spectral Mamba model, termed UNMamba, which leverages the strengths of Mamba to efficiently model long-range spatial-spectral dependencies with linear computational complexity, achieving superior image-level unmixing performance with small parameters and operations. Specifically, UNMamba first captures long-range spatial dependencies, followed by the extraction of global spectral features, forming long-range spatial-spectral dependencies, which are subsequently mapped into abundance maps. Then, the input image is reconstructed using the linear mixing model (LMM), incorporating weighted averages of multiple trainable random sequences and an endmember loss to learn endmembers. UNMamba is the first unmixing approach that introduces the state-space models (SSMs). Extensive experimental results demonstrate that, without relying on any endmember initialization techniques [such as vertex component analysis (VCA)], the proposed UNMamba achieves significantly high unmixing accuracy, outperforming state-of-the-art methods. Codes are available athttps://github.com/Preston-Dong/UNMamba.

In practical applications, the variations in imaging conditions along with changes in ground object states cause spectral shifts within the same class across different domains of hyperspectral images (HSIs), resulting in substantial domain distribution discrepancies. Additionally, the annotation process for HSIs is time-consuming, yielding an insufficient amount of labeled data relative to the needs of strong models, making them prone to overfitting during training. To address these issues, the shift-reduced sample expansion domain generalization network (SSEDGnet) is proposed. Sample diversity is first enhanced by generating expanded domain (ED) samples. Then, feature extraction is jointly performed on multiple source-domain (SD) samples and ED samples to learn domain-invariant representations, which enhances adaptability to unseen target domains (TDs). Specifically, by modeling the full imaging process from stimulation to response, including signal transmission and ground object reflection, the ground object reflection is separately extracted and used to directly generate ED samples through stimulation, thereby obtaining samples with reduced domain shift. Subsequently, feature extraction and fusion at different levels are carried out on both the SDs and EDs. Finally, the classifier conducts the classification. The experimental results on four public HSI datasets show that the proposed method effectively learns a model with superior generalization ability and stability, outperforming state-of-the-art methods. The code will be released soon on the site ofhttps://github.com/Cherrieqi/SSEDGnet

In recent years, unsupervised domain adaptation (UDA) based on deep learning has been widely applied to address the spectral shift problem in cross-scene hyperspectral image classification (HSIC). However, most existing UDA methods focus solely on learning from source domain (SD) or target domain (TD) features, without fully exploiting the valuable class-discriminative information in the TD, resulting in limited performance on target data. To tackle this issue, we propose a prototype-guided cross-domain cyclic self-training (PGCST) framework. Specifically, we combine domain adversarial training with prototype-guided domain adaptation (PGDA) to align both global and class-wise distributions across domains. To better exploit TD information, we introduce a mutual information maximization (MIM) strategy to enhance the compactness and discriminability of target features. Furthermore, we propose a novel pseudo-label selection method that incorporates a classification loss-based cyclic self-training (CST) mechanism to improve the model’s discriminative ability on target samples. Experimental results on two cross-scene hyperspectral datasets demonstrate that the proposed method outperforms several state-of-the-art approaches.

Semantic change detection (SCD) extends the binary change detection (BCD) task, as it not only locates change areas but also identifies change transition types. Recent research has verified that a multitask learning network performs well in tackling the SCD task, which jointly addresses the two subtasks of change localization and semantic identification. However, it remains a challenging to specifically optimize the distinct features of these two subtasks and establish their interaction to further enhance the overall performance of the multitask network efficiently. In this paper, we propose a novel SCD method that emphasizes the refinement and collaboration of difference and semantic features (RCDSF). Specifically, a difference feature refinement branch (DFRB) is designed to integrate temporal information and highlight the difference features. Simultaneously, a semantic context refinement branch (SCRB) is developed to extract multi-scale and cross-scale semantic details. Moreover, a simple yet effective feature interaction-fusion module (FIFM) is incorporated to coordinate the two subtasks, ensuring consistency while providing additional auxiliary information for each other. Comprehensive experiments on two public remote sensing image SCD datasets demonstrate that the proposed method outperforms the state-of-the-art algorithms. The code will be available at https://github.com/wanglinlin0219/RCDSF.

Remote sensing hyperspectral images (HSIs) may contain incomplete or corrupted spatial or spectral information, how to reconstruct the missing regions to obtain a complete HSI is a challenging topic. Existing HSI inpainting methods are less generalizable across different remote sensing HSI and often contain artifacts. In order to address this problem, a HSI inpainting method based on predictive image filtering integrated generative restoration is proposed, which can adaptively handle different scenes with dynamically predicted filtering kernels according to different input. The generator of the proposed predictive filtering integrated generative HSI inpainting network (PFGIN) comprises two interconnected collaborative branches: the kernel prediction network (KPN) and the filtering guided generative network (FGN). FGN provides features to KPN, and the KPN dynamically predicts the kernels of designed spatial-spectral filtering according to its input. Extensive experiments on the public datasets have demonstrated the effectiveness of the proposed method and its superiority over the other comparison methods. The code for PFGIN is publicly available at https://github.com/yinhuwu/PFGIN.

Infrared small target detection plays a crucial role in infrared search and tracking systems. However, current detection methods are limited by the small target size and low signal-to-noise ratio of infrared imagery. Furthermore, motion features for target detection are difficult to extract using simple frame subtraction due to poor imaging conditions. Therefore, we focus on the holistic interframe interaction to enhance the temporal feature and propose a spatiotemporal local contrast method in this letter. First, the motion-enhanced density peak clustering is employed to determine the robust localization of candidate targets, in which the density feature maps are generated by the preprocessing of non-consecutive three frames difference after image registration. Second, to reliably exploit interframe interactions across both non-consecutive and successive frames, a temporal domain saliency map is computed based on local regions from successive frames. Moreover, a spatial domain saliency map is obtained using a novel tri-layer local contrast measure. By fusing results from both domains, the infrared small targets are detected through adaptive threshold segmentation. Experimental results on four real sequences demonstrate that the proposed method can achieve better detection performance by target enhancement and background suppression than other spatiotemporal algorithms.

Detecting infrared small targets from complex background is a challenging task. Due to the low signal-to-noise ratio and few pixels of targets, it is difficult to get accurate edge segmentation, and the targets are easily mixed up by adjacent region. To overcome these problems, we propose a gradient-enhanced feature pyramid network (GEFPN) in this letter. Specifically, we first generate gradient information under the assistance of supplementary gradient enhancement (SGE) branch, which is conductive to highlight gradient magnitude and mitigate the inaccurate edge location of small targets. On the basis of this, the proposed network utilizes a dilated cross-stage partial module (DCSPM) to refine the multiscale features and encode supplemental gradient information into the main FPN structure. Moreover, we construct a patch attention fusion module (PAFM), which fully collects both spatial details and semantic information. The experimental results show that the proposed GEFPN can achieve excellent detection performance with mean intersection over union (IoU) reaching 0.939 and 0.732 on public NUDT-SIRST and SIRST-Aug datasets, respectively, and with 0.42 M parameters and inference speed of 67.47 FPS. The code of GEFPN is available at:https://github.com/xiyunqiao/irst3.

Detecting tiny objects in remote sensing images has been an intriguing yet challenging topic in the remote sensing image processing. While significant progress has been made in many studies, most existing methods focus on improving the accuracy of tiny object detection without particular consideration for computational complexity, which restricts their applicability in resource-limited condi-tions. Therefore, this paper aims to design a lightweight de-tection algorithm tailored for tiny objects in remote sensing images. First, we investigate the impact of the complexity of different components in deep learning-based object detec-tion models on the accuracy of tiny object detection, includ-ing the backbone and detection head. Then, a dedicated backbone for tiny object detection is proposed, achieving competitive detection accuracy while remaining lightweight. Moreover, we propose a lightweight detection head that in-corporates deformable convolution and optimize the chan-nel dimension. Finally, we combine the above methods to introduce a lightweight network, LTDNet, for tiny object detection in remote sensing images. Benefiting from the dedicated designs for the backbone and detection head spe-cifically for tiny objects, the proposed method can achieve competitive detection accuracy with very low parameters and computational complexity. Extensive experiments are conducted on the AI-TODv2 and LEVIR-Ship datasets, and the results demonstrate the effectiveness of our proposed method. Specifically, the proposed method achieves 54.6% AP50 on the AI-TODv2 dataset with only 4.85M parameters and 38.19G FLOPs. The code will be released soon on the site of https://github.com/dyl96/LTDNet.

Metal artifacts in computed tomography (CT) images can significantly degrade image quality and impede accurate diagnosis. Supervised metal artifact reduction (MAR) methods, trained using simulated datasets, often struggle to perform well on real clinical CT images due to a substantial domain gap. Although state-of-the-art semi-supervised methods use pseudo ground-truths generated by a prior network to mitigate this issue, their reliance on a fixed prior limits both the quality and quantity of these pseudo ground-truths, introducing confirmation bias and reducing clinical applicability. To address these limitations, we propose a novel radiologist-in-the-loop self-training framework for MAR, termed RISE-MAR, which can integrate radiologists' feedback into the semi-supervised learning process, progressively improving the quality and quantity of pseudo ground-truths for enhanced generalization on real clinical CT images. For quality assurance, we introduce a clinical quality assessor model that emulates radiologist evaluations, effectively selecting high-quality pseudo ground-truths for semi-supervised training. For quantity assurance, our self-training framework iteratively generates additional high-quality pseudo ground-truths, expanding the clinical dataset and further improving model generalization. Extensive experimental results on multiple clinical datasets demonstrate the superior generalization performance of our RISE-MAR over state-of-the-art methods, advancing the development of MAR models for practical application. The source code is available at https://github.com/Masaaki-75/rise-mar.

Most conventional crowd counting methods utilize a fully-supervised learning framework to establish a mapping between scene images and crowd density maps. They usually rely on a large quantity of costly and time-intensive pixel-level annotations for training supervision. One way to mitigate the intensive labeling effort and improve counting accuracy is to leverage large amounts of unlabeled images. This is attributed to the inherent self-structural information and rank consistency within a single image, offering additional qualitative relation supervision during training. Contrary to earlier methods that utilized the rank relations at the original image level, we explore such rank-consistency relation within the latent feature spaces. This approach enables the incorporation of numerous pyramid partial orders, strengthening the model representation capability. A notable advantage is that it can also increase the utilization ratio of unlabeled samples. Specifically, we propose a Deep Rank-consist Ent pyrAmid Model (DREAM), which makes full use of rank consistency across coarse-to-fine pyramid features in latent spaces for enhanced crowd counting with massive unlabeled images. In addition, we have collected a new unlabeled crowd counting dataset, FUDAN-UCC, comprising 4000 images for training purposes. Extensive experiments on four benchmark datasets, namely UCF-QNRF, ShanghaiTech PartA and PartB, and UCF-CC-50, show the effectiveness of our method compared with previous semi-supervised methods. The codes are available at https://github.com/bridgeqiqi/DREAM.

Gait recognition is a prominent biometric recognition technique extensively employed in public security. Appearance-based and model-based gait recognition are two categories of methods commonly used. Specifically, appearance-based methods, which use silhouettes to represent body information, typically outperform model-based methods that rely on skeleton data, making them more popular. Recently, the shift from single-frame templates to multiframe silhouettes has advanced appearance-based gait recognition with better spatiotemporal representation. However, there is a notable lack of comprehensive studies that deepen the understanding of multiframe appearance-based gait recognition methods. This article reviews various methods to trace the evolution of gait recognition. Furthermore, we unify various performant models in one framework, study the overlooked effects on data arrangement, and explore the scaling ability of existing methods. Besides the advancement in gait recognition, we also summarize the current challenges and future prospects to foster future research.

Class-agnostic object counting aims to count all objects in an image with respect to example boxes or class names, a.k.a few-shot and zero-shot counting. In this paper, we propose a generalized framework for both few-shot and zero-shot object counting based on detection. Our framework combines the superior advantages of two foundation models without compromising their zero-shot capability: (i) SAM to segment all possible objects as mask proposals, and (ii) CLIP to classify proposals to obtain accurate object counts. However, this strategy meets the obstacles of efficiency over-head and the small crowded objects that cannot be localized and distinguished. To address these issues, our framework, termed PseCo, follows three steps: point, segment, and count. Specifically, we first propose a class-agnostic object localization to provide accurate but least point prompts for SAM, which consequently not only reduces computation costs but also avoids missing small objects. Furthermore, we propose a generalized object classification that leverages CLIP image/text embeddings as the classifier, following a hierarhical knowledge distillation to obtain discriminative classifications among hierarchical mask proposals. Extensive experimental results on FSC-147, COCO, and LVIS demonstrate that PseCo achieves state-of-the-art performance in both few-shot/zero-shot object counting/detection.

Navigating in the latent space of StyleGAN has shown effectiveness for face editing. However, the resulting methods usually encounter challenges in complicated navigation due to the entanglement among different attributes in the latent space. To address this issue, this paper proposes a novel framework, termed SDFlow, with a semantic decomposition in original latent space using continuous conditional normalizing flows. Specifically, SDFlow decomposes the original latent code into different irrelevant variables by jointly optimizing two components: (i) a semantic encoder to estimate semantic variables from input faces and (ii) a flow-based transformation module to map the latent code into a semantic-irrelevant variable in Gaussian distribution, conditioned on the learned semantic variables. To eliminate the entanglement between variables, we employ a disentangled learning strategy under a mutual information framework, thereby providing precise manipulation controls. Experimental results demonstrate that SDFlow outperforms existing state-of-the-art face editing methods both qualitatively and quantitatively. The source code is available at https://github.com/phil329/SDFlow.

Video prediction, predicting future frames from the previous ones, has broad applications such as autonomous driving and weather forecasting. Existing state-of-the-art methods typically focus on extracting either spatial, temporal, or spatiotemporal features from videos. Different feature focuses, resulting from different network architectures, may make the resultant models excel at some video prediction tasks but perform poorly on others. Towards a more generic video prediction solution, we explicitly model these features in a unified encoder-decoder framework and propose a simple alternating Mixer (SIAM). The novelty of SIAM lies in the design of dimension alternating mixing (DaMi) blocks, which can model spatial, temporal, and spatiotemporal features through alternating the dimensions of the feature maps. Extensive experimental results demonstrate the superior performance of the proposed SIAM on four benchmark video datasets covering both synthetic and real-world scenarios.

Graph neural networks (GNNs) demonstrate impressive performance in graph classification, albeit exhibiting challenges in terms of generalizability and robustness. Despite its proven efficacy in enhancing the robustness and generalizability of GNNs, ensemble learning encounters practical limitations due to its extensive computational and memory requirements. This paper presents a novel low-cost ensemble learning method for graph classification, which utilizes a Graph Multi-Input and Multi-Output framework (G-MIMO) to allow multiple sub-networks within a single GNN simultaneously treated. Extensive experiments demonstrate that G-MIMO effectively enhances GNN performance across multiple graph classification tasks without introducing much computational overhead. G-MIMO outperforms existing graph augmentations and ensemble approaches, delivering a 3.33% and 4.25% increase in average accuracy over standard GCN and GIN models, respectively. The source code has been released at https://github.com/smurf-1119/GMIMO.

The remote sensing satellite system has steadily developed, and the availability of massive high-quality satellite remote sensing data has rapidly improved. The dynamic monitoring of land cover changes using hyperspectral data has received great attention. The existing change detection methods usually use the spatial correlation or spatial-spectral correlation of hyperspectral images, lacking an overall consideration of the three-dimensional temporal-spatial-spectral joint features, resulting in suboptimal change detection results. Given the considerations above, this paper proposes a SimAM multi-scale joint feature extraction network for hyperspectral image change detection. The proposed method first adopts SimAM multi-scale joint features network to consider the hyperspectral image as a whole, and extracts multi-scale temporal-spatial-spectral joint features from the network. Then, uses multi-scale feature weighted fusion module to weight different scale features after simple fusion, highlighting the change regions. Finally, adopts the batch-balance measurement module to measure the similarity of the bi-temporal fusion features and output the change detection result map. Experiments show that on two public datasets, the proposed method can achieve a good change detection effect.

UAV remote sensing image registration has a wide range of applications in the fields of fine geographic information extraction, environmental change monitoring, agriculture and forestry, serving as a fundamental and critical step in supporting various applications. However, existing registration methods often extract fewer feature points, uneven distribution and poor stability when faced with significant changes in land features. Accordingly, in this paper, we propose a UAV remote sensing image registration method based on an improved SuperPoint+SuperGlue deep learning network. We add a feature point extraction branch to the shallow feature map in the SuperPoint network, and add the newly extracted feature points to the existing feature points, and then perform non-maximum suppression (NMS) for further refinement. This enhances its capability to extract image feature points, resulting in more accurate and evenly distributed positions of these points. The experiments carried on two sets of real UAV datasets indicate that our method outperforms several typical registration approaches in terms of registration accuracy and detection efficiency.

Remote sensing image change detection has important applications in many fields. However, current studies mostly focus on identifying pixel-level changes. Although these methods can achieve better performance, this paradigm fails to determine changes in specific object instances due to the definition of the task itself. For this reason, we conduct preliminary exploration and propose a method named OBJ-CD, which can detect the changes of object instance. Specifically, OBJ-CD initially employs a lightweight Siamese object detector to detect objects within two temporal images. Subsequently, OBJ-CD calculates the metric matrix for the detected objects in these images. Finally, the conditions of the object instance change are limited by a certain threshold, and the final object-level change detection results can be obtained. We conducted several experiments on our constructed dataset, and the experimental results indicate that the proposed method can achieve object-level change detection with good performance.

Hyperspectral image classification (HSIC) is a pivotal technology in hyperspectral remote sensing, playing a widespread role in remote sensing applications. However, the limited number of labeled samples has always made hyperspectral image classification difficult. In response to this issue, researchers have delved into cross-domain classification studies. Moreover, significant progress on cross-domain HSIC has been made in recent years. Nevertheless, existing methods exhibit shortcomings, including inadequate exploitation of spectral and spatial information and a slow training speed, rendering them unsuitable for downstream application tasks. To address these challenges, this paper introduces a model of cross-domain few-shot learning with spectral-spatial split attention(S3A-CFSL). Channel attention and split attention are presented to emphasize effective spectral and spatial information for HSIC adaptively. Additionally, the ResNet variant, called ResNeSt, is employed to expedite the training speed of the model. Experimental results demonstrate notable enhancements in the proposed method's classification accuracy and model training speed across two public datasets.

With the development of deep learning, automated road detection has gradually become a popular topic in remote sensing image processing. However, existing road detection methods are still insufficient in occlusion problem, resulting in the occluded road being difficult to be detected and thus broken. In order to solve this problem, we propose a multi-resolution codec road detection network (MCRDnet). MCRDnet consists of a backbone with a codec structure, an edge extraction branch (EEB) and an information supplement module (ISM). The backbone is used to extract the features of the road. In addition, EEB is responsible for extracting edge information and interacting with the backbone to smooth the road edge regions. Moreover, ISM also adopts the codec structure but with the input of the images after 2-fold and 4-fold downsampling. The decoder of ISM plays the roles of image restoration and feature mapping at the same time, fusing features into the backbone to supplement the missing information affected by occlusion. The experimental results on the RNBD dataset demonstrate the effectiveness of the proposed method for mitigating the breakage phenomenon in road detection results.

The fusion of hyperspectral image (HSI) and LiDAR data for classification has gained widespread attention. However, the current fusion methods still have limitations on the use of heterogeneous data information and the interaction of heterogeneous data features. Therefore, we propose a classification method based on heterogeneous attention feature fusion network (HAFF-Net). Firstly, The convolutional neural network is employed to capture local spatial features from multi-scale inputs of heterogeneous data. Secondly, the heterogeneous feature attention module is designed to deeply fuse the extracted local spatial features and promote the interaction of heterogeneous data. Next, a transformer encoder is utilized to extract global spectral features. Finally, the two extracted features are classified by decision fusion using different classifiers. The proposed approach exhibits effectiveness through experimental results on two datasets.

There is abundant spectral and spatial information in Hyperspectral images (HSI). However, there exists a limitation of not using spatial information sufficiently in HSI classification. Besides, there is the limitation of mononuclear in feature extraction, resulting in insufficient feature extraction and insufficient utilization of data information. In view of these problems, a node similarity semi-supervised classification method of multiscale feature is proposed to break the limitation of mononuclear and achieve full extraction of spatial information. First, to extract pixel-level features, a three-dimensional (3-D) multiscale convolutional neural network (CNN) is used. Second, based on node similarity superpixel graph U-Net (NSGUNet) is proposed to extract superpixel-level features. Finally, the above two features are weighted fusing, the fused features are classified by sparse graph regularization. Experiments on three datasets illustrate that the proposed method is effective.

Automated road detection from the high-resolution remote sensing images (RSI) is always a hot topic. Particularly, the accurate and continuous road detection in RSI is challenging due to the tree and shadow shading and unsmooth road edges further hindering the accuracy of the road extraction. Considering that the dynamic snake convolution (DSConv) is able to capture the distinctive characteristics of tubular objects such as roads, and edge information extracted from road edge branch can enhance the smoothness of road edges, we propose a cascade automatic road detection method based on DSConv and edge branch named CasDeNet. Specifically, the DSConv is introduced as the foundational module for low-level feature extraction of CasDeNet, aiming to capture the intricate shapes of roads. Road edge information from the edge branch is incorporated to ensure the smoothness of the road edges. Experiments are conducted on the Ottawa dataset. The results show that the proposed CasDeNet can extract more coherent and accurate roads compared to other state-of-the-art (SOTA) methods and achieve the best results.

The explainability of deep neural networks (DNNs) is critical for trust and reliability in AI systems. Path-based attribution methods, such as integrated gradients (IG), aim to explain predictions by accumulating gradients along a path from a baseline to the target image. However, noise accumulated during this process can significantly distort the explanation. While existing methods primarily concentrate on finding alternative paths to circumvent noise, they overlook a critical issue: intermediate-step images frequently diverge from the distribution of training data, further intensifying the impact of noise. This work presents a novel Denoising Diffusion Path (DDPath) to tackle this challenge by harnessing the power of diffusionmodels for denoising. By exploiting the inherent ability of diffusion models to progressively remove noise from an image, DDPath constructs a piece-wise linear path. Each segment of this path ensures that samples drawn from a Gaussian distribution are centered around the target image. This approach facilitates a gradual reduction of noise along the path. We further demonstrate that DDPath adheres to essential axiomatic properties for attribution methods and can be seamlessly integrated with existing methods such as IG. Extensive experimental results demonstrate that DDPath can significantly reduce noise in the attributions—resulting in clearer explanations—and achieves better quantitative results than traditional path-based methods.

人工智能生成内容(AIGC)是近年来人工智能(AI)领域一个研究热点,它有望取代人类以较低成本高效率执行内容生成工作,如音乐、绘画、多模态内容生成、新闻文章、总结报告、股评摘要,以至元宇宙中的内容生成和数字人。AIGC为未来AI发展和实现提供了一条新的技术路径。 在此背景下,《信息与电子工程前沿(英文)》期刊组织了一期关于AIGC最新进展的特刊。本期特刊关注AIGC理论、算法、应用及相关领域。通过吸引高质量论文,我们希望帮助学术界和工业界研究人员更深入了解AIGC背后的基本理论及其潜在应用,激励更多研究人员加入并推进AIGC领域的研究。因此,我们就以下主题(但不限于)征集论文:(1)AI生成音乐;(2)AI生成绘画;(3)AI对话模型;(4)AI新闻摘要;(5)AI与元宇宙;(6)AI与数字人;(7)AI图像编辑;(8)AI生成短视频;(9)AI生成多媒体内容;(10)ChatGPT相关工作。经严格评审,选出12篇论文,包括1篇评论、1篇观点、3篇综述、6篇研究和1篇通讯。我们将其划分为3个主要部分:ChatGPT、扩散模型、提示学习和多模态。 总体而言,本期特刊涵盖了与AIGC开发和应用相关的广泛研究主题,包括人工智能图像/文本生成、三维内容创建、以用户为中心的图形设计、特定风格的音乐生成,以及与因果表征学习、高阶扩散模型相关的工作。此外,还详细调研了概率扩散模型、提示学习和ChatGPT。 最后,感谢所有作者对本期特刊的支持,特别感谢所有评审人对专刊投稿富有见地的意见和有益建议。

Hyperspectral image data contains rich spectral-spatial information which can be useful for various applications. Many methods have been proposed to classify the hyperspectral images. Nonetheless, the availability of limited training samples in traditional models frequently weakens their ability to handle the inherent complexity of the task. Deep Learning models has been successfully applied in the field of remote sensing. In this letter, we propose a vision transformer (ViT) based network called hierarchical attention transformer that combines the properties of local representation learning in 3D and 2D CNNs and potent global modeling capabilities in ViT. We leverage the efficiency of window based self-attention. Within each window, there are dedicated tokens that contribute to both local and global representation learning. The overall accuracy of the proposed model achieved 99.70%,99.89%,99.56%,81.75%and 99.59% on five dataset.

In the field of remote sensing, the detection of tiny objects has always been an interesting and highly regarded issue. Although many researchers have dedicated their efforts to studying this problem, it still presents numerous challenges due to the complexity of the environment in which tiny objects are presented in remote sensing images. To this end, we propose a remote sensing image tiny objects detection method based on explicit semantic guidance, with a specific focus on regions containing tiny objects. Specifically, we incorporate supervision of the tiny object regions during the training process. This supervision allowed us to extract tiny object regions, thereby forming an explicit attention map. This explicit attention map is employed to semantically modulate the feature map for detecting tiny objects, thus enhancing the regions containing tiny objects while suppressing the background. Extensive experiments are conducted on the AI-TODv2 dataset and the proposed method can achieve an AP of 24.6%. The experimental results demonstrate the effectiveness of the proposed tiny object detection method based on explicit semantic guidance. The code will be released soon on the site of https://github.com/dyl96/ESG_TODNet.

In practical applications, due to the high cost and difficulty of hyperspectral image (HSI) annotation, labels for the target domain (TD) may be either unavailable or insufficient in quantity. To address this issue, we propose a multi-source domain generalization two-branch network (MDGTnet) and train the model only using source domain (SD) HSIs with contrastive learning to classify an unknown TD image. MDGTnet consists of a classifier and two branches, which are intra-domain uniqueness extraction branch (intra-DUEB) and inter-domain commonality extraction branch (inter-DCEB). The intra-DUEB is responsible for mining internal attributes of each SD, which can be seen as imaging environmental characteristics. And the inter-DCEB is applied to extract generic features among different SDs. The features extracted by two branches are fused at different levels respectively to remove the influence of different imaging environments for discriminative class features. We have conducted extensive experiments on four public HSI datasets. The results show that the proposed method outperforms state-of-the-art methods. It can learn robust models and extract highly discriminative features, leading to excellent performance in HSI cross-domain classification. Especially on the Pavia Center dataset, the overall accuracy (OA) is 2.47% higher and kappa coefficient is 2.92% higher than the best results of the other methods. The code will be released soon on the site of https://github.com/Cherrieqi/MDGTnet.

Remote sensing hyperspectral images (HSIs) have been applied in a variety of fields. However, HSIs are susceptible to various types of noise which affect both their quality and subsequent analysis. Existing knowledge-driven methods are time-consuming and need handcrafted parameters, while data-driven methods require large amounts of training resources and lack interpretability. What’s more, most methods mainly focus on Gaussian noise rather than Poisson-Gaussian noise that matches better the real noise model. To address this issue, this paper proposes a multi-priors guided HSIs noise removal method that not only combines the benefits of traditional methods and deep learning methods but also considers the Poisson-Gaussian based mixed noise. Specifically, tensor subspace representation based on the guidance of the global spectral low-rank prior is employed to decompose the HSI into eigen-images and orthogonal spectral basis. Then a nonlocal-local aware network that incorporates the guidance of local and nonlocal self-similarity priors is constructed to remove the noise in the eigen-images effectively and efficiently. Extensive experiments demonstrate that our method achieves better quantitative and qualitative performance compared to the state-of-the-art methods.

Infrared small target suffers from weak features and complex background. The existing detection methods are usually unable to effectively extract global context and maintain features of infrared small targets. To overcome the problems, we design a non-local enhanced feature pyramid network (NLFPNet) based on an encoder-decoder framework by additionally global contextual features modeling. In the encoder stage, small targets features are extracted by a ResNet-18 based backbone network. Then, global contextual information of small targets is exploited by using a non-local enhanced pyramid pooling module (NLPPM), which is conductive to estimate the correlation between pixels in a wide range and enhance the global prior information. In the decoder stage, we obtain multi-level feature representation through asymmetric attention fusion module (AAFM), which reasonably modulates and maintains the shallow-level spatial details and deep-level semantic information. The experimental results show that the proposed NLFPNet can achieve an improved detection performance with mean intersection over union (IoU) of 0.739 and probability of detection (Pd) of 0.953 on the public SIRST-Aug dataset. The code is available at https://github.com/xiyunqiao/irst1.

Alzheimer's disease (AD) is one of the most common neurodegenerative disorders presenting irreversible progression of cognitive impairment. How to identify AD as early as possible is critical for intervention with potential preventive measures. Among various neuroimaging modalities used to diagnose AD, functional positron emission tomography (PET) has higher sensitivity than structural magnetic resonance imaging (MRI), but it is also costlier and often not available in many hospitals. How to leverage massive unpaired unlabeled PET to improve the diagnosis performance of AD from MRI becomes rather important. To address this challenge, this paper proposes a novel joint learning framework of unsupervised cross-modal synthesis and AD diagnosis by mining underlying shared modality information, improving the AD diagnosis from MRI while synthesizing more discriminative PET images. We mine underlying shared modality information in two aspects: diversifying modality information through the cross-modal synthesis network and locating critical diagnosis-related patterns through the AD diagnosis network. First, to diversify the modality information, we propose a novel unsupervised cross-modal synthesis network, which implements the inter-conversion between 3D PET and MRI in a single model modulated by the AdaIN module. Second, to locate shared critical diagnosis-related patterns, we propose an interpretable diagnosis network based on fully 2D convolutions, which takes either 3D synthesized PET or original MRI as input. Extensive experimental results on the ADNI dataset show that our framework can synthesize more realistic images, outperform the state-of-the-art AD diagnosis methods, and have better generalization on external AIBL and NACC datasets.

Tiny object detection in the field of remote sensing has always been a challenging and interesting topic. Despite many researchers have been working on this problem, it has not been well solved due to its complexity. In this paper, we analyze the reasons for the poor performance of deep learning-based object detection methods for tiny objects in remote sensing images. Moreover, we propose a new remote sensing image tiny object detection network based on object reconstruction and multiple receptive field adaptive feature enhancement module (MRFAFEM), called ORFENet. Detailedly, object reconstruction aims to reduce the information loss of tiny objects within deep neural networks, which is only used in the training phase and can be discarded in the inference phase. MRFAFEM is designed to enhance the features for detecting tiny objects by dynamically adjusting the multiple receptive field features. We have conducted several experiments on the AI-TODv2 and LEVIR-Ship datasets, both of which are proposed for tiny object detection in remote sensing images. The experimental results indicate the effectiveness of the proposed method. Specifically, the proposed ORFENet can achieve the AP of 24.8% on the AI-TODv2 dataset and 83.3% AP50 on the LEVIR-Ship dataset. The code will be released at https://github.com/dyl96/ORFENet.

Convolutional neural networks (CNNs) have performed notably in change detection (CD) tasks due to their superior learning and automatic feature extraction capabilities. However, they suffer from the limited receptive field and the weak modeling of long-range dependencies. Vision transformers (ViTs) excel in modeling long-range contexts and have been recently introduced in CD. Some works have combined CNN and transformers to obtain local-global information. However, these works do not fully consider the guidance and interactions from both local features (LFs) and global features (GFs). Most importantly, most of them involve a very large number of parameters and computational costs. To address these issues, in this article, we propose a lightweight CD network that mixes features across CNN and transformer (MixCDNet). We use EfficientNet as the backbone and design a novel mixing features block (MFB). First, we employ hierarchical feature extraction blocks, where local feature blocks (LFBs) and global feature blocks (GFBs) are utilized for extracting information at different spatial resolutions. Second, we propose to exploit bidirectional interactions across LFBs and GFBs branches to provide complementary clues while capturing LFs and GFs. Moreover, a skip-connection and fusion separable self-attention layer (SFSSL) is designed to obtain GFs with low complexity. Comprehensive experiments are conducted on three high-resolution remote sensing (HRRS) images CD datasets: LEVIR-CD, WHU-CD, and CDD. The results show the effectiveness of the proposed MixCDNet in improving the performance of existing CD methods with fewer parameters (0.32 M) and lower computation costs (1.59G FLOPs).

Mild cognitive impairment (MCI) is often at high risk of progression to Alzheimer's disease (AD). Existing works to identify the progressive MCI (pMCI) typically require MCI subtype labels, pMCI vs. stable MCI (sMCI), determined by whether or not an MCI patient will progress to AD after a long follow-up. However, prospectively acquiring MCI subtype data is time-consuming and resource-intensive; the resultant small datasets could lead to severe overfitting and difficulty in extracting discriminative information. Inspired by that various longitudinal biomarkers and cognitive measurements present an ordinal pathway on AD progression, we propose a novel Hybrid-granularity Ordinal PrototypE learning (HOPE) method to characterize AD ordinal progression for MCI progression prediction. First, HOPE learns an ordinal metric space that enables progression prediction by prototype comparison. Second, HOPE leverages a novel hybrid-granularity ordinal loss to learn the ordinal nature of AD via effectively integrating instance-to-instance ordinality, instance-to-class compactness, and class-to-class separation. Third, to make the prototype learning more stable, HOPE employs an exponential moving average strategy to learn the global prototypes of NC and AD dynamically. Experimental results on the internal ADNI and the external NACC datasets demonstrate the superiority of the proposed HOPE over existing state-of-the-art methods as well as its interpretability.

Low-dose computed tomography (CT) images suffer from noise and artifacts due to photon starvation and electronic noise. Recently, some works have attempted to use diffusion models to address the over-smoothness and training instability encountered by previous deep-learning-based denoising models. However, diffusion models suffer from long inference time due to a large number of sampling steps involved. Very recently, cold diffusion model generalizes classical diffusion models and has greater flexibility. Inspired by cold diffusion, this paper presents a novel COntextual eRror-modulated gEneralized Diffusion model for low-dose CT (LDCT) denoising, termed CoreDiff. First, CoreDiff utilizes LDCT images to displace the random Gaussian noise and employs a novel mean-preserving degradation operator to mimic the physical process of CT degradation, significantly reducing sampling steps thanks to the informative LDCT images as the starting point of the sampling process. Second, to alleviate the error accumulation problem caused by the imperfect restoration operator in the sampling process, we propose a novel ContextuaL Error-modulAted Restoration Network (CLEAR-Net), which can leverage contextual information to constrain the sampling process from structural distortion and modulate time step embedding features for better alignment with the input at the next time step. Third, to rapidly generalize the trained model to a new, unseen dose level with as few resources as possible, we devise a one-shot learning framework to make CoreDiff generalize faster and better using only one single LDCT image (un)paired with normal-dose CT (NDCT). Extensive experimental results on four datasets demonstrate that our CoreDiff outperforms competing methods in denoising and generalization performance, with clinically acceptable inference time. Source code is made available at https://github.com/qgao21/CoreDiff.

Metal implants and other high-density objects in patients introduce severe streaking artifacts in CT images, compromising image quality and diagnostic performance. Although various methods were developed for CT metal artifact reduction over the past decades, including the latest dual-domain deep networks, remaining metal artifacts are still clinically challenging in many cases. Here we extend the state-of-the-art dual-domain deep network approach into a quad-domain counterpart so that all the features in the sinogram, image, and their corresponding Fourier domains are synergized to eliminate metal artifacts optimally without compromising structural subtleties. Our proposed quad-domain network for MAR, referred to as Quad-Net, takes little additional computational cost since the Fourier transform is highly efficient, and works across the four receptive fields to learn both global and local features as well as their relations. Specifically, we first design a Sinogram-Fourier Restoration Network (SFR-Net) in the sinogram domain and its Fourier space to faithfully inpaint metal-corrupted traces. Then, we couple SFR-Net with an Image-Fourier Refinement Network (IFR-Net) which takes both an image and its Fourier spectrum to improve a CT image reconstructed from the SFR-Net output using cross-domain contextual information. Quad-Net is trained on clinical datasets to minimize a composite loss function. Quad-Net does not require precise metal masks, which is of great importance in clinical practice. Our experimental results demonstrate the superiority of Quad-Net over the state-of-the-art MAR methods quantitatively, visually, and statistically. The Quad-Net code is publicly available at https://github.com/longzilicart/Quad-Net.

Due to the homophily assumption in graph convolution networks (GCNs), a common consensus in the graph node classification task is that graph neural networks (GNNs) perform well on homophilic graphs but may fail on heterophilic graphs with many interclass edges. However, the previous interclass edges' perspective and related homo-ratio metrics cannot well explain the GNNs' performance under some heterophilic datasets, which implies that not all the interclass edges are harmful to GNNs. In this work, we propose a new metric based on the von Neumann entropy to reexamine the heterophily problem of GNNs and investigate the feature aggregation of interclass edges from an entire neighbor identifiable perspective. Moreover, we propose a simple yet effective Conv-Agnostic GNN framework (CAGNNs) to enhance the performance of most GNNs on the heterophily datasets by learning the neighbor effect for each node. Specifically, we first decouple the feature of each node into the discriminative feature for downstream tasks and the aggregation feature for graph convolution (GC). Then, we propose a shared mixer module to adaptively evaluate the neighbor effect of each node to incorporate the neighbor information. The proposed framework can be regarded as a plug-in component and is compatible with most GNNs. The experimental results over nine well-known benchmark datasets indicate that our framework can significantly improve performance, especially for the heterophily graphs. The average performance gain is 9.81%, 25.81%, and 20.61% compared with graph isomorphism network (GIN), graph attention network (GAT), and GCN, respectively. Extensive ablation studies and robustness analysis further verify the effectiveness, robustness, and interpretability of our framework. Code is available at https://github.com/JC-202/CAGNN.

Existing Graph Neural Networks (GNNs) follow the message-passing mechanism that conducts information interaction among nodes iteratively. While considerable progress has been made, such node interaction paradigms still have the following limitation. First, the scalability limitation precludes the broad application of GNNs in large-scale industrial settings since the node interaction among rapidly expanding neighbors incurs high computation and memory costs. Second, the over-smoothing problem restricts the discrimination ability of nodes, i.e., node representations of different classes will converge to indistinguishable after repeated node interactions. In this work, we propose a novel hop interaction paradigm to address these limitations simultaneously. The core idea is to convert the interaction target among nodes to pre-processed multi-hop features inside each node. We design a simple yet effective HopGNN framework that can easily utilize existing GNNs to achieve hop interaction. Furthermore, we propose a multi-task learning strategy with a self-supervised learning objective to enhance HopGNN. We conduct extensive experiments on 12 benchmark datasets in a wide range of domains, scales, and smoothness of graphs. Experimental results show that our methods achieve superior performance while maintaining high scalability and efficiency. The code is at https://github.com/JC-202/HopGNN.

Learning from noisy data is a challenging task that sig-nificantly degenerates the model performance. In this paper, we present TCL, a novel twin contrastive learning model to learn robust representations and handle noisy labels for classification. Specifically, we construct a Gaussian mixture model (GMM) over the representations by injecting the supervised model predictions into GMM to link label- free latent variables in GMM with label-noisy annotations. Then, TCL detects the examples with wrong labels as the out- of-distribution examples by another two-component GMM, taking into account the data distribution. We further propose a cross-supervision with an entropy regularization loss that bootstraps the true targets from model predictions to handle the noisy labels. As a result, TCL can learn discriminative representations aligned with estimated labels through mixup and contrastive learning. Extensive experimental results on several standard benchmarks and real-world datasets demonstrate the superior performance of TCL. In particular, TCL achieves 7.5% improvements on CIFAR-10 with 90% noisy label-an extremely noisy scenario. The source code is available at https://github.com/Hzzone/TCL.

Precipitation nowcasting uses previous rainfall observations to forecast future rainfall intensities in a local area. In rainfall data, the rain-less samples usually well exceed the heavy rainfall samples, and it causes the data imbalance problem in precipitation nowcasting tasks. In this paper, we find that if the imbalance ratio is fixed, tasks with higher mutual information make the nowcasting model more robust to the data imbalance problem. Based on this observation, we propose a mutual information-based reweighting strategy. The reweighting strategy allows the neural network models to achieve better performance on minorities without compromising the performance of majorities and overall nowcasting image quality. Extensive experimental results demonstrate that this proposed approach is effective and compatible with state-of-the-art models.

Noisy annotations such as missing annotations and location shifts often exist in crowd counting datasets due to multi-scale head sizes, high occlusion, etc. These noisy annotations severely affect the model training, especially for density map-based methods. To alleviate the negative impact of noisy annotations, we propose a novel crowd counting model with one convolution head and one transformer head, in which these two heads can supervise each other in noisy areas, called Cross-Head Supervision. The resultant model, CHS-Net, can synergize different types of inductive biases for better counting. In addition, we develop a progressive cross-head supervision learning strategy to stabilize the training process and provide more reliable supervision. Extensive experimental results on Shang-haiTech and QNRF datasets demonstrate superior performance over state-of-the-art methods. Code is available at https://github.com/RaccoonDML/CHSNet.

As a unique biometric that can be perceived at a distance, gait has broad applications in person authentication, social security and so on. Existing gait recognition methods suffer from changes in viewpoint and clothing and barely consider extracting diverse motion features, a fundamental characteristic in gaits, from gait sequences. This paper proposes a novel motion modeling method to extract the discriminative and robust representation. Specifically, we first extract the motion features from the encoded motion sequences in the shallow layer. Then we continuously enhance the motion feature in deep layers. This motion modeling approach is independent of mainstream work in building network architectures. As a result, one can apply this motion modeling method to any backbone to improve gait recognition performance. In this paper, we combine motion modeling with one commonly used backbone (GaitGL) as GaitGL-M to illustrate motion modeling. Extensive experimental results on two commonly-used crossview gait datasets demonstrate the superior performance of GaitGL-M over existing state-of-the-art methods.

This work presents a novel hybrid convolution-transformer framework for gait recognition, termed GaitCoTr. The developed framework captures the appearance and short-term temporal features by convolution and extracts the long-term temporal features by transformer architecture, achieving a comprehensive spatial-temporal representation of gait. To unleash the potential of this hybrid framework for extracting richness and generalized temporal features, we propose a new variant of transformer tailored for gait, including temporally shifted tokenization, length-flexible position embedding, and inter-frame encoder. In addition, we introduce an auxiliary task—view label prediction—aiming to disentangle view from ID information. Extensive experimental results on two well-known gait benchmark datasets, CASIA-B and GREW, demonstrate the superior performance of the proposed Gait-CoTr.

Facial attribute manipulation (FAM) aims to edit the semantic attributes of facial images according to the user’s requirements. Unfortunately, the majority of existing FAM methods struggle in meeting at least one of the two requirements: high reconstruction quality and high irrelevance preservation. To alleviate these two limitations, we propose a novel Disentangled nOn-linear latent navigation framework for FAM, termed DO-FAM. To promote the reconstruction quality, we leverage hypernetworks to fine-tune a pre-trained StyleGAN2 generator. To decouple entangled attributes, we propose a novel Disentangled nOn-Linear Latent transformation module, named DOLL, which consists of three components: (1) a decomposer to factorize input latent codes into two parts: attribute-related and attribute-unrelated; (2) a non-linear Latent Transformation Network (LTNet) to navigate the attribute-related latent codes to the target one with respect to the designed attribute(s); and (3) a latent classifier tasked with predicting latent codes’ attributes to guide the latent code navigation. Extensive experimental results on a widely-used benchmark facial editing dataset, CelebA-HQ, demonstrate the superiority of our method over state-of-the-art methods.

Online continual learning (CL) studies the problem of learning continuously from a single-pass data stream while adapting to new data and mitigating catastrophic forgetting. Recently, by storing a small subset of old data, replay-based methods have shown promising performance. Unlike previous methods that focus on sample storage or knowledge distillation against catastrophic forgetting, this paper aims to understand why the online learning models fail to generalize well from a new perspective of shortcut learning. We identify shortcut learning as the key limiting factor for online CL, where the learned features may be biased, not generalizable to new tasks, and may have an adverse impact on knowledge distillation. To tackle this issue, we present the online prototype learning (OnPro) framework for online CL. First, we propose online prototype equilibrium to learn representative features against shortcut learning and discriminative features to avoid class confusion, ultimately achieving an equilibrium status that separates all seen classes well while learning new classes. Second, with the feedback of online prototypes, we devise a novel adaptive prototypical feedback mechanism to sense the classes that are easily misclassified and then enhance their boundaries. Extensive experimental results on widely-used benchmark datasets demonstrate the superior performance of OnPro over the state-of-the-art baseline methods. Source code is available at https://github.com/weilllllls/OnPro.

Recent works for face editing usually manipulate the latent space of StyleGAN via the linear semantic directions. However, they usually suffer from the entanglement of facial attributes, need to tune the optimal editing strength, and are limited to binary attributes with strong supervision signals. This paper proposes a novel adaptive nonlinear latent transformation for disentangled and conditional face editing, termed AdaTrans. Specifically, our AdaTrans divides the manipulation process into several finer steps; i.e., the direction and size at each step are conditioned on both the facial attributes and the latent codes. In this way, AdaTrans describes an adaptive nonlinear transformation trajectory to manipulate the faces into target attributes while keeping other attributes unchanged. Then, AdaTrans leverages a predefined density model to constrain the learned trajectory in the distribution of latent codes by maximizing the likelihood of transformed latent code. Moreover, we also propose a disentangled learning strategy under a mutual information framework to eliminate the entanglement among attributes, which can further relax the need for labeled data. Consequently, AdaTrans enables a controllable face editing with the advantages of disentanglement, flexibility with non-binary attributes, and high fidelity. Extensive experimental results on various facial attributes demonstrate the qualitative and quantitative effectiveness of the proposed AdaTrans over existing state-of-the-art methods, especially in the most challenging scenarios with a large age gap and few labeled examples. The source code is available at https://github.com/Hzzone/AdaTrans.

Sparse-view computed tomography (CT)—using a small number of projections for tomographic reconstruction—enables much lower radiation dose to patients and accelerated data acquisition. The reconstructed images, however, suffer from strong artifacts, greatly limiting their diagnostic value. Current trends for sparse-view CT turn to the raw data for better information recovery. The resultant dual-domain methods, nonetheless, suffer from secondary artifacts, especially in ultra-sparse view scenarios, and their generalization to other scanners/protocols is greatly limited. A crucial question arises: have the image post-processing methods reached the limit? Our answer is not yet. In this paper, we stick to image post-processing methods due to great flexibility and propose global representation(GloRe) distillation framework for sparse-view CT, termed GloReDi. First, we propose to learn GloRe with Fourier convolution, so each element in GloRe has an image-wide receptive field. Second, unlike methods that only use the full-view images for supervision, we propose to distill GloRe from intermediate-view reconstructed images that are readily available but not explored in previous literature. The success of GloRe distillation is attributed to two key components: representation directional distillation to align the GloRe directions, and band-pass-specific contrastive distillation to gain clinically important details. Extensive experiments demonstrate the superiority of the proposed GloReDi over the state-of-the-art methods, including dual-domain ones. The source code is available at https://github.com/longzilicart/GloReDi.

Deep learning has strong learning ability to extract the features from Image datasets. In recent years, deep networks especially deep convolutional neural networks have revolutionized this field. When exposed to a huge number of datasets and their labels, deep learning techniques like Convolutional Neural Networks (CNNs) can produce precise categorization results. However, employing CNNs with scant labeled data might have a number of issues, including the problem of heavy overfitting. Convolutional Neural Networks are the backbone of modern deep learning architectures for the purpose of image classification. We solved image classification problem with different architectures and compare their performances. The objective of this work lies in the approach to study the given datasets. We use domain adaptation approach to highlight the underlying characteristics of these datasets and the different parameters (activation function, weights, regularization, neural simulation etc.) associated with these architectures. In our work, we use three datasets—RS19, UC Merced, and EuroSat images—were utilized in the CNN implementation to training the suggested model. The obtained results effectively demonstrate the local representation capacity of CNNs. Furthermore, this work shows that transfer learning improves classification outcomes in optical remote sensing images, particularly when the training sample is small.

With the development of hyperspectral sensors, there is an increasing amount of accessible hyperspectral data, and the classification task for land cover categories has gained significant attention. Existing classification methods typically extract features from either the pixel or superpixel perspective. However, using a single-scale feature extraction approach fails to simultaneously consider both local and global features of land cover, leading to suboptimal classification results. To address this issue, this paper proposes a parallel graph attention network model based on pixel and superpixel feature fusion (SSPGAT) for hyperspectral image classification, which leverages the fusion of pixel-level and superpixel-level features. The proposed approach first employs spectral convolutional layers to reduce the redundant spectral dimension. Then, it utilizes graph attention network (GAT) to extract local and global features of land cover separately from the pixel and superpixel perspectives. Finally, a fully connected network is employed to classify the fused features from both branches. Experimental results on two different datasets demonstrate the effectiveness of the proposed approach.

The development of deep learning technology provides an especially practical tool for hyperspectral image classification. However, the acquisition of labeled samples in a specific domain is usually time-consuming, which is not conducive to the training of neural network. In addition, different domains bring the phenomenon of ‘same object but different spectrum’ to hyperspectral images, which makes it difficult to directly learn from the available samples of other domains to serve a specific domain. To address this problem, we propose a deep domain adaptation network by aligning the second-order moment of source and target domain through cross-scene transfer learning. Specifically, we use abundant labeled samples in the source domain to train a 3DCNN with the purpose of identifying the target domain. Meanwhile, to reduce the distribution difference, we minimize the covariance distance between source domain and the target domain. The experimental results on two groups of hyperspectral images have shown that the proposed method can outperform several baseline methods.

The primary task of hyperspectral anomaly detection (HAD) is to distinguish targets with noticeable spectral variances from their surroundings. Background purification for dictionary construction is vital for representation based HAD. In this letter, we present a novel HAD method based on background purification via deep autoencoding gaussian mixture Model. First, the low-rank representation (LRR) model is applied to separate the sparse anomalies from the low-rank background component. Secondly, we innovatively apply the deep autoencoding gaussian mixture model (DAGMM) for dimensionality reduction and background purification, which combines the feature mapping and Gaussian density estimation for joint training in the deep latent space. In addition, we propose a dictionary construction strategy based on the gaussian mixture model, achieving the exclusion of possible anomalies and inclusion of background features. Experiments on two real datasets illustrated the superior performance for HAD task of the proposed method.

Infrared small target detection (ISTD) has been widely concerned in certain fields like astronomy, surveillance, and missile early warning system. ISTD is still a challenging task due to the complex backgrounds and small size of targets, which restrict the performance of the convolutional neural networks (CNN) in ISTD. To this end, a dual branch architecture which combines nested attention network and auxiliary transformer enhancement (NANetFormer) is proposed. The CNN-based branch uses channel-spatial-attention embedded U-Net++ architecture to obtain low-level local details of small targets and suppress background noises. The transformer-based branch applies hierarchical self-attention mechanism as an auxiliary enhancement encoder path. Furthermore, we design a local-global feature fusion module to make feature concentration of two branches. Experimental results show that proposed network achieves competitive results compared with other state-of-the-art methods.

Interpreting the decisions of deep learning models has been actively studied since the explosion of deep neural networks. One of the most convincing interpretation approaches is salience-based visual interpretation, such as Grad-CAM, where the generation of attention maps depends merely on categorical labels. Although existing interpretation methods can provide explainable decision clues, they often yield partial correspondence between image and saliency maps due to the limited discriminative information from one-hot labels. This paper develops a Language-Image COnsistency model for explainable image classification, termed LICO, by correlating learnable linguistic prompts with corresponding visual features in a coarse-to-fine manner. Specifically, we first establish a coarse global manifold structure alignment by minimizing the distance between the distributions of image and language features. We then achieve fine-grained saliency maps by applying optimal transport (OT) theory to assign local feature maps with class-specific prompts. Extensive experimental results on eight benchmark datasets demonstrate that the proposed LICO achieves a significant improvement in generating more explainable attention maps in conjunction with existing interpretation methods such as Grad-CAM. Remarkably, LICO improves the classification performance of existing models without introducing any computational overhead during inference.

Crowd counting has important applications in public safety and pandemic control. A robust and practical crowd counting system has to be capable of continuously learning with the newly incoming domain data in real-world scenarios instead of fitting one domain only. Off-the-shelf methods have some drawbacks when handling multiple domains: (1) the models will achieve limited performance (even drop dramatically) among old domains after training images from new domains due to the discrepancies in intrinsic data distributions from various domains, which is called catastrophic forgetting; (2) the well-trained model in a specific domain achieves imperfect performance among other unseen domains because of domain shift; (3) it leads to linearly increasing storage overhead, either mixing all the data for training or simply training dozens of separate models for different domains when new ones are available. To overcome these issues, we investigate a new crowd counting task in incremental domain training setting called lifelong crowd counting. Its goal is to alleviate catastrophic forgetting and improve the generalization ability using a single model updated by the incremental domains. Specifically, we propose a self-distillation learning framework as a benchmark (forget less, count better, or FLCB) for lifelong crowd counting, which helps the model leverage previous meaningful knowledge in a sustainable manner for better crowd counting to mitigate the forgetting when new data arrive. A new quantitative metric, normalized Backward Transfer (nBwT), is developed to evaluate the forgetting degree of the model in the lifelong learning process. Extensive experimental results demonstrate the superiority of our proposed benchmark in achieving a low catastrophic forgetting degree and strong generalization ability.

As one of the most fundamental topics in reinforcement learning (RL), sample efficiency is essential to the deployment of deep RL algorithms. Unlike most existing exploration methods that sample an action from different types of posterior distributions, we focus on the policy sampling process and propose an efficient selective sampling approach to improve sample efficiency by modeling the internal hierarchy of the environment. Specifically, we first employ clustering methods in the policy sampling process to generate an action candidate set. Then we introduce a clustering buffer for modeling the internal hierarchy, which consists of on-policy data, off-policy data, and expert data to evaluate actions from the clusters in the action candidate set in the exploration stage. In this way, our approach is able to take advantage of the supervision information in the expert demonstration data. Experiments on six different continuous locomotion environments demonstrate superior reinforcement learning performance and faster convergence of selective sampling. In particular, on the LGSVL task, our method can reduce the number of convergence steps by 46.7% and the convergence time by 28.5%. Furthermore, our code is open-source for reproducibility. The code is available at https://github.com/Shihwin/SelectiveSampling .

Deep learning based methods, such as the representative vision transformer and convolutional neural network structures, can characterize spatial-spectral features of hyperspectral images (HSI) well and achieve outstanding classification performance. Nevertheless, when land cover is complex, the intra-class spectral consistency may be weak and difficult to express effectively in the original data space, leading to potential bias regarding the validity of spatial-spectral information utilization. We propose a new method GSPFormer that first constructs a global spectral projection space to generate land cover more robust representations and enhance the spectral consistency in local neighborhoods. After that, a space aggregation idea is introduced to obtain the central pixel’s more abundant spectral feature expression for better classification by fusing all spectral features in the local neighborhood. Extensive experiments are conducted on various HSI datasets for evaluating the classification performance of GSPFormer and other state-of-the-art networks. Comparison results indicate the superiority of the proposed method not only in classification accuracy but also in the number of parameters and convergence. The code of GSPFormer will be found at https://github.com/Preston-Dong/GSPFormer.

Remote sensing image change detection refers to finding the changed regions from a pair of registered images. It has important applications in many fields. However, most methods based on convolutional neural networks and transformer have high complexity and cannot be effectively deployed on satellites or drones in practical applications. To address this issue, we propose an extremely lightweight change detection algorithm called ELW_CDNet. Its inference speed is very fast. This method is based on the extremely lightweight shufflenetv2. Moreover, considering that both global as well as local features play an important role in change detection, we design a light global-local feature enhancement module (LGLFEM) for reinforcing the features extracted by the backbone. Specifically, the global feature extraction module in LGLFEM is implemented using separable self-attention, which has linear complexity and very low computational effort. We conduct experiments on two change detection datasets. Compared with some state-of-the-art methods, the proposed method can achieve superior performance with extremely fast inference speed. On the LEVIR-CD dataset, it achieves an F1 score of 90.47%, an IoU of 82.60% and an FPS of 914 with 1.75M parameters and 1.91GFLOPs. The code will be released soon on the site of https://github.com/dyl96/ELW_CDNet.

Lightweight object detection and recognition models are extremely crucial for in-orbit applications, which is the most critical factor for whether deep learning-based object detection and recognition algorithms can be applied to remote sensing satellites for real-time or near real-time processing. Global information is extremely important for object detection and recognition of remote sensing images. However, due to the high computational cost, the existing CNN-based lightweight models over-emphasize on the extraction of local information, while ignoring the global information. For this reason, we propose a lightweight object detection and recognition model (Lightweight Global-Local Detection, LGLDet) based on the especially light global modeling structure. In LGLDet, light global-local module (LGLM) is proposed to extract the global and local information. The LGLM consists of Point2Patch Non-Local (P2PNL), local branch and skip connection. Specifically, P2PNL is proposed to reduce the computation of global long-range dependency modeling. In addition, the feature fusion part and detection head are also designed in a lightweight way. In the experiments, the proposed method can achieve optimal performance with fewer parameters and lower computational complexity than existing CNN-based lightweight models and transformer-based lightweight models with similar parameters or computational complexity. The code will be released on the site of https://github.com/dyl96/LGLDet.

Automatic road detection from remote sensing images has always been a significant research topic. It is of great value to many practical applications. However, there are still some problems need to be solved. First of all, most of existing road detection methods are inefficient because of the sequential processing of the decoder head. Secondly, some existing methods are unable to detect occluded road areas effectively. For this reason, we focus on the speed and occlusion problems in road detection network, and propose a new lightweight road detection method based on multi-scale convolution attention network (MSCAN) and coupled decoder head, LRDNet. In particular, LRDNet adopt multi-scale convolution attention network with large receptive field for feature extraction to solve the occlusion problem, and decode the road surface, road edge and road centerline in a coupled way to improve the speed of road detection and ensure that the road surface detection results have fewer burrs at the road edge. We have performed several experiments on the RNBD dataset. Compared with some state-of-the-art methods, the experimental results prove the validity of the proposed LRDNet. The code will be released soon on the site of https://github.com/dyl96/LRDNet.

Convolutional neural networks (CNNs) have been widely used with its powerful discriminative ability in change detection (CD), but most CNNs-based methods are still exploring ways to capture relatively long-range context in spatial-temporal domain. The recent vision transformer (ViT) models long-range dependencies based on self-attention mechanism, which has been applied in CD. However, such transformer-based architectures do not fully consider the potential of interdependencies among the high-level semantic feature maps and easily overlook local detail features, resulting in a non-compact interior of the large-scale change area and missing small changes. Therefore, we propose a new transformer based hybrid network called interact-feature transformer network with spatial detail enhancement module (IFTSDNet), which takes advantage of transformers to capture long-range context, and of CNNs to extract local information. We design an interact-feature transformer (IFT), which can not only obtain the global contextual information, but also achieve the interactions of high-level semantic feature maps. The spatial detail enhancement module (SDEM) with a group of various receptive fields is built to refine spatial features, which incorporates more discriminative feature representations. Comparative experiments prove the effectiveness of the proposed method, which shows better performance than four recent transformer-based methods. The code will be available at https://github.com/wanglinlin0219/IFTSDNet.

Existing deep clustering methods rely on either contrastive or non-contrastive representation learning for downstream clustering task. Contrastive-based methods thanks to negative pairs learn uniform representations for clustering, in which negative pairs, however, may inevitably lead to the class collision issue and consequently compromise the clustering performance. Non-contrastive-based methods, on the other hand, avoid class collision issue, but the resulting non-uniform representations may cause the collapse of clustering. To enjoy the strengths of both worlds, this paper presents a novel end-to-end deep clustering method with prototype scattering and positive sampling, termed ProPos. Specifically, we first maximize the distance between prototypical representations, named prototype scattering loss, which improves the uniformity of representations. Second, we align one augmented view of instance with the sampled neighbors of another view-assumed to be truly positive pair in the embedding space-to improve the within-cluster compactness, termed positive sampling alignment. The strengths of ProPos are avoidable class collision issue, uniform representations, well-separated clusters, and within-cluster compactness. By optimizing ProPos in an end-to-end expectation-maximization framework, extensive experimental results demonstrate that ProPos achieves competing performance on moderate-scale clustering benchmark datasets and establishes new state-of-the-art performance on large-scale datasets. Source code is available at https://github.com/Hzzone/ProPos.

To minimize the impact of age variation on face recognition, age-invariant face recognition (AIFR) extracts identity-related discriminative features by minimizing the correlation between identity- and age-related features while face age synthesis (FAS) eliminates age variation by converting the faces in different age groups to the same group. However, AIFR lacks visual results for model interpretation and FAS compromises downstream recognition due to artifacts. Therefore, we propose a unified, multi-task framework to jointly handle these two tasks, termed MTLFace, which can learn the age-invariant identity-related representation for face recognition while achieving pleasing face synthesis for model interpretation. Specifically, we propose an attention-based feature decomposition to decompose the mixed face features into two uncorrelated components-identity- and age-related features-in a spatially constrained way. Unlike the conventional one-hot encoding that achieves group-level FAS, we propose a novel identity conditional module to achieve identity-level FAS, which can improve the age smoothness of synthesized faces through a weight-sharing strategy. Benefiting from the proposed multi-task framework, we then leverage those high-quality synthesized faces from FAS to further boost AIFR via a novel selective fine-tuning strategy. Furthermore, to advance both AIFR and FAS, we collect and release a large cross-age face dataset with age and gender annotations, and a new benchmark specifically designed for tracing long-missing children. Extensive experimental results on five benchmark cross-age datasets demonstrate that MTLFace yields superior performance than state-of-the-art methods for both AIFR and FAS. We further validate MTLFace on two popular general face recognition datasets, obtaining competitive performance on face recognition in the wild. The source code and datasets are available at http://hzzone.github.io/MTLFace.

Noisy image segmentation using fuzzy c-means clustering (FCM) algorithm is an important direction in the field of image segmentation, among which FCM using local information or nonlocal information constraints has made some progress. However, as the noise density increases, the local information cannot restore the real image. The nonlocal information-based segmentation method overly retains the noise and thus produces false edges resulting in the degradation of image segmentation accuracy (SA). To address these problems, this article uses regional and local information to develop an adaptive FCM image segmentation algorithm. First, the regional information and local information are obtained by the regional information filtering method and the local information filtering method. Then, the weighted image is constructed in a certain way. Second, to make the objective function adaptively adjust the constraint ratio of the original image and the weighted image pixel by pixel, the degree of constraint is adaptively changed by using the difference between them as the coefficients of both through certain transformation. Finally, to speed up the convergence of the objective function, we add the number of clustering samples to the denominator of the objective function by certain transformations to reduce the number of iterations of the objective function. The proposed algorithm achieves more than 90% in image SA and mean Intersection-over-Union through synthetic and real-world image tests and acquires good performance in other indexes.

With the rapid development of earth observation technology, the fusion of multisource remote sensing (RS) data, has become an important research field in land cover classification. Particularly, the fusion of hyperspectral image (HSI) and light detection and ranging (LiDAR) data, which can provide complementary information to each other, has attracted an increasing attention for land cover classification tasks. However, most of the existing methods focus on multi-source information fusion on feature level and ignore the discrepancy of the land covers with the same label among different kinds of viewpoints, which limit the further improvement of classification performance. So, this paper proposes a novel HSI-LiDAR classification method based on multi-view feature learning and multi-level information fusion, which consists of a multi-view data representation (MVDR) strategy, a multi-branch dual-channel graph convolutional networks (MB-DCGCNs) model and a progressively high-confidence label assignment (PHCLA) scheme. MVDR is designed by jointly utilizing multiple attributes of spatial information and multi-source RS data to re-express multi-source land covers with diversity and complementarity. The MB-DCGCNs model which aims at integrating spectral-spatial-elevation information on feature level is employed on the above multi-view re-expressed HSI-LiDAR data to explore land cover feature representations from different views. Furthermore, a PHRLA scheme is proposed to classify the land covers with high-reliability by a combination of decision-level label prediction and pixel-level label assignment. Comparative experiments on two benchmark datasets with several state-of-the-art classification methods validate the effectiveness and superior performance of the proposed method.

Graph neural networks (GNNs) have been ubiquitous in graph node classification tasks. Most GNN methods update the node embedding iteratively by aggregating its neighbors' information. However, they often suffer from negative disturbances, due to edges connecting nodes with different labels. One approach to alleviate this negative disturbance is to use attention to learn the weights of aggregation, but current attention-based GNNs only consider feature similarity and suffer from the lack of supervision. In this article, we consider label dependency of graph nodes and propose a decoupling attention mechanism to learn both hard and soft attention. The hard attention is learned on labels for a refined graph structure with fewer interclass edges so that the aggregation's negative disturbance can be reduced. The soft attention aims to learn the aggregation weights based on features over the refined graph structure to enhance information gains during message passing. Particularly, we formulate our model under the expectation-maximization (EM) framework, and the learned attention is used to guide label propagation in the M-step and feature propagation in the E-step, respectively. Extensive experiments are performed on six well-known benchmark graph datasets to verify the effectiveness of the proposed method.

Facial attribute manipulation (FAM) aims to infer desired facial images by modifying specific attributes while keeping others unchanged. Existing works suffer from the entanglement of facial attributes, leading to unexpected artifacts and the loss of facial identity information after editing. To alleviate these issues, we propose a novel FAM framework based on StyleGAN, termed VR-FAM, which can meet the requirements of FAM—editing ability, distortion, and fidelity. First, we propose a variance-reduced encoder to make the latent space close to the one of StyleGAN. Second, we present a nonlinear latent transformation network, which can convert the source latent code to target latent code in line with the nonlinear latent space of StyleGAN. Experimentally, we evaluate the proposed FAM framework on the benchmark FFHQ dataset and demonstrate the improvement gain over the recently published models in terms of edit accuracy and fidelity.

Object detection for high resolution images has been an important cornerstone in remote sensing interpretation. Though considerable success has been made, there still exists issues for hierarchical semantic representation and integration, which limit the performance of existing methods. Therefore, we first analyze different integration patterns of deformable convolution on RetinaNet comprehensively to enhance the adaptability for the geometrical variations elegantly, which may provide the instructions of the future backbone designs for remote sensing images. Second, a feature pyramid network with filter mechanism (F- FPN) is proposed to strengthen the interaction of hierarchical semantics in feature reflow. Experiments on a challenging public dataset DIOR indicate the great performance.

Unmanned aerial vehicles (UAV) low-altitude remote sensing has been extensively applied in agriculture as an important way of monitoring the growth and physiological conditions of crops. Due to the limited imaging width, multispectral images captured by UAV need to be stitched in order to obtain the panoramic image of the whole area. However, sometimes it is difficult to stitch multispectral images by using the traditional stitching methods when there is blurring or uneven local brightness of the images. Accordingly, in this paper, we propose an effective and robust UAV multispectral image stitching method based on an improved As-Projective-As-Possible (APAP) algorithm via line segment correction. We handle the stitching problem in two steps: firstly, inter-band registration and global homography transformation are used to ensure the precision of stitching results. Then the line segment detection and correction based on line segment detector and least-square estimation is conducted, which can protect the unique line segment structure in farmland area. The experimental results substantiate the superiority of our method both visually and quantitatively when compared with state-of-the-art methods.

With the increase in the amount of remote sensing images (RSIs), deep learning (DL) has been used to the change detection (CD) task in remote sensing field and achieved good results. However, most existing methods do not take full advantage of the temporal dependence of the multi-temporal images. In this paper, we propose a novel method for CD, namely SNN-LSTM (a deep Siamese neural network (SNN) with convolutional Long Short-Term Memory (ConvLSTM) and channel attention module (CAM)), especially for capturing and representing spatial-temporal information effectively. It mainly contains three parts. First, a network based on Siamese convolutional architecture is designed to extract multi-level features. Then, a ConvLSTM block is introduced to further obtain time dependency of multi-temporal RSIs, and spatial information is also extracted simultaneously. Finally, CAM blocks are used to refine the extracted multi-level features, enhance the feature of changes, and eventually generate change map. The experiments are conducted on LEVIR-CD dataset, both visual results and quantitative assessment prove that the proposed method outperforms several state-of-the-art methods.

Hyperspectral images (HSIs) are inevitably influenced by various noise, including Gaussian noise, sparse noise and so on, which could degrade the HSIs and limit their applications greatly. Deep neural network (DNNs) based HSIs denoising methods have been widely used in recent years. However, the existing methods based on deep learning are mainly for Gaussian noise removal, and few are for mixed noise. Accordingly, we propose a two-stage cascade refined network consisting of two subnetworks for hyperspectral mixed Gaussian and sparse noise reduction. In the first stage, the spatial-spectral features are extracted by a feature extraction block based on attentions mechanism firstly. Then the multi-band noise is obtained by feeding the extracted features into the multi-band noise estimation subnetwork with encoder-decoder structure. Finally, the single-band denoising subnetwork in the second stage further refines the output of the previous subnetwork to accomplish single-band noise reduction. The experiments on HSI show that the superiority of the proposed method compared with four typical methods for mixed noise removal.

In the recent decade, multidimensional or tenso-rial time series have drawn increasing attention for their rich spatial and temporal information. In data collection, missing tensorial time instances can always occur at any random time steps, which cause the time steps to be not equally spaced, i.e., irregular. Furthermore, it is very likely that the tensorial time series are influenced by other tensor factors, yet from external resources. These driving tensor factors are referred to as tensor controls, whereas the influenced tensorial time series are named as tensor responses. Existing methods either cannot predict tensor responses using all available past tensor controls, or are not able to directly model continuous-time tensor-valued processes, incurring the curse-of-dimensionality issue. Therefore, we propose tensor neural controlled differential equations (TENCDEs) which incorporate tensor controls and explicitly define the continuous-time tensor-valued processes for tensor controls and tensor responses aiming at resolving irregular tensorial time series issues. TENCDE also effectively captures both spatial and temporal information, and avoids the curse-of-dimensionality issue. The unique solution and the consistency and convergence of solutions are also guaranteed. The experiments demonstrate the proposed TENCDE outperforms the existing models including the state-of-the-art methods.

Though single image super-resolution (SR) has witnessed incredible progress, the increasing model complexity impairs its applications in memory-limited devices. To solve this problem, prior arts have aimed to reduce the number of model parameters and sparsity has been exploited, which usually enforces the group sparsity constraint on the filter level and thus is not arbitrarily adjustable for satisfying the customized memory requirements. In this paper, we propose an individual kernel sparsity (IKS) method for memory-efficient and sparsity-adjustable image SR to aid deep network deployment in memory-limited devices. IKS performs model sparsity in the weight level that implicitly allocates the user-defined target sparsity to each individual kernel. To induce the kernel sparsity, a soft thresholding operation is used as a gating constraint for filtering the trivial weights. To achieve adjustable sparsity, a dynamic threshold learning algorithm is proposed, in which the threshold is updated by associated training with the network weight and is adaptively decayed with the guidance of the desired sparsity. This work essentially provides a dynamic parameter reassignment scheme with a given resource budget for an off-the-shelf SR model. Extensive experimental results demonstrate that IKS imparts considerable sparsity with negligible effect on SR quality. The code is available at: https://github.com/RaccoonDML/IKS.

Hyperspectral images (HSIs) can provide abundant and diverse features which are helpful for classification, such as spectral, texture, and shape features. Combining these features can enhance the ability to describe the characteristics of different classes of land covers. However, most of the existing methods usually stack features from multiple views to construct high-dimensional data and then learn information from it which may waste information inherent in different feature spaces. In this letter, we proposed a multiview spectral–spatial feature extraction (FE) and fusion framework for the analysis and classification of the HSIs. First, different and complementary spatial features extracted by extended multiattribute profiles (EMAPs), gray-level cooccurrence matrix (GLCM), and Gabor from the original HSI are, respectively, stacked with the spectral bands to construct multiview data set for one single scene of HSI. Thus, each sample can be represented in different spectral–spatial domains. Then, a semisupervised FE method, which combines local fisher discriminant analysis (LFDA) that explores discriminative information from limited labeled samples and the improved neighborhood preserving embedding (NPE) that aims at maintaining the local neighborhood structure from a global perspective, is applied on the multiview data set to eliminate redundant information and obtain multiview spectral–spatial features. Note that the improved NPE which adds spatial interpixel correlations to similarity measure between samples is applied on all the samples rather than unlabeled samples, and hence, the best spatial nearby neighbors for each sample could be found from the whole data. Finally, we integrate these multiview features with diversity and complementarity to construct the intact feature representation for each sample. The experimental results show that the proposed method outperforms the state-of-the-art multiview FE methods for HSI classification.

Hyperspectral images (HSIs) have been utilized in various fields due to abundant information, whose application is to a great extent limited by the number of labeled samples. Hence, how to exploit the diverse and complementary characteristics of multiple information from different views inherent in the HSIs is very critical for improving the classification performance under the condition of small-sized training set. This letter presents a novel method to tackle the small labeled sample size problem with semisupervised self-learning (S3L) and multiview information fusion. First, a sample augmentation scheme based on S3L and high-reliable neighborhood structure is designed for realizing training set enlargement. Thus, the pseudo-labeled samples with high-quality would be automatically picked out from the unlabeled data by exploiting complementary information from multiple views, i.e., semantic information, spectral bands, texture and geospatial information. Then, we retrain the classifier by the enlarged training set and generate an intermediate classification map. Finally, we simultaneously utilize the location and gradient information of the samples to adaptively realize the refined landcover classification. Experimental results on three widely-used data sets compared with several representative methods for small-sized HSI classification validate the effectiveness of the proposed method.

Due to the difficulty and time-consuming of labeling ground truth map in practical situations, unsupervised multiple change detection (MCD) for multispectral images (MSIs) have attracted much attention in recent years. One possible strategy to obtain multiple changes is to assign labels to the binary change result. However, some methods are difficult to obtain the accurate binary result because of the complexity of backgrounds; moreover, assigning labels is also a challenge owing to the limitation of the number of spectral channels in MSIs. Therefore, we propose a novel unsupervised MCD framework based on auto-updating multitemporal matrix factorization (AMMF) and spatiospectral channel augmentation (SSCA). In AMMF, the accurate binary change result can be detected based on joint matrix factorization, during which the distribution and subspace information of each temporal image are regularized to encode the spatiotemporal correlation. In SSCA, some novel augmentation strategies are introduced to increase the number of channels in MSIs to form the normalized high-dimensional maps for each temporal image based on nonlinear operations and convolutional sparse analysis, respectively. MCD can be achieved by integrating the binary change result and directional information that can be calculated by high-dimensional maps of different temporal images. Experiments are conducted on two real MSIs, indicating that the proposed framework performs well in detecting multiple changes.

Automatic road detection has always been an important research problem in the field of remote sensing image processing, which is of great significance for many applications. However, many road areas cannot be effectively detected and the existing road detection methods suffer from unsmooth edges due to shadow phenomena and the occlusion of objects such as trees. For this reason, we propose a cascaded automatic road detection network based on edge sensing module and attention module, called CasEANet. CasEANet consists of three tasks including road surface detection, road edge detection and road centerline extraction. An encoder-decoder structure integrating edge sensing module and attention module is applied to detect the road surface. In order to obtain smoother road edges, the edge sensing module is designed to enhance the perception of road edges. The attention module is employed to guide the network to reinforce the perception of global information, aiming to solve the problem of discontinuity in the detection of road surfaces that are obscured by trees. The centerline extraction is adopted to assist the training of road surface detection. Experiments performed on the RNBD dataset prove the effectiveness of CasEANet. Specially, the F1 score, overall accuracy and balanced error rates of the CasEANet are 0.946, 0.986 and 0.0219 respectively, outperform the other state-of-the-art methods. The code will release soon on https://github.com/HITLDY/CasEANet.

Automated shadow detection is an important research problem in the field of remote sensing image processing. The shadow regions seriously affect the interpretation of the remote sensing images. However, the existing methods have poor detection effect for small shadow regions, and the ability to distinguish between weakly illuminated regions and shadow regions is insufficient. For this reason, we propose a shadow detection network based on multiscale spatial attention mechanism for aerial remote sensing images, called MSASDNet. First, the backbone based on residual block is employed to extract the preliminary features of the input image. Then, we design a multiscale feature extraction module based on the spatial attention mechanism to extract multiscale features with spatial attention information, which can suppress the influence of complex nonshadow regions on the detection and improve the detection ability of small shadow regions. Finally, the decoder structure based on deconvolution is used to predict the shadow mask from the combined feature. Experiments performed on the Aerial Imagery dataset for Shadow Detection (AISD) dataset demonstrate the superiority of MSASDNet in terms of quantitative and qualitative comparison with several state-of-the-art methods. The code will release soon inhttps://github.com/HITLDY/MSASDNet.

Compared with visible and near-infrared images, the long-wave infrared region hyperspectral image (LWIR HSI) is more vulnerable to noise pollution in the acquisition process due to its specific imaging mode. In this letter, a new restoration method is proposed using total variation based on bidirectional estimation and brightness temperature smoothing (BBSTV), which can remove dead lines and restore junk bands effectively. The proposed method introduces the linear relation between brightness temperature and emissivity derived from radiative transfer model (RTM) to restoration processing. Besides, bilateral estimation is used to complete the loss information of noise-polluted bands to achieve a faster convergence speed of total variation (TV) method. Both simulated and real LWIR HSI experiments were conducted to verify the improvements of the BBSTV method in quantitative and qualitative ways.

This letter presents a novel method for hyperspectral image (HSI) restoration, which aims to improve the removal effectiveness of the sparse noise. In contrast to the existing approaches that employ the$L_{1}$-norm for tractable optimization, we apply the non-convex non-smooth$L_{0}$-norm to measure the sparsity of the impulse noise, stripes, deadlines, and other outliers accurately. By combining the low-rank and total variation (TV) priors to exploit the intrinsic properties of the clean HSI and using the patch scheme to preserve local features, the$L_{0}$-PLRTV restoration model is established. In order to deal with the optimization problem, we introduce an equivalent primal-dual formulation to reformulate the$L_{0}$-norm term, and develop a minimization approach for the objective function based on the alternating iterative method. The simulated and real data experiments confirm that the proposed algorithm can effectively reduce the sparse noise in HSI.

With the development of convolutional neural network (CNN), many CNN-based object detection methods have made a remarkable success in very high-resolution (VHR) remote sensing images (RSIs). However, the standard convolution has a fixed receptive field, which makes it deficient in dynamic feature capture; complex backgrounds may also lead to the degradation of detection performance. Accordingly, this letter proposes a novel multiscale semantic guidance network (MSGN) to tackle these problems, wherein, based on the deformable convolution, an improved feature extraction backbone is proposed to capture features dynamically. Moreover, features from different layers are used to ensure the ability for detecting multiscale objects. Furthermore, a multilevel semantic guidance filtering subnetwork is proposed based on the designed backward semantic guidance filtering (BSGF) module, to suppress the complex backgrounds. Experimental results show that the proposed MSGN has stronger robustness and a better accuracy for multiscale object detection, compared with other reference methods.

Unlike the conventional facial expressions, micro-expressions are involuntary and transient facial expressions capable of revealing the genuine emotions that people attempt to hide. Therefore, they can provide important information in a broad range of applications such as lie detection, criminal detection, etc. Since micro-expressions are transient and of low intensity, however, their detection and recognition is difficult and relies heavily on expert experiences. Due to its intrinsic particularity and complexity, video-based micro-expression analysis is attractive but challenging, and has recently become an active area of research. Although there have been numerous developments in this area, thus far there has been no comprehensive survey that provides researchers with a systematic overview of these developments with a unified evaluation. Accordingly, in this survey paper, we first highlight the key differences between macro- and micro-expressions, then use these differences to guide our research survey of video-based micro-expression analysis in a cascaded structure, encompassing the neuropsychological basis, datasets, features, spotting algorithms, recognition algorithms, applications and evaluation of state-of-the-art approaches. For each aspect, the basic techniques, advanced developments and major challenges are addressed and discussed. Furthermore, after considering the limitations of existing micro-expression datasets, we present and release a new dataset — calledmicro-and-macro expression warehouse(MMEW) — containing more video samples and more labeled emotion types. We then perform a unified comparison of representative methods on CAS(ME)$^2$for spotting, and on MMEW and SAMM for recognition, respectively. Finally, some potential future research directions are explored and outlined.

Gait is a unique biometric feature that can be recognized at a distance; thus, it has broad applications in crime prevention, forensic identification, and social security. To portray a gait, existing gait recognition methods utilize either a gait template which makes it difficult to preserve temporal information, or a gait sequence that maintains unnecessary sequential constraints and thus loses the flexibility of gait recognition. In this paper, we present a novel perspective that utilizes gait as a deep set, which means that a set of gait frames are integrated by a global-local fused deep network inspired by the way our left- and right-hemisphere processes information to learn information that can be used in identification. Based on this deep set perspective, our method is immune to frame permutations, and can naturally integrate frames from different videos that have been acquired under different scenarios, such as diverse viewing angles, different clothes, or different item-carrying conditions. Experiments show that under normal walking conditions, our single-model method achieves an average rank-1 accuracy of 96.1 percent on the CASIA-B gait dataset and an accuracy of 87.9 percent on the OU-MVLP gait dataset. Under various complex scenarios, our model also exhibits a high level of robustness. It achieves accuracies of 90.8 and 70.3 percent on CASIA-B under bag-carrying and coat-wearing walking conditions respectively, significantly outperforming the best existing methods. Moreover, the proposed method maintains a satisfactory accuracy even when only small numbers of frames are available in the test samples; for example, it achieves 85.0 percent on CASIA-B even when using only 7 frames. The source code has been released at https://github.com/AbnerHqC/GaitSet.

By utilizing physical models of the atmosphere collected from the current weather conditions, the numerical weather prediction model developed by the European Centre for Medium-range Weather Forecasts (ECMWF) can provide the indicators of severe weather such as heavy precipitation for an early-warning system. However, the performance of precipitation forecasts from ECMWF often suffers from considerable prediction biases due to the high complexity and uncertainty for the formation of precipitation. The bias correcting on precipitation (BCoP) was thus utilized for correcting these biases via forecasting variables, including the historical observations and variables of precipitation, and these variables, as predictors, from ECMWF are highly relevant to precipitation. The existing BCoP methods, such as model output statistics and ordinal boosting autoencoder, do not take advantage of both spatiotemporal (ST) dependencies of precipitation and scales of related predictors that can change with different precipitation. We propose an end-to-end deep-learning BCoP model, called the ST scale adaptive selection (SSAS) model, to automatically select the ST scales of the predictors via ST Scale-Selection Modules (S3M/TS2M) for acquiring the optimal high-level ST representations. Qualitative and quantitative experiments carried out on two benchmark datasets indicate that SSAS can achieve state-of-the-art performance, compared with 11 published BCoP methods, especially on heavy precipitation.

With the wide application of deep learning (DL), change detection (CD) for remote-sensing images (RSIs) has realized the leap from the traditional to the intelligent methods. However, many existing methods still need further improvement in practical applications, especially in increasing the effectiveness of feature extraction and reducing the model computational cost. In this article, we propose a novel deep multiscale Siamese network with parallel convolutional structure (PCS) and self-attention (SA) (MSPSNet), which has excellent capabilities of feature extraction and feature integration under an acceptable consumption. It mainly contains three subnetworks: deep multiscale feature extraction, feature integration by the PCS, and feature refinement based on the SA. In the first subnetwork, a deep multiscale Siamese network based on convolutional block is designed to depict the image features at different scales for different temporal images. In the subsequent subnetworks, a PCS model is proposed to integrate multiscale features of different temporal images, and then, an SA model is constructed to further enhance the representation of image information. Experiments are conducted on two public RSI datasets, indicating that the proposed framework performs well in detecting changes.

Anomaly detection in a hyperspectral image (HSI) has been actively researched in the field of remote sensing due to its significant application requirements. Traditional methods were based on the spatial models for the background to detect the anomaly targets. However, in detecting the spectral anomaly targets, they led to two problems: 1) the spatial characteristics of spectral anomaly targets are not obvious, which causes many false alarms in detection and 2) spectral anomaly usually occurs in the local band of targets, while the rest of the spectrum is similar to the ones of surrounding backgrounds, which leads to missed detection. This article proposes a novel hyperspectral anomaly detection method for spectral anomaly targets based on spatial and spectral constraints (SASCs). This model finds suspected anomaly target part as spatial anomaly results through SASCs. Then, the feedback process determines the spectral anomaly through the spectral difference between the tested pixel and the surrounding background. It is fed back to the spatial anomaly results to obtain final detection results. Furthermore, in order to enlarge the spectral difference between anomaly and background while suppressing the background, the optimal order of fractional Fourier transform (FrFT) is determined by combining spatial anomaly results with the uncertainty principle, which is used in FrFT of HSI. Experimental results show that the proposed method suppresses the background and reduces the false alarm rate. The feedback mechanism effectively reduces the missing detection rate, achieving a promising detection accuracy.

Anomaly detection is a hot topic in the field of hyperspectral image (HSI) processing, which aims to find anomalous targets through spatial and spectral differences from a complex background. However, there still remains two unsolved problems in anomaly detection methods: 1) since the global image covers more complex scenes, it inevitably suffers a lot of false alarms in the anomaly detection and 2) the sensitivity of the target to the global mode or the local mode detector is different. A detector that only considers one mode usually misses some targets, which makes the detection rate low. This article proposes an anomaly detection method for HSIs with potential target region extraction. The potential target region is extracted through the combination of the tensor robust principal component analysis (RPCA) and the curvature circle model. The obtained subregions are used in the region traversing method to filter out the areas not containing the target. In addition, in order to adapt to the different targets contained in the region, the proposed method combines the isolated forest of the global model and the statistical correlation of the local model, thereby enhancing the separability of the target and the background. Experimental results show that the proposed method can effectively extract potential target regions and greatly reduce the false alarm rate. Furthermore, the proposed method can highlight the target for a higher detection rate and accuracy.

Anomaly detection is an active topic in hyperspectral image processing. Recently, low-rank representation (LRR)-based approaches have shown satisfactory results in wide anomaly detection applications. However, the existing LRR methods still have the following two problems: i) Setting a fixed value as a termination condition of the iterative constraint often results in the loss of target information, leading to a low detection rate with some missing targets. ii) Noise after LRR still remains in the sparse part, which increases false alarms. This paper proposes the tensor approximation with LRR and the kurtosis correlation constraint method for anomaly detection. The hyperspectral image is regarded as a third-order tensor for the LRR process. In the optimization process, the background suppression degree is obtained through the background dictionary to determine the iteration termination condition. After the iterative optimization is completed, the low-rank tensor that can fully represent the background is obtained. Also, the difference between the original hyperspectral image tensor and the low-rank tensor is used as the input of the kurtosis correlation constraint. The kurtosis correlation constraint compares the similarity between the current pixel and its surrounding pixels to detect the anomaly, where the kurtosis in the high-order statistical feature is introduced to avoid the interference of noise. The experimental results illustrate that the proposed method can retain the complete target information to highlight targets while suppressing background.

In hyperspectral image classification, deep learning (DL) based on abundant training samples has demonstrated its significance in classification performance. However, due to the limitation of available samples and the imbalance/similarity of classes in small-sized datasets, data-driven DL algorithms can hardly extract representative and effective features for interclass classification, and the subtle diagnostic spectral features for intraclass classification are easily covered or lost in the iterative feature extraction (FE). The restricted FE of interclass/intraclass results in the accuracy reduction and performance limitation of refined classification. To mitigate these issues, an attention network with multiscale receptive fields (MRFs) is proposed, embedding an inversion subnet for relative water content retrieval (RWCR). In classification, the three critical parts in the proposed network, namely, MRFs, embedded subnet, and multiple-attention mechanism, are responsible for multiscale feature merging, relative water content (RWC) feature enhancement, and paying attention to bands, channels, and multiscale features, respectively. The ablation studies on small-sized datasets show the accuracy improvements of interclass and intraclass in refined classification, which verifies the effectiveness of critical parts for extracting representative features and taking RWC features as the diagnostic biochemical signature from unbalanced and similar classes. The comparison results with typical DL models demonstrate the superiority of the proposed network. Moreover, the competitive advantage of the proposed network is demonstrated in comparison with traditional and state-of-the-art HSI classification methods.

Hypersharpening (namely, hyperspectral (HS) and multispectral (MS) image fusion) aims at enhancing the spatial resolution of HS image via an auxiliary higher resolution MS image. Currently, numerous hypersharpening methods are proposed successively, among which the unmixing-based approaches have been widely researched and demonstrated their effectiveness in the spectral fidelity aspect. However, existing unmixing-based fusion methods substantially employ mathematical techniques to solve the spectral mixture model, without taking full advantage of the collaborative spatial–spectral information that is usually helpful for abundance estimation improvement. To overcome this drawback, in this article, a novel unmixing-based HS and MS image fusion method, via a convolutional neural network (CNN), is proposed to promote spectral fidelity. The main idea of this work is to use CNN to fully explore the spatial information and the spectral information of both HS and MS images simultaneously, thereby enhancing the accuracy of estimating the abundance maps. Experiments on four simulated and real remote sensing data sets demonstrate that the proposed method is beneficial to the spectral fidelity of the fused images compared with some state-of-the-art algorithms. Meanwhile, it is also easy to implement and has a certain advantage in running time.

Image ordinal estimation is to predict the ordinal label of a given image, which can be categorized as an ordinal regression (OR) problem. Recent methods formulate an OR problem as a series of binary classification problems. Such methods cannot ensure that the global ordinal relationship is preserved since the relationships among different binary classifiers are neglected. We propose a novel OR approach, termed convolutional OR forest (CORF), for image ordinal estimation, which can integrate OR and differentiable decision trees with a convolutional neural network for obtaining precise and stable global ordinal relationships. The advantages of the proposed CORF are twofold. First, instead of learning a series of binary classifiers independently, the proposed method aims at learning an ordinal distribution for OR by optimizing those binary classifiers simultaneously. Second, the differentiable decision trees in the proposed CORF can be trained together with the ordinal distribution in an end-to-end manner. The effectiveness of the proposed CORF is verified on two image ordinal estimation tasks, i.e., facial age estimation and image esthetic assessment, showing significant improvements and better stability over the state-of-the-art OR methods.

Intensity modulation and direct detection (IM-DD) based optical wireless communication (OWC), requires the modulated signal to be real and non-negative. To satisfy the requirements, this paper proposes two types of mixed orthogonal frequency division multiplexing (X-OFDM) waveforms. The Hermitian symmetry (HS) characteristic of the sub-carriers in the frequency domain, guarantees the signal in the time domain to be real, which reduces the spectral efficiency to 1/2. For the odd sub-carriers in the frequency domain, the signal in the time domain after the inverse fast fourier transform (IFFT) is antisymmetric. For the even sub-carriers in the frequency domain, the signal in the time domain after the IFFT is symmetric. Based on the antisymmetric and symmetric characteristics, the two types of X-OFDM waveforms are designed to guarantee the signal in the time domain to be non-negative. With$M$sub-carriers in the frequency domain, the generated signal in the time domain has$3M/2$points, which further reduces the spectral efficiency to 1/3. The numerical simulations show that, the two types of X-OFDM waveforms greatly enhance the power efficiency considering the OWC channel with the signal-dependent noise and/or the signal-independent noise.

To minimize the effects of age variation in face recognition, previous work either extracts identity-related discriminative features by minimizing the correlation between identity- and age-related features, called age-invariant face recognition (AIFR), or removes age variation by transforming the faces of different age groups into the same age group, called face age synthesis (FAS); however, the former lacks visual results for model interpretation while the latter suffers from artifacts compromising downstream recognition. Therefore, this paper proposes a unified, multi-task framework to jointly handle these two tasks, termed MTL-Face, which can learn age-invariant identity-related representation while achieving pleasing face synthesis. Specifically, we first decompose the mixed face features into two uncorrelated components—identity- and age-related features—through an attention mechanism, and then decorrelate these two components using multi-task training and continuous domain adaption. In contrast to the conventional one-hot encoding that achieves group-level FAS, we propose a novel identity conditional module to achieve identity-level FAS, with a weight-sharing strategy to improve the age smoothness of synthesized faces. In addition, we collect and release a large cross-age face dataset with age and gender annotations to advance AIFR and FAS. Extensive experiments on five benchmark cross-age datasets demonstrate the superior performance of our proposed MTLFace over state-of-the-art methods for AIFR and FAS. We further validate MTLFace on two popular general face recognition datasets, showing competitive performance for face recognition in the wild. The source code and dataset are available at https://github.com/Hzzone/MTLFace.

Gait recognition plays a vital role in human identification since gait is a unique biometric feature that can be perceived at a distance. Although existing gait recognition methods can learn gait features from gait sequences in different ways, the performance of gait recognition suffers from insufficient labeled data, especially in some practical scenarios associated with short gait sequences or various clothing styles. It is unpractical to label the numerous gait data. In this work, we propose a self-supervised gait recognition method, termed SelfGait, which takes advantage of the massive, diverse, unlabeled gait data as a pre-training process to improve the representation abilities of spatiotemporal backbones. Specifically, we employ the horizontal pyramid mapping (HPM) and micro-motion template builder (MTB) as our spatiotemporal backbones to capture the multi-scale spatiotemporal representations. Experiments on CASIA-B and OU-MVLP benchmark gait datasets demonstrate the effectiveness of the proposed SelfGait compared with four state-of-the-art gait recognition methods. The source code has been released at https://github.com/EchoItLiu/SelfGait.

Although impressive results have been achieved for age progression and regression, there remain two major issues in generative adversarial networks (GANs)-based methods: 1) conditional GANs (cGANs)-based methods can learn various effects between any two age groups in a single model, but are insufficient to characterize some specific patterns due to completely shared convolutions filters; and 2) GANs-based methods can, by utilizing several models to learn effects independently, learn some specific patterns, however, they are cumbersome and require age label in advance. To address these deficiencies and have the best of both worlds, this paper introduces a dropout-like method based on GAN (RoutingGAN) to route different effects in a high-level semantic feature space. Specifically, we first disentangle the age-invariant features from the input face, and then gradually add the effects to the features by residual routers that assign the convolution filters to different age groups by dropping out the outputs of others. As a result, the proposed RoutingGAN can simultaneously learn various effects in a single model, with convolution filters being shared in part to learn some specific effects. Experimental results on two benchmarked datasets demonstrate superior performance over existing methods both qualitatively and quantitatively.

The progression of lung cancer implies the intrinsic ordinal relationship of lung nodules at different stages—from benign to unsure then to malignant. This problem can be solved by ordinal regression methods, which is between classification and regression due to its ordinal label. However, existing convolutional neural network-based ordinal regression methods only focus on modifying classification head based on a randomly sampled mini-batch of data, ignoring the ordinal relationship resided in the data itself. In this paper, we propose a Meta Ordinal Weighting Network (MOW-Net) to explicitly align each training sample with a meta ordinal set (MOS) containing a few samples from all classes. During the training process, the MOW-Net learns a mapping from samples in MOS to corresponding class-specific weight. We further propose a meta cross-entropy loss to optimize the network in a meta-learning scheme. Experimental results demonstrate that the MOW-Net achieves better accuracy than the state-of-the-art ordinal regression methods, especially for the unsure class.

Positive-unlabeled (PU) learning is a learning paradigm when only positive and unlabeled data are available in the training stage. This paradigm is particularly useful for the applications that the negative samples are hard to define or expensive to obtain. We propose a simple yet effective and scalable method to address PU learning with a novel asymmetric loss. The proposed asymmetric loss behaviors differently for the prediction errors of the labeled and unlabeled samples, and thus encourages the identification of the negative examples from the unlabeled set. For the PU learning with SCAR assumption, neither hyper-parameter nor class prior is required to be tuned or known. For the situation with selection bias on the labeled samples, we propose a heuristic method to automatically choose the hyper-parameter according to the class prior on the training data. Compared with previous approaches, our method only requires a slight modification of the conventional cross-entropy loss and is compatible with various deep neural networks in an end-to-end way. Extensive experiments on synthetic and real-world datasets with and without SCAR assumption verify the superior performance of the proposed method.

Hyperspectral image (HSI) classification methods based on deep learning (DL) algorithms have achieved significant improvements on abundant samples. However, due to the limitation of practically available samples, the difficulty of representative feature extraction from small-sized samples and the loss of subtle diagnostic features in DL iteration results in the accuracy reduction of interclass and intraclass in refined classification, respectively. To address these issues, a water retrieval embedded deep network is proposed in this paper. The relative water content retrieval (RWCR) of the proposed network is embedded as a subnet, which is responsible for extracting subtle diagnostic features of relative water content (RWC) to enhance the representation of features in classification. The experimental results verify the effectiveness of RWCR for improving the interclass and intraclass accuracy in refined classification. Moreover, the superiority of the proposed network is also demonstrated in comparison with state-of-the-art methods.

With the development of remote sensing imaging technology, more and more high-resolution remote sensing images can be easily obtained. As a significant processing step in 3D reconstruction of remote sensing images, stereo matching has always been a hot spot in the field of stereo vision. However, the efficiency and accuracy of stereo matching algorithms remain to be improved, particularly the efficiency. To this end, we propose a remote sensing image stereo matching algorithm based on sparse coding and dictionary learning (SCDL), which can greatly improve the matching efficiency and achieve higher matching accuracy. In SCDL, positive and negative samples are introduced into dictionary learning phase. A constraint term is designed in the objective function to discriminate correct and incorrect matching points. Experimental results show that the proposed algorithm performed well both in efficiency and accuracy.

In hyperspectral image (HSI) classification, the distinction of similar classes has always been a focus of research. In this paper, a new classification module named class confusion merging (CCM) is proposed to improve the classification accuracy, especially for classes with the similar spectral feature. In CCM processing, the merging matrix is firstly constructed based on the confusion matrix to measure the similarity between different classes. Then similar classes are merged as big categories. Finally, for each big category, soft band selection is implemented based on the spectral difference of contained classes for reclassification. To evaluate the performance of CCM, real image experiments are conducted in comparison with no CCM module hyperspectral classification methods. The experiment results demonstrate that the CCM module can improve the classifier performance by providing higher classification accuracy.

Age progression and regression aim to synthesize photorealistic appearance of a given face image with aging and rejuvenation effects, respectively. Existing generative adversarial networks (GANs) based methods suffer from the following three major issues: 1) unstable training introducing strong ghost artifacts in the generated faces, 2) unpaired training leading to unexpected changes in facial attributes such as genders and races, and 3) non-bijective age mappings increasing the uncertainty in the face transformation. To overcome these issues, this paper proposes a novel framework, termed AgeFlow, to integrate the advantages of both flow-based models and GANs. The proposed AgeFlow contains three parts: an encoder that maps a given face to a latent space through an invertible neural network, a novel invertible conditional translation module (ICTM) that translates the source latent vector to target one, and a decoder that reconstructs the generated face from the target latent vector using the same encoder network; all parts are invertible achieving bijective age mappings. The novelties of ICTM are two-fold. First, we propose an attribute-aware knowledge distillation to learn the manipulation direction of age progression while keeping other unrelated attributes unchanged, alleviating unexpected changes in facial attributes. Second, we propose to use GANs in the latent space to ensure the learned latent vector indistinguishable from the real ones, which is much easier than traditional use of GANs in the image domain. Experimental results demonstrate superior performance over existing GANs-based methods on two benchmarked datasets. The source code is available at https://github.com/Hzzone/AgeFlow.

Knowledge distillation is a simple but effective method for model compression, which obtains a better-performing small network (Student) by learning from a well-trained large network (Teacher). However, when the difference in the model sizes of Student and Teacher is large, the gap in capacity leads to poor performance of Student. Existing methods focus on seeking simplified or more effective knowledge from Teacher to narrow the Teacher-Student gap, while we address this problem by Student's self-boosting. Specifically, we propose a novel distillation method named Self-boosting Feature Distillation (SFD), which eases the Teacher-Student gap by feature integration and self-distillation of Student. Three different modules are designed for feature integration to enhance the discriminability of Student's feature, which leads to improving the order of convergence in theory. Moreover, an easy-to-operate self-distillation strategy is put forward to stabilize the training process and promote the performance of Student, without additional forward propagation or memory consumption. Extensive experiments on multiple benchmarks and networks show that our method is significantly superior to existing methods.

Change detection (CD) for multitemporal remote sensing images can find change trends and reveal the development patterns. However, the typical methods based on algebraic operation, image transformation, or segmentation may not yield satisfactory results due to the spectral variability and noise complexity. In a sense, it can be considered that multitemporal images are composed of unchanged and changed regions, as well as noise. In the light of this, a stepwise subtraction method for CD is proposed, based on auto-updating multitemporal matrix factorization (AMMF) and spectral constraint, to remove the unchanged regions and noise from the original images step by step. The unchanged regions are first identified by AMMF, during which the distribution and subspace information of each temporal image are regularized to encode the spatio-temporal correlation. Then, mean shift smoothness is adopted as a spectral constraint so as to remove the noise. In this way, the changed regions have been highlighted so that the change map can be obtained by a postsegmentation method. Experiments have been conducted on three multitemporal data sets, including images from Quick Bird, aerial, and GF-1, indicating that the proposed method is effective and robust, which is superior to some state-of-the-art methods.

The richness of spectral information in multitemporal hyperspectral images (MHSIs) offers the possibility to effectively detect subtle changes and properties of grounds. However, severe spectral variabilities and inadequate spatial co-exploitation capabilities limit the performance of existing methods due to differences in acquisition times and conditions. Therefore, this article proposes a strategy of joint unmixing and multitemporal spatial information coguidance (JUC) to fully exploit the spatio-temporal-spectral features. First, a multitemporal joint unmixing method is used to achieve endmembers’ extraction and abundance estimation. Wherein the method adds spectral perturbed regularization when compared to the traditional unmixing strategy, making it robust to spectral variability. Second, we propose a multitemporal coguidance method that highlights the contrast between changed and unchanged regions and removes the noise by transferring the common structure information between the multitemporal first principal component map and the abundance difference maps. It will obtain an enhanced abundance difference maps and achieve effective combination of multitemporal spatial information. The final change result can be obtained by combining and thresholding these enhanced abundance difference maps. Experiments on some data sets demonstrate that the proposed algorithm is sufficiently valid and robust for multitemporal images, especially for data containing spectral variabilities and obvious structures.

Hyperspectral image (HSI) classification appro- aches achieve significant improvements with the proposal and application of deep learning algorithms. However, due to the end-to-end structure of the deep learning model, the exploration of intrinsic physical-chemical properties in HSI data is insufficient, which restricts the extraction of diagnostic features and impedes the improvement of intraclass classification performance. Moreover, the synergetic spectral-spatial feature extraction in deep learning model is limited owing to the difference between HSI spectral and spatial dimensions, which also hinders the refinement of performance. In order to mitigate these issues, an attention symbiotic neural network (ASNN) based on relative water content (RWC) retrieval (RWCR) is proposed for HSI refined classification in this article. ASNN is a multisupervised deep learning model that is able to extract spectral-spatial and biochemical features from the multilabel input data simultaneously. The augmented multilabel data, consisting of original spectral-spatial labels and RWC labels, are generated in the RWCR inversion model, which contains the proposed spectral index [red edge slope (RES)] calculation and the proposed adaptive grading algorithm. There are two critical parts in ASNN, soft band selection (SBS) module and dimensionality-varied feature extraction (DVFE) module, which are responsible for attention assignment and synergistic spectral-spatial feature extraction, respectively. The experimental results on real HSI data verify the effectiveness of RES, SBS, and DVFE in ablation studies. It is also demonstrated that ASNN has the capacity for improving intraclass and interclass accuracy in refined classification and providing a competitive advantage in comparison with several state-of-the-art methods.

Face aging is to render a given face to predict its future appearance, which plays an important role in the information forensics and security field as the appearance of the face typically varies with age. Although impressive results have been achieved with conditional generative adversarial networks (cGANs), the existing cGANs-based methods typically use a single network to learn various aging effects between any two different age groups. However, they cannot simultaneously meet three essential requirements of face aging-including image quality, aging accuracy, and identity preservation-and usually generate aged faces with strong ghost artifacts when the age gap becomes large. Inspired by the fact that faces gradually age over time, this paper proposes a novel progressive face aging framework based on generative adversarial network (PFA-GAN) to mitigate these issues. Unlike the existing cGANs-based methods, the proposed framework contains several sub-networks to mimic the face aging process from young to old, each of which only learns some specific aging effects between two adjacent age groups. The proposed framework can be trained in an end-to-end manner to eliminate accumulative artifacts and blurriness. Moreover, this paper introduces an age estimation loss to take into account the age distribution for an improved aging accuracy, and proposes to use the Pearson correlation coefficient as an evaluation metric measuring the aging smoothness for face aging methods. Extensively experimental results demonstrate superior performance over existing (c)GANs-based methods, including the state-of-the-art one; e.g., PFA-GAN reduces the aging estimation errors by 0.23 and 0.35 and increases the identity preservation rates by 0.49 and 0.63 on two benchmarked datasets compared to the second best method for the challenging face aging from 30- to 51+. The source code is available at https://github.com/Hzzone/PFA-GAN.

Owing to the limits of incident energy and hardware system, hyperspectral (HS) images always suffer from low spatial resolution, compared with multispectral (MS) or panchromatic (PAN) images. Therefore, image fusion has emerged as a useful technology that is able to combine the characteristics of high spectral and spatial resolutions of HS and PAN/MS images. In this paper, a novel HS and PAN image fusion method based on convolutional neural network (CNN) is proposed. The proposed method incorporates the ideas of both hyper-sharpening and MS pan-sharpening techniques, thereby employing a two-stage cascaded CNN to reconstruct the anticipated high-resolution HS image. Technically, the proposed CNN architecture consists of two sub-networks, the detail injection sub-network and unmixing sub-network. The former aims at producing a latent high-resolution MS image, whereas the latter estimates the desired high-resolution abundance maps by exploring the spatial and spectral information of both HS and MS images. Moreover, two model-training fashions are presented in this paper for the sake of effectively training our network. Experiments on simulated and real remote sensing data demonstrate that the proposed method can improve the spatial resolution and spectral fidelity of HS image, and achieve better performance than some state-of-the-art HS pan-sharpening algorithms.

This article examines how to automate the determination of the coagulant dosage for water treatment plants. Whilst most of the processes for water treatment are automated, determining the coagulant dosage, required for reducing turbidity, depends on well-trained and experienced operators. Based on a time-series data set provided by the Shanghai municipal investment water production company, this article comprehensively surveys existing coagulant prediction methods and utilizes an auto-adjustable and time-consistent model to incorporate the operators' experience. Compared to existing methods, the algorithm introduced in this article produced a better accuracy for predicting the coagulant dosage. Moreover, this article demonstrates that taking seasonal effects into account can approximate operator behavior more accurately. To examine the robustness of the identified models, this article examines the model performance based on water drawn from different locations/sources.

Deep learning-based methods have achieved promising performance in early detection and classification of lung nodules, most of which discard unsure nodules and simply deal with a binary classification-malignant vs benign. Recently, an unsure data model (UDM) was proposed to incorporate those unsure nodules by formulating this problem as an ordinal regression, showing better performance over traditional binary classification. To further explore the ordinal relationship for lung nodule classification, this paper proposes a meta ordinal regression forest (MORF), which improves upon the state-of the-art ordinal regression method, deep ordinal regression forest (DORF), in three major ways. First, MORF can alleviate the biases of the predictions by making full use of deep features while DORF needs to fix the composition of decision trees before training. Second, MORF has a novel grouped feature selection (GFS) module to re-sample the split nodes of decision trees. Last, combined with GFS, MORF is equipped with a meta learning based weighting scheme to map the features selected by GFS to tree-wise weights while DORF assigns equal weights for all trees. Experimental results on LIDC-IDRI dataset demonstrate superior performance over existing methods, including the state of-the-art DORF.

Face aging is of great importance for cross-age recognition and entertainment-related applications. Recently, conditional generative adversarial networks (cGANs) have achieved impressive results for face aging. Existing cGANs-based methods usually require a pixel-wise loss to keep the identity and background consistent. However, minimizing the pixel-wise loss between the input and synthesized images likely resulting in a ghosted or blurry face. To address this deficiency, this paper introduces an Attention Conditional GANs (AcGANs) approach for face aging, which utilizes attention mechanism to only alert the regions relevant to face aging. In doing so, the synthesized face can well preserve the background information and personal identity without using the pixel-wise loss, and the ghost artifacts and blurriness can be significantly reduced. Based on the benchmarked dataset Morph, both qualitative and quantitative experiment results demonstrate superior performance over existing algorithms in terms of image quality, personal identity, and age accuracy. Codes are available on https://github.com/JensonZhu14/AcGAN.

Feature extraction (FE) is very important for hyperspectral image (HSI) dimension deduction before classification. Compared to supervised FE methods, unsupervised FE methods do not need any labeled samples. So unsupervised FE methods are more suitable to extract hyperspectral features for classification. However, most unsupervised FE methods only make use of spectral information of HSI while ignoring spatial information. Hence, the classification accuracy of conventional unsupervised FE methods is unsatisfactory. In this paper, we propose an unsupervised FE method which is based on the framework of Neighborhood Preserving Embedding (NPE) for HSI classification. The proposed method uses both spectral and spatial information of HSI to measure the similarity between samples when constructing the adjacency graph. Experimental results on typical datasets show that the proposed FE method outperforms other conventional unsupervised FE methods on the same classification task significantly. Moreover, the proposed method has competitive performance to some deep learning methods which based on spectral-spatial features.

Hyperspectral image (HSI) target detection has become an increasingly important research topic while still facing various aspects of difficulties, such as the influence of spectral variability, the deficiency of available samples and the limited capability of utilizing spectral-spatial information. In this paper, a novel HSI target detection method combining spectral and spatial information is proposed. Siamese convolutional neural network (S-CNN) is applied as spectral feature extractor according to the spectral similarity, in which pixel pairs generated from the source domain HSI are transferred to target domain so as to solve the problem of insufficient samples. Then spatial post-processing guided by the initial result is used to combining spatial context information to further improve the detection performance. Experimental results show that the proposed method performs well in hyperspectral target detection.

The combination of spectral information and spatial context is known to be a suitable way in improving classification accuracy for hyperspectral image. In this paper, a novel method using PCA and spatial filtering for the classification of hyperspectral image is proposed. Firstly, PCA is used to extract spectral information from the hyperspectral image. Secondly, spatial filters containing a set of 2-D Gabor filters and rolling guidance filters (RGF) are convolved with the principal components to extract the subtle spatial texture and edge features respectively. Thirdly, the obtained features are concatenated together as a feature cube to be classified by SVM. The proposed method is thus named as PCA-GR. Experimental results on two real hyperspectral image data sets demonstrate the significant advantages of the proposed method over the compared ones.

Crowd counting is a highly challenging problem in computer vision and machine learning. Most previous methods have focused on consistent density crowds, i.e., either a sparse or a dense crowd, meaning they performed well in global estimation while neglecting local accuracy. To make crowd counting more useful in the real world, we propose a new perspective, named pan-density crowd counting, which aims to count people in varying density crowds. Specifically, we propose the Pan-Density Network (PaDNet) which is composed of the following critical components. First, the Density-Aware Network (DAN) contains multiple subnetworks pretrained on scenarios with different densities. This module is capable of capturing pandensity information. Second, the Feature Enhancement Layer (FEL) effectively captures the global and local contextual features and generates a weight for each density-specific feature. Third, the Feature Fusion Network (FFN) embeds spatial context and fuses these density-specific features. Further, the metrics Patch MAE (PMAE) and Patch RMSE (PRMSE) are proposed to better evaluate the performance on the global and local estimations. Extensive experiments on four crowd counting benchmark datasets, the ShanghaiTech, the UCF-CC-50, the UCSD, and the UCFQNRF, indicate that PaDNet achieves state-of-the-art recognition performance and high robustness in pan-density crowd counting.

As one of the efficient approaches to deal with big data, divide-and-conquer distributed algorithms, such as the distributed kernel regression, bootstrap, structured perception training algorithms, and so on, are proposed and broadly used in learning systems. Some learning theories have been built to analyze the feasibility, approximation, and convergence bounds of these distributed learning algorithms. However, less work has been studied on the stability of these distributed learning algorithms. In this paper, we discuss the generalization bounds of distributed learning algorithms from the view of algorithmic stability. First, we introduce a definition of uniform distributed stability for distributed algorithms and study the distributed algorithms' generalization risk bounds. Then, we analyze the stability properties and generalization risk bounds of a kind of regularization-based distributed algorithms. Two generalization distributed risks obtained show that the generalization distributed risk bounds for the difference between their generalization distributed and empirical distributed/leave-one-computer-out risks are closely related to the size of samples n and the amount of working computers m as O(m/n1/2) . Furthermore, the results in this paper indicate that, for a good generalization regularized distributed kernel algorithm, the regularization parameter λ should be adjusted with the change of the term m/n1/2. These theoretic discoveries provide the useful guidance when deploying the distributed algorithms on practical big data platforms. We explore our theoretic analyses through two simulation experiments. Finally, we discuss some problems about the sufficient amount of working computers, nonequivalence, and generalization for distributed learning. We show that the rules for the computation on one single computer may not always hold for distributed learning.

As a unique biometric feature that can be recognized at a distance, gait has broad applications in crime prevention, forensic identification and social security. To portray a gait, existing gait recognition methods utilize either a gait template, where temporal information is hard to preserve, or a gait sequence, which must keep unnecessary sequential constraints and thus loses the flexibility of gait recognition. In this paper we present a novel perspective, where a gait is regarded as a set consisting of independent frames. We propose a new network named GaitSet to learn identity information from the set. Based on the set perspective, our method is immune to permutation of frames, and can naturally integrate frames from different videos which have been filmed under different scenarios, such as diverse viewing angles, different clothes/carrying conditions. Experiments show that under normal walking conditions, our single-model method achieves an average rank-1 accuracy of 95.0% on the CASIA-B gait dataset and an 87.1% accuracy on the OU-MVLP gait dataset. These results represent new state-of-the-art recognition accuracy. On various complex scenarios, our model exhibits a significant level of robustness. It achieves accuracies of 87.2% and 70.4% on CASIA-B under bag-carrying and coat-wearing walking conditions, respectively. These outperform the existing best methods by a large margin. The method presented can also achieve a satisfactory accuracy with a small number of frames in a test sample, e.g., 82.5% on CASIA-B with only 7 frames. The source code has been released at https://github.com/AbnerHqC/GaitSet.

With to the development of sensors, the fusion of features from multisource data becomes an interesting but challenging problem. In this paper, the fusion of hyperspectral imagery (HSI) and light detection and ranging (LiDAR) data is investigated with a novel and simplified deep learning architecture, named the dual-branch convolutional neural network (DB-CNN). More specifically, a 3D CNN framework as one of the two branches is used to extract spectral-spatial features simultaneously from HSI, which can keep three-dimensional structural characteristics of HSI. Another one is 2D CNN with cascade blocks, which is developed to extract elevation feature from LiDAR data, and it can exploit the multiscale features. Finally, the features of two branches will be flattened and stacked, and then sent to the fully connected layers. The experiments show that the proposed DB-CNN method can effectively fuse the HSI and LiDAR data, and yield higher classification performance than some existing methods.

The combination of the spectral and spatial features is received wide attention in hyperspectral image (HSI) classification. And the multiscale-strategy is an effective way in improving the classification accuracy for HSI due to the various sizes of land covers, which can capture more intrinsic information. For this reason, a multiscale spectral-spatial unified network (MSSN) with two-branch architecture is proposed for hyperspectral image classification. Different from other networks mainly focusing on the multiscale spatial features, the MSSN can jointly extract the multiscale spectral-spatial features, which is based on the reason that features of different layers in CNN correspond to different scales. In the implementation of the MSSN, the 1D CNN and 2D CNN are used to extract the spectral and spatial features respectively. Then the features of the corresponding layers in the two branches will be integrated to the fully-connected layers and finally sent to the classification layers. Experiments on two benchmark HSIs demonstrate that the proposed MSSN can yield a competitive performance compared with other existing methods.

Target detection has become an increasingly important research topic in hyperspectral image (HSI) processing with most existing detecting algorithms focus on the information of required targets while fail to utilize the background of input data. In this paper, we propose a novel target detection method for HSI, which deals with the detection problem from a new contextual perspective. Based on an initial detection result, the background of HSI is extracted by introducing the joint convolutional analysis and synthesis (JCAS) sparse representation method. Then, the initial result is corrected through a background guided approach, false alarm is greatly reduced. Experimental results show that the proposed method performs well in hyperspectral target detection.

This paper introduces a framework to deal with the distribution of descriptive features, which preserves the advantages of the vectorial representation and computational efficiency of histogram-based techniques, and inherits the rigorous theoretical guarantee and competitive performance of metric-based ones. The methods developed under this framework describe the underlying distribution of a set of features as a vectorial feature by utilizing random features. Moreover, the proposed methods asymptotically converge to metric-based methods in terms of the similarity and distance and, depending on a specific kernel function, reduce to histogram-based methods. The experimental results show the benefits of a comparable performance on categorization tasks compared to conventional metric-based methods at a significantly reduced computational cost.

Gait recognition is of great importance in the fields of surveillance and forensics to identify human beings since gait is the unique biometric feature that can be perceived efficiently at a distance. However, the accuracy of gait recognition to some extent suffers from both the variation of view angles and the deficient gait templates. On one hand, the existing cross-view methods focus on transforming gait templates among different views, which may accumulate the transformation error in a large variation of view angles. On the other hand, a commonly used gait energy image template loses temporal information of a gait sequence. To address these problems, this paper proposes multi-task generative adversarial networks (MGANs) for learning view-specific feature representations. In order to preserve more temporal information, we also propose a new multi-channel gait template, called period energy image (PEI). Based on the assumption of view angle manifold, the MGANs can leverage adversarial training to extract more discriminative features from gait sequences. Experiments on OU-ISIR, CASIA-B, and USF benchmark data sets indicate that compared with several recently published approaches, PEI + MGANs achieves competitive performance and is more interpretable to cross-view gait recognition.

Automated crowd counting is valuable for intelligent transportation systems, as it can help to improve the emergency planning and prevent congestion in transit hubs such as train stations and airports. Semi-supervised crowd counting aims to estimate the number of pedestrians in an ongoing scene using a combination of a small number of labeled frames and a large number of unlabeled ones. However, existing methods do not incorporate ways to effectively select informative frames as labeled training samples, resulting in low accuracy on unseen crowd scenes. We propose a submodular method to select the most informative frames from the image sequences of crowds. Specifically, the method selects the most representative images to guarantee the information coverage, by maximizing the similarities between the group of selected images and the image sequence. In addition, these frames are chosen to avoid redundancies and preserve diversity. Finally, our semi-supervised method incorporates graph Laplacian regularization and spatiotemporal constraints. Extensive experiments on three benchmark data sets demonstrate that our proposed approach achieves higher accuracy compared with the state-of-the-art regression methods and competitive performance with deep convolutional models, especially when the number of labeled data is exceptionally small.

Currently, big data is a new and hot object of research. In particular, the development of the Internet of things (IoT) results in a sharp increase in data. Enormous amounts of networking sensors are constantly collecting and transmitting data for storage and processing in the cloud including remote sensing data, environmental data, geographical data, etc. Road information extraction from remote sensing data is mainly researched in this paper. Roads are typical man-made objects. Extracting roads from remote sensing imagery has great significance in various applications such as GIS data updating, urban planning, navigation, and military. In this paper a multistage and multifeature method to extract roads and detect road intersections from high-resolution remotely sensed imagery based on tensor voting is presented. Firstly, the input remote sensing image is segmented into two groups including road candidate regions and nonroad regions using template matching; then we can obtain preliminary road map. Secondly, nonroad regions are removed by geometric characteristics of road (large area and long strip). Thirdly, tensor voting is used to overcome the broken roads and discontinuities caused by the different disturbing factors and then delete the nonroad areas that are mixed into the road areas due to mis-segmentation, improving the completeness of extracted roads. And then, all the road intersections are extracted by using tensor voting. The experiments are conducted on different remote sensing images to test the effectiveness of our method. The experimental results show that our method can get more complete and accurate extracted results than the state-of-the-art methods.

The spectrum of water is directly related to the composition of the water, so water quality can be estimated by spectral reflectance. In this paper, the inversion model of water quality parameters for remote sensing images is established, based on which the status of water pollution is analyzed. Firstly, the spectral information of water in studied area is extracted from remote sensing images. Then, the relationship between spectral reflectance and water quality parameters is modeled. The model is evaluated by the fitting SSE (Sum of the Squared Errors), R-square, RMSE (the Root Mean Squared Errors) and Adjusted R-square. When R-square and Adjusted R-square are close to 1 and the SSE and RMSE are close to 0, inversion model is considered to be built up successfully. The experiments are conducted on 14 Landsat 8 OLI images in recent four years and the results show that the model method can realize the analysis and monitoring of the water quality. The R-square of the permanganate and dissolved oxygen were 0.96 and 0.80 respectively, which satisfied the requirements of the application.

Hyperspectral images (HSI) contain a wealth of spectral and spatial information, spectral-spatial combination is an effective way in improving the classification accuracy for HSI. To characterize the variability of spatial features at different scales better, a multiscale spectral-spatial classification method with adaptive filtering (MSAF) is proposed. The proposed method consists of the following four steps. Firstly, the spectral features are extracted by a feature selection algorithm. Secondly, the adaptive edge-preserving filtering with different scales are conducted on each feature, and then several stacks of data blocks containing spatial information can be obtained. Thirdly, the combinations of the spectral and spatial data blocks are classified using support vector machine (SVM). Finally, a post-processing is conducted to improve the classification results further. The experiments on the hyperspectral data demonstrate that the proposed method can improve the classification accuracy significantly compared to the SVM classifier, especially need less parameters than the spectral-spatial EPF method.

Change detection techniques for remote sensing images are increasingly applied to many fields, such as disaster monitoring, vegetation coverage analysis and so on. How to improve the accuracy of detection has been a critical topic that confuse the researchers for a long time. In this paper, a method combining multiscale segmentation and fusion for high-resolution images is presented. The strategy of multiscale segmentation is to segment the same image several times under different scales, and then extract the features of objects. After, the features are used as inputs of change detection. The final results are achieved by decision-level fusion. The experiments show that, comparing with other typical methods, the method proposed in this paper has a superior performance in change detection for high-resolution images.

Multi-resolution analysis (MRA) is a useful technique for multispectral (MS) image pan-sharpening. To overcome the limitation of the conventional wavelet-based method, multi-directional MRA approaches have been proposed to provide an efficient directional information representation. In this paper, we present a new hybrid method for MS image pan-sharpening that uses non-subsampled contourlet transform (NSCT) as the MRA tool, and followed by a high-pass detail-injection model. Particularly, to further enhance the spatial resolution of the MS image, and in the meantime, produce a high spectral quality, we take account of the dissimilarity between the spatial details of MS image and decomposed panchromatic image of NSCT, and introduce a fast and effective means to deal with the problem. Experiments on four remote sensing data sets show the superiority of the presented approach over other typical pan-sharpening methods.

This paper proposes a target detector based on kernel sparse and spatial constraint for hyperspectral imagery (HSI). Due to the nonlinear and structural features of HSI data, sparse representation and spatial constraint are taken into consideration. Firstly, we construct a dictionary to represent the target pixels within a small neighborhood by a linear combination of samples. Then, these targets pixels are projected into the high-dimensional feature space through kernel function and orthogonal matching pursuit (OMP) are kernelized to obtain recovered sparse coefficient vector. By comparing the residuals of background and target to determine the type of pixel, the preliminary detection result can be achieved. Lastly, a spatial over-complete basis matrix is used to revise the initial detection result. The experimental results show that the proposed detector has better detection performance than several typical detectors.

Region-of-interest (ROI) detection techniques are of great importance in the analysis of remote sensing images, especially in target detection, since the size of the image to be dealt with grows substantially with the improvement of spatial resolution. Most of current studies are not aiming at the specific type of object area detection, and the processed images are rather small compared to the size of the raw data acquired by high-resolution satellite. In this letter, a hierarchical task-driven ROI detection method, based on saliency and density, is proposed to address the detection of the potential object areas in large-scale remote sensing images. The proposed saliency and density-based detection method (SDBD) integrates bottom-up and top-down strategies, where the saliency-based multilevel histogram contrast is presented in the bottom-up phase to obtain the preliminary regions, while the centroid density distribution index (CDDI) is defined in the top-down scheme to refine the previous results. Specifically, superpixel segmentation is introduced in this letter to narrow down the ROI candidates. SDBD is capable of extracting ROI of different objects by adjusting the threshold of CDDI. The experiments are conducted on two data sets to extract ROIs of storage tanks and residence. Experimental results demonstrate that the proposed method is effective in identifying ROI in large-scale data.

In the past decades, with the development of polarized hyperspectral imaging system, target detection and recognition for polarized hyperspectral images (PHSIs) have attracted more and more attention. Most conventional target detection algorithms of PHSI are based on the Stokes vector, which mainly take advantage of the spectral information, and ignore the continuous variability of polarized dimension, being similar to spectrum. In fact, the PHSI include multidimensional features of polarization, spectrum, space, and radiation, and these provide more discriminable information about target and background than traditional spectrum or intensity ones. Hence, tensor, which can keep the complete information of PHSI, is introduced to represent such high-dimensional data. In this letter, the tensor canonical polyadic (CP) decomposition is adopted to extract the spectral and polarized features. There are two ways to realize the target detection through CP decomposition. One is to construct a fourth-order tensor matched filtering (FTMF), and FTMF is applied to the original data directly without extracting the Stokes vector. The other is to reconstruct the PHSI into a new 3-D data, then matched filter algorithm of hyperspectral image is applied to this data to detect the targets, for short CPMF. The experimental results show that the proposed methods achieve the joint utilization of spectrum and polarization and are more suitable and effective for the target detection of the PHSI.

Building shape, as a key structured element, plays a significant role in various urban remote sensing applications. However, because of high complexity and intraclass variations between building structures, the capability of building shape description and recognition becomes limited or even impoverished. In this letter, a novel deep embedding network is proposed for building shape recognition, which combines the strength of the unsupervised feature learning of convolutional neural networks (CNNs) and a novel triplet loss. Specifically, we take advantage of the strong discriminative power of CNNs to learn an efficient building shape representation for shape recognition. With this deep embedding network, the high-dimensional image space can be mapped into a low-dimensional feature space, and the deep features can effectively reduce the intraclass variations while increasing the interclass variation between different building shape images. Afterward, the derived deep features are exploited for the process of building shape recognition. This method consists of two stages. In the first stage, for standard building shape image queries stored in the shape primitives library and the building shape data set, two sets of deep features are extracted with the deep embedding network. In the second stage, we formulate the shape recognition task into a feature matching problem and the final building shape recognition results can be achieved by set-to-set feature matching method. Experiments on the VHR-10 and UCML data sets demonstrate the effectiveness and precision of the proposed method.

Polarized hyperspectral images (PHSIs) possess multidimensional information, including space, spectrum, and polarization, and in the past decades, target detection and recognition for PHSIs have attracted more and more attention. However, most target detection methods of PHSIs are based on the Stokes vector, and derived from the target detection of HSIs, which mainly take advantage of the spectral information and ignore the continuous variability of polarized dimension, being similar to spectrum. Hence, in order to take full advantage of the multidimensional information of PHSIs, we combine tensor decomposition and joint sparse representation, and propose a joint sparse tensor representation (JSTR) method for the target detection of PHSI, which can remove the redundancy and noise, and also realize the joint utilization of spectral, polarized, and spatial information. And the experiments on the PHSI data have validated the practicability and effectiveness of JSTR for the target detection of PHSIs.

Unlike conventional facial expressions, microexpressions are instantaneous and involuntary reflections of human emotion. Because microexpressions are fleeting, lasting only a few frames within a video sequence, they are difficult to perceive and interpret correctly, and they are highly challenging to identify and categorize automatically. Existing recognition methods are often ineffective at handling subtle face displacements, which can be prevalent in typical microexpression applications due to the constant movements of the individuals being observed. To address this problem, a novel method called the Facial Dynamics Map is proposed to characterize the movements of a microexpression in different granularity. Specifically, an algorithm based on optical flow estimation is used to perform pixel-level alignment for microexpression sequences. Each expression sequence is then divided into spatiotemporal cuboids in the chosen granularity. We also present an iterative optimal strategy to calculate the principal optical flow direction of each cuboid for better representation of the local facial dynamics. With these principal directions, the resulting Facial Dynamics Map can characterize a microexpression sequence. Finally, a classifier is developed to identify the presence of microexpressions and to categorize different types. Experimental results on four benchmark datasets demonstrate higher recognition performance and improved interpretability.

Local image features can be assumed to be drawn from an unknown distribution. For image classification, such features are compared through the histogram-based model or the metric-based model. By quantizing these local features into a set of histograms, the histogram-based model is convenient and has vectorial representation of image but information could be lost in vector quantization. Unlike the histogram-based model, the metric-based model estimates the metrics over the underlying distribution of local features immediately, achieving better predictive performance. However, the model requires higher computational cost and loses the benefit of vectorial representation of image.

The polarized hyperspectral images (PHSI) include the multidimensional information of polarization, spectral, spatial and radiant features, which provide more information about objects and background than traditional spectrum or intensity ones. However, the conventional object detection algorithms of PHSI are based on the Stokes vector, which mainly take advantage of the spectral information, and often ignore the continuous variation characteristics of polarized dimension, being similar to the spectrum. What is more, when the data is large, it will increase the calculative difficulty and error. Hence, a fourth-order tensor matched filtering (FTMF) is proposed in this paper, which is applied to the original data directly without extracting the Stokes vector, and achieves the combined utilization of polarization and spectrum. The experimental results also show that the proposed method in this paper is more suitable for the object detection of the PHSI.

Road is the skeleton of the city, which is usually elongated. Extracting road information is of great significance in urban planning. Tensor voting can detect the geometric characteristics of the typical objects in the image. In this paper in order to improve the accuracy of road extraction in high-resolution remote sensing image, road is extracted preliminary using morphological edge segmentation and shape index feature extraction, then tensor voting is introduced to purify the road information and detect road intersections. Voting in stick field can detect and amplify the stick saliency of road pixels, the gaps can also be connected after tensor voting. On the basis of pure road extraction result above, the voting in ball field is used to detect the road intersections. The experiments on remote sensing images show that the proposed method in this paper is superior to traditional morphological post-processing method, it can precisely extract the roads, and detect all the road intersections at the same time.

Image aesthetics assessment is essential to multimedia applications such as image retrieval, and personalized image search and recommendation. Primarily relying on visual information and manually-supplied ratings, previous studies in this area have not adequately utilized higher-level semantic information. We incorporate additional textual phrases from user comments to jointly represent image aesthetics utilizing multimodal Deep Boltzmann Machine. Given an image, without requiring any associated user comments, the proposed algorithm automatically infers the joint representation and predicts the aesthetics category of the image. We construct the AVA-Comments dataset to systematically evaluate the performance of the proposed algorithm. Experimental results indicate that the proposed joint representation improves the performance of aesthetics assessment on the benchmarking AVA dataset, comparing with only visual features.

Application Driven Network(ADN) is a new paradigm that provides on-demand differentiated services for applications. A physical network in ADN is sliced into various logically isolated sub-networks. Each network slice can have its own network architecture and protocol to serve one application exclusively. ADN enhances the user experience while keeping the resource efficiency by further imposing multiplexing among these logically isolated sub-networks.

Pan-sharpening is designed to estimate multi-/hyperspectral (MS/HS) images that would have been observed with a sensor at higher resolution. It is a very important issue for many remote sensing and mapping applications. This letter proposes an improved pan-sharpening algorithm based on the ARSIS concept under the assumption that missing information of a low-resolution MS/HS image is linked to the high frequencies of MS/HS and panchromatic (PAN) images. The main object of this letter is to exhibit a multilevel interband structure model that better considers the inherent relationship between the hierarchical structures of MS/HS and PAN images. Several groups of data sets are used to demonstrate the performance of the proposed method. The results show that the proposed method outperforms the existing ARSIS-based and some other fusion techniques and can be extended to HS image sharpening as well.

Space partitioning trees, which sequentially divide and subdivide a space into disjoint subsets using splitting hyperplanes, play a key role in accelerating the query of samples in the cybernetics and computer vision domains. Associated methods, however, suffer from the curse of dimensionality or stringent assumptions on the data distribution. This paper presents a new concept, termed kernel dimension reduction-tree (KDR-tree), that relies on linear projections computed based on an unsupervised kernel dimension reduction approach. The proposed concept does not rely on any assumption on the data distribution and can capture higher-order statistical information encapsulated within the data. This paper then develops two variants of the KDR-tree concept: 1) to handle residual data [i.e., the residual-based KDR-tree (rKDR-tree) algorithm] and 2) to cope with larger datasets, [i.e., the sampling-based KDR-tree (sKDR-tree) algorithm]. By directly comparing the KDR-tree concept to competitive techniques, involving several benchmark datasets, this paper shows that the sKDR-tree yields a better performance for non-Gaussian distributed datasets. Based on the analysis of three datasets, this paper highlights, experimentally, that the rKDR-tree has the potential to discover the intrinsic dimension. This paper also provides a theoretical analysis about the KDR-tree concept to outline why it outperforms existing techniques if the data distribution is non-Gaussian.

In this paper, we propose a self-learning approach for remote sensing image classification. The main work of this paper aims at providing a new framework of semisupervised learning technique for multiple-source synergetic classification, thereby improving the classification accuracy under the condition of small samples. Considering the high spectral resolution of a hyperspectral (HS) image and the high spatial resolution of a panchromatic (PAN) image, the proposed approach combines image segmentation with an active learning algorithm and adopts a standard active learning method for a self-learning strategy, in which the learning algorithm automatically selects informative unlabeled samples by itself according to their collaborative spatial-spectral features and the predicted information of a spectral-based classifier. This way, no extra cost of human expertise is required for labeling the selected pixels when compared with conventional active learning methods. Experiments on three data sets, including HS and PAN images, indicate that our proposed approach has a great enhancement on overall classification accuracy compared with classical supervised algorithms and turns out to be a promising strategy in synergetic classification of HS and PAN images.

Most of the compression methods for remote sensing images are often designed under the guidance of mean square error. However, for the vision-related applications, high peak-signal-to-noise ratio (PSNR) does not mean good visual quality. On the other hand, existing compression methods that considering the human visual system (HVS) are usually designed for natural images, without taking the unique characteristics of remote sensing images into account. Focusing on this problem, we present a novel HVS-based adaptive scanning (HAS) scheme for the compression of remote sensing images. First, after the wavelet transform, a retina-based visual sensitivity model is established, and then, the visual weighting mask is generated. Second, for the weighted transformed image, an adaptive scanning method is proposed, which provides different scanning orders among subbands and within a subband, respectively. The former focuses on organizing the codestream according to the importance of weighted subbands, and the latter aims at preserving the direction information of an image as much as possible. Finally, the binary tree codec is utilized. Experimental results show that, as compared with other scan-based compression methods, the proposed HAS-based compression method can provide better visual quality, which makes it more desirable in vision-related applications for remote sensing images.

Pansharpening algorithms have been widely discussed since the poor spatial resolution of the hyperspectral images limits their use in identifying textures or some other detailed information of different ground types in many applications such as sophisticated interpretation. To overcome the defect is of the top priority in this topic, for the conventional pansharpening methodologies usually suffer both spectral and spatial distortion severely after the processing. In this paper, a hyperspectral images resolution enhancement method aiming at distortion optimization is presented. Quality-based Segmented Principal Component Transform (QSPCT) is proposed to minimize the distortion, after which the injection model is employed. The potential of the proposed method is further demonstrated through comparative analysis against a number of standard pan-sharpening algorithms on EO-1 data sets.

In change detection analysis, it is important to distinguish the real change targets and pseudo change targets accurately. Supervised change detection has been regarded as the best way to reduce the effects of pseudo change information. This is because human visual system has the ability to find the real changes. By imitating human visual characteristic, visual attention mechanism can bring the improvement of accuracy and speed of unsupervised change detection. In this paper, a change detection approach based on visual attention mechanism is proposed to reduce the influence of pseudo change information. Experiments show that the proposed method significantly reduces the false alarm rate and missed alarm rate and also shows insensitive to noise.

High resolution remote sensing imagery can provide more useful information, such as spectral, shape and texture information. However, traditional pixel-based image classification approaches may suffer the increase of within-class spectral variation with improved spatial resolution. This paper presents a novel method which combines the optimal segmentation scale with Bag-of-Visual Words (BOV) representation for object-oriented classification. More precisely, an improved estimation of scale parameter (ESP) tool is adopted to determine the optimal parameters in multi-scale image segmentation. BOV is introduced to construct the midlevel representations instead of low-level features for object description. Then Support vector machine (SVM) is used for classification. And the experiments are conducted on high spatial resolution images to validate the proposed algorithm.

Determining the number of pedestrians from video surveillance has become a very important task in recent years. Available techniques in support of this task include regression-based approaches, which have shown a satisfactory performance in estimating this number from a crowd of pedestrians. However, most of these approaches suffer from treating a surveillance video as a sequence of separate frames, resulting in some temporal information being lost. To address this issue, this paper proposes a semisupervised methodology to extract temporal consistency in a continuous sequence of unlabeled frames. In addition to the temporal consistency, this paper also employs spatial consistency in the sum of pedestrians in subgroups, or subblobs, to determine the total number of pedestrians, or the ground truth. This is effectively achieved by incorporating regularization terms in the objective function to account for temporal and spatial consistencies. The experimental results show that the proposed technique, based on temporal and spatial consistencies, is more robust and can be trained with relatively few labeled frames (e.g., ten frames).

Generally, dimensionality reduction methods, such as Principle Component Analysis (PCA) and Negative Matrix Factorization (NMF), are always applied as the preprocessing part in hyperspectral image classification so as to classify the constituent elements of every pixel in the scene efficiently. The results, however, would suffer the loss of detailed information inevitably. In this paper, deep learning frameworks, restricted Boltzmann machine (RBM) model and its deep structure deep belief networks (DBN), are introduced in hyperspectral image processing as the feature extraction and classification approach. The experiments are conducted on an airborne hyperspectral image. Further in the experiments, spatial-spectral classification is also practiced. Meanwhile, SVM with and without some classical feature extraction methods adopting before classification are employed as comparison. The results show the superior performance of the proposed approach.

The image registration is an important technology in remote sensing image processing and applications. However, in recent years it is yet a challenging task especially to highresolution remote sensing images which contain more similar objects such as smooth areas, repetitive structures. In order to solve this problem, a novel registration method is proposed using multi-feature and multi-stage strategy in this paper. Firstly local features, the local density distribution and spatial relation, are employed to deal with the ambiguity in the similar objects and avoid mismatches, which are described by the DAISY and the RPCP (relative polar coordinates of point) respectively. Secondly the multi-stage strategy is used to avoid exhaustive search process and also improves the matching performance. Finally experimental results have shown the effectiveness of the proposed method.

Fusion of remote sensing images has received considerable attention during the last two decades, due to the contradiction between the spatial and spectral resolutions of remote sensing images. In this paper, a new fusion method based on Modified Gram-Schmidt (MGS) and stationary wavelet transform (SWT) has been presented to merge multispectral and panchromatic images. In order to show the effectiveness of the proposed method, experiments have been conducted on two datasets of QuickBird and WorldView-II, compared with several typical methods. Experimental results show that the proposed method performs better than the other methods both in the enhancement of spatial details and the preservation of spectral information.

This paper investigates the segmentation problem for very high resolution airborne synthetic aperture radar (SAR) images. In addition to the instinct speckles, these images show two extra characteristics: scene complexity and intensity inhomogeneity, which make segmentation more difficult. An unsupervised solution is proposed based on level set method. First, a new level set evolution method is put forward, it can get global minimum without initial contour, thus can handle complex images automatically. And the new evolution function also introduces the localizing idea from region-scalable-fitting (RSF) model to deal with the intensity inhomogeneity. Then the two segmentation results for background and targets are fused. The experimental results on real images demonstrate the effectiveness of the proposed method.

Canonical correlation analysis (CCA) is a well established technique for identifying linear relationships among two variable sets. Kernel CCA (KCCA) is the most notable nonlinear extension but it lacks interpretability and robustness against irrelevant features. The aim of this article is to introduce two nonlinear CCA extensions that rely on the recently proposed Hilbert-Schmidt independence criterion and the centered kernel target alignment. These extensions determine linear projections that provide maximally dependent projected data pairs. The paper demonstrates that the use of linear projections allows removing irrelevant features, whilst extracting combinations of strongly associated features. This is exemplified through a simulation and the analysis of recorded data that are available in the literature.

Classification is one of the most important applications in the field of remote sensing. How to improve the accuracy of classification is the critical topic that has long obsessed the researchers. In this paper, a fusion method based on a synergic use of hyperspectral data and Polarimetric SAR (PolSAR) data is presented. This method consists of two main parts, feature-level fusion and decision-level fusion. In feature-level, parallel feature combination strategy is introduced to classification of remote sensing images. Results of feature-level fusion are used as inputs of decision-level fusion based on fuzzy set theory. The final results are compared with processing of single level and single data set, and it shows that the synergic method proposed in this paper has a superior performance in joint classification of hyperspectral and polarimetric SAR data.

Most lossy compression methods for hyperspectral image (HSI) usually compress the data in this way that focus on preserving low frequency information. However, for some applications such as edge detection, the information which belongs to high frequency is more useful. Thus, a new pixel of interest (POI)-preserving-based HSI compression scheme is proposed. The concept of POI is proposed because some pixels are significant in preserving the main high frequency. Firstly, the POI extraction is performed by unmixing and the mixed pixels are viewed as POI, then the mask of the pixel of interest (MPI) is generated. Secondly, the compression scheme based on the POI preserving is conducted. The spatial and spectral redundancies are reduced, respectively, then a POI-lifting strategy is adopted for preserving the main high frequency information. Finally, bit allocation and encoding to the transformed HSI is performed by SPIHT_TCIRA algorithm, followed by the contextual adaptive arithmetic coder (CAAC). Experiments are implemented using the HSI acquired by the ROSIS Sensor. Results indicate that compared with the common compression method, the POI-preserving-based compression method can keep the key high-frequency information more effectively.

The structure representation of data distribution plays an important role in understanding the underlying mechanism of generating data. In this paper, we propose the prime discriminant simplicial complex (PDSC) by utilizing persistent homology to capture such structures. Assuming that each class is represented with a prime simplicial complex, we classify unlabeled samples based on the nearest projection distances from the samples to the simplicial complexes. We also extend the extrapolation ability of these complexes with a projection constraint term. Experiments in simulated and practical datasets indicate that, compared with several published algorithms, the proposed PDSC approaches achieve promising performance without losing structure representation.

In this work, we present an interactive system for visual analysis of urban traffic congestion based on GPS trajectories. For these trajectories we develop strategies to extract and derive traffic jam information. After cleaning the trajectories, they are matched to a road network. Subsequently, traffic speed on each road segment is computed and traffic jam events are automatically detected. Spatially and temporally related events are concatenated in, so-called, traffic jam propagation graphs. These graphs form a high-level description of a traffic jam and its propagation in time and space. Our system provides multiple views for visually exploring and analyzing the traffic condition of a large city as a whole, on the level of propagation graphs, and on road segment level. Case studies with 24 days of taxi GPS trajectories collected in Beijing demonstrate the effectiveness of our system.

To broaden the future application of CHANG'E-1 imagery, including hyperspectral imagery (low spatial resolution of 200m) and CCD imagery (relatively high spatial resolution of 120m), an ARSIS-based method for spatial-spectral fusion is proposed in this paper, which aims at combine high spatial and high spectral resolution. Firstly, ARSIS concept is employed, in which Àtrous wavelet is used to describe images at different resolutions for multiresolution analysis. Secondly, Pulse Coupled Neural Network (PCNN) is employed to search and model a relationship between the high frequencies of the images to be fused for missing information. The ARSIS method preserves the spectral content of the original image for its very definition, and Àtrous wavelet and PCNN prove to be effective means to implement it on CHANG'E-1 Imagery. The experimental results demonstrate that the visual improvement and spectral fidelity of the proposed method outperform many conventional methods of image fusion.

Advances in sensor and computer technology are revolutionizing the way that remote sensing data with hundreds or even thousands of channels for the same area on the surface of the earth is collected, managed and analyzed. In this paper, the classical Spectral Angle Mapper (SAM) algorithm, which is fit for parallel and distributed computing, is implemented by using Graphic Processing Units (GPU) and distributed cluster respectively to accelerate the computations. A quantitative performance comparison between Compute Unified Device Architecture (CUDA) and Matlab platform is given by analyzing result of different parallel architectures' implementation of the same SAM algorithm.

As inherent optical properties (IOPs) are directly related to the constituents in the water, the condition of water quality can be reflected by fundamental IOPs absorption and scattering coefficients. And these values can be derived by analytically inverting the remote sensing spectral reflectance. In this paper, the relations between the remote sensing reflectance and water quality information are established, and the model parameters of water quality are obtained by stochastic optimization. Based on Threshold Accepting algorithm, a method with the improved searching strategy and new optimization criteria is proposed to find optimal parameters for the inversion model. The experiments conducted on the simulated data and real data, which indicate that through the division of optimization parameters and the use of different search methods, the accuracy of inversion and operational efficiency can be improved.

To improve the classification accuracy of image in which many classes have the similar spectrum, this paper presents a new hierarchical classification scheme for hyperspectral images (HSI). The Spectral Angle Mapping (SAM) is firstly used to combine the similar classes into large classes. Next the hierarchical classifier classifies the image with large classes and then divides every large class into normal classes further. For every large class, the most suitable feature extraction method and classifier are chosen empirically. Meanwhile, a new band selection is proposed to help every large class find the bands which can better reflect the differences of classes according to the characteristic of spectrum. Experiments are conducted on a 103-band ROSIS image of University of Pavia. The experimental results show that the hierarchical classifier is better than the single classifier used only once. Especially when the spectra of the given classes are so similar that the traditional classifiers couldn't divide them thoroughly, the proposed classifier can make it. Moreover, the hierarchical classifier can do more efficiently because it excludes some redundant bands and concentrates on the bands with slight differences.

A new scheme integrating segmentation into classification to analyze hyperspectral images is presented in this paper, particularly for images with a very few number of labels and largely adjacent spatial structures. Using pixel-wise semi-supervised support vector machine, the image is classified, and segmented by modified C-V level set in this method. Afterwards, classification and segmentation images are combined with neighborhood voting. Experiments are conducted on a 200-band AVIRIS image of the Northwestern Indiana's Indian Pine site. The integration of the spatial information from the level set segmentation provides classification images with more homogeneous regions and improves the classification accuracy, comparing to the general pixel-wise supervised and semi-supervised classification.

Gait Energy Image (GEI) is an efficient template for human identification by gait. However, such a template loses temporal information in a gait sequence, which is critical to the performance of gait recognition. To address this issue, we develop a novel temporal template, named Chrono-Gait Image (CGI), in this paper. The proposed CGI template first extracts the contour in each gait frame, followed by encoding each of the gait contour images in the same gait sequence with a multichannel mapping function and compositing them to a single CGI. To make the templates robust to a complex surrounding environment, we also propose CGI-based real and synthetic temporal information preserving templates by using different gait periods and contour distortion techniques. Extensive experiments on three benchmark gait databases indicate that, compared with the recently published gait recognition approaches, our CGI-based temporal information preserving approach achieves competitive performance in gait recognition with robustness and efficiency.

Polarimetric hyperspectral images can provide spectral, spatial, and polarimetric information of a scene, which are unique and comprehensive for remote sensing applications such as growth monitoring of crops, analysis of water quality, and geology mapping, etc. The researches on polarimetric hyperspectral imaging mechanism and on image characteristics are of great importance for further information extraction and utilization of the images. The purposes of this paper are to analyze the mechanism of polarimetric hyperspectral imaging and to model such a process. The outcome of the paper will help designers and users of a polarimetric hyperspectral imaging system to further understand the system and take full advantages of it. In this paper, a polarimetric hyperspectral imaging model is proposed, in which the influence of skylight on polarization is considered, and subpixel model, polarized reflectance models, and the classical fast canopy reflectance model are combined to model the vegetation canopy. Then, a simulated scene that includes a woodland area with low shrubbery and a road is obtained by using the imaging model. Experiments analyze and discuss the simulation condition and parameters of the imaging models, the uniqueness, and usefulness of the integration of polarimetric and spectral information.

In this paper, a new method for improving the classification performance of hyperspectral images with the aid of derivative information is investigated. First, spectral features are filtered and derivatives of different orders at different sampling intervals are computed. Then, the suitable spectral magnitude features and different derivative features are chosen by using segmented principle component analysis feature extraction method with optimal parameters, and are stacked to constitute a new feature cube. Finally, the efficacy of the spectral derivatives in improving the classification performance of the hyperspectral data is testified using support vector machine for AVIRIS hyperspectral data. The experimental results show that the proposed method can improve the classification accuracy compared to the traditional classification techniques with spectral magnitude features even on very small training samples.

In recent years, remote sensing images with high resolution are increasingly applied in change detection and disaster assessment. Compared with the traditional pixel-based methods, object-oriented image processing techniques have attracted more attention for high resolution images. In this paper, we aim to research the object-oriented change detection for urban area. A new multi-scale segmentation algorithm is proposed so as to obtain accurate image objects, and a pre-processing step is adopted to improve the computation efficiency. In order to testify the performance of proposed method, experiments are conducted on QuickBird images. The experimental results show that accurate image objects and changed area can be acquired in appropriate scales.

China's first lunar orbiter Chang'e-1 (CE-1) has the capability to concurrently capture hyperspectral imagery with resolution of 200 m along with CCD imagery with a higher resolution of 120 m. To better serve for future lunar terrain exploration, and preparation work for the landing of probe vehicle, resolution of hyperspectral images from CE-1 needs to be enhanced. In order to make use of information from a high-resolution CCD sensor, this paper introduces a resolution enhancing method combining MAP estimation with SMM, and develops an iteration optimization algorithm and a Possion optimization algorithm. Through a set of experiments, it is demonstrated that the proposed method is more effective for practical use compared with conventional methods in resolution enhancement of hyperspectral imageries of CE-1.

An efficient superresolution technique through spatial-spectral data fusion for hyperspectral (HS) imagery is proposed in this letter. The spatial and spectral contents of an HS image are extracted using a linear mixture model and a fully constrained least squares unmixing technique. These data are then combined using a spatial correlation model through a learning-based superresolution mapping (SRM) algorithm. The proposed spatial correlation model realistically simulates a mapping model between the low-resolution (LR) HS image and its subsampled version ( LR2HS image) to train the designed SRM algorithm for mapping from the LR to high resolution. The experiments on real HS images validate the accuracy and low complexity of the proposed autonomous technique for key information detection in HS imagery.

Estimating the number of pedestrians in surveillance images and videos has important applications in intelligent transportation systems. This problem is particularly challenging when the scenes are densely crowded, in which the techniques of tracking a single pedestrian has limited effectiveness. Alternative approaches employ statistical learning algorithms to infer pedestrian counts directly from visual features computed on images or scenes. In this paper, we describe a system for predicting pedestrian counts that significantly extends the utility of those ideas. Our approach incorporates a richer set of features for statistical modeling. While these features give rise to regression problems in a high-dimensional space, we leverage learning techniques to reduce dimensionality while still attaining high accuracy for predicting the number of pedestrians. Empirical results have validated our strategy. Specifically, our system outperforms state-of-the-art methods on standard benchmark tasks by a large margin.

For the last two decades, intelligent transportation systems (ITS) have emerged as an efficient way of improving the performance of transportation systems, enhancing travel security, and providing more choices to travelers. A significant change in ITS in recent years is that much more data are collected from a variety of sources and can be processed into various forms for different stakeholders. The availability of a large amount of data can potentially lead to a revolution in ITS development, changing an ITS from a conventional technology-driven system into a more powerful multifunctional data-driven intelligent transportation system (D2ITS) : a system that is vision, multisource, and learning algorithm driven to optimize its performance. Furthermore, D2ITS is trending to become a privacy-aware people-centric more intelligent system. In this paper, we provide a survey on the development of D2ITS, discussing the functionality of its key components and some deployment issues associated with D2ITS Future research directions for the development of D2ITS is also presented.

A lot of nonlinear embedding techniques have been developed to recover the intrinsic low-dimensional manifolds embedded in the high-dimensional space. However, the quantitative evaluation criteria are less studied in literature. The embedding quality is usually evaluated by visualization which is subjective and qualitative. The few existing evaluation methods to estimate the embedding quality, neighboring preservation rate for example, are not widely applicable. In this paper, we propose several novel criteria for quantitative evaluation, by considering the global smoothness and co-directional consistence of the nonlinear embedding algorithms. The proposed criteria are geometrically intuitive, simple, and easy to implement with a low computational cost. Experiments show that our criteria capture some new geometrical properties of the nonlinear embedding algorithms, and can be used as a guidance to deal with the embedding of the out-of-samples.

Most partitioning algorithms iteratively partition a space into cells that contain underlying linear or nonlinear structures using linear partitioning strategies. The compactness of each cell depends on how well the (locally) linear partitioning strategy approximates the intrinsic structure. To partition a compact structure for complex data in a nonlinear context, this paper proposes a nonlinear partition strategy. This is a principal curve tree (PC-tree), which is implemented iteratively. Given that a PC passes through the middle of the data distribution, it allows for partitioning based on the arc length of the PC. To enhance the partitioning of a given space, a residual version of the PC-tree algorithm is developed, denoted here as the principal component analysis tree (PCR-tree) algorithm. Because of its residual property, the PCR-tree can yield the intrinsic dimension of high-dimensional data. Comparisons presented in this paper confirm that the proposed PC-tree and PCR-tree approaches show a better performance than several other competing partitioning algorithms in terms of vector quantization error and nearest neighbor search. The comparison also shows that the proposed algorithms outperform competing linear methods in total average coverage which measures the nonlinear compactness of partitioning algorithms.

This paper proposes a two-stage registration method for SAR and optical images based on multi-features and multi-constraints. In the first stage, closed regions are extracted automatically to achieve the coarse mapping parameters as geometrical restriction. In the second stage, Harris corner points and cross-road features are extracted, and then correlation analysis and mutual information are utilized to match the corresponding control points. After that, multi-constraints are used to delete the false matched points. The retained ones are served as ground control points for registration. The experimental results show that the method can reduce the possibility of false matching effectively and the registration error is within one pixel.

Hyperspectral image provides rich information of ground covers, which has been brought great attention. Combining hyperspectral spatial and spectral constraint, an image segmentation method based on region active contour is proposed in this paper. The energy function in Chan-Vese's method is improved and both spatial and spectral information are employed. Spatial term of the function is restricted by global spatial information and intensity consistency in homogeneous region, while spectral term is restricted by spectral similarity of the target. Finally the image is segmented by minimizing the energy function. In order to verify the effectiveness of the proposed method, experiments are conducted on AVIRIS data. Results show that the method can extract indistinct contours of interested target in hyperspectral image, and reduce the influence caused by mixed pixels and the shadows around the target, and also adapts to complex scene in some degree.

The determination of principal curves relies on the arc-length as a global index to describe the middle of the data distribution. With a non-constant data distribution, however, curves that are constructed by the approach introduced in reference may not reflect the middle of data distribution, as demonstrated in this article. This is particularly so for curve segments that have a large curvature and a high data density. To overcome this problem, the paper revisits the projection of the samples onto the curve by incorporating Riemannian distances. This analysis suggests estimating the density value of each sample relative to its neighbors and utilize this value to compute the projection index for the curve. The use of density values, in turn, allows penalizing distances between samples along with the arc-length. In a similar fashion to conventional principal curves algorithms, for example proposed by Hastie and Stuetzle and Tibshirani, the incorporation of Riemannian distances gives rise to an iterative algorithm that includes a projection and a self-consistent step. Application studies to simulated and experimental data sets shows that the proposed modification has the potential to outperform existing algorithms in areas of high curvature under an non-constant data distribution.

In this letter, we propose a novel remote sensing image registration method by optimizing the Speeded Up Robust Features (SURF) and developing a new similarity measure function based on trajectories generated from Lissajous figures. Compared with SURF which has a low feature-matching rate in some complex cases, the retrofitted SURF algorithm is more robust and accurate. The algorithm greatly improves the correct matching rate to over 80%. Furthermore, the recognition capability of the similarity measure is enhanced by using a trajectory disturbance strategy, which is a significant displacement in the trajectory induced by a minor error of the transformation parameters. Experiments show the promising performance of the proposed image registration method.

Hyperspectral images (HSI) regularly contain hundreds of bands, which are of different importance in the application. Most HSI compression methods usually deal with most bands in the same way, and they do not take the difference of different bands into consideration, which may cause the loss of important spectral information. In order to preserve the spectral information of interest for applications, a new band-of-interest (BOI)-preserving-based HSI compression method is proposed. The conception of BOI is proposed because some bands are significant in the specific applications, and BOI selection methods are chosen according to application requirements. BOI selection is first performed according to application measurements. Then, BOI information is fed into recursive bidirection prediction (RBP) and set partition in hierarchical trees (SPIHT) compression scheme which uses RBP for spectral decorrelation followed by SPIHT algorithm for coding the resulting decorrelated residual images. More bits are allocated to BOI to preserve BOI by two approaches, respectively. Compress BOI and non-BOI bands directly with low distortion and high distortion, respectively, and compress all bands with low distortion and perform a postcompression truncation. Experiments are implemented with different settings using AVIRIS images. Results indicate that the proposed two methods both can achieve excellent compression efficiency and reconstructed quality. In addition, they can improve the application effect in both material classification and target recognition. Compared with non-BOI compression algorithm, at the compression ratio of 80, the proposed methods improve the classification accuracy by 2% and target recognition accuracy by 9%.

Hyperspectral imaging (HSI) systems can acquire both spectral and spatial information of ground surface simultaneously and have been used in a variety of applications such as object detection, material identification, land cover classification, etc. Through simulation of a HSI process, it is in favor of finding key contributors to optimize system performance and sensor design. Although it is difficult to be expressed accurately, a hyperspectral image simulator could be built on the premise of some simplifications. In this paper, a complex scenario that includes a woodland area with low shrubbery and a woodland area surrounded by high building walls is simulated. The simulated scene model is built, in which a bidirectional reflectance distribution function (BRDF) is introduced and reflectance functions for ground materials are represented using a combination of canopy BRDF and a subpixel fraction model. To improve our previous work, we introduce in the model the secondary illumination based on geometric analysis methods to simulate the effects of reflected solar radiance from other buildings, which improves atmospheric modeling on the basis of nonhomogeneous and non-Lambertian surface. Experiments and analysis show that reasonable results could be achieved.

We study the problem of projecting high-dimensional tensor data on an unspecified Riemannian manifold onto some lower dimensional subspace We note that, technically, the low-dimensional space we compute may not be a subspace of the original high-dimensional space. However, it is convenient to envision it as a subspace when explaining the algorithms. without much distorting the pairwise geodesic distances between data points on the Riemannian manifold while preserving discrimination ability. Existing algorithms, e.g., ISOMAP, that try to learn an isometric embedding of data points on a manifold have a nonsatisfactory discrimination ability in practical applications such as face and gait recognition. In this paper, we propose a two-stage algorithm named tensor-based Riemannian manifold distance-approximating projection (TRIMAP), which can quickly compute an approximately optimal projection for a given tensor data set. In the first stage, we construct a graph from labeled or unlabeled data, which correspond to the supervised and unsupervised scenario, respectively, such that we can use the graph distance to obtain an upper bound on an objective function that preserves pairwise geodesic distances. Then, we perform some tensor-based optimization of this upper bound to obtain a projection onto a low-dimensional subspace. In the second stage, we propose three different strategies to enhance the discrimination ability, i.e., make data points from different classes easier to separate and make data points in the same class more compact. Experimental results on two benchmark data sets from the University of South Florida human gait database and the Face Recognition Technology face database show that the discrimination ability of TRIMAP exceeds that of other popular algorithms. We theoretically show that TRIMAP converges. We demonstrate, through experiments on six synthetic data sets, its potential ability to unfold nonlinear manifolds in the first stage.

Unlike other biometric authentication methods, gait recognition is noninvasive and effective from a distance. However, the performance of gait recognition will suffer in the low-resolution (LR) case. Furthermore, when gait sequences are projected onto a nonoptimal low-dimensional subspace to reduce the data complexity, the performance of gait recognition will also decline. To deal with these two issues, we propose a new algorithm called superresolution with manifold sampling and backprojection (SRMS), which learns the high-resolution (HR) counterparts of LR test images from a collection of HR/LR training gait image patch pairs. Then, we incorporate SRMS into a new algorithm called multilinear tensor-based learning without tuning parameters (MTP) for LR gait recognition. Our contributions include the following: 1) With manifold sampling, the redundancy of gait image patches is remarkably decreased; thus, the superresolution procedure is more efficient and reasonable. 2) Backprojection guarantees that the learned HR gait images and the corresponding LR gait images can be more consistent. 3) The optimal subspace dimension for dimension reduction is automatically determined without introducing extra parameters. 4) Theoretical analysis of the algorithm shows that MTP converges. Experiments on the USF human gait database and the CASIA gait database show the increased efficiency of the proposed algorithm, compared with previous algorithms.

In this paper, we propose a novel face hallucination method to reconstruct a high-resolution face image from a lowresolution observation based on a set of high- and lowresolution local training image pairs. Instead of basing on probabilistic or manifold learning models, the proposed method synthesizes the high-resolution image patch using the same position image patches of training image pairs. A cost function is formulated to obtain the optimal weights of the training image position-patches and the high-resolution patches are reconstructed using the same weights. The final high-resolution facial image is formed by integrating the hallucinated patches. Experiments show that the proposed method without residue compensation generates higherquality images than some methods.

With much richer information than multispectral image, hyperspectral image has been applied in many aspects such as agriculture, environment, military etc. But higher spectral resolution is accompanied by a huge volume of image data, which will result in excessive computing time and data complexity for transmission and storage, so it is necessary to compress hyperspectral image. Because of limit of spatial resolution, some targets that we are interested in are usually in small size and belong to high frequency. In order to preserve such information as much as possible, a hierarchical compression method with protection strategy for hyperspectral image is proposed in this paper, which can protect information of interest (IOI) to some extent, including spatial and spectral IOI. The experimental results show that under the same compression rate, the proposed method can achieve better performance in target detection application than typical SPIHT method.

Multi-sensor image fusion can overcome the limitation of single data source, enhance auto-interpretation capability and improve image quality and usability. In this paper a fusion method based on transferable parameters is presented. The parameters transferability is based on spatial dependence and structural similarity between higher and lower resolution images. The key techniques include fusion model establishment and optimal parameter searching. In order to validate the effectiveness of the proposed method, experiments have been conducted on Landsat TM and SPOT PAN images. The experimental results show that the presented method performs better than some conventional fusion methods both in spatial resolution enhancement and in spectral information preservation.

In order to support spaceborne hyperspectral sensor system design, an end-to-end simulation model for spaceborne hyperspectral image generation starting from the airborne image has been developed in this paper. Airborne image after being resampled both in the space and spectrum performs as the at-sensor radiance that is the input of the sensor model. Sensor model is the main part of proposed model. According to the sensor's imaging process, the simulation is divided into four sub-modules, which are optics, detector, electronics parts and system noise. Based on the theory of optical transfer function (OTF), each sub-module can be treated as a spatial filter and thus its simulation can be realized in the spatial frequency domain. Using parameters of spaceborne sensor Hyperion as well as the image acquired by Airborne Visible and Infrared Imaging Spectrometer (AVIRIS), the validity of the proposed model for sensor design and operation are also demonstrated in the paper.

This paper proposes a method of dimensionality reduction and classification based on ant colony algorithm for hyperspectral remote sensing image. The high-dimensional hyperspectral data space is decomposed into several low-dimensional data subspace by ant colony algorithm (ACA) in terms of the correlation between bands. Then principal component analysis is used in subspace to extract features, whereafter the classification of hyperspectral image is carried out by maximum likelihood classifier. The experiments show that comparing with the method of dimensionality reduction which doesn't use ACA decomposition (i.e. standard PCA), the method proposed is more reasonable, and reserves more useful information, has the higher classification accuracy.

In this paper, a new algorithm is proposed for resolution enhancement in hyperspectral images (HSIs). The key techniques are included: spectral unmixing and superresolution mapping, by which spatial and spectral information of HSIs is substantially fused. The proposed algorithm first represents each pixel in scene as a linear combination of landcover spectra and noise. Then, a fully constrained least squares algorithm is used to obtain the proportion of each landcover in each pixel, i.e., abundance, subjecting to two constraints: nonnegativity and sum-to-one. After that, superresolution mapping is performed on high-resolution grids according to spectral unmixing abundances of each landcover and following spatial correlation of clutters. Thus, by reasonably integrating spatial and spectral information of landcovers in HSIs, the proposed algorithm realizes resolution enhancement of the HSIs based on a back-propagation neural network. The proposed algorithm is independent from thea prioriinformation associated with original HSIs, i.e., a main merit of the algorithm. In order to evaluate the performance of the new algorithm, numerical experiments are conducted on both simulated images and real HSIs collected by the Airborne Visible/Infrared Imaging Spectrometer. The proposed algorithm is compared with the traditional method in the experiments. The experimental results prove that the proposed algorithm effectively enhances the resolution of HSIs and indicate its applicability.

This paper revisits the construction of principal curves. Although they have a solid theoretical foundation as a nonlinear extension to principal components, this paper shows that they are difficult to implement in practice if the data distribution is sparse and uneven or if the data contain outliers. These issues may hamper the application of principal curves to an intelligent transportation system. To address these problems, this paper introduces an adaptive constraint K-segment principal curve (ACKPC) algorithm that can be applied in the presence of uneven and sparse distributions, as well as outliers. The benefits of the ACKPC algorithm are as follows: (1) It utilizes predefined endpoints of the curve to reduce the computational effort, and (2) it shows to be less sensitive to parameter settings and outliers. These benefits are demonstrated using two benchmark studies and experimental data from a freeway traffic stream system as well as recorded data from a Global Positioning System (GPS) data from a low-precision GPS receiver.

Based on the basic principle of PolInSAR and the coherent scattering model of random volume over ground, the inversion of forest parameters of PolInSAR can be characterized by a six-dimensional non-linear parameter optimization problem. However, the global optimal can't be obtained using the traditional gradient-based optimization algorithms. Therefore, a global optimization inversion scheme of forest parameters of PolInSAR based on genetic algorithm is presented. We generate a validity test using SIR-C L-band repeat-pass PolInSAR data of the area of Tien Shan, China. The preliminary results accord with the range of the parameters of the fact. Performances of different GAs and effects of different parameters are compared. SGA is influenced by the mutation rate strongly, but GA with tournament of two generations is independent of the mutation rate.

In this paper we present a target detection method for hyperspectral images using feature extraction based on independent component analysis (ICA). This method makes good use of the high order statistic of image data and greatly overcome the spectral signature variability. ICA aims to find a linear representation of the observed data in order that the components are statistically independent, or as independent as possible. Such an independent component can capture the intrinsic structure of data and extract image features, including target feature that will be used in detection. First each pixel, which is assumed to be a linear mixture of target and background spectra, is projected onto the orthogonal background subspace to remove the background spectral portion from the corresponding pixel spectrum. Then the targets in the background-removed image are estimated through matched filtering with the feature of target component extracted by ICA. The method has been testified on airborne visible and infrared imaging spectrometer (AVIRIS) data. The experimental results show that targets are successfully separated from the background, demonstrating the good performance of this method to detect targets in hyperspectral images.

In this paper, a novel image change detection algorithm based on clustering characteristic of 2-D histogram formed by pixel gray levels and the local average gray levels is proposed. First, the 2-D histogram is segmented into two initial clusters representing change region and unchanged region respectively by using classical segmentation method. Then, the traditional 2-D maximum entropy principle is improved properly to adjust the initial clusters. Finally, changes are detected according to the two relative more accurate clusters that have been adjusted. Theoretical analysis and experimental results show that the proposed algorithm has more accurate detection precision, stronger anti-noise capability and faster computation than traditional 2-D maximum entropy algorithm.

In this paper, change detection for urban area based on multiple sensor information fusion is studied. The images acquired at different times are used to identify the differences during the process of city construction and reconstruction. The proposed method in this paper consists of three steps: image pre-processing and registration, information fusion for multiple sensor images and image change detection. Wavelet packet transform is used in the fusion stage. More detailed features can be extracted by high frequency components as well as low frequency components decomposition. The experiments are carried on optical and SAR images taken at different periods. The results show that the method based on multi-source information fusion can effectively detect changed regions.

We have proposed to use the method of principal curves to describe and analyze the interaction among freeway traffic-stream variables and their joint behaviors without utilizing conventional assumptions made on the functional forms of interactions, as in previous studies. As a nonparameter modeling approach, the performance of the proposed method depends only on the data used and involves no assumed knowledge regarding the relationship among the traffic-stream variables. First, we discuss the basic algorithm for data analysis using principal curves and the corresponding data filter algorithm for determining principal curves for application in traffic-steam analysis. Second, a case study is used to compare the performance of the proposed method to that of the classical model proposed by Greenshields; results indicate that the proposed model is better than the classical one in both data accuracy and curve shape. Finally, the traffic-stream models generated with principal curves at different locations and lanes are compared with each others and the three-dimensional traffic-stream models developed from principal curves are discussed. Clearly, our results have demonstrated the feasibility and advantages of applying principal curves in freeway traffic-stream modeling and analysis.

A kernel based nonlinear subspace projection (KNSP) method is proposed for reduction of hyperspectral image dimensionality. This method involves three steps: subspace partition of full data space, feature extraction based on kernel principal component analysis (KPCA) in subspace and feature selection based on class separability criterion. The main merit of the proposed method is that it is more suitable for feature extraction than linear principal component analysis (PCA) and segmented principal component: transformation (SPCT), in particular, when hyperspectral data have nonlinear characteristics. In order to testify the effectiveness of the KNSP method for reduction of hyperspectral image dimensionality, hyperspectral image classification is performed on AVIRIS data. Experimental results show that when the hyperspectral dimensionality is reduced to a few features, the average classification accuracy of the new method is higher than those of PCA and SPCT methods.

Classification is one of the most important tasks for remote sensing image processing. Most of the existing supervised classification methods are based on traditional statistics, which can provide ideal results when sample size is tending to infinity. However, only finite samples can be acquired in practice. In addition, many methods are constrained by high data dimension of hyperspectral images. In this paper, a novel learning method, the support vector machine (SVM), is applied to hyperspectral data classification. This method does not suffer the limitations of data dimensionality and limited samples. The foundations of the SVM have been developed by Vapnik (1995) and are gaining popularity in field of machine learning due to many attractive features and promising empirical performance. In our experiment, the support vectors, which are critical for classification, are obtained by learning from the training samples. Choosing appropriate kernel function and suitable parameters, better classification results are obtained.

Data fusion is widely used in many fields in the last two decades. With the development of the hyperspectral sensor technique, the concept of data fusion is introduced into the classification investigation of hyperspectral data recently. Consensus theory is one of the data fusion methods, in which how to properly choose and assign the weights is very important for the improvement of fusion classification accuracy. In this paper, a new method of hyperspectral image classification is studied, which is realized by two key steps: division of data sources based on adaptive subspace decomposition (ASD) and fusion classification based on consensus theory. In order to testify the effectiveness of the proposed method, computer simulations are conducted on AVIRIS data. The experiment investigation shows that the classification result in our new method is improved compared with both the equal weights and conventional approach in the full data space.

This paper proposed a novel adaptive subspace decomposition (ASD) method for hyperspectral data dimensionality reduction. The new method is mainly based on the criterions of the correlation matrix and the variability ratio of eigenvalues and it can overcome the disadvantages of the conventional Principal Component Analysis (PCA) method. To evaluate the effectiveness of the new method, experiments are conducted on AVIRIS data. The data dimensionality is reduced from 100 to 5 bands. When applied to classification, the results show that the new method keeps more detail information than the conventional PCA method and can get higher classification accuracy.