Visual abductive reasoning (VAR) is a challenging task that requires AI systems to infer the most likely explanation for incomplete visual observations. While recent MLLMs develop strong general-purpose multimodal reasoning capabilities, they remain fall short in abductive inference, as compared to human beings. To bridge this gap, we draw inspiration from the interplay between verbal and pictorial abduction in human cognition, and propose to strengthen abduction of MLLMs by mimicking such dual-mode behavior. Concretely, we introduce AbductiveMLLM comprising of two synergistic components: REASONER and IMAGINER. The REASONER operates in the verbal domain. It first explores a broad space of possible explanations using a blind LLM and then prunes visually incongruent hypotheses based on cross-modal causal alignment. The remaining hypotheses are introduced into the MLLM as targeted priors, steering its reasoning toward causally coherent explanations. The IMAGINER, on the other hand, further guides MLLMs by emulating human-like pictorial thinking. It conditions a text-to-image diffusion model on both the input video and the REASONER’s output embeddings to “imagine” plausible visual scenes that correspond to verbal explanation, thereby enriching MLLMs' contextual grounding. The two components are trained jointly in an end-to-end manner. Experiments on standard VAR benchmarks show that AbductiveMLLM achieves state-of-the-art performance, consistently outperforming traditional solutions and advanced MLLMs.

Remote sensing change detection (CD) has achieved remarkable progress in recent years. However, little attention has been paid to generalizable change detection (GCD) methods that can effectively generalize to unseen scenarios or domains beyond the training distribution. The major challenges in GCD arise from domain diversity and bitemporal domain shifts in remote sensing images, caused by variations in imaging platforms, acquisition times, geographic regions, and observed events. To tackle these challenges, we propose GenCD, a GCD framework built upon vision foundation models (VFMs). Specifically, GenCD introduces two key components: (1) a Low-Rank Exchange Adaptation (LREA) strategy of VFMs that aligns bitemporal representations while preserving the generalization capacity of VFMs on single-temporal inputs; and (2) a Token-Guided Feature Refinement (TGFR) mechanism that leverages an input-independent token as a guide to refine difference features, improving the discrimination between changed and unchanged regions. We conduct extensive cross-dataset evaluations on eight diverse datasets across three binary CD tasks: land cover, land use, and building-only CD. The results consistently demonstrate the superior generalization of GenCD over SoTA methods, highlighting its effectiveness in GCD.

We challenge the assumption that complex instruction-guided segmentation tasks necessitate equally complex and explicit supervision. This paper introduces RISE (Reasoning via Implicit Self-supervised Emergence), a framework that learns intricate compositional reasoning, spanning spatial relations to world knowledge, without a single ground-truth mask. To achieve this, RISE employs reinforcement learning with GRPO guided by a single, strikingly simple reward: the semantic alignment score between the textual instruction and the predicted image region. Our primary discovery is the implicit emergence of a high-quality chain-of-thought process from this minimalist signal. Within a structured format, the model autonomously learns to understand instructions by accessing its latent knowledge, inferring spatial relationships—capabilities inherent in its architecture but unlocked by our simple objective. Remarkably, our emergent reasoning yields highly competitive results: RISE achieves 58.7 gIoU on the ReasonSeg benchmark, on par with methods using geometric rewards. Furthermore, we show extreme data efficiency: a variant trained on only 2,000 ImageNet-label pairs establishes a new state-of-the-art for annotation-free referring segmentation with 79.6 cIoU on RefCOCO.

Shadows are dark areas, typically rendering low illumination intensity. Admittedly, the infrared image can provide robust illumination cues that the visible image lacks, but existing methods ignore the collaboration between heterogeneous modalities. To fill this gap, we propose a weakly supervised shadow removal network with a spherical feature space, dubbed S2-ShadowNet, to explore the best of both worlds for visible and infrared modalities. Specifically, we employ a modal translation (visible-to-infrared) model to learn the cross-domain mapping, thus generating realistic infrared samples. Then, Swin Transformer is utilized to extract strong representational visible/infrared features. Simultaneously, the extracted features are mapped to the smooth spherical manifold, which alleviates the domain shift through regularization. Well-designed similarity loss and orthogonality loss are embedded into the spherical space, prompting the separation of private visible/infrared features and the alignment of shared visible/infrared features through constraints on both representation content and orientation. Such a manner encourages implicit reciprocity between modalities, thus providing a novel insight into shadow removal. Notably, ground truth is not available in practice, thus S2-ShadowNet is trained by cropping shadow and shadow-free patches from the shadow image itself, avoiding stereotypical and strict pair data acquisition. More importantly, we contribute a largescale weakly supervised shadow removal benchmark that makes shadow removal independent of specific scenario constraints possible. Extensive experiments demonstrate that S2-ShadowNet outperforms state-of-the-art methods in both qualitative and quantitative comparisons. The code and benchmark are available at https://github.com/chi-kaichen/S2-ShadowNet.

As a bridge between the moon and human perception, the lunar optical image reflects lunar topography, geology, and evolution. Unfortunately, the permanent shadow regions (PSRs) near the lunar poles suffer from information contamination due to insufficient illumination. Low-light enhancement is a subjective process whose target is tied to human visual perception. However, existing low-light enhancement methods often operate as opaque “black box”, lacking transparency and failing to accommodate diverse perceptual preferences. To this end, we explore a PSRs Low-Light Enhancer (PSRs-LLE) that treats low-light enhancement as a Markov decision process, thereby dynamically fitting perceptual preferences. Specifically, a deep Q network as an agent integrates multiple user-friendly attributes (e.g., brightness, contrast, chroma, and detail) through actions recursion (i.e., a candidate set of image enhancement operations). Such transparent and specific action sequences satisfy customization preferences of users while providing convincing interpretability, compared with the “black box” paradigm of deep learning. More importantly, a well-designed non-reference loss function liberates PSRs-LLE from the dilemma of virtual assumptions and paired data, which further enhances usability. Extensive experiments demonstrate that PSRs-LLE outperforms state-of-the-art methods in both qualitative and quantitative comparisons.

Scene text reading is a crucial task for scene understanding. Text detection, as a fundamental task in scene text reading, has recently garnered significant attention. Among various approaches, segmentation-based methods stand out for their flexible pixel-level prediction capabilities. However, two main issues remain. 1) These methods treat all text instances as a pixel set during training, causing the features of large-scale instances to dominate the model optimization process. As a result, the optimization deviates from the instance-level objectives. 2) Segmentation methods filter candidates based on pixel-level class scores, whereas what is needed is an evaluation of whether an instance is text, which also deviates from the original goals. To address these issues, we propose an Instance-Equal Feature Guide Module (IEFGM), a Cross-Level Feature Interaction Module (CLIFM), and a Pixel-Instance Fusion Discriminator (PIFD) to balance optimization strategies with practical goals. The IEFGM introduces instance-level features and positional information, guiding the model to treat instances of different scales equally at the feature level. The CLIFM encourages feature interaction across different levels, enabling the model to recognize text from various perspectives. Unlike existing methods that filter candidates using pixel-level results, the PIFD integrates both instance-level and pixel-level information to identify candidate regions, aligning with the original goals of text detection. A series of ablation studies demonstrates the effectiveness of the proposed modules. Extensive experiments across six datasets from different scenes demonstrate that our method outperforms existing state-of-the-art approaches.

To pursue an efficient text assembling process, existing methods detect texts via the shrink-mask expansion strategy. However, the shrinking operation loses the visual features of text margins and confuses the foreground and background difference, which brings intrinsic limitations to recognize text features. We follow this issue and design Text-Pass Filter (TPF) for arbitrary-shaped text detection. It segments the whole text directly, which avoids the intrinsic limitations. It is noteworthy that different from previous whole text region-based methods, TPF can separate adhesive texts naturally without complex decoding or post-processing processes, which makes it possible for real-time text detection. Concretely, we find that the band-pass filter allows through components in a specified band of frequencies, called its passband but blocks components with frequencies above or below this band. It provides a natural idea for extracting whole texts separately. By simulating the band-pass filter, TPF constructs a unique feature-filter pair for each text. In the inference stage, every filter extracts the corresponding matched text by passing its pass-feature and blocking other features. Meanwhile, considering the large aspect ratio problem of ribbon-like texts makes it hard to recognize texts wholly, a Reinforcement Ensemble Unit (REU) is designed to enhance the feature consistency of the same text and to enlarge the filter's recognition field to help recognize whole texts. Furthermore, a Foreground Prior Unit (FPU) is introduced to encourage TPF to discriminate the difference between the foreground and background, which improves the feature-filter pair quality. Experiments demonstrate the effectiveness of REU and FPU while showing the TPF's superiority.

The rapid growth of dataset scales has been a key driver in advancing deep learning research. However, as dataset scale increases, the training process becomes increasingly inefficient due to the presence of low-value samples, including excessive redundant samples, overly challenging samples, and inefficient easy samples that contribute little to model improvement. To address this challenge, we propose Scale Efficient Training (SeTa) for large datasets, a dynamic sample pruning approach that losslessly reduces training time. To remove low-value samples, SeTa first performs random pruning to eliminate redundant samples, then clusters the remaining samples according to their learning difficulty measured by loss. Building upon this clustering, a sliding window strategy is employed to progressively remove both overly challenging and inefficient easy clusters following an easy-to-hard curriculum. We conduct extensive experiments on large-scale synthetic datasets, including ToCa, SS1M, and ST+MJ, each containing over 3 million samples. SeTa reduces training costs by up to 50% while maintaining or improving performance, with minimal degradation even at 70% cost reduction. Furthermore, experiments on various scale real datasets across various backbones (CNNs, Transformers, and Mambas) and diverse tasks (instruction tuning, multi-view stereo, geo-localization, composed image retrieval, referring image segmentation) demonstrate the powerful effectiveness and universality of our approach. Code is available at https://github.com/mrazhou/SeTa.

Full Parameter Fine-Tuning (FPFT) has become the preferred method for adapting LLMs to downstream tasks due to its exceptional performance. Current methods primarily utilize zeroth-order optimizers or integrate gradient computation and updates to conserve GPU memory. However, they fail to consider the optimizer states information (e.g., momentum, variance), leading to suboptimal convergence and instability during training. To address this, we propose a Quantized Memory-Efficient Full-Parameter Tuning with Sign descent optimization training framework (SQ-MEFT). Firstly, we construct a novel optimizer that uses the sign of momentum as the update amount to maximize the potential of momentum. In addition, to better maintain memory efficiency, we apply 4-bit quantization to the momentum while synchronously computing and updating gradients. When trained with mixed precision, our optimizer can reduce the total memory footprint by up to 7× compared to AdamW.

Scene graph generation (SGG) in satellite imagery (SAI) benefits promoting understanding of geospatial scenarios from perception to cognition. In SAI, objects exhibit great variations in scales and aspect ratios, and there exist rich relationships between objects (even between spatially disjoint objects), which makes it attractive to holistically conduct SGG in large-size very-high-resolution (VHR) SAI. However, there lack such SGG datasets. Due to the complexity of large-size SAI, mining triplets subject, relationship, object heavily relies on long-range contextual reasoning. Consequently, SGG models designed for small-size natural imagery are not directly applicable to large-size SAI. This paper constructs a large-scale dataset for SGG in large-size VHR SAI with image sizes ranging from 512 × 768 to 27,860 × 31,096 pixels, named STAR (Scene graph generaTion in lArge-size satellite imageRy), encompassing over 210K objects and over 400K triplets. To realize SGG in large-size SAI, we propose a context-aware cascade cognition (CAC) framework to understand SAI regarding object detection (OBD), pair pruning and relationship prediction for SGG. We also release a SAI-oriented SGG toolkit with about 30 OBD and 10 SGG methods which need further adaptation by our devised modules on our challenging STAR dataset. The dataset and toolkit are available at: https://linlin-dev.github.io/project/STAR.

Scene text reading is an essential component of scene understanding. As its fundamental requirement, text detection has garnered increasing attention. Segmenting the text kernel and extending it to reconstruct text instances is efficient and effective among the various methods. However, the incomplete semantic features of text kernels and the high similarity between kernels and texts make it hard to extract kernels from images accurately. Considering the above, we propose an efficient text detector, termed CIC, which comprises a bidirectional information transfer module (BITM), a dual knowledge integration module (DKIM), and a cross-verification module (CVM). The former generates collaborative information between the predicted text and kernel via the proposed differentiable adaptive gap operator. It forces mutual restraint and collaborative progress between the predictions of text and kernel. Unlike BITM, DKIM designs a knowledge fuse scheme, which helps to locate kernels accurately under the guidance of the complete semantic feature of texts. Intuitively, as the kernel is generated by shrinking the text, the kernel pixel is only presented in the text area. Based on this criterion, the CVM further utilizes text predictions to constrain kernel predictions and reduce false positive predictions. Ablation experiments demonstrate the effectiveness of the proposed BITM, DKIM, and CVM. Extensive experiments show the proposed CIC outperforms existing state-of-the-art (SOTA) methods on five public datasets from different scenes. The code is available at https://github.com/fengmulin/CIC.

Pursuing efficient text shape representations helps scene text detection models focus on compact foreground regions and optimize the contour reconstruction steps to simplify the whole detection pipeline. Current approaches either represent irregular shapes via box-to-polygon strategy or decomposing a contour into pieces for fitting gradually, the deficiency of coarse contours or complex pipelines always exists in these models. Considering the above issues, we introduceEdgeTextto fit text contours compactly while alleviating excessive contour rebuilding processes. Concretely, it is observed that the two long edges of texts can be regarded as smooth curves. It allows us to build contours via continuous and smooth edges that cover text regions tightly instead of fitting piecewise, which helps avoid the two limitations in current models. Inspired by this observation, EdgeText formulates the text representation as the edge approximation problem via parameterized curve fitting functions. In the inference stage, our model starts with locating text centers, and then creating curve functions for approximating text edges relying on the points. Meanwhile, truncation points are determined based on the location features. In the end, extracting curve segments from curve functions by using the pixel coordinate information brought by truncation points to reconstruct text contours. Furthermore, considering the deep dependency of EdgeText on text edges, a bilateral enhanced perception (BEP) module is designed. It encourages our model to pay attention to the recognition of edge features. Additionally, to accelerate the learning of the curve function parameters, we introduce a proportional integral loss (PI-loss) to force the proposed model to focus on the curve distribution and avoid being disturbed by text scales. Ablation experiments demonstrate that EdgeText can fit scene texts compactly and naturally. Comparisons show that EdgeText is superior to existing methods on multiple public datasets. Code is available at https://github.com/omtcyang/EdgeTD.

Shadow removal is an essential task for remote sensing imagery analysis, which is tricky due to spatial irregular and inhomogeneous degradation distribution. Unfortunately, current shadow removal pipelines face challenges with suboptimal performance and insufficient interpretability. To this end, we unleash the long-sequence modeling potential of State Space Models (SSMs) in the context of shadow removal. Coupled with the accurate perception of traditional Retinex decomposition towards illumination, the well-designed RSMamba enjoys the best of both worlds between superior competitiveness and theoretical intuitiveness. Specifically, RSMamba mimics the retina and cerebral cortex to explore illumination and reflectance. The former drives the selective scan mechanism to enhance the response towards contamination, while the latter serves as a tool to preserve illumination fidelity. In addition, contour and gradient regularizations of illumination and reflectance components reflect the spatial opponency of shadows, which are consistent with the center-surround opponent receptive field of the human visual system. Such a manner incorporates the domain knowledge of neurophysiological mechanisms into neural networks, providing new insights into shadow removal. Extensive experiments demonstrate that RSMamba outperforms state-of-the-art methods.

Remote sensing shadow removal task aims to effectively restore key regions of an image obscured by shadows. However, the spatial non-uniformity of shadow distribution presents significant challenges to this task. To address this issue, we propose RMMamba, a shadow removal network based on the SS2D architecture. The core concept of RMMamba involves balancing the spatial non-uniformity of shadow distribution, thereby optimizing the utilization efficiency of non-shadow pixel information across different windows. Specifically, RMMamba employs a random pixel shuffling operation to evenly disperse pixels from shadow regions with pronounced spatial non-uniformity into non-shadow areas, ensuring a more balanced spatial distribution of shadow and non-shadow pixels within each window. Subsequently, a shared weight local State Space Model (SS2D) is employed to integrate non-shadow pixel features uniformly distributed around shadow pixels, consequently effectively relighting shadow pixels. Reverse shuffling operations are then applied to restore the processed image to its original pixel order. Coupled with CP-FFN, a lightweight feedforward network incorporating color priors, RMMamba effectively restores color in shadow regions. More importantly, given the difficulty of acquiring remote sensing shadow samples with corresponding ground truth and shadow masks, we leverage the game GTA to control its shadow renderer and create SRGTA, a synthetic fully supervised dataset, hence providing a new benchmark for the performance evaluation of remote sensing shadow removal algorithms. Extensive experiments conducted on SRGTA and UAV-SC have demonstrated the outstanding performance of RMMamba. The code and SRGTA dataset are publicly available at https://github.com/xgd-cj/RMMamba.

Automated road segmentation from remote sensing imagery remains a fundamental challenge in Earth observation systems. The primary bottleneck lies in acquiring dense pixel-wise annotations, which is both labor-intensive and time-prohibitive. This article presents DualStrip-Net, a novel deep learning framework for weakly supervised and semi-supervised road segmentation that effectively handles both sparse annotations and limited labeled data. Unlike conventional convolutional neural network (CNN)-based segmentation methods that lack explicit road topology modeling, DualStrip-Net exploits the inherent linear topology of road networks through a dual-stream architecture that combines patch-level annotation strategy and strip-based feature learning. The framework captures road characteristics through orthogonal strip processing in horizontal and vertical orientations. The proposed DualStrip Learning mechanism enables robust feature representation of road structures through complementary views. Extensive evaluations on the DeepGlobe, Massachusetts, and CHN6-CUG benchmark datasets demonstrate that DualStrip-Net achieves superior performance in both weakly supervised and semi-supervised settings. Notably, with only 20% of labeled training data, our method outperforms the supervised-only baselines on both Massachusetts and CHN6-CUG datasets. The code is available athttps://github.com/jasonnhu/DualStrip-Net/.

Advancements in few-shot segmentation (FSS) for remote sensing images have significantly improved the ability to binarization parse novel classes using only a few supports. Generalized few-shot segmentation (GFSS), a challenging and practical task, has recently attracted research attention. It involves recognizing base and novel classes while segmenting multiple categories in a query. Most GFSS methods adopt a two-stage approach: base classifier training and novel classifier registering. However, they encounter two key challenges: the data scale disparity between base and novel classes and significant intraclass variation in remote sensing images. In this article, we present a dual-view classifier evolution (DiCE) method. Our approach utilizes the well-trained base classifier to allocate attention within the novel classifier, effectively addressing the disparities between the two. Simultaneously, it fosters context-driven interactions between the query and the classifier, tailoring sample-specific classifiers to mitigate intraclass variations. Furthermore, we propose a binocular hybrid training (BHT) mechanism that integrates normal base training with episodic training, endowing the model with the ability to adapt to few-shot tasks. Extensive experiments on the iSAID-$5^{i}$dataset demonstrate the superior performance of DiCE.

Few-shot segmentation (FSS) techniques enhance pixel-level interpretation of unseen classes while reducing reliance on extensive labeled data. However, FSS still faces significant limitations in practical applications: it is restricted to segmenting novel classes and relies on manually constructed support-query pairs during inference. We are the first to introduce the generalized few-shot segmentation (GFSS) task to remote sensing analysis. It enables simultaneous segmentation of base and novel classes without manual prior interventions. The most intuitive construction is to extend a pretrained base classifier with a novel classifier. Nevertheless, since the latter is aggregated from a limited number of supports while the former is trained on abundant data, this disparity inevitably introduces a base class bias, leading to suboptimal segmentation results. This article proposes a background-aware self-mining prototype learning (BSPL) strategy to address the issues above. Specifically, we design a dynamic prototype update mechanism during training to enhance the model’s adaptability in few-shot scenarios and thereby mitigate the base class bias. Considering the intraclass variation and complex background elements in remote sensing images, we customize segmentation guidance for each query through background-aware self-mining, achieving more precise segmentation performance. Compared to peer algorithms, extensive experiments demonstrate that BSPL achieves the best overall segmentation performance for both base and novel classes, indicating its significant practicality.

Referring Remote Sensing Image Segmentation (RRSIS) aims to establish pixel-level interpretation of specific regions queried by textual expressions, bridging textual semantics and intelligent analysis of remote sensing imagery. In contrast to natural scenarios, the intricate backgrounds in remote sensing scenarios result in low target-background contrast, often leading to semantic dispersion in segmented regions. Furthermore, conventional cross-attention-based referring image segmentation (RIS) methods struggle to bridge the modal gap, hindering fine-grained alignment between linguistic descriptions and geographical features. To overcome these challenges, we present a pioneering Entity-Guided Attention Twisting Network (Enti-TwistNet) for RRSIS. Our framework first introduces a SAM-inspired Entity Guidance (SEG) module that extracts spatially constrained entity prompts through a self-reasoning mask generation mechanism, constructing a comprehensive entity-visual-text tri-modal information cube. Subsequently, during cross-modal interaction, we propose a Dual-phase Attention-Twisting (DAT) mechanism: (1) initially sequential channel-wise scanning to facilitate cross-modal semantic propagation; (2) Subsequently, twist attention to the spatial dimension, integrating entity guidance to enhance the representation of irregular geographic boundaries. Extensive experiments on two widely used benchmarks, RefSegRS and RRSIS-D, demonstrate that Enti-TwistNet achieves significant performance improvements over existing state-of-the-art models.

Few-Shot Segmentation (FSS) in remote sensing aims to achieve segmentation of novel categories in query images using limited annotated support images. Despite extensive research, the significant intra-class differences of remote sensing targets continue to hinder progress in this field. Pre-trained vision-language models (VLMs) possess strong generalization capabilities, and their cross-modal information can effectively mitigate intra-class variance issues. However, VLMs rarely focus on dense prediction tasks, and the complexity of remote sensing imagery limits the effectiveness of existing attempts on FSS tasks. To address this issue, this article proposes an Implicit CLIP Prior Decoupling Network (ICPD-Net), which mines effective cross-modal priors from VLMs and leverages ranking information to improve visual metric strategies. Specifically, the Implicit Prior Decoupling Module (IPDM) utilizes ambiguous foreground-background vision-language similarities to construct class-agnostic prompts, while employing a prior learner to mine implicit vision-language priors that alleviate intra-class differences. To fully leverage cross-modal information, the Reliable Feature Fusion Module (RFFM) utilizes vision-language priors to obtain high-confidence query features for fusion with support features, and further mitigating intra-class differences through self-support paradigm. Finally, the Dual Visual Priors Module (DVPM) introduces a novel rank information prior for visual feature measurement. This approach constructs an effective metric learning method by combining the ranking relationships of Euclidean distances between support-query features with the Normalized Discounted Cumulative Gain (NDCG) algorithm, while comprehensively exploring visual metric relationships through traditional cosine similarity prior. Extensive experiments on iSAID-5iand DLRSD-5idemonstrate that our method achieves significant improvements. Particularly under the 1-shot setting, our approach shows exceptional effectiveness, outperforming state-of-the-art methods by up to 11.48% on the iSAID-5i dataset. The code of ICPD-Net is available at https://github.com/yeh15/ICPD-Net.

Haze removal in remote sensing (RS) images has become increasingly vital due to their capacity to contain essential information for accurate geospatial analysis. Notably, this phenomenon is particularly pronounced in both spatial and spectrum distributions of buildings, complex terrain, and landforms. Inspired by the success of generative models in enhancing details incrementally and suppressing noise, we propose a deep frequency-guided denoising diffusion model for RS imagery dehazing. The pixel-level generative capability of the diffusion model is fully leveraged, and the fast Fourier transform is utilized to extract frequency-domain information. This enables the separate mining of semantic information from RS images in both spatial and spectral domains. Concurrently, the continuity of the image in the frequency domain is ensured without altering the diffusion process, thus achieving detail retention while improving overall clarity. Furthermore, to address the scarcity of physically realistic training data for spatially heterogeneous atmospheric degradation, we construct a Random Haze Distribution Dataset for Remote Sensing dehazing (RHDRS). RHDRS randomly simulates the spatial distribution and thickness of haze, containing 4,500 hazy images along with the corresponding ground truths. Experiments demonstrate that our approach outperforms existing state-of-the-art techniques. The dataset and the code can be accessed at https://github.com/Junjie-LLL/DFG-DDM.

At present, many infrared target detection approaches focus on designing modules that address the two key characteristics of targets: their weak signals and small size. However, these approaches often fail to fully leverage guided learning for weak and small target content, resulting in sub-optimal detection performance, particularly in terms of shape preservation and target positioning. To tackle this challenge, this paper proposes a multi-branch mutual-guiding learning network (MMLNet) that enhances the accuracy of infrared target detection, even in the absence of clear morphological and textural features in images. The method consists of three branches: edge, positioning, and detection, each of which is designed with a specialized module from a unique perspective. In the detection branch, we introduce a multi-dimensional lossless encoder optimized through a downsampling strategy and multi-level feature fusion to mitigate feature loss in small targets. In the positioning branch, a target positioning strategy is proposed to explicitly identify candidate targets from the image by means of a learnable multi-kernel pattern. In the edge branch, a simple architecture is adopted to enhance the ability of the model to preserve the target shape. To effectively utilize the knowledge of different branches, a mutual-guiding fusion module is developed to adjust information within and between branches. The manner adaptively utilizes the specific knowledge from each input branch. Experiment results demonstrate that the proposed method achieves comparable performance, and the visualization results show the advantages of our method in shape preservation and positioning of the targets. Our code is publicly available at https://github.com/qianngli/MMLNet.

Existing object detection methods is constrained by the high annotation costs, particularly in remote sensing due to the diversity of targets and the large scale of data. Visual-Prompted Interactive Object Detection can enhance the efficiency of data annotation by leveraging user-provided visual prompts to iteratively refine detection results. However, current interactive annotation frameworks are hindered by their reliance on simple feature fusion strategies, which limit their ability to capture fine-grained semantic relationships. Moreover, more advanced fusion methods face computational complexity challenges, making them unsuitable for high-resolution feature spaces commonly encountered in remote sensing imagery. To address these limitations, we propose InterMamba, an efficient framework for interactive object detection in remote sensing images. InterMamba integrates the VMamba backbone and a novel Cross Vision Selective Scan Module (Cross-VSSM) to achieve linear-complexity multi-scale feature fusion, reducing memory consumption while capturing fine-grained details in high-resolution feature spaces. To further enhance interaction flexibility and detection precision, a hybrid Gaussian heatmap generation method is proposed to encodes user-provided point and bounding box annotations. Meanwhile, a User Interaction Loss function further optimizes detection accuracy in dense scenarios by aligning localization and classification with user guidance. Our experiments demonstrate that InterMamba consistently outperforms existing methods in mean Average Precision (mAP). In terms of enhancing precision and reducing annotation costs, InterMamba establishes a robust solution for interactive remote sensing object detection. Code will be available at https://github.com/lsjhaha/InterMamba.

Cross-resolution change detection (CRCD) identifies changes between bitemporal images with different resolutions, which provide better adaptability to real-world applications than conventional change detection (CD). Existing CRCD methods first align the resolution of different temporal images, then employ the Siamese network. Experiments conducted from a frequency-domain perspective validate that resize operations disrupt the data distribution and result in performance degradation of the Siamese network. Further experiments reveal the resolution-invariant temporal and spatial unequal relationship between bitemporal images. Specifically, spatial specificity information within a specific temporal domain is more critical for CRCD, i.e., high-frequency components in a specific temporal domain are closely related to change label. And this unequal relationship exhibits invariance in resolution. On this basis, we propose the Fourier and wavelet transform-based inequality Siamese network (FWISN) to address the performance degradation observed in Siamese networks on CRCD, leveraging the inequality between bitemporal images to improve network performance. FWISN includes a frequency reconstruction (FRC) stage, in which high-frequency components of a given temporal-domain image are extracted and reconstructed using our proposed high-frequency attention (HFA) module. We further propose the wavelet transform-based frequency learning block (WFB), which enhances high-frequency features and is integrated into both the encoder (WFB-E) and the decoder (WFB-D) of the network. The experiments demonstrate state-of-the-art performance, compared with methods specifically designed for cross-resolution tasks, FWISN achieving$F1$/intersection over union (IoU) improvements of 2.15/3.51, 2.50/4.59, and 1.93/1.73 on the LEVIR-CD ($4 \times $), SV-CD ($8 \times $), and DE-CD ($3.3\times $) tests, respectively. Furthermore, in the continuous CRCD task, FWISN achieves$F1$/IoU improvements of 8.39/11.24 on the LEVIR-CD ($8 \times $) test. Our code will be public onhttps://github.com/blacksheep182/NN_FWISN

Few-Shot Object Detection (FSOD) aims to recognize novel class targets using limited annotated data. Conventional approaches rely on extensive base class training, followed by fine-tuning where few instances from both base and novel classes are sampled for each category. Although they demonstrate remarkable performance in natural image domains, the specificity of remote sensing scenarios poses two critical challenges for FSOD: 1) The morphological differences between remote sensing images and natural images are significant, leading to a loss of structural priors in the Region Proposal Network (RPN). This makes it difficult for structural priors pretrained on natural images to generalize to remote sensing images, especially for novel class with scarce data; 2) Differences in imaging conditions lead to appearance variations among similar objects, leading to sparse visual features are insufficient to represent the common semantic structure of the entire class. To solve problems above, we introduce an innovative framework named ST-FSOD. Primarily, we introduce the SA-RPN module, which leverages efficient pixel association capability to generate high-quality foreground object proposals. Subsequently, through a text guiding learner module (TGL), we use textual labels of each category to generate image-agnostic text-guided prototypes. The enhanced text prototypes are fused with visual features to complement the sparse visual features. Extensive experiments conducted on the DIOR, NWPU VHR-10 and RSOD benchmarks demonstrate that the proposed method consistently surpasses strong baselines and achieves superior performance compared to previous state-of-the-art (SOTA) approaches. Our project will be open-sourced soon on https://github.com/wdcjhyy/ST-FSOD.

Few-shot segmentation (FSS) for remote sensing (RS) imagery leverages supporting information from limited annotated samples to achieve query segmentation of novel classes. Previous efforts are dedicated to mining segmentation-guiding visual cues from a constrained set of support samples. However, they still struggle to address the pronounced intra-class differences in RS images, as sparse visual cues make it challenging to establish robust class-specific representations. In this article, we propose a holistic semantic embedding (HSE) approach that effectively harnesses general semantic knowledge, i.e., class description (CD) embeddings. Instead of the naive combination of CD embeddings and visual features for segmentation decoding, we investigate embedding the general semantic knowledge during the feature extraction stage. Specifically, in HSE, a spatial dense interaction (SDI) module allows the interaction of visual support features with CD embeddings along the spatial dimension via self-attention. Furthermore, a global content modulation (GCM) module efficiently augments the global information of the target category in both support and query features, thanks to the transformative fusion of visual features and CD embeddings. These two components holistically synergize CD embeddings and visual cues, constructing a robust class-specific representation. Through extensive experiments on the standard FSS benchmark, the proposed HSE approach demonstrates superior performance compared to peer work, setting a new state-of-the-art.

Automatically monitoring building changes can assist human experts in disaster rescue, urban planning, and resource protection. Consequently, much research focuses on building change detection. Recently, the methods based on deep learning have achieved impressive performance. However, most of them ignore the effect of bitemporal image misalignment, which is prone to lead to false detection. To this end, a building change detection model with context alignment and structure perception (CASP) is proposed. First, imitating the brain logic of humans to identify changes, a bitemporal interactive alignment module (BIAM) is designed, which suppresses the spatial dislocation noise via a bidirectional reference-guided feature aggregation strategy. Building on this, a difference-induced alignment module (DIAM) is introduced to mitigate the adverse impact of misalignment errors further and improve the accuracy of building change detection. Second, a structure-aware feature fusion module is developed and integrated into the feature encoder, to enhance the discrimination of building representations and highlight the specificity of the proposed method. Extensive experiments on three representative building change detection datasets are implemented to verify the superiority of the above improvements. The quantitative and qualitative results demonstrate that the proposed method achieves competitive performance. The code is available athttps://github.com/ptdoge/CASP.

Oriented object detection in remote sensing images (RSIs) relies heavily on costly annotated data. To alleviate this challenge, we propose a straightforward yet powerful approach for semi-supervised oriented object detection, termed Adjacent Teacher. Drawing inspiration from the First Law of Geography, “Everything is related to everything else, but near things are more related than distant things”. We observe that, in the adjacent space of RSIs, there is a widespread phenomenon that objects of the same category or closely related exhibit a clustered distribution and are roughly aligned in orientation. This discovery is generalized as the Adjacent Spatial Consistency Prior (ASCP), which reflects the consistent correlation between the categories and orientations of objects in the adjacent space of RSIs. Building on the ASCP, two novel modules are introduced: Low-Confidence Pseudo-Label Mining (LPM) and Pseudo-Label Angle Correcting (PAC). LPM boosts the number of reliable pseudo-labels by exploring low-confidence pseudo-labels that conform to the ASCP. PAC improves the quality of pseudo-labels by correcting their angles to satisfy the ASCP. With these, Adjacent Teacher achieves state-of-the-art results on the DOTA-v1.5, SODA-A and FAIR1M datasets, showing reduced missed detection rates and improved bounding box accuracy. Furthermore, the proposed method seamlessly integrates with existing pseudo-label-based semi-supervised oriented object detection models, significantly enhancing their performance. The code will be available at https://github.com/Xia-tao/Adjacent-Teacher.

Remote sensing change detection (RSCD) plays a crucial role in applications such as environmental monitoring and urban planning. With the emergence of foundational vision-language models like CLIP, there is growing interest in integrating textual information into vision tasks. However, in the RSCD domain, limited efforts have been made to effectively leverage textual cues, and challenges persist in capturing differential features. To address these issues, this study proposes the Multimodal Difference Augmentation learning for remote sensing change detection model (MdaCD) that fully exploits textual information and enhances differential feature learning. MdaCD introduces a CLIP-Guided Masking process to direct textual descriptions toward image differences, and a Multimodal Fusion and Difference Augmentation process to integrate and refine differential features across modalities. The CLIP-Guided Masking process applies masking to bi-temporal image pairs before generating text prompts, enabling a more targeted analysis of changes. Meanwhile, the Multimodal Fusion and Difference Augmentation process computes a fused attention map to integrate visual and textual cues, effectively amplifying relevant differences. By applying Difference Augment functions, the differential features from both visual and textual embeddings are further refined and strengthened. The effectiveness of the proposed MdaCD model is validated through extensive experiments on two public RSCD datasets, where it achieves state-of-the-art performance with IoU scores of 84.88% on LEVIR-CD and 71.96% on SYSU-CD. The code and pretrained models of this work will be publicly available at https://github.com/haoyangofficial/MdaCD.

The rapid advancement of deep generative models (DGMs) has significantly advanced research in computer vision, providing a cost-effective alternative to acquiring vast quantities of expensive imagery. However, existing methods predominantly focus on synthesizing remote sensing (RS) images aligned with real images in a global layout view, which limits their applicability in RS image object detection (RSIOD) research. To address these challenges, we propose a multiclass and multiscale object (MMO) image generator based on DGMs, termed MMO-IG, designed to generate RS images with supervised object labels from global and local aspects simultaneously. Specifically, from the local view, MMO-IG encodes various RS instances using an iso-spacing instance map (ISIM). During the generation process, it decodes each instance region with iso-spacing value in ISIM—corresponding to both background and foreground instances—to produce RS images through the denoising process of diffusion models. Considering the complex interdependencies among MMOs, we construct a spatial-cross dependence knowledge graph (SCDKG). This ensures a realistic and reliable multidirectional distribution among MMOs for region embedding, thereby reducing the discrepancy between source and target domains. Besides, we propose a structured object distribution instruction (SODI) to guide the generation of synthesized RS image content from a global aspect with SCDKG-based ISIM together. Extensive experimental results demonstrate that our MMO-IG exhibits superior generation capabilities for RS images with dense MMO-supervised labels, and RS detectors pretrained with MMO-IG show excellent performance on real-world datasets. Code is available athttps://github.com/omtcyang/MMO-IG.

Research on remote sensing multi-view stereo has significantly advanced the development of large-scale 3D urban reconstruction. However, existing frameworks encounter challenges with blurred edge details when processing aerial image, which impedes the accuracy of depth estimation. To address these limitations, we propose RC-MVS, the deep estimation network specifically tailored for remote sensing multi-view stereo tasks. This network aims to enhance the geometric details within the view space while effectively reducing match noise, achieving high-precision depth estimation. Specifically, we introduce a refined cascade framework that integrates geometric details with semantic information, ensuring both global structural consistency and local feature expressiveness. During the feature extraction phase, we redesign the feature space construction process and introduce a denoising feature pyramid module. This module reduces feature inconsistency and employs multiple denoising strategies to purify feature representations, thereby enhancing the accuracy of the matching process. Furthermore, to achieve progressive optimization of the depth range, we propose a progressive cross-layer fusion module. This module progressively fuses low-resolution cost volumes, reducing domain shifts between different data dimensions, thereby enhancing the understanding of fine structures within the depth map and the broader context. Experimental results show that the RC-MVS model performs exceptionally well on the LuoJia-MVS and WHU datasets, achieving superior quantitative and qualitative performance.

The application of learning-based Multi-view Stereo (MVS) depth estimation methods has achieved significant results in large-scale 3D reconstruction benchmarks. However, adjacent terrains in aerial image interfere with depth estimation along building edges during matching process, leading to inaccurate results. To address these challenges, we propose a new end-to-end MVS network, named FuS-MVSNet, which fuses monocular depth probability as a semantic guidance into the multi-view geometry-based MVS framework. By combining the strengths of geometric consistency and local semantics, FuS-MVSNet achieves notable enhancements in both accuracy and robustness. Specifically, we first construct a monocular branch based on the pre-trained Depth Anything model to perform monocular metric depth estimation. The non-shared parameters ensure that the depth estimation process is independent of multi-view branch, focusing exclusively on semantic depth inference. Subsequently, to incorporate monocular features into the multi-view network, we introduce a volume adaptive fusion module, which adaptively integrates monocular feature volumes into the standard cost volume via an attention mechanism and guides the cost volume regularization. Finally, confidence-based dynamic selection between the two prediction branches ensures the selection of the more robust branch result under challenging conditions. Qualitative and quantitative results indicate that we achieve competitive performance on multiple benchmarks, including the WHU and LuoJia-MVS datasets.

Remote sensing image captioning (RSIC) aims to generate accurate and concise textual descriptions for remote sensing (RS) images. It plays a significant role in the analysis of earth observation data. The success of Vision-and-Language Pre-training (VLP) models provides the foundation for their transfer to the RSIC task. To reduce the cost of transferring VLP models to downstream tasks, numerous Parameter-Efficient Transfer Learning (PETL) techniques have been proposed. However, most of them focus on fine-tuning general-purpose foundation models without fully considering the unique characteristics of remote sensing data. In this paper, we introduce PE-RSIC, a novel PETL framework tailored for RSIC. Specifically, the framework builds on a pre-trained BLIP-2 model while further designing a lightweight Cross-modal RS adapter (CRS-Adapter) and a Class Prompt. During training, all parameters of the pre-trained model remain frozen, and the newly added CRS-Adapter modules are updated to efficiently transfer vision-and-language knowledge from the natural domain to the RS domain. The Class Prompt is obtained by projecting the vision-encoded [CLS] token into the decoder, guiding the model to generate more accurate captions. This approach enables the model to capture critical RS class features that might be lost during the query decoding process, with only a minimal increase in parameters. Extensive experiments show that our PE-RSIC framework outperforms full fine-tuning while utilizing only 5% of the trainable parameters.

SAR-to-RGB translation, which transforms Synthetic Aperture Radar (SAR) images into visually interpretable RGB counterparts, is critical for enhancing applications in visual analysis, deep learning, and multi-source data fusion. However, existing methods often fail to preserve both global structural integrity and fine-grained local textures. This deficiency stems from weak feature extraction and the lack of a robust layout framework, leading to outputs with information loss, geometric distortions, and unnatural textures. To overcome these limitations, we propose the Robust Layout-based Iterative Diffusion Model (RLI-DM), a novel three-stage framework for high-fidelity translation. The framework begins with an Optical Reconstruction Module that employs a conditional diffusion model to ensure precise spectral mapping. At its core, the Geometric Robustness Module leverages a Brownian bridge model that we train to derive a noise-resilient layout, overcoming the limitations of conventional edge detection and significantly enhancing global structural fidelity. Finally, this robust layout guides a Customized Multi-Level Refinement Module to iteratively reconstruct local textures, ensuring structural clarity and cross-feature consistency. Extensive experiments on multiple benchmark datasets demonstrate that RLI-DM achieves state-of-the-art performance, significantly outperforming existing methods in both structural integrity and perceptual quality.

Scene text reading plays a crucial role in scene understanding. As its precondition task, scene text detection has garnered increasing interest from researchers. Segmentation-based text detection methods have gained prominence due to their adaptable pixel-level predictions. Many existing methods predict the shrink mask and utilize the Vatti clipping algorithm to reconstruct text contours. However, the shrink mask only focuses on the global geometry feature and shrinks the same distance everywhere, which neglects local contour information and disrupts the instance shape feature. In addition, the post-processing based on the Vatti clipping algorithm heavily relies on the predictions and is relatively complex, causing suboptimal performance in both detection accuracy and efficiency. To address the above problems, we propose an efficient and effective method named Magnetic Text Detector (MTD), inspired by magnetism. It is constructed by a text representation method flexible mask (FM) and a magnetic pull module (MPM). Unlike the shrink mask and concentric mask, the former concerns the local contours and shrinks unfixed distances on different positions, which avoids the truncation issue while preserving distinctiveness from the text regions. The latter generates magnetic fields and pulls pole points of FM to the text contour by magnetism. This allows accurate reconstruction of text contours, even when predictions deviate from the actual text severely, while saving 50% of the post-processing time approximately. Several ablation studies verify the effectiveness of the proposed FM and MPM. Extensive experiments show that our MTD achieves state-of-the-art (SOTA) methods on multiple datasets from different scenes. The code is available at https://github.com/fengmulin/MTD.

Current object detectors often suffer performance degradation when applied to cross-domain scenarios, particularly under challenging visual conditions such as nighttime scenes. This is primarily due to the I3 problems: Inadequate sampling of instance-level features, Indistinguishable feature representation across domains and Inaccurate generation for identical category participation. To address these challenges, we propose a domain-adaptive detection framework that enables robust generalization across different visual domains without introducing any additional inference overhead. The framework comprises three key components. Specifically, the centerness-category consistency sampler alleviates inadequate sampling by selecting representative instance-level features, while the paired centerness consistency loss enforces alignment between classification and localization. Second, VLM-based orthogonality enhancement leverages frozen vision-language encoders with an orthogonal projection loss to improve cross-domain feature distinguishability. Third, hallucination feature generator synthesizes robust instance-level features for missing categories, ensuring balanced category participation across domains. Extensive experiments on multiple datasets covering various domain adaptation and generalization settings demonstrate that our method consistently outperforms state-of-the-art detectors, achieving up to 5.5 mAP improvement, with particularly strong gains in nighttime adaptation.

Texts on the intelligent transportation scene include mass information. Fully harnessing this information is one of the critical drivers for advancing intelligent transportation. Unlike the general scene, detecting text in transportation has extra demand, such as a fast inference speed, except for high accuracy. Most existing real-time text detection methods are based on the shrink mask, which loses some geometry semantic information and needs complex post-processing. In addition, the previous method usually focuses on correct output, which ignores feature correction and lacks guidance during the intermediate process. To this end, we propose an efficient multi-scene text detector that contains an effective text representation similar mask (SM) and a feature correction module (FCM). Unlike previous methods, the former aims to preserve the geometric information of the instances as much as possible. Its post-progressing saves 50% of the time, accurately and efficiently reconstructing text contours. The latter encourages false positive features to move away from the positive feature center, optimizing the predictions from the feature level. Some ablation studies demonstrate the efficiency of the SM and the effectiveness of the FCM. Moreover, the deficiency of existing traffic datasets (such as the low-quality annotation or closed source data unavailability) motivated us to collect and annotate a traffic text dataset, which introduces motion blur. In addition, to validate the scene robustness of the SM-Net, we conduct experiments on traffic, industrial, and natural scene datasets. Extensive experiments verify it achieves (SOTA) performance on several benchmarks. The code and dataset are available at:https://github.com/fengmulin/SMNet.

Due to the diversity of scene text in aspects such as font, color, shape, and size, accurately and efficiently detecting text is still a formidable challenge. Among the various detection approaches, segmentation-based approaches have emerged as prominent contenders owing to their flexible pixel-level predictions. However, these methods typically model text instances in a bottom-up manner, which is highly susceptible to noise. In addition, the prediction of pixels is isolated without introducing pixel-feature interaction, which also influences the detection performance. To alleviate these problems, we propose a multi-information level arbitrary-shaped text detector consisting of a focus entirety module (FEM) and a perceive environment module (PEM). The former extracts instance-level features and adopts a top-down scheme to model texts to reduce the influence of noises. Specifically, it assigns consistent entirety information to pixels within the same instance to improve their cohesion. In addition, it emphasizes the scale information, enabling the model to distinguish varying scale texts effectively. The latter extracts region-level information and encourages the model to focus on the distribution of positive samples in the vicinity of a pixel, which perceives environment information. It treats the kernel pixels as positive samples and helps the model differentiate text and kernel features. Extensive experiments demonstrate the FEM's ability to efficiently support the model in handling different scale texts and confirm the PEM can assist in perceiving pixels more accurately by focusing on pixel vicinities. Comparisons show the proposed model outperforms existing state-of-the-art approaches on four public datasets.

The irregular contour representation is one of the tough challenges in scene text detection. Although segmentation-based methods have achieved significant progress with the help of flexible pixel prediction, the overlap of geographically close texts hinders detecting them separately. To alleviate this problem, some shrink-based methods predict text kernels and expand them to restructure texts. However, the text kernel is an artificial object with incomplete semantic features that are prone to incorrect or missing detection. In addition, different from the general objects, the geometry features (aspect ratio, scale, and shape) of scene texts vary significantly, which makes it difficult to detect them accurately. To consider the above problems, we propose an effective spotlight text detector (STD), which consists of a spotlight calibration module (SCM) and a multivariate information extraction module (MIEM). The former concentrates efforts on the candidate kernel, like a camera focus on the target. It obtains candidate features through a mapping filter and calibrates them precisely to eliminate some false positive samples. The latter designs different shape schemes to explore multiple geometric features for scene texts. It helps extract various spatial relationships to improve the model's ability to recognize kernel regions. Ablation studies prove the effectiveness of the designed SCM and MIEM. Extensive experiments verify that our STD is superior to existing state-of-the-art methods on various datasets, including ICDAR2015, CTW1500, MSRA-TD500, and Total-Text.

Few-shot learning aims to generalize the recognizer from seen categories to an entirely novel scenario. With only a few support samples, several advanced methods initially introduce class names as prior knowledge for identifying novel classes. However, obstacles still impede achieving a comprehensive understanding of how to harness the mutual advantages of visual and textual knowledge. In this paper, we set out to fill this gap via a coherent Bidirectional Knowledge Permeation strategy called BiKop, which is grounded in human intuition: a class name description offers a moregeneralrepresentation, whereas an image captures thespecificityof individuals. BiKop primarily establishes a hierarchical joint general-specific representation through bidirectional knowledge permeation. On the other hand, considering the bias of joint representation towards the base set, we disentangle base-class-relevant semantics during training, thereby alleviating the suppression of potential novel-class-relevant information. Experiments on four challenging benchmarks demonstrate the remarkable superiority of BiKop, particularly outperforming previous methods by a substantial margin in the 1-shot setting (improving the accuracy by 7.58% onminiImageNet).

Solving the Multi-View Stereo (MVS) problem is a cornerstone in computer vision, with depth map estimation and fusion being one of the most critical approaches. The depth confidence map is pivotal in ensuring the precision and completeness of the reconstruction outcomes. These algorithms frequently encounter a trade-off between completeness and accuracy in the confidence map, which can significantly impair the final reconstruction results. This paper analyzes the causes and phenomena of these issues, namely Confidence Jitter, Confidence Gap, and Confidence Disappearance. From these insights, a multi-view stereo network named CF-MVSNet is introduced, comprising three essential components. Firstly, the method mitigates the Confidence Jitter problem through two confidence fusion strategies. Secondly, it narrows the depth sampling space to near sub-pixel levels, addressing the Confidence Gap through neighborhood-average pooling. Lastly, the algorithm tackles the Confidence Disappearance problem resulting from multi-scale classification and regression with a loss function named CL. Our proposed method demonstrates superior performance across two critical metrics: the completeness of the depth map and the accuracy of the reconstructed point cloud, outperforming current state-of-the-art MVS methods.

This paper proposes **GO4Align**, a multi-task optimization approach that tackles task imbalance by explicitly aligning the optimization across tasks. To achieve this, we design an adaptive group risk minimization strategy, comprising two techniques in implementation: (i) dynamical group assignment, which clusters similar tasks based on task interactions; (ii) risk-guided group indicators, which exploit consistent task correlations with risk information from previous iterations. Comprehensive experimental results on diverse benchmarks demonstrate our method's performance superiority with even lower computational costs.

Anchor-free object detectors have recently received increasing research attention in the field of aerial scene object detection, due to their high flexibility and practicality. Anchor-free detectors typically depend on the feature pyramid network (FPN) to alleviate the challenge of significant variations in object scales in aerial contexts. Despite establishing a multi-scale feature pyramid, existing FPN-based methods treat each aerial object as an indivisible entity solely managed by a single-scale representation. However, they fail to take into account the distinct characteristics of various components within an instance. To this end, this letter proposes a novel anchor-free detector, namely IMPR-Det, which can integrally mix multi-scale pyramid representations for different components of an instance, thus boosting the fine-grained object representation capability. Specifically, IMPR-Det fundamentally introduces a more advanced detection head with an adaptive routing mechanism for pixel-level multi-scale feature assignment, instead of previous instance-level assignment. Experimental results demonstrate the superiority of the proposed method over its counterparts, in terms of both accuracy and efficiency, for object detection in aerial images.

Remote sensing shadow removal is an open issue. Previous studies focus on working in the spatial dimension, ignoring the potential of the Fourier dimension, while illumination degradation typically exists in the amplitude component. To address this limitation, our insight is a fresh dual-stage Fourier-based network (NeFour), which explores the best of both worlds between frequency and spatial information. In the frequency stage, we investigate the positive correlation between amplitude and brightness from channel and spatial statistics. Coupled with implicitly defined normalization, a controllable fitting amplitude transform map recreates the illumination. In the spatial stage, the inverted dark channel prior with 3-D coordinates serves as modulation matrices that naturally reveal the spatial distribution of shadows, thus elegantly eliminating shadow remnants. With ingenious design, NeFour achieves nontrivial performance against state-of-the-art shadow removal methods in terms of both visual perception and quantitative evaluation. The code is publicly available athttps://github.com/chi-kaichen/NeFour.

Change detection is a prevalent technique in remote sensing image analysis for investigating geomorphological evolution. The modeling and analysis of difference features are crucial for the precise detection of land cover changes. In order to extract difference features, previous work has either directly computed them through differential operations or implicitly modeled them via feature fusion. However, these rudimentary strategies rely heavily on a high degree of congruence within the bitemporal feature space, which results in the model’s diminished capacity to capture subtle variations induced by factors such as differences in illumination. In response to this challenge, the concept of 3-D neighborhood difference convolution (3D-NDC) is proposed for robustly aggregating the intensity and gradient information of features. Furthermore, to delve into the deep disparities within bitemporal instance features, we propose a novel paradigm for differential feature extraction based on 3D-NDC, termed 3-D neighborhood cross-differencing. This strategy is dedicated to exploring the interplay of cross-temporal features, thereby unveiling the inherent disparities among various land cover characteristics. In addition, a detail-focused refinement (DfR) decode based on the Laplace operator has been designed to synergize with the 3-D neighborhood cross-differencing, aiming to improve the detail performance of change instances. This integration forms the basis of a new change detection framework, named ChangeLN. Extensive experiments demonstrate that ChangeLN significantly outperforms other state-of-the-art change detection methods. Moreover, the 3-D neighborhood cross-difference strategy exhibits the potential for integration into other change detection frameworks to improve detection performance. Open code is available fromhttps://github.com/weiAI1996/3DNCD_ChangeLN.

Infrared small target detection (IRSTD) plays a critical role in applications such as night navigation and fire rescue. Its primary purpose is to extract small targets from cluttered backgrounds. While deep learning-based methods have made great advancements in this field, there are still some limitations. One common issue is that the detected target shape tends to be smooth, and extremely small targets may not be effectively detected due to background interference. This article proposes an edge-guided perception network (EGPNet) for IRSTD to alleviate this trouble. To maintain the information of small targets, EGPNet utilizes a multiscale feature progressive fusion (MFPF) encoder to extract features. This progressive fusion manner enhances semantic information and contextual correlation. Considering that the detected target shapes may result in smoothing effect, an edge-guided image refinement module (EIRM) is incorporated to improve the integrity of the target shape. Moreover, we introduce a local target amplifier (LTA) to boost the visibility and representation of targets, while suppressing the clutter background interference. The experimental results illustrate that the proposed model can detect the targets with small and weak in different scenes well. Our code is publicly available athttps://github.com/qianngli/EGPNet.

Significant progress has been recently achieved in the field of remote sensing image (RSI) change detection based on data-driven deep learning. Fully supervised models have limitations on the availability of massive annotated training data, while semi-supervised change detection (SSCD) has garnered increasingly widespread attention. Nevertheless, existing SSCD methods do not categorize the types of remote sensing aleatoric uncertainty (RSAU), let alone investigate the impact of uncertainty on performance. To this end, we define RSAU for SSCD and introduce the progressive uncertainty-aware and uncertainty-guided framework (PUF). It consists of two crucial components to perceive and guide the RSAU in the training stage. The first component, i.e., progressive uncertainty-aware learning (PUAL), decodes and quantifies the uncertainty inherent in the samples from the weak branch. The second one, i.e., uncertainty-guided multiview learning (UML), generates multiple image pairs designed for distortion and mixing for the strong branch. UML utilizes the uncertainty values derived from PUAL to offer guidance throughout the training process, which discerns and learns discriminative features from high-quality samples. Extensive experiments are conducted on three multiclass and building change detection (CD) benchmarks, i.e., CDD, SYSU, and LEVIR-CD. Furthermore, we propose a small dataset to enhance the understanding of aleatoric uncertainty, namely, LEVIR-AU. The proposed PUF consistently achieves state-of-the-art (SOTA) performance. The dataset and codes are available athttps://github.com/shenjh0/PUF.

Shadow removal from remote sensing images is still an open issue. Recently, deep network training on unpaired data is preferable since corresponding ground truths of shadow images are not available in practice. Nevertheless, unsupervised shadow removal research for remote sensing imagery is limited by the scarcity of publicly available benchmarks. This paper proposes an unsupervised progressive network (UP-ShadowGAN) to jointly learn decoupled features for shadow removal and color transfer. UP-ShadowGAN explores the mapping between shadow and shadow-free domains through adversarial learning and cycle consistency constraint. In particular, we employ progressive learning to decompose the overall mapping process into more manageable shadow removal and color transfer steps. Specifically, the realistic illumination is restored by propagating spatial context between shadow and shadow-free nodes. Coupled with a multi-color space aggregation strategy, diverse color space representations alleviate color deviation caused by spatial inconsistency. More importantly, we contribute the first unpaired remote sensing shadow removal dataset (URSSR), which encourages future exploration. Extensive experiments demonstrate that UP-ShadowGAN competes favorably with state-of-the-art methods. The dataset and code are available at https://github.com/chi-kaichen/UP-ShadowGAN.

The objective of Semantic Change Detection (SCD) is to discern intricate changes in land cover while simultaneously identifying their semantic categories. Prior research has shown that using multiple independent branches for the distinct tasks of change localization and semantic recognition is a reliable approach to solving the SCD problem. Nevertheless, conventional SCD architectures rely heavily on a high degree of consistency within the bi-temporal feature space when modeling difference features, inevitably resulting in false positives or missed alerts within change areas. In this paper, we introduce a SCD framework called the Cross-Differential Semantic Consistency (CdSC) network. CdSC is designed to mine deep discrepancies in bi-temporal instance features while preserving their semantic consistency. Specifically, the 3D-Cross-Difference module, incorporating 3D convolutions, explores the interaction of cross-temporal features, revealing inherent differences among various land features. Simultaneously, deep semantic representations are further utilized to enhance the local correlation of difference information, thereby improving the model’s discriminative capabilities within change regions. Incorporating principles from contrastive learning, a Semantic Co-Alignment loss is introduced to increase intra-class consistency and inter-class distinctiveness of dual-temporal semantic features, thereby addressing the challenges posed by semantic disparities. Extensive experiments on two SCD datasets demonstrate that CdSC outperforms other state-of-the-art SCD methods significantly in both qualitative and quantitative evaluations. The code and dataset are available at https://github.com/weiAI1996/CdSC.

Image captioning is a fundamental vision-language task with wide-ranging applications in daily life. The existing methods often struggle to accurately interpret the semantic information in remote sensing images due to the complexity of backgrounds. Target region masks can effectively reflect the shape characteristics of targets and their potential interrelationships. Therefore, incorporating and fully integrating these features can significantly improve the quality of generated captions. However, researchers are hindered by the lack of relevant datasets that contain corresponding object masks. It is natural to ask the following: how to efficiently introduce and utilize object masks? In this article, we provide potential target masks for the publicly available remote sensing image caption (RSIC) datasets, enabling models to utilize the regional features of targets for RSIC. Meanwhile, a novel RSIC algorithm is proposed that combines regional positional features with fine-grained semantic information, abbreviated as$\text {S}^{2}$CPNet. To effectively capture the semantic information from image and position relationship from mask, respectively, the semantic and spatial feature enhancement submodules are introduced at the ends of encoder branches, respectively. Furthermore, the cross-view feature fusion module is designed to integrate regional features and semantic information efficiently. Then, a target recognition decoder is developed to enhance the ability of model to identify and describe critical targets in images. Finally, we improve the caption generation decoder by adaptively merging textual information with visual features to generate more accurate descriptions. Our model achieves satisfactory results on three RSIC datasets compared with the existing method. The related datasets and code will be open-sourced inhttps://github.com/CVer-Yang/SSCPNet.

Remote sensing image captioning aims to describe the crucial objects from remote sensing images in the form of natural language. The inefficient utilization of object texture and semantic features in images, along with the ineffective cross-modal alignment between image and text features, are the primary factors that impact the model to generate high-quality captions. To alleviate this trouble, this paper presents a network for remote sensing image captioning, namely HCNet, including hierarchical feature aggregation and cross-modal feature alignment. Specifically, a hierarchical feature aggregation module is proposed to obtain a comprehensive representation of vision features, which is beneficial for producing accurate descriptions. Considering the disparities between different modal features, we design a cross-modal feature interaction module in the decoder to facilitate feature alignment. It can fully utilize cross-modal features to localize critical objects. Besides, a cross-modal feature align loss is introduced to realize the alignment between image and text features. Extensive experiments show our HCNet can achieve satisfactory performance. Especially, we demonstrate significant performance improvements of +14.15% CIDEr score on NWPU datasets compared to existing approaches. The source code is publicly available at https://github.com/CVer-Yang/HCNet.

Road extraction from remote sensing images (RSIs) holds significant application value in various aspects of daily scenarios. However, it is still challenging to extract high-quality road results from RSIs due to the interference of objects sharing similar structures with roads in the background and the occlusion caused by surroundings. To alleviate these problems, a road extraction network based on the global-local Context perception and Cross spatial-scale feature interaction is proposed ($\text {C}^{2}$Net). First, a global-local context perception module (GLCPM) is incorporated to capture the overall topology features of the road, which aims to improve the ability of the model to discriminate between roads and similar objects. Then, the cross spatial-scale feature interaction module is designed in the skip connection to effectively aggregate full-scale features without loss of feature information, which can provide rich and accurate road structural features for the decoder. Experiments conducted on public road datasets demonstrate that$\text {C}^{2}$Net outperforms existing methods in terms of comprehensive metrics such as intersection over union (IoU) and the$F1$-score. The results indicate that$\text {C}^{2}$Net can produce road results with superior connectivity and quality. The source code will be publicly available athttps://github.com/CVer-Yang/CCNet.

Object detection plays a crucial role in scene understanding and has extensive practical applications. In the field of remote sensing object detection, both detection accuracy and robustness are of significant concern. Existing methods heavily rely on sophisticated adversarial training strategies that tend to improve robustness at the expense of accuracy. However, detection robustness is not always indicative of improved accuracy. Therefore, in this paper, we research how to enhance robustness, while still preserving high accuracy, or even improve both simultaneously, with simple vanilla adversarial training or even in the absence thereof. In pursuit of a solution, we first conduct an exploratory investigation by shifting our attention from adversarial training, referred to as adversarial fine-tuning, to adversarial pretraining. Specifically, we propose a novel pretraining paradigm, namely structured adversarial self-supervised (SASS) pretraining, to strengthen both clean accuracy and adversarial robustness for object detection in remote sensing images. At a high level, SASS pretraining aims to unify adversarial learning and self-supervised learning into pretraining and encode structured knowledge into pretrained representations for powerful transferability to downstream detection. Moreover, to fully explore the inherent robustness of vision Transformers and facilitate their pretraining efficiency, by leveraging the recent masked image modeling (MIM) as the pretext task, we further instantiate SASS pretraining into a concise end-to-end framework, named structured adversarial MIM (SA-MIM). SA-MIM consists of two pivotal components, structured adversarial attack and structured MIM (S-MIM). The former establishes structured adversaries for the context of adversarial pretraining, while the latter introduces a structured local-sampling global-masking strategy to adapt to hierarchical encoder architectures. Comprehensive experiments on three different datasets have demonstrated the significant superiority of the proposed pretraining paradigm over previous counterparts for remote sensing object detection. More importantly, regardless of with or without adversarial fine-tuning, it enables simultaneous improvements on detection accuracy and robustness as expected, promisingly alleviating the dependence on complicated adversarial fine-tuning.

Binary object change detection aims to monitor the evolution of the object of interest in a fixed region. Constructing a relevant dataset for deep learning models is strenuous. In the existing datasets, there is usually an imbalance between changed and unchanged samples, as well as a restricted diversity within the changed samples. Aiming at that, some methods utilize unpaired images used for object segmentation to generate pseudo-bitemporal images for change detection. However, due to the existence of the domain gap between different data sources, the model obtained by these methods can not well generalize to the real bitemporal images. Inspired by them but to avoid the domain difference, we explore how to directly use the unpaired images within a real change detection dataset to complement changed samples. In detail, a concise metric-based framework is designed, which consists of two branches, a projector and a predictor. The framework obtains the change map by computing the distance between the bitemporal embedding outputted by the projector. Meanwhile, instructed by an indirect semantic supervision module (ISSM) specially designed, the predictor can generate the semantic confidence map distinguishing the pixels in an image into two categories. Based on the output of the framework, an unpaired image prototype contrast module (UIPCM) is proposed. It enriches the diversity of the change samples for training by combining the prototypes in unpaired images at the feature level, leading to alleviating the imbalance between changed and unchanged samples. Besides, a dual margin contrastive loss (DMCL) is adopted during training. It can reduce the constraint on the consistency of bitemporal embedding in unchanged regions. The benefits and the superiority of the proposed method are demonstrated on two well-recognized datasets. The code is available at https://github.com/ptdoge/UIPC.

Learnable multiview stereo (MVS) aerial image depth estimation has obtained great success in 3-D digital urban reconstruction. Currently, most depth estimation methods in the large-scale sense heavily involve adapting the general MVS framework. However, these methods often overlook the cross-view interval and limited viewpoint inherent in aerial images data. In this article, we introduce an learning-based MVS method for aerial image depth estimation, which enhances visual consistency to address the insufficient accuracy caused by the characteristics of aerial image data, namely, AggrMVS. First, an optical flow-guided feature extraction module is introduced to map the dynamic relationship between reference and source images. It explicitly captures edge information of different depth components to guide the cost volume regularization. Second, a cross-view volume fusion module is proposed to enhance the interaction among reference volumes, further improving the aggregation ability of the source volume. Furthermore, AggrMVS achieves refined aerial image depth estimation results with a lightweight cascade architecture. Since low-altitude oblique aerial datasets currently lack, we reconstruct a multicategory synthetic aerial scene benchmark from general MVS datasets. The benchmark dataset is available athttps://github.com/ToscW/BlendedUAV. Experiments on public and proposed datasets confirm that AggrMVS outperforms other MVS depth estimation methods in terms of qualitative and quantitative aspects.

Remote sensing (RS) image change captioning is a visual semantic understanding task that has received increasing attention. The change captioning methods are required to understand the visual information of the images and capture the most significant difference between them, then describe it in natural language. Most existing methods mainly focus on improving the difference feature encoder or language decoder, while ignoring the visual feature extractor. The current feature extractors suffer from several issues, including 1) domain gap between pre-training on single temporal natural images and downstream bi-temporal RS task, 2) limited difference feature modeling in the implicit single-stream network, and 3) high computational costs caused by extracting features for each temporal phase image under the dual-stream extractor. To address these issues, we propose a Single-stream Extractor Network (SEN). It consists of a single-stream extractor pre-trained on bi-temporal RS images using contrastive learning to mitigate the domain gap and high computational cost. Additionally, to improve feature modeling for difference information, we propose a shallow feature embedding (SFE) module and a cross attention guided difference (CAGD) module, which enhance the representation of temporal features and extract the difference features explicitly. Extensive experiments and visualizations demonstrate the effectiveness and advanced performance of SEN. The code and model weights are available at https://github.com/mrazhou/SEN.

With the increasing availability of cameras in vehicles, obtaining license plate (LP) information via on-board cameras has become feasible in traffic scenarios. LPs play a pivotal role in vehicle identification, making automatic LP detection (ALPD) a crucial area within traffic analysis. Recent advancements in deep learning have spurred a surge of studies in ALPD. However, the computational limitations of on-board devices hinder the performance of real-time ALPD systems for moving vehicles. Therefore, we propose a real-time frame-by-frame LP detector focusing on real-time accurate LP detection. Specifically, video frames are categorized into keyframes and non-keyframes. Keyframes are processed by a deeper network (high-level stream), while non-keyframes are handled by a lightweight network (low-level stream), significantly enhancing efficiency. To achieve accurate detection, we design a knowledge distillation strategy to boost the performance of low-level stream and a feature propagation method to introduce the temporal clues in video LP detection. Our contributions are: (1) A real-time frame-by-frame LP detector for video LP detection is proposed, achieving a competitive performance with popular one-stage LP detectors. (2) A simple feature-based knowledge distillation strategy is introduced to improve the low-level stream performance. (3) A spatial-temporal attention feature propagation method is designed to refine the features from non-keyframes guided by the memory features from keyframes, leveraging the inherent temporal correlation in videos. The ablation studies show the effectiveness of knowledge distillation strategy and feature propagation method.

As a unique identity of vehicle, License Plate (LP) facilitates the intelligent transportation in many fields, such as traffic enforcement, intelligent transportation dispatching, etc. Recently, the LP detectors are trained by supervised learning which is directly guided by manual annotations and lacks the use of visual knowledge in image content, limiting the further development of detection performance. Inspired by the contrast and comparison in perception of human beings, a contrastive learning method is introduced into license plate detection task and we propose an end-to-end Contrastive License Plate Detector (CLPD). In CLPD, a special contrastive triad for contrastive learning is designed which aims to decouple the foregrounds and backgrounds. Based on this triad, a contrastive learning branch is introduced into the license plate detection pipeline to prompt the feature expression ability of backbone and extracting more discriminative features for detection. This contrastive learning branch is jointly trained with supervised learning branch for detection and it is only used in training, keeping the efficiency in inference. The experiment results show that the proposed CLPD improves the detection accuracy compared to baselines and other license plate detectors significantly on three datasets. The ablation studies further explore the potential of CLPD. In addition, the proposed CLPD has generalization to improve the performance on different baselines. And the visualization results in latent space verify our proposed CLPD aggregates features tightly and extracts discriminative features effectively.

Most existing traffic sign-related works are dedicated to detecting and recognizing part of traffic signs separately, which fails to analyze the global semantic logic among signs and may convey inaccurate traffic instruction information. Following the above issues, we propose a traffic sign interpretation (TSI) task, which aims to interpret global semantic interrelated traffic signs (e.g., driving instruction-related texts, symbols, and guide panels) into a natural language for providing complete traffic instruction support to autonomous or assistant driving. Meanwhile, considering the lack of an effective framework for the proposed TSI task in existing works, we design a multi-task learning architecture (TSI-arch) to detect and recognize various traffic signs with drastic changes in sizes and aspect ratios. Meanwhile interpreting these signs into a natural language like a human according to Chinese design criteria of road traffic signs. Furthermore, the absence of a public TSI available dataset prompts us to build a traffic sign interpretation dataset, namely TSI-CN. The dataset consists of real road scene images, which are captured from the highway and the urban way in China from a driver’s perspective. It contains rich location labels of texts, symbols, and guide panels, and the corresponding natural language description labels. Experiments on our TSI-CN dataset demonstrate that the TSI task is achievable and the TSI architecture can interpret traffic signs from scenes successfully even if there is a complicated semantic logic among signs.

Motion deblurring is an important topic in the field of image enhancement, which has widespread applications including video surveillance, object detection, etc. Many algorithms are designed for motion deblurring and achieve remarkable performance. However, mainstream motion blur datasets are collected under normal weather and illuminance conditions, i.e., normal domain, ignoring their variations. As a result, current methods perform poorly in dynamic real-world scenes. To address these issues, we study the work in two aspects. First, we collect the real-world motion blur dataset with a well-designed collection device from various angles, focal lengths, and street scenes. Considering its domain is single, it is augmented via a Domain Transfer Strategy (DTS) to construct a Multi-Domain dataset (MD dataset), expanding the domains of the collected dataset. Second, we propose a Multi-Domain Adaptive Deblur Network (MDADNet) with two modules. The one is the Domain Adaptation (DA) module that exploits domain invariant features to stabilize the performance of the MDADNet in multiple domains. The other is the Meta Deblurring (MDB) module that employs the auxiliary branch to enhance the deblurring ability. It also enables the MDADNet to update parameters during the testing stage, improving the generalizations of the MDADNet. Extensive experimental results demonstrate that the MD-trained methods significantly strengthen the motion deblurring ability in multiple domains. Particularly, the proposed MDADNet achieves state-of-the-art performance on the MD dataset and public motion blur datasets.

In the past few years, visual tracking methods with convolution neural networks (CNNs) have gained great popularity and success. However, the convolution operation of CNNs struggles to relate spatially distant information, which limits the discriminative power of trackers. Very recently, several Transformer-assisted tracking approaches have emerged to alleviate the above issue by combining CNNs with Transformers to enhance the feature representation. In contrast to the methods mentioned above, this article explores a pure Transformer-based model with a novel semi-Siamese architecture. Both the time-space self-attention module used to construct the feature extraction backbone and the cross-attention discriminator used to estimate the response map solely leverage attention without convolution. Inspired by the recent vision transformers (ViTs), we propose the multistage alternating time-space Transformers (ATSTs) to learn robust feature representation. Specifically, temporal and spatial tokens at each stage are alternately extracted and encoded by separate Transformers. Subsequently, a cross-attention discriminator is proposed to directly generate response maps of the search region without additional prediction heads or correlation filters. Experimental results show that our ATST-based model attains favorable results against state-of-the-art convolutional trackers. Moreover, it shows comparable performance with recent "CNN + Transformer" trackers on various benchmarks while our ATST requires significantly less training data.

To pursue comprehensive performance, recent text detectors improve detection speed at the expense of accuracy. They adopt shrink-mask-based text representation strategies, which leads to a high dependence of detection accuracy on shrink-masks. Unfortunately, three disadvantages cause unreliable shrink-masks. Specifically, these methods try to strengthen the discrimination of shrink-masks from the background by semantic information. However, the feature defocusing phenomenon that coarse layers are optimized by fine-grained objectives limits the extraction of semantic features. Meanwhile, since both shrink-masks and the margins belong to texts, the detail loss phenomenon that the margins are ignored hinders the distinguishment of shrink-masks from the margins, which causes ambiguous shrink-mask edges. Moreover, false-positive samples enjoy similar visual features with shrink-masks. They aggravate the decline of shrink-masks recognition. To avoid the above problems, we propose a zoom text detector (ZTD) inspired by the zoom process of the camera. Specifically, zoomed-out view module (ZOM) is introduced to provide coarse-grained optimization objectives for coarse layers to avoid feature defocusing. Meanwhile, zoomed-in view module (ZIM) is presented to enhance the margins recognition to prevent detail loss. Furthermore, sequential-visual discriminator (SVD) is designed to suppress false-positive samples by sequential and visual features. Experiments verify the superior comprehensive performance of ZTD.

Due to the domain gap between the public large-scale datasets and actual scenes, the crowd counting models trained on the common datasets have a significant performance degradation when applying in practical applications. To address the above issue, one of the solution is to label additional data from the novel scenes, which is time-consuming and impractical for multiple scenes. Another solution is to utilize domain adaptation approaches to adapt a well-trained model to novel scenes. However, most of these approaches focus on appearance adaptation while the background and the crowd distribution is not adapted. In this paper, we propose a weakly-supervised method with real-synthetic hybrid data which only requires a small portion of unlabelled real images and auto-generated synthetic labelled images for training. First, the hybrid data is generated based on background from the real scene and random distributed synthetic persons. Second, an initialized counter is trained based on the hybrid data and the crowd distribution is predicted based on the predictions on real images. Then, a better crowd counter is trained based on new hybrid data generated from updated crowd distribution. The process is iterated until convergence. Extensive experiments demonstrate the effectiveness of the proposed method.

Recently, segmentation-based text detection methods develop rapidly, which achieve competitive accuracy and detection speed. However, these methods are hard to fit text instances accurately, which leads to the decrease of model performance. Meanwhile, the poor perception of the text center by the boundary pixels further affects the detection accuracy. We follow the issues and design an efficient framework for arbitrary-shaped text detection, which is constructed based on Optimal Kernel Representation (OKR) and Pixel Enhancement Module (PEM). Specifically, OKR is proposed to fit texts with optimal kernels. It erodes texts according to the corresponding geometric characteristics, which is simpler and more accurate compared with previous methods. PEM is used to enhance the perception of boundary pixels to the virtual character centers of text, thus improving the cohesion of the whole instance. Particularly, PEM only participates in the training process, which brings no extra computation costs to inference. Ablation experiments show the effectiveness of OKR and PEM. Comparisons on serveral benchmarks verify that our efficient detector is superior to the existing state-of-the-art (SOTA) methods.

We present the RSSOD-Bench dataset for salient object detection (SOD) in optical remote sensing imagery. While SOD has achieved success in natural scene images with deep learning, research in SOD for remote sensing imagery (RSSOD) is still in its early stages. Existing RSSOD datasets have limitations in terms of scale, and scene categories, which make them misaligned with real-world applications. To address these shortcomings, we construct the RSSOD-Bench dataset, which contains images from four different cities in the USA1. The dataset provides annotations for various salient object categories, such as buildings, lakes, rivers, highways, bridges, aircraft, ships, athletic fields, and more. The salient objects in RSSOD-Bench exhibit large-scale variations, cluttered backgrounds, and different seasons. Unlike existing datasets, RSSOD-Bench offers uniform distribution across scene categories. We benchmark 23 different state-of-the-art approaches from both the computer vision and remote sensing communities. Experimental results demonstrate that more research efforts are required for the RSSOD task.

Recently, graph neural networks (GNNs) have drawn a surge of investigations in deep graph clustering. Nevertheless, existing approaches predominantly are inclined to semantic-agnostic since GNNs exhibit inherent limitations in capturing global underlying semantic structures. Meanwhile, multiple objectives are imposed within one latent space, whereas representations from different granularities may presumably conflict with each other, yielding severe performance degradation for clustering. To this end, we propose a novel Multi-Level Graph Contrastive Prototypical Clustering (MLG-CPC) framework for end-to-end clustering. Specifically, a Prototype Discrimination (ProDisc) objective function is proposed to explicitly capture semantic information via cluster assignments. Moreover, to alleviate the issue of objectives conflict, we introduce to perceive representations of different granularities within individual feature-, prototypical-, and cluster-level spaces by the feature decorrelation, prototype contrast, and cluster space consistency respectively. Extensive experiments on four benchmarks demonstrate the superiority of the proposed MLG-CPC against the state-of-the-art graph clustering approaches.

This paper focuses on the data-insufficiency problem in multi-task learning within an episodic training setup. Specifically, we explore the potential of heterogeneous information across tasks and meta-knowledge among episodes to effectively tackle each task with limited data. Existing meta-learning methods often fail to take advantage of crucial heterogeneous information in a single episode, while multi-task learning models neglect reusing experience from earlier episodes. To address the problem of insufficient data, we develop Heterogeneous Neural Processes (HNPs) for the episodic multi-task setup. Within the framework of hierarchical Bayes, HNPs effectively capitalize on prior experiences as meta-knowledge and capture task-relatedness among heterogeneous tasks, mitigating data-insufficiency. Meanwhile, transformer-structured inference modules are designed to enable efficient inferences toward meta-knowledge and task-relatedness. In this way, HNPs can learn more powerful functional priors for adapting to novel heterogeneous tasks in each meta-test episode. Experimental results show the superior performance of the proposed HNPs over typical baselines, and ablation studies verify the effectiveness of the designed inference modules.

Recently, fully convolutional networks (FCNs) have contributed significantly to salient object detection in optical remote sensing images (RSIs). However, owing to the limited receptive fields of FCNs, accurate and integral detection of salient objects in RSIs with complex edges and irregular topology is still challenging. Moreover, suffering from the low contrast and complicated background of RSIs, existing models often occur ambiguous or uncertain recognition. To remedy the above problems, we propose a novel hybrid modeling approach, i.e., uncertainty-aware graph reasoning with global collaborative learning (UG2L) framework. Specifically, we propose a graph reasoning pipeline to model the intricate relations among RSI patches instead of pixels, and introduce an efficient graph reasoning block (GRB) to build graph representations. On top of it, a global context block (GCB) with a linear attention mechanism is proposed to explore the multiscale and global context collaboratively. Finally, we design a simple yet effective uncertainty-aware loss (UAL) to enhance the model’s reliability for better prediction of saliency or non-saliency. Experimental and visual results on three datasets show the superiority of the proposed UG2L. Code is available at https://github.com/lyf0801/UG2L.

Building change detection aims to identify the change in buildings in the same geographic area. Recently, many methods based on deep learning (DL) have achieved encouraging performance. However, some challenges remain in effectively exploiting the temporal–spatial correlation and achieving good discrimination in the neighborhood of the edge. To relieve these issues, we develop a selective attention module (SAM) to model the relationship between the semantic and the state (i.e., unchanged or changed) of the pixel, which is integrated into an existing metric learning-based architecture. Moreover, inspired by recent advances in contrastive learning, we present a novel edge neighborhood contrastive learning method to force the network to learn discriminative and compact features, leading to improving the accuracy of building change detection. Experimental results demonstrate that our method achieves competitive performance in terms of objective metrics and visual comparisons.

In the past few decades, license plate detection and recognition (LPDR) systems have made great strides relying on Convolutional Neural Networks (CNN). However, these methods are evaluated on small and non-representative datasets that perform poorly in complex natural scenes. Besides, most of existing license plate datasets are based on a single image, while the information source in the actual application of license plates is frequently based on video. The mainstream algorithms also ignore the dynamic clue between consecutive frames in the video, which makes the LPDR system have a lot of room for improvement. In order to solve these problems, this paper constructs a large-scale video-based license plate dataset named LSV-LP, which consists of 1,402 videos, 401,347 frames and 364,607 annotated license plates. Compared with other data sets, LSV-LP has stronger diversity, and at the same time, it has multiple sources due to different collection methods. There may be multiple license plates in a frame, which is more in line with complex natural scenes. Based on the proposed dataset, we further design a new framework that explores the information between adjacent frames, called MFLPR-Net. In addition to these, we release the annotation tools for license plates or vehicles in videos. By evaluating the performance of MFLPR-Net and some mainstream methods, it is proved that the proposed model is superior to other LPDR systems.In order to be more intuitive, we put some samples on https://drive.google.com/file/d/1udqRddpJZMpTdHHQdwZRll6vaYALUiql/view?usp=sharingGoogle Drive. The whole dataset is available at https://github.com/Forest-art/LSV-LP.

Scene Text Detection (STD) has applied in many fields successfully. One of the important applications of STD is License Plate Detection (LPD). As a unique identity of vehicle, License Plate (LP) facilitates the intelligent transportation in many fields, such as traffic enforcement, intelligent transportation dispatching, etc. However, there are many scene texts similar to LPs causing misjudgment of LP detector. To alleviate these disturbances, more discriminative features are necessary. In latent feature space, discriminative features should aggregate into a tight cluster to widen decision boundary. We assume three perspectives about how to aggregate features and boost feature expression. From these assumptions, a special contrastive triad is designed. Then, we propose a Self-Constrained Contrastive Aggregation (SCCA) method to lead the feature aggregation in latent space and boost the feature expression of backbone. The proposed SCCA is jointly trained with supervised learning for detection to improve the detection performance. The experiments show that our proposed SCCA prompts the baseline significantly and exceeds recent LP detectors, reaching 99.7 on both F1-score and AP on UFPR-ALPR dataset. Meanwhile, we compare the self-constrained contrastive learning with vanilla contrastive learning in experiments and visualize their LP features. The results show that our proposed SCCA reaches better performance and verifies our assumptions are reasonable.

Remote sensing image captioning (RSIC), which describes a remote sensing image with a semantically related sentence, has been a cross-modal challenge between computer vision and natural language processing. For visual features extracted from remote sensing images, global features provide the complete and comprehensive visual relevance of all the words of a sentence simultaneously, while local features can emphasize the discrimination of these words individually. Therefore, not only global features are important for caption generation but also local features are meaningful for making the words more discriminative. In order to make full use of the advantages of both global and local features, in this article, we propose an attention-based global-local captioning model (GLCM) to obtain global-local visual feature representation for RSIC. Based on the proposed GLCM, the correlation of all the generated words and the relation of each separate word and the most related local visual features can be visualized in a similarity-based manner, which provides more interpretability for RSIC. In the extensive experiments, our method achieves comparable results in UCM-captions and superior results in Sydney-captions and RSICD which is the largest RSIC dataset.

Haze superimposes a veil over remote sensing images, which severely limits the extraction of valuable military information. To this end, we present a novel trinity model to restore realistic surface information by integrating the merits of both prior-based and deep learning-based strategies. Concretely, the critical insight of our Trinity-Net is to investigate how to incorporate prior information into CNNs and Swin Transformer for reasonable estimation of haze parameters. Then, haze-free images are obtained by reconstructing the remote sensing image formation model. Although Swin Transformer has shown tremendous potential in the dehazing task, which typically results in ambiguous details. We devise a gradient guidance module that naturally inherits structure priors of gradient maps, guiding the deep model to generate visually pleasing details. In light of the generality of image formation parameters, we successfully promote Trinity-Net to natural image dehazing and underwater image enhancement tasks. Notably, the acquisition of large-scale remote sensing hazy images and natural hazy images in military scenes is not feasible in practice. To bridge this gap, we construct aRemote Sensing Image Dehazing Benchmark(RSID) and aNatural Image Dehazing Benchmark(NID), including 1000 real-world hazy images with corresponding ground truth images, respectively. To our knowledge, this is the first exploration to develop dehazing benchmarks in the military field, alleviating the dilemma of data scarcity. Extensive experiments on three vision tasks illustrate the superiority of our Trinity-Net against multiple state-of-the-art methods. The datasets and code are available at https://github.com/chi-kaichen/Trinity-Net.

Road connectivity is vital in road extraction for accurate vehicle navigation. However, the segmentation-based methods fail to model the connectivity resulting in broken road segments. Therefore, we propose a Location-Guided Network (LGNet) for promoting connectivity performance in a very effective and efficient way. Specifically, an auxiliary Road Location Prediction (RLP) task is designed to obtain global road connectivity information, which improves the performance of road segmentation. The RLP can predict the location coordinates of the whole roads with row anchors and column anchors. By aggregating the global location context to the segmentation branch with a location-guided decoder (LG-Decoder), the features can finally capture the connectivity of each road segment. Overall, LGNet has the following advantages: 1) The proposed RLP and LCG can plug into any encoder-decoder network and achieve an impressive performance. 2) High computational efficiency. In comparison with the multi-branch method, our proposed LGNet requires about 6× fewer GFLOPs. 3) The superior road connectivity performance. A series of experiments are conducted on two road extraction data sets (SpaceNet and DeepGlobe), confirming the effectiveness of the LGNet.

Few-shot remote sensing scene classification is of high practical value in real situations where data are scarce and annotated costly. The few-shot learner needs to identify new categories with limited examples, and the core issue of this assignment is how to prompt the model to learn transferable knowledge from a large-scale base dataset. Although current approaches based on transfer learning or meta-learning have achieved significant performance on this task, there are still two problems to be addressed: (i) as an essential characteristic of remote sensing images, spatial rotation insensitivity surprisingly remains largely unexplored; (ii) the high distribution uncertainty of hard samples reduces the discriminative power of the model decision boundary. Stimulated by these, we propose a corresponding end-to-end framework termed a Hard Sample Learning (HSL) and Multi-view Integration (MI) Network (HSL-MINet). First, the MI module contains a pretext task introduced to guide the knowledge transfer, and a multiview-attention mechanism used to extract correlational information across different rotation views of images. Second, aiming at increasing the discrimination of the model decision boundary, the HSL module is designed to evaluate and select hard samples via a class-wise adaptive threshold strategy, and then decrease the uncertainty of their feature distributions by a devised triplet loss. Extensive evaluations on NWPU-RESISC45, WHU-RS19, and UCM datasets show that the effectiveness of our HSL-MINet surpasses the former state-of-the-art approaches.

Few-shot segmentation endeavors to utilize a minimal amount of annotated samples (support) to guide the segmentation of unseen objects (query). Previous techniques primarily employ asupport-to-queryparadigm, neglecting to sufficiently leverage the mutual representation between query and support images, which leaves models suffering from intra-class variations and background interference in remote sensing images. This paper proposes a Holistic Mutual Representation Enhancement (HMRE) method to bridge these gaps. First, a Dual Activation (DA) module is devised to establish information symmetry between the two branches and forms the foundation for mutual representation enhancement. Subsequently, the holistic mutual enhancement is jointly constructed by the Global Semantic (GS) and Spatial Dense (SD) mutual enhancement modules. In the prediction stage for segmentation, we integrate the enhanced mutual representation into the Mutual-Fusion Decoder to activate the homologous object regions bidirectionally. To expedite the replication of investigation in this task, we further create a corresponding benchmark Flood-3i. The whole dataset is attainable at https://drive.google.com/drive/folders/1FMAKf2sszoFKjq0UrUmSLnJDbwQSpfxR. Extensive experiments on two benchmarks iSAID-5i and Flood-3i demonstrate the superiority of our proposed method, which also sets a new state-of-the-art.

Super-resolution (SR) is one of the powerful techniques to improve image quality for low-resolution (LR) hyperspectral image (HSI) with insufficient detail and noise. Traditional methods typically perform simple cascade or addition during the fusion of the auxiliary high-resolution RGB and LR HSI. As a result, the abundant HR RGB details are not utilized as a priori information to enhance the HSI feature representation, leaving room for further improvements. To address this issue, we propose an RGB-induced feature modulation network for HSI SR (IFMSR). Considering that similar patterns are common in images, a multi-corresponding patch aggregation is designed to globally assemble this contextual information, which is beneficial for feature learning. Besides, to adequately exploit plentiful HR RGB details, an RGB-induced detail enhancement (RDE) module and a deep cross-modality feature modulation (CFM) module are proposed to transfer the supplementary materials from RGB to HSI. These modules can provide a more direct and instructive representation, leading to further edge recovery. Experiments on several datasets demonstrate that our approach achieves comparable performance under more realistic degradation condition. Our code is publicly available at https://github.com/qianngli/IFMSR.

Hyperspectral image super-resolution (SR) using auxiliary RGB image has obtained great success. Currently, most methods respectively train single model to handle different scale factors, which may lead to the inconsistency of spatial and spectral contents when converted to the same size. In fact, the manner ignores the exploration of potential interdependence among different scale factors in single model. To this end, we propose a multi-scale factor joint learning for hyperspectral image super-resolution (MulSR). Specifically, to take advantage of the inherent priors of spatial and spectral information, a deep architecture using single scale factor is designed by terms of symmetrical guided encoder (SGE) to explore the hyperspectral image and RGB image. Considering that there are obvious differences in texture details at various scale factors, another architecture is proposed which is basically the same as above, except that its scale factor is larger. On this basis, a multi-scale information interaction (MII) unit is modeled between two architectures by a direction-aware spatial context aggregation (DSCA) module. Besides, the contents generated by the model with multi-scale factor are combined to build a learnable feedback compensation correction (LFCC). The difference is fed back to the architecture with large scale factor, forming an interactive feedback joint optimization pattern. This calibrates the representation of spatial and spectral contents in the reconstruction process. Experiments on synthetic and real datasets demonstrate that our MulSR shows superior performance in terms of qualitative and quantitative aspects. Our code is publicly available at https://github.com/qianngli/MulSR.

Current state-of-the-art remote sensing salient object detectors always require high-resolution spatial context to ensure excellent performance, which incurs enormous computation costs and hinders real-time efficiency. In this work, we propose a universal super-resolution assisted learning (SRAL) framework to boost performance and accelerate the inference efficiency of existing approaches. To this end, we propose to reduce the spatial resolution of the input remote sensing images (RSIs), which is model-agnostic, and can be applied to existing algorithms without extra computation cost. Specifically, a transposed saliency detection decoder (TSDD) is designed to upsample interim features progressively. On top of it, an auxiliary super-resolution decoder (ASRD) is proposed to build a multitask learning (MTL) framework to investigate an efficient complementary paradigm of saliency detection and super-resolution. Furthermore, a novel task-fusion guidance module (TFGM) is proposed to effectively distill domain knowledge from the super-resolution auxiliary task to the salient object detection task in optical RSIs. The presented ASRD and TFGM can be omitted in the inference phase without any extra computational budget. Extensive experiments on three datasets show that the presented SRAL with 224×224 input is superior to more than 20 algorithms. Moreover, it can be successfully generalized to existing typical networks with significant accuracy improvements in a parameter-free manner. Codes and models are available at https://github.com/lyf0801/SRAL.

Existing remote sensing image salient object detection (RSI-SOD) methods widely perform object-level semantic understanding with pixel-level supervision, but ignore the image-level scene information. As a fundamental attribute of RSIs, the scene has a complex intrinsic correlation with salient objects, which may bring hints to improve saliency detection performance. However, existing RSI-SOD datasets lack both pixel- and image-level labels, and it is non-trivial to effectively transfer the scene domain knowledge for more accurate saliency localization. To address these challenges, we first annotate the image-level scene labels of three RSI-SOD datasets inspired by remote sensing scene classification. On top of it, we present a novel scene-guided dual-stream network (SDNet), which can perform cross-task knowledge distillation from the scene classification to facilitate accurate saliency detection. Specifically, a scene knowledge transfer module (SKTM) and a conditional dynamic guidance module (CDGM) are designed for extracting saliency key area as spatial attention from the scene subnet and guiding the saliency subnet to generate scene-enhanced saliency features, respectively. Finally, an object contour awareness module (OCAM) is introduced to enable the model to focus more on irregular spatial details of salient objects from the complicated background. Extensive experiments reveal that our SDNet outperforms over 20 state-of-the-art algorithms on three datasets. Moreover, we prove that the proposed framework is model-agnostic, and its extension to six baselines can bring significant performance benefits. Code will be available at https://github.com/lyf0801/SDNet.

Deep learning-based object detection has recently played a vital role in both computer vision and Earth observation communities. However, the performance of modern object detectors is highly limited by the quantity and quality of manually labeled training samples. Furthermore, compared to object detection in natural scenes, Remote Sensing Object Detection (RSOD) faces two specific critical challenges. 1) Densely arranged instances: geospatial objects tend to be densely packed in remote sensing scenarios. 2) Large variations in object scale: the wide field of the bird’s eye view leads to dramatic variations in object scale across various categories. The above issues bring significant difficulties to attaining manual annotations for deep learning-based RSOD. To this end, in this paper, we turn our attention from fully-supervised RSOD to semi-supervised RSOD, and propose a novel framework based on the teacher-student paradigm, namely Prospective Consistent Teacher (PCT), which includes three crucial components,i.e., Weighted Dense-Proposal Learning (WDPL), Mean-Consistency-based Proposal Pruning (MCPP), and EM-based Fitting Policy (EFP). Specifically, WDPL re-weights the dense proposals with box confidences, while MCPP ranks the student proposals with consistency analysis to select discriminative and consistent boxes. EFP can automatically set thresholds for pseudo labels and improve the consistent information of the teacher network. Extensive experimental results on two challenging public datasets,i.e., DOTA and DIOR, have demonstrated the reduced reliance of our proposed method on large amounts of labeled data for the task of RSOD.

Change detection is a critical task in remote sensing to monitor the state of the surface on Earth. This field has been dominated by deep learning-based methods recently. Many models that model the temporal-spatial correlation in bitemporal images through the non-local interaction between bitemporal features achieve impressive performance. However, under complex scenes including multiple change types or weakly discriminate objects, they suffer from achieving discriminative fusion of information due to the weak semantic discrimination of the bitemporal representations. Aiming at this problem, a difference-guided aggregation network (DGANet) is proposed, where two key modules are injected, i.e., a difference-guided aggregation module (DGAM) and a weighted metric module (WMM). The bitemporal features in DGAM are aggregated with the guidance of their differences, which focuses on their change relevance and relaxes their semantic distinction. Therefore, the fused features are change-relevant and discriminative. WMM aims to achieve adaptive distance computation between the bitemporal features by dynamic feature attention in different dimensions. It is helpful to suppress the pseudo-changes. Besides, a change magnitude contrastive loss (CMCL) is introduced to employ the dependency of bitemporal pixels in different bitemporal images, which further enhances the representation quality of the model. Meanwhile, it is further extended in this work. The effectiveness of the three improvements is demonstrated by extensive ablation studies. The results on three datasets widely used illustrate that our method achieves satisfactory performance.

Object detection in remote-sensing images is a crucial task in the fields of Earth observation and computer vision. Despite impressive progress in modern remote-sensing object detectors, there are still three challenges to overcome: 1) complex background interference; 2) dense and cluttered arrangement of instances; and 3) large-scale variations. These challenges lead to two key deficiencies, namely, coarse features and coarse samples, which limit the performance of existing object detectors. To address these issues, in this article, a novel coarse-to-fine framework (CoF-Net) is proposed for object detection in remote-sensing imagery. CoF-Net mainly consists of two parallel branches, namely, coarse-to-fine feature adaptation (CoF-FA) and coarse-to-fine sample assignment (CoF-SA), which aim to progressively enhance feature representation and select stronger training samples, respectively. Specifically, CoF-FA smoothly refines the original coarse features into multispectral nonlocal fine features with discriminative spatial–spectral details and semantic relations. Meanwhile, CoF-SA dynamically considers samples from coarse to fine by progressively introducing geometric and classification constraints for sample assignment during training. Comprehensive experiments on three public datasets demonstrate the effectiveness and superiority of the proposed method.

Object detection is a fundamental task in remote sensing image analysis and scene understanding. Previous remote sensing object detectors are typically based on convolutional neural networks (CNNs), whose performance is significantly limited by the intrinsic locality of convolution operations. The emergence of vision Transformers brings potential solutions to this problem, which have the capability to be a solid alternative to CNNs. However, three crucial obstacles hinder the application and performance of Transformers in the task of remote sensing object detection, i.e., 1) high computational complexity, especially for high-resolution remote sensing images, 2) training-and sample-inefficiency caused by lack of inductive bias, and 3) difficulty in learning arbitrary orientation knowledge of geospatial objects. To address these issues, in this paper, a novel efficient inductive vision Transformer framework is proposed for oriented object detection in remote sensing imagery. This framework follows the hierarchical feature pyramid structure and makes threefold contributions, as follows. 1) Spatial redundancy in remote sensing images is fully explored and an adaptive multi-grained routing mechanism is proposed to facilitate token sparsity in Transformers, which can dramatically reduce the computational cost without comprising the accuracy. 2) A compact dual-path encoding architecture, where both global long-range dependencies and local semantic relations are jointly and complementarily captured, is proposed to enhance inductive bias in Transformers. 3) An angle tokenization technique is proposed to promote the encoding, embedding, and learning of direction knowledge for oriented objects in remote sensing scenarios. In this work, the above three contributions are instantiated in an advanced Transformer-based object detector, namely EIA-PVT. Comprehensive experiments on two publicly available datasets have demonstrated its effectiveness and superiority for oriented object detection in remote sensing images.

Crowd localization is to predict each instance head position in crowd scenarios. Since the distance of pedestrians being to the camera are variant, there exists tremendous gaps among scales of instances within an image, which is called the intrinsic scale shift. The core reason of intrinsic scale shift being one of the most essential issues in crowd localization is that it is ubiquitous in crowd scenes and makes scale distribution chaotic. To this end, the paper concentrates on access to tackle the chaos of the scale distribution incurred by intrinsic scale shift. We propose Gaussian Mixture Scope (GMS) to regularize the chaotic scale distribution. Concretely, the GMS utilizes a Gaussian mixture distribution to adapt to scale distribution and decouples the mixture model into sub-normal distributions to regularize the chaos within the sub-distributions. Then, an alignment is introduced to regularize the chaos among sub-distributions. However, despite that GMS is effective in regularizing the data distribution, it amounts to dislodging the hard samples in training set, which incurs overfitting. We assert that it is blamed on the block of transferring the latent knowledge exploited by GMS from data to model. Therefore, a Scoped Teacher playing a role of bridge in knowledge transform is proposed. What' s more, the consistency regularization is also introduced to implement knowledge transform. To that effect, the further constraints are deployed on Scoped Teacher to derive feature consistence between teacher and student end. With proposed GMS and Scoped Teacher implemented on four mainstream datasets of crowd localization, the extensive experiments demonstrate the superiority of our work. Moreover, comparing with existing crowd locators, our work achieves state-of-the-art via F1-measure comprehensively on four datasets.

Existing real-time text detectors reconstruct text contours by shrink-masks only. Though they simplify the framework and can make the model run fast, the strong dependence on shrink-masks leads to unreliable detection results (e.g., miss detection and overdetection). Moreover, these methods ignore the information from surrounding pixels, which causes sensitive shrink-masks and accelerates the reliability decline of detection results. Considering the above problems, we construct an effective and efficient text detection network, termed as Reinforcement Shrink-Mask for Text Detection (RSMTD), which strengthens the model's ability to recognize texts while enjoying a high detection speed. Specifically, an effective text representation strategy (Reinforcement Shrink-Mask, RSM) is designed to decouple texts and shrink-masks. RSM builds texts through shrink-masks and reinforcement offsets to ensure stable detection results encountering shrink-masks that deviate from the ground-truth. It is worth noting that reinforcement offsets can force our method to focus on the foreground shapes to bring precise shrink-mask edges. For the robustness improvement of shrink-masks, Super-pixel Window (SPW) is proposed to encourage RSMTD to utilize the surroundings of each pixel to predict shrink-masks. Particularly, SPW treats the interval regions between texts and shrink-masks as background, which helps to suppress interval regions and to avoid text adhesion. Moreover, a lightweight feature merging branch is constructed to further accelerate the inference process. As demonstrated in the experiments, our method is superior to existing state-of-the-art (SOTA) methods in both detection accuracy and speed on multiple benchmarks.

Irregular-shaped texts bring challenges to Scene Text Detection (STD). Although existing regression-based approaches achieve comparable performances, they fail to cover some highly curved ribbon-like text lines. Inspired by morphology, we found that the leaf vein can easily cover various geometries. Specifically, lateral and thin veins are emitted to margin along main vein gradually with the leaf growth. This process can decompose a concave object into consecutive convex regions, which are easier to fit. Hence, the leaf vein is suitable for representing highly curved texts. Considering the aforementioned advantage, we design a leaf vein-based text representation method (LVT), where text contour is treated as leaf margin and represented through main, lateral, and thin veins. We further construct a detection framework based on LVT, namely LeafText. In the text reconstruction stage, LeafText simulates the leaf growth process to rebuild text contours. It grows main veins in Cartesian coordinates to locate texts roughly at first. Then, lateral and thin veins are generated along the main vein growth direction in polar coordinates. They are responsible for generating the coarse contour and refining it, respectively. Meanwhile, Multi-Oriented Smoother (MOS) is designed to smooth the main vein for ensuring reliable growth directions of lateral and thin veins. Additionally, a global incentive loss is proposed to enhance the predictions of lateral and thin veins. Ablation experiments demonstrate LVT can fit irregular-shaped texts precisely and verify the effectiveness of MOS and global incentive loss. Comparisons show that LeafText is superior to existing state-of-the-art (SOTA) methods on MSRA-TD500, CTW1500, Total-Text, and ICDAR2015 datasets.

Recently, crowd counting using supervised learning achieves a remarkable improvement. Nevertheless, most counters rely on a large amount of manually labeled data. With the release of synthetic crowd data, a potential alternative is transferring knowledge from them to real data without any manual label. However, there is no method to effectively suppress domain gaps and output elaborate density maps during the transferring. To remedy the above problems, this article proposes a domain-adaptive crowd counting (DACC) framework, which consists of a high-quality image translation and density map reconstruction. To be specific, the former focuses on translating synthetic data to realistic images, which prompts the translation quality by segregating domain-shared/independent features and designing content-aware consistency loss. The latter aims at generating pseudo labels on real scenes to improve the prediction quality. Next, we retrain a final counter using these pseudo labels. Adaptation experiments on six real-world datasets demonstrate that the proposed method outperforms the state-of-the-art methods.

Pedestrian counting is a fundamental tool for under-standing pedestrian patterns and crowd flow analysis. Existing works (e.g., image-level pedestrian counting, cross-line crowd counting et al.) either only focus on the image-level counting or are constrained to the manual annotation of lines. In this work, we propose to conduct the pedes-trian counting from a new perspective - Video Individual Counting (VIC), which counts the total number of individual pedestrians in the given video (a person is only counted once). Instead of relying on the Multiple Object Tracking (MOT) techniques, we propose to solve the problem by decomposing all pedestrians into the initial pedestrians who existed in the first frame and the new pedestrians with separate identities in each following frame. Then, an end-to-end Decomposition and Reasoning Network (DRNet) is designed to predict the initial pedestrian count with the density estimation method and reason the new pedestrian's count of each frame with the differentiable optimal transport. Extensive experiments are conducted on two datasets with congested pedestrians and diverse scenes, demonstrating the effectiveness of our method over baselines with great superiority in counting the individual pedestrians. Code: https://github.com/taohan10200/DRNet.

The popular convolutional neural networks (CNNs) have been successfully used in very high-resolution remote sensing image semantic segmentation. However, these networks often suffer from performance limitations. First, although deeper networks usually provide better feature representation, they may cause parameter redundancy and the inefficient use of prior knowledge. Secondly, attention-based networks often only focus on weighting different features of a single sample but ignore the correlation of all samples in training set, thus leading to the loss of global information. To address above issues, we propose two simple yet effective global evolution strategies. The first is knowledge enhancement. This strategy can reactivate invalid convolutional kernels through convergence of different models and make full use of prior knowledge from the network to improve its feature representation. The second is a dict-attention module that greatly enhances the generalization of networks by learning and inferring the global relationship among different samples through the dictionary unit. As a result, a novel global evolution network (GENet) is designed based on knowledge enhancement and dict-attention for remote sensing image semantic segmentation. Experiments demonstrate that the proposed GENet is not only superior to popular networks in segmentation accuracy.

Geospatial object detection is an essential task in remote sensing community. One-stage methods based on deep learning have faster running speed but cannot reach higher detection accuracy than two-stage methods. In this paper, to achieve excellent speed/accuracy trade-off for geospatial object detection, a single-shot balanced detector is presented. First, a balanced feature pyramid network (BFPN) is designed, which can balance semantic information and spatial information between high-level and shallow-level features adaptively. Second, we propose a task-interactive head (TIH). It can reduce the task misalignment between classification and regression. Extensive experiments show that the improved detector obtains significant detection accuracy with considerable speed on two benchmark datasets.

Detecting irregular-shaped text instances is the main challenge for text detection. Existing approaches can be roughly divided into top-down and bottom-up perspective methods. The former encodes text contours into unified units, which always fails to fit highly curved text contours. The latter represents text instances by a number of local units, where the complicated network and post-processing lead to slow detection speed. In this paper, to detect arbitrary-shaped text instances with high detection accuracy and speed simultaneously, we propose a Bidirectional Perspective strategy based Network (BiP-Net). Specifically, a new text representation strategy is proposed to represent text contours from a topdown perspective, which can fit highly curved text contours effectively. Moreover, a contour connecting (CC) algorithm is proposed to avoid the information loss of text contours by rebuilding interval contours from a bottom-up perspective. The experimental results on MSRA-TD500, CTW1500, and ICDAR2015 datasets demonstrate the superiority of BiP-Net against several state-of-the-art methods.

In recent years, deep convolutional neural networks have achieved amazing results on multiple tasks. However, these complex network models often require significant computation resources and energy costs, so that they are difficult to deploy to power-constrained devices, such as IoT systems, mobile phones, embedded devices, etc. Aforementioned challenges can be overcome through model compression like network pruning. In this paper, we propose an adaptive channel pruning module (ACPM) to automatically adjust the pruning rate with respect to each channel, which is more efficient to prune redundant channel parameters, as well as more robust to datasets and backbones. With one-shot pruning strategy design, the model compression time can be saved significantly. Extensive experiments demonstrate that ACPM makes tremendous improvement on both pruning rate and accuracy, and also achieves the state-of-the-art results on a series of different networks and benchmarks.

The distortion of high-frequency information is the most fundamental problem of dynamic scene blur, which leads to the degradation of image quality. However, most deep-based methods fail to show satisfactory results because of ignoring the importance of image structural information (high-frequency and low-frequency preception) in deblurring. In this paper, we propose a high-frequency and low-frequency information fusion deblurring network (HLFNet) that uses edge perception as a guide. The proposed HLFNet consists of the high-frequency information network (HFNet) and the low-frequency information network (LFNet). Besides, we adopt the proposed multi-scale atrous convolution (MSA) block into LFNet, which can effectively reduce the number of model parameters while expanding the receptive fields. Extensive experiments show that the proposed model can achieve state-of-the-art results with smaller parameters and shorter inference time on the public datasets.

Crowd understanding has widespread applications, including video surveillance, crowd monitoring. Unlike existing coarse-grained crowd understanding methods(e.g., counting people in images), crowd instance segmentation can provide more precise results (pixel-wise segmentation for each person in images). However, crowd instance segmentation demands a considerable amount of pixel-wise labeled data, which is very time-consuming and challenging to annotate accurate human instance masks in the crowd scene. In this paper, we propose a data generator and labeler to automatically generate synthetic crowd instance segmentation data. Then based on it, we build a large-scale synthetic crowd instance segmentation dataset called "GCIS Dataset". Besides, we demonstrate two approaches that utilize the synthetic GCIS dataset to advance the performance of crowd instance segmentation: 1)supervised crowd instance segmentation: pretrain crowd instance segmentation models on GCIS dataset, then finetune on other real data. It can remarkably boost the model’s real-world performance; 2) crowd instance segmentation via domain adaption: transfer the synthetic GCIS dataset to photo-realistic images, then train the model together with transformed data and real data, which shows better performance when tested on real-world data. Extensive experiments show the validity of the synthetic GCIS dataset for crowd instance segmentation. The dataset and source code will be released online.

Motion deblurring is challenging due to the fast movements of the object or the camera itself. Existing methods usually try to liberate it by training CNN model or Generative Adversarial Networks(GAN). However, their methods can’t restore the details very well. In this paper, a Deblurring Transformer based on Generative Adversarial Network(DTransGAN) is proposed to improve the deblurring performance of the vehicles under the surveillance camera scene. The proposed DTransGAN combines the low-level information and the high-level information through skip connection, which saves the original information of the image as much as possible to restore the details. Besides, we replace the convolution layer in the generator with the swin transformer block, which could pay more attention to the reconstruction of details. Finally, we create the vehicle motion blur dataset. It contains two parts, namely the clear image and the corresponding blurry image. Experiments on public datasets and the collected dataset report that DTransGAN achieves the state-of-the-art for motion deblurring task.

Anomaly detection (AD) on hyperspectral images has been widely researched in recent decades due to its high practicability and wide range of application scenarios. Such AD methods derived from low-rank matrix decomposition (LRMD) have appeared rapidly and been applied effectively. However, most of them focused on the use of spectral information and neglected the abundant spatial characteristics. In this letter, a spectral-spatial total variation (TV) (SSTV) regularized low-rank matrix decomposition method with a Schatten 1/2 quasi-norm ($S_{1/2}$) and denoising is proposed. First, to exploit the hyperspectral imagery (HSI) characteristics from the spectral perspective, we propose the low-rank matrix decomposition method with$S_{1/2}$norm and image denoising modules. Second, we incorporate the SSTV regularization by employing a 2-D TV (TV) spatially and 1-D TV along the spectral dimension to realize the maximized utilization of spatial characteristics of HSI. Finally, the alternating direction multiplier method (ADMM) is brought in the calculating process to attain the consequent detection results. The superiority of the proposed method has been demonstrated by the excellent performance on three real datasets.

Anomaly detection has been drawing a great deal of attention by virtue of its practicability among the hyperspectral research area. Low-rank representation (LRR) has been widely employed to detect anomalies from hyperspectral imagery (HSI) effectively while a great number of methods derived from LRR replace rank function with a nuclear norm, which gives rise to a certain amount of error. In this letter, we propose a Schatten 1/2 quasi-norm ($S_{1/2}$) regularized LRR (SRLRR) method with an improved algorithm of establishing the dictionary for hyperspectral anomaly detection. First,$S_{1/2}$regularization is proposed to substitute the initial nuclear norm to approximate the rank function. Second, an improved dictionary construction algorithm based on K-Means++ clustering is presented to integrate the model and improve the performance. Finally, the optimization algorithm through alternating direction multiplier method (ADMM) incorporating a half threshold operator is introduced to attain the eventual results. Our method has been testified on three typical data sets and demonstrates the eminent performance.

Hyperspectral images can provide spectral characteristics related to the physical properties of different materials, which arouses great interest in many fields. Band selection (BS) could effectively solve the problem of high dimensions and redundant information of HSI data. However, most BS methods utilize a single measurement criterion to evaluate band importance so that the assessment of bands is not comprehensive. To dispose of these issues, we propose the hypergraph spectral clustering band selection (HSCBS) method in this letter. First, a novel hypergraph construction method is proposed to combine bands selected by different priority criteria. Second, based on the hypergraph Laplacian matrix, an unsupervised band selection model named HSCBS is presented to cluster the bands into compact clusters with high within-class similarity and low between-class similarity. The results of comprehensive experimental on two public real datasets demonstrate the effectiveness of HSCBS.

As a significant and fundamental task in the remote sensing field, object detection has received increasing attention and research studies. However, geospatial object detection is still a challenge owing to the dramatic variation in object scales, intraclass differences, and interclass similarity from multiscale and multiclass objects. To deal with these problems, an end-to-end feature-reflowing pyramid network (FRPNet) is proposed in this letter. FRPNet has two advantages that contribute to improve object detection accuracy. First, we embed a nonlocal block into the backbone in order to get the relevancy between different regions of the geospatial image for obtaining discriminative features. Furthermore, a feature-reflowing pyramid structure is proposed to generate high-quality feature presentation for each scale through fusing fine-grained features from the adjacent lower level, which improves the detection capability for multiscale and multiclass objects. Experiments on a public remote sensing data set DIOR illustrate that FRPNet can significantly improve the performance when compared to several state-of-the-art detection approaches in terms of mean average precision (mAP).

Unlike object detection in natural images that usually achieved great success, remote sensing imagery has its own challenges to detect and localize multiclass objects, such as large-scale change, uncertain direction, and high density. The context information of the objects is very worthwhile for solving these challenges in remote sensing images. In this letter, we propose a context-driven detection network (CDD-Net) to improve the accuracy of multiclass object detection in remote sensing images. For capturing the local neighboring objects and features, a local context feature network (LCFN) is proposed to learn the local context of the region of interest. Meanwhile, a hybrid attention pyramid network (HAPN) is designed, which can steer the focus to more valuable features. The HAPN inserts a squeeze and excitation block (SEB) and three asymmetric convolution blocks (ACBs) in the feature pyramid network (FPN). The experimental results over the DOTA-v1.5 data set demonstrate that the proposed CDD-Net yields promising results.

Compared with general optical images, remote sensing images (RSIs) capture large areas from high altitudes with a bird’s eye view, which is responsible for the many categories and scale variations of objects in the images, as well as the abundant scene information. Although the complexity of the RSIs presents a significant challenge to the object detection task, the complexity presents opportunities as well. The RSIs contain plenty of object-related context information, which is valuable for boosting the object detection performance. To address the existing issue of poor context utilization in RSIs, we propose a hierarchical context embedding network (HCENet) in this letter. First, we construct a semantic feature pyramid, in which the semantic context aggregation module (SFAM) integrates the semantic contexts included in the adjacent layers of features with a novel feature fusion mechanism. Furthermore, the scene-level context embedding module (SLCEM) extracts the scene context of the overall image by a simple design and is utilized to guide feature classification. Finally, we outperform the popular object detectors on the publicly available DOTA-v1.5 dataset, achieving superior performance.

Accurate object detection in remote sensing images is an essential part of automatic extraction, analysis, and understanding of image information, which potentially plays a significant role in a number of practical applications. However, the scale diversity in remote sensing images presents a substantial challenge for object detection, regarded as one of the crucial problems to be solved. To extract multiscale feature representations and sufficiently exploit semantic context information, this letter proposes a semantic context-aware network (SCANet) model for multiscale object detection. We propose two novel modules, called receptive field-enhancement module (RFEM) and semantic context fusion module (SCFM), to enhance the performance of SCANet. The RFEM dedicates to more robust multiscale feature extraction by paying attention to distinct receptive fields through multibranch different convolutions. For the purpose of utilizing the semantic context information contained in the scene to guide the network to better detection accuracy, the SCFM integrates the semantic context features from the upper level with the lower level features and delivers them hierarchically. Experiments demonstrate that, compared with the state-of-the-art approaches, the SCANet yields superior detection results on the DOTA-v1.5 data set.

Binarized neural networks (BNNs) have drawn significant attention in recent years, owing to great potential in reducing computation and storage consumption. While it is attractive, traditional BNNs usually suffer from slow convergence speed and dramatical accuracy-degradation on large-scale classification datasets. To minimize the gap between BNNs and deep neural networks (DNNs), we propose a new framework of designing BNNs, dubbed Hyper-BinaryNet, from the aspect of enhanced information-flow. Our contributions are threefold: 1) Considering the capacity-limitation in the backward pass, we propose an 1-bit convolution module named HyperConv. By exploiting the capacity of auxiliary neural networks, BNNs gain better performance on large-scale image classification task. 2) Considering the slow convergence speed in BNNs, we rethink the gradient accumulation mechanism and propose a hyper accumulation technique. By accumulating gradients in multiple variables rather than one as before, the gradient paths for each weight increase, which escapes BNNs from the gradient bottleneck problem during training. 3) Considering the ill-posed optimization problem, a novel gradient estimation warmup strategy, dubbed STE-Warmup, is developed. This strategy prevents BNNs from the unstable optimization process by progressively transferring neural networks from 32-bit to 1-bit. We conduct evaluations with variant architectures on three public datasets: CIFAR-10/100 and ImageNet. Compared with state-of-the-art BNNs, Hyper-BinaryNet shows faster convergence speed and outperforms existing BNNs by a large margin.

Data clustering, which is to partition the given data into different groups, has attracted much attention. Recently various effective algorithms have been developed to tackle the task. Among these methods, non-negative matrix factorization (NMF) has been demonstrated to be a powerful tool. However, there are still some problems. First, the standard NMF is sensitive to noises and outliers. Although$\ell _{2,1}$norm based NMF improves the robustness, it is still affected easily by large noises. Second, for most graph regularized NMF, the performance highly depends on the initial similarity graph. Third, many graph-based NMF models perform the graph construction and matrix factorization in two separated steps. Thus the learned graph structure may not be optimal. To overcome the above drawbacks, we propose a robust bi-stochastic graph regularized matrix factorization (RBSMF) framework for data clustering. Specifically, we present a general loss function, which is more robust than the commonly used$L_2$and$L_1$functions. Besides, instead of keeping the graph fixed, we learn an adaptive similarity graph. Furthermore, the graph updating and matrix factorization are processed simultaneously, which can make the learned graph more appropriate for clustering. Extensive experiments have shown the proposed RBSMF outperforms other state-of-the-art methods.

Detecting moving objects from the video sequences has been treated as a challenging computer vision task, since the problems of dynamic background, multi-scale moving objects and various noise interference impact the corresponding feasibility and efficiency. In this paper, a novel spatio-temporal online matrix factorization (STOMF) method is proposed to detect multi-scale moving objects under dynamic background. To accommodate a wide range of the real noise distractions, we apply a specific mixture of exponential power (MoEP) distributions to the framework of low-rank matrix factorization (LRMF). For the optimization of solution algorithm, a temporal difference motion prior (TDMP) model is proposed, which estimates the motion matrix and calculates the weight matrix. Moreover, a partial spatial motion information (PSMI) post-processing method is further designed to implement multi-scale objects extraction in varieties of complex dynamic scenes, which utilizes partial background and motion information. The superiority of the STOMF method is validated by massive experiments on practical datasets, as compared with state-of-the-art moving objects detection approaches.

Visual tracking is a very critical issue in computer vision and video processing. For Discriminative Correlation Filter (DCF)-based tracking methods, it is very essential and meaningful to adaptively incorporate reliable target and surrounding information from video frames. However, most existing DCF-based trackers solely rely on pre-defined and fixed constraints such as a binary mask or quadratic function-based regularization to improve the discrimination. Unfortunately, such attempts fail to adjust the constraints according to the change of tracking circumstance in the video sequence, and thus lead to the lack of reliability of learned filters. To mitigate these problems, we present a novel DCF-based tracking method that introduces an adaptive target-and-surrounding soft mask (ATSM) into the learning formula. The adaptive soft mask that is represented by float numbers contains the detail information for both target region and its surrounding information: first, for the background area, it introduces meaningful background information and suppressing uninformative one; second, for the target area inside the bounding box, it helps to focus on the reliable area and repress the rapidly changing area; third, the target-and-surrounding soft mask is adaptively adjusted based on the variations of the target and its surrounding during the tracking process. By jointly modeling the filter and the adaptive soft mask, our ATSM tracker achieves an efficient integration of meaningful information of both foreground and background and performs favorably against state-of-the-art algorithms on seven well-known benchmarks.

Linear discriminant analysis (LDA) is a well-known technique for supervised dimensionality reduction and has been extensively applied in many real-world applications. LDA assumes that the samples are Gaussian distributed, and the local data distribution is consistent with the global distribution. However, real-world data seldom satisfy this assumption. To handle the data with complex distributions, some methods emphasize the local geometrical structure and perform discriminant analysis between neighbors. But the neighboring relationship tends to be affected by the noise in the input space. In this research, we propose a new supervised dimensionality reduction method, namely, locality adaptive discriminant analysis (LADA). In order to directly process the data with matrix representation, such as images, the 2-D LADA (2DLADA) is also developed. The proposed methods have the following salient properties: 1) they find the principle projection directions without imposing any assumption on the data distribution; 2) they explore the data relationship in the desired subspace, which contains less noise; and 3) they find the local data relationship automatically without the efforts for tuning parameters. The performance of dimensionality reduction shows the superiorities of the proposed methods over the state of the art.

The location information of road and lane lines is the supremely important thing for the automatic drive and auxiliary drive. The detection accuracy of these two elements dramatically affects the reliability and practicality of the whole system. In real applications, the traffic scene can be very complicated, which makes it particularly challenging to obtain the precise location of road and lane lines. Commonly used deep learning-based object detection models perform pretty well on the lane line and road detection tasks, but they still encounter false detection and missing detection frequently. Besides, existing convolution neural network (CNN) structures only pay attention to the information flow between layers, while it cannot fully utilize the spatial information inside the layers. To address those problems, we propose an attention-based spatial segmentation network for traffic scene understanding. We use the convolutional attention module to improve the network's understanding capacity of spatial location distribution. Spatial CNN (SCNN) obtains through the information flow within one single convolutional layer and improves the spatial relationship modeling ability of the network. The experimental results demonstrate that this method effectively improves the neural network's application ability of the spatial information, thereby improving the effect of traffic scene understanding. Furthermore, a pixel-level road segmentation dataset called NWPU Road Dataset is built to help improve the process of traffic scene understanding.

In this article, we focus on the unsupervised multiview feature selection, which tries to handle high-dimensional data in the field of multiview learning. Although some graph-based methods have achieved satisfactory performance, they ignore the underlying data structure across different views. Besides, their predefined Laplacian graphs are sensitive to the noises in the original data space and fail to obtain the optimal neighbor assignment. To address the above problems, we propose a novel unsupervised multiview feature selection model based on graph learning, and the contributions are three-fold: 1) during the feature selection procedure, the consensus similarity graph shared by different views is learned. Therefore, the proposed model can reveal the data relationship from the feature subset; 2) a reasonable rank constraint is added to optimize the similarity matrix to obtain more accurate information; and 3) an autoweighted framework is presented to assign view weights adaptively, and an effective alternative iterative algorithm is proposed to optimize the problem. Experiments on various datasets demonstrate the superiority of the proposed method compared to the state-of-the-art methods.

Graph-based clustering aims to partition the data according to a similarity graph, which has shown impressive performance on various kinds of tasks. The quality of similarity graph largely determines the clustering results, but it is difficult to produce a high-quality one, especially when data contain noises and outliers. To solve this problem, we propose a robust rank constrained sparse learning (RRCSL) method in this article. The$L_{2,1}$-norm is adopted into the objective function of sparse representation to learn the optimal graph with robustness. To preserve the data structure, we construct an initial graph and search the graph within its neighborhood. By incorporating a rank constraint, the learned graph can be directly used as the cluster indicator, and the final results are obtained without additional postprocessing. In addition, the proposed method cannot only be applied to single-view clustering but also extended to multiview clustering. Plenty of experiments on synthetic and real-world datasets have demonstrated the superiority and robustness of the proposed framework.

Recently, crowd counting draws much attention on account of its significant meaning in congestion control, public safety, and ecological surveys. Although the performance is improved dramatically due to the development of deep learning, the scales of these networks also become larger and more complex. Moreover, a large model also entails more time to train for better performance. To tackle these problems, this article first constructs a lightweight model, which is composed of an image feature encoder and a simple but effective decoder, called the pixel shuffle decoder (PSD). PSD ends with a pixel shuffle operator, which can display more density information without increasing the number of convolutional layers. Second, a density-aware curriculum learning (DCL) training strategy is designed to fully tap the potential of crowd counting models. DCL gives each predicted pixel a weight to determine its predicting difficulty and provides guidance on obtaining better generalization. Experimental results exhibit that PSD can achieve outstanding performance on most mainstream datasets while training under the DCL training framework. Besides, we also conduct some experiments about adopting DCL on existing typical crowd counters, and the results show that they all obtain new better performance than before, which further validates the effectiveness of our method.

Small aerial object detection plays an important role in numerous computer vision tasks, including remote sensing, early warning systems, and visual tracking. Despite existing moving object detection techniques that can achieve reasonable results in normal size objects, they fail to distinguish the small objects from the dynamic background. To cope with this issue, a novel method is proposed for accurate small aerial object detection under different situations. Initially, the block segmentation is introduced for reducing frame information redundancy. Meanwhile, a random projection feature (RPF) is proposed for characterizing blocks into feature vectors. Subsequently, a moving direction estimation based on feature vectors is presented to measure the motions of blocks and filter out the major directions. Finally, variable search region clustering (VSRC), together with the color feature difference, is designed for extracting pixelwise targets from the remaining moving direction blocks. The comprehensive experiments demonstrate that our approach outperforms the level of state-of-the-art methods upon the integrity of small aerial objects, especially on the dynamic background and scale variation targets.

Fuzzy c-means (FCM) algorithms with spatial information have been widely applied in the field of image segmentation. However, most of them suffer from two challenges. One is that the introduction of fixed or adaptive single neighboring information with narrow receptive field limits contextual constraints leading to clutter segmentations. The other is that the incorporation of superpixels with wide receptive field enlarges spatial coherency leading to block effects. To address these challenges, we propose fuzzy STUDENT’S t-distribution model based on richer spatial combination (FRSC) for image segmentation. In this article, we make two significant contributions. The first is that both the narrow and wide receptive fields are integrated into the objective function of FRSC, which is convenient to mine image features and distinguish local difference. The second is that the rich spatial combination under STUDENT’S t-distribution ensures that spatial information is introduced into the updated parameters of FRSC, which is helpful in finding a balance between the noise-immunity and detail-preservation. Experimental results on synthetic and publicly available images further demonstrate that the proposed FRSC addresses successfully the limitations of FCM algorithms with spatial information, and provides better segmentation results than state-of-the-art clustering algorithms.

The popular Siamese convolutional neural networks (CNNs) for remote sensing (RS) image change detection (CD) often suffer from two problems. First, they either ignore the original information of bitemporal images or insufficiently utilize the difference information between bitemporal images, which leads to the low tightness of the changed objects. Second, Siamese CNNs always employ dual-branch encoders for CD, which increases computational cost. To address the above issues, this article proposes a network based on difference enhancement and spatial–spectral nonlocal (DESSN) for CD in very-high-resolution (VHR) images. This article makes threefold contributions. First, we design a difference enhancement (DE) module that can effectively learn the difference representation between foreground and background to reduce the impact of irrelevant changes on the detection results. Second, we present a spatial–spectral nonlocal (SSN) module that is different from vanilla nonlocal because multiscale spatial global features are incorporated to model the large-scale variation of objects during CD. The module can be used to strengthen the edge integrity and internal tightness of changed objects. Third, the asymmetric double convolution with Ghost (ADCG) module is exploited instead of standard convolution. The ADCG can not only refine the edge information of the changed objects, since horizontal and vertical convolutional kernels have good contour preservation advantages, but also greatly reduce the computational complexity of the proposed model. The experiments on two public VHR CD datasets demonstrate that the proposed network can provide higher detection accuracy and requires smaller memory usage than state-of-the-art networks.

Single hyperspectral image (HSI) super-resolution (SR) methods using a auxiliary high-resolution (HR) RGB image have achieved great progress recently. However, most existing methods aggregate the information of RGB image and HSI early during input or shallow feature extraction, whose difference between two images has not been treated and discussed. Although a few methods combine both the image features in the middle layer of the network, they fail to make full use of the two inherent properties, i.e., rich spectra of HSI and HR content of RGB image, to guide model representation learning. To address these issues, in this article, we propose a dual-stage learning approach for HSI SR to learn a general spatial–spectral prior and image-specific details, respectively. In the coarse stage, we fully take advantage of two adjacent bands and RGB image to build the model. During coarse SR, a symmetrical feature propagation approach is developed to learn the inherent content of each image over a relatively long range. The symmetrical structure encourages the two streams to better retain their particularity. Meanwhile, it can realize the information interaction by the adaptive local block aggregation (ALBA) module. To learn image-specific details, a back-projection refinement network is embedded in the structure, which further improves the performance in fine stage. The experiments on four benchmark datasets demonstrate that the proposed approach presents excellent performance over the existing methods. Our code is publicly available athttps://github.com/qianngli/SFPN.

Benefiting from the development of convolutional neural networks (CNNs), many excellent algorithms for object detection have been presented. Remote sensing object detection (RSOD) is a challenging task mainly due to: 1) complicated background of remote sensing images (RSIs) and 2) extremely imbalanced scale and sparsity distribution of remote sensing objects. Existing methods cannot effectively solve these problems with excellent detection accuracy and rapid speed. To address these issues, we propose an adaptive balanced network (ABNet) in this article. First, we design an enhanced effective channel attention (EECA) mechanism to improve the feature representation ability of the backbone, which can alleviate the obstacles of complex background on foreground objects. Then, to combine multiscale features adaptively in different channels and spatial positions, an adaptive feature pyramid network (AFPN) is designed to capture more discriminative features. Furthermore, considering that the original FPN ignores rich deep-level features, a context enhancement module (CEM) is proposed to exploit abundant semantic information for multiscale object detection. Experimental results on three public datasets demonstrate that our approach exhibits superior performance over baseline by only introducing less than 1.5M extra parameters.

Remote sensing image scene classification plays a significant role in remote sensing image analysis. Aiming at the problems of large transformation and scale variation of background and key objects in remote sensing images, we propose a neural architecture search (NAS) method based on attention search space. The network adaptively searches convolution, pooling, and attention operations in the appropriate layers. To ensure the stability of the searching process, a multistage network progressive fusion search method is proposed, which discards useless operations in stages, reduces the burden of search algorithm, and improves the search efficiency. Finally, paying attention to the association information between objects and scenes, a bottom-up multiscale fusion network connection strategy is proposed to fully reuse the semantics of multiscale feature maps in each stage. The experimental results show that the proposed method performs better than the manual method and the current neural network architecture search method.

Recently, salient object detection in optical remote sensing images (RSI-SOD) has attracted great attention. Benefiting from the success of deep learning and the inspiration of natural SOD task, RSI-SOD has achieved fast progress over the past two years. However, existing methods usually suffer from the intrinsic problems of optical RSIs, 1) cluttered background; 2) scale variation of salient objects; 3) complicated edges and irregular topology. To remedy these problems, we propose a hybrid feature aligned network (HFANet) jointly modeling boundary learning to detect salient objects effectively. Specifically, we design a hybrid encoder by unifying two components to capture global context for mitigating the disturbance of complex background. Then, to detect multiscale salient objects effectively, we propose a Gated Fold-ASPP (GF-ASPP) to extract abundant context in the deep semantic features. Furthermore, an adjacent feature aligned module (AFAM) is presented for integrating adjacent features with unparameterized alignment strategy. Finally, we propose a novel interactive guidance loss (IGLoss) to combine saliency and edge detection, which can adaptively perform mutual supervision of the two sub-tasks to facilitate detection of salient objects with blurred edges and irregular topology. Adequate experimental results on three optical RSI-SOD datasets reveal that the presented approach exceeds 18 state-of-the-art ones. All codes and detection results are available athttps://github.com/lyf0801/HFANet.

Hyperspectral image (HSI) clustering, which aims at dividing hyperspectral pixels into clusters without labeled training data, has drawn significant attention in practical applications. Recently, many graph-based clustering methods, which construct an adjacent graph to model the data relationship, have shown dominant performance. However, the high dimensionality of HSI data makes it hard to construct the pairwise adjacent graph. Besides, abundant spatial structures are often overlooked during the clustering procedure. In order to better handle the high dimensionality problem and preserve the spatial structures, this paper proposes a novel unsupervised approach called spatial-spectral clustering with anchor graph (SSCAG) for HSI data clustering. The SSCAG has the following contributions: 1) the multiscale filtering module is utilized to smooth the homogeneous regions, so that it can increase the similarity and consistency of neighboring pixels and capture the multiple views of a local region with different scales; 2) a new similarity metric is proposed to embed the spatial-spectral features into the combined adjacent graph, which can mine the intrinsic property structure of HSI data; 3) the AG-based strategy is adopted to construct the adjacent graph by a neighbor assignment scheme without hyperparameters, and its optimization employs SVD to replace eigenvalue decomposition to reduce the computational complexity. Extensive experiments on three public HSI datasets show that the proposed SSCAG is competitive against the state-of-the-art approaches.

With practical applications such as environment surveillance, agricultural production, and disaster assessment, accurate object detection in remote sensing images is in high demand. Precise detection of object instances in remote sensing images remains considerably challenging due to dense instance stacking, large-scale variations, and complex backgrounds. To solve the mentioned issues, a novel global context-weaving network (GCWNet) is developed for object detection in remote sensing images. We propose two novel modules for feature extraction and refinement, which include the global context aggregation module (GCAM) and the feature refinement module (FRM). GCAM assembles a global context with high-level and low-level features through feature weaving, which facilitates dense object detection. Meanwhile, FRM convolves multiple receptive fields by combining different branches, thereby further refining the features and improving the feature distinction at different scales. Furthermore, we design to alleviate the sample imbalanced problem during training using focal loss and balanced L1 loss to improve object classification and regression, respectively. The experimental results indicate that GCWNet achieves superior performance in object classification and localization on the DOTA-v1.5 dataset, which illustrates the superiority of GCWNet.

Visual question answering (VQA) for remote sensing scene has great potential in intelligent human-computer interaction system. Although VQA in computer vision has been widely researched, VQA for remote sensing data (RSVQA) is still in its infancy. There are two characteristics that need to be specially considered for the RSVQA task. 1) No object annotations are available in RSVQA datasets, which makes it difficult for models to exploit informative region representation; 2) There are questions with clearly different difficulty levels for each image in the RSVQA task. Directly training a model with questions in a random order may confuse the model and limit the performance. To address these two problems, in this paper, a multi-level visual feature learning method is proposed to jointly extract language-guided holistic and regional image features. Besides, a self-paced curriculum learning (SPCL)-based VQA model is developed to train networks with samples in an easy-to-hard way. To be more specific, a language-guided SPCL method with a soft weighting strategy is explored in this work. The proposed model is evaluated on three public datasets, and extensive experimental results show that the proposed RSVQA framework can achieve promising performance. Code will be available at https://gitlab.lrz.de/ai4eo/reasoning/VQA-easy2hard.

Hyperspectral image produces high spectral resolution at the sacrifice of spatial resolution. Without reducing the spectral resolution, improving the resolution in the spatial domain is a very challenging problem. Motivated by the discovery that hyperspectral image exhibits high similarity between adjacent bands in a large spectral range, in this paper, we explore a new structure for hyperspectral image super-resolution (DualSR), leading to a dual-stage design, i.e., coarse stage and fine stage. In coarse stage, five bands with high similarity in a certain spectral range are divided into three groups, and the current band is guided to study the potential knowledge. Under the action of alternative spectral fusion mechanism, the coarse SR image is super-resolved in band-by-band. In order to build model from a global perspective, an enhanced back-projection method via spectral angle constraint is developed in fine stage to learn the content of spatial-spectral consistency, dramatically improving the performance gain. Extensive experiments demonstrate the effectiveness of the proposed coarse stage and fine stage. Besides, our network produces state-of-the-art results against existing works in terms of spatial reconstruction and spectral fidelity. Our code is publicly available at https://github.com/qianngli/DualSR.

Recently fast arbitrary-shaped text detection has become an attractive research topic. However, most existing methods are non-real-time, which may fall short in intelligent systems. Although a few real-time text methods are proposed, the detection accuracy is far behind non-real-time methods. To improve the detection accuracy and speed simultaneously, we propose a novel fast and accurate text detection framework, namely CM-Net, which is constructed based on a new text representation method and a multi-perspective feature (MPF) module. The former can fit arbitrary-shaped text contours by concentric mask (CM) in an efficient and robust way. The latter encourages the network to learn more CM-related discriminative features from multiple perspectives and brings no extra computational cost. Benefiting the advantages of CM and MPF, the proposed CM-Net only needs to predict one CM of the text instance to rebuild the text contour and achieves the best balance between detection accuracy and speed compared with previous works. Moreover, to ensure that multi-perspective features are effectively learned, the multi-factor constraints loss is proposed. Extensive experiments demonstrate the proposed CM is efficient and robust to fit arbitrary-shaped text instances, and also validate the effectiveness of MPF and constraints loss for discriminative text features recognition. Furthermore, experimental results show that the proposed CM-Net is superior to existing state-of-the-art (SOTA) real-time text detection methods in both detection speed and accuracy on MSRA-TD500, CTW1500, Total-Text, and ICDAR2015 datasets.

Cross-domain crowd counting (CDCC) is a hot topic due to its importance in public safety. The purpose of CDCC is to alleviate the domain shift between the source and target domain. Recently, typical methods attempt to extract domain-invariant features via image translation and adversarial learning. When it comes to specific tasks, we find that the domain shifts are reflected in model parameters' differences. To describe the domain gap directly at the parameter level, we propose a neuron linear transformation (NLT) method, exploiting domain factor and bias weights to learn the domain shift. Specifically, for a specific neuron of a source model, NLT exploits few labeled target data to learn domain shift parameters. Finally, the target neuron is generated via a linear transformation. Extensive experiments and analysis on six real-world data sets validate that NLT achieves top performance compared with other domain adaptation methods. An ablation study also shows that the NLT is robust and more effective than supervised and fine-tune training. Code is available at https://github.com/taohan10200/NLT.

Many CNN-based segmentation methods have been applied in lane marking detection recently and gain excellent success for a strong ability in modeling semantic information. Although the accuracy of lane line prediction is getting better and better, lane markings' localization ability is relatively weak, especially when the lane marking point is remote. Traditional lane detection methods usually utilize highly specialized handcrafted features and carefully designed postprocessing to detect the lanes. However, these methods are based on strong assumptions and, thus, are prone to scalability. In this work, we propose a novel multitask method that: 1) integrates the ability to model semantic information of CNN and the strong localization ability provided by handcrafted features and 2) predicts the position of vanishing line. A novel lane fitting method based on vanishing line prediction is also proposed for sharp curves and nonflat road in this article. By integrating segmentation, specialized handcrafted features, and fitting, the accuracy of location and the convergence speed of networks are improved. Extensive experimental results on four-lane marking detection data sets show that our method achieves state-of-the-art performance.

Remote sensing image scene classification has attracted great attention because of its wide applications. Although convolutional neural network (CNN)-based methods for scene classification have achieved excellent results, the large-scale variation of the features and objects in remote sensing images limits the further improvement of the classification performance. To address this issue, we present multiscale representation for scene classification, which is realized by a global-local two-stream architecture. This architecture has two branches of the global stream and local stream, which can individually extract the global features and local features from the whole image and the most important area. In order to locate the most important area in the whole image using only image-level labels, a weakly supervised key area detection strategy of structured key area localization (SKAL) is specially designed to connect the above two streams. To verify the effectiveness of the proposed SKAL-based two-stream architecture, we conduct comparative experiments based on three widely used CNN models, including AlexNet, GoogleNet, and ResNet18, on four public remote sensing image scene classification data sets, and achieve the state-of-the-art results on all the four data sets. Our codes are provided in https://github.com/hw2hwei/SKAL.

Hyperspectral image exhibits low spatial resolution due to the limitation of imaging system. Improving it without an auxiliary high resolution (HR) image still remains a challenging problem. Recently, although many deep learning-based hyperspectral image super-resolution (SR) methods have been proposed, they make the insufficient utilization of adjacent bands to improve the reconstruction performance. To address this issue, we explore a new structure for hyperspectral image SR via adjacent spectral fusion strategy. Inspired by the high similarity among adjacent bands, neighboring band partition is proposed to divide the adjacent bands into several groups. Through the current band, the adjacent bands is guided to enhance the exploration ability. To explore more complementary information, an alternative fusion mechanism, i.e., intra-group fusion and inter-group fusion, is designed, which helps to recover the missing details in the current band. Experiments demonstrate that our approach produces the state-of-the-art results over the existing approaches.

Single blind image deblurring caused by a combination of multiple factors has been one of the most challenging visual tasks. Recently, many essential methods of this task are based on deep learning networks and have achieved high performance. However, most of them only apply norm pixel-wise L1-loss function as the guide of training, which is not suitable or effective enough. In this paper, we propose Multiple Auxiliary Networks (MANet) for single blind image deblurring to assist norm L1-loss function and enhance the quality of the deblurring image. The main branch of our MANet is an encoder-decoder structure made up of residual blocks, and the three auxiliary branches are the edge prediction branch, the multi-scale refinement branch, and the perceptual loss branch. The experimental results demonstrate that the proposed MANet can obtain better deblurring performance with more details than state-of-the-art methods. The code is released at github.com/ZERO2ER0/MANet.

The popular deep convolutional networks used for image super-resolution (SR) reconstruction often increase the network depth and employ attention mechanism to improve image reconstruction effect. However, these networks suffer from two problems. The first is the deeper network easily causes higher computational cost and more GPU memory usage. The second is traditional attention mechanism often misses the spatial information of images leading the loss of image detail information. To address these issues, we propose a lightweight non-local network (LNLN) for image super resolution in this paper. The proposed network makes two contributions. First, we use non-local module instead of normal attention module to obtain larger receptive field and extract more comprehensive feature information, which is helpful for improving image SR reconstruction results. Secondly, we use the depthwise separable convolution (DSC) instead of the vanilla convolution to reconstruct the residual block, which greatly reduces the number of parameters and computational cost. The proposed LNLN and comparative networks are evaluated on five commonly public datasets, and experiments demonstrate that the proposed LNLN is superior to state-of-the-art networks in terms of reconstruction performance, the number of parameters and storage space.

Unsupervised cross-domain person re-identification (re-ID) is a challenging task, because it is an open-set problem with completely unknown person identities in the target domain. Existing methods attempt to tackle the challenge by transferring image style across domains or generating pseudo labels in the target domain, whereas the valuable information in multiple domains (i.e., source domain, style-transferred data, and target domain) is not taken fully into consideration. To this end, we propose a novel multi-domain synchronous refinement (MDSR) network, where valuable knowledge from multiple domains is sufficiently exploited and refined to enforce the discriminative ability of the model. MDSR network contains two complementary modules dedicated to source-to-target domain adaptation and style-transferred data to the target domain adaptation, respectively. The domain adaptive knowledge from two modules is aggregated in the final stage. Extensive experiments verify our method achieves significant improvements over the state-of-the-art approaches on multiple unsupervised domain adaptative person re-ID tasks.

Road extraction from satellite images is usually corrupted with several disconnected segments so that it does not satisfy the real application. The segmentation-based methods fail to correct separated roads due to the incompleteness information. Therefore, this paper introduces auxiliary Road Location Prediction(RLP), a task leveraging global context information to help road segmentation infer each road segment. The auxiliary task has two branches: horizontal location prediction and vertical location prediction which can predict locations of all the roads. By combining road segmentation and RLP, road extraction performance is effectively improved. As a result, the additional training signals help the primary road segmentation task to aggregate surrounding scene information to reason about its connectivity. The experiments on two public datasets have demonstrated the effectiveness of the proposed method.

Hyperspectral image super-resolution (SR) methods are continually being refreshed due to deep neural networks. Despite this, the existing works barely explore more spatial information using mixed 2D/3D convolution. Moreover, they do not make full use of multi-domain features to realize information complementation. To tackle these challenges, we propose a hyperspectral image SR approach via multi-domain feature learning. To be specific, a multi-domain feature learning strategy using 2D/3D unit is presented to explore spatial and spectral information by alternate manner. To recover the more details, the edge body generation mechanism (EBGM) is introduced to learn the high frequency information, which generates the edge prior. Besides, the multi-domain feature fusion (MDFF) is designed to fully integrated hierarchical know ledge from different 2D/3D units, leading to further achieve information complementation. Experiments demonstrate that our approach attains the better performance over the state-of-the-art methods.

Graph-based semi-supervised learning methods, which deal well with the situation of limited labeled data, have shown dominant performance in practical applications. However, the high dimensionality of hyperspectral images (HSI) makes it hard to construct the pairwise adjacent graph. Besides, the fine spatial features that help improve the discriminability of the model are often overlooked. To handle the problems, this paper proposes a novel spatial-spectral HSI classification method via multiple random anchor graphs ensemble learning (RAGE). Firstly, the local binary pattern is adopted to extract the more descriptive features on each selected band, which preserves local structures and subtle changes of a region. Secondly, the adaptive neighbors assignment is introduced in the construction of anchor graph, to reduce the computational complexity. Finally, an ensemble model is built by utilizing multiple anchor graphs, such that the diversity of HSI is learned. Extensive experiments show that RAGE is competitive against the state-of-the-art approaches.

For license plate detection (LPD), most of the existing work is based on images as input. If these algorithms can be applied to multiple frames or videos, they can be adapted to more complex unconstrained scenes. In this paper, we propose a LPD framework for detecting license plates in multiple frames or videos, called AWFA-LPD, which effectively integrates the features of nearby frames. Compared with image based detection models, our network integrates optical flow extraction module, which can propagate the features of local frames and fuse with the reference frame. Moreover, we concatenate a non-link suppression module after the detection results to post-process the bounding boxes. Extensive experiments demonstrate the effectiveness and efficiency of our framework.

Many fields in the real world involve a lot of structured data, such as social networks, transportation networks, communication networks etc., and their structures carry important information about the characteristics of the data. However, how to use its structural information to analyse and process the data efficiently has caused continuous research in the field. A graph provides an important means for dealing with structured data. It can describe the geometric structure of data intuitively and flexibly, especially in the representation of spatial irregular data. Graph learning refers to machine learning on graphs, which mainly utilises machine learning algorithms to extract the relevant features of graphs. In recent years, combined with specific applications, researchers have conducted in-depth research on graph learning and proposed various approaches. This Special Issue aims to introduce the latest studies in graph learning for computer vision and proposes new theories and approaches to solve the existing problems. It received a number of submissions from researchers in the field, which all went through a rigorous review process. After several rounds of review, six papers were accepted. These papers cover a variety of fields, such as medicine, remote sensing, and data mining. Specific tasks include image segmentation, knowledge graph reasoning, clustering, and image classification. These accepted papers are mainly divided into two categories. The first category covers the graph learning method guided by optimisation, which obtains the graph structure by establishing a clear model and solving the corresponding optimisation problem. The second category is the deep learning-oriented graph learning method, which combines convolutional neural network and graph neural network to construct the model. In the first paper of the Special Issue by Wang et al. entitled An Enhanced 3D U-Net with Graph-based Refining for Segmentation of Gastrointestinal Stromal Tumours, the authors propose a segmentation algorithm using an improved 3D U-Net to segment gastrointestinal stromal tumours. To enhance information transmission, multiple skip connections are attached into same size feature maps between an encoder and a decoder. Due to difficulties in tumour labelling and other reasons, the author transforms the small intestinal segmentation model into a gastrointestinal stromal tumour segmentation model. Since fully convolutional networks typically suffer from inaccuracies around the boundaries of small structures, the graph neural network is introduced to refine segmentation results. Experiments demonstrate that the proposed method presents superior performance over traditional U-Net. The second paper by Ma et al. entitled Hybrid Attention Mechanism for Few-Shot Relational Learning of Knowledge Graphs, develops a few-shot relationship learning framework. The authors first design an entity-enhanced encoder with weak attention networks and self-attention mechanisms to explore the influence of different levels for source entities. The local graph structure is then utilised to enhance the embedding of the source entity by combining explicit and implicit features. Finally, the model parameters are optimised to infer real entities in the candidate set of similar entities obtained by a loop-processing matching processor. The authors provide extensive experiments and confirm the excellent accuracy of the proposed model. The third paper by Zhao et al. entitled Incremental Multi-View Correlated Feature Learning Based on Non-Negative Matrix Factorization, studies multi-view data. The authors present an incremental multi-view correlated feature learning approach based on non-negative matrix factorization to analyse the uncorrelated items in each view. The algorithm separates uncorrelateditems across views and constructs incremental joint learning with uncorrelated and correlated features to study the common features for multi-view data. Subsequently, the authors design an incremental objective function and derive an effective updating scheme. The proposed method is proved to converge effectively, and its complexity is discussed. They evaluate the proposed solution on real-world datasets and report excellent performance in comparison with the existing state-of-the-art solutions. The fourth paper by Hu et al. entitled Complete/Incomplete Multi-view Subspace Clustering via Soft Block-Diagonal-Induced Regularizer, concentrates on complete and incomplete multi-view clustering problems. The proposed method adopts the self-representation model to individually construct the similarity graphs for each view. To fuse a shared affinity matrix for all views, the authors design the soft block-diagonal-induced regulariser to encourage the generation of a matrix with K diagonal blocks. Considering the incomplete multi-view data, the proposed method effectively utilises some indicator matrices to accurately mark the missing instances in each view. The authors analyse the complexity and convergence of the proposed method on four public datasets and demonstrate that it is better than the most advanced complete/incomplete clustering methods. The fifth paper by Gong et al. entitled Few-shot Learning with Relation Propagation and Constraint, aims to extract valuable information of pair-wise correlation between sparse training samples. The authors state that transductive relation propagation simply propagates the pair-wise relation without relation constraints. Thus, the paper develops a constrained relation–propagation network to capture the accurate relation so as to generate discriminative relational representations. To constrain the pair-wise relation, the proposed method introduces a relation constraint module to regularise the distilled relations between samples, which helps to calibrate the propagated correlation information. Extensive experiments conducted on several benchmark datasets indicate that the proposed method achieves remarkable performance compared to few-shot learning methods. The last paper by Guo et al. entitled CNN-Combined Graph Residual Network with Multilevel Feature Fusion for Hyperspectral Image Classification introduces graph convolutional networks to obtain more superpixel-level features with a topological structure. This paper develops an effective CNN-combined graph residual network with a multilevel feature fusion strategy. The main idea is to learn superpixeltopological information by using the graph residual network and pixel information by using the convolutional neural network. The strategy can adequately leverage the superpixel level and pixel-level features and capture the class boundary features, which further enhances the generalisation performance. Experiments report highly competitive performance in comparison to existing hyperspectral image classification approaches. The papers selected in this Special Issue highlight the extensive study of graph learning in computer vision. We hope that these papers can promote the theoretical study of graph learning as well as provide new ideas for more researchers who are committed to graph learning.

With the development of modern life, all kinds of social activities have become more frequent. These social activities are often attended by a wide range of people, which puts forward high requirements for effective management and ensures the safety of the people involved in the activities. As an effective auxiliary measure, crowd understanding and analysis has been concerned by more and more researchers. The basic idea is to extract the key information from video sequences/images, and use digital image processing technology to study and analyse the behaviour characteristics and patterns of people in the region of interest. Currently, it has a wide range of applications in the fields of economy, public security, and so forth. This Special Issue aims to introduce the latest works in crowd understanding and analysis, and proposes new theory and approaches to solve the existing problems. The Special Issue received a number of submissions from researchers in the field. Finally, 27 papers were accepted after careful peer reviews and revisions. The accepted papers can be broadly divided into six sections according to the types of tasks, including Section 1: Crowd behaviour detection and recognition, Section 2: Crowd counting, Section 3: Object detection and recognition, and Section 4: Object tracking, and Section 5: Other tasks. “Human behaviour recognition with mid-level representations for crowd understanding and analysis” of Sun et al. uses mid-level semantic concepts to represent human actions from videos and argues that these semantic attributes enable the construction of more descriptive methods for human action recognition. The idea is verified on three challenging datasets, and the experimental results demonstrate that their method achieves better results than the baseline methods on human action recognition. “Class structure-aware adversarial loss for cross-domain human action recognition” of Chen et al. proposes a class structure-aware adversarial loss, which aims to address the issue that the existing adversarial-based approaches ignore the underlying coherence of class structure across domains. This paper incorporates category information into the adversarial learning branch to capture the fine-grained alignment of each class, effectively avoiding the false mix-up of samples from different categories in the embedding space. Experiments show significant improvement compared to the baseline. “Dual-view 3D human pose estimation without camera parameters for action recognition” of Liu et al. proposes a dual-view single-person 3D pose estimation method without camera parameters. This method first uses the 2D pose estimation network to estimate the 2D joint point coordinates from two images with different views, and then inputs them into the 3D regression network to generate the final 3D joint point coordinates. Experiments show that this method is effective. “Deep social force network for anomaly event detection” of Yang et al. develops a deep social force network by exploiting both social force extraction and deep motion coding. This network can discover the interaction force of particles to learn the deep social force features. The experiments on UCF-Crime and ShanghaiTech datasets demonstrate that this method can predict the temporal localization of anomaly events and outperform the state-of-the-art methods. “Anomaly detection in video sequences: a benchmark and computational model” of Wan et al. contributes a new Large-scale Anomaly Detection (LAD) dataset as the benchmark for anomaly detection in video sequences. This paper formulates anomaly detection as a fully supervised learning problem and proposes a multi-task deep neural network to solve it. Experimental results show that the proposed method outperforms state-of-the-art anomaly detection competitors on the proposed dataset and other public datasets. “Behaviour detection in crowded classroom scenes via enhancing features robust to scale and perspective variations” of Liu et al. proposes two modules to tackle the large variations of humans in scale and pose perspective, namely Scale Attention Aggregation module and RoI Spatial Transformation module. Besides, this paper constructs a new classroom human behaviour detection dataset with 1500 images based on a third-person perspective. The proposed method is verified on the proposed dataset. Experimental results demonstrate the effectiveness of the proposed method with better mAP values. “Crowd activity recognition in live video streaming via 3D-ResNet and region graph convolution network” of Kang et al. presents a crowd activity recognition method to identify and supervise the crowd activity from live videos. The method utilizes 3D-ResNet and ReGCN to extract deep spatiotemporal features and the correlation or external knowledge between crowd content, respectively. In addition to the above work, the authors construct a real-world video dataset called BJUT-CAD that includes eight kinds of crowd activity videos collected from live video websites. Experiments on BJUT-CAD and CAE datasets verify the effectiveness of this work. “Latent label mining for group activity recognition in basketball videos” of Wu et al. proposes a latent label mining strategy for group activity recognition in basketball videos. This paper aims at mining the latent labels of motion patterns from the frames and further combining two levels of supervision signal to obtain effective spatio-temporal representation. Experimental results demonstrate that the proposed algorithm achieves state-of-the-art performance. “A deep learning method for video-based action recognition” of Zhang et al. aims to address a key issue, that is, how to convert spatial and temporal information into an effective representation to infer actions. This paper employs boundary compensation on the basis of a deep neural network to achieve action proposal. Based on the resultant action proposals, a two-stream network with a spatio-temporal structure is adopted for the action recognition task. The experimental results show the competitive performance of the proposed method over the state-of-the-art methods. “MSR-FAN: Multi-scale residual feature-aware network for crowd counting” of Zhao et al. proposes a framework that combines the multi-scale features using multiple receptive field sizes and learns the feature-aware information on each image. This method effectively alleviates perspective distortion and the varying scales in congested scene images, which helps the algorithm crowd counting correctly. Experiment results on benchmark datasets indicate that the proposed approach outperforms the existing competitors. “MFP-Net: Multi-scale feature pyramid network for crowd counting” of Lei et al. introduces a feature pyramid fusion module and a feature attention-aware module. Two modules can extract different levels of fine-grained information; local and global context information, respectively. It enhances the correlation of different features and improves robustness to background noise effectively. Experiments show that the proposed method not only provides better crowd counting results than comparative models, but also requires fewer parameters. “Multi-level features extraction network with gating mechanism for crowd counting” of Zeng et al. designs a novel crowd counting model, which integrates multi-level information from multiple levels, such as appearance, scale, and context. To avoid interference from confusing information, a simple and effective multi-channel gated unit is proposed to adaptively select features at different levels of the network. Extensive experiments and evaluations clearly illustrate that the approach is superior. “Learn from object counting: crowd counting with meta-learning” of Zan et al. develops an efficient algorithm to extract the meta-information via utilizing object counting data in few-shot scenes. This method successfully explores the shared information between the crowd counting task and the object counting task, thus improving the performance and convergence rate of the crowd counting task. Comprehensive experiments on two datasets verify the effectiveness of the proposed approach. “Crowd estimation using key-point matching with support vector regression” of Ekanayake et al. proposes a novel key-point-based moving object detection in noisy backgrounds. This work mainly uses the key-point matching of continuous frames and optical flow density to identify moving objects in video sequences. The moving blobs are then generated via morphological operations. The experiments are compared with recent regression-based and CNN methods, and it is verified that the proposed work is superior. “A novel face recognition method based on fusion of LBP and HOG” of Chen et al. proposes an improved fusion local feature extraction algorithm called CS-NWALBP+HOG. This study not only smoothens noise sensitivity of the LBP operator, but also reduces the original computational complexity, as well as strengthens the description ability for image gradient direction information. Several experiments eventually demonstrate that the designed algorithm shows more robust performance under complex illumination conditions. “Multi-view intrinsic low-rank representation for robust face recognition and clustering” of Shen et al. considers the problem that the most existing methods ignore the specific local structure of different views. To address this issue, the paper proposes a multi-view low-rank representation method which exploits both intrinsic relationships and specific local structures of different views simultaneously. Experiments on several datasets demonstrate the effectiveness of this method in classification and clustering. “Multi-dimensional weighted cross-attention network in crowded scenes” of Xie et al. proposes an end-to-end anchor-free network, namely Multidimensional Weighted Cross-Attention Network, which can perform real-time human detection in crowded scenes. The designed model does not require manual intervention, and reduces the sizeable computational resource cost due to the anchor boxes mapping during the training process. Experiments reveal that the improved strategy achieves state-of-the-art results in the anchor-free methods. “Part-level attention networks for cross-domain person re-identification” of Zhao et al. uses the diversified spatial semantic feature in pixel-level learning in the target domain to improve the generality and adaptability of the model. Combined with partial branches, this method proves that it is effective to add an attention cascade module to the backbone network. Experiments indicate that the proposed approach has better recognition ability and robustness in cross-domain aspect. “MFNet-LE: Multilevel fusion network with Laplacian embedding for face presentation attacks detection” of Niu et al. proposes a face presentation attack detection method by incorporating a multilevel fusion structure and Laplacian loss into shallow CNNs. This allows the proposed model to learn more discriminative features under the joint supervision of softmax and Laplacian loss, which improves the detection ability. Experiments demonstrate the effectiveness of the proposed method. “Real-time automatic helmet detection of motorcyclists in urban traffic using improved YOLOv5 detector” of Jia et al. presents an automatic helmet detection that contains two steps. This work first utilizes improved YOLOv5 detector to detect motorcycles. Then, the motorcycle detected in the above step is input, and the improved YOLOv5 detector is used again to detect whether the rider is wearing a helmet. The proposed method is evaluated on constructed dataset, and the results show that it is superior to other detection methods. “Multi-label learning based target detecting from multi-frame data” of Mei et al. regards target detection from time series data as a multi-label problem to design the model. In this method, a background subtraction tracker is presented to track the slightly moving object in videos, which are based on Gaussian mixture model background subtraction and integral image. Experimental results show that the proposed method attains better performance. “Contrastive learning of graph encoder for accelerating pedestrian trajectory prediction training” of Yao et al. proposes a graph contrastive accelerating encoder. It accelerates the pedestrian trajectory prediction training process of spatiotemporal graph transformer networks. This method makes the pedestrian trajectory prediction error the lowest in the obviously early training steps, and makes the final performance reach the state-of-the-art level. “Multiple object tracking based on multi-task learning with strip attention” of Song et al. believes that it is difficult to strike a balance between accuracy and efficiency by embedding the re-identification (re-ID) model into the target tracking task. To enhance the overall tracking performance, a one-shot multiple object tracking is proposed based on multi-task learning, which contains two homogeneous branches of object detection and re-ID. By the fine-grained features extraction in pedestrian recognition, it benefits overall tracking framework improvement in both speed and robustness. The experiments show that the proposed method attains superior performance in more evaluation metrics. “Cross-modal semantic correlation learning by Bi-CNN network” of Wang et al. presents a novel cross modal retrieval framework, which integrates feature learning and latent space embedding. It aims to generate specific representations consistent with cross-modal tasks. It helps to reduce the differences in the distribution of categories in different modalities. Experiments on three real-word datasets show that the proposed work is superior to the popular methods. “Adaptive colour restoration and detail retention for image enhancement” of He et al. aims to overcome the problem of colour distortion caused by low illumination and fog. Considering the issue, this paper develops a multi-channel fusion-based adaptive image colour restoration method. To generate human-consistent observations, the detailed retention-based method is applied to enhance the details. Experiments demonstrate that the results are effective and outperform the compared methods both in visual and objective evaluations. “Image encryption algorithm for crowd data based on a new hyperchaotic system and Bernstein polynomial” of Jiang et al. designs a new two-dimensional chaotic system with hyperchaotic behaviour based on the Chebyshev system and the infinite collapse system. To protect the crowd image data, an image cryptosystem combined with the SVD and Bernstein polynomial is proposed. Security analyses indicate that this method has higher encryption efficiency and the visual quality of steganography image can reach 39 dB. “CA-PMG: Channel attention and progressive multi-granularity training network for fine-grained visual classification” of Zhao et al. designs a framework for visual classification for the subtle intra-class object variations. This model can be trained efficiently in an end-to-end manner without bounding box or part annotations. Extensive experiments on three challenging fine-grained datasets demonstrate that the approach obtains state-of-the-art performance. The papers selected in this Special Issue highlight the extensive study of crowd understanding and analysis. It is hoped that these papers can play a role in promoting theoretical research. Meanwhile, there are many challenges in this field that need to be further studied, such as the robustness of algorithms in cross-scenarios, the interaction between groups and individuals in different scenarios, and so forth. In addition, key problems such as the running time of the algorithm also need to be considered. These works will be helpful when applied to research in the real world. Lead Guest Editor Professor Qi Wang, Northwestern Polytechnical University, Xi'an, China Guest Editors Assistant Professor Bo Liu, Auburn University, USA Dr. Jianzhe Lin, University of British Columbia, Canada

With the benefits from deep learning technology, generating captions for remote sensing images has become achievable, and great progress has been made in this field in the recent years. However, a large-scale variation of remote sensing images, which would lead to errors or omissions in feature extraction, still limits the further improvement of caption quality. To address this problem, we propose a denoising-based multi-scale feature fusion (DMSFF) mechanism for remote sensing image captioning in this letter. The proposed DMSFF mechanism aggregates multiscale features with the denoising operation at the stage of visual feature extraction. It can help the encoder-decoder framework, which is widely used in image captioning, to obtain the denoising multiscale feature representation. In experiments, we apply the proposed DMSFF in the encoder-decoder framework and perform the comparative experiments on two public remote sensing image captioning data sets including UC Merced (UCM)-captions and Sydney-captions. The experimental results demonstrate the effectiveness of our method.

In the last decade, crowd counting and localization attract much attention of researchers due to its wide-spread applications, including crowd monitoring, public safety, space design, etc. Many convolutional neural networks (CNN) are designed for tackling this task. However, currently released datasets are so small-scale that they can not meet the needs of the supervised CNN-based algorithms. To remedy this problem, we construct a large-scale congested crowd counting and localization dataset, NWPU-Crowd, consisting of 5,109 images, in a total of 2,133,375 annotated heads with points and boxes. Compared with other real-world datasets, it contains various illumination scenes and has the largest density range ( 0 ∼ 20,033). Besides, a benchmark website is developed for impartially evaluating the different methods, which allows researchers to submit the results of the test set. Based on the proposed dataset, we further describe the data characteristics, evaluate the performance of some mainstream state-of-the-art (SOTA) methods, and analyze the new problems that arise on the new data. What's more, the benchmark is deployed at https://www.crowdbenchmark.com/, and the dataset/code/models/results are available at https://gjy3035.github.io/NWPU-Crowd-Sample-Code/.

With the development of deep neural networks, the performance of crowd counting and pixel-wise density estimation is continually being refreshed. Despite this, there are still two challenging problems in this field: 1) current supervised learning needs a large amount of training data, but collecting and annotating them is difficult and 2) existing methods cannot generalize well to the unseen domain. A recently released synthetic crowd dataset alleviates these two problems. However, the domain gap between the real-world data and synthetic images decreases the models' performance. To reduce the gap, in this article, we propose a domain-adaptation-style crowd counting method, which can effectively adapt the model from synthetic data to the specific real-world scenes. It consists of multilevel feature-aware adaptation (MFA) and structured density map alignment (SDA). To be specific, MFA boosts the model to extract domain-invariant features from multiple layers. SDA guarantees the network outputs fine density maps with a reasonable distribution on the real domain. Finally, we evaluate the proposed method on four mainstream surveillance crowd datasets, Shanghai Tech Part B, WorldExpo'10, Mall, and UCSD. Extensive experiments are evidence that our approach outperforms the state-of-the-art methods for the same cross-domain counting problem.

Hyperspectral image super-resolution (SR) methods based on deep learning have achieved significant progress recently. However, previous methods lack the joint analysis between spectrum and horizontal or vertical direction. Besides, when both 2D and 3D convolution are in the network, the existing models cannot effectively combine the two. To address these issues, in this article, we propose a novel hyperspectral image SR method by exploring the relationship between 2D/3D convolution (ERCSR). Our method alternately employs 2D and 3D units to solve the problem of structural redundancy by sharing spatial information during reconstruction for existing model, which can enhance the learning ability of 2D spatial domain. Importantly, compared with the network using 3D units, i.e., 2D units are replaced by 3D units, it can not only reduce the size of the model but also improve the performance of the model. Furthermore, to exploit the spectrum fully, the split adjacent spatial and spectral convolution (SAEC) is designed to parallelly explore information between spectrum and horizontal or vertical direction in space. Experiments on widely used benchmark datasets demonstrate that the proposed approach outperforms state-of-the-art SR algorithms across different scales in terms of quantitative and qualitative analysis.

The fusion of hyperspectral (HS) and multispectral (MS) images designed to obtain high-resolution HS (HRHS) images is a very challenging work. A series of solutions has been proposed in recent years. However, the similarity in the structure of the HS image has not been fully used. In this article, we present a novel HS and MS image-fusion method based on nonlocal low-rank tensor approximation and sparse representation. Specifically, the HS image and the MS image are considered the spatially and spectrally degraded versions of the HRHS image, respectively. Then, the nonlocal low-rank constraint term is adopted in order to form the nonlocal similarity and the spatial-spectral correlation. Meanwhile, we add the sparse constraint term to describe the sparsity of abundance. Thus, the proposed fusion model is established and its optimization is solved by alternative direction method of multipliers (ADMM). The experimental results on three synthetic data sets and one real data set show the advantages of the proposed method over several state-of-the-art competitors.

Recently, remote sensing image captioning (RSIC) has drawn an increasing attention. In this field, the encoder-decoder-based methods have become the mainstream due to their excellent performance. In the encoder-decoder framework, the convolutional neural network (CNN) is used to encode a remote sensing image into a semantic feature vector, and a sequence model such as long short-term memory (LSTM) is subsequently adopted to generate a content-related caption based on the feature vector. During the traditional training stage, the probability of the target word at each time step is forcibly optimized to 1 by the cross entropy (CE) loss. However, because of the variability and ambiguity of possible image captions, the target word could be replaced by other words like its synonyms, and therefore, such an optimization strategy would result in the overfitting of the network. In this article, we explore the overfitting phenomenon in the RSIC caused by CE loss and correspondingly propose a new truncation cross entropy (TCE) loss, aiming to alleviate the overfitting problem. In order to verify the effectiveness of the proposed approach, extensive comparison experiments are performed on three public RSIC data sets, including UCM-captions, Sydney-captions, and RSICD. The state-of-the-art result of Sydney-captions and RSICD and the competitive results of UCM-captions achieved by TCE loss demonstrate that the proposed method is beneficial to RSIC.

Remote sensing image captioning (RSIC), which aims at generating a well-formed sentence for a remote sensing image, has attracted more attention in recent years. The general framework for RSIC is the encoder–decoder architecture containing two submodels of encoder and decoder. Although the significant performance is obtained, the encoder–decoder architecture is a black-box model with a lack of explainability. To overcome this drawback, in this article, we propose a new explainable word–sentence framework for RSIC. The proposed word–sentence framework consists of two parts: word extractor and sentence generator, where the former extracts the valuable words in the given remote sensing image, while the latter organizes these words into a well-formed sentence. The proposed framework decomposes RSIC into a word classification task and a word sorting task, which is more in line with human intuitive understanding. On the basis of the word–sentence framework, some ablation experiments are conducted on the three public RSIC data sets of Sydney-captions, UCM-captions, and RSICD to explore the specific and effective network structures. In order to evaluate the proposed word–sentence framework objectively, we further conduct some comparative experiments on these three data sets and achieve comparable results in comparison with the encoder–decoder-based methods.

Hyperspectral images can provide dozens to hundreds of continuous spectral bands, so the richness of information has been greatly improved. However, these bands lead to increasing complexity of data processing, and the redundancy of adjacent bands is large. Recently, although many band selection methods have been proposed, this task is rarely handled through the context information of the whole spectral bands. Moreover, the scholars mainly focus on the different numbers of selected bands to explain the influence by accuracy measures, neglecting how many bands to choose is appropriate. To tackle these issues, we propose a fast neighborhood grouping method for hyperspectral band selection (FNGBS). The hyperspectral image cube in space is partitioned into several groups using coarse-fine strategy. By doing so, it effectively mines the context information in a large spectrum range. Compared with most algorithms, the proposed method can obtain the most relevant and informative bands simultaneously as subset in accordance with two factors, such as local density and information entropy. In addition, our method can also automatically determine the minimum number of recommended bands by determinantal point process. Extensive experimental results on benchmark data sets demonstrate the proposed FNGBS achieves satisfactory performance against state-of-the-art algorithms.

The fusion of a low-spatial-resolution hyperspectral image (HSI) (LR-HSI) with its corresponding high-spatial-resolution multispectral image (MSI) (HR-MSI) to reconstruct a high-spatial-resolution HSI (HR-HSI) has been a significant subject in recent years. Nevertheless, it is still difficult to achieve the cross-mode information fusion of spatial mode and spectral mode when reconstructing HR-HSI for the existing methods. In this article, based on a convolutional neural network (CNN), an interpretable spatial-spectral reconstruction network (SSR-NET) is proposed for more efficient HSI and MSI fusion. More specifically, the proposed SSR-NET is a physical straightforward model that consists of three components: 1) cross-mode message inserting (CMMI); this operation can produce the preliminary fused HR-HSI, preserving the most valuable information of LR-HSI and HR-MSI; 2) spatial reconstruction network (SpatRN); the SpatRN concentrates on reconstructing the lost spatial information of LR-HSI with the guidance of spatial edge loss (Lspat); and 3) spectral reconstruction network (SpecRN); the SpecRN pays attention to reconstruct the lost spectral information of HR-MSI under the constraint of spatial edge loss (Lspec). Comparative experiments are conducted on six HSI data sets of Urban, Pavia University (PU), Pavia Center (PC), Botswana, Indian Pines (IP), and Washington DC Mall (WDCM), and the proposed SSR-NET achieves the superior or competitive results in comparison with seven state-of-the-art methods. The code of SSR-NET is available at https://github.com/hw2hwei/SSRNET.

Indoor scene images usually contain scattered objects and various scene layouts, which make RGB-D scene classification a challenging task. Existing methods still have limitations for classifying scene images with great spatial variability. Thus, how to extract local patch-level features effectively using only image label is still an open problem for RGB-D scene recognition. In this article, we propose an efficient framework for RGB-D scene recognition, which adaptively selects important local features to capture the great spatial variability of scene images. Specifically, we design a differentiable local feature selection (DLFS) module, which can extract the appropriate number of key local scene-related features. Discriminative local theme-level and object-level representations can be selected with DLFS module from the spatially-correlated multi-modal RGB-D features. We take advantage of the correlation between RGB and depth modalities to provide more cues for selecting local features. To ensure that discriminative local features are selected, the variational mutual information maximization loss is proposed. Additionally, the DLFS module can be easily extended to select local features of different scales. By concatenating the local-orderless and global-structured multi-modal features, the proposed framework can achieve state-of-the-art performance on public RGB-D scene recognition datasets.

The conventional subspace clustering method obtains explicit data representation that captures the global structure of data and clusters via the associated subspace. However, due to the limitation of intrinsic linearity and fixed structure, the advantages of prior structure are limited. To address this problem, in this brief, we embed the structured graph learning with adaptive neighbors into the deep autoencoder networks such that an adaptive deep clustering approach, namely, autoencoder constrained clustering with adaptive neighbors (ACC_AN), is developed. The proposed method not only can adaptively investigate the nonlinear structure of data via a parameter-free graph built upon deep features but also can iteratively strengthen the correlations among the deep representations in the learning process. In addition, the local structure of raw data is preserved by minimizing the reconstruction error. Compared to the state-of-the-art works, ACC_AN is the first deep clustering method embedded with the adaptive structured graph learning to update the latent representation of data and structured deep graph simultaneously.

Context information is critical for image semantic segmentation. Especially in indoor scenes, the large variation of object scales makes spatial-context an important factor for improving the segmentation performance. Thus, in this paper, we propose a novel variational context-deformable (VCD) module to learn adaptive receptive-field in a structured fashion. Different from standard ConvNets, which share fixed-size spatial context for all pixels, the VCD module learns a deformable spatial-context with the guidance of depth information: depth information provides clues for identifying real local neighborhoods. Specifically, adaptive Gaussian kernels are learned with the guidance of multimodal information. By multiplying the learned Gaussian kernel with standard convolution filters, the VCD module can aggregate flexible spatial context for each pixel during convolution. The main contributions of this work are as follows: 1) a novel VCD module is proposed, which exploits learnable Gaussian kernels to enable feature learning with structured adaptive-context; 2) variational Bayesian probabilistic modeling is introduced for the training of VCD module, which can make it continuous and more stable; 3) a perspective-aware guidance module is designed to take advantage of multi-modal information for RGB-D segmentation. We evaluate the proposed approach on three widely-used datasets, and the performance improvement has shown the effectiveness of the proposed method.

For pixel-level crowd understanding, it is time-consuming and laborious in data collection and annotation. Some domain adaptation algorithms try to liberate it by training models with synthetic data, and the results in some recent works have proved the feasibility. However, we found that a mass of estimation errors in the background areas impede the performance of the existing methods. In this paper, we propose a domain adaptation method to eliminate it. According to the semantic consistency, a similar distribution in deep layer's features of the synthetic and real-world crowd area, we first introduce a semantic extractor to effectively distinguish crowd and background in high-level semantic information. Besides, to further enhance the adapted model, we adopt adversarial learning to align features in the semantic space. Experiments on three representative real datasets show that the proposed domain adaptation scheme achieves the state-of-the-art for cross-domain counting problems.

Recently abnormality detection in musculoskeletal radio-graphs has attracted many attentions. For abnormality detection, it is crucial to locate the most important area in the musculoskeletal radiographs. To achieve this goal, we propose a key area localization mechanism (KALM) for abnormality detection for the first time in this paper. The proposed KALM explicitly defines the process of selecting the most important area from the whole image with using only image-level label. Based on KALM, we further present a joint global and local feature representation strategy for abnormality detection which takes as input both the entire image and the selected local area. The experimental results based on several classical convolutional neural network (CNN) architectures of MURA, the largest abnormality detection dataset of musculoskeletal radiographs, demonstrate the effectiveness of our KALM.

Video frame interpolation achieves temporal super-resolution by generating smooth transitions between frames. Although great success has been achieved by deep neural networks, the synthesized images stills suffer from poor visual appearance and unsatisfactory artifacts. In this paper, we propose a novel network structure that leverages residue refinement and adaptive weight to synthesize in-between frames. The residue refinement technique is used for optical flow and image generation for higher accuracy and better visual appearance, while the adaptive weight map combines the forward and backward warped frames to reduce the artifacts. Moreover, all submodules in our method are implemented by U-Net with less depths, so the efficiency is guaranteed. Experiments on public datasets demonstrate the effectiveness and superiority of our method over the state-of-the-art approaches.

Recently, lots of deep networks are proposed to improve the quality of predicted super-resolution (SR) images, due to its widespread use in several image-based fields. However, with these networks being constructed deeper and deeper, they also cost much longer time for training, which may guide the learners to local optimization. To tackle this problem, this paper designs a training strategy named Pixel-level Self-Paced Learning (PSPL) to accelerate the convergence velocity of SISR models. PSPL imitating self-paced learning gives each pixel in the predicted SR image and its corresponding pixel in ground truth an attention weight, to guide the model to a better region in parameter space. Extensive experiments proved that PSPL could speed up the training of SISR models, and prompt several existing models to obtain new better results. Furthermore, the source code is available at https://github.com/Elin24/PSPL.

Graph-based clustering is an advanced clustering techniuqe, which partitions the data according to an affinity graph. However, the graph quality affects the clustering results to a large extent, and it is difficult to construct a graph with high quality, especially for data with noises and outliers. To solve this problem, a robust rank constrained sparse learning method is proposed in this paper. The L2,1-norm objective function of sparse representation is introduced to learn the optimal graph with robustness. To preserve the data structure, the graph is searched within the neighborhood of the initial graph. By incorporating a rank constraint, the learned graph can be directly used as the cluster indicator and the final results is obtained without additional post-processing. Plenty of experiments on real-world data sets have proved the superiority and the robustness of the proposed approach.

Dynamic scene deblurring is a challenging problem due to the various blurry source. Many deep learning based approaches try to train end-to-end deblurring networks, and achieve successful performance. However, the architectures and parameters of these methods are unchanged after training, so they need deeper network architectures and more parameters to adapt different blurry images, which increase the computational complexity. In this paper, we propose a local correlation block (LCBlock), which can adjust the weights of features adaptively according to the blurry inputs. And we use it to construct a dynamic scene deblurring network named LCNet. Experimental results show that the proposed LC-Net produces compariable performance with shorter running time and smaller network size, compared to state-of-the-art learning-based methods.

Crowd counting has a wide range of applications such as video surveillance and public safety. Many existing methods only focus on improving the accuracy of counting but ignore the importance of density maps. It's no doubt that a high-quality density map contains more information such as localization and movement of the crowd. In this paper, we propose a multi-supervised parallel network (MSPNet) to achieve high accuracy of crowd counting and generate high-quality density maps. We conduct multiple supervisions in the training process, which can supplement the details lost in pooling and up-sampling operations to improve the quality of density maps. In addition, to reduce the impact of background noise, the attention mechanism is employed to help the network focus on the crowd. Extensive experiments on two mainstream benchmarks show that MSPNet achieves significantly improvement over the state-of-the-art in terms of counting accuracy and the quality of density maps.

This paper proposes a novel metric-based deep learning method to solve the few-shot learning problem. It models the relation between images as high dimensional vector, and trains a network module to judge, when given two relational features, which one indicates a stronger connection between the image objects. By training such a network module, we introduce a comparative mechanism into the metric space, i.e., the similarity score of any two images is computed after seeing other images in the same task. Further more, we propose to incorporate a batch classification loss into episodic training to mitigate the hard training problem that occurs when embedding network is going deeper. Experiments demonstrate that the proposed network can achieve promising performance.

License plate recognition (LPR) is one of the essential components in intelligent transportation systems. Although the image processing algorithms for LPR have been extensively studied in the past several years, the recognition performance is still not satisfactory especially in unconstrained complex scenes. In order to tackle this issue, a novel deep multi-task learning-based method is proposed in this paper by introducing contextual information in multiple license plate frames. Specifically, an end-to-end trainable multi-task architecture, namely IQ-STAN, is developed by joint license plate recognition and image quality scoring. Moreover, we propose an image quality-guided spatio-temporal attention mechanism, which is utilized in the frame-level feature representation during the phase of plate recognition. Extensive experiments are conducted and the competitive results demonstrate the effectiveness of our proposed framework.

License plate recognition (LPR) has attracted considerable attention due to its widespread applications in real life. Although numerous approaches based on image processing have been presented in the past few years, it is still an urgent issue to perform the LPR task efficiently in complex and unconstrained scenarios. To remedy this problem, an efficient quality-aware license plate recognition algorithm is proposed by introducing the siamese networks for plate stream recognition and quality awareness in the traffic videos. Moreover, we explore three progressive architectures for efficient and accurate recognition. Knowledge distillation is adopted to compress the quality awareness network and make it lightweight. Extensive experiments have demonstrated the impressive performance and efficiency of the proposed method.

Short-range motion features and long-range dependencies are two complementary and vital cues for action recognition in videos, but it remains unclear how to efficiently and effectively extract these two features. In this paper, we propose a novel network to capture these two features in a unified 2D framework. Specifically, we first construct a Short-range Temporal Interchange (STI) block, which contains a Channels-wise Temporal Interchange (CTI) module for encoding short-range motion features. Then a Graph-based Regional Interchange (GRI) module is built to present long-range dependencies using graph convolution. Finally, we replace original bottleneck blocks in the ResNet with STI blocks and insert several GRI modules between STI blocks, to form a Multi-range Feature Interchange (MFI) Network. Practically, extensive experiments are conducted on three action recognition datasets (i.e., Something-Something V1, HMDB51, and UCF101), which demonstrate that the proposed MFI network achieves impressive results with very limited computing cost.

Hyperspectral (HS) and multispectral (MS) image fusion is an important task to construct an HS image with high spatial and spectral resolutions. In this paper, we present a novel HS and MS fusion method using non-convex low rank tensor approximation and total variation regularization. In specific, the Laplace based low-rank model is formed to exploit spatial-spectral correlation and nonlocal similarity of the HS image, and the second-order total variation is used to describe the local smoothness structure in the spatial domain and adjacent bands. Also, an effective optimization algorithm is designed for the proposed model. In the experiments, we demonstrate the superiority of the proposed method compared to several state-of-the-art approaches.

Unlabeled data learning has attracted considerable attention recently. However, it is still elusive to extract the expected high-level semantic feature with mere unsupervised learning. In the meantime, semi-supervised learning (SSL) demonstrates a promising future in leveraging few samples. In this paper, we combine both to propose an Unsupervised Semantic Aggregation and Deformable Template Matching (USADTM) framework for SSL, which strives to improve the classification performance with few labeled data and then reduce the cost in data annotating. Specifically, unsupervised semantic aggregation based on Triplet Mutual Information (T-MI) loss is explored to generate semantic labels for unlabeled data. Then the semantic labels are aligned to the actual class by the supervision of labeled data. Furthermore, a feature pool that stores the labeled samples is dynamically updated to assign proxy labels for unlabeled data, which are used as targets for cross-entropy minimization. Extensive experiments and analysis across four standard semi-supervised learning benchmarks validate that USADTM achieves top performance (e.g., 90.46% accuracy on CIFAR-10 with 40 labels and 95.20% accuracy with 250 labels). The code is released at https://github.com/taohan10200/USADTM.

Hyperspectral images (HSIs) usually have a high spectral resolution but low spatial resolution due to hardware limitations, while multispectral images (MSIs) usually have a low spectral resolution but high spatial resolution. To obtain an image with a high resolution both in spectral and spatial domains, a general strategy is image fusion. A variety of methods have been proposed on this, but these methods generally cannot achieve good performance due to the incomplete overlapping wavelength of the HSI and the MSI. To solve this problem, this letter proposes a novel HSI fusion method based on band simulation. The proposed method expands MSI using spectral unmixing and acquires high-resolution images based on linear least squares. The experimental results on two hyperspectral data sets show that the proposed method outperforms the competitors, especially when the overlapping wavelength of the HSI and the MSI is small.

Detecting coherent groups is fundamentally important for crowd behavior analysis. In the past few decades, plenty of works have been conducted on this topic, but most of them have limitations due to the insufficient utilization of crowd properties and the arbitrary processing of individuals. In this study, a Multiview-based Parameter Free framework (MPF) is proposed. Based on the L1-norm and L2-norm, we design two versions of the multiview clustering method, which is the main part of the proposed framework. This paper presents the contributions on three aspects: (1) a new structural context descriptor is designed to characterize the structural properties of individuals in crowd scenes; (2) a self-weighted multiview clustering method is proposed to cluster feature points by incorporating their orientation and context similarities; and (3) a novel framework is introduced for group detection, which is able to determine the group number automatically without any parameter or threshold to be tuned. The effectiveness of the proposed framework is evaluated on real-world crowd videos, and the experimental results show its promising performance on group detection. In addition, the proposed multiview clustering method is also evaluated on a synthetic dataset and several standard benchmarks, and its superiority over the state-of-the-art competitors is demonstrated.

Crowd counting from a single image is a challenging task due to high appearance similarity, perspective changes, and severe congestion. Many methods only focus on the local appearance features and they cannot handle the aforementioned challenges. In order to tackle them, we propose a perspective crowd counting network (PCC Net), which consists of three parts: 1) density map estimation (DME) focuses on learning very local features of density map estimation; 2) random high-level density classification (R-HDC) extracts global features to predict the coarse density labels of random patches in images; and 3) fore-/background segmentation (FBS) encodes mid-level features to segments the foreground and background. Besides, the Down, Up, Left, and Right (DULR) module is embedded in PCC Net to encode the perspective changes on four directions (DULR). The proposed PCC Net is verified on five mainstream datasets, which achieves the state-of-the-art performance on the one and attains the competitive results on the other four datasets. The source code is available at https://github.com/gjy3035/PCC-Net.

Band selection is one of the most important technique in the reduction of hyperspectral image (HSI). Different from traditional feature selection problem, an important characteristic of it is that there is usually strong correlation between neighboring bands, that is, bands with close indexes. Aiming to fully exploit this prior information, a novel band selection method called optimal neighborhood reconstruction (ONR) is proposed. In ONR, band selection is considered as a combinatorial optimization problem. It evaluates a band combination by assessing its ability to reconstruct the original data, and applies a noise reducer to minimize the influence of noisy bands. Instead of using some approximate algorithms, ONR exploits a recurrence relation that underlies the optimization target to obtain the optimal solution in an efficient way. Besides, we develop a parameter selection approach to automatically determine the parameter of ONR, ensuring it is adaptable to different data sets. In experiments, ONR is compared with some state-of-the-art methods on six HSI data sets. The results demonstrate that ONR is more effective and robust than the others in most of the cases.

People in crowd scenes always exhibit consistent behaviors and form collective motions. The analysis of collective motion has motivated a surge of interest in computer vision. Nevertheless, the effort is hampered by the complex nature of collective motions. Considering the fact that collective motions are formed by individuals, this paper proposes a new framework for both quantifying and detecting collective motion by investigating the spatio-temporal behavior of individuals. The main contributions of this work are threefold: 1) an intention-aware model is built to fully capture the intrinsic dynamics of individuals; 2) a structure-based collectiveness measurement is developed to accurately quantify the collective properties of crowds; 3) a multistage clustering strategy is formulated to detect both the local and global behavior consistency in crowd scenes. Experiments on real world data sets show that our method is able to handle crowds with various structures and time-varying dynamics. Especially, the proposed method shows nearly 10% improvement over the competitors in terms of NMI, Purity and RI. Its applicability is illustrated in the context of anomaly detection and semantic scene segmentation.

Graph clustering plays an important role in data mining. Based on an input data graph, data points are partitioned into clusters. However, most existing methods keep the data graph fixed during the clustering procedure, so they are limited to exploit the implied data manifold and highly dependent on the initial graph construction. Inspired by the recent development on manifold learning, this paper proposes an Adaptive Consistency Propagation (ACP) method for graph clustering. In order to utilize the features captured from different perspectives, we further put forward the Multi-view version of the ACP model (MACP). The main contributions are threefold: (1) the manifold structure of input data is sufficiently exploited by propagating the topological connectivities between data points from near to far; (2) the optimal graph for clustering is learned by taking graph learning as a part of the optimization procedure; and (3) the negotiation among the heterogeneous features is captured by the multi-view clustering model. Extensive experiments on real-world datasets validate the effectiveness of the proposed methods on both single-and multi-view clustering, and show their superior performance over the state-of-the-arts.

Non-negative Matrix Factorization (NMF) has been one of the most popular clustering techniques in machine leaning, and involves various real-world applications. Most existing works perform matrix factorization on high-dimensional data directly. However, the intrinsic data structure is always hidden within the low-dimensional subspace. And, the redundant features within the input space may affect the final result adversely. In this paper, a new unsupervised matrix factorization method, Discrimination-aware Projected Matrix Factorization (DPMF), is proposed for data clustering. The main contributions are threefold: (1) The linear discriminant analysis is jointly incorporated into the unsupervised matrix factorization framework, so the clustering can be accomplished in the discriminant subspace. (2) The manifold regularization is introduced to perceive the geometric information, and the ℓ2,1-norm is utilized to improve the robustness. (3) An efficient optimization algorithm is designed to solve the proposed problem with proved convergence. Experimental results on one toy dataset and eight real-world benchmarks show the effectiveness of the proposed method.

Human actions captured in video sequences contain two crucial factors for action recognition, i.e., visual appearance and motion dynamics. To model these two aspects, Convolutional and Recurrent Neural Networks (CNNs and RNNs) are adopted in most existing successful methods for recognizing actions. However, CNN based methods are limited in modeling long-term motion dynamics. RNNs are able to learn temporal motion dynamics but lack effective ways to tackle unsteady dynamics in long-duration motion. In this work, we propose a memory-augmented temporal dynamic learning network, which learns to write the most evident information into an external memory module and ignore irrelevant ones. In particular, we present a differential memory controller to make a discrete decision on whether the external memory module should be updated with current feature. The discrete memory controller takes in the memory history, context embedding and current feature as inputs and controls information flow into the external memory module. Additionally, we train this discrete memory controller using straight-through estimator. We evaluate this end-to-end system on benchmark datasets (UCF101 and HMDB51) of human action recognition. The experimental results show consistent improvements on both datasets over prior works and our baselines.

RGB image classification has achieved significant performance improvement with the resurge of deep convolutional neural networks. However, mono-modal deep models for RGB image still have several limitations when applied to RGB-D scene recognition. 1) Images for scene classification usually contain more than one typical object with flexible spatial distribution, so the object-level local features should also be considered in addition to global scene representation. 2) Multi-modal features in RGB-D scene classification are still under-utilized. Simply combining these modal-specific features suffers from the semantic gaps between different modalities. 3) Most existing methods neglect the complex relationships among multiple modality features. Considering these limitations, this paper proposes an adaptive crossmodal (ACM) feature learning framework based on graph convolutional neural networks for RGB-D scene recognition. In order to make better use of the modal-specific cues, this approach mines the intra-modality relationships among the selected local features from one modality. To leverage the multi-modal knowledge more effectively, the proposed approach models the inter-modality relationships between two modalities through the cross-modal graph (CMG). We evaluate the proposed method on two public RGB-D scene classification datasets: SUN-RGBD and NYUD V2, and the proposed method achieves state-of-the-art performance.

Visual-to-auditory sensory substitution devices can assist the blind in sensing the visual environment by translating the visual information into a sound pattern. To improve the translation quality, the task performances of the blind are usually employed to evaluate different encoding schemes. In contrast to the toilsome human-based assessment, we argue that machine model can be also developed for evaluation, and more efficient. To this end, we firstly propose two distinct cross-modal perception model w.r.t. the late-blind and congenitally-blind cases, which aim to generate concrete visual contents based on the translated sound. To validate the functionality of proposed models, two novel optimization strategies w.r.t. the primary encoding scheme are presented. Further, we conduct sets of human-based experiments to evaluate and compare them with the conducted machine-based assessments in the cross-modal generation task. Their highly consistent results w.r.t. different encoding schemes indicate that using machine model to accelerate optimization evaluation and reduce experimental cost is feasible to some extent, which could dramatically promote the upgrading of encoding scheme then help the blind to improve their visual perception ability.

Recently, counting the number of people for crowd scenes is a hot topic because of its widespread applications (e.g. video surveillance, public security). It is a difficult task in the wild: changeable environment, large-range number of people cause the current methods can not work well. In addition, due to the scarce data, many methods suffer from over-fitting to a different extent. To remedy the above two problems, firstly, we develop a data collector and labeler, which can generate the synthetic crowd scenes and simultaneously annotate them without any manpower. Based on it, we build a large-scale, diverse synthetic dataset. Secondly, we propose two schemes that exploit the synthetic data to boost the performance of crowd counting in the wild: 1) pretrain a crowd counter on the synthetic data, then finetune it using the real data, which significantly prompts the model's performance on real data; 2) propose a crowd counting method via domain adaptation, which can free humans from heavy data annotations. Extensive experiments show that the first method achieves the state-of-the-art performance on four real datasets, and the second outperforms our baselines. The dataset and source code are available at https://gjy3035.github.io/GCC-CL/.

In this paper, we present iDVO (inertia-embedded deep visual odometry), a self-supervised learning based monocular visual odometry (VO) for road vehicles. When modelling the geometric consistency within adjacent frames, most deep VO methods ignore the temporal continuity of the camera pose, which results in a very severe jagged fluctuation in the velocity curves. With the observation that road vehicles tend to perform smooth dynamic characteristics in most of the time, we design the inertia loss function to describe the abnormal motion variation, which assists the model to learn the consecutiveness from long-term camera ego-motion. Based on the recurrent convolutional neural network (RCNN) architecture, our method implicitly models the dynamics of road vehicles and the temporal consecutiveness by the extended Long Short-Term Memory (LSTM) block. Furthermore, we develop the dynamic hard-edge mask to handle the non-consistency in fast camera motion by blocking the boundary part and which generates more efficiency in the whole non-consistency mask. The proposed method is evaluated on the KITTI dataset, and the results demonstrate state-of-the-art performance with respect to other monocular deep VO and SLAM approaches.

Recently, the analysis of remote sensing images has attracted a lot of attention. In the domain of scene classification, deep learning methods, especially convolutional networks (CNNs), currently achieve the best results. Although the classification performance has reached a high level, there are still some factors limiting the improvement of classification accuracy. Based on obeservation of remote sensing scene images, we fing that some scenes are quite similar though they belong to different classes. To improve the classification performance between different scenes with similar characteristics, we propose a significant Feature Sparsity Layer that can be esaily embedded into various convolutional network architectures. The proposed layer can inhibit the confusing features meanwhile stress the discriminative features, and it is used to sparse the multi-layer feature map, which is extracted by the convolutional layers. The proposed method achieves the state-of-the-art results on three datasets UC Merced Land Use, Aerial Image Data and OPTIMAL-31, and competitive result on dataset WHU-RS19.

In this paper, a novel introspective learning strategy for remote sensing scene classification is proposed. Through this strategy, the neural network used for classification can introspectively generate negative samples. In most training deep neural networks, negative samples are rarely noticed. We are the first to actively introduce negative samples into the remote sensing scene classification tasks. The goal of this paper is to analyze the effect of introspective negative samples on remote sensing scene classification tasks. Experiments demonstrate that the introduction of negative samples in training can effectively improve the classification accuracy and robustness. In addition, we found that our method can effectively against invalid remote sensing images.

Hyperspectral unmixing, by extracting the fractional abundances of endmembers from the hyperspectral image (HSI), has raised wide attention in recent years. In last decade, nonnegative matrix factorization (NMF) have been intensively studied for solving spectral unmixing problem. In this paper, we extend the multilayer NMF method by incorporating the L1/4sparsity constraint, named L1/4-MLNMF. The L1/4regularizer induces sparsity effectively. We propose an iterative estimation algorithm for L1/4-MLNMF, which provides sparser and more accurate results than MLNMF. Experiments on a synthetic dataset and a real dataset show that the prposed method outperforms the similar competitors.

With the rapid development of artificial satellite, a large number of high resolution remote sensing images can be easily obtained now. Recently, remote sensing image captioning, which aims to generate accurate and concise descriptive sentences for remote sensing images, has been promoted by template-based model and encoder-decoder model with several related datasets released. Based on an encoder-decoder model, we propose a training mechanism of multi-scale cropping for remote sensing image captioning in this paper, which can extract more fine-grained information from remote sensing images and enhance the generalization performance of the base model. The experimental results on two datasets UCM-captions and Sydney-captions demonstrate that the proposed approach availably improves the performances in describing high resolution remote sensing images.

In this work, we propose a compact multi-task architecture based on deep learning for remote sensing scene classification and image quality assessment (IQA) simultaneously. The model can be trained in an end-to-end manner, and the robustness of classification is improved in our method. More importantly, by exploiting IQA and super-resolution, the accurate classification results can be obtained even if the images are distorted or with low quality. To the best of our knowledge, it is the first successful attempt to associate IQA with scene classification in a unified multi-task architecture. Our method is evaluated on the expanded UC Merced Land-Use dataset after data augmentation. In comparison with some other methods, the experimental results show that the proposed structure makes a great improvement on both classification and IQA.

Mahalanobis metric learning is one of the most popular methods for person re‐identification. Most existing metric learning methods regularly formulate the person re‐identification as an unconstrained optimisation problem and the constraints on the Mahalanobis matrix are seldom imposed. In addition, weights are often used to model the relationships between different variables but they often suffer from boundedness caused by their hand‐designed feature. Taking the above two disadvantages into consideration, the authors propose a new metric learning method for person re‐identification, which formulates the metric learning problem as a constrained optimisation problem by imposing a constraint on the linear transformation matrix. Furthermore, they treat the weights as unknown variables and introduce a weight learning method instead of designing weight intuitively. Finally, they evaluate the proposed method on two challenging person re‐identification databases and show that it performs favourably against the state‐of‐the‐art approaches.

Dynamic scene classification has been extensively studied in computer vision due to its widespread applications. The key to dynamic scene classification lies in jointly characterizing spatial appearance and temporal dynamics to achieve informative representation, which remains an outstanding task in the literature. In this paper, we propose a unified framework to extract spatial and temporal features for dynamic scene representation. More specifically, we deploy two variants of deep convolutional neural networks to encode spatial appearance and short-term dynamics into short-term deep features (STDF). Based on STDF, we propose using the autoregressive moving average model to extract long-term frequency features (LTFF). By combining STDF and LTFF, we establish the long-short-term feature (LSTF) representations of dynamic scenes. The LSTF characterizes both spatial and temporal patterns of dynamic scenes for comprehensive and information representation that enables more accurate classification. Extensive experiments on three-dynamic scene classification benchmarks have shown that the proposed LSTF achieves high performance and substantially surpasses the state-of-the-art methods.

We are now living in an era of visual information where data is unceasingly generated and pushed into consumption at astounding rates. A remarkable portion of this sensory input comes in the form of videos streaming from large-scale surveillance infrastructures as well as consumer-grade monitoring systems. The sheer amount of ground-based, aerial and mobile video surveillance data demands fittingly competent, accurate, effective techniques to extract useful cues and provide assistance for detection, prevention, and intervention tasks in traffic, safety, security, defense, forensic, health, biology, ethology, and retail space management applications.

Hyperspectral image (HSI) classification, which aims to assign an accurate label for hyperspectral pixels, has drawn great interest in recent years. Although low-rank representation (LRR) has been used to classify HSI, its ability to segment each class from the whole HSI data has not been exploited fully yet. LRR has a good capacity to capture the underlying low-dimensional subspaces embedded in original data. However, there are still two drawbacks for LRR. First, the LRR does not consider the local geometric structure within data, which makes the local correlation among neighboring data easily ignored. Second, the representation obtained by solving LRR is not discriminative enough to separate different data. In this paper, a novel locality- and structure-regularized LRR (LSLRR) model is proposed for HSI classification. To overcome the above-mentioned limitations, we present locality constraint criterion and structure preserving strategy to improve the classical LRR. Specifically, we introduce a new distance metric, which combines both spatial and spectral features, to explore the local similarity of pixels. Thus, the global and local structures of HSI data can be exploited sufficiently. In addition, we propose a structural constraint to make the representation have a near-block-diagonal structure. This helps to determine the final classification labels directly. Extensive experiments have been conducted on three popular HSI data sets. And the experimental results demonstrate that the proposed LSLRR outperforms other state-of-the-art methods.

Scene classification of remote sensing images has drawn great attention because of its wide applications. In this paper, with the guidance of the human visual system (HVS), we explore the attention mechanism and propose a novel end-to-end attention recurrent convolutional network (ARCNet) for scene classification. It can learn to focus selectively on some key regions or locations and just process them at high-level features, thereby discarding the noncritical information and promoting the classification performance. The contributions of this paper are threefold. First, we design a novel recurrent attention structure to squeeze high-level semantic and spatial features into several simplex vectors for the reduction of learning parameters. Second, an end-to-end network named ARCNet is proposed to adaptively select a series of attention regions and then to generate powerful predictions by learning to process them sequentially. Third, we construct a new data set named OPTIMAL-31, which contains more categories than popular data sets and gives researchers an extra platform to validate their algorithms. The experimental results demonstrate that our model makes great promotion in comparison with the state-of-the-art approaches.

Change detection (CD) is an important application of remote sensing, which provides timely change information about large-scale Earth surface. With the emergence of hyperspectral imagery, CD technology has been greatly promoted, as hyperspectral data with high spectral resolution are capable of detecting finer changes than using the traditional multispectral imagery. Nevertheless, the high dimension of the hyperspectral data makes it difficult to implement traditional CD algorithms. Besides, endmember abundance information at subpixel level is often not fully utilized. In order to better handle high-dimension problem and explore abundance information, this paper presents a general end-to-end 2-D convolutional neural network (CNN) framework for hyperspectral image CD (HSI-CD). The main contributions of this paper are threefold: 1) mixed-affinity matrix that integrates subpixel representation is introduced to mine more cross-channel gradient features and fuse multisource information; 2) 2-D CNN is designed to learn the discriminative features effectively from the multisource data at a higher level and enhance the generalization ability of the proposed CD algorithm; and 3) the new HSI-CD data set is designed for objective comparison of different methods. Experimental results on real hyperspectral data sets demonstrate that the proposed method outperforms most of the state of the arts.

Semantic segmentation, a pixel-level vision task, is rapidly developed by using convolutional neural networks (CNNs). Training CNNs requires a large amount of labeled data, but manually annotating data is difficult. For emancipating manpower, in recent years, some synthetic datasets are released. However, they are still different from real scenes, which causes that training a model on the synthetic data (source domain) cannot achieve a good performance on real urban scenes (target domain). In this paper, we propose a weakly supervised adversarial domain adaptation to improve the segmentation performance from synthetic data to real scenes, which consists of three deep neural networks. A detection and segmentation (DS) model focuses on detecting objects and predicting segmentation map; a pixel-level domain classifier (PDC) tries to distinguish the image features from which domains; and an object-level domain classifier (ODC) discriminates the objects from which domains and predicts object classes. PDC and ODC are treated as the discriminators, and DS is considered as the generator. By the adversarial learning, DS is supposed to learn domain-invariant features. In experiments, our proposed method yields the new record of mIoU metric in the same problem.

Although traffic sign detection has been studied for years and great progress has been made with the rise of deep learning technique, there are still many problems remaining to be addressed. For complicated real-world traffic scenes, there are two main challenges. First, traffic signs are usually small-sized objects, which makes them more difficult to detect than large ones; second, it is hard to distinguish false targets which resemble real traffic signs in complex street scenes without context information. To handle these problems, we propose a novel end-to-end deep learning method for traffic sign detection in complex environments. Our contributions are as follows: 1) we propose a multi-resolution feature fusion network architecture which exploits densely connected deconvolution layers with skip connections, and can learn more effective features for a small-size object and 2) we frame the traffic sign detection as a spatial sequence classification and regression task, and propose a vertical spatial sequence attention module to gain more context information for better detection performance. To comprehensively evaluate the proposed method, we experiment on several traffic sign datasets as well as the general object detection dataset, and the results have shown the effectiveness of our proposed method.

Unsupervised feature selection is fundamentally important for processing unlabeled high-dimensional data, and several methods have been proposed on this topic. Most existing embedded unsupervised methods just emphasize the data structure in the input space, which may contain large noise. Therefore, they are limited to perceive the discriminative information implied within the low-dimensional manifold. In addition, these methods always involve several parameters to be tuned, which is time-consuming. In this paper, we present a self-tuned discrimination-aware (STDA) approach for unsupervised feature selection. The main contributions of this paper are threefold: 1) it adopts the advantage of discriminant analysis technique to select the valuable features; 2) it learns the local data structure adaptively in the discriminative subspace to alleviate the effect of data noise; and 3) it performs feature selection and clustering simultaneously with an efficient optimization strategy, and saves the additional efforts to tune parameters. Experimental results on a toy data set and various real-world benchmarks justify the effectiveness of STDA on both feature selection and data clustering, and demonstrate its promising performance against the state of the arts.

Policy evaluation algorithms are essential to reinforcement learning due to their ability to predict the performance of a policy. However, there are two long-standing issues lying in this prediction problem that need to be tackled: off-policy stability and on-policy efficiency. The conventional temporal difference (TD) algorithm is known to perform very well in the on-policy setting, yet is not off-policy stable. On the other hand, the gradient TD and emphatic TD algorithms are off-policy stable, but are not on-policy efficient. This paper introduces novel algorithms that are both off-policy stable and on-policy efficient by using the oblique projection method. The empirical experimental results on various domains validate the effectiveness of the proposed approach.

Spectral clustering has been widely used in various aspects, especially the machine learning fields. Clustering with similarity matrix and low-dimensional representation of data is the main reason of its promising performance shown in spectral clustering. However, such similarity matrix and low-dimensional representation directly derived from input data may not always hold when the data are high dimensional and has complex distribution. First, the similarity matrix simply based on the distance measurement might not be suitable for all kinds of data. Second, the low-dimensional representation might not be able to reflect the manifold structure of the original data. In this brief, we propose a novel linear space embedded clustering method, which uses adaptive neighbors to address the above-mentioned problems. Linearity regularization is used to make the data representation a linear embedded spectral. We also use adaptive neighbors to optimize the similarity matrix and clustering results simultaneously. Extensive experimental results show promising performance compared with the other state-of-the-art algorithms.

Random projection is a popular machine learning algorithm, which can be implemented by neural networks and trained in a very efficient manner. However, the number of features should be large enough when applied to a rather large-scale data set, which results in slow speed in testing procedure and more storage space under some circumstances. Furthermore, some of the features are redundant and even noisy since they are randomly generated, so the performance may be affected by these features. To remedy these problems, an effective feature selection method is introduced to select useful features hierarchically. Specifically, a novel criterion is proposed to select useful neurons for neural networks, which establishes a new way for network architecture design. The testing time and accuracy of the proposed method are improved compared with traditional methods and some variations on both classification and regression tasks. Extensive experiments confirm the effectiveness of the proposed method.

Forward Vehicle Collision Warning (FCW) is one of the most important functions for autonomous vehicles. In this procedure, vehicle detection and distance measurement are core components, requiring accurate localization and estimation. In this paper, we propose a simple but efficient forward vehicle collision warning framework by aggregating monocular distance measurement and precise vehicle detection. In order to obtain forward vehicle distance, a quick camera calibration method which only needs three physical points to calibrate related camera parameters is utilized. As for the forward vehicle detection, a multi-scale detection algorithm that regards the result of calibration as distance priori is proposed to improve the precision. Intensive experiments are conducted in our established real scene dataset and the results have demonstrated the effectiveness of the proposed framework.

Processing and fusing information among multi-modal is a very useful technique to achieving high performance in many computer vision problem. In order to tackle multi-modal information more effectively, we introduce a novel framework for multi-modal fusion: Cross-modal Message Passing (CMMP). Specifically, we propose a cross-modal message passing mechanism to fuse two-stream network for action recognition, which composes of an appearance modal network (RGB image) and a motion modal (optical flow image) network. The objectives of individual networks in this framework are two-fold: a standard classification objective and a competing objective. The classification object ensures that each modal network predicts the true action category while the competing objective encourages each modal network to outperform the other one. We quantitatively show that the proposed CMMP fuse the traditional two-stream network more effectively, and outperforms all existing two-stream fusion method on UCF-101 and HMDB-51 datasets.

Graph clustering aims to group the data into clusters according to a similarity graph, and has received sufficient attention in computer vision. As the basis of clustering, the quality of graph affects the results directly. In this paper, a Robust Adaptive Sparse Learning (RASL) method is proposed to improve the graph quality. The contributions made in this paper are three fold: (1) the sparse representation technique is employed to enforce the graph sparsity, and the l2,1 norm is introduced to improve the robustness; (2) the intrinsic manifold structure is captured by investigating the local relationship of data points; (3) an efficient optimization algorithm is designed to solve the proposed problem. Experimental results on various real-world benchmark datasets demonstrate the promising results of the proposed graph-based clustering method.

Low-rank representation (LRR) has been applied widely in most fields due to its considerable ability to explore the low-dimensional subspace embedding in high-dimensional data. However, there are still some problems that LRR can't effectively exploit the local structure and the representation for the given data is not discriminative enough. To tackle the above issues, we propose a novel locality and structure constrained low-rank representation (LSLRR) for hyperspectral image (HSI) classification. First, a distance metrics, which combines spectral and spatial similarity, is proposed to constrain the local structure. This makes two pixels in HSI with small distance have high similarity. Second, we exploit the classwise block-diagonal structure for the training data to learn the more discriminative representation for the testing data. And the experimental results verify the effectiveness and superiority of LSLRR comparing with other state-of-the-art methods.

Hyperspectral dimensionality reduction by optimal band selection attracts wide attention recently because a few pivotal and physically meaningful bands can not only represent the whole image cube without losing effectiveness but also mitigate the computational burden. In this paper, we construct an efficient searching strategy based on the clonal selection principle to optimize a geometry-based criterion named maximum ellipsoid volume (MEV). The main contributions are two-fold: 1) a subtle relationship that can accelerate the calculation of the criterion and 2) an evolutionary strategy to relieve the heavy computational burden of obtaining the desired bands from numerous quality candidates. The experimental result on a real hyperspectral data demonstrates that the proposed method is effective.

Scene classification of very high resolution remote sensing images is becoming more and more important because of its wide range of applications. However, previous works are mainly based on handcrafted features which do not have enough adaptability and expression ability. In this paper, inspired by the attention mechanism of human visual system, we propose a novel attention based network (AttNet) for scene classification. It can focus selectively on some key areas of images so that it can abandon redundant information. Essentially, AttNet gives a way to readjust the signal of supervision, and it is one of the first successful attempts on visual attention for remote sensing scene classification. Our method is evaluated on the UC Merced Land-Use Dataset, in comparison with some state-of-the-art methods. The experimental result shows that the proposed method makes a great improvement on both convergence speed and classification accuracy, and it also shows the effectiveness of visual attention for this task.

Recently, deep learning methods have dominated many fields thanks to its powerful discriminative feature learning ability. While for hyperspectral images (HSI) analysis, these deep neural networks methods suffer from overfitting as the number of labeled training samples are limited. Thus more efficient neural network architecture should be designed to improve the performance of HSI classification task. In this paper, a novel attention inception module is introduced to extract features dynamically from multi-resolution convolutional filters. The AI-NET constructed by stacking the proposed attention inception module can adaptively learn the network architecture by dynamically routing between the attention inception modules. By exploiting different spatial size convolutional filters and dynamic CNN architecture, more representative feature can be learned with limited training samples. Extensive experimental results have shown that the proposed method can adaptively adjust the network architecture and obtain better classification performance.

Deep learning is an effective tool for handling high-dimensional data and modeling nonlinearity, which can tackle the hyperspectral data well. Usually deep learning methods need a large number of training samples. However, there is no labeled data for training in change detection (CD). Considering these, this paper develops an unsupervised Robust PCA network (RPCANet) for hyperspectral image CD task. The main contributions of this work are twofold: 1) An unsupervised convolutional neural networks named RPCANet is proposed to handle the hyperspectral image CD; 2) An effective CD framework using the RPCANet and change vector analysis (CVA) is designed to achieve better CD performance with more powerful features. Experimental results on real hyperspectral data sets demonstrate the effectiveness of the proposed method.

Band selection, as an effective and popular dimensional reduction methods for hyperspectral image (HSI), has raised wide attention in recent years. In this paper, we propose a novel band selection method called optimal neighboring reconstruction (ONR). Compared to conventional methods, ONR mainly has following advantages. 1) It is globally optimal, which means the best combination of bands towards the designed objective function can be achieved. 2) It sufficiently exploits the neighboring structure among bands, so can effectively reduce the redundancy while maintaining the discrimination among bands. Experiments on three real data sets show that the proposed method has excellent performance.

Point set registration (PSR) is a crucial problem in computer vision and pattern recognition. Existing PSR methods cannot align point sets robustly due to degradations, such as deformation, noise, occlusion, outlier, and multi-view changes. In this paper, we present a self-selected regularized Gaussian fields criterion for nonrigid point matching. Unlike most existing methods, we formulate the registration problem as a sparse approximation task with low rank constraint in reproducing kernel Hilbert space (RKHS). A self-selected mechanism is used to dynamically assign real-valued label for each point in an accuracy-aware weighting manner, which makes the model focus more on the reliable points in position. Based on the label, an equivalent matching number optimization is embedded into the non-rigid criterion to enhance the reliability of the approximation. Experimental results show that the proposed method can achieve a better result in both registration accuracy and correct matches compared to state-of-the-art approaches.

Cross-scene regression tasks, such as congestion level detection and crowd counting, are useful but challenging. There are two main problems, which limit the performance of existing algorithms. The first one is that no appropriate congestion-related feature can reflect the real density in scenes. Though deep learning has been proved to be capable of extracting high level semantic representations, it is hard to converge on regression tasks, since the label is too weak to guide the learning of parameters in practice. Thus, many approaches utilize additional information, such as a density map, to guide the learning, which increases the effort of labeling. Another problem is that most existing methods are composed of several steps, for example, feature extraction and regression. Since the steps in the pipeline are separated, these methods face the problem of complex optimization. To remedy it, a deep metric learning-based regression method is proposed to extract density related features, and learn better distance measurement simultaneously. The proposed networks trained end-to-end for better optimization can be used for crowdedness regression tasks, including congestion level detection and crowd counting. Extensive experiments confirm the effectiveness of the proposed method.

Band selection, by choosing a set of representative bands in a hyperspectral image, is an effective method to reduce the redundant information without compromising the original contents. Recently, various unsupervised band selection methods have been proposed, but most of them are based on approximation algorithms which can only obtain suboptimal solutions toward a specific objective function. This paper focuses on clustering-based band selection and proposes a new framework to solve the above dilemma, claiming the following contributions: 1) an optimal clustering framework, which can obtain the optimal clustering result for a particular form of objective function under a reasonable constraint; 2) a rank on clusters strategy, which provides an effective criterion to select bands on existing clustering structure; and 3) an automatic method to determine the number of the required bands, which can better evaluate the distinctive information produced by certain number of bands. In experiments, the proposed algorithm is compared with some state-of-the-art competitors. According to the experimental results, the proposed algorithm is robust and significantly outperforms the other methods on various data sets.

Street scene understanding is an essential task for autonomous driving. One important step toward this direction is scene labeling, which annotates each pixel in the images with a correct class label. Although many approaches have been developed, there are still some weak points. First, many methods are based on the hand-crafted features whose image representation ability is limited. Second, they cannot label foreground objects accurately due to the data set bias. Third, in the refinement stage, the traditional Markov random filed inference is prone to over smoothness. For improving the above problems, this paper proposes a joint method of priori convolutional neural networks at superpixel level (called as “priori s-CNNs”) and soft restricted context transfer. Our contributions are threefold: 1) a priori s-CNNs model that learns priori location information at superpixel level is proposed to describe various objects discriminatingly; 2) a hierarchical data augmentation method is presented to alleviate data set bias in the priori s-CNNs training stage, which improves foreground objects labeling significantly; and 3) a soft restricted MRF energy function is defined to improve the priori s-CNNs model's labeling performance and reduce the over smoothness at the same time. The proposed approach is verified on CamVid data set (11 classes) and SIFT Flow Street data set (16 classes) and achieves a competitive performance.

In recent years, intelligent vehicles have been a hot topic for both research and industry communities. Since the whole system is a comprehensive integration of many advanced techniques, their respective development and improvement become fundamentally important.

Road detection from the perspective of moving vehicles is a challenging issue in autonomous driving. Recently, many deep learning methods spring up for this task, because they can extract high-level local features to find road regions from raw RGB data, such as convolutional neural networks and fully convolutional networks (FCNs). However, how to detect the boundary of road accurately is still an intractable problem. In this paper, we propose siamesed FCNs (named “s-FCN-loc”), which is able to consider RGB-channel images, semantic contours, and location priors simultaneously to segment the road region elaborately. To be specific, the s-FCN-loc has two streams to process the original RGB images and contour maps, respectively. At the same time, the location prior is directly appended to the siamesed FCN to promote the final detection performance. Our contributions are threefold: 1) An s-FCN-loc is proposed that learns more discriminative features of road boundaries than the original FCN to detect more accurate road regions. 2) Location prior is viewed as a type of feature map and directly appended to the final feature map in s-FCN-loc to promote the detection performance effectively, which is easier than other traditional methods, namely, different priors for different inputs (image patches). 3) The convergent speed of training s-FCN-loc model is 30% faster than the original FCN because of the guidance of highly structured contours. The proposed approach is evaluated on the KITTI road detection benchmark and one-class road detection data set, and achieves a competitive result with the state of the arts.

Group detection is fundamentally important for analyzing crowd behaviors, and has attracted plenty of attention in artificial intelligence. However, existing works mostly have limitations due to the insufficient utilization of crowd properties and the arbitrary processing of individuals. In this paper,we propose the Multiview-based Parameter Free (MPF) approach to detect groups in crowd scenes. The main contributions made in this study are threefold: (1) a new structural context descriptor is designed to characterize the structural property of individuals in crowd motions; (2) an self-weighted multiview clustering method is proposed to cluster feature points by incorporating their motion and context similarities;(3) a novel framework is introduced for group detection, which is able to determine the group number automatically without any parameter or threshold to be tuned. Extensive experiments on various real world datasets demonstrate the effectiveness of the proposed approach, and show its superiority against state-of-the-art group detection techniques.

The analysis of collective motion has attracted many researchers in artificial intelligence. Though plenty of works have been done on this topic, the achieved performance isstill unsatisfying due to the complex nature of collective motions. By investigating the similarity of individuals, this paper proposes a novel framework for both quantifying and detecting collective motions. Our main contributions are threefold: (1) the time-varying dynamics of individuals are deeply investigated to better characterize the individual motion; (2) a structure-based collectiveness measurement is designed toprecisely quantify both individual-level and scene-level properties of collective motions; (3) a multi-stage clustering strategy is presented to discover a more comprehensive understanding of the crowd scenes, containing both local and global collective motions. Extensive experimental results on realworld data sets show that our method is capable of handling crowd scenes with complicated structures and various dynamics, and demonstrate its superior performance against state-of-the-art competitors.

Group detection aims to classify pedestrians into categories according to their motion dynamics. It's fundamental for analyzing crowd behaviors and involves a wide range of applications. In this paper, we propose a Anchor-based Manifold Ranking (AMR) method to detect groups in crowd scenes. Our main contributions are threefold: (1) the topological relationship of individuals are effectively investigated with a manifold ranking method; (2) global consistency in crowds are accurately recognized by a coherent merging strategy; (3) the number of groups is decided automatically based on the similarity graph of individuals. Experimental results show that the proposed framework is competitive against the state-of-the-art methods.

Person re-identification is a critical yet challenging task in video surveillance which intends to match people over non-overlapping cameras. Most metric learning algorithms for person re-identification use symmetric matrix to project feature vectors into the same subspace to compute the similarity while ignoring the discrepancy between views. To solve this problem, we proposed an asymmetric cross-view matching algorithm with dictionary learning to alleviate the variations in human appearance across different views. Not only the views' dictionaries but also the persons' dictionary codes are constrained. Moreover, the `between-class' and the `within-class' distance are taken into consideration which makes the forming dictionary codes more robust and discriminative than the original feature vectors. The effectiveness of our approach is validated on the VIPeR and CUHK01 datasets. Experimental results show the proposed algorithm achieves compelling performance and asymmetric model plays an important role in the proposed approach.

In this paper, a new perspective of congestion is presented to promote the development of traffic video analysis. Our main contributions are threefold: a) An unified and quantifiable definition of congestion is proposed to describe the traffic state in video. b) Based on the definition, a congestion dataset which contains multiple traffic scenes is constructed as a platform for the research community. At the same time, a precise labeling method is introduced to get the ground truth of congestion level accurately. c) An algorithm based on Inverse Perspective Mapping (IPM) and pairwise regression is proposed to analyze traffic videos and serves as a baseline. We further compare the proposed method with two deep learning methods. Intensive experiments justify the effectiveness of the proposed method.

Dimensionality reduction plays an important role in solving the “curse of the dimensionality” and attracts a number of researchers in the past decades. In this paper, we proposed a new supervised linear dimensionality reduction method named largest center-specific margin (LCM) based on the intuition that after linear transformation, the distances between the points and their corresponding class centers should be small enough, and at the same time the distances between different unknown class centers should be as large as possible. On the basis of this observation, we take the unknown class centers into consideration for the first time and construct an optimization function to formulate this problem. In addition, we creatively transform the optimization objective function into a matrix function and solve the problem analytically. Finally, experiment results on three real datasets show the competitive performance of our algorithm.

Scene parsing is an important task in computer vision and many issues still need to be solved. One problem is about the non-unified framework for predicting things and stuff and the other one refers to the inadequate description of contextual information. In this paper, we address these issues by proposing a Hierarchical Deep Probability Analysis(HDPA) method which particularly exploits the power of probabilistic graphical model and deep convolutional neural network on pixel-level scene parsing. To be specific, an input image is initially segmented and represented through a CNN framework under Gaussian pyramid. Then the graphical models are built under each scale and the labels are ultimately predicted by structural analysis. Three contributions are claimed: unified framework for scene labeling, hierarchical probabilistic graphical modeling and adequate contextual information consideration. Experiments on three benchmarks show that the proposed method outperforms the state-of-the-arts in scene parsing.

Road detection from the perspective of moving vehicles is a challenging issue in autonomous driving. Recently, many deep learning methods spring up for this task because they can extract high-level local features to find road regions from raw RGB data, such as Convolutional Neural Networks (CNN) and Fully Convolutional Networks (FCN). However, how to detect the boundary of road accurately is still an intractable problem. In this paper, we propose a siamesed fully convolutional network (named as “s-FCN-loc”) based on VGG-net architecture, which is able to consider RGB-channel, semantic contour and location prior simultaneously to segment road region elaborately. To be specific, the s-FCN-loc has two streams to process original RGB images and contour maps respectively. At the same time, the location prior is directly appended to the last feature map to promote the final detection performance. Experiments demonstrate that the proposed s-FCN-loc can learn more discriminative features of road boundaries and converge 30% faster than the original FCN during the training stage. Finally, the proposed approach is evaluated on KITTI road detection benchmark, and achieves a competitive result.

Hyperspectral anomaly detection is playing an important role in remote sensing field. Most conventional detectors based on the Reed-Xiaoli (RX) method assume the background signature obeys a Gaussian distribution. However, it is definitely hard to be satisfied in practice. Moreover, background statistics is susceptible to contamination of anomalies in the processing windows, which may lead to many false alarms and sensitiveness to the size of windows. To solve these problems, a novel sparse dictionary learning hyperspectral anomaly detection method with capped norm constraint is proposed. Contributions are claimed in threefold: 1) requiring no assumptions on the background distribution makes the method more adaptive to different scenes; 2) benefiting from the capped norm our method has a stronger distinctiveness to anomalies; and 3) it also has better adaptability to detect different sizes of anomalies without using the sliding dual window. The extensive experimental results demonstrate the desirable performance of our method.

Band selection, by choosing a set of representative bands in hyperspectral images (HSI), is concerned to be an effective method to eliminate the “Hughes phenomenon”. In this paper, we present a global optimal clustering-based band selection (GOC) algorithm based on the hypothesis that all the bands in a cluster are continuous at their wavelengths. After the clustering result is obtained, we propose a greedy-based method to select representative bands in each cluster, trying to minimize the linear reconstruction error. Experiment on a real HSI dataset shows that the proposed method outperforms the state-of-the-art competitors.

Linear Discriminant Analysis (LDA) is a popular technique for supervised dimensionality reduction, and its performance is satisfying when dealing with Gaussian distributed data. However, the neglect of local data structure makes LDA inapplicable to many real-world situations. So some works focus on the discriminant analysis between neighbor points, which can be easily affected by the noise in the original data space. In this paper, we propose a new supervised dimensionality reduction method, Locality Adaptive Discriminant Analysis (LADA), to lean a representative subspace of the data. Compared to LDA and its variants, the proposed method has three salient advantages: (1) it finds the principle projection directions without imposing any assumption on the data distribution; (2) it’s able to exploit the local manifold structure of data in the desired subspace; (3) it exploits the points’ neighbor relationship automatically without introducing any additional parameter to be tuned. Performance on synthetic datasets and real-world benchmark datasets demonstrate the superiority of the proposed method.

Representing high-volume and high-order data is an essential problem, especially in machine learning field. Although existing two-dimensional (2D) discriminant analysis achieves promising performance, the single and linear projection features make it difficult to analyze more complex data. In this paper, we propose a novel convolutional two-dimensional linear discriminant analysis (2D LDA) method for data representation. In order to deal with nonlinear data, a specially designed Convolutional Neural Networks (CNN) is presented, which can be proved having the equivalent objective function with common 2D LDA. In this way, the discriminant ability can benefit from not only the nonlinearity of Convolutional Neural Networks, but also the powerful learning process. Experiment results on several datasets show that the proposed method performs better than other state-of-the-art methods in terms of classification accuracy.

Linear discriminant analysis (LDA) is a popular technique for supervised dimensionality reduction, but with less concern about a local data structure. This makes LDA inapplicable to many real-world situations, such as hyperspectral image (HSI) classification. In this letter, we propose a novel dimensionality reduction algorithm, locality adaptive discriminant analysis (LADA) for HSI classification. The proposed algorithm aims to learn a representative subspace of data, and focuses on the data points with close relationship in spectral and spatial domains. An intuitive motivation is that data points of the same class have similar spectral feature and the data points among spatial neighborhood are usually associated with the same class. Compared with traditional LDA and its variants, LADA is able to adaptively exploit the local manifold structure of data. Experiments carried out on several real hyperspectral data sets demonstrate the effectiveness of the proposed method.

Video-based vehicle detection and tracking is one of the most important components for intelligent transportation systems. When it comes to road junctions, the problem becomes even more difficult due to the occlusions and complex interactions among vehicles. In order to get a precise detection and tracking result, in this paper we propose a novel tracking-by-detection framework. In the detection stage, we present a sequential detection model to deal with serious occlusions. In the tracking stage, we model group behavior to treat complex interactions with overlaps and ambiguities. The main contributions of this paper are twofold: 1) shape prior is exploited in the sequential detection model to tackle occlusions in crowded scene and 2) traffic force is defined in the traffic scene to model group behavior, and it can assist to handle complex interactions among vehicles. We evaluate the proposed approach on real surveillance videos at road junctions and the performance has demonstrated the effectiveness of our method.

Anomaly detection from a driver's perspective when driving is important to autonomous vehicles. As a part of Advanced Driver Assistance Systems (ADAS), it can remind the driver about dangers in a timely manner. Compared with traditional studied scenes such as a university campus and market surveillance videos, it is difficult to detect an abnormal event from a driver's perspective due to camera waggle, abidingly moving background, drastic change of vehicle velocity, etc. To tackle these specific problems, this paper proposes a spatial localization constrained sparse coding approach for anomaly detection in traffic scenes, which first measures the abnormality of motion orientation and magnitude, respectively, and then fuses these two aspects to obtain a robust detection result. The main contributions are threefold, as follows. 1) This work describes the motion orientation and magnitude of the object, respectively, in a new way, which is demonstrated to be better than the traditional motion descriptors. 2) The spatial localization of an object is taken into account considering the sparse reconstruction framework, which utilizes the scene's structural information and outperforms the conventional sparse coding methods. 3) Results of motion orientation and magnitude are adaptively weighted and fused by a Bayesian model, which makes the proposed method more robust and able to handle more kinds of abnormal events. The efficiency and effectiveness of the proposed method are validated by testing on nine difficult video sequences that we captured ourselves. Observed from the experimental results, the proposed method is more effective and efficient than the popular competitors and yields a higher performance.

Video-based traffic sign detection, tracking, and recognition is one of the important components for the intelligent transport systems. Extensive research has shown that pretty good performance can be obtained on public data sets by various state-of-the-art approaches, especially the deep learning methods. However, deep learning methods require extensive computing resources. In addition, these approaches mostly concentrate on single image detection and recognition task, which is not applicable in real-world applications. Different from previous research, we introduce a unified incremental computational framework for traffic sign detection, tracking, and recognition task using the mono-camera mounted on a moving vehicle under non-stationary environments. The main contributions of this paper are threefold: (1) to enhance detection performance by utilizing the contextual information, this paper innovatively utilizes the spatial distribution prior of the traffic signs; (2) to improve the tracking performance and localization accuracy under non-stationary environments, a new efficient incremental framework containing off-line detector, online detector, and motion model predictor together is designed for traffic sign detection and tracking simultaneously; and (3) to get a more stable classification output, a scale-based intra-frame fusion method is proposed. We evaluate our method on two public data sets and the performance has shown that the proposed system can obtain results comparable with the deep learning method with less computing resource in a near-real-time manner.

Hyperspectral image (HSI) classification is a crucial issue in remote sensing. Accurate classification benefits a large number of applications such as land use analysis and marine resource utilization. But high data correlation brings difficulty to reliable classification, especially for HSI with abundant spectral information. Furthermore, the traditional methods often fail to well consider the spatial coherency of HSI that also limits the classification performance. To address these inherent obstacles, a novel spectral-spatial classification scheme is proposed in this paper. The proposed method mainly focuses on multitask joint sparse representation (MJSR) and a stepwise Markov random filed framework, which are claimed to be two main contributions in this procedure. First, the MJSR not only reduces the spectral redundancy, but also retains necessary correlation in spectral field during classification. Second, the stepwise optimization further explores the spatial correlation that significantly enhances the classification accuracy and robustness. As far as several universal quality evaluation indexes are concerned, the experimental results on Indian Pines and Pavia University demonstrate the superiority of our method compared with the state-of-the-art competitors.

Hyperspectral anomaly detection (AD) is an important problem in remote sensing field. It can make full use of the spectral differences to discover certain potential interesting regions without any target priors. Traditional Mahalanobis-distance-based anomaly detectors assume the background spectrum distribution conforms to a Gaussian distribution. However, this and other similar distributions may not be satisfied for the real hyperspectral images. Moreover, the background statistics are susceptible to contamination of anomaly targets which will lead to a high false-positive rate. To address these intrinsic problems, this paper proposes a novel AD method based on the graph theory. We first construct a vertex- and edge-weighted graph and then utilize a pixel selection process to locate the anomaly targets. Two contributions are claimed in this paper: 1) no background distributions are required which makes the method more adaptive and 2) both the vertex and edge weights are considered which enables a more accurate detection performance and better robustness to noise. Intensive experiments on the simulated and real hyperspectral images demonstrate that the proposed method outperforms other benchmark competitors. In addition, the robustness of the proposed method has been validated by using various window sizes. This experimental result also demonstrates the valuable characteristic of less computational complexity and less parameter tuning for real applications.

Hyperspectral image (HSI) involves vast quantities of information that can help with the image analysis. However, this information has sometimes been proved to be redundant, considering specific applications such as HSI classification and anomaly detection. To address this problem, hyperspectral band selection is viewed as an effective dimensionality reduction method that can remove the redundant components of HSI. Various HSI band selection methods have been proposed recently, and the clustering-based method is a traditional one. This agglomerative method has been considered simple and straightforward, while the performance is generally inferior to the state of the art. To tackle the inherent drawbacks of the clustering-based band selection method, a new framework concerning on dual clustering is proposed in this paper. The main contribution can be concluded as follows: 1) a novel descriptor that reveals the context of HSI efficiently; 2) a dual clustering method that includes the contextual information in the clustering process; 3) a new strategy that selects the cluster representatives jointly considering the mutual effects of each cluster. Experimental results on three real-world HSIs verify the noticeable accuracy of the proposed method, with regard to the HSI classification application. The main comparison has been conducted among several recent clustering-based band selection methods and constraint-based band selection methods, demonstrating the superiority of the technique that we present.

Saliency detection has been a hot topic in recent years, and many efforts have been devoted in this area. Unfortunately, the results of saliency detection can hardly be utilized in general applications. The primary reason, we think, is unspecific definition of salient objects, which makes that the previously published methods cannot extend to practical applications. To solve this problem, we claim that saliency should be defined in a context and the salient band selection in hyperspectral image (HSI) is introduced as an example. Unfortunately, the traditional salient band selection methods suffer from the problem of inappropriate measurement of band difference. To tackle this problem, we propose to eliminate the drawbacks of traditional salient band selection methods by manifold ranking. It puts the band vectors in the more accurate manifold space and treats the saliency problem from a novel ranking perspective, which is considered to be the main contributions of this paper. To justify the effectiveness of the proposed method, experiments are conducted on three HSIs, and our method is compared with the six existing competitors. Results show that the proposed method is very effective and can achieve the best performance among the competitors.

Crowd system has motivated a surge of interests in many areas of multimedia, as it contains plenty of information about crowd scenes. In crowd systems, individuals tend to exhibit collective behaviors, and the motion of all those individuals is called collective motion. As a comprehensive descriptor of collective motion, collectiveness has been proposed to reflect the degree of individuals moving as an entirety. Nevertheless, existing works mostly have limitations to correctly find the individuals of a crowd system and precisely capture the various relationships between individuals, both of which are essential to measure collectiveness. In this article, we propose a collectiveness-measuring method that is capable of quantifying collectiveness accurately. Our main contributions are threefold: (1) we compute relatively accurate collectiveness by making the tracked feature points represent the individuals more precisely with a point selection strategy; (2) we jointly investigate the spatial-temporal information of individuals and utilize it to characterize the topological relationship between individuals by manifold learning; (3) we propose a stability descriptor to deal with the irregular individuals, which influence the calculation of collectiveness. Intensive experiments on the simulated and real world datasets demonstrate that the proposed method is able to compute relatively accurate collectiveness and keep high consistency with human perception.

Abnormal behavior detection in crowd scenes is continuously a challenge in the field of computer vision. For tackling this problem, this paper starts from a novel structure modeling of crowd behavior. We first propose an informative structural context descriptor (SCD) for describing the crowd individual, which originally introduces the potential energy function of particle's interforce in solid-state physics to intuitively conduct vision contextual cueing. For computing the crowd SCD variation effectively, we then design a robust multi-object tracker to associate the targets in different frames, which employs the incremental analytical ability of the 3-D discrete cosine transform (DCT). By online spatial-temporal analyzing the SCD variation of the crowd, the abnormality is finally localized. Our contribution mainly lies on three aspects: 1) the new exploration of abnormal detection from structure modeling where the motion difference between individuals is computed by a novel selective histogram of optical flow that makes the proposed method can deal with more kinds of anomalies; 2) the SCD description that can effectively represent the relationship among the individuals; and 3) the 3-D DCT multi-object tracker that can robustly associate the limited number of (instead of all) targets which makes the tracking analysis in high density crowd situation feasible. Experimental results on several publicly available crowd video datasets verify the effectiveness of the proposed method.

Anomaly detection has been an important topic in hyperspectral image analysis. This technique is sometimes more preferable than the supervised target detection because it requires no a priori information for the interested materials. Many efforts have been made in this topic; however, they usually suffer from excessive time cost and a high false-positive rate. There are two major problems that lead to such a predicament. First, the construction of the background model and affinity estimation are often overcomplicated. Second, most of these methods have to impose a stringent assumption on the spectrum distribution of background; however, these assumptions cannot hold for all practical situations. Based on this consideration, this paper proposes a novel method allowing for fast yet accurate pixel-level hyperspectral anomaly detection. We claim the following main contributions: construct a high-order 2-D crossing approach to find the regions of rapid change in the spectrum, which runs without any a priori assumption; and design a low-complexity discrimination framework for fast hyperspectral anomaly detection, which can be implemented by a series of filtering operators with linear time cost. Experiments on three different hyperspectral images containing several pixel-level anomalies demonstrate the superiority of the proposed detector compared with the benchmark methods.

Hyperspectral images have been proved to be effective for a wide range of applications; however, the large volume and redundant information also bring a lot of inconvenience at the same time. To cope with this problem, hyperspectral band selection is a pertinent technique, which takes advantage of removing redundant components without compromising the original contents from the raw image cubes. Because of its usefulness, hyperspectral band selection has been successfully applied to many practical applications of hyperspectral remote sensing, such as land cover map generation and color visualization. This paper focuses on groupwise band selection and proposes a new framework, including the following contributions: 1) a smart yet intrinsic descriptor for efficient band representation; 2) an evolutionary strategy to handle the high computational burden associated with groupwise-selection-based methods; and 3) a novel MTSP-based criterion to evaluate the performance of each candidate band combination. To verify the superiority of the proposed framework, experiments have been conducted on both hyperspectral classification and color visualization. Experimental results on three real-world hyperspectral images demonstrate that the proposed framework can lead to a significant advancement in these two applications compared with other competitors.

Hyperspectral image classification is one of the most significant topics in remote sensing. A large number of methods have been proposed to improve the classification accuracy. However, the improvement often comes at the cost of higher complexity. In this work, we mainly focus on the Markov Random Fields related paradigm, which involves a demanding energy minimization procedure. Traditional methods are prone to employ the advanced optimization techniques. On the contrary, this paper is in defense of a simple yet efficient method for hyperspectral image classification, Iterated Conditional Mode, which has been generally considered inferior to other state-of-the-art methods. Our purpose is successfully achieved by tackling two inherent drawbacks of ICM, sensitive label initialization and local minimum. We apply our method to three real-world hyperspectral images, and compare the results with those of state-of-the-art methods. The comparisons show that the proposed method outperforms its competitors.

Visual tracking in condition of occlusion, appearance or illumination change has been a challenging task over decades. Recently, some online trackers, based on the detection by classification framework, have achieved good performance. However, problems are still embodied in at least one of the three aspects: 1) tracking the target with a single region has poor adaptability for occlusion, appearance or illumination change; 2) lack of sample weight estimation, which may cause overfitting issue; and 3) inadequate motion model to prevent target from drifting. For tackling the above problems, this paper presents the contributions as follows: 1) a novel part-based structure is utilized in the online AdaBoost tracking; 2) attentional sample weighting and selection is tackled by introducing a weight relaxation factor, instead of treating the samples equally as traditional trackers do; and 3) a two-stage motion model, multiple parts constraint, is proposed and incorporated into the part-based structure to ensure a stable tracking. The effectiveness and efficiency of the proposed tracker is validated upon several complex video sequences, compared with seven popular online trackers. The experimental results show that the proposed tracker can achieve increased accuracy with comparable computational cost.

Although numerous trackers have been designed to adapt to the nonstationary image streams that change over time, it remains a challenging task to facilitate a tracker to accurately distinguish the target from the background in every frame. This paper proposes a robust superpixel-based tracker via depth fusion, which exploits the adequate structural information and great flexibility of mid-level features captured by superpixels, as well as the depth-map's discriminative ability for the target and background separation. By introducing graph-regularized sparse coding into the appearance model, the local geometrical structure of data is considered, and the resulting appearance model has a more powerful discriminative ability. Meanwhile, the similarity of the target superpixels' neighborhoods in two adjacent frames is also incorporated into the refinement of the target estimation, which helps a more accurate localization. Most importantly, the depth cue is fused into the superpixel-based target estimation so as to tackle the cluttered background with similar appearance to the target. To evaluate the effectiveness of the proposed tracker, four video sequences of different challenging situations are contributed by the authors. The comparison results demonstrate that the proposed tracker has more robust and accurate performance than seven ones representing the state-of-the-art.

Saliency detection has raised much interest in computer vision recently. Many visual saliency models have been developed for individual images, video clips, and image pairs. However, image sequence, one most general occasion in the real world, is not explored yet. A general image sequence is different from video clips whose temporal continuity is maintained and image pairs where common objects exist. It might contain some similar low-level properties while completely distinct contents. Traditional saliency detection methods will fail on these general sequences. Based on this consideration, this paper investigates the shortcomings of the classical saliency detection methods, which significantly limit their advantages: 1) inability to capture the natural connections among sequential images, 2) over-reliance on motion cues, and 3) restriction to image pairs/videos with common objects. In order to address these problems, we propose a framework that performs the following contributions: 1) construct an image data set as benchmark through a rigorously designed behavioral experiment, 2) propose a neural activity trace aware saliency model to capture the general connections among images, and 3) design a novel measure to handle the low-level clues contained among sequential images. Experimental results demonstrate that the proposed saliency model is associated with a tremendous advancement compared with traditional methods when dealing with the general image sequence.

Automatic detection of salient objects in visual media (e.g., videos and images) has been attracting much attention. The detected salient objects can be utilized for segmentation, recognition, and retrieval. However, the accuracy of saliency detection remains a challenge. The reason behind this challenge is mainly due to the lack of a well-defined model for interpreting saliency formulation. To tackle this problem, this letter proposes to detect salient objects based on selective contrast. Selective contrast intrinsically explores the most distinguishable component information in color, texture, and location. A large number of experiments are thereafter carried out upon a benchmark dataset, and the results are compared with those of 12 other popular state-of-the-art algorithms. In addition, the advantage of the proposed algorithm is also demonstrated in a retargeting application.

Saliency detection has been a hot topic in recent years. Its popularity is mainly because of its theoretical meaning for explaining human attention and applicable aims in segmentation, recognition, etc. Nevertheless, traditional algorithms are mostly based on unsupervised techniques, which have limited learning ability. The obtained saliency map is also inconsistent with many properties of human behavior. In order to overcome the challenges of inability and inconsistency, this paper presents a framework based on multiple-instance learning. Low-, mid-, and high-level features are incorporated in the detection procedure, and the learning ability enables it robust to noise. Experiments on a data set containing 1000 images demonstrate the effectiveness of the proposed framework. Its applicability is shown in the context of a seam carving application.

Saliency detection has been an attractive topic in recent years. The reliable detection of saliency can help a lot of useful processing without prior knowledge about the scene, such as content-aware image compression, segmentation, etc. Although many efforts have been spent in this subject, the feature expression and model construction are far from perfect. The obtained saliency maps are therefore not satisfying enough. In order to overcome these challenges, this paper presents a new psychologic visual feature based on differential threshold and applies it in a supervised Markov-random-field framework. Experiments on two public data sets and an image retargeting application demonstrate the effectiveness, robustness, and practicability of the proposed method.

The phenomenon of color discrepancy between image pairs happens frequently in stereo vision systems. This inconsistence in color domain may cause difficulties when identifying point correspondence to reconstruct the scene depth. In this paper, we propose a robust algorithm to correct the color discrepancy between images. The proposed algorithm neither requires a color calibration chart/object which is a tedious procedure, nor explicitly compensates for the image as a whole, which possibly give bad correction results in local areas of an image. Instead, we correct the image region by region. Experiments show that the presented color correction algorithm is effective and efficient.