Multimodal emotion recognition is gaining prominence as the fusion of heterogeneous signals enhances both robustness and accuracy. Current vision-based approaches rarely incorporate objective neural measurements to achieve high-accuracy identification, partly because of the common disparity in modality convergence within multimodal systems. To bridge this gap, we introduce a prototype-based method for rebalancing modalities that adaptively modulates the learning pace of each one, facilitating a more synergistic integration. Evaluations on two public affective computing datasets confirm the efficacy of our approach; it attains leading-edge results in modality-constrained settings and presents a viable solution for emotion recognition.

The remote sensing image scene classification continues to face significant challenges due to high intraclass diversity and interclass similarity. Existing methods mainly use semantic associations between images to establish deep semantic associations between classes, ignoring the rich high-level semantic knowledge contained in the label text. This high-level information are especially valuable for distinguishing between confusing categories, as it enables the model to capture both similarity and distinctive features effectively. In this article, we introduce a novel approach that incorporates label semantic information and proposes a plug-and-play framework to guide classification model learning of intraclass and interclass relationships. Specifically, our framework includes a dynamic soft label module (DSLM), which uses textual semantics to facilitate classification model learning of interclass relationships via soft labels at the target level. In addition, we design a coarse-to-fine contrastive module (CFCM) to integrate textual semantics into contrastive learning, guiding the model in capturing intraclass and interclass relationships at the feature level. Our framework is compatible with both convolutional neural network (CNN)-based and vision transformer (ViT)-based classification architectures and is employed solely during training to minimize computational overhead. Experimental results on four datasets validate the effectiveness of our approach.

Remote sensing scene classification is an important yet challenging task of remote sensing image interpretation. In recent years, the development of convolutional neural networks (CNNs) has significantly improved the accuracy of this study. However, most of the methods based on CNNs use discrete label representation, which ignores the potential correlations between scenes, resulting in insufficient generalization ability of the model. This paper proposes a model based on inter-class association information mining, named Correlation Label Embedding Network (CLENet). Specifically, we mine inter-class association information from the network’s predictions and continuously adjust them by back-propagation during the network training process to obtain continuous label representations. Unlike other methods, we distill the potential correlations between scenes as the supervisory signal of the network to guide the feature selection process of the network. This forces the output of the network to learn more scene-related features to adapt to complex application scenarios and improve the generalization performance of the model. Additionally, to enhance the discriminative nature of the model, we design regular metric terms based on the learned interclass association information. Compared with the state-of-the-art scene classification methods, the experimental results validate the potential of CLENet models on remote sensing image scene classification.

Vehicle detection in infrared aerial images is vital for both military and civilian applications, as infrared imaging remains effective under low-light conditions and various adverse weather scenarios. However, the longer wavelengths of long-wave infrared, compared to visible light, make diffraction more noticeable, leading to low-frequency degradation of vehicle information. Thermal radiation from the environment and optical system leads to higher noise in infrared images. Additionally, atmospheric transport models for various weather conditions can degrade infrared images to different extents. These factors lead to a reduced signal-to-noise ratio, which complicates the extraction of clear features. To overcome these challenges, we propose the Dual-Tuning Network (DTNet), an advanced framework for vehicle detection in infrared aerial images, developed based on the mechanisms of infrared imaging. Specifically, the core component of DTNet is the Dual-Tuning Block (DTBlock), which works alongside the Dynamic Guided Filtering Module (DGFM) and the Point Spread Recovery Module (PSRM) for feature extraction. DTBlock decomposes feature maps into low- and high-frequency components with learnable low-pass filters. DGFM eliminates disturbance from the optical system and background thermal radiation in the high-frequency component of feature maps, while preserving the details and texture of vehicles. PSRM aggregates vehicle features in the low-frequency component of feature maps, which is proposed with reference to diffraction and atmospheric models. The concept of Dual-Tuning refers to enhancing the signal and suppressing interference in the low and high frequency parts, respectively. Experimental results on the DroneVehicle public dataset for infrared vehicle detection indicate that our proposed approach achieves state-of-the-art (SOTA) performance. Moreover, extensive ablation studies confirm the superior capability of our DTNet in robust feature extraction from infrared images.

Point clouds obtained from 3D scanners are often noisy and cannot be directly used for subsequent high-level tasks. In this article, we propose a novel point cloud optimization method capable of denoising and homogenizing point clouds. Our idea is based on the assumption that the noise is generally much smaller than the effective signal. We perform noise perturbation on the noisy point cloud to get a new noisy point cloud, called self-variation point cloud. The noisy point cloud and self-variation point cloud have different noise distribution, but the same point cloud distribution. We compute the potential commonality between two noisy point clouds to obtain a clean point cloud. To implement our idea, we propose a Self-Variation Capture Network (SVCNet). We perturb the point cloud features in the latent space to obtain self-variation feature vectors, and capture the commonality between two noisy feature vectors through the feature aggregation and averaging. In addition, an edge constraint module is introduced to suppress low-pass effects during denoising. Our denoising method does not take into account the noise characteristics, and can filter the drift noise located on the underlying surface, resulting in a uniform distribution of the generated point cloud. The experimental results show that our algorithm outperforms the current state-of-the-art algorithms, especially in generating more uniform point clouds. In addition, extended experiments demonstrate the potential of our algorithm for point clouds upsampling.

Multi-frame matching decision is a key step of high-precision scene matching navigation. The traditional decision algorithms mainly adopts hierarchical filtering method. When dealing with multi-frame sequences with low matching quality, it is hard to meet the precision requirements. In view of this problem, an intelligent multi-frame matching consistency decision algorithm based on SVM_bagging is proposed. Firstly, the features of correlation plane formed by matching are split and combined to form a variety of training datasets and train multiple independent support vector machine (SVM) models based on the bagging idea of ensemble learning. Then, a hard voting mechanism is adopted to combine the decision results of all SVM models for consistency decision. The experimental results on Sentinel_1-2 optical/SAR data show that the proposed algorithm can further reduce the false positive rate and false negative rate by more than 60% on the basis of the traditional decision algorithm.

For ship targets in down-sight conditions, the aspect ratio is uneven. Therefore, horizontal box labeling can lead to the interference of too much background information, making the target modeling inaccurate. To address this issue, a Rotated Discriminative Scale Space Tracker (roDSST) based on DSST is proposed. Firstly, the sampling method has been refined. The rotated target area is sampled using bilinear interpolation to reduce background interference and avoid quantization errors. Secondly, an angle filter is introduced to predict the rotation angle of the bounding box. Finally, the template update mechanism is improved by always retaining a certain weight of the initial template. The experimental results show that roDSST has superior performance in tracking accuracy and robustness on downward-looking ship sequences.

Infrared sensors are highly valuable in both civil and military applications due to their immunity to weather and time conditions. Among the various targets in infrared remote sensing images, vehicle targets are the most prevalent and significant. However, Existing deep learning-based detection algorithms face challenges such as high false alarm rates and low efficiency when applied to infrared weak vehicle targets detection. To address the aforementioned issues, we propose a Multi-scale Adaptive Feature Enhancement Network, named MAFE-Net, which focuses on highlighting significant features of vehicles and the correlation between targets and their surrounding environments. It consists of two significant designs: the self-adaptive coordinate attention (SCA) for feature enhancement and the multi-scale self-attention module (MSM) for feature fusion. In the meantime, we created the IRSWV dataset, which comprises 4,770 infrared aircraft-captured images. Comparing our algorithm with other current mainstream target detection algorithms on the IRSWV dataset, our algorithm demonstrates significant advantages.

Remote sensing ship detection is of great importance for the applications of maritime transport and precision guidance. However, the background interference on land, the large similarity between ship classes, and the arbitrary orientation of ships all pose serious challenges for inshore ship detection. To solve the problem of complex and diverse land backgrounds and sea conditions, we use the contextual attention module to enhance the ship target features. In addition, a rotation-invariant feature extraction and fusion module is introduced to make the features directionally invariant and increase the inter-class distance by fusing global features with local features. Finally, the feature decoupled module is redesigned to resolve the conflict between target detection and fine-grained recognition. Experimental results show that our proposed integrated model for inshore ship detection and fine-grained recognition improves mAP by about 6% compared to the current baseline model.

Aircraft detection in synthetic aperture radar (SAR) image is a very important but challenging question. Due to the multi-scale characteristics of aircraft and the complex background of airports in SAR images, the detection process often encounters challenges of false alarms and missed detections. Meanwhile, the presence of SAR image noise and the discrete distribution characteristics of scatterers can also contribute to incomplete aircraft target detection results. To address these problems, we proposed a novel multi-scale approach based on Swin Transformer. It employs a shifted window-based self-attention mechanism to extract the correlated features between scatter points. Moreover, to better enhance and integrate the multi-scale information among various level features, we incorporated an enhanced neck network with a four-layer feature pyramid and proposed an Adaptive Feature Fusion Network (AFFN) in our approach. Experiments are conducted on the GaoFen-3 (GF3) SAR aircraft datasets, and the results show the effectiveness of the proposed method.

Recently, few-shot learning (FSL) has received increasing attention because of difficulties in sample collection in some application scenarios, such as maritime surveillance using synthetic aperture radar (SAR) or infrared images. In real situations of such scenarios, it is a common requirement that the model can recognize novel classes incrementally, namely class-incremental learning (CIL). Considering the above requirement, a novel problem that recognizes novel classes incrementally when both the base and novel class samples are scarce is proposed in this article. It is called complete few-shot CIL (C-FSCIL) for distinguishing from the FSCIL that assumes sufficient samples of base classes. Specifically, the following challenges of C-FSCIL are focused on: 1) distance measurement is used for recognizing novel classes incrementally, but the encoder is difficult to be learned well when base class samples are scarce, making some features unsuitable for calculating the distance, decreasing the performance and 2) the catastrophic forgetting problem becomes more difficult to be alleviated than that in FSCIL because of the scarcity of base class samples. To tackle both challenges, mine-distill-prototypes (MDP) algorithm is proposed, which consists of two parts: 1) prototypes-distillation (PD) network is proposed to learn to distill the features and prototypes into a lower dimensional in which ineffective features are eliminated and 2) the prototypes-weight (PW) network and the prototypes-selection (PS) training strategy are proposed for the catastrophic forgetting problem, which aims to capture the relationship between the base and novel prototypes. The superior performance of the proposed algorithm is demonstrated by the experiments on three datasets.

Infrared vehicle detection based on aerial images has significant applications in military and civilian fields for the perception ability under low-light and foggy conditions. However, it remains challenging due to the following characteristics. First, infrared textures and edges are blurred, implying a plenty of low-frequency signals and a shortage of detailed descriptions. Second, objects in infrared images present different patterns depending on their thermal radiations, which hamper the feature extraction of convolution kernels. Third, infrared images lack color information, which means fewer features for classification and regression can be used. Inspired by cognitive neuroscience that humans perceive the entirety from low-frequency information and discern details from high-frequency information, we devise a new framework for oriented infrared vehicle detection called I2MDet (Infrared Information Mining Detector) to tackle the above challenges. It consists of two significant designs: the kaleidoscope module and the semantic feature supplement module (SFSM). In the kaleidoscope module, we explore the effect of kernel sizes and dilation rates on frequency information mining with kaleidoscope-like equivalent kernels. Features in this module are extracted by adaptive involution operators instead of convolution kernels to deal with multiple patterns. The SFSM provides the network with features beneficial for classification and regression. On the one hand, the network is guided to learn more meaningful features under semantic supervision. On the other hand, features output by the SFSM supplement the detection head with semantic information. Experimental results on the public dataset DroneVehicle demonstrate that our proposed approach achieves outstanding performance on oriented infrared vehicle detection.

Aircraft detection in remote sensing images plays an essential role in military and civil applications. Many aircraft in remote sensing images are small targets and are covered by cloud and fog, which make it difficult to extract sufficient feature information for aircraft detection. In this paper, a single-stage object detection model YOLM (You can Look More) has achieved better remote sensing aircraft detection performance by extracting more features. YOLM includes an enhanced neck network with a four-layer feature pyramid and a path aggregation network. More layers enable the model to extract more detailed information. To use the background information around the aircraft as a supplement to aircraft features, an attention module which can obtain the image context information of aircraft is designed. Experiments are conducted on the aircraft subset of FAIR1M data set and some aircraft images we obtained, and the proposed model has out-performed many baseline methods such as YOLO V5s.

Aircraft is easily covered by clouds in optical remote sensing images. It is a challenge to detect the aircraft and recognize its sub-categories in this situation. However, the methods proposed by the current research are mainly applied to high-quality images, which do not perform well on cloudy images. In this paper, an adaptive feature aggregation network called AFA-Net is proposed to solve this problem. We design a mixed self-attention module that adaptively focuses on the uncovered parts of the aircraft and its neighborhood from space and channel in feature maps. Experiments were done on the Optical Image Aircraft Detection and Recognition Data Set of the 3rdTianzhibei Challenge. Compared with the most advanced object detection algorithms, the proposed approach achieves state-of-the-art performance.

As an important part of the air defense and anti-missile system, the surface-to-air missile sites (SAMSs) have important military application value. The existing deep learning-based detection algorithms have the problems of high false alarm rate and low efficiency when applied to large-scale remote sensing images. To address this issue, in this work we propose a multi-task detection and recognition network, including classification branch and detection branch. The classification branch selects the suspected target area from the large-scale remote sensing image, and the detection branch detects and recognizes the target in the suspected area, achieving precise positioning from coarse to fine. In addition, we propose a density clustering algorithm to post-process the detection results, which effectively reduces the false alarm rate of the algorithm. Finally, we propose a SAMS detection and recognition dataset (DSAMS), which divides the SAMSs into three parts: the launch fielding, the launch bunker and the control and guide plain. Comparing our algorithm with other current mainstream target detection algorithms on the DSAMS dataset, our algorithm has significant advantages.

Military-civilian attribute recognition of ships in synthetic aperture radar (SAR) imagery plays an important role in marine surveillance. However, high-quality labeled data are hard to obtain for SAR ships, which hinder the development of deep learning models. Considering that models directly transferred from labeled optical images cannot achieve satisfactory performance for SAR applications due to the great discrepancy of different modalities, we propose a two-stage transfer learning method by combining the data-level and feature-level knowledge transfer. First, CycleGAN is adopted in the first stage to transfer the labeled optical image domain to the intermediate SAR-like image domain with the attribute labels. Then, a novel network called Domain Transfer using Adversarial learning and Metric learning (DTAM) is proposed to realize the task of military-civilian ship recognition by the domain adaption of the intermediate domain and the target SAR domain with joint adversarial learning and metric learning. To validate the proposed method, we establish a high-resolution SAR ship recognition dataset (HRSSRD), containing SAR and optical images of military and civilian ships. The experimental results show that the proposed two-stage architecture exhibits promising performance on the problem of SAR military-civilian ship recognition.

Image paragraph captioning (IPC) aims to generate a fine-grained paragraph to describe the visual content of an image.Significant progress has been made by deep neural networks, in which the attention mechanism plays an essential role.However, conventional attention mechanisms tend to ignore the past alignment information, which often results in problems of repetitive captioning and incomplete captioning.In this paper, we propose an Interactive key-value Memoryaugmented Attention model for image Paragraph captioning (IMAP) to keep track of the attention history (salient objects coverage information) along with the update-chain of the decoder state and therefore avoid generating repetitive or incomplete image descriptions.In addition, we employ an adaptive attention mechanism to realize adaptive alignment from image regions to caption words, where an image region can be mapped to an arbitrary number of caption words while a caption word can also attend to an arbitrary number of image regions.Extensive experiments on a benchmark dataset (i.e., Stanford) demonstrate the effectiveness of our IMAP model.

Vehicle detection is one of the hot topics in lane detection and vehicle counting. Many works have been done on it and some data sets with satellite images and aerial images are proposed. However, the ratio of the vehicle targets to the background is small and the detection results are unsatisfied. In this paper, a bottom enhanced RetinaNet model named En-RetinaNet is proposed to get better performance on vehicle detection. The EnRetinaNet includes an enhanced feature pyramid network(FPN) and a bottom-top fusion before the region proposal network. Enhanced feature pyramid network adds a bottom layer to the feature pyramid network to exploit more local features. Bottom-top fusion is utilized to get a better fusion of the bottom layers and top layers. In order to get a high ratio of the objects to the images, we take a sliding window mechanism on the testing images. We discuss the effects of the training input crop size on the final results and choose a moderate size of the training input. With all the above work done, we get an improvement on the UCAS—AOD data set in contrast to the RetinaNet.

Ship detection and fine-grained recognition in large-format remote sensing image are an important research direction in the field of remote sensing image detection. But less research has been done in this area. Aiming at this problem, this paper constructs a large-format remote sensing image ship target dataset with ship category information, and proposes a background filtering network and a ship fine-grained classification network. The background filtering network is used to quickly filter out the background area, and the ship fine-grained classification network is used to detect ship targets and distinguish ship categories. Compared with the previous method, the method proposed in this paper can significantly improve the efficiency of ship target detection in large-format remote sensing images, while also improving the detection accuracy.

Recent image captioning approaches are typically trained on generation-based or retrieval-based approaches. Both methods have their advantages but limited by the disadvantages. In this paper, we propose a Unified Generation-Retrieval framework for Image Captioning (UGRIC) by using adversarial learning. Different from previous methods, the proposed UGRIC model leverages the informative contents of N-best response candidates provided by the retrieval-based model to enhance the generation-based method. In addition, to further improve the informativeness of the generated caption, we employ copying mechanism to choose words from the retrieved candidate captions and put them into proper positions of the output sequence. Experiments on MSCOCO dataset demonstrate the effectiveness of the UGRIC model through various evaluation metrics.\footnoteCode and data are available at: \urlhttp://tinyurl.com/y6z2x6ho.

Saliency object detection has been a very active research topic recently, due to its extensive applications in image compression, scene understanding, image retrieval, and so forth. The overwhelming majority of existing computational models are designed based on computer vision techniques by using a lot of image cues and priors. In fact, salient object detection is derived from the biological perceptual mechanism, and biological evidence shows that the object-based saliency stems from the spread of the spatial attention. Inspired by this, we attempt to utilize the emerging spread mechanism of object attention to construct a new computational model. A novel Laplacian regularized kernel regression diffusion model is proposed to fulfill the spread process. The proposed diffusion model, which is able to fully capture both global and local structures of the image, thereby allows for effective propagation of spatial attention with visual grouping cues, yielding a well-structured object-based saliency map. Experimental results demonstrate that our method can achieve encouraging performance in comparison with the state-of-the-art methods.

Extraction of objects such as buildings from very high resolution(VHR) remote sensing imagery is an important task nowadays. In practical application, the extraction precision is normally satisfied through interactive manual input. Therefore, we propose a semi-automatic building extraction framework with an energy minimization model, which includes two stages: the first stage generates the coarse information of foreground and background, and the second stage extracts the building object by applying energy minimization model. More specifically, in the first stage, the VHR imagery is grouped into small superpixels which are roughly merged into background or foreground according to a line drawn manually. Based on the coarse information of the first stage, we apply the energy minimization model to obtain the precise building object using graph cut optimization. Experimental results indicate our building extraction framework can precisely extract the buildings with different shapes.

This letter proposes a simple but effective approach to automatically learn a multilayer image feature for satellite image scene classification. Different from the hand-crafted features which are empirically designed but lack high generalization ability, the proposed approach can autonomously extract the data-dependent feature. The presented feature extraction algorithm is composed of two layers, and the bases of these two layers are uniformly learned by a plain $K$-means clustering algorithm. Coincidentally, the feature extraction performance of the aforementioned two layers is consistent with visual processing of human visual cortex. More specifically, the first layer can generate edgelike bases, which are analogous to the neuron responses of primary visual cortex (V1), and the second layer can produce cornerlike bases, which resemble the neuron responses of visual extrastriate cortical area two (V2). The proposed feature extraction approach can automatically extract not only simple structure features (e.g., edges) but also complex structure features (e.g., corners and junctions). The learned feature is further discriminated by the linear support vector machine classifier for scene classification. In order to fairly demonstrate the validity of the proposed feature extraction approach, its satellite image scene classification performance is evaluated on the public UCM-21 data set. Experimental results show that the proposed approach can outperform several recent state-of-the-art approaches.

This letter proposes a novel supervised approach for accurate built-up area detection from high-resolution remote sensing images. In existing supervised built-up area detection approaches based on block-based image interpretation, the determination of the block size and the pursuit of the pixel-level result are not well addressed. Concerning these issues, this letter proposes a complete and systematic approach. It first utilizes multikernel learning to incorporate multiple features to implement the block-level image interpretation. Then, multifield integrating (i.e., the image interpretation results using different block sizes are fused) is proposed to obtain the block-level result. On the basis of the achieved result of the second step, multihypothesis voting is finally presented for working toward the pixel-level built-up area detection result through multihypothesis superpixel representation and graph smoothing. The proposed approach has been validated in the ZY-3 and GF-1 satellite images, and experimental results show that the proposed approach can outperform the state-of-the-art approaches.

With the development of high-resolution imagery, ship detection in optical satellite images has attracted a lot of research interest because of the broad applications in fishery management, vessel salvage, etc. Major challenges for this task include cloud, wave, and wake clutters, and even the variability of ship sizes. In this letter, we propose an unsupervised ship detection method toward overcoming these existing issues. Visual saliency, which focuses on highlighting salient signals from scenes, is applied to extract candidate regions followed by a homogeneous filter presented to confirm suspected ship targets with complete profiles. Then, a novel descriptor, ship histogram of oriented gradient, which characterizes the gradient symmetry of ship sides, is provided to discriminate real ships. Experimental results on numerous panchromatic satellite images demonstrate the good performance of our method compared to state-of-the-art methods.

Autonomous underwater vehicles (AUVs) have been widely used in diverse contexts, especially military affairs. The smooth operation of the AUV requires accurate localization of surrounding objects, especially the aerial objects. In this letter, a novel and practical method is presented for aerial object localization by using gravity and gravity gradient anomaly. Different from the state-of-the-art methods, such as GPS, radar, and laser, the proposed method runs in a passive manner and achieves AUV invisibility without energy emission. Compared with the object localization methods based on gravity and gravity gradient inversion, the proposed method is more practical as no large area gravity and gravity gradient measurements are needed to estimate the object mass. Experimental results demonstrate that the proposed method performs better than the existing methods.

In this letter, we propose a simple yet effective approach, named Progressive Vector Field Consensus (PVFC), for addressing the problem of finding more true feature correspondences between images. The key idea is to progressively perform feature matching based on Vector Field Consensus, and hence greatly boost the number of true matches as well as avoid false matches. More specifically, it uses matching results on a small putative correspondence set with high inlier ratio to guide the matching on a large putative correspondence set which probably covers the whole true correspondences. We model the transformation between images in a reproducing kernel Hilbert space, and a sparse approximation is applied to the transformation to avoid high computational complexity. Our results quantitatively show that our PVFC outperforms state-of-the-art methods, both in accuracy and in efficiency. Moreover, the progressive framework is general and can be applied to other cases for robust estimation.

Feature matching, which refers to establishing reliable correspondence between two sets of features (particularly point features), is a critical prerequisite in feature-based registration. In this paper, we propose a flexible and general algorithm, which is called locally linear transforming (LLT), for both rigid and nonrigid feature matching of remote sensing images. We start by creating a set of putative correspondences based on the feature similarity and then focus on removing outliers from the putative set and estimating the transformation as well. We formulate this as a maximum-likelihood estimation of a Bayesian model with hidden/latent variables indicating whether matches in the putative set are outliers or inliers. To ensure the well-posedness of the problem, we develop a local geometrical constraint that can preserve local structures among neighboring feature points, and it is also robust to a large number of outliers. The problem is solved by using the expectation-maximization algorithm (EM), and the closed-form solutions of both rigid and nonrigid transformations are derived in the maximization step. In the nonrigid case, we model the transformation between images in a reproducing kernel Hilbert space (RKHS), and a sparse approximation is applied to the transformation that reduces the method computation complexity to linearithmic. Extensive experiments on real remote sensing images demonstrate accurate results of LLT, which outperforms current state-of-the-art methods, particularly in the case of severe outliers (even up to 80%).

In this paper, a novel model based on feature activity weighted decorrelation cues is proposed for visual saliency detection in natural images. It consists of two parts: the feature decorrelation and feature information-activity. For the first part, Laplacian sparse coding and low-rank decomposition are used to extract decorrelated features from the scenes. For the second part, Incremental Coding Length is applied to measure the information-activity contained in features, which is then employed to weight the decorrelated features. Finally, visual saliency is estimated through a max pooling strategy. Experimental results on a publicly available benchmark demonstrate the effectiveness of our proposed model with good performance against the state-of-the-art methods.

Gravity navigation, with its independent, passive, concealment and all weather, has become one of the best options of aided inertial navigation system (INS). Precise local gravity or gravity anomaly reference maps will greatly improve the accuracy of autonomous underwater vehicles (AUVs) navigation. Due to the lack of measured gravity data, the previous methods generally used digital elevation model (DEM) to model gravity anomaly reference maps, however, which neglected the impact of terrain density in homogeneity. In this paper, a novel and practical method is proposed for modeling a reference map which takes a full consideration of the terrain density difference. Experimental results show that the proposed method performs better than the existing methods.

This paper addresses the automatic building extraction problem from high-resolution remote sensing images. The buildings in remote sensing images generally represent different shapes (i.e., simple rectangular or complex hybrid shape), it is intractable to extract all the buildings with different shapes once. Therefore, we adopt a hierarchical extraction style with a visual graphical topic model embedded, which includes two stages: the first stage detects the simple rectangular buildings and the second stage extracts the complex hybrid buildings. More specifically, the first stage is mainly responsible for regular buildings detection, and unsupervised visual graphical topic model (i.e., replicated softmax restricted boltzmann machine) and supervised discriminative model learning, and the second stage is mainly in charge of complex buildings extraction using the learned semantic feature mapping and discriminative model. Experimental results show that the second stage can obviously improve the building detection rate with slightly increasing the false alarm rate.

To improve computational efficiency in synthetic aperture radar (SAR) image matching, a fast image matching method using a novel two-step searching strategy (coarse-to-fine) is proposed in this letter. This method is based on two-column histogram (TCH) hashing and improved random sample consensus (RANSAC). First, coarse matching is conducted using a novel TCH hashing, which is notable for its robustness and speed. Compared with the discrete cosine transform used in perceptual hashing, TCH describes SAR images more accurately and rapidly. Then, in the refining stage, key points are detected and described in the coarser scales using scale-invariant feature transform. The Euclidean distance strategy and the improved RANSAC based on prior energy function (P-RANSAC) are then employed to implement matching. On the basis of prior information, a model of energy function has been constructed to improve sampling strategy. Experimental results on various SAR images show that the proposed approach outperforms the state-of-the-art algorithms in SAR image matching.

The smooth operation of autonomous underwater vehicles (AUVs) relies heavily on the accurate detection of surrounding objects. Toward this end, this letter presents a novel method for underwater object detection based on the gravity gradient differential and the gravity gradient differential ratio caused by the relative motion between the AUV and the object. Unlike the existing techniques, the proposed method works in a passive manner and achieves AUV invisibility without energy emission. In addition, for the proposed method, no gravity map or gravity gradient map is required, which improves its practicality. Experimental results demonstrate that the proposed method performs better than the existing methods.

Visual saliency is attracting more and more research attention since it is beneficial to many computer vision applications. In this paper, we propose a novel bottom-up saliency model for detecting salient objects in natural images. First, inspired by the recent advance in the realm of statistical thermodynamics, we adopt a novel mathematical model, namely, the maximal entropy random walk (MERW) to measure saliency. We analyze the rationality and superiority of MERW for modeling visual saliency. Then, based on the MERW model, we establish a generic framework for saliency detection. Different from the vast majority of existing saliency models, our method is built on a purely region-based strategy, which is able to yield high-resolution saliency maps with well preserved object shapes and uniformly highlighted salient regions. In the proposed framework, the input image is first over-segmented into superpixels, which are taken as the primary units for subsequent procedures, and regional features are extracted. Then, saliency is measured according to two principles, i.e., uniqueness and visual organization, both implemented in a unified approach, i.e., the MERW model based on graph representation. Intensive experimental results on publicly available datasets demonstrate that our method outperforms the state-of-the-art saliency models.

In this paper, we propose an efficient algorithm, called vector field consensus, for establishing robust point correspondences between two sets of points. Our algorithm starts by creating a set of putative correspondences which can contain a very large number of false correspondences, or outliers, in addition to a limited number of true correspondences (inliers). Next, we solve for correspondence by interpolating a vector field between the two point sets, which involves estimating a consensus of inlier points whose matching follows a nonparametric geometrical constraint. We formulate this a maximum a posteriori (MAP) estimation of a Bayesian model with hidden/latent variables indicating whether matches in the putative set are outliers or inliers. We impose nonparametric geometrical constraints on the correspondence, as a prior distribution, using Tikhonov regularizers in a reproducing kernel Hilbert space. MAP estimation is performed by the EM algorithm which by also estimating the variance of the prior model (initialized to a large value) is able to obtain good estimates very quickly (e.g., avoiding many of the local minima inherent in this formulation). We illustrate this method on data sets in 2D and 3D and demonstrate that it is robust to a very large number of outliers (even up to 90%). We also show that in the special case where there is an underlying parametric geometrical model (e.g., the epipolar line constraint) that we obtain better results than standard alternatives like RANSAC if a large number of outliers are present. This suggests a two-stage strategy, where we use our nonparametric model to reduce the size of the putative set and then apply a parametric variant of our approach to estimate the geometric parameters. Our algorithm is computationally efficient and we provide code for others to use it. In addition, our approach is general and can be applied to other problems, such as learning with a badly corrupted training data set.

We present a new point matching algorithm for robust nonrigid registration. The method iteratively recovers the point correspondence and estimates the transformation between two point sets. In the first step of the iteration, feature descriptors such as shape context are used to establish rough correspondence. In the second step, we estimate the transformation using a robust estimator called L_2E. This is the main novelty of our approach and it enables us to deal with the noise and outliers which arise in the correspondence step. The transformation is specified in a functional space, more specifically a reproducing kernel Hilbert space. We apply our method to nonrigid sparse image feature correspondence on 2D images and 3D surfaces. Our results quantitatively show that our approach outperforms state-of-the-art methods, particularly when there are a large number of outliers. Moreover, our method of robustly estimating transformations from correspondences is general and has many other applications.

Infrared small-target detection plays an important role in image processing for infrared remote sensing. In this letter, different from traditional algorithms, we formulate this problem as salient region detection, which is inspired by the fact that a small target can often attract attention of human eyes in infrared images. This visual effect arises from the discrepancy that a small target resembles isotropic Gaussian-like shape due to the optics point spread function of the thermal imaging system at a long distance, whereas background clutters are generally local orientational. Based on this observation, a new robust directional saliency-based method is proposed incorporating with visual attention theory for infrared small-target detection. Experimental results demonstrate that the proposed algorithm outperforms the state-of-the-art methods for real infrared images with various typical complex backgrounds.

Given a set of high-resolution remote sensing images covering different scenes, we propose an unsupervised approach to simultaneously detect possible built-up areas from them. The motivation behind is that the frequently recurring appearance patterns or repeated textures corresponding to common objects of interest (e.g., built-up areas) in the input image data set can help us discriminate built-up areas from others. With this inspiration, our method consists of two steps. First, we extract a large set of corners from each input image by an improved Harris corner detector. Afterward, we incorporate the extracted corners into a likelihood function to locate candidate regions in each input image. Given a set of candidate build-up regions, in the second stage, we formulate the problem of build-up area detection as an unsupervised grouping problem. The candidate regions are modeled through texture histogram, and the grouping problem is solved by spectrum clustering and graph cuts. Experimental results show that the proposed approach outperforms the existing algorithms in terms of detection accuracy.

This letter presents an image deformation method based on Moving Regularized Least Squares optimization. The user controls the deformation by simply choosing a set of point handles in the input image, and also the target positions that the source point handles should be deformed to. The deformation function in our method is nonrigid and specified in a functional space, more specifically a reproducing kernel Hilbert space. The proposed method possesses three characteristics: 1) it is able to create detail-preserving and intuitive deformations; 2) the solution of the deformation function has a simple closed-form; 3) it is extremely computationally efficient which can be performed in real-time (less than 0.1 milliseconds per frame for an image of size 500 ×500). We compare our method to a state-of-the-art method which is modeled by rigid transformations; the qualitative and quantitative results demonstrate the benefits of using the nonrigid formulation in aspects of both accuracy and efficiency. Moreover, the proposed method is general and it can be applied to other applications for interpolation.

We propose a method for removing mismatches from given putative point correspondences in image pairs. Our algorithm aims to recover the underlying coherent spatial mapping which related to inliers. The thin-plate spline (TPS) is chosen to parameterize the coherent spatial mapping, and we formulate the solution of it as a maximum likelihood problem. The mismatches could be successfully removed after the EM algorithm, which we used for solving the problem, converges. The quantitative results on various experimental data demonstrate that our method outperforms many state-of-the-art methods. Moreover, the proposed method is also able to handle the case that image pairs contain non-rigid motions.

We propose a method for vector field learning with outliers, called vector field consensus (VFC). It could distinguish inliers from outliers and learn a vector field fitting for the inliers simultaneously. A prior is taken to force the smoothness of the field, which is based on the Tiknonov regularization in vector-valued reproducing kernel Hilbert space. Under a Bayesian framework, we associate each sample with a latent variable which indicates whether it is an inlier, and then formulate the problem as maximum a posteriori problem and use Expectation Maximization algorithm to solve it. The proposed method possesses two characteristics: 1) robust to outliers, and being able to tolerate 90% outliers and even more, 2) computationally efficient. As an application, we apply VFC to solve the problem of mismatch removing. The results demonstrate that our method outperforms many state-of-the-art methods, and it is very robust.

In this paper, we present a robust and efficient approach to extract motion layers from a pair of images with large disparity motion. First, motion models are established as: 1) initial SIFT matches are obtained and grouped into a set of clusters using our developed topological clustering algorithm; 2) for each cluster with no less than three matches, an affine transformation is estimated with least-square solution as tentative motion model; and 3) the tentative motion models are refined and the invalid models are pruned. Then, with the obtained motion models, a graph cuts based layer assignment algorithm is employed to segment the scene into several motion layers. Experimental results demonstrate that our method can successfully segment scenes containing objects with large interframe motion or even with significant interframe scale and pose changes. Furthermore, compared with the previous method invented by Wills and its modified version, our method is much faster and more robust.

Anomaly detection in hyperspectral images is investigated using local tangent space alignment (LTSA) for dimensionality reduction (DR) in conjunction with a minimum distance detector. The LTSA is implemented for large images by constructing a manifold with training data and employing the out-of-sample extension for testing data. The training data that should represent all the background types are generated by the recursive hierarchical segmentation (RHSEG) algorithm and the elimination of the very small segments that may represent anomalies. Experimental results indicate that the LTSA is able to distinguish anomalies from background using a small number of features in the embedded space, and the LTSA-based detector has superior anomaly detection performance to the well-known RX and kernel RX detectors.

A novel method of underwater object detection based on gravity gradient is presented, which can be used on autonomous underwater vehicles (AUVs) to detect abnormal objects underwater. Gravity gradient anomalies of partial area, which are caused by the object, can be measured by a gravity gradiometer on an AUV. Then, anomalies can be inversed with a gravity gradient inversion algorithm, so the mass and barycenter of an object can be estimated. Simulation results show that approximate information of an object can be provided by the proposed method.

Approaches to combine local manifold learning (LML) and thek-nearest-neighbor (kNN) classifier are investigated for hyperspectral image classification. Based on supervised LML (SLML) andkNN, a new SLML-weightedkNN (SLML-WkNN) classifier is proposed. This method is appealing as it does not require dimensionality reduction and only depends on the weights provided by the kernel function of the specific ML method. Performance of the proposed classifier is compared to that of unsupervised LML (ULML) and SLML for dimensionality reduction in conjunction with thekNN (ULML-kNN and SLML-kNN). Three LML methods, locally linear embedding (LLE), local tangent space alignment (LTSA), and Laplacian eigenmaps, are investigated with these classifiers. In experiments with Hyperion and AVIRIS hyperspectral data, the proposed SLML-WkNN performed better than ULML-kNN and SLML-kNN, and the highest accuracies were obtained using weights provided by supervised LTSA and LLE.