In recent years, the rapid development of remote sensing (RS) technology has led to a drastic increase in the availability of RS images. This calls for the need to develop new methods able to effectively and efficiently retrieve the required instances from a massive amount of RS imagery. In retrieval tasks, finding the nearest-neighbor sample of the retrieval query is a fundamental research topic. Exhaustive comparison is the simplest method to accomplish this task. However, due to the involved computational complexity and memory limitations, this solution is no longer feasible in large data retrieval tasks. As an important branch of approximate nearest-neighbor retrieval (NNR), hash algorithms transform high-dimensional data into low-bit expressions (hash codes) with elements of 0 and 1 to reduce storage and computational costs. Hash algorithms aim to preserve the same nearest-neighbor relationship between the learned hash codes and the original data. Existing hash algorithms are divided into two classes: shallow and deep methods. Furthermore, deep hash algorithms can be divided into (semi-) supervised and unsupervised algorithms. In this article, representative hash-based RS image retrieval (HBRSIR) methods are reviewed, studying the application of hashing in other areas of the RS community and introducing available datasets and evaluation metrics for RS image retrieval (RSIR). The performance of representative and cross-modal hashing methods is validated using two common RSIR datasets (UCMerced and AID) and a cross-modal dataset (DSRSID). Prospects of future work summarizing HBRSIR are also provided.

Hashing for localization (HfL) is an effective method for fast localizing specific scenes in a large-scale remote sensing image. Key to its efficiency arises from a comprehensive deep hashing network that generates representational binary hash codes for image patches cropped from the remote sensing image. On the other hand, this paper will investigate the problem of encrypting the remote sensing image against the HfL task. We refer to the new task as encrypting hashing against localization (EHaL). We characterize the EHaL task in term of two cues: (I) An encrypted image patch is supposed to appear as visually similar to its original image patch as possible; (II) The hash code generated by the deep hashing network for the encrypted image patch is supposed to be not close to its original class but close to a different class. Following the two cues, we develop an encrypted patch generator, which is trained in an adversarial fashion. Based on the encrypted patch generator, we propose two remote sensing image encryption frameworks that can cause non-localization and mis-localization to the HfL task separately. Experiments validates the effectiveness of our method. Reproducible executions are given at https://github.com/JingpengHan/EHaL.

In this paper, we undertake the task of fast geo-localization of a query ground image by using geo-tagged aerial images. To this end, we propose a hashing strategy that fast searches the database of geo-tagged aerial images for the ground image’s matches, whose geo-tags are exploited to estimate the ground geographic location. Specifically, we commence by converting the aerial images into ground-view aerial images that have the common angle of view (i.e., horizontal view) with the ground image. We then develop a feature extraction model and a hash encoder for generating hash codes for the images. Based on these models, the ground image and the geo-tagged aerial images are transformed to hash codes that comprehensively reflect their visual content similarity. Fast searching the geo-tagged aerial image database for the ground image’s matches is conducted subject to small Hamming distance between the hash codes. We extract a geographical cluster from the matched aerial images subject to their geo-tags. In this way, the geographic location of the ground image is efficiently retrieved according to the geographical cluster. Experiments on two datasets validate the efficiency and effectiveness of our proposed framework. We have released our implementation code at https://github.com/taoyiminR/Hashing_for_geo-localization for public evaluation.

The point process is a solid framework to model sequential data, such as videos, by exploring the underlying relevance. As a challenging problem for high-level video understanding, weakly supervised action recognition and localization in untrimmed videos have attracted intensive research attention. Knowledge transfer by leveraging the publicly available trimmed videos as external guidance is a promising attempt to make up for the coarse-grained video-level annotation and improve the generalization performance. However, unconstrained knowledge transfer may bring about irrelevant noise and jeopardize the learning model. This article proposes a novel adaptability decomposing encoder-decoder network to transfer reliable knowledge between the trimmed and untrimmed videos for action recognition and localization by bidirectional point process modeling, given only video-level annotations. By decomposing the original features into the domain-adaptable and domain-specific ones based on their adaptability, trimmed-untrimmed knowledge transfer can be safely confined within a more coherent subspace. An encoder-decoder-based structure is carefully designed and jointly optimized to facilitate effective action classification and temporal localization. Extensive experiments are conducted on two benchmark data sets (i.e., THUMOS14 and ActivityNet1.3), and the experimental results clearly corroborate the efficacy of our method.

We investigate the problem of efficiently localizing sketch depicted scenes in a remote sensing image dataset. We pose the problem as that of remote sensing image retrieval with sketch queries and explore the use of hashing techniques to achieve efficient retrieval. Given two training datasets of sketches and remote sensing images that have a common set of class labels, we develop a hashing strategy that coconstructs two hash code books for the sketches and the remote sensing images separately. The hash code book coconstruction strategy encourages hash codes for the sketches and remote sensing images from different classes to be far away from one another and those from the same class to be close. This property is maintained by two cohesion intensive cues: 1) an interclass pairwise disperse cue (InterPDC) and 2) an intraclass pairwise balance cue (IntraPBC). We use the two coconstructed hash code books for training two linear mapping models that generate hash codes for sketches and remote sensing images separately. Sorting the Hamming distance between the sketch hash codes and the remote sensing image hash codes renders efficient remote sensing image retrieval with sketch queries. This enables localizing the sketch depicted scenes in the remote sensing image dataset. In addition, our method can also be used for fast localizing sketch depicted scenes in a remote sensing image of large size. Extensive experiments on public datasets validate the effectiveness and efficiency of our method.

Advanced remote-sensing instruments produce massively large scenes from the surface of the earth, with very high spatial resolution and dimensionality. Developing methods for efficiently localizing specific objects in a large-scale scene presents a significant challenge, mainly because of the high computational requirements involved. To tackle this issue, we propose a new hashing for localization (HfL) framework that efficiently searches for specific objects in the large-scale scene. It begins by dividing the scene into a large number of overlapping local patches. A lightweight deep hash model, referred to as a tiny hashing network (THNet), encodes the local patches into hash codes. The Hamming distances between the hash code of an object image, i.e., an image containing the specific class of objects to be localized in the scene, and those of all local patches are computed. Small values of the Hamming distance indicate local patches that are similar to the object image. The positions of these local patches in the large-scale scene reflect the regional locations of the specific objects. The hash codes are binary and do not take up much space, and the Hamming distance carries very low-computational overheads. Further, we exploit a class center loss as the THNet training objective, which can comprehensively manage multiple object classes. These features mean that the HfL framework can localize specific objects very quickly, regardless of the size of the scene. Extensive experiments validate the effectiveness and efficiency of the framework. For instance, HfL can find objects in a remote-sensing image of 19584$\times$19584 pixels in only 4.388 s (on a single RTX2080ti), with remarkable localization results. The source codes and datasets are available athttps://github.com/lrhan/HfL, together providing a baseline for fast localizing objects in a large-scale scene.

Weakly supervised action recognition and localization for untrimmed videos is a challenging problem with extensive applications. The overwhelming irrelevant background contents in untrimmed videos severely hamper effective identification of actions of interest. In this paper, we propose a novel multi-instance multi-label modeling network based on spatio-temporal pre-trimming to recognize actions and locate corresponding frames in untrimmed videos. Motivated by the fact that person is the key factor in a human action, we spatially and temporally segment each untrimmed video into person-centric clips with pose estimation and tracking techniques. Given the bag-of-instances structure associated with video-level labels, action recognition is naturally formulated as a multi-instance multi-label learning problem. The network is optimized iteratively with selective coarse-to-fine pre-trimming based on instance-label activation. After convergence, temporal localization is further achieved with local-global temporal class activation map. Extensive experiments are conducted on two benchmark datasets, i.e. THUMOS14 and ActivityNet1.3, and experimental results clearly corroborate the efficacy of our method when compared with the state-of-the-arts.

Fast and accurate remote sensing image retrieval from large data archives has been an important research topic in the remote sensing research literature. Recently, hashing-based remote sensing image retrieval has attracted extreme attention because of its efficient search capabilities. Especially, deep remote sensing image hashing algorithms have been developed based on convolutional neural networks (CNNs) and have shown effective retrieval performance. However, implementing a deep hashing network tends to be highly expensive in terms of storage space and computing resources to be suitable for on-orbit remote sensing image retrieval, which usually operates on resource-limited devices such as satellites and unmanned aerial vehicles (UAVs). To address this limitation, we propose to hash a deep network that in turn hashes remote sensing images. Specifically, we develop a quantized deep learning to hash (QDLH) framework for large-scale remote sensing image retrieval. The weights and activation functions in the QDLH framework are binarized to low-bit representations, which require comparatively much less storage space and computing resources. The QDLH results in a lightweight deep neural network for effective remote sensing image hashing. We conduct extensive experiments on two public remote sensing image data sets by incorporating several state-of-the-art network architectures into our QDLH methodology for remote sensing image hashing. The experimental results demonstrate that the proposed QDLH is effective in saving hardware resources in terms of both storage and computation. Moreover, superior remote sensing image retrieval performance is also achieved by our QDLH, compared with state-of-the-art deep remote sensing image hashing methods.

Learning to hash is a popular technique for fast similarity search on a large-scale image database. However, many hashing methods do not achieve satisfactory results because of the quantization loss in the straightforward binary code generation procedure. In order to address this problem, we propose a novel hash code reconstruction framework for existing unsupervised hashing methods. In our proposed approach, the hash codes are generated through reconstructing the original images with relaxed hamming vector representation, such that the final learned codes will be more approximate to characterize the intrinsic image structure. Moreover, our proposed hash code reconstruction algorithm is very efficient for computing, which can be generalized to various hashing methods. Extensive experiments are conducted on four public image datasets by incorporating our proposed scheme with different hashing methods, and the comparison results have shown that significant performance improvements can be achieved with minor additional time cost for fast similarity search task.

With the rapid progress of satellite and aerial vehicle technologies, large-scale remote sensing (RS) image retrieval has recently become an important research issue in geosciences. Hashing-based searching approaches have been widely employed in content-based image retrieval tasks. However, most hash schemes compromise between learning efficiency and retrieval accuracy, and can thus barely satisfy the precise requirements in RS data analysis. To address these shortcomings, we introduce a partial randomness scheme for learning hash functions, which is referred to as partial randomness hashing (PRH). Specifically, for constructing hash functions, a part of model parameter values are randomly generated and the remaining ones are trained based on RS images. The randomness enables an efficient hash function construction and the trained model parameters encode characteristics from RS images. The coplay between random and trained model parameters results in both efficient and effective learning scheme for constructing hash functions. Experiments on two large public RS image data sets have shown that our PRH method outperforms state of the arts in terms of both learning efficiency and retrieval accuracy.