The large variation of viewpoint and irrelevant content around the target always hinder accurate image retrieval and its subsequent tasks. In this paper, we investigate an extremely challenging task: given a ground-view image of a landmark, we aim to achieve cross-view geo-localization by searching out its corresponding satellite-view images. Specifically, the challenge comes from the gap between ground-view and satellite-view, which includes not only large viewpoint changes (some parts of the landmark may be invisible from front view to top view) but also highly irrelevant background (the target landmark tend to be hidden in other surrounding buildings), making it difficult to learn a common representation or a suitable mapping. To address this issue, we take advantage of drone-view information as a bridge between ground-view and satellite-view domains. We propose a Peer Learning and Cross Diffusion (PLCD) framework. PLCD consists of three parts: 1) a peer learning across ground-view and drone-view to find visible parts to benefit ground-drone cross-view representation learning; 2) a patch-based network for satellite-drone cross-view representation learning; 3) a cross diffusion between ground-drone space and satellite-drone space. Extensive experiments conducted on the University-Earth and University-Google datasets show that our method outperforms state-of-the-arts significantly.

Most recent approaches for the zero-shot cross-modal image retrieval map images from different modalities into a uniform feature space to exploit their relevance by using a pre-trained model. Based on the observation that manifolds of zero-shot images are usually deformed and incomplete, we argue that the manifolds of unseen classes are inevitably distorted during the training of a two-stream model that simply maps images from different modalities into a uniform space. This issue directly leads to poor cross-modal retrieval performance. We propose a bi-directional random walk scheme to mining more reliable relationships between images by traversing heterogeneous manifolds in the feature space of each modality. Our proposed method benefits from intra-modal distributions to alleviate the interference caused by noisy similarities in the cross-modal feature space. As a result, we achieved great improvement in the performance of the thermal v.s. visible image retrieval task. The code of this paper: https://github.com/fyang93/cross-modal-retrieval

Diffusion is commonly used as a ranking or re-ranking method in retrieval tasks to achieve higher retrieval performance, and has attracted lots of attention in recent years. A downside to diffusion is that it performs slowly in comparison to the naive k-NN search, which causes a non-trivial online computational cost on large datasets. To overcome this weakness, we propose a novel diffusion technique in this paper. In our work, instead of applying diffusion to the query, we precompute the diffusion results of each element in the database, making the online search a simple linear combination on top of the k-NN search process. Our proposed method becomes 10∼ times faster in terms of online search speed. Moreover, we propose to use late truncation instead of early truncation in previous works to achieve better retrieval performance.

In this paper we present a novel approach to improve temporal matching kernel (TMK) for video retrieval tasks. TMK has the ability to align videos during retrieval, but provides little to none retrieval performance improvement over baseline methods. We discovered that TMK cannot discriminate between a true match case in which two videos have long, consecutive segments of similar frames and a false match case in which two videos contain non-consecutive segments of randomly similar frames. Our proposed burst-survive temporal matching kernel adopts a novel shuffle strategy to rule out false match cases, with the assistance of multiple periods selected from Fibonacci series. As a result, we achieved significant performance improvement on the EVVE dataset.

It is common that TV audiences want to quickly browse scenes with certain actors in TV series. Since 2016, the TREC Video Retrieval Evaluation (TRECVID) Instance Search (INS) task has started to focus on identifying a target person in a target scene simultaneously. In this paper, we name this kind of task as P-S INS (Person-Scene Instance Search). To find out P-S instances, most approaches search person and scene separately, and then directly combine the results together by addition or multiplication. However, we find that person and scene INS modules are not always effective at the same time, or they may suppress each other in some situations. Aggregating the results shot after shot is not a good choice. Luckily, for the TV series, video shots are arranged in chronological order. We extend our focus from time point (single video shot) to time slice (multiple consecutive video shots) in the time-line. Through detecting salient time slices, we prune the data. Through evaluating the importance of salient time slices, we boost the aggregation results. Extensive experiments on the large-scale TRECVID INS dataset demonstrate the effectiveness of the proposed method.

Person re-identification (REID) is an important task in video surveillance and forensics applications. Most of previous approaches are based on a key assumption that all person images have uniform and sufficiently high resolutions. Actually, various low-resolutions and scale mismatching always exist in open world REID. We name this kind of problem as Scale-Adaptive Low Resolution Person Re-identification (SALR-REID). The most intuitive way to address this problem is to increase various low-resolutions (not only low, but also with different scales) to a uniform high-resolution. SR-GAN is one of the most competitive image super-resolution deep networks, designed with a fixed upscaling factor. However, it is still not suitable for SALR-REID task, which requires a network not only synthesizing high-resolution images with different upscaling factors, but also extracting discriminative image feature for judging person’s identity. (1) To promote the ability of scale-adaptive upscaling, we cascade multiple SRGANs in series. (2) To supplement the ability of image feature representation, we plug-in a reidentification network. With a unified formulation, a Cascaded Super-Resolution GAN (CSR-GAN) framework is proposed. Extensive evaluations on two simulated datasets and one public dataset demonstrate the advantages of our method over related state-of-the-art methods.

We propose a fast event retrieval method in video databases based on temporal match kernel. The basic idea of the baseline is to maximize the score function for all possible relative timestamps. However, considering the stability and computational cost, we simplify the similarity score by calculating the energy of the score function. In this way, maximizing the score function which is time-consuming and sensitive to noise is avoided. We derive the simplified energy formulation by using Parseval's theorem, which describes the unitarity of Fourier transform. Further, by our formulation, the problem becomes a simple nearest neighbor search, and we can use product quantization (PQ) to accelerate the computation. We evaluate our method on EVVE dataset for event retrieval. The experimental results show that our approach is much faster than the baseline, with a higher mAP as well. Comparison with state of the art demonstrates the efficacy of our approach.

This paper presents a method that embeds stability-sensitive filter in the temporal matching kernel with explicit feature maps (TE). The added filter improves the robustness of TE to noise for content-based video retrieval. Originally TE embeds temporal information of frame descriptors by using explicit feature mapping in a fixed length video vector by using a temporal invariant match kernel. TE matches relevant videos and at the same time estimates the temporal offset by referring only to these vectors. It has achieved state-of-the-art performance in video event retrieval. However, since TE defines video similarity as a sum of frame-wise similarities, we argue that TE is not so robust to noise. Usually matching videos have consecutive frames in correspondence, which result in series of high similarities. A simple sum of frame-wise similarities cannot differentiate this situation from that of non-consecutive high similarities which seems to lead to false matches. To circumvent this problem, we propose to apply a non-linear low-pass filter to the frame-wise similarities. In order to enjoy the benefit of the TE framework, we further develop a technique to embed the filter into frame descriptors. Consequently the video similarity can be evaluated as the dot product of video vectors with a stability-sensitive filter enabled. As the original TE, our method is seamlessly compatible with query expansion with temporal consistency checking. Our approach is evaluated on the EVVE dataset for particular event retrieval, it achieves state-of-the-art performance.