Instance-level image retrieval is the task of searching in a large database for images that match an object in a query image. To address this task, systems usually rely on a retrieval step that uses global image descriptors, and a subsequent step that performs domain-specific refinements or reranking by leveraging operations such as geometric verification based on local features. In this work, we propose Reranking Transformers (RRTs) as a general model to incorporate both local and global features to rerank the matching images in a supervised fashion and thus replace the relatively expensive process of geometric verification. RRTs are lightweight and can be easily parallelized so that reranking a set of top matching results can be performed in a single forward-pass. We perform extensive experiments on the Revisited Oxford and Paris datasets, and the Google Landmarks v2 dataset, showing that RRTs outperform previous reranking approaches while using much fewer local descriptors. Moreover, we demonstrate that, unlike existing approaches, RRTs can be optimized jointly with the feature extractor, which can lead to feature representations tailored to downstream tasks and further accuracy improvements. The code and trained models are publicly available at https://github.com/uvavision/RerankingTransformer.

Outfit recommendation requires the answers of some challenging outfit compatibility questions such as 'Which pair of boots and school bag go well with my jeans and sweater?'. It is more complicated than conventional similarity search, and needs to consider not only visual aesthetics but also the intrinsic fine-grained and multi-category nature of fashion items. Some existing approaches solve the problem through sequential models or learning pair-wise distances between items. However, most of them only consider coarse category information in defining fashion compatibility while neglecting the fine-grained category information often desired in practical applications. To better define the fashion compatibility and more flexibly meet different needs, we propose a novel problem of learning compatibility among multiple tuples (each consisting of an item and category pair), and recommending fashion items following the category choices from customers. Our contributions include: 1) Designing a Mixed Category Attention Net (MCAN) which integrates both fine-grained and coarse category information into recommendation and learns the compatibility among fashion tuples. MCAN can explicitly and effectively generate diverse and controllable recommendations based on need. 2) Contributing a new dataset IQON, which follows eastern culture and can be used to test the generalization of recommendation systems. Our extensive experiments on a reference dataset Polyvore and our dataset IQON demonstrate that our method significantly outperforms state-of-the-art recommendation methods.

Product quantization (PQ) and residual quantization (RQ) have been successfully used to solve fast nearest neighbor search problems thanks to their exponentially reduced complexities of both storage and computation with respect to the codebook size, Recent efforts have been focused on employing optimization strategies and seeking more effective models. Based on the observation that randomness typically increases in subsequent residual spaces, we propose a new strategy, called, transformed RQ (TRQ), that jointly learns a local transformation per residual cluster with an ultimate goal to further reduce overall quantization errors. Additionally we propose a hybrid approximate nearest search method based on the proposed TRQ and PQ. We show that our methods achieve significantly better accuracy on nearest neighbor search than both the original and the optimized PQ on several benchmark datasets.

In contrast to classic supervised learning methods that demand pre-defined class labels, pairwise encoding or side-information encoding merely requires pairwise similarity information to drive feature learning, which makes it very appealing for many fundamental tasks such as dimensionality reduction and semi-supervised learning. In this paper, we present a novel bimarginal pairwise encoding model, along with deep autoencoder, to learn nonlinear embedding for the aforementioned tasks. The new method learns powerful features that preserve critical pairwise information in a semi-supervised manner. It has achieved better performance on the well-known yet hard to make improvement benchmark MINIST compared with other methods in the same category, i.e. Autoencoder [4], Invariant Mapping for Dimensionality Reduction [1], Neighborhood Component Analysis [3], and Fixed Bi-Margin Pairwise Encoding [11].

The product quantization (PQ) performance degrades on read-world data due to the severity of dependence between feature groups. Meanwhile, tree structured vector quantization (TSVQ) often supply lower distortion than other structured vector quantizers; yet it is prohibitive to learning compact codes like PQ does considering its codebook storage. In this paper, we propose a hybrid model dubbed as product tree quantization (PTQ) that aims to relax the PQ constraints while to retain the tree-structured codebooks with reasonable size. We first show that our methods can achieve significantly better quantization performance on several large scale benchmarks. We then demonstrate the advantage for very large scale ANN search; for instance, on a 1-billion scale dataset, we have achieved on average 4% improvement in accuracy than the existing state of the art methods.

This paper presents BABAZ, an audio search system to search modified segments in large databases of music or video tracks. It is based on an efficient audio feature matching system which exploits the reciprocal nearest neighbors to produce a per-match similarity score. Temporal consistency is taken into account based on the audio matches, and boundary estimation allows the precise localization of the matching segments. The method is mainly intended for video retrieval based on their audio track, as typically evaluated in the copy detection task of TRECVID evaluation campaigns. The evaluation conducted on music retrieval shows that our system is comparable to a reference audio fingerprinting system for music retrieval, and significantly outperforms it on audio-based video retrieval, as shown by our experiments conducted on the dataset used in the copy detection task of TRECVID'2010 campaign.

This paper presents a k-means based algorithm for approximate nearest neighbor search. The proposed Embedded k-Means algorithm is a two-level clustered index structure which consists of two groups of centroids; additionally, an inverted file is used for recording of the assignments. The coarse-to-fine structure achieves high search efficiency using multi-assignment operations on the coarse level. At the query stage, pruning strategies are utilized to balance the trade-off between search qualities and speeds. Our algorithm is able to explore the neighborhood space of a given query efficiently. Due to its good recall/selectivity and memory efficiency, the proposed algorithm is scalable and is able to process very large databases. Experimental results on SIFT and GIST image descriptor datasets show search performance better and comparable to the state-of-the-art approaches with lower memory usage and complexity.