In this paper, aiming at two key problems of instance-level image retrieval, i.e., the distinctiveness of image representation and the generalization ability of the model, we propose a novel deep architecture - Multiple Saliency and Channel Sensitivity Network(MSCNet). Specifically, to obtain distinctive global descriptors, an attention-based multiple saliency learning is first presented to highlight important details of the image, and then a simple but effective channel sensitivity module based on Gram matrix is designed to boost the channel discrimination and suppress redundant information. Additionally, in contrast to most existing feature aggregation methods, employing pre-trained deep networks, MSCNet can be trained in two modes: the first one is an unsupervised manner with an instance loss, and another is a supervised manner, which combines classification and ranking loss and only relies on very limited training data. Experimental results on several public benchmark datasets, i.e., Oxford buildings, Paris buildings and Holidays, indicate that the proposed MSCNet outperforms the state-of-the-art unsupervised and supervised methods.

Convolutional Neural Network (CNN) has been proven to be an effective feature extractor for multiple computer vision tasks such as image classification and object detection etc. However, image retrieval in realistic scenarios, usually faces large-scale unlabeled datasets, thus the learning of a good model is often infeasible. In this paper, we propose a novel and interpretable image representation via spatial-channel weighting for aggregated deep convolutional features. Specifically, we first determine discriminative regions of an image by computing the Indicator matrix, and then, the distinctive features are extracted from salient areas by calculating the Gram matrix, in which high-order features are learnt. Finally, a compact image representation is generated by fusing spatial saliency and channel sensitivity of CNN features. The experimental results on several benchmark datasets, i.e., Oxford buildings, Paris buildings and Holidays, indicate that the proposed approach outperforms state-of-the-art methods based on pre-trained deep networks.