Contrastive Language–Image Pre-training (CLIP) jointly optimizes an image encoder and a text encoder, yet its semantic supervision can blur the distinction between visually different images that share similar captions, hurting instance-level image retrieval. We study two strategies, two-stage fine-tuning (2SFT) and multi-caption-image pairing (MCIP) that strengthen CLIP models for content-based image retrieval while preserving their cross-modal strengths. 2SFT first adapts the image encoder for retrieval and then realigns the text encoder. MCIP injects multiple pseudo-captions per image so that class labels sharpen retrieval and the extra captions keep text alignment. This extended version augments the original SISAP24 study with experiments on additional models, a systematic investigation of key hyperparameters of the presented approach, insights into the effects of the methods on the model, and more a detailed report on training setting and costs. Across four CLIP model families, the proposed methods boost image-to-image retrieval accuracy without sacrificing text-to-image performance, simplifying large-scale multimodal search systems by allowing them to store one embedding per image while being effective in image-to-image and text-to-image search.

1199

As datasets and the dimensionality of feature vectors continue to grow, Approximate Nearest Neighbor Search (ANNS) in large multimedia databases becomes increasingly relevant. Graph-based approaches have demonstrated to offer the best trade-off between retrieval precision and search time. Despite their ability to deliver search times several orders of magnitude faster than exact search techniques, existing methods suffer from slow constructions speeds or high memory requirements. This paper presents a continuous refining Exploration Graph (crEG), a novel approach for rapidly constructing a compact exploration graph with state-of-the-art search performance. Additionally, it provides the ability to enhance its effectiveness even further through an optional edge optimization algorithm. Both algorithms are specifically designed to produce and operate on undirected graphs with even degrees and guarantee graph connectivity at any time - a property particularly valuable for exploratory search, where the query is part of the database elements. Although such queries provide an advantageous starting point for graph search algorithms, they have been rarely considered in the context of ANNS, yet are crucial for recommendation and exploration systems. Our experiments demonstrate high efficiency in ANNS does not necessarily translate to a good performance in exploratory search.

The increasing amount of visual data shared online highlights the importance of organizing and finding related content. However, current efforts to improve visual search and image classification lack support for exploratory image search. Sorting images by similarity offers a solution, allowing users to view several hundred images at once.

Due to the size of today’s image collections it can be challenging to fully understand their content. Recent technological advances have enabled efficient visual search. These systems use joint visual and textual feature vectors to identify similar images based on image queries or text descriptions. Despite their effectiveness, high-dimensional feature vectors can lead to long search times for large collections. In this demonstration, we propose a solution that significantly reduces search times and increases the efficiency of the search system. By combining two separate image graphs, our method provides fast approximate nearest neighbor search and allows seamless visual exploration of the entire collection in real time through a standard web browser, using familiar navigation techniques such as zooming and dragging, common in systems like Google Maps.

Recent advances in computer vision research led to large vision foundation models that generalize to a broad range of image domains and perform exceptionally well in various image based tasks. However, content-based image-to-image retrieval is often overlooked in this context. This paper investigates the effectiveness of different vision foundation models on two challenging nearest neighbor search-based tasks: zero-shot retrieval and k-NN classification. A benchmark for evaluating the performance of various vision encoders and their pre-training methods is established, where significant differences in the performance of these models are observed. Additionally, we propose a fine-tuning regime that improves zero-shot retrieval and k-NN classification through training with a combination of large publicly available datasets without specializing in any data domain. Our results show that the retrained vision encoders have a higher degree of generalization across different search-based tasks and can be used as general-purpose embedding models for image retrieval.

Over the last years, we have published papers about intuitive image graph navigation and showed how to build static hierarchical image graphs efficiently. In this paper, we showcase new results and present techniques to dynamically construct and manipulate these kinds of graphs. They connect similar images and perform well in retrieving tasks regardless of the number of nodes. By applying an improved fast self-sorting map algorithm, entire image collections (structured in a graph) can be explored with a user interface resembling common navigation services.

We present a new approach to visually browse very large sets of untagged images. High quality image features are generated using transformed activations of a convolutional neural network. These features are used to model image similarities, from which a hierarchical image graph is build. We show how such a graph can be constructed efficiently. In our experiments we found best user experience for navigating the graph is achieved by projecting sub-graphs onto a regular 2D image map. This allows users to explore the image collection like an interactive map.