In a data-driven world, efficiently organizing and mapping relationships between objects is crucial. Graphs are powerful tools for modeling these connections, being widely used in social networks, telecommunications, and biology. However, graph-based methods often face high computational costs, particularly in memory and space usage. To address this, graph embedding techniques, also referred to as Network Representation Learning, encode graph information into lower-dimensional representations while preserving structural aspects. Traditional methods, however, lack interpretable dimensions. RaDE (Rank Diffusion Embedding) introduces a new approach using rank-based information, with a key step being the selection of a representative subset of nodes to provide interpretability for its dimensions and improve retrieval tasks. Despite its potential, RaDE’s original proposal did not fully explore the effectiveness of representative subset selection across different classes or evaluate embeddings in tasks like classification and clustering. Inspired by RaDE, this work introduces GRaCE (Graph and Rank-based Contextual Embeddings), a fully unsupervised framework that generates interpretable embeddings by leveraging robust rank-based measures for representative subset selection and node embedding. GRaCE surpasses RaDE and Original Features across diverse datasets, including textual and image collections, excelling in retrieval, classification, and clustering tasks, considering state-of-the-art Transformer models as feature descriptors and Graph Convolutional Networks models in classification tasks.

The large and growing amount of digital data creates a pressing need for approaches capable of indexing and retrieving multimedia content. A traditional and fundamental challenge consists of effectively and efficiently performing nearest-neighbor searches. After decades of research, several different methods are available, including trees, hashing, and graph-based approaches. Most of the current methods exploit learning to hash approaches based on deep learning. In spite of effective results and compact codes obtained, such methods often require a significant amount of labeled data for training. Unsupervised approaches also rely on expensive training procedures usually based on a huge amount of data. In this work, we propose an unsupervised data-independent approach for nearest neighbor searches, which can be used with different features, including deep features trained by transfer learning. The method uses a rank-based formulation and exploits a hashing approach for efficient ranked list computation at query time. A comprehensive experimental evaluation was conducted on seven public datasets, considering deep features based on CNNs and Transformers. Both effectiveness and efficiency aspects were evaluated. The proposed approach achieves remarkable results in comparison to traditional and state-of-the-art methods. Hence, it is an attractive and innovative solution, especially when costly training procedures need to be avoided.

The representation of multimedia content experienced tremendous advances in the last decades. Mainly supported by deep learning models, impressive results have been obtained. However, despite such advances in representation, the definition of similarity has been neglected. Effectively computing the similarity between representations remains a challenge. Traditional distance functions, such as the Euclidean distance, are not able to properly consider the relevant similarity information encoded in the dataset manifold. In fact, manifolds are essential to perception in many scenarios, such that exploiting the underlying structure of dataset manifolds plays a central role in multimedia content understanding and retrieval. In this paper, we present a framework for unsupervised distance learning which provides easy and uniform access to methods capable of considering the dataset manifold for redefining similarity. Such methods perform context-sensitive similarity learning based on more global measures, capable of improving the effectiveness of retrieval and machine learning tasks. The framework can use distance, similarity, or ranking information both as input and output and compute traditional retrieval effectiveness measures. Implemented as a wrapper in Python, the framework allows integration with a large number of Python libraries while keeping a back-end in C++ for efficiency. The paper also discusses diverse applications of the methods available in the pyUDLF framework, including image re-ranking, video retrieval, person re-ID, and pre-processing of distance measurements for clustering and classification.

Impressive advances in acquisition and sharing technologies have made the growth of multimedia collections and their applications almost unlimited. However, the opposite is true for the availability of labeled data, which is needed for supervised training, since such data is often expensive and time-consuming to obtain. While there is a pressing need for the development of effective retrieval and classification methods, the difficulties faced by supervised approaches highlight the relevance of methods capable of operating with few or no labeled data. In this work, we propose a novel manifold learning algorithm named Rank Flow Embedding (RFE) for unsupervised and semi-supervised scenarios. The proposed method is based on ideas recently exploited by manifold learning approaches, which include hypergraphs, Cartesian products, and connected components. The algorithm computes context-sensitive embeddings, which are refined following a rank-based processing flow, while complementary contextual information is incorporated. The generated embeddings can be exploited for more effective unsupervised retrieval or semi-supervised classification based on Graph Convolutional Networks. Experimental results were conducted on 10 different collections. Various features were considered, including the ones obtained with recent Convolutional Neural Networks (CNN) and Vision Transformer (ViT) models. High effective results demonstrate the effectiveness of the proposed method on different tasks: unsupervised image retrieval, semi-supervised classification, and person Re-ID. The results demonstrate that RFE is competitive or superior to the state-of-the-art in diverse evaluated scenarios.

Despite the impressive advances obtained by supervised deep learning approaches on retrieval and classification tasks, how to acquire labeled data for training remains a challenging bottleneck. In this scenario, the need for developing more effective content-based retrieval approaches capable of taking advantage of multimodal information and advances in unsupervised learning becomes imperative. Based on such observations, we propose two novel approaches that combine Graph Convolutional Networks (GCNs) with rank-based manifold learning methods. The GCN models were trained in an unsupervised way, using the Deep Graph Infomax algorithm, and the proposed approaches employ recent rank-based manifold learning methods. Multimodal information is exploited through pre-trained CNNs via transfer learning for extracting audio, image, and video features. The proposed approaches were evaluated on three public action recognition datasets. High-effective results were obtained, reaching relative gains up to +29.44% of MAP compared to baseline approaches without GCNs. The experimental evaluation also considered classical and recent baselines in the literature.

Effectively measuring similarity among data samples represented as points in high-dimensional spaces remains a major challenge in retrieval, machine learning, and computer vision. In these scenarios, unsupervised manifold learning techniques grounded on rank information have been demonstrated to be a promising solution. However, various methods rely on rank correlation measures, which often depend on a proper definition of neighborhood size. On current approaches, this definition may lead to a reduction in the final desired effectiveness. In this work, a novel rank correlation measure robust to such variations is proposed for manifold learning approaches. The proposed measure is suitable for diverse scenarios and is validated on a Manifold Learning Algorithm based on Correlation Graph (CG). The experimental evaluation considered 6 datasets on general image retrieval and person Re-ID, achieving results superior to most state-of-the-art methods.

Image and multimedia retrieval has established as a prominent task in an increasingly digital and visual world. Mainly supported by decades of development on hand-crafted features and the success of deep learning techniques, various different feature extraction and retrieval approaches are currently available. However, the frequent requirements for large training sets still remain as a fundamental bottleneck, especially in real-world and large-scale scenarios. In the scarcity or absence of labeled data, choosing what retrieval approach to use became a central challenge. A promising strategy consists in to estimate the effectiveness of ranked lists without requiring any groundtruth data. Most of the existing measures exploit statistical analysis of the ranked lists and measure the reciprocity among lists of images in the top positions. This work innovates by proposing a new and self-supervised method for this task, the Deep Rank Noise Estimator (DRNE). An algorithm is presented for generating synthetic ranked list data, which is modeled as images and provided for training a Convolutional Neural Network that we propose for effectiveness estimation. The proposed model is a variant of the DnCNN (Denoiser CNN), which intends to interpret the incorrectness of a ranked list as noise, which is learned by the network. Our approach was evaluated on 5 public image datasets and different tasks, including general image retrieval and person re-ID. We also exploited and evaluated the complementary between the proposed approach and related rank-based approaches through fusion strategies. The experimental results showed that the proposed method is capable of achieving up to 0.88 of Pearson correlation with MAP measure in general retrieval scenarios and 0.74 in person re-ID scenarios.

Machine learning approaches have achieved remarkable advances over the last decades, especially in supervised learning tasks such as classification. Meanwhile, multimedia data and applications experienced an explosive growth, becoming ubiquitous in diverse domains. Due to the huge increase in multimedia data collections and the lack of labeled data in several scenarios, creating methods capable of exploiting the unlabeled data and operating under weakly supervision is imperative. In this work, we propose a rank-based model to exploit contextual information encoded in the unlabeled data in order to perform weakly supervised classification. We employ different rank-based correlation measures for identifying strong similarities relationships and expanding the labeled set in an unsupervised way. Subsequently, the extended labeled set is used by a classifier to achieve better accuracy results. The proposed weakly supervised approach was evaluated on multimedia classification tasks, considering several combinations of rank correlation measures and classifiers. An experimental evaluation was conducted on 4 public image datasets and different features. Very positive gains were achieved in comparison with various semi-supervised and supervised classifiers taken as baselines when considering the same amount of labeled data.

Summary Despite the continuous advances in image retrieval technologies, performing effective and efficient content‐based searches remains a challenging task. Unsupervised iterative re‐ranking algorithms have emerged as a promising solution and have been widely used to improve the effectiveness of multimedia retrieval systems. Although substantially more efficient than related approaches based on diffusion processes, these re‐ranking algorithms can still be computationally costly, demanding the specification and implementation of efficient big multimedia analysis approaches. Such demand associated with the significant potential for parallelization and highly effective results achieved by recently proposed re‐ranking algorithms creates the need for exploiting efficiency vs effectiveness trade‐offs. In this article, we introduce a class of unsupervised iterative re‐ranking algorithms and present a model that can be used to guide their implementation and optimization for parallel architectures. We also analyze the impact of the parallelization on the performance of four algorithms that belong to the proposed class: Contextual Spaces, RL‐Sim, Contextual Re‐ranking, and Cartesian Product of Ranking References. The experiments show speedups that reach up to 6.0×, 16.1×, 3.3×, and 7.1× for each algorithm, respectively. These results demonstrate that the proposed parallel programming model can be successfully applied to various algorithms and used to improve the performance of multimedia retrieval systems.

Despite the major advances on feature development for low and mid-level representations, a single visual feature is often insufficient to achieve effective retrieval results in different scenarios. Since diverse visual properties provide distinct and often complementary information for a same query, the combination of different features, including handcrafted and learned features, has been establishing as a relevant trend in image retrieval. An intrinsic difficulty task consists in selecting and combining features that provide a high-effective result, which is often supported by supervised learning methods. However, in the absence of labeled data, selecting and fusing features in a completely unsupervised fashion becomes an essential, although very challenging task. The proposed genetic algorithm employs effectiveness estimation measures as fitness functions, making the evolutionary process fully unsupervised. Our approach was evaluated considering 3 public datasets and 35 different descriptors achieving relative gains up to +53.96% in scenarios with more than 8 billion possible combinations of rankers. The framework was also compared to different baselines, including state-of-the-art methods.

Accurately ranking images and multimedia objects are of paramount relevance in many retrieval and learning tasks. Manifold learning methods have been investigated for ranking mainly due to their capacity of taking into account the intrinsic global manifold structure. In this paper, a novel manifold ranking algorithm is proposed based on the hypergraphs for unsupervised multimedia retrieval tasks. Different from traditional graph-based approaches, which represent only pairwise relationships, hypergraphs are capable of modeling similarity relationships among a set of objects. The proposed approach uses the hyperedges for constructing a contextual representation of data samples and exploits the encoded information for deriving a more effective similarity function. An extensive experimental evaluation was conducted on nine public datasets including diverse retrieval scenarios and multimedia content. Experimental results demonstrate that high effectiveness gains can be obtained in comparison with the state-of-the-art methods.

Due to the growing availability of unlabeled data and the difficulties in obtaining labeled data, the use of semi-supervised learning approaches becomes even more promising. The capacity of taking into account the dataset structure is of crucial relevance for effectively considering the unlabeled data. In this paper, a novel classifier is proposed through a manifold learning approach. The graph is constructed based on a new hybrid similarity measure which encodes both supervised and unsupervised information. Next, strongly connected components are computed and used to analyze the dataset manifold. The classification is performed through a voting scheme based on primary (labeled) and secondary (unlabeled) voters. An experimental evaluation is conducted, considering various datasets, diverse situations of training/test dataset sizes and comparison with baselines. The proposed method achieved positive results in most of situations.

The increasing amount of multimedia data collections available today evinces the pressing need for methods capable of indexing and retrieving this content. Despite the continuous advances in multimedia features and representation models, to establish an effective measure for comparing different multimedia objects still remains a challenging task. While supervised and semi-supervised techniques made relevant advances on similarity learning tasks, scenarios where labeled data are non-existent require different strategies. In such situations, unsupervised learning has been established as a promising solution, capable of considering the contextual information and the dataset structure for computing new similarity/dissimilarity measures. This article extends a recent unsupervised learning algorithm that uses an iterative re-ranking strategy to take advantage of different k -Nearest Neighbors (kNN) sets and rank correlation measures. Two novel approaches are proposed for computing the kNN sets and their corresponding top- k lists. The proposed approaches were validated in conjunction with various rank correlation measures, yielding superior effectiveness results in comparison with previous works. In addition, we also evaluate the ability of the method in considering different multimedia objects, conducting an extensive experimental evaluation on various image and video datasets.

Due to the increasing availability of image and multimedia collections, unsupervised post-processing methods, which are capable of improving the effectiveness of retrieval results without the need of user intervention, have become indispensable. This paper presents the Unsupervised Distance Learning Framework (UDLF), a software which enables an easy use and evaluation of unsupervised learning methods. The framework defines a broad model, allowing the implementation of different unsupervised methods and supporting diverse file formats for input and output. Seven different unsupervised methods are initially available in the framework. Executions and experiments can be easily defined by setting a configuration file. The framework also includes the evaluation of the retrieval results exporting visual output results, computing effectiveness and efficiency measures. The source-code is public available, such that anyone can freely access, use, change, and share the software under the terms of the GPLv2 license.

Various unsupervised learning methods have been proposed with significant improvements in the effectiveness of image search systems. However, despite the relevant effectiveness gains, these approaches commonly require high computation efforts, not addressing properly efficiency and scalability requirements. In this paper, we present a novel unsupervised learning approach for improving the effectiveness of image retrieval tasks. The proposed method is also scalable and efficient as it exploits parallel and heterogeneous computing on CPU and GPU devices. Extensive experiments were conducted considering five different public image collections and several descriptors. This rigorous experimental protocol evaluates the effectiveness, efficiency, and scalability of the proposed approach, and compares it with previous methods. Experimental results demonstrate that high effectiveness gains (up to +29%) can be obtained requiring small run times.