Climate change is a heated discussion topic in public arenas such as social media. Both texts and visuals play key roles in the debate, as they can complement, contradict, or reinforce each other in nuanced ways. It is therefore urgently needed to study the messages as multimodal objects to better understand the polarized debate about climate change impacts and policies. Multimodal representation models such as CLIP are known to be able to transfer knowledge across domains and modalities, enabling the investigation of textual and visual semantics together. Yet they are not directly able to distinguish the nuances between supporting and sceptic climate change stances. This paper explores a simple but effective strategy combining modality fusion and domain-knowledge enhancing to prepare CLIP-based models with knowledge of climate change stances. A multimodal Dutch Twitter dataset is collected and experimented with the proposed strategy, which increased the macro-average F1 score across stances from 51% to 86%. The outcomes can be applied in both data science and public policy studies, to better analyse how the combined use of texts and visuals generates meanings during debates, in the context of climate change and beyond.

Abstract The need for effective and efficient search and rescue operations is more important than ever as the frequency and severity of disasters increase due to the escalating effects of climate change. Recognizing the value of personal knowledge and past experiences of experts, in this paper, we present findings of an investigation of how past knowledge and experts’ experiences can be effectively integrated with current search and rescue practices to improve rescue planning and resource allocation. A special focus is on investigating and demonstrating the potential associated with integrating knowledge graphs and case-based reasoning as a viable approach for search and rescue decision support. As part of our investigation, we have implemented a demonstrator system using a Norwegian search and rescue dataset and case-based and concept-based similarity retrieval. The main contribution of the paper is insight into how case-based and concept-based retrieval services can be designed to improve the effectiveness of search and rescue planning. To evaluate the validity of ranked cases in terms of how they align with the existing knowledge and insights of search and rescue experts, we use evaluation measures such as precision and recall. In our evaluation, we observed that attributes, such as the rescue operation type, have high precision, while the precision associated with the objects involved is relatively low. Central findings from our evaluation process are that knowledge-based creation, as well as case- and concept-based similarity retrieval services, can be beneficial in optimizing search and rescue planning time and allocating appropriate resources according to search and rescue incident descriptions.

Knowledge Graphs have been established as one the most relevant representations to encode knowledge, with relevant applications in the public and private sectors. One common research direction concerning the analysis of created knowledge graphs relies on the assumption that their intrinsic properties and structure are similar to what is observed in complex networks. However, studies concerning identifying typical complex network structures in knowledge graphs are lacking in the literature. This paper bridges this gap by analyzing commonly and recently used knowledge graphs in the semantic web field, seeking to demonstrate their complex network properties. Evaluation involving DBpedia and Wikidata data confirms the occurrence of intrinsic complex network structures in their respective knowledge graphs.

The use of nearest neighbors graphs has been leading to considerable gains over classic approaches (e.g., tree-indexing-based methods) in approximate nearest neighbor searches. Classical searches on these graphs are conduced in a greedy way by moving at each step to the neighbor (of current vertex) with the lowest distance to the query. In this work, we explore the combination of topological properties of graphs and the distance itself through a Genetic Programming framework to obtain a better indicator for selecting the next vertex in the search process. Our objective is to minimize the scan rate needed to reach the true nearest neighbors. Experimental results, conducted with three different graph-based methods over a large textual collection, show significant gains of the proposed approach over the classic search algorithm on graphs.

Text representation models are the fundamental basis for information retrieval and text mining tasks. Although different text models have been proposed, they typically target specific task aspects in isolation, such as time efficiency, accuracy, or applicability for different scenarios. Here we present Bag of Textual Graphs (BoTG), a general text representation model that addresses these three requirements at the same time. The proposed textual representation is based on a graph‐based scheme that encodes term proximity and term ordering, and represents text documents into an efficient vector space that addresses all these aspects as well as provides discriminative textual patterns. Extensive experiments are conducted in two experimental scenarios—classification and retrieval—considering multiple well‐known text collections. We also compare our model against several methods from the literature. Experimental results demonstrate that our model is generic enough to handle different tasks and collections. It is also more efficient than the widely used state‐of‐the‐art methods in textual classification and retrieval tasks, with a competitive effectiveness, sometimes with gains by large margins.

This paper presents an efficient diffusion-based re-ranking approach. The proposed method propagates contextual information defined in terms of top-ranked objects of ranked lists in a diffusion process. That makes the method suitable for large scale real-world collections. Experiments were conducted considering public image collections, several descriptors, and comparisons with state-of-the-art methods. Experimental results demonstrate that the proposed method provides high effectiveness gains with low computational costs.

PhenoVis is framework for the visual phenological analysis of forest ecosystems. It contains the chronological percentage maps (CPM), a novel representation that is capable of discovering additional patterns by encoding percentage distributions of the data. Two types of masks are used in PhenoVis: a community mask , which considers all plant species in the image; and a species mask , associated with a given plant species. Among the several images taken at different times of the day, the image taken at noon is preferred for the analysis because it minimizes shadow effects. Therefore, only one image per day is used. The analysis considers the chromatic co- efficients associated with each pixel in the image. In PhenoVis we associate different colors with each bucket of the percentage histogram. The histogram granularity defines the size of a given bucket of the percentage distribution. The number of buckets is given by the number of colors available, and the range of the distribution is given by the IOI. The percentage map of a single input image consists of a normalized stacked bar chart. The chronological percentage map consists of a sequence of percentage maps stacked in chronological order, from top to bottom (portrait) or left to right (landscape).

This paper presents an approach to combine rank aggregation techniques using a soft computing technique -- Genetic Programming -- in order to improve the results in Information Retrieval tasks. Previous work shows that by combining rank aggregation techniques in an agglomerative way, it is possible to get better results than with individual methods. However, these works either combine only a small set of lists or are performed in a completely ad-hoc way. Therefore, given a set of ranked lists and a set of rank aggregation techniques, we propose to use a supervised genetic programming approach to search combinations of them that maximize effectiveness in large search spaces. Experimental results conducted using four datasets with different properties show that our proposed approach reaches top performance in most datasets. Moreover, this cross-dataset performance is not matched by any other baseline among the many we experiment with, some being the state-of-the-art in learning-to-rank and in the supervised rank aggregation tasks. We also show that our proposed framework is very efficient, flexible, and scalable.

Ranking accurately collection images is the main objective of Content-based Image Retrieval (CBIR) systems. In fact, the set of images ranked at the first positions generally defines the effectiveness of provided search services, i.e., they are used for assessing automatically the quality of search systems as this set usually contains the collection images that are of interest. Recently, the use of ranking information (e.g., rank correlation) has been used in different research initiatives with the objective of improving the effectiveness of image retrieval tasks. This paper presents a broad rank correlation analysis for unsupervised distance learning on image retrieval tasks. Various well-known rank correlation measures are considered and two new measures are proposed. Several experiments were conducted considering various image datasets involving shape, color, and texture descriptors. Experimental results demonstrate that ranking information can be exploited for distance learning tasks successfully. Evaluated approaches yield better results in terms of effectiveness than various state-of-the-art algorithms.

Various unsupervised learning methods have been proposed with significant improvements in the effectiveness of image search systems. However, despite the relevant effectiveness gains, these approaches commonly require high computation efforts, not addressing properly efficiency and scalability requirements. In this paper, we present a novel unsupervised learning approach for improving the effectiveness of image retrieval tasks. The proposed method is also scalable and efficient as it exploits parallel and heterogeneous computing on CPU and GPU devices. Extensive experiments were conducted considering five different public image collections and several descriptors. This rigorous experimental protocol evaluates the effectiveness, efficiency, and scalability of the proposed approach, and compares it with previous methods. Experimental results demonstrate that high effectiveness gains (up to +29%) can be obtained requiring small run times.

This paper presents a novel unsupervised learning approach that takes into account the intrinsic dataset structure, which is represented in terms of the reciprocal neighborhood references found in different ranked lists. The proposed Reciprocal kNN Distance defines a more effective distance between two images, and is used to improve the effectiveness of image retrieval systems. Several experiments were conducted for different image retrieval tasks involving shape, color, and texture descriptors. The proposed approach is also evaluated on multimodal retrieval tasks, considering visual and textual descriptors. Experimental results demonstrate the effectiveness of proposed approach. The Reciprocal kNN Distance yields better results in terms of effectiveness than various state-of-the-art algorithms.

This work is developed in the context of the placing task of the MediaEval 2011 initiative. The objective is to geocode (or geotag) a set of videos, i.e., automatically assign geographical coordinates to them. This paper presents an architecture for multimodal geocoding that exploits both visual and textual descriptions associated with videos. This work also describes our efforts regarding the implementation of this architecture to demonstrate its applicability. Conducted experiments show how our multimodal approach enhances the results compared to relying on a single modality.

Vote-based algorithms are very popular in tasks based on image local-descriptors, including object matching, panoramic stitching and near-duplicate detection. On this paper, we focus on the latter application, proposing a Bayesian approach, which allows giving a probabilistic interpretation to the distances between local descriptors in the feature space. That contrasts with traditional schemes, in which the distances are used to establish a simple unweighted vote count. Near-duplicate detection is demanded for a myriad of applications: metadata retrieval in cultural institutions, detection of copyright violations, duplicate elimination in storage, etc. The majority of current solutions are based either on voting algorithms, which are very precise, but expensive; or on the use of visual dictionaries, which are efficient, but less precise. Contrarily to raw-vote based systems, our scheme performs few database accesses; and contrarily to dictionary-based systems, it allows a fine control of the compromise between precision and efficiency. In our experiments, it yields 99% accuracy with less than 10 database accesses, in contrast with the hundreds needed in raw-voting schemes.

This paper presents a novel approach for video representation, called bag-of-scenes. The proposed method is based on dictionaries of scenes, which provide a high-level representation for videos. Scenes are elements with much more semantic information than local features, specially for geotagging videos using visual content. Thus, each component of the representation model has self-contained semantics and, hence, it can be directly related to a specific place of interest. Experiments were conducted in the context of the MediaEval 2011 Placing Task. The reported results show our strategy compared to those from other participants that used only visual content to accomplish this task. Despite our very simple way to generate the visual dictionary, which has taken photos at random, the results show that our approach presents high accuracy relative to the state-of-the art solutions.

This paper introduces Hypercurves, a flexible framework for pro- viding similarity search indexing to high throughput multimedia services. Hypercurves efficiently and effectively answers k-nearest neighbor searches on multigigabyte high-dimensional databases. It supports massively parallel processing and adapts at runtime its parallelization regimens to keep answer times optimal for either low and high demands. In order to achieve its goals, Hypercurves introduces new techniques for selecting parallelism configurations and allocating threads to computation cores, including hyperthreaded cores. Its efficiency gains are throughly validated on a large database of multimedia descriptors, where it presented near linear speedups and superlinear scaleups. The adaptation reduces query response times in 43% and 74% for both platforms tested, when compared to the best static parallelism regimens.

The objective of Content-based Image Retrieval (CBIR) systems is to return the most similar images given an image query. In this scenario, accurately ranking collection images is of great relevance. In general, CBIR systems consider only pairwise image analysis, that is, compute similarity measures considering only pair of images, ignoring the rich information encoded in the relations among several images. This paper presents a novel re-ranking approach based on contextual spaces aiming to improve the effectiveness of CBIR tasks, by exploring relations among images. In our approach, information encoded in both distances among images and ranked lists computed by CBIR systems are used for analyzing contextual information. The re-ranking method can also be applied to other tasks, such as: (i) for combining ranked lists obtained by using different image descriptors (rank aggregation); and (ii) for combining post-processing methods. We conducted several experiments involving shape, color, and texture descriptors and comparisons to other post-processing methods. Experimental results demonstrate the effectiveness of our method.

Similarity search in high-dimensional metric spaces is a key operation in many applications, such as multimedia databases, image retrieval, object recognition, and others. The high dimensionality of the data requires special index structures to facilitate the search. Most of existing indexes are constructed by partitioning the data set using distance-based criteria. However, those methods either produce disjoint partitions, but ignore the distribution properties of the data; or produce non-disjoint groups, which greatly affect the search performance. In this paper, we study the performance of a new index structure, called Ball-and-Plane tree (BP-tree), which overcomes the above disadvantages. BP-tree is constructed by recursively dividing the data set into compact clusters. Distinctive from other techniques, it integrates the advantages of both disjoint and non-disjoint paradigms in order to achieve a structure of tight and low overlapping clusters, yielding significantly improved performance. Results obtained from an extensive experimental evaluation with real-world data sets show that BP-tree consistently outperforms state-of-the-art solutions.

Researchers have been studying and developing teaching materials for information retrieval (IR), such as [3]. Toolkits also have been built that provide hands-on experience to students. For example, IR-Toolbox [4] is an effort to close the gap between the students' understanding of IR concepts and real-life indexing and search systems. Such tools might be good for helping students in non-technical areas such as in the Library and Information Science field to develop their conceptual model of search engines. However, they do not cover emerging topics and skills, such as content-based image retrieval (CBIR) and fusion search. Although there is open source software (such as those in http://www.searchtools.com/tools/tools-opensource.html) that can be used to teach basic and advanced IR topics, they require a student to have high-level technical knowledge and to spend a long time to gain a practical understanding of these topics.

In this paper, we propose a novel framework using Genetic Programming to combine image database descriptors for content-based image retrieval (CBIR). Our framework is validated through several experiments involving two image databases and specific domains, where the images are retrieved based on the shape of their objects.

Content-Based Image Retrieval (CBIR) presents several challenges and has been subject to extensive research from many domains, such as image processing or database systems. Database researchers are concerned with indexing and querying, whereas image processing experts worry about extracting appropriate image descriptors. Comparatively little work has been done on designing user interfaces for CBIR systems. This, in turn, has a profound effect on these systems since the concept of image similarity is strongly influenced by user perception. This paper describes an initial effort to fill this gap, combining recent research in CBIR and Information Visualization, studied from a Human-Computer Interface perspective. It presents two visualization techniques based on Spiral and Concentric Rings implemented in a CBIR system to explore query results. The approach is centered on keeping user focus on both the query image, and the most similar retrieved images. Experiments conducted so far suggest that the proposed visualization strategies improves system usability.