Retrieval in 3D point clouds is a challenging task that consists in retrieving the most similar point clouds to a given query within a reference of 3D points. Current methods focus on comparing descriptors of point clouds in order to identify similar ones, without any particular index structure. Due to the complexity of this latter step, here we focus on the acceleration of the retrieval by adapting the Differentiable Search Index (DSI), a transformer-based approach initially designed for text information retrieval, for 3D point cloud retrieval. Our approach generates 1D identifiers based on the point descriptors, enabling direct retrieval in constant time. To adapt DSI to 3D data, we integrate Vision Transformers to map descriptors to these identifiers while incorporating positional and semantic encoding. The approach is evaluated for place recognition on a public benchmark comparing its retrieval capabilities against state-of-the-art methods, in terms of quality and speed of returned point clouds.

While pre-trained image autoencoders are increasingly utilized in computer vision, the application of inverse graphics in 2D latent spaces has been under-explored. Yet, besides reducing the training and rendering complexity, applying inverse graphics in the latent space enables a valuable interoperability with other latent-based 2D methods. The major challenge is that inverse graphics cannot be directly applied to such image latent spaces because they lack an underlying 3D geometry. In this paper, we propose an Inverse Graphics Autoencoder (IG-AE) that specifically addresses this issue. To this end, we regularize an image autoencoder with 3D-geometry by aligning its latent space with jointly trained latent 3D scenes. We utilize the trained IG-AE to bring NeRFs to the latent space with a latent NeRF training pipeline, which we implement in an open-source extension of the Nerfstudio framework, thereby unlocking latent scene learning for its supported methods. We experimentally confirm that Latent NeRFs trained with IG-AE present an improved quality compared to a standard autoencoder, all while exhibiting training and rendering accelerations with respect to NeRFs trained in the image space. Our project page can be found at https://ig-ae.github.io .

SUMAC 2025 is the 7th edition of the workshop on analySis, Understanding and proMotion of heritAge Contents. It is held in Dublin, Ireland, on 27 October and is co-located with the 33rd ACM International Conference on Multimedia. The workshop's objective is to present and discuss the latest and most significant trends, challenges, and advances in the fields of machine learning, signal processing, multimodal techniques, and human-machine interaction. The workshop is dedicated to the valorization of cultural heritage, with an emphasis on unlocking and access to the big data of the past. A representative scope of Computer Science methodologies dedicated to the processing of multimedia heritage contents and their exploitation is covered by the works presented, with the ambition of advancing and raising awareness about this fully developing research field.

Location retrieval based on visual information is to retrieve the location of an agent (e.g.human, robot) or the area they see by comparing their observations with a certain representation of the environment.Existing methods generally treat the problem as a content-based image retrieval problem and have demonstrated promising results in terms of localization accuracy.However, these methods are challenging to scale up due to the volume of reference data involved; and the image descriptions might not be easily understandable/communicable for humans to describe surroundings.Considering that humans often use less precise but easily produced qualitative spatial language and high-level semantic landmarks when describing an environment, a coarseto-fine qualitative location retrieval method is proposed in this work to quickly narrow down the initial location of an agent by exploiting the available information in large-scale open data.This approach describes and indexes a location/place using the perceived qualitative spatial relations between ordered pairs of covisible landmarks from the perspective of viewers, termed as 'qualitative place signatures' (QPS).The usability and effectiveness of the proposed method were evaluated using openly available datasets, together with simulated observations by considering different types perception errors.

As the number of digitized geographic iconographic heritage collections increases, their global use is under-exploited by their lack of structure at large scale, which does not facilitate their access nor their understanding. Using automatic image retrieval methods appears to be the solution to bring structure by building links between contents, within and between collections. This paper presents an overview of methods for image retrieval applied to geographic iconographic heritage collections, both from the perspectives of image content description and of post-processing re-ranking. The article evaluates features and methods to identify their efficiency when faced with a challenging dataset. Moreover, new re-ranking approaches exploiting structuring information (scene geometry, metadata) are proposed to improve retrieval without having to adapt image descriptors to the specific data (retraining, fine-tuning, etc.) for every new specific collection.

SUMAC 2023 is the fifth edition of the workshop on analySis, Understanding and proMotion of heritAge Contents. It is held in Ottawa, Canada on November 2, 2023 and is co-located with the 31st ACM International Conference on Multimedia. The workshop's objective is to present and discuss the latest and most significant trends, challenges and advances in the fields of machine learning, signal processing, multimodal techniques and human-machine interaction. The workshop is dedicated to the valorization of cultural heritage, with the emphasis on the unlocking of and access to the big data of the past. A representative scope of Computer Science methodologies dedicated to the processing of multimedia heritage contents and their exploitation is covered by the works presented, with the ambition of advancing and raising awareness about this fully developing research field. The complete SUMAC'23 workshop proceedings are available at: https://dl.acm.org/doi/proceedings/10.1145/3581783.3610949.

In this article, we present two online platforms developed for the structuring and valorization of old geographical iconographic collections: a multimodal search engine for their indexing, retrieval and interlinking, and a 3D navigation platform for their visualization in spatial context. In particular, we show how the joint use of these functionalities, guided by geolocation, brings structure and knowledge to the manipulated collections. In the demonstrator, they consist of 54,000 oblique aerial photographs from several French providers (national archives, a museum and a mapping agency).

SUMAC 2022 is the fourth edition of the workshop on Structuring and Understanding of Multimedia heritAge Contents. It is held in Lisboa, Portugal on October 10th, 2022 and is co-located with the 30th ACM International Conference on Multimedia. Its objective is to present and discuss the latest and most significant trends and challenges in the analysis, structuring and understanding of multimedia contents dedicated to the valorization of heritage, with the emphasis on the unlocking of and access to the big data of the past. A representative scope of Computer Science methodologies dedicated to the processing of multimedia heritage contents and their exploitation is covered by the works presented, with the ambition of advancing and raising awareness about this fully developing research field. The complete SUMAC'22 workshop proceedings are available at: https://dl.acm.org/doi/proceedings/10.1145/3552464

SUMAC 2021 is the third edition of the workshop on Structuring and Understanding of Multimedia heritAge Contents. It is held in Chengdu, China on October 20th, 2021 and is co-located with the 29th ACM International Conference on Multimedia. Its objective is to present and discuss the latest and most significant trends and challenges in the analysis, structuring and understanding of multimedia contents dedicated to the valorization of heritage, with the emphasis on the unlocking of and access to the big data of the past. A representative scope of Computer Science methodologies dedicated to the processing of multimedia heritage contents and their exploitation is covered by the works presented, with the ambition of advancing and raising awareness about this fully developing research field.

This paper introduces a multi-modal network that learns to retrieve by content vertical aerial images of French urban and rural territories taken about 15 years apart. This means it should be invariant against a big range of changes as the (natural) landscape evolves over time. It leverages the original images and semantically segmented and labeled regions. The core of the method is a Siamese network that learns to extract features from corresponding image pairs across time. These descriptors are discriminative enough, such that a simple kNN classifier on top, suffices as final geo-matching criteria. The method outperformed SOTA “off-the-shelf' image descriptors GEM and ResNet50 on the new aerial images dataset.

SUMAC 2020 is the second edition of the workshop on Structuring and Understanding of Multimedia heritAge Contents. It is held in Seattle, USA on October 12th, 2020 and is co-located with the 28th ACM International Conference on Multimedia; this year, due to the sanitary crisis, it is organized virtually. Its objective is to present and discuss the latest and most significant trends and challenges in the analysis, structuring and understanding of multimedia contents dedicated to the valorization of heritage, with the emphasis on the unlocking of and access to the big data of the past. A representative scope of Computer Science methodologies dedicated to the processing of multimedia heritage contents and their exploitation is covered by the works presented, with the ambition of advancing and raising awareness about this fully developing research field.

We present a new method for image-only camera relocalisation composed of a fast image indexing retrieval step followed by pose refinement based on ICP (Iterative Closest Point). The first step aims to find an initial pose for the query by evaluating images similarity with low dimensional global deep descriptors. Subsequently, we predict with a fully convolutional deep encoder-decoder neural network a dense depth map from the image query. We use this depth map to create a local point cloud and refine the initial query pose using an ICP algorithm.We demonstrate the effectiveness of our new approach on various indoor scenes. Compared to learned pose regression methods, our proposal can be used on multiple scenes without the need of a specific weights-setup for each scene, while showing equivalent results.

We propose a new approach for outdoor large scale image based localization that can deal with challenging scenarios like cross-season, cross-weather, day/night and long-term localization. The key component of our method is a new learned global image descriptor, that can effectively benefit from scene geometry information during training. At test time, our system is capable of inferring the depth map related to the query image and use it to increase localization accuracy. We are able to increase recall@1 performances by 2.15% on cross-weather and long-term localization scenario and by 4.24% points on a challenging winter/summer localization sequence versus state-of-the-art methods. Our method can also use weakly annotated data to localize night images across a reference dataset of daytime images.

SUMAC 2019 is the first workshop on Structuring and Understanding of Multimedia heritAge Contents. It is held in Nice, France on October 21, 2019 and is co-located with the 27th ACM International Conference on Multimedia. Its objective is to present and discuss the latest and most significant trends and challenges in the analysis, structuring and understanding of multimedia contents dedicated to the valorization of heritage, with the emphasis on the unlocking of and access to the big data of the past. A representative scope of Computer Science methodologies dedicated to the processing of multimedia heritage contents and their exploitation is covered by the works presented, with the ambition of advancing and raising awareness about this fully developing research field.

Visual localization is a useful alternative to standard localization techniques. In a typical scenario, features are extracted from images captured by cameras and compared with geo-referenced databases. Location information is then inferred from the matching results. Conventional schemes mainly use low-level visual features. They offer good accuracy but suffer from scalability issues. In order to assist localization in large urban areas, this work explores a different path by utilizing high-level semantic information. It is found that object information in a street view can facilitate localization. A novel descriptor scheme called “semantic signature” is proposed to summarize this information. A semantic signature consists of type and angle information of visible objects at a spatial location. Several metrics and protocols are proposed for signature comparison and retrieval. They illustrate different trade-offs between accuracy and complexity. Extensive simulation results confirm the potential of the proposed scheme in large-scale applications.

It has been now widely assessed in the literature that both multi/hyperspectral optical images and 3D lidar point clouds are necessary inputs for tree species based forest stand detection. Nevertheless, no comprehensive analysis of the genuine relevance of each data source has been performed so far: existing strategies are limited to a single spatial and spectral resolution. This paper investigates which is the optimal combination of geospatial optical images and lidar point clouds. A supervised semantic segmentation framework is fed with various sources (multispectral satellite and airborne images, hyperspectral airborne images, low, medium and high density lidar point clouds), ablation cases are defined, and the discrimination performance of several fusion schemes is assessed under a challenging mountainous area in France.

Forest stands are a basic unit of analysis for forest inventory and mapping. Stands are defined as large forested areas of homogeneous tree species composition and age. Their accurate delineation is usually performed by human operators through visual analysis of very high resolution (VHR) infra-red and visible images. This task is tedious, highly time consuming, and needs to be automated for scalability and efficient updating purposes. The most appropriate fusion of two remote sensing modalities (lidar and multispectral images) is investigated here. The multispectral images give information about the tree species while 3D lidar point clouds provide geometric information. The fusion is operated at three different levels within a semantic segmentation workflow: over-segmentation, classification, and regularization. Results show that over-segmentation can be performed either on lidar or optical images without performance loss or gain, whereas fusion is mandatory for efficient semantic segmentation. Eventually, the fusion strategy dictates the composition and nature of the forest stands, assessing the high versatility of our approach.

Due to the large diversity of existing feature descriptors in content-based image retrieval, the image contents can be better represented by the joint use of several descriptors in order to explore their potentially complementary characteristics. This paper presents and discusses a strategy for fusion of the different multidimensional features involved, based on inverted multi-indices and dedicated to similarity search. Image descriptors are quantized separately and efficiently through dimension reduction techniques, before being combined in the inverted multi-indices. To exhibit its effectiveness, the proposal is evaluated on two datasets having different contents and sizes, facing several state-of-the-art approaches of image descriptor fusion. The obtained results reconfirm that the joint use of several descriptions improves similarity search, and show that our fusion proposal outperforms other solutions, while manipulating lower or similar volumes of features.

A knowledge on spatial relationships between objects present in a given collection of images can provide interesting information to improve classical CBIR tasks such as object detection and localization, by reducing the searching areas of the object relatively to one or several given objects. In this paper, we propose a representation of the knowledge on relationships existing between symbolic objects in a collection of images. None exhaustively, these relationships can be co-occurrences of objects or different kinds of spatial relationships between them in images. We present a graph-based representation of this knowledge and its associated operations and properties. This work was evaluated on the public symbolic image database LabelMe. The experiments show its relevance for object detection and localization.

This paper presents the prototype of a personal photo album manager that we build over three years. It integrates state of the art techniques of both database systems and computer vision to propose to publishers a privacy preserving sharable photo album manager where the publisher decides what each user is allowed to see. In addition, the photo album manager provides tools for an easy semantic annotation, classification and querying of the images.

In this article, we are interested in accelerating similarity search in collections of image contents described in high dimensional spaces. The presented framework, called HiPeR (for Hierarchical Progressive Retrieval), is based onahierarchyof subspaces and indexes: it performs similarity search across feature spaces of different dimensions, by beginning with the lowest dimensions up to the highest ones, with the aim of minimizing the effects of curse of dimensionality. HiPeR significantly accelerates exact retrieval even with the best indexes, and also enables progressive retrieval, i.e. the possibility to provide results to the user progressively, with refinements until userpsilas satisfaction.

This paper presents applications for dealing with videos on the Web, using an efficient technique for video copy detection in large archives. Managing videos on the Web is the source of two exciting challenges: the respect of the copyright and the linkage of multiple videos. We present a technique called ViCopT for video copy tracking which is based on labels of behavior of local descriptors computed along video. The results obtained on large amount of data (270 hours of videos from the Internet) are very promising, even with a large video database (700 hours): ViCopT displays excellent robustness to various severe signal transformations, making it able to identify copies accurately from highly similar videos, as well as to link similar videos, in order to reduce redundancy or to gather the metadata associated. Finally, we also show that ViCopT goes further by detecting segments having the same background, with the aim of linking videos of the same category, like forecast weather programs or particular TV shows.

This paper presents an efficient approach for copies detection in a large videos archive consisting of several hundred of hours. The video content indexing method consists of extracting the dynamic behavior on the local description of interest points and further on the estimation of their trajectories along the video sequence. Analyzing the low-level description obtained allows to highlight trends of behaviors and then to assign a label of behavior to each local descriptor. Such an indexing approach has several interesting properties: it provides a rich, compact and generic description, while labels of behavior provide a high-level description of the video content. Here, we focus on video Content Based Copy Detection (CBCD). Copy detection is problematic as similarity search problem but with prominent differences. To be efficient, it requires a dedicated on-line retrieval method based on a specific voting function. This voting function must be robust to signal transformations and discriminating versus high similarities which are not copies. The method we propose in this paper is a dedicated on-line retrieval method based on a combination of the different dynamic contexts computed during the off-line indexing. A spatio-temporal registration based on the relevant combination of detected labels is then applied. This approach is evaluated using a huge video database of 300 hours with different video tests. The method is compared to a state-of-the art technique in the same conditions. We illustrate that taking labels into account in the specific voting process reduces false alarms significantly and drastically improves the precision.

This paper presents a robust and generic approach of object tracking in video sequences. Here, the object to track is described by considering two wellknown image primitives: first, its content is described with Points of interest. Such points are automatically extracted and then characterized according to a selective spatial appearance-based model. Second, the object envelope is described with a Snake. The originality of the SAP approach consists in a complementary use of these two primitives: the snake allows to reduce the points tracking to a limited area in each frame, and the spatial point description is exploited during the snake tracking, making the process robust to wide occlusions. Since no model of trajectory is considered, the approach is robust to wide motions of object and camera. The relevance of this approach has been evaluated on several video streams. Results obtained with the most representative of them are presented in this paper. The algorithms involved have been implemented with the aim of achieving near real-time performance. 1

For content-based image retrieval (CBIR), traditional approaches of image matching involve global descriptions of the color image. When considering particular tasks like object recognition or partial queries, more local characterizations must be employed. In this context, image description based on points of interest appear best adapted. The point characterization which proved reliable is based on combinations of the Hilbert's differential invariants. For gray value images, such a description used to be considered up to third order. Generalizations to color images were previously proposed for stereovision and image retrieval. Some of them propose to consider the invariants only at first order, while others consider higher order invariants and compute some combinations of them to achieve illumination changes invariance. We discuss the advantages and drawbacks of these different choices, with the aim of proposing an optimal use of color points of interest for image retrieval.

In this paper we present a new method for point matching in stereoscopic color images. Our approach consists first in characterizing points of interest using differential invariants. Then we define additional first order invariants using color information, which make sufficient the characterization till first order. In addition, we make our description robust to important image transformations like rotation, range of viewpoint and linear illumination variations. Second, we propose a new incremental technique for point matching using our characterization, which works robustly and rapidly whatever the number of points to be matched. Our stereo matching scheme is evaluated using stereo color images, with viewpoint and illumination variations. The very good results obtained clearly show the pertinence of our approach. Our color characterization produces a high rate of good matches, even though only first order derivatives are used. Results on images holding many points show that our matchi...

In this paper, we present a new method for matching points in stereoscopic, uncalibrated color images. Our approach works on point primitives and consists first in using a point characterization based on differential invariants, defined until now for gray value images. Then we define additional invariants of first order, exploiting color information. The characterization obtained is invariant to orthogonal transformations of the image. In addition, we make it robust to affine transformations of illumination. Second, we propose a simple and efficient scheme for point matching, using our differential characterization. Finally, we present matching and epipolar geometry results obtained on complex scenes. The results clearly show the pertinence of our approach. We are able to match points robustly and rapidly, using only derivatives till first order.

We present a new method for matching points in stereoscopic, uncalibrated color images. Our approach consists of characterizing points of interest using differential invariants. We define additional invariants of first order, exploiting color information. We show that this contribution makes the characterization sufficient for first order. In addition, we make our description robust to usual transformations of image. We present a robust generalization of a gray level corner detector to the case of color images. We also propose a simple and efficient scheme for matching these points, using our characterization. Finally, we present matching results and the epipolar geometry obtained on complex scenes, which clearly show the pertinence of our approach. We are able to match points robustly and rapidly, using only first order derivatives.