Small, fast, and lightweight drones present significant challenges for traditional RGB cameras due to their limitations in capturing fast-moving objects, especially under challenging lighting conditions. Event cameras offer an ideal solution, providing high temporal definition and dynamic range, yet existing benchmarks often lack fine temporal resolution or drone-specific motion patterns, hindering progress in these areas. This paper introduces the Florence RGB-Event Drone dataset (FRED), a novel multimodal dataset specifically designed for drone detection, tracking, and trajectory forecasting, combining RGB video and event streams. FRED features more than 7 hours of densely annotated drone trajectories, using 5 different drone models and including challenging scenarios such as rain and adverse lighting conditions. We provide detailed evaluation protocols and standard metrics for each task, facilitating reproducible benchmarking. The authors hope FRED will advance research in high-speed drone perception and multimodal spatiotemporal understanding.

Event cameras offer significant advantages over traditional frame-based sensors, including higher temporal resolution, lower latency and dynamic range. However, efficiently converting event streams into formats compatible with standard computer vision pipelines remains a challenging problem, particularly in the presence of noise. In this paper, we propose Spike-TBR, a novel event-based encoding strategy based on Temporal Binary Representation (TBR), addressing its vulnerability to noise by integrating spiking neurons. Spike-TBR combines the frame-based advantages of TBR with the noise-filtering capabilities of spiking neural networks, creating a more robust representation of event streams. We evaluate four variants of Spike-TBR, each using different spiking neurons, across multiple datasets, demonstrating superior performance in noise-affected scenarios while improving the results on clean data. Our method bridges the gap between spike-based and frame-based processing, offering a simple noise-resilient solution for event-driven vision applications. • We present Spike-TBR an enhanced version of Temporal Binary Representation. • By using spiking neurons we make the frame-based representation resilient to noise. • We obtain state-of-the-art results on four different neuromorphic datasets.

In the literature, several 3D face datasets have been collected, aiming at advancing the field of 3D face analysis from different perspectives. Data collection generally follows specific research needs, and the existing 3D face datasets all have different characteristics that are tailored for investigating different tasks, encompassing face recognition, facial expressions and emotions analysis, 3D face reconstruction. However, the majority of these datasets are either collected with high-resolution scanners, or consumer level devices, such as the Kinect, the latter being motivated by the burdensome and costly process of collecting high-quality scans. Differently from 2D imagery, the difference in resolution in 3D data represents a non negligible problem that is under-investigated, and still prevents the successful development of methods that can work in real scenarios. In this paper, we propose a new 3D face dataset, named “Florence Multi-Resolution 3D Facial Expression” (Florence 3DMRE), which aims at bridging the gap between high- and low-resolution 3D face datasets. Its peculiarity consists in (1) including high-resolution (HR) models obtained with a HR scanner, and paired samples collected with a Kinect sensor, (2) LR and HR scans are synchronized and capture extreme and asymmetric facial deformations as used in facial rehabilitation exercises. In total, our dataset consists of 14 subjects, each performing 19 complex and asymmetric expressions. For each of them, we collected a high-resolution scan, and an RGB-D sequence. Finally, to highlight the value of the dataset and the challenges it introduces, we use the collected data to perform baseline experiments for cross-resolution 3D face recognition and reconstruction. The dataset is released for research purposes only, and complies to GDPR for data treatment. The dataset can be found at this link.

The 3D Morphable Model (3DMM) is a powerful statistical tool for representing 3D face shapes. To build a 3DMM, a training set of face scans in full point-to-point correspondence is required, and its modeling capabilities directly depend on the variability contained in the training data. Thus, to increase the descriptive power of the 3DMM, establishing a dense correspondence across heterogeneous scans with sufficient diversity in terms of identities, ethnicities, or expressions becomes essential. In this manuscript, we present a fully automatic approach that leverages a 3DMM to transfer its dense semantic annotation across raw 3D faces, establishing a dense correspondence between them. We propose a novel formulation to learn a set of sparse deformation components with local support on the face that, together with an original non-rigid deformation algorithm, allow the 3DMM to precisely fit unseen faces and transfer its semantic annotation. We extensively experimented our approach, showing it can effectively generalize to highly diverse samples and accurately establish a dense correspondence even in presence of complex facial expressions. The accuracy of the dense registration is demonstrated by building a heterogeneous, large-scale 3DMM from more than 9,000 fully registered scans obtained by joining three large datasets together.

We propose an automatic method to estimate self-reported pain based on facial landmarks extracted from videos. For each video sequence, we decompose the face into four different regions and the pain intensity is measured by modeling the dynamics of facial movement using the landmarks of these regions. A formulation based on Gram matrices is used for representing the trajectory of landmarks on the Riemannian manifold of symmetric positive semi-definite matrices of fixed rank. A curve fitting algorithm is used to smooth the trajectories and temporal alignment is performed to compute the similarity between the trajectories on the manifold. A Support Vector Regression classifier is then trained to encode extracted trajectories into pain intensity levels consistent with self-reported pain intensity measurement. Finally, a late fusion of the estimation for each region is performed to obtain the final predicted pain level. The proposed approach is evaluated on two publicly available datasets, the UNBCMcMaster Shoulder Pain Archive and the Biovid Heat Pain dataset. We compared our method to the state-of-the-art on both datasets using different testing protocols, showing the competitiveness of the proposed approach.

The accurate (quantitative) analysis of face deformations in 3D is a problem of increasing interest for the many applications it may have. In particular, defining a 3D model of the face that can deform to a 2D target image, while capturing local and asymmetric deformations is still a challenge in the existing literature. Computing a measure of such local deformations may represent a relevant index for monitoring rehabilitation exercises that are used in Parkinson’s and Alzheimer’s disease or in recovering from a stroke. In this study, we present a complete framework that allows the construction of a 3D Morphable Shape Model (3DMM) of the face and its fitting to a target RGB image. The model has the specific characteristic of being based on localized components of deformation; the fitting transformation is performed from 3D to 2D and is guided by the correspondence between landmarks detected in the target image and landmarks manually annotated on the average 3DMM. The fitting has also the peculiarity of being performed in two steps, disentangling face deformations that are due to the identity of the target subject from those induced by facial actions. In the experimental validation of the method, we used the MICC-3D dataset that includes 11 subjects each acquired in one neutral pose plus 18 facial actions that deform the face in localized and asymmetric ways. For each acquisition, we fit the 3DMM to an RGB frame with an apex facial action and to the neutral frame, and computed the extent of the deformation. Results indicated that the proposed approach can accurately capture the face deformation even for localized and asymmetric ones. The proposed framework proved the idea of measuring the deformations of a reconstructed 3D face model to monitor the facial actions performed in response to a set of target ones. Interestingly, these results were obtained just using RGB targets without the need for 3D scans captured with costly devices. This opens the way to the use of the proposed tool for remote medical monitoring of rehabilitation.

In this paper, a model is presented to extract statistical summaries to characterize the repetition of a cyclic body action, for instance a gym exercise, for the purpose of checking the compliance of the observed action to a template one and highlighting the parts of the action that are not correctly executed (if any). The proposed system relies on a Riemannian metric to compute the distance between two poses in such a way that the geometry of the manifold where the pose descriptors lie is preserved; a model to detect the begin and end of each cycle; a model to temporally align the poses of different cycles so as to accurately estimate the cross-sectional mean and variance of poses across different cycles. The proposed model is demonstrated using gym videos taken from the Internet.

We propose an automatic method for pain intensity measurement from video. For each video, pain intensity was measured using the dynamics of facial movement using 66 facial points. Gram matrices formulation was used for facial points trajectory representations on the Riemannian manifold of symmetric positive semi-definite matrices of fixed rank. Curve fitting and temporal alignment were then used to smooth the extracted trajectories. A Support Vector Regression model was then trained to encode the extracted trajectories into ten pain intensity levels consistent with the Visual Analogue Scale for pain intensity measurement. The proposed approach was evaluated using the UNBC McMaster Shoulder Pain Archive and was compared to the state-of-the-art on the same data. Using both 5-fold cross-validation and leave-one-subject-out cross-validation, our results are competitive with respect to state-of-the-art methods.

No abstract available.

3D cameras for face capturing are quite common today thanks to their ease of use and affordable cost. The depth information they provide is mainly used to enhance face pose estimation and tracking, and face-background segmentation, while applications that require finer face details are usually not possible due to the low-resolution data acquired by such devices. In this paper, we propose a framework that allows us to derive high-quality 3D models of the face starting from corresponding low-resolution depth sequences acquired with a depth camera. To this end, we start by defining a solution that exploits temporal redundancy in a short-sequence of adjacent depth frames to remove most of the acquisition noise and produce an aggregated point cloud output with intermediate level details. Then, using a 3DMM specifically designed to support local and expression-related deformations of the face, we propose a two-steps 3DMM fitting solution: initially the model is deformed under the effect of landmarks correspondences; subsequently, it is iteratively refined using points closeness updating guided by a mean-square optimization. Preliminary results show that the proposed solution is able to derive 3D models of the face with high visual quality; quantitative results also evidence the superiority of our approach with respect to methods that use one step fitting based on landmarks.

Face recognition from two-dimensional (2D) still images and videos is quite successful even with “in the wild” conditions. Instead, less consolidated results are available for the cases in which face data come from non-conventional cameras, such as infrared or depth. In this article, we investigate this latter scenario assuming that a low-resolution depth camera is used to perform face recognition in an uncooperative context. To this end, we propose, first, to automatically select a set of frames from the depth sequence of the camera because they provide a good view of the face in terms of pose and distance. Then, we design a progressive refinement approach to reconstruct a higher-resolution model from the selected low-resolution frames. This process accounts for the anisotropic error of the existing points in the current 3D model and the points in a newly acquired frame so that the refinement step can progressively adjust the point positions in the model using a Kalman-like estimation. The quality of the reconstructed model is evaluated by considering the error between the reconstructed models and their corresponding high-resolution scans used as ground truth. In addition, we performed face recognition using the reconstructed models as probes against a gallery of reconstructed models and a gallery with high-resolution scans. The obtained results confirm the possibility to effectively use the reconstructed models for the face recognition task.

The analysis of human gait is more and more investigated due to its large panel of potential applications in various domains, like rehabilitation, deficiency diagnosis, surveillance and movement optimization. In addition, the release of depth sensors offers new opportunities to achieve gait analysis in a non-intrusive context. In this paper, we propose a gait analysis method from depth sequences by analyzing separately each step so as to be robust to gait duration and incomplete cycles. We analyze the shape of the motion trajectory as signature of the gait and consider shape variations within a Riemannian manifold to learn step models. During classification, the derivation of each performed step is evaluated in an online manner to qualitatively analyze the gait. Experiments are carried out in the context of abnormal gait detection and person re-identification trough gait recognition. Results demonstrated the potential of the method in both scenarios.

Performing face recognition across 3D scans with different resolution is now attracting an increasing interest thanks to the introduction of a new generation of depth cameras, capable of acquiring color/depth images over time. In fact, these devices acquire and provide depth data with much lower resolution compared with the 3D high-resolution scanners typically used for face recognition applications. If data are acquired without user cooperation, the problem is even more challenging, and the gap of resolution between probe and gallery scans can yield to a severe loss in terms of recognition accuracy. Based on these premises, we propose a method to build a higher resolution 3D face model from 3D data acquired by a low-resolution scanner. This face model is built using data acquired when a person passes in front of the scanner, without assuming any particular cooperation. The 3D data are registered and filtered by combining a model of the expected distribution of the acquisition error with a variant of the lowess method to remove outliers and build the final face model. The proposed approach is evaluated in terms of accuracy of face reconstruction and face recognition.

Recognizing human actions in 3-D video sequences is an important open problem that is currently at the heart of many research domains including surveillance, natural interfaces and rehabilitation. However, the design and development of models for action recognition that are both accurate and efficient is a challenging task due to the variability of the human pose, clothing and appearance. In this paper, we propose a new framework to extract a compact representation of a human action captured through a depth sensor, and enable accurate action recognition. The proposed solution develops on fitting a human skeleton model to acquired data so as to represent the 3-D coordinates of the joints and their change over time as a trajectory in a suitable action space. Thanks to such a 3-D joint-based framework, the proposed solution is capable to capture both the shape and the dynamics of the human body, simultaneously. The action recognition problem is then formulated as the problem of computing the similarity between the shape of trajectories in a Riemannian manifold. Classification using k-nearest neighbors is finally performed on this manifold taking advantage of Riemannian geometry in the open curve shape space. Experiments are carried out on four representative benchmarks to demonstrate the potential of the proposed solution in terms of accuracy/latency for a low-latency action recognition. Comparative results with state-of-the-art methods are reported.

Face recognition based on the analysis of 3D scans has been an active research subject over the last few years. However, the impact of the resolution of 3D scans on the recognition process has not been addressed explicitly, yet being an element of primal importance to enable the use of the new generation of consumer depth cameras for biometric purposes. In fact, these devices perform depth/color acquisition over time at standard frame-rate, but with a low resolution compared to the 3D scanners typically used for acquiring 3D faces in recognition applications. Motivated by these considerations, in this paper, we define a super-resolution approach for 3D faces by which a sequence of low-resolution 3D face scans is processed to extract a higher resolution 3D face model. The proposed solution relies on the scaled iterative closest point procedure to align the low-resolution scans with each other, and estimates the value of the high-resolution 3D model through a 2D box-spline functions approximation. To evaluate the approach, we built-and made it publicly available-the Florence Superface dataset that collects high-resolution and low-resolution data for about 50 different persons. Qualitative and quantitative results are reported to demonstrate the accuracy of the proposed solution, also in comparison with alternative techniques.

An original solution to 3D face recognition, which supports face matching also in the case of probes with varying expressions and missing parts is proposed in this work. Distinguishing traits of the face are captured by first extracting 3D keypoints of the face scan, then measuring how the face surface changes in the neighborhood of the keypoints using a local descriptor. To this end, an adaptation of the meshDOG detector to the case of 3D faces is proposed, together with a multi-ring geometric histogram descriptor. Face similarity is then evaluated by comparing local keypoint descriptors across inlier pairs of matching keypoints between probe and gallery scans. Experiments have been performed on the Bosphorus database, showing competitive results with respect to existing solutions for 3D face biometrics.

In this work, we propose and experiment an original solution to 3D face recognition that supports face matching also in the case of probe scans with missing parts. In the proposed approach, distinguishing traits of the face are captured by first extracting 3D keypoints of the scan and then measuring how the face surface changes in the keypoints neighborhood using local shape descriptors. In particular: 3D keypoints detection relies on the adaptation to the case of 3D faces of the meshDOG algorithm that has been demonstrated to be effective for 3D keypoints extraction from generic objects; as 3D local descriptors we used the HOG descriptor and also proposed two alternative solutions that develop, respectively, on the histogram of orientations and the geometric histogram descriptors. Face similarity is evaluated by comparing local shape descriptors across inlier pairs of matching keypoints between probe and gallery scans. The face recognition accuracy of the approach has been first experimented on the difficult probes included in the new 2D/3D Florence face dataset that has been recently collected and released at the University of Firenze, and on the Binghamton University 3D facial expression dataset. Then, a comprehensive comparative evaluation has been performed on the Bosphorus, Gavab and UND/FRGC v2.0 databases, where competitive results with respect to existing solutions for 3D face biometrics have been obtained.

In this work, we propose and experiment a 3-D face recognition approach capable of performing accurate face matching also in the case where just parts of probe scans are available. This is obtained through an original face representation and matching solution that first extracts keypoints of the 3-D depth image of the face and then measures how the face depth changes along facial curves connecting pairs of keypoints. Face similarity is evaluated by sparse comparison of facial curves defined across inlier pairs of matching keypoints between probe and gallery scans. In doing so, a statistical model is also proposed to associate facial curves of the gallery scans with a saliency measure so that curves that model characterizing traits of some subjects are distinguished from curves that are frequently observed in the face of many different subjects. Following recent related work, the recognition accuracy of the approach is experimented using two datasets, both comprising scans with missing parts: the Face Recognition Grand Challenge v2.0 dataset combined with the University of Notre Dame probes; the Gavab dataset.

Among different approaches for 3D face recognition, solutions based on local facial characteristics are very promising, mainly because they can manage facial expression variations by assigning different weights to different parts of the face. However, so far, a few works have investigated the individual relevance that local features play in 3D face recognition with very simple solutions applied in the practice. In this article, a local approach to 3D face recognition is combined with a feature selection model to study the relative relevance of different regions of the face for the purpose of discriminating between different subjects. The proposed solution is experimented using facial scans of the Face Recognition Grand Challenge dataset. Results of the experimentation are two-fold: they quantitatively demonstrate the assumption that different regions of the face have different relevance for face discrimination and also show that the relevance of facial regions changes for different ethnic groups.

In this paper, the problem of person-independent facial expression recognition is addressed on 3D shapes. To this end, an original approach is proposed that computes SIFT descriptors on a set of facial landmarks of depth images, and then selects the subset of most relevant features. Using SVM classification of the selected features, an average recognition rate of 77.5% on the BU-3DFE database has been obtained. Comparative evaluation on a common experimental setup, shows that our solution is able to obtain state of the art results.

In this paper, we present a novel approach to 3D face matching that shows high effectiveness in distinguishing facial differences between distinct individuals from differences induced by nonneutral expressions within the same individual. The approach takes into account geometrical information of the 3D face and encodes the relevant information into a compact representation in the form of a graph. Nodes of the graph represent equal width isogeodesic facial stripes. Arcs between pairs of nodes are labeled with descriptors, referred to as 3D Weighted Walkthroughs (3DWWs), that capture the mutual relative spatial displacement between all the pairs of points of the corresponding stripes. Face partitioning into isogeodesic stripes and 3DWWs together provide an approximate representation of local morphology of faces that exhibits smooth variations for changes induced by facial expressions. The graph-based representation permits very efficient matching for face recognition and is also suited to being employed for face identification in very large data sets with the support of appropriate index structures. The method obtained the best ranking at the SHREC 2008 contest for 3D face recognition. We present an extensive comparative evaluation of the performance with the FRGC v2.0 data set and the SHREC08 data set.

In the past few years, several research works have addressed the problems posed by vision-assisted navigation systems. Basically, these systems allow a tourists navigating in an urban environment to take the photograph of a scene and submit it to the navigation system that will recognize what monument is represented in the photograph. However, solutions proposed so far focus on the recognition of approximately planar structures (e.g. building facades) and are inadequate to support the recognition task if the object of interest has a generic 3D structure such as a statue. In this paper we present a model to support the recognition of generic monuments that is based on the information-theoretic notion of mutual information to quantify the saliency of invariant local descriptors. Results are demonstrated on a real-database also in comparison with the baseline method that performs matching without any measure of saliency.

An original approach to represent 2D and 3D faces using radial geodesic distances (RGDs) is proposed in this work. In 3D, the RGD of a generic point of the face surface is computed as the length of the geodesic connecting the point with a reference point along a radial direction. In 2D, the RGD of a pixel with respect to a reference pixel accounts for the difference of gray level intensities of the two pixels and the Euclidean distance between them. Support Vector Machines (SVMs) are used to perform face recognition using 2D- and 3D-RGDs. Due to the high dimensionality of face representations based on RGDs, embedding into lower-dimensional spaces is applied before SVMs classification. Experimental results are reported for 3D-3D and 2D-3D face recognition using the proposed approach.

In this work, we shortly describe an original 3D face recognition approach and its performance as resulted from the 3D Shape Retrieval Contest of 3D Face Scans organized by SHREC 2008 with the support of the Network of Excellence AIM@SHAPE. In particular, the evaluation shows that the proposed approach attains the highest performance on the SHREC 2008 data set.

In this paper, we propose an original framework for representing 2D and 3D face information using geodesic distances. This aims to define a representation enabling the direct comparison between 2D face images of an individual against its 3D face model. This representation is extracted by measuring geodesic distances in 2D and 3D. In 3D, the geodesic distance between two points on a surface is computed as the length of the shortest path connecting the two points. In 2D, the geodesic distance between two pixels is computed based on the differences of gray level intensities along the segment connecting the two pixels. Experimental results are shown to demonstrate the viability of the proposed solution.

Retrieval by content of 3-D models is becoming more and more important due to the advancements in 3-D hardware and software technologies for acquisition, authoring and display of 3-D objects, their ever-increasing availability at affordable costs, and the establishment of open standards for 3-D data interchange. In this paper, we present a new method, referred to as Spin Image Signatures, that develops on the original spin images approach, with adaptations to support effective retrieval by content. According to the method proposed, a set of spin images is derived for each model, to obtain a view-independent description of its 3-D shape and a signature is evaluated for each spin image in the set. Clustering is hence performed on the set of Spin Image Signatures to obtain a compact representation. Experimental results are presented, showing the effectiveness of the Spin Image Signatures method for retrieval, also in comparison with other methods, and its sensitivity to model deformations.

In this paper, we propose an original framework for three dimensional face representation and matching for identification purposes. Basic traits of a face are encoded by extracting curves of salient ridges and ravines from the surface of a dense mesh. A compact graph representation is then extracted from these curves through an original modeling technique capable to quantitatively measure spatial relationships between curves in a three dimensional space. In this way, face recognition is obtained by matching 3D graph representations of faces. Experimental results on a 3D face database show that the proposed solution attains high recognition accuracy and is quite robust to facial expression and pose changes

In the past few years, there has been an increasing availability of technologies for the acquisition of digital 3D models of real objects and the consequent use of these models in a variety of applications, in medicine, engineering, and cultural heritage. In this framework, content-based retrieval of 3D objects is becoming an important subject of research, and finding adequate descriptors to capture global or local characteristics of the shape has become one of the main investigation goals. In this article, we present a comparative analysis of a few different solutions for description and retrieval by similarity of 3D models that are representative of the principal classes of approaches proposed. We have developed an experimental analysis by comparing these methods according to their robustness to deformations, the ability to capture an object's structural complexity, and the resolution at which models are considered.

Along with images and videos, 3D models have raised a certain interest for a number of reasons, including advancements in 3D hardware and software technologies, their ever decreasing prices and increasing availability, affordable 3D authoring tools, and the establishment of open standards for 3D data interchange. The resulting proliferation of 3D models demands for tools supporting their effective and efficient management, including archival and retrieval. In order to support effective retrieval by content of 3D objects and enable retrieval by object parts, information about local object structure should be combined with spatial information on object surface. In this paper, as a solution to this requirement, we present a method relying on curvature correlograms to perform description and retrieval by content of 3D objects. Experimental results are presented both to show results of sample queries by content and to compare-in terms of precision/recall figures-the proposed solution to alternative techniques.

A solution for part segmentation of 3D objects is proposed in this paper. The approach is targeted to identify salient visual parts of a mesh by determining its main protrusions and discarding, at the same time, parts originated by un-relevant local properties. This is obtained by first breaking the 3D mesh into seed regions according to the sum of geodesic distances between vertices, then by using topological and curvature information to refine the number of regions and their boundaries. In so doing, effective segmentation is regarded as a prerequisite to enable retrieval of 3D objects based on similarity of parts. Experimental results show the applicability of the proposed solution to complex shapes and its effectiveness in the identification of object parts.

Along with image and video libraries, archives of 3D models have recently gained increasing attention. Accordingly, there is an increasing demand for solutions enabling retrieval of 3D models based on global properties as well as properties of object parts. In particular, retrieval based on object parts relies on segmentation of 3D objects into their constituent parts. This is a challenging task, as the identification of object parts should conform to human perceptual judgement. Therefore, definition of models and solutions that enable decomposition of 3D objects into perceptually relevant parts is a fundamental step to enable effective retrieval based on object parts. However, a few approaches have been proposed to support segmentation of 3D meshes into perceptually relevant parts. In this paper we propose a model based on pulse-coupled oscillator networks. Preliminary experiments are reported to demonstrate the validity and potential of the proposed solution.

Along with images and videos, 3D models have recently gained increasing attention for a number of reasons: advancements in 3D hardware and software technologies; their ever decreasing prices and increasing availability; affordable 3D authoring tools; the establishment of open standards for 3D data interchange. The ever increasing availability of 3D models demands tools to support their effective and efficient management. Among these tools, those enabling content-based retrieval play a key role. We present a novel approach to 3D content-based retrieval that is based on spin images. Spin images are used to derive a view-independent description of both database and query objects: a set of spin images is first created for each object; then, a descriptor is evaluated for each spin image in the set; clustering is performed on the set of image-based descriptors of each object to achieve a compact representation of the object, thus allowing for efficient indexing and matching. Experimental results are presented for a test database of about 300 models. These results indicate that spin images can be successfully exploited for content-based retrieval of 3D objects

Along with images and videos, 3D models have recently gained increasing attention for a number of reasons: advancements in 3D hardware and software technologies, their ever decreasing prices and increasing availability, affordable 3D authoring tools, and the establishment of open standards for 3D data interchange. In this paper we address the problem of content-based retrieval of 3D models. The solution we present here relies on the description of each model by means a curvature map: after an initial pre-processing of the model, differential properties of points on the surface of the 3D object are evaluated; the model surface is then warped into an ellipsoid, and is mapped onto a 2D image retaining curvature information of the original model. Matching is performed by comparing the 2D map of the query against the 2D maps of the database models. This method has been implemented in a prototype system supporting retrieval by content of 3D objects through a Web interface.

Exploitation of information repositories available on the Internet requires users to separately query each repository and manually gather retrieved results. Such a solution could be simplified by using a centralized server that acts as a gateway between the user and repositories: the centralized server forwards the user query to federated repositories and fuses retrieved documents for presentation to the user. To perform these tasks efficiently, the centralized server should perform two main functions: resource selection and data fusion. The former is required to forward the user query only to the repositories that are candidate to contain relevant documents. The latter is used to gather all retrieved documents and conveniently arrange them for presentation to the user. In the case of image repositories, data fusion is particularly challenging owing to the difficulty to normalize document scores returned by different repositories. In this paper a novel solution is presented for fusion of results returned by different image repositories. Experimental results are presented that show the potential of the proposed approach.

Callan, Jamie (ed.): SIGIR '03: Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval. New York: ACM, 2003, 363–364. - ISBN: 978-1-58113-646-3 Online also available at: https://doi.org/10.1145/860435.860502

In this paper we report on our experience in the detection and recognition of soccer highlights in videos using hidden Markov models. A first approach relies on camera motion only, whereas a second one also includes information regarding the location of players on the playing field. While the former approach requires less information, the latter has proven to be more precise. Our experimental evaluation yields interesting results.

Content based retrieval from distributed libraries raises new and challenging issues with respect to retrieval from a single repository. In particular, an effective management of distributed libraries develops upon three main processes: resource description (extraction of descriptors that qualify the content of a given archive), resource selection (given a user query, analyze resource descriptions and select the resources that contain relevant documents) and results merging (organize and present items returned by individual libraries). So far, these issues have been mainly addressed for text archives. We present a solution to resource descriptors extraction, developing on the use of techniques for multidimensional data indexing. In particular, we implement and compare the extraction of resource descriptors computed through two different indexing approaches; namely m-tree indexing and fuzzy clustering. Comparative results are presented for a test database of about 1000 images.

Image databases are widely exploited in a number of different contexts, ranging from history of art, through medicine, to education. Existing querying paradigms are based either on the usage of textual strings, for high-level semantic queries or on 2D visual examples for the expression of perceptual queries. Semantic queries require manual annotation of the database images. Instead, perceptual queries only require that image analysis is performed on the database images in order to extract salient perceptual features that are matched with those of the example. However, usage of 2D examples is generally inadequate as effective authoring of query images, attaining a realistic reproduction of complex scenes, needs manual editing and sketching ability. Investigation of new querying paradigms is therefore an important-yet still marginally investigated-factor for the success of content-based image retrieval. In this paper, a novel querying paradigm is presented which is based on usage of 3D interfaces exploiting navigation and editing of 3D virtual environments. Query images are obtained by taking a snapshot of the framed environment and by using the snapshot as an example to retrieve similar database images. A comparative analysis is carried out between the usage of 3D and 2D interfaces and their related query paradigms. This analysis develops on a user test on retrieval efficiency and effectiveness, as well as on an evaluation of users' satisfaction.

Recent advances in management of multimedia digital libraries enable effective retrieval of information in the form of audio, image and video. Many archives of 3D objects already exist and are expected to grow both in relevance and size. However, retrieval of information in the form of 3D objects has received limited attention. We address the problem of effective description and retrieval of 3D data representing intracellular structures. These structures are represented as image stacks, where an image stack is constituted by a set of 2D images representing sections of a cellular body at different heights. In the proposed approach 2D visual feature descriptors and hidden Markov models are combined to obtain a representation model which is able to distinguish such intracellular structures as Golgi, nucleus, endoplasmic reticulum and lysosomes. Preliminary results are presented to show the effectiveness of the proposed representation model.

Recent advances in management of multimedia digital libraries enable effective retrieval of information in the form of audio, image and video. However, retrieval of information in the form of 3D objects has received limited attention so far. Yet many archives of 3D objects already exist and are expected to grow both in relevance and size. In this paper, we address the problem of effective description and retrieval of 3D data representing intracellular structures. These structures are represented in the form of image stacks, being an image stack a set of 2D images representing planar sections of a cellular body at different heights. In the proposed method, 2D visual feature descriptors and Hidden Markov Models are combined to obtain a representation model which is able to distinguish such intracellular structures as Golgi, nucleus, endoplasmic reticulum and lysosomes. Preliminary results are presented to show the effectiveness of the proposed representation model.

Systems for content based image retrieval typically support access to database images through the query-by-example paradigm. This includes query-by-image and query-by-sketch. Since query-by-sketch can be difficult in some cases-lack of sketching abilities, difficulty to detect distinguishing image features-generally querying is performed through the query-by-image paradigm. A limiting factor of this paradigm is that a single sample image rarely includes all and only the characterizing elements the user is looking for. In this paper a system is presented that supports query-by-image using multiple image examples. Examples can be positive and negative and can be edited in order to disregard irrelevant image features.

An important problem in accessing and retrieving visual information is to provide efficient similarity matching in large databases. Though much work is being done on the investigation of suitable perceptual models and the automatic extraction of features, little attention is given to the combination of useful representations and similarity models with efficient index structures. In this paper we propose retrieval by shape similarity using local descriptors and effective indexing. Shapes are partitioned into tokens in correspondence with their protrusions, and each token is modeled according to a set of perceptually salient attributes. Shape indexing is obtained by arranging shape tokens into a suitably modified M-tree index structure. Two distinct distance functions model respectively, token and shape perceptual similarity. Examples from a prototype system and computational experiences are reported for both retrieval accuracy and indexing efficiency. Shape retrieval has been tested under shape scaling, orientation changes, and partial shape occlusions. A comparative analysis of different indexing structures, for shape retrieval is presented.

Videos convey information through several planes of communication, encompassing what is represented in the images how the images are linked together and how the subject is imaged. This feature is stressed in commercials where colors, editing effects, rhythms, and object motion are exploited to influence human purchasing habits. In this paper, based on researches in the marketing field, a link is formalized between low level features of a commercial video and feelings that the video would inspire in the observer. This link is used to define high level indices capturing the main semantics of the video. These indices are embedded in a video retrieval system to support access to a database of video based on their semantics.

The development of a system supporting querying of image databases by color content tackles a major design choice about properties of colors which are referenced within user queries. On the one hand, low-level properties directly reflect numerical features and concepts tied to the machine representation of color information. On the other hand, high-level properties address concepts such as the perceptual quality of colors and the sensations that they convey. Color-induced sensations include warmth, accordance or contrast, harmony, excitement, depression, anguish etc. In particular, paintings are an example where the message is contained more in the high-level color qualities and spatial arrangements than in the physical properties of colors. Starting from this observation, Johannes Itten (1961) introduced a formalism to analyze the use of color in art and the effects that this induces on the user's psyche. In this paper, we present a system which translates the Itten theory into a formal language that allows us to express the semantics associated with the combination of chromatic properties of color images.

Effective image retrieval by content from database requires that visual image properties are used instead of textual labels to properly index and recover pictorial data. Retrieval by shape similarity, given a user-sketched template is particularly challenging, owing to the difficulty to derive a similarity measure that closely conforms to the common perception of similarity by humans. In this paper, we present a technique which is based on elastic matching of sketched templates over the shapes in the images to evaluate similarity ranks. The degree of matching achieved and the elastic deformation energy spent by the sketch to achieve such a match are used to derive a measure of similarity between the sketch and the images in the database and to rank images to be displayed. The elastic matching is integrated with arrangements to provide scale invariance and take into account spatial relationships between objects in multi-object queries. Examples from a prototype system are expounded with considerations about the effectiveness of the approach and comparative performance analysis.

Image retrieval by contents from database is a major research subject in advanced multimedia systems. Effective image retrieval by contents requires that visual image properties are used instead of textual labels to properly index and recover pictorial data. Retrieval by shape similarity given a user-sketched template is particularly challenging, owing to the difficulty to derive a similarity measure that closely conforms to the common perception of similarity by humans. In this paper we present a technique which takes into account global shape properties and is based on elastic matching of sketched templates over the shapes in the images to evaluate similarity ranks. The degree of matching achieved and the elastic deformation energy spent by the sketch to achieve such a match are used to derive a measure of similarity between the sketch and the images in the database and to rank images to be displayed.

Efficient image retrieval by contents from database requires that selective access methods are provided to prune out uninteresting items during the search process. Image indexing based on visual features is particularly challenging, owing to the difficulty to derive a representation of the shapes that closely models the visual appearance perceived by humans. In this paper we present a novel approach for indexing planar and closed curves, on the basis on their visual appearance. A hierarchical model of the curve is derived from its multi-scale analysis, and it is used to provide a description of the curve which is able to distinguish between its structural parts and its details. To cope with the inherent uncertainty of shape appearance, fuzzy sets are used to represent the visual attributes of the shapes. In this way the shapes which share similar structural parts can be gathered and embedded into an index structure which retains the imprecision of shape description by embedding fuzziness in the index itself.