Recently, significant efforts were made towards Variational Autoencoder (VAE) -based prototypical Self Explainable Models (SEM) for image classification. The principle is to learn class-specific prototypes that can be projected back into the image space thanks to the decoding branch of a VAE. However, existing VAE-based SEM fail to represent properly the distribution of training samples in the embedding space requiring to define specific additional constraints as diversity or orthogonality. In this work, we propose to define the prototypes as the components of a Gaussian Mixture of VAE (GMVAE) that is an approximation of the distribution of training samples. We show that this definition allows to produce relevant and diverse prototypes providing a probabilistic explanation of the model without assigning prototypes to a specific class. We support our definition with extensive experimentation and comparison with previous self-explainable approaches.

In this paper, we introduce Slope-Track. Slope-Track is a novel multiple object tracking (MOT) dataset designed to reflect the complexities of real ski slope environments. The dataset has over 96,000 frames collected from 10 different ski resorts under various weather and visibility conditions. Slope-Track addresses significant challenges in slope monitoring, including small object sizes, occlusions, fast and irregular motion, and low appearance consistency. It is densely annotated with bounding boxes and object identities, facilitating the evaluation of detection and tracking algorithms. We analyze the dataset’s characteristics comparing it to the existing MOT datasets. The results demonstrate that Slope-Track encapsulates a combination of challenges found in other datasets. Additionally, we benchmark a range of existing tracking algorithms and propose a new module that improves motion-based association by dealing with the specific shape of trajectories along ski slopes. Our results demonstrate that incorporating appearance features can have a mixed impact, depending on how they are used within each tracking algorithm. In contrast, motion-based methods and spatial association strategies show more reliable performance. Overall, we provide a challenging benchmark for evaluating and improving multi-object tracking systems in real-world outdoor environments. The dataset and code can be found at https://slopetrack.github.io/ .

Digital dance and theater archives are difficult to analyse and understand because few tools can accurately extract 3D motion from monocular images, which are often low-quality in heritage collections. We present two cloud-ready pipelines that transform low-quality videos into temporally dense SMPL-X reconstructions, per-frame segmentation masks, depth maps, and estimated camera trajectories. The PREMIERE pipeline targets archival material, while the MultiPerson pipeline is tuned for high-frame-rate smartphone or action camera recordings with up to five performers. Both pipelines couple Segment Anything v2.1, Neural Localizer Fields pose recovery, MoGe-based depth cues, WiLoR hand refinement, and VGGT for camera parameters estimation. This is followed by scale-aware optimization and RBF smoothing. Extensive testing on the AIST++ dance data set confirms that the resulting 3D assets surpass earlier monocular baselines in pose fidelity and temporal stability, while remaining robust to motion blur, extreme lighting, and human occlusion on stage film. Outputs conform to a unified geometry schema that feeds the Horizon PREMIERE project tools and two open-source WebGL viewers, enabling browser-based playback, VR/MR immersion, and high-resolution render export with no additional recording hardware. All codes, evaluation scripts, and viewers will be released under an open licence, offering a reproducible, extensible foundation for large-scale motion mining and the preservation of intangible cultural heritage. See interactive results on our project page: https://www.couleur.org/PREMIERE/JMTA/ .

Computational color constancy has seen strong improvement these last years due to the emergence of large labeled datasets. However, the models trained on images acquired by some cameras show low generalization power when tested on images acquired by other cameras. Indeed, since the light chromaticities are device dependent, the training distribution is very spread out when mixing different sensors. In this paper, we propose to inform the network that this complex distribution is a set of simpler distributions, one for each considered camera. For this purpose, we create a Siamese architecture trained with a specific contrastive loss. This loss enforces the model to predict light chromaticities in the same sensor distribution, when considering images acquired by the same camera and in different distributions for images from different sensors. The key idea consists in learning a specific color distance that is sensitive to only sensor variations and not to lighting variations. This learned distance is a nice tool to control if two chromaticity points are in the same sensor distribution or not. We test this original training process in the context of cross-camera color constancy and we show that it outperforms the alternatives on three datasets.

Deep learning for automated cell imaging analysis has become a tool of choice to process large amounts of data. But many of these methods lack explainability, slowing down their deployment for tasks such as diagnosis. We present a prototype-based framework to analyze structural changes which addresses the specific challenges of explainability in the context of cell imaging. Our method relies on classification between two distinct cell populations in a weakly supervised context where no label for individual cells is available. Our model extracts typical features from each population, representing intra-cellular structure, and provides an explanation on its classification decision by creating visualization of the local textures corresponding to the structures of interest. We show a real application where it effectively highlights a change in the organization of the actin content of the cells.

Bilinear pooling is a feature aggregation step applied after the convolutional layers of a deep network and encodes a matrix of local features into a fixed-size bilinear representation. It improves performance in many image classification tasks. Since its emergence, this pooling has seen two major improvements: Compact Bilinear Pooling (CBP) and square-root normalization. Recently, the combination of these two elements has been widely studied. However, due to the lack of good normalization solutions, existing combination approaches showed less efficiency when they are plugged into different networks and less compatibility when they work with existing CBP techniques. To solve this problem, in this paper, we propose to apply Newton iterations, a fast square-root normalization method, to produce a new normalized matrix calledpseudo square-rooted matrix. Subsequently, the new matrix allows a CBP technique to encode itself into a compact and normalized bilinear representation. In order to further accelerate the normalization process, our approach has two variants which can handle feature matrix extracted by different networks. Tested on three fine-grained image classification datasets, it provides competitive classification performance while consuming less computational time than other prior works.

The classical color constancy algorithms concentrate only on the color of a grey surface to estimate the light color and to white balance the image. In this paper, we show that the quality of the whole process can be clearly improved by predicting and correcting the colors of a set of reference surfaces. The ground truth of the surface color under white light can be easily obtained with a set of images acquired by the considered camera under sun light. Thus, we design a deep network to predict the colors of the reference surfaces of a color checker as if it had been in the scene at acquisition time. We show that our solution improves the two steps of the color constancy process on 9 datasets and we claim that being able to synthetically insert a color chart in any image can help for many other tasks.

Color constancy is the ability of the human visual system to perceive constant colors for a surface despite changes in the spectrum of the illumination. In computer vision, the main approach consists in estimating the illuminant color and then to remove its impact on the color of the objects. Many image processing algorithms have been proposed to tackle this problem automatically. However, most of these approaches are handcrafted and mostly rely on strong empirical assumptions, e.g., that the average reflectance in a scene is gray. State-of-the-art approaches can perform very well on some given datasets but poorly adapt on some others. In this paper, we have investigated how neural networks-based approaches can be used to deal with the color constancy problem. We have proposed a new network architecture based on existing successful hand-crafted approaches and a large number of improvements to tackle this problem by learning a suitable deep model. We show our results on most of the standard benchmarks used in the color constancy domain.

Domain adaptation (DA) has gained a lot of success in the recent years in computer vision to deal with situations where the learning process has to transfer knowledge from a source to a target domain. In this paper, we introduce a novel unsupervised DA approach based on both subspace alignment and selection of landmarks similarly distributed between the two domains. Those landmarks are selected so as to reduce the discrepancy between the domains and then are used to non linearly project the data in the same space where an efficient subspace alignment (in closed-form) is performed. We carry out a large experimental comparison in visual domain adaptation showing that our new method outperforms the most recent unsupervised DA approaches.

We present a supervised learning to rank algorithm that effectively orders images by exploiting the structure in image sequences. Most often in the supervised learning to rank literature, ranking is approached either by analysing pairs of images or by optimizing a list-wise surrogate loss function on full sequences. In this work we propose MidRank, which learns from moderately sized sub-sequences instead. These sub-sequences contain useful structural ranking information that leads to better learnability during training and better generalization during testing. By exploiting sub-sequences, the proposed MidRank improves ranking accuracy considerably on an extensive array of image ranking applications and datasets.

Estimating the age of historical photographs is a challenging task for human beings. Only recently this task has been addressed in computational image analysis perspective. The characteristics of the device used to acquire each photograph are discriminative features for this task. We aim at extracting such characteristics from a historical color photographs. The acquisition device mainly effects two properties of the colors: the distribution of their derivatives and the angles drawn by three consecutive pixels in the RGB space. We propose two color features that take advantage of these observations. We show that these two color descriptors (namely color derivatives and color angles) attain the state-of-the-art in the context of image dating.

Color description is a challenging task because of large variations in RGB values which occur due to scene accidental events, such as shadows, shading, specularities, illuminant color changes, and changes in viewing geometry. Traditionally, this challenge has been addressed by capturing the variations in physics-based models, and deriving invariants for the undesired variations. The drawback of this approach is that sets of distinguishable colors in the original color space are mapped to the same value in the photometric invariant space. This results in a drop of discriminative power of the color description. In this paper we take an information theoretic approach to color description. We cluster color values together based on their discriminative power in a classification problem. The clustering has the explicit objective to minimize the drop of mutual information of the final representation. We show that such a color description automatically learns a certain degree of photometric invariance. We also show that a universal color representation, which is based on other data sets than the one at hand, can obtain competing performance. Experiments show that the proposed descriptor outperforms existing photometric invariants. Furthermore, we show that combined with shape description these color descriptors obtain excellent results on four challenging datasets, namely, PASCAL VOC 2007, Flowers-102, Stanford dogs-120 and Birds-200.

We propose a method to acquire multispectral data with handheld devices with front-mounted RGB cameras. We propose to use the display of the device as an illuminant while the camera captures images illuminated by the red, green and blue primaries of the display. Three illuminants and three response functions of the camera lead to nine response values which are used for reflectance estimation. Results are promising and show that the accuracy of the spectral reconstruction improves in the range of 30-40% over the spectral reconstruction based on a single illuminant. Furthermore, we propose to compute sensor-illuminant aware linear basis by discarding the part of the reflectances that falls in the sensor-illuminant null-space. We show experimentally that optimizing reflectance estimation on these new basis functions decreases the RMSE significantly over basis functions that are independent to sensor-illuminant. We conclude that, multispectral data acquisition is potentially possible with consumer hand-held devices such as tablets, mobiles, and laptops, opening up applications which are currently considered to be unrealistic.

International audience

Bag-of-words-based image classification approaches mostly rely on low level local shape features. However, it has been shown that combining multiple cues such as color, texture, or shape is a challenging and promising task which can improve the classification accuracy. Most of the state-of-the-art feature fusion methods usually aim to weight the cues without considering their statistical dependence in the application at hand. In this paper, we present a new logistic regression-based fusion method, called LRFF, which takes advantage of the different cues without being tied to any of them. We also design a new marginalized kernel by making use of the output of the regression model. We show that such kernels, surprisingly ignored so far by the computer vision community, are particularly well suited to achieve image classification tasks. We compare our approach with existing methods that combine color and shape on three datasets. The proposed learning-based feature fusion process clearly outperforms the state-of-the art fusion methods for image classification.

In this paper, we propose a compact illumination invariant color descriptor. Recent papers have shown that the rank measures of the pixels within a color image are invariant across illumination changes. We exploit this characteristic by measuring the rank correlation between different color components for pixels located at a particular distance from each other. This measure which takes into account both the color distribution and the spatial interactions between the pixels is stable across illumination changes. Furthermore, we show that 18 correlation measures are almost sufficient to discriminate 1000 objects and provide better results than classical invariant indexes which require much more memory space.

In the context of object recognition, it is useful to extract, from the images, efficient indexes that are insensitive to the illumination conditions, to the camera scale factor and to the 2D position and orientation of the object. In this paper, we propose to cope with this invariance problem by normalizing the images according to these parameters in a preprocessing step. This spatio-colorimetric normalization transforms the images so that each pixel get a new position and a new color. These position and color are evaluated according to both the colors and the relative positions of all the pixels in the original image. The comparison of two images is then processed by evaluating the sum of the similarity measures between local indexes extracted at the same positions and scales in the two images. The invariance and the discriminating power of our approach is assessed on a public database.

In this paper, we propose an original scheme to retrieve among all the target images of a database, those which contain the same object as that represented by the query image, these images being acquired under different illumination conditions. Rather than considering the color vectors of the pixels to characterize the images, we propose to introduce and exploit the concept of spatial ranks of CCD sensor responses. Indeed, these values are preserved in case of illumination changes and they take into account both the colors of the pixels and the spatial interactions between them in the image. Since we can not determine these ranks from a color image, we propose to estimate their probabilities of occurrences thanks to fuzzy functions. These probabilities are used by our object recognition scheme whose effectiveness is assessed with a public database that contains images of objects acquired under different illuminations