Digital dance and theater archives are difficult to analyse and understand because few tools can accurately extract 3D motion from monocular images, which are often low-quality in heritage collections. We present two cloud-ready pipelines that transform low-quality videos into temporally dense SMPL-X reconstructions, per-frame segmentation masks, depth maps, and estimated camera trajectories. The PREMIERE pipeline targets archival material, while the MultiPerson pipeline is tuned for high-frame-rate smartphone or action camera recordings with up to five performers. Both pipelines couple Segment Anything v2.1, Neural Localizer Fields pose recovery, MoGe-based depth cues, WiLoR hand refinement, and VGGT for camera parameters estimation. This is followed by scale-aware optimization and RBF smoothing. Extensive testing on the AIST++ dance data set confirms that the resulting 3D assets surpass earlier monocular baselines in pose fidelity and temporal stability, while remaining robust to motion blur, extreme lighting, and human occlusion on stage film. Outputs conform to a unified geometry schema that feeds the Horizon PREMIERE project tools and two open-source WebGL viewers, enabling browser-based playback, VR/MR immersion, and high-resolution render export with no additional recording hardware. All codes, evaluation scripts, and viewers will be released under an open licence, offering a reproducible, extensible foundation for large-scale motion mining and the preservation of intangible cultural heritage. See interactive results on our project page: https://www.couleur.org/PREMIERE/JMTA/ .

Computational color constancy has seen strong improvement these last years due to the emergence of large labeled datasets. However, the models trained on images acquired by some cameras show low generalization power when tested on images acquired by other cameras. Indeed, since the light chromaticities are device dependent, the training distribution is very spread out when mixing different sensors. In this paper, we propose to inform the network that this complex distribution is a set of simpler distributions, one for each considered camera. For this purpose, we create a Siamese architecture trained with a specific contrastive loss. This loss enforces the model to predict light chromaticities in the same sensor distribution, when considering images acquired by the same camera and in different distributions for images from different sensors. The key idea consists in learning a specific color distance that is sensitive to only sensor variations and not to lighting variations. This learned distance is a nice tool to control if two chromaticity points are in the same sensor distribution or not. We test this original training process in the context of cross-camera color constancy and we show that it outperforms the alternatives on three datasets.

Bilinear pooling is a feature aggregation step applied after the convolutional layers of a deep network and encodes a matrix of local features into a fixed-size bilinear representation. It improves performance in many image classification tasks. Since its emergence, this pooling has seen two major improvements: Compact Bilinear Pooling (CBP) and square-root normalization. Recently, the combination of these two elements has been widely studied. However, due to the lack of good normalization solutions, existing combination approaches showed less efficiency when they are plugged into different networks and less compatibility when they work with existing CBP techniques. To solve this problem, in this paper, we propose to apply Newton iterations, a fast square-root normalization method, to produce a new normalized matrix calledpseudo square-rooted matrix. Subsequently, the new matrix allows a CBP technique to encode itself into a compact and normalized bilinear representation. In order to further accelerate the normalization process, our approach has two variants which can handle feature matrix extracted by different networks. Tested on three fine-grained image classification datasets, it provides competitive classification performance while consuming less computational time than other prior works.

The classical color constancy algorithms concentrate only on the color of a grey surface to estimate the light color and to white balance the image. In this paper, we show that the quality of the whole process can be clearly improved by predicting and correcting the colors of a set of reference surfaces. The ground truth of the surface color under white light can be easily obtained with a set of images acquired by the considered camera under sun light. Thus, we design a deep network to predict the colors of the reference surfaces of a color checker as if it had been in the scene at acquisition time. We show that our solution improves the two steps of the color constancy process on 9 datasets and we claim that being able to synthetically insert a color chart in any image can help for many other tasks.

International audience

Copy sensitive graphical codes are used as anti-counterfeiting solution in packaging and document protection. Their security is funded on a design hard-to-predict after print and scan. In practice there exist different designs. Here random codes printed at the printer resolution are considered. We suggest an estimation of such codes by using neural networks, an in-trend approach which has however not been studied yet in the present context. In this paper, we test a state-of-the-art architecture efficient in the binarization of handwritten characters. The results show that such an approach can be successfully used by an attacker to provide a valid counterfeited code so fool an authentication system.

Color constancy is the ability of the human visual system to perceive constant colors for a surface despite changes in the spectrum of the illumination. In computer vision, the main approach consists in estimating the illuminant color and then to remove its impact on the color of the objects. Many image processing algorithms have been proposed to tackle this problem automatically. However, most of these approaches are handcrafted and mostly rely on strong empirical assumptions, e.g., that the average reflectance in a scene is gray. State-of-the-art approaches can perform very well on some given datasets but poorly adapt on some others. In this paper, we have investigated how neural networks-based approaches can be used to deal with the color constancy problem. We have proposed a new network architecture based on existing successful hand-crafted approaches and a large number of improvements to tackle this problem by learning a suitable deep model. We show our results on most of the standard benchmarks used in the color constancy domain.

A workflow is proposed for Cultural Heritage applications in which the fusion of 3D and 2D visual data is required. Using data acquired by cheap, standard devices, like a 3D scanner having a low quality 2D camera in it, and a high resolution DSLR camera, one can produce high quality color calibrated 3D model for documenting purpose. The proposed processing workflow combines a novel region based calibration method with an ICP alignment used for refining the results. It works on 3D data, that do not necessarily contain intensity information in them, and 2D images of a calibrated camera. These can be acquired with commercial 3D scanners and color cameras without any special constraint. In contrast with the typical solutions, the proposed method is not using any calibration patterns or markers. The efficiency and robustness of the proposed calibration method has been confirmed on both synthetic and real data.

Abstract The objective of colour mapping or colour transfer methods is to recolour a given image or video by deriving a mapping between that image and another image serving as a reference. These methods have received considerable attention in recent years, both in academic literature and industrial applications. Methods for recolouring images have often appeared under the labels of colour correction, colour transfer or colour balancing, to name a few, but their goal is always the same: mapping the colours of one image to another. In this paper, we present a comprehensive overview of these methods and offer a classification of current solutions depending not only on their algorithmic formulation but also their range of applications. We also provide a new dataset and a novel evaluation technique called ‘evaluation by colour mapping roundtrip’. We discuss the relative merit of each class of techniques through examples and show how colour mapping solutions can have been applied to a diverse range of problems.

Recent advances in depth imaging sensors provide easy access to the synchronized depth with color, called RGB-D image. In this paper, we propose an unsupervised method for indoor RGB-D image segmentation and analysis. We consider a statistical image generation model based on the color and geometry of the scene. Our method consists of a joint color-spatial-directional clustering method followed by a statistical planar region merging method. We evaluate our method on the NYU depth database and compare it with existing unsupervised RGB-D segmentation methods. Results show that, it is comparable with the state of the art methods and it needs less computation time. Moreover, it opens interesting perspectives to fuse color and geometry in an unsupervised manner.

One of the main challenges in a projector-camera system is to be able to project on an arbitrary surface and compensate color changes due to the color of the projection surface. This task becomes dramatically more difficult in case the projection surface changes dynamically. Even a slight displacement of the surface breaks the photometric compensation and visible artifacts may occur. An approach that is able to compensate for a dynamic scene is of great importance since it expands the application range of such a system. In this paper, a method that requires only a single frame in order to adapt its photometric compensation to the new surface is demonstrated. Furthermore, it is compared with another state-of-the-art method that claims one-frame delay compensation. In addition, the complex characterization of a DLP projector is addressed. The results prove that not only our approach outperforms the pre-existing method but also it is robust against challenging surfaces with sharp and saturated color patches, as the ones in a real environment.

Automatic detection of moving objects is an important task for aerial surveillance. It has been a popular and well-studied subject for the computer vision community, but is still a challenge. The method we introduce targets surveillance low-altitude mini and micro-UAVs. We take advantage of the inherent image motion on footage captured by such aerial vehicles. Our method confronts Optical Flow vectors and an estimated Flow in order to detect independently moving pixels. This motion-based approach is robust to operational conditions and to the geometric properties of the scene. The efficiency of the method was computed on the VIVID database. The moving areas detected will make the tracking task more robust and efficient.

The architecture of computational models of visual attention designed for videos is generally the result of a direct extension of techniques dedicated to static images. These models try to extract the salient areas of dynamic scenes (i.e. areas that may capture visual attention) by fusing static saliency maps computed frame per frame with saliency maps independently obtained from dynamic features. The problem is that there is no evidence for assuming that visual saliency of videos can be accurately identified from such a fusion process. In addition, there is no guarantee that visual saliency is the same for still and dynamic scenes. Then we propose to investigate this issue for short videos from the perspective of color information that has been clearly identified as an important salient property in static images.

We address the problem of compensating color differences in image stitching. We explicitly color correct input images before image stitching by leveraging sparse correspondences. Our approach is twofold. First, for each geometric correspondence, we locally collect and process color information to be robust against low accuracy in geometric correspondences. Second, for all collected colors, we fit a global model that compensates complex color changes. Despite complex cases between input images (changes of illumination condition and imaging devices) qualitative experiments show good stitching of our method compared to recent methods in the literature.

Model Based Clustering (MBC) is a method that estimates a model for the data and produces probabilistic clustering. In this paper, we propose a novel MBC method to cluster three dimensional directional features. We assume that the features are generated from a finite statistical mixture model based on the von Mises-Fisher (vMF) distribution. The core elements of our proposed method are: (a) generate a set of vMF Mixture Models (vMFMM) and (b) select the optimal model using a parsimony based approach with information criteria. We empirically validate our proposed method by applying it on simulated data. Next, we apply it to cluster image normals in order to perform depth image analysis.

We propose a 3D structure estimation framework that adopts the slanted-planes representation in order to provide a dense estimation. The proposed approach fuses sparse 3D reconstructed point cloud obtained using several feature matching methods and noisy dense optical flow in order to perform accurate structure fitting and visually appealing results. We formulate the problem as a weighted total least square model that takes into account the occlusion boundaries between neighboring planes. We also propose an extended flow-based superpixel segmentation which is adaptive to the sparse feature points density for more balanced reconstruction. To validate our approach, we present 3D models obtained using the KITTI dataset [1] compared with other methods.

In this paper, we propose an unsupervised clustering method for axially symmetric directional unit vectors. Our method exploits the Watson distribution and Bregman Divergence within a Model Based Clustering framework. The main objectives of our method are: (a) provide efficient solution to estimate the parameters of a Watson Mixture Model (WMM), (b) generate a set of WMMs and (b) select the optimal model. To this aim, we develop: (a) an efficient soft clustering method, (b) a hierarchical clustering approach in parameter space and (c) a model selection strategy by exploiting information criteria and an evaluation graph. We empirically validate the proposed method using synthetic data. Next, we apply the method for clustering image normals and demonstrate that the proposed method is a potential tool for analyzing the depth image.

Visual saliency detection is an important step in many computer vision applications, since it reduces further processing steps to regions of interest. Saliency detection in still images is a well-studied topic. However, videos scenes contain more information than static images, and this additional temporal information is an important aspect of human perception. Therefore, it is necessary to include motion information in order to obtain spatio-temporal saliency map for a dynamic scene. In this paper, we introduce a new spatio-temporal saliency detection method for dynamic scenes based on dynamic textures computed with local binary patterns. In particular, we extract local binary patterns descriptors in two orthogonal planes (LBP-TOP) to describe temporal information, and color features are used to represent spatial information. The obtained three maps are finally fused into a spatio-temporal saliency map. The algorithm is evaluated on a dataset with complex dynamic scenes and the results show that our proposed method outperforms state-of-art methods.

We present an adaptive weight based superpixel segmentation method for the goal of creating mesh representation that respects the 3D scene structure. We propose a new fusion framework which employs both dense optical flow and color images to compute the probability of boundaries. The main contribution of this work is that we introduce a new color and optical flow pixel-wise weighting model that takes into account the non-linear error distribution of the depth estimation from optical flow. Experiments show that our method is better than the other state-of-art methods in terms of smaller error in the final produced mesh.

Visual saliency is an important research topic in computer vision applications, which helps to focus on regions of interest instead of processing the whole image. Detecting visual saliency in still images has been widely addressed in literature. However, visual saliency detection in videos is more complicated due to additional temporal information. A spatio-temporal saliency map is usually obtained by the fusion of a static saliency map and a dynamic saliency map. The way both maps are fused plays a critical role in the accuracy of the spatio-temporal saliency map. In this paper, we evaluate the performances of different fusion techniques on a large and diverse dataset and the results show that a fusion method must be selected depending on the characteristics, in terms of color and motion contrasts, of a sequence. Overall, fusion techniques which take the best of each saliency map (static and dynamic) in the final spatio-temporal map achieve best results.

The authors present a novel approach to improve three‐dimensional (3D) structure estimation from an image stream in urban scenes. The authors consider a particular setup, where the camera is installed on a moving vehicle. Applying traditional structure from motion (SfM) technique in this case generates poor estimation of the 3D structure because of several reasons such as texture‐less images, small baseline variations and dominant forward camera motion. The authors idea is to introduce the monocular depth cues that exist in a single image, and add time constraints on the estimated 3D structure. The scene is modelled as a set of small planar patches obtained using over‐segmentation, and the goal is to estimate the 3D positioning of these planes. The authors propose a fusion scheme that employs Markov random field model to integrate spatial and temporal depth features. Spatial depth is obtained by learning a set of global and local image features. Temporal depth is obtained via sparse optical flow based SfM approach. That allows decreasing the estimation ambiguity by forcing some constraints on camera motion. Finally, the authors apply a fusion scheme to create unique 3D structure estimation.

The perception of video is different from that of image because of the motion information in video. Motion objects lead to the difference between two neighboring frames which is usually focused on. By far, most papers have contributed to image saliency but seldom to video saliency. Based on scene understanding, a new video saliency detection model with multi-features is proposed in this paper. First, background is extracted based on binary tree searching, then main features in the foreground is analyzed using a multi-scale perception model. The perception model integrates faces as a high level feature, as a supplement to other low-level features such as color, intensity and orientation. Motion saliency map is calculated using the statistic of the motion vector field. Finally, multi-feature conspicuities are merged with different weights. Compared with the gaze map from subjective experiments, the output of the multi-feature based video saliency detection model is close to gaze map.

This work presents a robust technique for tracking a set of detected points on a human face. Facial features can be manually selected or automatically detected. We present a simple and efficient method for detecting facial features such as eyes and nose in a color face image. We then introduce a tracking method which, by employing geometric constraints based on knowledge about the configuration of facial features, avoid the loss of points caused by error accumulation and tracking drift. Experiments with different sequences and comparison with other tracking algorithms, show that the proposed method gives better results with a comparable processing time.

In this paper, we propose a compact illumination invariant color descriptor. Recent papers have shown that the rank measures of the pixels within a color image are invariant across illumination changes. We exploit this characteristic by measuring the rank correlation between different color components for pixels located at a particular distance from each other. This measure which takes into account both the color distribution and the spatial interactions between the pixels is stable across illumination changes. Furthermore, we show that 18 correlation measures are almost sufficient to discriminate 1000 objects and provide better results than classical invariant indexes which require much more memory space.

In the context of object recognition, it is useful to extract, from the images, efficient indexes that are insensitive to the illumination conditions, to the camera scale factor and to the 2D position and orientation of the object. In this paper, we propose to cope with this invariance problem by normalizing the images according to these parameters in a preprocessing step. This spatio-colorimetric normalization transforms the images so that each pixel get a new position and a new color. These position and color are evaluated according to both the colors and the relative positions of all the pixels in the original image. The comparison of two images is then processed by evaluating the sum of the similarity measures between local indexes extracted at the same positions and scales in the two images. The invariance and the discriminating power of our approach is assessed on a public database.

This article proposes a local photometric model that compensates for specular highlights and lighting variations due to position and intensity changes. We define clearly on which assumptions it is based, according to widely used reflection models. Moreover, its theoretical validity is studied according to few configurations of the scene geometry (lighting, camera and object relative locations). Next, this model is used to improve the robustness of points tracking in luminance images with respect to specular highlights and lighting changes.

In this article, we study the color appearance of the ink printed on a background, according to both its concentration and the background color. We find some attributes, the concentration quotients ratios, that are more invariant to the ink concentration than simple color attributes. Our work deals with traceability of porcine products. We have to detect the animal identifier, printed with ink on the pork rind. Using the concentration quotients ratios, our segmentation technique succeeds for any quantity of ink and any hue of pork rind. This technique could be applied to segment any set of pixels, that are colorimetrically and spatially close, but not necessarily all connected.

The aim of this paper is to present different algorithms, based on a combination of two structures of graph and of two color image processing methods, in order to segment color images. The structures used in this study are the region adjacency graph and the line graph associated.We will see how these structures can enhance segmentation processes such as region growing or watershed transformation. The principal advantage of these structures is that they give more weight to adjacency relationships between regions than usual methods. Let us note nevertheless that this advantage leads in return to adjust more parameters than other methods to best refine the result of the segmentation.We will show that this adjustment is necessarily image dependent and observer dependent.

In order to optimize the codebook used by the vector quantization compression scheme, we have developed a process based on the max-min algorithm. This process optimizes color space partitioning from vector blocks selected iteratively within the training set according to three algorithms. The partitioning algorithm is based on the nearest neighbor query. The selection algorithm searches the furthest color of the nearest vector block of the training set already computed. A centroid process generates the codebook in refining the vector block selection. In order to counterbalance cases of study for which the centroid process modifies the vector block selection, we have introduced three tests. These tests restrict the training set from which representative colors can be selected.

Color image quantization is a process of selecting a set of colors to display an image with some representative colors, this without noticeable perceived difference. Whatever the matter we deal with; (1) the definition of a relevant quantization process, or (2) the evaluation of the possible errors linked to this quantization, the problem remains the same: the definition of an objective criterion which takes into account the perceptual aspect of the degradation observation between an original image and its quantized representation. In this paper we deal with the problem of the quantization errors evaluation in taking into account two new criteria: the LSE measure and the SCAP measure. Knowing that at least these new evaluation criteria will be used in quantization techniques in order to minimize the possible degradations that would be perceived.>

In order to extend the field of pattern segmentation applications and to improve the accuracy of the pattern segmentation processes, we propose to use several criteria linked to a local spatiocolor analysis. Indeed, it is proved that spatial interactions between adjacent pixels play an important role in pattern analysis. Consequently, we must take into account the spatial distribution of color information to obtain a relevant pattern segmentation. With such informations it becomes easier to deal with the aspect of local color similarity and dissimilarity between spatial adjacent colors. These two notions may be used to characterize each pattern independently of their surrounding patterns. In that way, we can use a local analysis based on the neighborhood of each pixel of the pattern under study, provided this neighborhood is restricted to the spatiocolor distribution of this pattern.>