International audience

Abstract Herein, the problem of vehicle re‐identification using distance comparison of images in CNN latent spaces is addressed. First, the impact of the distance metrics, comparing performances obtained with different metrics is studied: the minimal Euclidean distance ( MED ), the minimal cosine distance ( MCD ) and the residue of the sparse coding reconstruction ( RSCR ). These metrics are applied using features extracted from five different CNN architectures, namely ResNet18, AlexNet, VGG16, InceptionV3 and DenseNet201. We use the specific vehicle re‐identification dataset VeRi to fine‐tune these CNNs and evaluate results. Overall, independently of the CNN used, MCD outperforms MED , commonly used in the literature. These results are confirmed on other vehicle retrieval datasets. Second, the state‐of‐the‐art image‐to‐track process (I2TP) is extended to a track‐to‐track process (T2TP). The three distance metrics are extended to measure distance between tracks, enabling T2TP. T2TP and I2TP are compared using the same CNN models. Results show that T2TP outperforms I2TP for MCD and RSCR. T2TP combining DenseNet201 and MCD ‐based metrics exhibits the best performances, outperforming the state‐of‐the‐art I2TP‐based models. Finally, experiments highlight two main results: i) the impact of metric choice in vehicle re‐identification, and ii) T2TP improves the performances compared with I2TP, especially when coupled with MCD ‐based metrics.

Determining the links between large amounts of video data with no prior knowledge of the camera positions is a hard task to automate. From a collection of videos acquired from static cameras simultaneously, we propose a method for finding groups of videos with overlapping fields of view. Each video is first processed individually: at regular time steps, objects are detected and are assigned a category and an appearance descriptor. Next, the video is split into cells at different resolutions and we assign to each cell its story: it consists of the list of objects detected in the cell over time. Once the stories are established for each video, the links between cells of different videos are determined by comparing their stories: two cells are linked if they show simultaneous detections of objects of the same category with similar appearances. Pairs of videos with overlapping fields of view are identified using these links between cells. A link graph is finally returned, in which each node represents a video, and the edges indicate pairs of overlapping videos. The approach is evaluated on a set of 63 real videos from both public datasets and live surveillance videos, as well as on 84 synthetic videos, and shows promising results.

In this paper, we propose an approach for segmentation of challenging depth images. We first use a semi-automatic segmentation algorithm that only takes a user-defined rectangular area as an input. The quality of the segmentation is very heterogeneous at this stage, and unsufficient to efficiently train a neural network. We thus introduce a learning process that takes this imperfect nature of data into account, by iteratively filtering the dataset to only keep the best segmented images. We show this method improves the neural network's performance by a significant amount.

We present an approach for ranking a collection of videos with overlapping fields of view. The ranking depends on how they allow to visualize as best as possible, i.e. with significant details, a trajectory query drawn in one of the videos. The proposed approach decomposes each video into cells and aims at estimating a correspondence map between cells from different videos using the linear correlation between their functions of activity. These latter are obtained during a training session by detecting objects in the videos and computing the coverage rate between the objects and the cells over time. The main idea is that two areas from two different videos that systematically offer presence of objects simultaneously are very likely to correspond to each other. Then, we use the correspondence between cells to find the reformulated trajectory in the other videos. Finally, we rank the videos based on the visibility they offer. We show promising results by testing three aspects: the correspondence maps, the reformulation and the ranking.

Navigating in a 3D networked virtual environment with a six-degree of freedom on a mobile device can be disorientating and challenging to users. In this technical demonstration, we show how 3D bookmarks can be used to simplify such interactions. Our system integrates the 3D bookmarks into a DASH-based network virtual environment in a DASH-compliant manner and is available on both the desktop and mobile.

In the context of 2D/3D registration, this paper introduces an approach that allows for matching features detected in two different modalities, photographs, and 3D models, by using a common 2D representation. More precisely, 2D images are matched with a set of depth images representing the 3D model. After introducing the concept of Curvilinear Saliency, which is related to curvature estimation, we propose a new ridge and valley detector for depth images rendered from 3D models. A variant of this detector is adapted to photographs, first by considering multi-scale features and second by integrating the focus curve principle. Finally, a registration algorithm determines the correct view of the 3D model and, thus, the pose of the photograph. This approach relies on the Histogram of Curvilinear Saliency (HCS), an adaptation of the Histogram of Oriented Gradients (HOG) to the proposed features in 2D and 3D. The presented results highlight both the quality of the features detected in terms of repeatability and the interest of the approach for registration and pose estimation.

We demonstrate the use of DASH, a widely-deployed standard for streaming video content, for streaming 3D content in an NVE (Networked Virtual Environment) consisting of 3D geometry and associated textures. We have developed a DASH client for NVE to show how NVE benefits from the advantages of DASH: it offers a scalable, easy-to-deploy 3D streaming framework. In our system, the 3D content is first statically partitioned into compliant DASH data, and metadata is provided in order for the client to manage which data to download. Based on a proposed utility metric for geometry and texture at the different resolution, the client can choose the content to request depending on its viewpoint. We effectively provide a Web-based client to navigate through our sample 3D scene, while deriving the streaming requests from its computation of the necessary online parameters, in a receiver-driven manner.

DASH is now a widely deployed standard for streaming video content due to its simplicity, scalability, and ease of deployment. In this paper, we explore the use of DASH for a different type of media content -- networked virtual environment (NVE), with different properties and requirements. We organize a polygon soup with textures into a structure that is compatible with DASH MPD (Media Presentation Description), with a minimal set of view-independent metadata for the client to make intelligent decisions about what data to download at which resolution. We also present a DASH-based NVE client that uses a view-dependent and network dependent utility metric to decide what to download, based only on the information in the MPD file. We show that DASH can be used on NVE for 3D content streaming. Our work opens up the possibility of using DASH for highly interactive applications, beyond its current use in video streaming.

We introduce a new application for annotating images, with the purpose of constituting training datasets for machine learning algorithms. Our open-source software is meant to be easily used and deployed, configured to meet the annotation needs of any use case, and embeddable in crowdsourcing campaigns using the Amazon Mechanical Turk service.

In surveillance applications, humans and vehicles are the most important common elements studied. In consequence, detecting and matching a person or a car that appears on several videos is a key problem. Many algorithms have been introduced and nowadays, a major relative problem is to evaluate precisely and to compare these algorithms, in reference to a common ground-truth. In this paper, our goal is to introduce a new dataset for evaluating multi-view based methods. This dataset aims at paving the way for multidisciplinary approaches and applications such as 4D-scene reconstruction, object identification/tracking, audio event detection and multi-source meta-data modeling and querying. Consequently, we provide two sets of 25 synchronized videos with audio tracks, all depicting the same scene from multiple viewpoints, each set of videos following a detailed scenario consisting in comings and goings of people and cars. Every video was annotated by regularly drawing bounding boxes on every moving object with a flag indicating whether the object is fully visible or occluded, specifying its category (human or vehicle), providing visual details (for example clothes types or colors), and timestamps of its apparitions and disappearances. Audio events are also annotated by a category and timestamps.

Interactive segmentation consists in building a pixel-wise partition of an image, into foreground and background regions, with the help of user inputs. Most state-of-the-art algorithms use scribble-based interactions to build foreground and background models, and very few of these work focus on the usability of the scribbling interaction. In this paper we study the outlining interaction, which is very intuitive to non-expert users on touch devices. We present an algorithm, built upon the existing GrabCut algorithm, which infers both foreground and background models out of a single outline. We conducted a user study on 20 participants to demonstrate the usability of this interaction, and its performance for the task of interactive segmentation.

Object recognition based on local features computed at multiple locations is robust to occlusions, strong viewpoint changes and object deformations. These features should be repeatable, precise and distinctive. We present an operator for repeatable feature detection on depth images (relative to 3D models) as well as 2D intensity images. The proposed detector is based on estimating the curviness saliency at multiple scales in each kind of image. We also propose quality measures that evaluate the repeatability of the features between depth and intensity images. The experiments show that the proposed detector outperforms both the most powerful, classical point detectors (e.g., SIFT) and edge detection techniques.

A 3D bookmark in a networked virtual environment (NVE) provides a navigation aid, allowing the user to move quickly from its current viewpoint to a bookmarked viewpoint by simply clicking on the bookmark. In this paper, we first validate the positive impact that 3D bookmarks have in easing navigation in a 3D scene. Then, we show that, in the context of a NVE that streams content on demand from server to client, navigating with bookmarks leads to lower rendering quality at the bookmarked viewpoint, due to lower locality of data. We then investigate into how prefetching the 3D data at the bookmarks and precomputation of visible faces at the bookmarks help to improve the rendering quality.

This paper explores processing techniques to deal with noisy data in crowdsourced object segmentation tasks. We use the data collected with Click'n'Cut, an online interactive segmentation tool, and we perform several experiments towards improving the segmentation results. First, we introduce different superpixel-based techniques to filter users' traces, and assess their impact on the segmentation result. Second, we present different criteria to detect and discard the traces from potential bad users, resulting in a remarkable increase in performance. Finally, we show a novel superpixel-based segmentation algorithm which does not require any prior filtering and is based on weighting each user's contribution according to his/her level of expertise.

Reducing seek latency by predicting what the users will access is important for user experience, particularly during video browsing, where users seek frequently to skim through a video. Much existing research strived to predict user access pattern more accurately to improve the prefetching hit rate. This paper proposed a different approach whereby the prefetch hit rate is improved by biasing the users to seek to prefetched content with higher probability, through changing the video player user interface. Through a user study, we demonstrated that our player interface can lead to up to 4$\times$ more seeks to bookmarked segments and reduce seek latency by 40\%, compared to a video player interface commonly used today. The user study also showed that the user experience and the understanding of the video content when browsing is not compromised by the changes in seek behavior.

In this paper we present the Toulouse Vanishing Points Dataset, a public photographs database of Manhattan scenes taken with an iPad Air 1. The purpose of this dataset is the evaluation of vanishing points estimation algorithms. Its originality is the addition of Inertial Measurement Unit (IMU) data synchronized with the camera under the form of rotation matrices. Moreover, contrary to existing works which provide vanishing points of reference in the form of single points, we computed uncertainty regions. The Toulouse Vanishing Points Dataset is publicly available at http://ubee.enseeiht.fr/tvpd

Videos are becoming more and more a tool of communication. There are how-to videos, people are discussing actions of others based on their recorded performance, e.g., in soccer, or they simply record videos of great moments and show them to friends and family. In this paper we focus on very specific how-to videos and present a novel, web based annotation tool, that combines (i) zoom, (ii) drawing, and (iii) temporal social bookmarking in video streams. Moreover, we present a short study on the usefulness of the tool to communicate general concepts of a specific video game based on a captured game session.

We consider an emerging situation where multiple cameras are filming the same event simultaneously from a diverse set of angles. The captured videos provide us with the multiple view geometry and an understanding of the 3D structure of the scene. We further extend this understanding by introducing the concept of 3D interest map in this paper. As most users naturally film what they find interesting from their respective viewpoints, the 3D structure can be annotated with the level of interest, naturally crowdsourced from the users. A 3D interest map can be understood as an extension of saliency maps in the 3D space that captures the semantics of the scene. We evaluate the idea of 3D interest maps on two real datasets, taken from the environment or the cameras that are equipped enough to have an estimation of the poses of cameras and a reasonable synchronization between them. We study two aspects of the 3D interest maps in our evaluation. First, by projecting them into 2D, we compare them to state-of-the-art saliency maps. Second, to demonstrate the usefulness of the 3D interest maps, we apply them to a video mashup system that automatically produces an edited video from one of the datasets.

In this demonstration, we show an automated mobile video mashup system that takes a set of videos filming the same scene as input, and generate an output mashup video consisting of temporally coherent clips selected from these input videos. The key difference of our system over the existing state-of-the-art is that it can generate virtual close-up shots and three camera operations: zooming in, zooming out, and panning, automatically. To achieve this, the system first computes the motion maps of the input videos and then determines a set of rectangles that correspond to highly interesting regions (in terms of motion). The choice of which shot types to use is done heuristically, ensuring diversity and coherency in the content presented in the mashup.

High-resolution panoramic video with a wide field-of-view is popular in many contexts. However, in many examples, like surveillance and sports, it is often desirable to zoom and pan into the generated video. A challenge in this respect is real-time support, but in this demo, we present an end-to-end real-time panorama system with interactive zoom and panning. Our system installed at Alfheim stadium, a Norwegian premier league soccer team, generates a cylindrical panorama from five 2K cameras live where the perspective is corrected in real-time when presented to the client. This gives a better and more natural zoom compared to existing systems using perspective panoramas and zoom operations using plain crop. Our experimental results indicate that virtual views can be generated far below the frame-rate threshold, i.e., on a GPU, the processing requirement per frame is about 10 milliseconds.

Panorama video is becoming increasingly popular, and we present an end-to-end real-time system to interactively zoom and pan into high-resolution panoramic videos. Compared to existing systems using perspective panoramas with cropping, our approach creates a cylindrical panorama. Here, the perspective is corrected in real-time, and the result is a better and more natural zoom. Our experimental results also indicate that such zoomed virtual views can be generated far below the frame-rate threshold. Taking into account recent trends in device development, our approach should be able to scale to a large number of concurrent users in the near future.

Online galleries of 3D models typically provide two ways to preview a model before the model is downloaded and viewed by the user: (i) by showing a set of thumbnail images of the 3D model taken from representative views (or keyviews); (ii) by showing a video of the 3D model as viewed from a moving virtual camera along a path determined by the content provider. We propose a third approach called preview streaming for mesh-based 3D objects: by streaming and showing parts of the mesh surfaces visible along the virtual camera path. This article focuses on the preview streaming architecture and framework and presents our investigation into how such a system would best handle network congestion effectively. We present three basic methods: (a) stop-and-wait , where the camera pauses until sufficient data is buffered; (b) reduce-speed , where the camera slows down in accordance to reduce network bandwidth; and (c) reduce-quality , where the camera continues to move at the same speed but fewer vertices are sent and displayed, leading to lower mesh quality. We further propose two advanced methods: (d) keyview-aware , which trades off mesh quality and camera speed appropriately depending on how close the current view is to the keyviews, and (e) adaptive-zoom , which improves visual quality by moving the virtual camera away from the original path. A user study reveals that our keyview-aware method is preferred over the basic methods. Moreover, the adaptive-zoom scheme compares favorably to the keyview-aware method, showing that path adaptation is a viable approach to handling bandwidth variation.

Publishers of 3D models online typically provide two ways to preview a model before the model is downloaded and viewed by the user: (i) by showing a set of thumbnail images of the 3D model taken from representative views (or keyviews); (ii) by showing a video of the 3D model as viewed from a moving virtual camera along a path determined by the content provider. We propose a third approach called preview streaming for mesh-based 3D object: by streaming and showing parts of the mesh surfaces visible along the virtual camera path. This paper focuses on the preview streaming architecture and framework, and presents our investigation into how such a system would best handle network congestion effectively. We study three basic methods: (a) stop-and-wait, where the camera pauses until sufficient data is buffered; (b) reduce-speed, where the camera slows down in accordance to reduce network bandwidth; and (c) reduce-quality, where the camera continues to move at the same speed but fewer vertices are sent and displayed, leading to lower mesh quality. We further propose a keyview-aware method that trades off mesh quality and camera speed appropriately depending on how close the current view is to the keyviews. A user study reveals that our keyview-aware method is preferred over the basic methods.

A C2Tag refers to a set of concentric circles of different radii. C2Tags have been recently introduced in computer vision, in particular for camera calibration, as they offer highly interesting photometric and geometric properties, compared to the classical “checkerboard” tags. In this work, we propose the general paradigm of camera tracking based on a planar marker consisting of at least two C2Tags. All the involved steps are described: detection, identification, 2D reconstruction, calibration and 3D reconstruction. Our contribution is to introduce the following missing steps required to deal with long video sequences under time constraints: key-frame selection, bundle adjustment and intermediate pose refinement.

This paper introduces a new paradigm for interacting with zoomable video. Our interaction technique reduces the number of zooms and pans required by providing recommended viewports to the users, and replaces multiple zoom and pan actions with a simple click on the recommended viewport. The usefulness of our technique is visible in the quality of the recommended viewport, which needs to match the user intention, track movement in the scene, and properly frame the scene in the video. To this end, we propose a hybrid method where content analysis is complimented by the implicit feedback of a community of users in order to recommend viewports. We first compute preliminary sets of recommended viewports by analyzing the content of the video. These viewports allow tracking of moving objects in the scene, and are framed without violating basic aesthetic rules. To improve the relevance of the recommended viewports, we collect viewing statistics as users view a video, and use the viewports they select to reinforce the importance of certain recommendations and penalize others. New recommendations that are not previously recognized by content analysis may also emerge. The resulting recommended viewports converge towards the regions in the video that are relevant to users. A user study involving 70 participants shows that an user interface incorporating with our paradigm leads to more number of zooms, into more informative regions with fewer interactions.

This paper presents an algorithm for segmenting the hair region in uncontrolled, real life conditions images. Our method is based on a simple statistical hair shape model representing the upper hair part. We detect this region by minimizing an energy which uses active shape and active contour. The upper hair region then allows us to learn the hair appearance parameters (color and texture) for the image considered. Finally, those parameters drive a pixel-wise segmentation technique that yields the desired (complete) hair region. We demonstrate the applicability of our method on several real images.

Screen size and display resolution limit the experience of watching videos on mobile devices. The viewing experience can be improved by determining important or interesting regions within the video (called regions of interest, or ROIs) and displaying only the ROIs to the viewer. Previous work focuses on analyzing the video content using visual attention model to infer the ROIs. Such content-based technique, however, has limitations. In this paper, we propose an alternative paradigm to infer ROIs from a video. We crowdsource from a large number of users through their implicit viewing behavior using a zoom and pan interface, and infer the ROIs from their collective wisdom. A retargeted video, consisting of relevant shots determined from historical users behavior, can be automatically generated and replayed to subsequent users who would prefer a less interactive viewing experience. This paper presents how we collect the user traces, infer the ROIs and their dynamics, group the ROIs into shots, and automatically reframe those shots to improve the aesthetics of the video. A user study with 48 participants shows that our automatically retargeted video is of comparable quality to one handcrafted by an expert user

Nowadays, commercial websites use an increasing amount of rich media ads, since these ads grab users' attention more easily. Ad overloading has a counterproductive effect and we advocate a more parsimonious and finely-tuned use of ads. In this paper we stress the importance of using ads in the right sequence and at the right time. We also present how to use our web adaptation system for a classic web marketing multivariable test. The experimental results show the usefulness of optimizing banners' sequences and timing.

We present MARCH, a complete solution for enhanced cultural heritage discovery using the mobile phone. Simply point the camera of a mobile device at prehistoric cave engravings. Then MARCH augments the captured images with the expert's drawings, highlighting in real time the animal engravings, which are almost impossible to observe with the naked eye. We have created a mobile augmented reality application which runs at 14 FPS for 320x240 frames on a Nokia N95 smartphone. We describe the optimizations and the requirements needed to obtain these results on mobile devices.

This paper presents a satellite image compression scheme based on a post-processing of the wavelet transform of images. The bandelet transform is a directional post-processing of wavelet coefficients. Thanks to a low computational complexity, this transform is a good candidate for future on-board satellite image compression systems. First, we analyze the ability of the bandelets to exploit directional correlations between wavelet coefficients. This study leads to an improved post-processing with a better decorrelation of adjacent wavelet coefficients in the vertical or in the horizontal direction taking into account the wavelet subband orientations. To perform even better decorrelation, bases are also build by principal component analysis (PCA). This results in an improved compression performance without increasing the computational complexity.

This paper deals with the optimization of a new technique of image compression. After the wavelet transform of an image, blocks of coefficients are further linearly decomposed using a basis selected in a dictionary. This dictionary is known by both the encoder and the decoder. This approach is a generalization of the bandelet transform. This paper investigates the problem of the best basis selection. On each block of wavelet coefficients, this selection is made by minimization of a Lagrangian rate-distortion criterion. Theoretical expressions of the optimal Lagrangian multiplier can be computed based on asymptotic hypotheses. A nearly exhaustive search of the optimal Lagrangian multiplier is done for the compression of high resolution satellite images. This numerical study validates the asymptotic theoretical expressions but as well provides a refined expression of the Lagrangian multiplier. At last, the compression results obtained using those different expressions are compared to the optimal compression results obtained with the exhaustive search.

Recent developments of hardware capabilities on mobile devices have made Location Based Services(LBS) very popular. This triggered an increasing need of rich visual content in mobile guide applications.This paper presents two original approaches for using representative visualizations: artistic views which represent an arbitrary deformation made by an artist and perspective views(3D-like) obtained from 3D models. Both approaches are based on learning GPS-to-image relations. We show an efficient use of the thin-plate spline for registering GPS coordinates with images. We also show the implementation of our guiding system on two mobile platforms.

This paper studies the performance of the ZC contexts in JPEG2000. A quality criterion based on the mutual information is introduced. According to this criterion, there exist ZC contexts which are not effective in any bit-plane. They can thus be merged with others without significant loss of the compression efficiency. Since the contextual conditional probabilities strongly depend on the bit-planes, an adaptation of the contexts to the bit-planes is also proposed. First, an optimal context quantizer is used on each bit-planes, then a simple greedy entropy minimization procedure allows to combine the ZC contexts in JPEG2000. The results are compared with a global combination of JPEG2000 ZC contexts. Tests have been performed on a real satellite image data base. With only four ZC contexts, the PSNR is less than 0.01 dB lower than the PSNR with the nine ZC contexts.

Abstract We present a real‐time system that outputs a multi‐layer augmented video from a normal video stream. The input sequence is captured in a known environment: a panoramic image (the reference background) is provided before shooting. We use background tracking to register each input frame with this reference, and background subtraction to segment the foreground objects. Our background tracking method is a coarse‐to‐fine integration of three state‐of‐the‐art algorithms that we make resistant to occlusions. In particular, we introduce an algebraic technique to properly adapt the Jurie and Dhome (JD) tracker (the most robust of them). We report experimental results on real and synthetic data which validate our approach. We generate our augmented video sequences by compositing layers: a natural or synthetic background and several natural or synthetic foregrounds. Copyright © 2006 John Wiley & Sons, Ltd.

Interaction with hypermedia documents is a required feature for new sophisticated yet flexible multimedia applications. This paper presents an innovative adaptive technique to stream hypervideo that takes into account user behaviour. The objective is to optimize hypervideo prefetching in order to reduce the latency caused by the network. This technique is based on a model provided by a Markov Decision Process approach. The problem is solved using two methods: classical stochastic dynamic programming algorithms and reinforcement learning. Experimental results under stochastic network conditions are very promising.

A moving object in front of a stereovision system provides range views of its surface. Assuming that we can reconstruct a sequence of partial 3D descriptions of the whole surface, a rigid displacement of the object between successive views can be estimated. This is known as 3D surface registration problem. The closest relevant works are presented and consist in iterative matching and then registration. Unfortunately, they ignore all the particular features and origins of the 3D maps. Hence this paper attacks the registration of stereo based 3D maps. This knowledge leads to use widely the backprojection of the 3D data onto the sensor image planes to model and solve the problem. In contrast to previous approaches, a stochastic scheme iterates to refine a rough estimate and to impose the consistency of the solution both in 3D and image spaces.