Accurate sensor calibration is crucial for autonomous systems, yet its uncertainty quantification remains under-explored. We present the first approach to integrate un-certainty awareness into online extrinsic calibration, com-bining Monte Carlo Dropout with Conformal Prediction to generate prediction intervals with a guaranteed level of coverage. Our method proposes a framework to en-hance existing calibration models with uncertainty quan-tification, compatible with various network architectures. Validated on KITTI (RGB Camera-LiDAR) and DSEC (Event Camera-LiDAR) datasets, we demonstrate effective-ness across different visual sensor types, measuring performance with adapted metrics to evaluate the efficiency and reliability of the intervals. By providing calibration pa-rameters with quantifiable confidence measures, we offer insights into the reliability of calibration estimates, which can greatly improve the robustness of sensor fusion in dy-namic environments and usefully serve the Computer Vision community.

Event cameras capture changes of illumination in the observed scene rather than accumulating light to create images. Thus, they allow for applications under high-speed motion and complex lighting conditions, where traditional frame-based sensors show their limits with blur and over- or under-exposed pixels. Thanks to these unique properties, they represent nowadays an highly attractive sensor for ITS-related applications. Event-based optical flow (EBOF) has been studied following the rise in popularity of these neuromorphic cameras. The recent arrival of high-definition neuromorphic sensors, however, challenges the existing approaches, because of the increased resolution of the events pixel array and a much higher throughput. As an answer to these points, we propose an optimized framework for computing optical flow in real-time with both low- and high-resolution event cameras. We formulate a novel dense representation for the sparse events flow, in the form of the “inverse exponential distance surface”. It serves as an interim frame, designed for the use of proven, state-of-the-art frame-based optical flow computation methods. We evaluate our approach on both low- and high-resolution driving sequences, and show that it often achieves better results than the current state of the art, while also reaching higher frame rates, 250Hz at 346$\times260$pixels and 77Hz at 1280$\times720$pixels.

Lane detection plays a crucial role in any autonomous driving system. Currently commercialized vehicles offer lane keep assist and lane departure warning via integrated smart cameras, deployed for road markings detection. These sensors alone, however, do not generally ensure adequate performance for higher autonomy levels. In this paper, a multi-sensor tracking approach for generic lane boundaries is proposed. This solution is based on well-established filtering techniques and supports a flexible clothoid spline representation. It relies on fine-tuned measurement models, tailored on collected data from both off-the-shelf and prototype smart sensors. The implementation takes into account real-time constraints and ADAS ECUs scarcity of resources. The result is finally validated against lane-level ground truth and experimental data acquisitions.

In traditional LIDAR processing pipelines, a point-cloud is split into clusters, or objects, which are classified afterwards. This supposes that all the objects obtained by clustering belong to one of the classes that the classifier can recognize, which is hard to guarantee in practice. We thus propose an evidential end-to-end deep neural network to classify LIDAR objects. The system is capable of classifying ambiguous and incoherent objects as unknown, while only having been trained on vehicles and vulnerable road users. This is achieved thanks to an evidential reformulation of generalized logistic regression classifiers, and an online filtering strategy based on statistical assumptions. The training and testing were realized on LIDAR objects which were labelled in a semi-automatic fashion, and collected in different situations thanks to an autonomous driving and perception platform.

Low-rank matrix completion aims to recover matrices with missing entries and has attracted considerable attention from machine learning researchers. Most of the existing methods, such as weighted nuclear-norm-minimization-based methods and Qatar Riyal (QR)-decomposition-based methods, cannot provide both convergence accuracy and convergence speed. To investigate a fast and accurate completion method, an iterative QR-decomposition-based method is proposed for computing an approximate singular value decomposition. This method can compute the largest r(r > 0) singular values of a matrix by iterative QR decomposition. Then, under the framework of matrix trifactorization, a method for computing an approximate SVD based on QR decomposition (CSVDQR)-based L2,1-norm minimization method (LNM-QR) is proposed for fast matrix completion. Theoretical analysis shows that this QR-decomposition-based method can obtain the same optimal solution as a nuclear norm minimization method, i.e., the L2,1-norm of a submatrix can converge to its nuclear norm. Consequently, an LNM-QR-based iteratively reweighted L2,1-norm minimization method (IRLNM-QR) is proposed to improve the accuracy of LNM-QR. Theoretical analysis shows that IRLNM-QR is as accurate as an iteratively reweighted nuclear norm minimization method, which is much more accurate than the traditional QR-decomposition-based matrix completion methods. Experimental results obtained on both synthetic and real-world visual data sets show that our methods are much faster and more accurate than the state-of-the-art methods.

In inductive transfer learning, fine-tuning pre-trained convolutional networks substantially outperforms training from scratch. When using fine-tuning, the underlying assumption is that the pre-trained model extracts generic features, which are at least partially relevant for solving the target task, but would be difficult to extract from the limited amount of data available on the target task. However, besides the initialization with the pre-trained model and the early stopping, there is no mechanism in fine-tuning for retaining the features learned on the source task. In this paper, we investigate several regularization schemes that explicitly promote the similarity of the final solution with the initial model. We show the benefit of having an explicit inductive bias towards the initial model, and we eventually recommend a simple $L^2$ penalty with the pre-trained model being a reference as the baseline of penalty for transfer learning tasks.

In many learning tasks, including semantic image segmentation, performance can be effectively improved through the fine-tuning of a pre-trained convolutional network, instead of training from scratch. With fine-tuning, the underlying assumption is that the pre-trained model extracts generic features, which are at least partially relevant for solving a segmentation task, but that would be difficult to extract from the smaller amount of data that is available for training in urban driving scenes segmentation. However, besides the initialization with the pre-trained model and the early stopping, there is no mechanism in classical fine-tuning approaches for keeping the generic features. Even worse, the standard weight decay drives the parameters towards the origin and affects the learned features. In this paper, we show that a simple regularization that uses the pre-trained model as a reference consistently improves the performance when applied to semantic urban driving scene segmentation. Experiments are done on the Cityscapes dataset, with four different architectures of convolutional networks.

Pedestrian detection is a well-studied problem. Even though many datasets contain challenging case studies, the performances of new methods are often only reported on cases of reasonable difficulty. In particular, the issue of small scale pedestrian detection is seldom considered. In this paper, we focus on the detection of small scale pedestrians, i.e., those that are at far distance from the camera. We show that classical features used for pedestrian detection are not well suited for our case of study. Instead, we propose a convolutional neural network based method to learn the features with an end-to-end approach. Experiments on the Caltech Pedestrian Detection Benchmark showed that we outperformed existing methods by more than 10% in terms of log-average miss rate.

Head pose is an important indicator of a person's visual focus of attention (VFoA). A traditional way to recognize VFoA is to consider accurate head pose or gaze estimations. However, these estimations usually degrade drastically in middle or low resolution video data. In this paper, a joint estimation of head pose and VFoA is proposed to address this issue; both head pose and VFoA are iteratively refined until convergence. This approach is evaluated in a specific scenario involving children around a table playing together with toys. Datasets are acquired and annotated by psychologists in Peking university. The experimental results demonstrate the usefulness of the join estimation process to recognize visual focus of attention in middle resolution video sequences.

Precise localization is an essential issue for autonomous driving applications, where GPS-based systems are challenged to meet requirements such as lane-level accuracy. This paper introduces a new visual-based localization approach in dynamic traffic environments, focusing on and exploiting properties of structured roads like straight roads or intersections. Such environments show several line segments on lane markings, curbs, poles, building edges, etc., which demonstrate the road's longitude, latitude and vertical directions. Based on this observation, we define a Road Structural Feature (RSF) as sets of segments along three perpendicular axes together with feature points. At each video frame, the proper road structure (or multiple road structures in case of an intersection) is predicted based on the geometric information given by a 2D map. The RSF is then detected from line segments and points extracted from the image, and used to estimate the pose of the vehicle. Experiments are conducted using video streams collected on major roads in downtown Beijing, which are structured and with intense dynamic traffic. GPS/IMU data have been collected and synchronized with the video streams as a reference in validation. The results show good performance compared with that of a more traditional visual odometry method. Future work will be addressed on using visual approach to improve GPS localization accuracy.

This paper presents a novel segmentation approach for extracting faces from videos. Under an active learning framework, the segmentation is conducted automatically without human interactions. A small portion of pixels are first labeled as face or non-face. Given these labeled samples, a semi-supervised spline regression model is then applied to obtain the face region. Based on the segmentation result, new pixels are selected and labeled. These two steps perform iterately until convergence. The main novelty is that color and depth data are combined to provide the labeling information. Our approach is validated via comparisons with state-of-the-art methods on real videos captured from the commodity Kinect camera.

It has become a well-known technology that 3D measurement of a large environment could be achieved by using a number of 2D LIDARs on a mobile platform. In such a system, calibration is essential for making collaborative use of different LIDAR data, while existing methods usually require modifications to the environments, such as putting calibration targets, or rely on special facilities, which is labor intensive and put many restrictions to potential applications. This research aims at developing a calibration method for multiple 2D LIDAR sensing systems, which could be conducted in a general outdoor environment using the features of a nature scene. Special focus is cast on solving the noisy sensing in a complex environment and the occlusions caused by largely different sensor viewpoints. A multi-type geometric feature based calibration algorithm is proposed, which extracts the features such as points, lines, planes and quadrics from the 3D points of each LIDAR sensing. Transformation parameters from each sensor to the frame on a moving platform is estimated by matching the multi-type features. Experiments are conducted using the data sets of an intelligent vehicle platform (POSS-V) through a driving in the campus of Peking University. Results of calibrating two LIDAR sensors with largely different viewpoints are presented, and the accuracy and robustness concerning noisy feature extractions are examined intensively.

It becomes a well-known technology that a low-level map of complex environment containing 3D laser points can be generated using a robot with laser scanners. Given a cloud of 3D laser points of an urban scene, this paper proposes a method for locating the objects of interest, e.g. traffic signs or road lamps, by computing object-based saliency. Our major contributions are: 1) a method for extracting simple geometric features from laser data is developed, where both range images and 3D laser points are analyzed; 2) an object is modeled as a graph used to describe the composition of geometric features; 3) a graph matching based method is developed to locate the objects of interest on laser data. Experimental results on real laser data depicting urban scenes are presented; efficiency as well as limitations of the method are discussed.

This paper presents a system to automatically generate car sample dataset for visual-based car detector training. The dataset contains multi-view car samples labeled with the car's pose, so that a view-discriminative training and car detection is also available. There are mainly two parts in the system: laser-based car detection and tracking generates motion trajectories of on-road cars, and then visual samples are extracted by fusing the detection and tracking results with visual-based detection. A multi-modal sensor system is developed for the omni-directional data collection on a test-bed vehicle. By processing the data of experiment conducted on the freeway of Beijing, a large number of multi-view car samples with pose information were generated. The samples' quality is evaluated by applying it in a visual car detector's training and testing procedure.

Lane change is one of the most principle driving behaviors on structure roads. It frequently happens in daily driving. A key issue in lane change technique is trajectory planning, where a set of trajectories describing possible vehicle motions are generated by applying a parametric function, and by uniformly sampling the end states in configuration space; the trajectories are then examined to find an optimal one for execution. However, such a trajectory set has poor efficiency due to the large sample number. Many trajectories in this set seldom happen in real human driving behaviors. In this research, lane change trajectories are collected from real driving data of different drivers. Their statistics are analyzed, through which, a simplified trajectory set is generated. Experiment results show that the trajectory set has much less number of samples but can still guarantee to cover usual lane change behaviors of human being.

This research aims at generating an omnidirectional perception at the host vehicle's surroundings, extracting accurate and continuous motion trajectories of the nearby vehicles using low cost laser scanners. A system of detecting and tracking on-road vehicles using multiple laser scanners is developed, where focuses are cast on solving data association of simultaneous measurements from multiple sensors at different viewpoints, and state estimation in case of partial observations in dense dynamic situations. Experimental results in freeways in Beijing are presented, system efficiency is demonstrated, where motion trajectories describing driving behaviors such as overtaking, lane changing and other interactions between driving objects are captured. In addition, the accuracy in vehicle detection and tracking is examined using a reference vehicle with a ground truth GPS.

Keypoint matching is a standard tool to solve the correspondence problem in vision applications. However, in 3-D face tracking, this approach is often deficient because the human face complexities, together with its rich viewpoint, nonrigid expression, and lighting variations in typical applications, can cause many variations impossible to handle by existing keypoint detectors and descriptors. In this paper, we propose a new approach to tailor keypoint matching to track the 3-D pose of the user head in a video stream. The core idea is to learn keypoints that are explicitly invariant to these challenging transformations. First, we select keypoints that are stable under randomly drawn small viewpoints, nonrigid deformations, and illumination changes. Then, we treat keypoint descriptor learning at different large angles as an incremental scheme to learn discriminative descriptors. At matching time, to reduce the ratio of outlier correspondences, we use second-order color information to prune keypoints unlikely to lie on the face. Moreover, we integrate optical flow correspondences in an adaptive way to remove motion jitter efficiently. Extensive experiments show that the proposed approach can lead to fast, robust, and accurate 3-D head tracking results even under very challenging scenarios.

Ferns was recently proposed to address the problem of fast keypoint matching [1]. In estimating homography, it often yields many similar matching scores under large viewpoint change or when the object is highly repetitive. However, we observe that keypoints having similar scores are usually far away from each other. Based on this fact, we propose a Softferns approach, in which Ferns and homography can alternatively refine each other. Extensive experiments prove that Softferns is a very general improvement over Ferns.

Descriptor matching is recently used in optical flow to handle large motion displacements. While doing so is a success, motion jitter often arises from lack of temporal consistency. However, when accounting for the consistency, it may violate our expectation of handling arbitrary motion shifts. In this paper, we propose a new approach to remove this controversy. At its core is a hierarchical fusion of descriptor matching and optical flow to determine the true local motion displacements. Applications to estimating homography and 3D head pose verify that the approach can well adapt to either large or small motion displacements.

The dynamical recognition of facial gestures and expressions in image sequences is an important and challenging problem. Most of the existing methods adopt the following paradigm. First, facial feature movements are retrieved from the images, then the facial expression is recognized based on these retrieved movements. In contrast to this mainstream approach, this paper introduces a new approach allowing the simultaneous retrieval of facial feature movements and expression using a particle filter adopting multi-class and multi-person dynamics. The proposed fast scheme is either as robust as, or more robust than existing ones in a number of respects. We provide evaluations of performance to show the feasibility and robustness of the proposed approach.

This paper presents an approach that incorporates Canonical Correlation Analysis (CCA) for monocular 3D face pose and facial animation estimation. The CCA is used to find the dependency between texture residuals and 3D face pose and facial gesture. The texture residuals are obtained from observed raw brightness shape-free 2D image patches that we build by means of a parameterized 3D geometric face model. This method is used to correctly estimate the pose of the face and the model's animation parameters controlling the lip, eyebrow and eye movements (encoded in 15 parameters). Extensive experiments on tracking faces in long real video sequences show the effectiveness of the proposed method and the value of using CCA in the tracking context.

We present in this paper a solution for 3D face and facial feature tracking using canonical correlation analysis and a 3D geometric model. This model is controlled with 17 parameters (6 for the 3D pose, and 11 for facial animation), and is used to crop out reference 2D shape free texture maps from the incoming input frames. Model parameters are updated via image registration in the texture map space. For registration, we use CCA to learn and exploit the dependency between texture residuals and model parameter corrections. We compare tracking results using two kinds of texture maps: one local (image patches around selected vertices of the 3D model), and one global (the whole image patch under the 3D model). Experiments evaluating the effectiveness on the approaches are reported.

A quick and reliable model-based head motion tracking scheme is presented. In this approach, rigid head motion and non-rigid facial animation are robustly tracked simultaneously by statistically analyzing the local regions of several representative facial features. The features are defined and operated based on a mesh model that helps maintain a global constraint on the local features and avoid the time-consuming appearance computation. A statistical model is computed from a moderate training set that is obtained by synthesizing different poses from a given standard initial image. During tracking, feature-based local distributions are obtained directly from the video frames and the troublesome feature detection or model rendering process is avoided. The observed distribution is compared with the pre-computed statistical model and the tracking is achieved by minimizing an error function based on the maximum likelihood principle. Experimental results show that this tracking strategy is robust to a wide range of head motion, facial animation and partial occlusion. The tracking can be conducted in nearly real-time and is easy to recover from failures.

In this work, we address the problem of tracking faces and facial actions in a single video sequence. The main contributions of the paper are as follows. First, we develop a particle filter based framework for tracking the global 3-D motion of a face using a statistical facial appearance model. Second, we propose a framework for tracking the 3-D face pose as well as the local motion of inner features of the face due for instance to spontaneous facial actions, using an adaptive appearance model. We allow the statistics of the facial appearance as well as the dynamics to be adaptively updated during tracking. Third, we propose a variant of the second framework based on a heuristic search. Tracking real video sequences demonstrated the effectiveness of the developed methods. Accurate tracking was obtained even in the presence of perturbing factors including significant head pose and facial expression variations, occlusions, and illumination changes

In this paper, we address the problem of tracking a near-frontal view face and its facial features in a video sequence. For this purpose, a particle filtering scheme is proposed, where the distribution of observations is derived from an active appearance model. The dynamics are adaptive in the sense that they are guided by a deterministic search, and the explored area of the state space is adjusted to the quality of the prediction. The number of particles is adapted accordingly, which enables a substantial gain in computing time. In order to account for occlusions, the observation model uses a robust distance measure. Experiments on real video show encouraging results.

In this paper, a new video watermarking scheme is proposed. We adapt two watermarking algorithms, originally proposed for still image watermarking, to embed data in a set of statistically independent sources, extracted from a video sequence. We show the interest of applying such an approach to video watermarking. The proposed scheme offers good robustness against MPEG compression attack, as well as an important capacity level. We consider, in this paper, data hiding in digital TV channels where data are compressed using MPEG-2. The main contribution of our study is to compare two different watermarking approaches applied to error prediction frames transformed by ICA.

The recognition of facial gestures and expressions in image sequences is an important and challenging problem. Most of the existing methods adopt the following paradigm. First, facial actions/features are retrieved from the images, and then facial expressions are recognized based on the retrieved temporal parameters. Unlike this main strewn, this paper introduces a new approach allowing the simultaneous recovery of facial actions and expression using a particle filter adopting, multiclass dynamics that are conditioned on the expression. For each frame in the video sequence, our approach is split in two consecutive stages. In the first stage, the 3D head pose is recovered using a deterministic registration technique based on online appearance models. In the second stage, the facial actions as well as the facial expression are simultaneously recovered using the stochastic framework with, mixed states. The proposed fast scheme is either as robust as existing ones or more robust with respect to many regards. Experimental results show the feasibility and robustness of the proposed approach

In this paper, we address the recognition of facial expressions in continuous videos. We introduce a fast and texture-independent approach that exploits facial action parameters estimated by an appearance-based 3D tracker. We represent the learned facial actions associated with different facial expressions by time series. We adopt a dynamic time warping technique for recognizing the facial expressions using similarity measures between the learned time series and the tracked ones. The developed approach can be used online and can deal with non-frontal face images since it is view-independent. Preliminary experiments demonstrated the effectiveness of the developed method.

Traditionally, the structure from motion (SFM) problem has been solved using feature correspondences. This approach requires reliably detected and tracked features between images taken from widespread locations. In this paper, we present a new paradigm to the SFM problem for planar scenes. The novelty of the paradigm lies in the fact that instead of image feature correspondences, only image derivatives are used. We use robust statistics in order to get robust solutions in presence of outliers. Experiments on both synthetic and real image sequences demonstrated the effectiveness of the developed methods.

Independent component analysis (ICA) techniques have been recently used in different watermarking schemes. However, performance of an ICA video watermarking scheme in comparison with those using classical domains, such as the discrete Fourier transform (DCT) domain, is still not clear. In this paper, we attempt to fill this gap. Therefore, we propose a video watermarking scheme, using an informed trellis, applied in two transformed domains obtained by using respectively the DCT transform and an ICA coding technique. We show that, for both domains, the scheme offers a good robustness against MPEG-2 compression, as well as an important capacity level. We consider in this paper data hiding in digital TV channels where data are compressed using MPEG-2.

This paper addresses the issue of face representations for f acial expression analysis and synthesis. In this context, a global appea rance model is used and two bilinear factorization models are subsequently proposed to separate expression and identity factors from the global appearance parameters. A feature extraction technique inspired from the above representations is then proposed which consists in automatically computing the optimal identity and expression components that best adapt to an unknown target face. The proposed representation can be seen as an alternative to the costly AAM gradient matrix construction and iterative search and is exploited i n the context of facial expression control. Results are compared with the ones obtained using bilinear factorization and linear regression in the space o f AAM parameters.

Firstly, this paper answered one interesting question: what is orthogonal in ICA? We discussed the non-orthogonality of ICA transformation and claimed that the independent components (ICs) by the FastICA algorithm are orthogonal. Based on this orthogonality, an orthogonal ICA representation (OICA) of images was proposed. Experiments on ORL database, Yale database and CMU database with OICA were performed. More work is needed to discover any advantage of OICA.

Facial expression interpretation, recognition and analysis is a key issue in visual communication and man to machine interaction. In this paper, we present a technique for extracting appearance parameters from a natural image or video sequence, which allow reproduction of natural looking expressive synthetic faces. This technique was used to perform face synthesis and tracking in video sequences as well as facial expression recognition and control.

This paper presents two semi-blind watermarking techniques, based on sets of significant wavelet coefficients, in the three lowest detail subbands of a cover-image. The first method modifies the magnitude of triplets of significant coefficients, according to a sequence of information bits. We show that multiple marks can be embedded in an image. The second method considers rectangular blocks of coefficients, and a uniform quantization-based watermarking scheme. The extraction algorithm requires one to know the position of the sets of watermarked coefficients in order to retrieve the hidden sequence of information bits. Experimental results show the robustness of the two methods, to different kinds of classical signal processing attacks.

We present the partitioning of grey-scale images using rectangular partitions, and its use for image prediction by motion compensation. Generally, a good prediction of a video frame can be made if an accurate motion estimation is computed, most often by using block matching algorithms (BMA). We propose to use an adaptive rectangular partition in order to improve the motion estimation and compensation steps. Rectangular partitioning presents the advantage of being well adapted to the image texture and to generate variable size blocks. Taking into account these properties, we test different adaptations of the classical BMA to the rectangular partitioning, in order to improve the motion compensation performance. We consider the simple translational BMA and extend this algorithm by using spatial transformations in order to cope with rotations and scalings. The last algorithm is furthermore optimized by taking into account the shape of the rectangles in the partition. Simulations show that the last method provides higher PSNR than the two other.

Our paper presents a watermarking scheme based on an insertion of similarities. In the first part different watermarking techniques are presented and classed. In the second part our scheme is described in its spatial and frequential implantations. Finally the different results and perspectives of the work are outlined.

Presents a new scheme for fractal image compression based on adaptive Delaunay triangulation. Such a partition is computed on an initial set of points obtained with a split and merge algorithm in a grey level dependent way. The triangulation is thus fully flexible and returns a limited number of blocks allowing good compression ratios. Moreover, a second original approach is the integration of a classification step based on a modified version of the Lloyd algorithm (vector quantization) in order to reduce the encoding complexity. The vector quantization algorithm is implemented on pixel histograms directly generated from the triangulation. The aim is to reduce the number of comparisons between the two sets of blocks involved in fractal image compression by keeping only the best representative triangles in the domain blocks set. Quality coding results are achieved at rates between 0.25-0.5 b/pixel depending on the nature of the original image and on the number of triangles retained.

This paper presents a new partitioning scheme for fractal image coding, based on triangles and quadrilaterals. The aim is to have the advantage of the triangles over the square and rectangular blocks, in terms of adaptivity to the image (reduction of the block effect), and to reduce the blocks number by grouping neighboring triangles into quadrilaterals. Quadrilaterals permit a reduction of the number of local contractive affine transformations composing the fractal transform, and thus to increase the compression ratio, while preserving the visual quality of the decoded image.

The principle of attractor image coding presented in this paper is based on the theory of IFS (iterated function systems). The algorithm exploits piece-wise similarities between block of different sizes. To improve the algorithms based on regular and square blocks, we propose an adaptive Delaunay triangulation of the image support. The originality of the method is to map the triangles on specific parts of the image in order to have lots of similarities between them.