Abstract Perception systems in autonomous vehicles need to accurately detect and classify objects within their surrounding environments. Numerous types of sensors are deployed on these vehicles, and the combination of such multimodal data streams can significantly boost performance. The authors introduce a novel sensor fusion framework using deep convolutional neural networks. The framework employs both camera and LiDAR sensors in a multimodal, multiview configuration. The authors leverage both data types by introducing two new innovative fusion mechanisms: element‐wise multiplication and multimodal factorised bilinear pooling. The methods improve the bird's eye view moderate average precision score by +4.97% and +8.35% on the KITTI dataset when compared to traditional fusion operators like element‐wise addition and feature map concatenation. An in‐depth analysis of key design choices impacting performance, such as data augmentation, multi‐task learning, and convolutional architecture design is offered. The study aims to pave the way for the development of more robust multimodal machine vision systems. The authors conclude the paper with qualitative results, discussing both successful and problematic cases, along with potential ways to mitigate the latter.

The classical color constancy algorithms concentrate only on the color of a grey surface to estimate the light color and to white balance the image. In this paper, we show that the quality of the whole process can be clearly improved by predicting and correcting the colors of a set of reference surfaces. The ground truth of the surface color under white light can be easily obtained with a set of images acquired by the considered camera under sun light. Thus, we design a deep network to predict the colors of the reference surfaces of a color checker as if it had been in the scene at acquisition time. We show that our solution improves the two steps of the color constancy process on 9 datasets and we claim that being able to synthetically insert a color chart in any image can help for many other tasks.

Abstract A two stage real-time hand gesture recognition system is presented. It combines a machine learning trained detection step with a colour processing contour shape validation step. The detection step is done with either Adaboost Cascades or Support Vector Machines using HOG features. The system achieves a low false positive rate and a sufficient true positive rate necessary for robust real-time performance. It performs well compared to MobileNets a state of the art Neural Network for mobile real-time applications.

Camera and Lidar processing have been revolutionized with the rapid development of deep learning model architectures. Automotive radar is one of the crucial elements of automated driver assistance and autonomous driving systems. Radar still relies on traditional signal processing techniques, unlike camera and Lidar based methods. We believe this is the missing link to achieve the most robust perception system. Identifying drivable space and occupied space is the first step in any autonomous decision making task. Occupancy grid map representation of the environment is often used for this purpose. In this paper, we propose PolarNet, a deep neural model to process radar information in polar domain for open space segmentation. We explore various input-output representations. Our experiments show that PolarNet is a effective way to process radar data that achieves state-of-the-art performance and processing speeds while maintaining a compact size.

Keeping the safe distance from the leading vehicle is crucial for transportation companies with a fleet of old cars. While modern Advanced Driver Assistant Systems (ADAS) might be able to estimate the distance from the front-leading vehicle, traditional ADAS do not usually offer this feature. An alternative solution is to monitor the distance using smartphones that are attached to a place such as a sun visor. The basic idea behind this approach is to detect the front-leading vehicle using the smartphone camera and estimate its distance from the car. Although SSD can achieve real-time performance on powerful GPUs, it remains challenging to run this model in real-time on mobile devices. In this paper, we propose a monocular distance estimator for forward-leading vehicles using a smartphone which is faster and more accurate than the state-of-the-art SSD detector. Specifically, we propose a layer-wise method to generate more efficient default boxes for the SSD and develop a lightweight method for estimating the distance accurately. Our experiments show that the proposed method reduces the number of default boxes by an average of 38.4% while it improves the detection rate and the processing speed compared to the original SSD. Moreover, our monocular distance estimator provides a proper safety buffer zone when the distance is greater than 20 meters. A sample video is available at https://youtu.be/-ptvfabBZWA.

In this paper, we investigate effective neural network layers for optical flow estimation and in particular for omni-directional optical flow. Optical flow has many applications in computer graphics, augmented reality and in 3D modeling. We create a simple dataset that enables us to efficiently assess the effectiveness of different neural network layers for optical flow. Based on this small-sized diagnostic dataset, FlowCLEVR, we conclude that a deformable convolution layer is highly effective in reducing motion and occlusion boundary blur. Based on these results, we are able to design modifications to various existing network architectures improving their performance. We demonstrate improved performance on FlowCLEVR, on standard datasets for optical flow in planar images and on a novel omni-directional optical flow dataset. We also extend our work to omni-directional stereo.

The difference between sample distributions of public data sets and specific scenes can be very significant. As a result, the deployment of generic human detectors in real-world scenes most often leads to sub-optimal detection performance. To avoid the labor-intensive task of manual annotations, we propose a semi-supervised approach for training deep convolutional networks on partially labeled data. To exploit a large amount of unlabeled target data, the knowledge learnt from public data sets is transferred to new model training by adapting an auxiliary detector to the target scene. We hypothesize that the components of the auxiliary detector capture essential human characteristics useful for constructing a scene-adapted detector. A selective ensemble algorithm is proposed to select a subset of the components relevant to the target scene for recombination. The resulting model is applied for collecting high-confidence samples from unlabeled target data. Furthermore, a deep convolutional network is trained by progressively labeling and selecting new training samples in a self-paced way. The detailed experimental evaluation verifies the effectiveness and superiority of the proposed approach in scene-specific human detection.

This paper addresses the problem of color reduction which aims at computing a compact representation of a color coordinate system. By capitalizing on studies that have suggested the existence of eleven focal colors, we conducted subjective experiments which exploited the categorical nature of human color perception. This paper describes a novel color reduction scheme based on human perception and graph transduction and proposes a look-up table that reduces the RGB color coordinate system into eleven salient colors. Objective results on standard datasets show improvements over previously proposed methods.

Although many visual attention models have been proposed, very few saliency models investigated the impact of audio information. To develop audio-visual attention models, researchers need to have a ground truth of eye movements recorded while exploring complex natural scenes in different audio conditions. They also need tools to compare eye movements and gaze patterns between these different audio conditions. This paper describes a toolbox that answer these needs by proposing a new eye-tracking dataset and its associated analysis ToolBox that contains common metrics to analysis eye movements. Our eye-tracking dataset contains the eye positions gathered during four eye-tracking experiments. A total of 176 observers were recorded while exploring 148 videos (mean duration = 22 s) split between different audio conditions (with or without sound) and visual categories (moving objects, landscapes and faces). Our ToolBox allows to visualize the temporal evolution of different metrics computed from the recorded eye positions. Both dataset and ToolBox are freely available to help design and assess visual saliency models for audiovisual dynamic stimuli.

The number of video surveillance cameras has increased by a large amount in recent years. There is therefore a need to process the captured videos such that human operators can quickly review the activities recorded by a camera over a long period of time. We propose in this paper an approach for producing video summaries, an abbreviated video preserving the important elements of interest. We introduce a dataset as well as three evaluation metrics for quantifying the performance of a video summary with respect to compression length, the amount of activity retained and the amount of activity that is packed into each frame of the summary.

Saliency models provide heatmaps highlighting the probability of each pixel to attract human gaze. To define image's important regions, features maps are extracted. The rarity, surprise or contrast are computed leading to conspicuity maps, showing important regions of each feature map. The final saliency map is obtained by merging these maps. The fusion process is usually a linear combination of the maps where the coefficients show their importance. We propose a novel generic fusion mechanism based on 1) using a rarity-based attention module and 2) using neural networks to achieve the fusion. The first layer of the NN merges the weighted feature maps into a saliency map. The second layer takes into account the spatial information. The approach is compared to 8 models using 4 different comparison metrics on open state-of-the-art databases.

Modeling human attention has been arousing a lot of interest due to its numerous applications. The process that allows us to focus on some more important stimuli is defined as the “attention”. Seam carving is an approach to resize images or video sequences while preserving the semantic content. To define what is important, gradient was first used but due to limitations of this approach, saliency models, which are able to predict whether regions in the images attract human attention, are now used. Most of them are optimized to conform a ground truth like eye tracking but there is no way to know the efficiency of the saliency models applied in a specific application like in seam carving. In this paper, we propose a criterion, based on the quantity of geometric deformation and the image's reduction, which evaluate the quality of a resizing by seam carving. This criterion is applied on evaluation of image (SCES) or video (SCED) resizing. We validate our criterion with subjective evaluation and used it to rank state of the art saliency models for seam carving. Evaluation of the image by SCES gives a Spearman correlation of −0.92196 and a Pearson correlation of −0.8812. For the video, the final SCED gives a Spearman correlation of −0.81351 and a Pearson correlation of −0.80581.

In this paper, we present a real-time approach for 3D object detection using a single, mobile and uncalibrated camera. We develop our algorithm using a feature-based method based on two novel naive Bayes classifiers for viewpoint and feature matching. Our algorithm exploits the specific structure of various binary descriptors in order to boost feature matching by conserving descriptor properties (e.g., rotational and scale invariance, robustness to illumination variations and real-time performance). Unlike state-of-the-art methods, our novel naive classifiers only require a database with a small memory footprint because we store efficiently encoded features. In addition, we also improve the indexing scheme to speed up the matching process. Because our database is built from powerful descriptors, only a few images need to be 'learned' and constructing a database for a new object is highly efficient.

This paper introduces a high efficient local spatiotemporal descriptor, called gradient boundary histograms (GBH). The proposed GBH descriptor is built on simple spatio-temporal gradients, which are fast to compute. We demonstrate that it can better represent local structure and motion than other gradient-based descriptors, and significantly outperforms them on large realistic datasets. A comprehensive evaluation shows that the recognition accuracy is preserved while the spatial resolution is greatly reduced, which yields both high efficiency and low memory usage.

Gender recognition and age classification are important applications of face analysis. The vast majority of the existing solutions focus on a single visual descriptor which often encodes only a certain characteristic of the image regions (e.g., shape, or texture, or color, etc.). In this paper, we propose a novel framework for gender and age classification, which facilitates the integration of multiple feature types and therefore allows for taking advantage of various sources of visual information. Furthermore, in the proposed method, only the regions that can best separate face images of different demographic classes (with respect to age and gender) contribute to the face representations, which in turn, improves the classification and recognition accuracies. Experiments performed on a challenging publicly available database validate the effectiveness of our proposed solution and show its superiority over the existing state-of-the-art methods.

Local spatio-temporal features and bag-of-features representations have become popular for action recognition. A recent trend is to use dense sampling for better performance. While many methods claimed to use dense feature sets, most of them are just denser than approaches based on sparse interest point detectors. In this paper, we explore sampling with high density on action recognition. We also investigate the impact of random sampling over dense grid for computational efficiency. We present a real-time action recognition system which integrates fast random sampling method with local spatio-temporal features extracted from a Local Part Model. A new method based on histogram intersection kernel is proposed to combine multiple channels of different descriptors. Our technique shows high accuracy on the simple KTH dataset, and achieves state-of-the-art on two very challenging real-world datasets, namely, 93% on KTH, 83.3% on UCF50 and 47.6% on HMDB51.

Comparison of image processing techniques is critically important in deciding which algorithm, method, or metric to use for enhanced image assessment. Image fusion is a popular choice for various image enhancement applications such as overlay of two image products, refinement of image resolutions for alignment, and image combination for feature extraction and target recognition. Since image fusion is used in many geospatial and night vision applications, it is important to understand these techniques and provide a comparative study of the methods. In this paper, we conduct a comparative study on 12 selected image fusion metrics over six multiresolution image fusion algorithms for two different fusion schemes and input images with distortion. The analysis can be applied to different image combination algorithms, image processing methods, and over a different choice of metrics that are of use to an image processing expert. The paper relates the results to an image quality measurement based on power spectrum and correlation analysis and serves as a summary of many contemporary techniques for objective assessment of image fusion algorithms.

In this paper, we propose an edge-based junction detector. In addition to detecting the locations of junctions, this operator specifies their orientations as well. In this respect, a junction is defined as a meeting point of two or more ridges in the gradient domain into which an image can be transformed through Gaussian derivative filters. To accelerate the detection process, two binary edge maps are produced; a thick-edge map is obtained by imposing a threshold on the gradient magnitude image, and another thin-edge map is obtained by calculating the local maxima. Circular masks are centered at putative junctions in the thick-edge map, and the so-called circumferential anchors or CA points are detected in the thin map. Radial lines are scanned to determine the presence of junctions. Comparisons are made with other well-known detectors. This paper proposes a new formula for measuring the detection accuracy. In addition, the so-called junction coordinate systems are introduced. Our operator has been successfully used to solve many problems such as wide-baseline matching, 3-D reconstruction, camera parameter enhancing, and indoor and obstacle localization.

This paper addresses the problem of camera pose recovery from spherical images. The 3D information is extracted from a set of panoramas sparsely distributed over a scene of interest. We present an algorithm to recover the position of omni-directional cameras in a scene using pair-wise essential matrices. First, all rotations with respect to the world frame are found using an incremental bundle adjustment procedure, thus achieving what we called cube alignment. The structure of the scene is then computed using a full bundle adjustment. During this step, the previously computed panorama orientations, used to feed the global optimization process, are further refined. Results are shown for indoor and outdoor panorama sets.

This paper describes a method for image rectification of a trinocular setup. The rectification method used is an extension of a recent approach based on the fundamental matrix to generate the correcting homographies in the case of a stereo pair. The extended method uses the fact that the triplet of images can be treated as two pairs and that homographies are simply projections of the different images planes onto new planes. Rectification thus becomes a matter of deciding which plane is the common one and what transformation or homography is to be applied to each image

Measuring image similarity is important in many applications. Different algorithms propose to compare images using pixel-based mean square error methods others use structure-based image quality index. We present, here, a new feature-based approach that utilizes image phase congruency measurement to quantify the assessment of the similarities or differences between two images

The purpose of this study is to investigate a variational method for joint segmentation and parametric estimation of image motion by basis function representation of motion and level set evolution. The functional contains three terms. One term is of classic regularization to bias the solution toward a segmentation with smooth boundaries. A second term biases the solution toward a segmentation with boundaries which coincide with motion discontinuities, following a description of motion discontinuities by a function of the image spatio-temporal variations. The third term refers to region information and measures conformity of the parametric representation of the motion of each region of segmentation to the image spatio-temporal variations. The components of motion in each region of segmentation are represented as functions in a space generated by a set of basis functions. The coefficients of the motion components considered combinations of the basis functions are the parameters of representation. The necessary conditions for a minimum of the functional, which are derived taking into consideration the dependence of the motion parameters on segmentation, lead to an algorithm which condenses to concurrent curve evolution, implemented via level sets, and estimation of the parameters by least squares within each region of segmentation. The algorithm and its implementation are verified on synthetic and real images using a basis of cosine transforms.

A method for finding correspondences between widely separated views is presented. The proposed solution consists in estimating the local perspective distortion between the neighborhoods of junctions. To this end, a formulation is proposed, based on a constrained minimization involving an estimated fundamental matrix. An application is also proposed to fundamental matrix recovery using crude camera pose estimates. 1

This paper presents a new junction detection operator that defines junctions as points where linear ridges in the gradient domain intersect. The radial lines that compose the junction are therefore identified by searching, in a circular neighborhood, for directional maxima of the intensity gradient. The proposed algorithm operates on two binary edge maps, the computational complexity of the detection process is then considerably reduced.

We propose a method to augment live video based on the tracking of natural features, and the online estimation of the trinocular geometry. Previous without-marker approaches require the computation of camera pose to render virtual objects. The strength of our proposed method is that it doesn 7 require tracking of camera pose, and exploits the usual advantages of marker-based approaches for a fast implementation. A 3-view AR system is used to demonstrate our approach. It consists of an uncalibrated camera that moves freely inside the scene of interest, and of three reference frames taken at the time of system initialization. As the camera is moving, image features taken from an initial triplet set are tracked throughout the video sequence. And the trifocal tensor associated with each frame is estimated online. With this tensor, the square pattern that was visible in the reference frames is transferred to the video. This invisible pattern is then used by the ARToolkit to embed virtual objects.

A method for quickly and reliably selecting and matching points from three views of a scene is presented. The points that are selected are based on the concept of epipolar gradients, and consist in stable and relevant image features. Then, the selected points are matched using edge transfer, resulting in a measure of consistency for point triplets and the edges on which they lie, with the camera system's trinocular geometry. This matching scheme is invariant to image deformations due to changes in viewpoint.

This paper presents a new operator for corner detection. This operator uses a variant of the morphological closing operator, which we have called asymmetrical closing. It consists of the successive application of different morphological transformations using different structuring elements. Each of these structuring elements used to probe the image under study is tuned to affect corners of different orientation and brightness. We found that this kind of approach, based on brightness comparisons, leads to better quality results than others and is achieved at a lower computational cost.

In this paper we propose a parallel network which gradually computes the three-dimensional (3-D) structure of a moving scene from its image sequence, using an incremental scheme based upon a constraint called the maximal rigidity principle. At each instant an internal model (i.e., current estimate) of the 3-D structure is updated, based upon the observations accumulated until then. The updating process favors rigid transformations but tolerates a limited amount of deviation from rigidity. This deviation eventually leads the internal model to converge towards the actual 3-D structure of the scene, An application of this network to the problem of structure from two views is also presented. The main advantage of this architecture is its ability to accurately estimate the 3-D structure of a scene, at a low computational cost. Testing has been successfully performed on synthetic data as well as real image sequences.