Bird's Eye View (BEV) maps have gained popularity in the autonomous driving field due to their ability to offer an information-rich and easily interpretable representation of the environment, playing a crucial role in various tasks such as motion planning, perception, and sensor fusion. In this paper, we propose a novel framework that integrates deep learning with grid filtering and feature engineering to achieve multi-object tracking and panoptic segmentation. The framework also validates object detections and removes spurious instances, resulting in a more robust environmental representation that includes tracked objects and semantic segmentation in both perspective and BEV domains. Furthermore, we introduce an original hybrid data association function that combines deep learning-based and handcrafted features to enhance object tracking accuracy. To support this approach, we present a dedicated dataset designed for training the deep learning model used in data association comprising four object categories: cars, pedestrians, riders, and animals. The proposed solution, implemented in C++, is integrated with contemporary deep learning frameworks and has been evaluated on international benchmarks such as Cityscapes and KITTI.

Monocular depth estimation (MDE) is one of the most difficult tasks in computer vision. The problem becomes even more complicated in case of aerial images due to the high complexity and lack of structure present in such scenarios. State-of-the-art MDE methods can cope with such environments only by using high amounts of resources. In this work, we try to provide a more resource-aware alternative by dynamically inserting scene priors through semantic features into the network. To this end, we initially propose a novel dynamic semantic-aware module, that combines features extracted from the RGB image with a dynamically weighted semantic map. The weights are gradually modified, according to the iteration number inside the training process. By pursuing this methodology, we initially predict the depth for larger objects inside the scene. As the training resumes, smaller objects are further emphasized, thus increasing object boundary confidences. We prove that this dynamic adaptation mechanism leads to a better and faster convergence. The adaptation technique is applied for various learning frameworks, by introducing three novel semantic-aware losses for depth-based regression, classification, and ordinal regression. We show the results on a large set of synthetic and real-life aerial images, captured in various scenarios and thus we prove the effectiveness of our approach in terms of convergence time, depth accuracy, and prediction time.

In this paper we present a novel multi-object tracking and segmentation approach that works on thermal images and is able to track objects at bounding-box and instance mask levels. Furthermore, we present a novel object validation module, which is necessary because only specific classes of objects are tracked and classifiers and detectors can be subjected to errors such as misclassifications, false detections, erroneous instance masks and missed detections. One of the key difficulties in multi-target tracking is the unknown correspondences between measurements and targets also known as the data association problem. To address this issue the proposed object tracker uses a feature engineering data association approach that exploits multiple features which include structure, appearance, size, context, and motion in the region given by the instance mask. Moreover, an original strategy has been designed for dealing with motion uncertainty, based on optical flow and multiple motion models to better predict the future position of objects in the scene. The proposed method runs in real-time and has been evaluated on an international thermal tracking benchmark showing competitive results.

Depth-aware video panoptic segmentation tackles the inverse projection problem of restoring panoptic 3D point clouds from video sequences, where the 3D points are augmented with semantic classes and temporally consistent instance identifiers. We propose a novel solution with a multi-task network that performs monocular depth estimation and video panoptic segmentation. Since acquiring ground truth labels for both depth and image segmentation has a relatively large cost, we leverage the power of unlabeled video sequences with self-supervised monocular depth estimation and semi-supervised learning from pseudo-labels for video panoptic segmentation. To further improve the depth prediction, we introduce panoptic-guided depth losses and a novel panoptic masking scheme for moving objects to avoid corrupting the training signal. Extensive experiments on the Cityscapes-DVPS and SemKITTI-DVPS datasets demonstrate that our model with the proposed improvements achieves competitive results and fast inference speed.

We present a novel self-distillation based self-supervised monocular depth estimation (SD-SSMDE) learning framework. In the first step, our network is trained in a self-supervised regime on high-resolution images with the photometric loss. The network is further used to generate pseudo depth labels for all the images in the training set. To improve the performance of our estimates, in the second step, we re-train the network with the scale invariant logarithmic loss supervised by pseudo labels. We resolve scale ambiguity and inter-frame scale consistency by introducing an automatically computed scale in our depth labels. To filter out noisy depth values, we devise a filtering scheme based on the 3D consistency between consecutive views. Extensive experiments demonstrate that each proposed component and the self-supervised learning framework improve the quality of the depth estimation over the baseline and achieve state-of-the-art results on the KITTI and Cityscapes datasets.

We propose a novel solution for the task of video panoptic segmentation, that simultaneously predicts pixel-level semantic and instance segmentation and generates clip-level instance tracks. Our network, named VPS-Transformer, with a hybrid architecture based on the state-of-the-art panoptic segmentation network Panoptic-DeepLab, combines a convolutional architecture for single-frame panoptic segmentation and a novel video module based on an instantiation of the pure Transformer block. The Transformer, equipped with attention mechanisms, models spatio-temporal relations between backbone output features of current and past frames for more accurate and consistent panoptic estimates. As the pure Transformer block introduces large computation overhead when processing high resolution images, we propose a few design changes for a more efficient compute. We study how to aggregate information more effectively over the space-time volume and we compare several variants of the Transformer block with different attention schemes. Extensive experiments on the Cityscapes-VPS dataset demonstrate that our best model improves the temporal consistency and video panoptic quality by a margin of 2.2%, with little extra computation.

Environment perception by computing the depth is a key task for unmanned aerial vehicle (UAV) type systems. Due to the limited load they can carry, most drones are equipped with a single camera. This prevents general-purpose depth perception methods based either on light detection and ranging (LiDAR) or stereo reconstruction to be effectively used on such platforms. Due to the success of convolutional neural networks (CNNs), monocular depth estimation (MDE) methods have become more and more trustworthy, so their usage on drones is convenient. However, very few such methods have been proposed in the literature, mainly due to the few existing constraints and high diversity that unstructured aerial environments pose. To bridge this gap, we propose a novel approach for MDE, capable to work on aerial images. The method initially proposes an original CNN, particularly adapted to such scenarios. This is done by finding an optimal feature extractor, introducing a new scene understanding module, a new loss and a novel softmax transformation layer that facilitate a better convergence. Furthermore, since both short- and long-range accuracy is required for a robust UAV perception, we introduce a learning-based correction method that redistributes the depth points across the entire depth interval. The proposed CNN gives accurate results, while the additional refinement further improves the accuracy with only a few additional computational resources (around 1–2 ms). We initially show the capabilities of our method on synthetic images captured in unstructured aerial scenarios. Then, we prove that our method can work in real-life situations, computing depth from a single image (at multiple pitch angles) captured by a drone flying in a series of field and forest-like environments. In all these situations, the depth is densely estimated with increased accuracy and reliability.

The European UP-Drive project addresses transportation-related challenges by providing key contributions that enable fully automated vehicle navigation and parking in complex urban areas, which results in a safer, inclusive, affordable and environmentally friendly transportation system. For this purpose, the project consortium developed a prototype electrical vehicle equipped with cameras and LiDARs sensors that is capable to autonomously drive around the city and find available parking spots. In UP-Drive, we created an accurate, robust and redundant multi-modal environment perception system that provides 360° coverage around the vehicle. This paper summarizes the work of the project related to the surround view semantic perception using fisheye and narrow field-of-view semantic virtual cameras. Deep learning-based semantic, instance and panoptic segmentation networks, which satisfy requirements in accuracy and efficiency have been developed and integrated into the final prototype. The UP-Drive automated vehicle has been successfully demonstrated in urban areas after extensive experiments and numerous field tests.

Depth estimation plays a crucial role in robotic applications that require environment perception. With the introduction of convolutional neural networks, monocular depth estimation (MDE) methods have become viable alternatives to LiDAR and stereo reconstruction-based solutions. Such methods require less equipment, fewer resources and do not need additional sensor alignment requirements. However, due to the ill-posed formulation of MDE, such algorithms can only rely on learning mechanisms, which makes them less reliable and less robust. In this work we propose a novel method to cope with the lack of geometric constraints inherent to monocular depth computation. Towards this goal, we initially mathematically transform the feature vectors from the last layer inside a MDE CNN such that a 3D stereo-like cost volume is generated. We then adapt the semi-global stereo optimization to the aforementioned volume, global consistency of the map being ensured. Furthermore, we enhance the results by adding a sub-pixel stereo post-processing be means of interpolation functions, a larger range of depth values being obtained. Our method can be applied to any classification-based MDE, experiments showing an increase in accuracy with an additional time cost of only 8 ms on a regular GPU, making the technique usable for real-time applications.

Environment perception for driving applications requires very accurate sensors especially when dealing with depth measurements. LiDAR is the most trustworthy sensor in this domain, but it suffers from disadvantages in terms of number of scene points and their temporal alignment. These issues are especially relevant when dealing with long-range measurements, where each 3D point is crucial. As an alternative, in this work we focus on camera-based depth perception for objects at large distance by using stereo reconstruction and monocular depth estimation. Towards improving the capabilities of camera-based, we initially introduce a taxonomy to categorize all types of camera-based depth perception methods with respect to their long-range capabilities. We then present a correction method that works for both stereo and monocular depth perception algorithms that output a depth in discrete setting (most suitable for realtime applications). We show that our method improves the precision for such algorithms for objects at large distances without affecting the near-range accuracy. The method requires only several additional operations, preserving the real-time capabilities of the underlying algorithms.

In this paper we propose a fast fully convolutional neural network for panoptic segmentation that can provide an accurate semantic and instance-level representation of the environment in the 2D space. We tackle panoptic segmentation as a dense classification problem and generate masks for stuff classes as well as for each instance of things classes. Our network employs a shared backbone and Feature Pyramid Network for multi-scale feature extraction which we extend with dual-decoders that learn background and foreground specific masks. Guided by object proposals, the panoptic head assembles location-sensitive prototype masks using a learned weighting scheme. Our solution runs in real-time, in 82 ms on high resolution images, making it suitable for robotic applications and automated driving. Extensive experiments on the Cityscapes dataset demonstrate that our panoptic segmentation network is robust and accurate, with 57.3% PQ and 76.9% mIoU.

In this paper, we propose a novel semantic segmentation-based stereo reconstruction method that can keep up with the accuracy of the state-of-the art approaches while running in real time. The solution follows the classic stereo pipeline, each step in the stereo workflow being enhanced by additional information from semantic segmentation. Therefore, we introduce several improvements to computation, aggregation, and optimization by adapting existing techniques to integrate additional surface information given by each semantic class. For the cost computation and optimization steps, we propose new genetic algorithms that can incrementally adjust the parameters for better solutions. Furthermore, we propose a new post-processing edge-aware filtering technique relying on an improved convolutional neural network (CNN) architecture for disparity refinement. We obtain the competitive results at 30 frames/s, including segmentation.

Environment perception for automated vehicles is achieved by fusing the outputs of different sensors such as cameras, LIDARs and RADARs. Images provide a semantic understanding of the environment at object level using instance segmentation, but also at background level using semantic segmentation. We propose a fully convolutional residual network based on Mask R-CNN to achieve both semantic and instance level recognition. We aim at developing an efficient network that could run in real-time for automated driving applications without compromising accuracy. Moreover, we compare and experiment with two different backbone architectures, a classification type of network and a faster segmentation type of network based on dilated convolutions. Experiments demonstrate top results on the publicly available Cityscapes dataset.

We propose a new real-time depth upsampling method based on convolutional neural networks (CNNs) that uses the local context provided by semantic information. Two solutions based on convolutional networks are introduced, modeled according to the level of sparsity given by the depth sensor. While first CNN upsamples data from a partial-dense input, the second one uses dilated convolutions as means to cope with sparse inputs from cost-effective depth sensors. Experiments over data extracted from Kitti dataset highlight the performance of our methods while running in real-time (11 ms for the first case and 17 ms for the second) on a regular GPU.

In this paper we propose a novel boosting-based sliding window solution for object detection which can keep up with the precision of the state-of-the art deep learning approaches, while being 10 to 100 times faster. The solution takes advantage of multisensorial perception and exploits information from color, motion and depth. We introduce multimodal multiresolution filtering of signal intensity, gradient magnitude and orientation channels, in order to capture structure at multiple scales and orientations. To achieve scale invariant classification features, we analyze the effect of scale change on features for different filter types and propose a correction scheme. To improve recognition we incorporate 2D and 3D context by generating spatial, geometric and symmetrical channels. Finally, we evaluate the proposed solution on multiple benchmarks for the detection of pedestrians, cars and bicyclists. We achieve competitive results at over 25 frames per second.

In this paper we introduce a novel multimodal boosting based solution for semantic segmentation of traffic scenarios. Local structure and context are captured from both monocular color and depth modalities in the form of image channels. We define multiple channel types at three different levels: low, intermediate and high order channels. The low order channels are computed using a multimodal multiresolution filtering scheme and capture structure and color information from lower receptive fields. For the intermediate order channels, we employ deep convolutional channels that are able to capture more complex structures, having a larger receptive field. The high order channels are scale invariant channels that consist of spatial, geometric and semantic channels. These channels are enhanced by additional pyramidal context channels, capturing context at multiple levels. The semantic segmentation is achieved by a boosting based classification scheme over superpixels using multi-range channel features and pyramidal context features. A pre-segmentation is used to generate semantic channels as input for more powerful final segmentation. The final segmentation is refined using a superpixel-level dense conditional random field. The proposed solution is evaluated on the Cityscapes segmentation benchmark and achieves competitive results at low computational costs. It is the first boosting based solution that is able to keep up with the performance of deep learning based approaches.

Lately stereo matching has become a key aspect in autonomous driving, providing highly accurate solutions at relatively low cost. Top approaches on state of the art benchmarks rely on learning mechanisms such as convolutional neural networks (ConvNets) to boost matching accuracy. We propose a new real-time stereo reconstruction method that uses a ConvNet for semantically segmenting the driving scene. In a “divide and conquer” approach this segmentation enables us to split the large heterogeneous traffic scene into smaller regions with similar features. We use the segmentation results to enhance Census Transform with an optimal census mask and the SGM energy optimization step with an optimal Pi penalty for each predicted class. Additionally, we improve the sub-pixel accuracy of the stereo matching by finding optimal interpolation functions for each particular segment class. In both cases we propose new stochastic optimization methods based on genetic algorithms that can incrementally adjust the parameters for better solutions. Tests performed on Kitti and real traffic scenarios show that our method outperforms the accuracy of previous solutions.

Scene labeling enables very sophisticated and powerful applications for autonomous driving. Training classifiers for this task would not be possible without the existence of large datasets of pixelwise labeled images. Manually annotating a large number of images is an expensive and time consuming process. In this paper, we propose a new semi-automatic annotation tool for scene labeling tailored for autonomous driving. This tool significantly reduces the effort of the annotator and also the time spent to annotate the data, while at the same time it offers the necessary features to produce precise pixel-level semantic labeling. The main contribution of our work represents the development of a complex annotation framework able to generate automatic annotations for 20 classes, which the user can control and modify accordingly. Automatic annotations are obtained in two separate ways. First, we employ a pixelwise fully-connected Conditional Random Field (CRF). Second, we perform grouping of similar neighboring superpixels based on 2D appearance and 3D information using a boosted classifier. Polygons represent the manual correction mechanism for the automatic annotations.

Pedestrian detection and semantic segmentation are high potential tasks for many real-time applications. However most of the top performing approaches provide state of art results at high computational costs. In this work we propose a fast solution for achieving state of art results for both pedestrian detection and semantic segmentation. As baseline for pedestrian detection we use sliding windows over cost efficient multiresolution filtered LUV+HOG channels. We use the same channels for classifying pixels into eight semantic classes. Using short range and long range multiresolution channel features we achieve more robust segmentation results compared to traditional codebook based approaches at much lower computational costs. The resulting segmentations are used as additional semantic channels in order to achieve a more powerful pedestrian detector. To also achieve fast pedestrian detection we employ a multiscale detection scheme based on a single flexible pedestrian model and a single image scale. The proposed solution provides competitive results on both pedestrian detection and semantic segmentation benchmarks at 8 FPS on CPU and at 15 FPS on GPU, being the fastest top performing approach.

The hepatocellular carcinoma (HCC) is a frequent malignant liver tumour and one of the main causes of death. Detecting the HCC evolution phases is an important issue, aiming the early diagnosis of this tumour and patient monitoring with maximum accuracy. Our objective is to discover the evolution stages of HCC, through unsupervised classification techniques, using advanced texture analysis methods. In this work, we assessed the role that the Haralick features derived from the Complex Extended Textural Microstructure Co-occurrence Matrices (CETMCM) have in the unsupervised detection of the HCC evolution stages. A textural model for these phases was also generated. The obtained results were validated by supervised classifiers, well known for their performance, such as the Multilayer Perceptron (MLP), Support Vector Machines (SVM), respectively decision trees and they were also compared with the previously obtained results in this domain. The final classification accuracy was about 90%.

In this paper we describe a fast solution for the semantic segmentation of traffic scenarios. We propose a multiresolution filtering scheme over LUV + HOG image channels using high pass and low pass filtered channels at multiple scales. To add spatial context, we extend the filtered channels with horizontal and vertical position channels. We introduce multi-range classification features that capture local structure and context for achieving fast semantic segmentation of traffic scenarios. Binary boosting based pixel classifiers are trained for each semantic class. Finally, we use these classifiers to provide the unary potential term in a dense Conditional Random Field. We evaluate the proposed solution on the CamVid traffic scene segmentation benchmark and achieve state of art results at 25 FPS, being the fastest top performing approach.

The accuracy of stereo matching algorithms is one of the key aspects in autonomous driving nowadays. In case of large distances, sub-pixel accurate solutions are required, especially for algorithms in discrete settings. It has been previously shown that a strong correlation between the matching algorithm and the sub-pixel interpolation method exists, and there are ways to determine it. Unfortunately all methodologies presented so far are laborious and time-consuming. We present here a novel sub-pixel disparity correction technique based on applying histogram matching through the use of generated Look-up Tables (LUTs). Our method is flexible, fast and produces more accurate results than previous solutions in the discrete domain. Although we show the improvements over the Semi-Global matching algorithm, it can be adapted to other matching algorithms that preserve constant misalignment for any kind of 3D scenarios. The proposed method was tested on multiple systems and datasets (Synthetic images, Traffic scenes, Middlebury images, KITTI images) and we show that we can find LUTs that outperform the accuracy of previous solutions on all these sets. The histogram matching procedure lacks in complexity and results indicate a strict dependency of a particular LUT to the underlying stereo matching and the stereo vision system, but not on the image composition.

In this work we present an obstacle detection and tracking solution applied to Automated Guided Vehicles (AGVs) in industrial environments. The proposed method relies on information provided by an omnidirectional stereo vision system enabling 360 degree perception around the AGV. The stereo data is transformed into a classified digital elevation map (DEM). Based on this intermediate representation we are able to generate a set of obstacle hypotheses, each represented by a 3D cuboid and a free-form polygonal model. The cuboidal model is used for the classification of each hypothesis as “Pedestrian”, “AGV”, “Large Obstacle” or “Small Obstacle”, while the free-form polylines are used for object motion estimation relying on an Iterative Closest Point (ICP) method. The obtained measurements are subjected to a Kalman filter based tracking approach, in which the data association takes into account also the classification results.

In this paper we propose an approach for obstacle localization and recognition using omnidirectional stereovision applied to autonomous fork-lifts in industrial environments. We use omnidirectional stereovision with two fisheye cameras for the 3D perception of the surrounding environment. Using the reconstructed 3D points, a Digital Elevation Map (DEM) is constructed consisting of a 2.5D grid of elevation cells. Each cell is then classified as ground or obstacle. Further, we use the classified DEM to generate obstacle hypotheses. To ensure a higher detection rate we also propose a fast sliding window based approach relying on the monocular fisheye intensity image. The detections from both approaches are merged and are subjected to a tracking mechanism. Finally each obstacle is classified using boosting over Visual Codebook type features. The classification is refined using the classification history available from tracking. The presented approaches are integrated into a 3D visual perception system for AGVs and are of real time performance.

This paper presents SmartCoDrive, an Android application which performs driving assistance functions: 3D lane detection and tracking, forward obstacle detection, obstacle tracking. With this mobile application we wish to increase the adoption rate of driving assistance systems and to provide a viable and cheap solution for every driver, that will be able to use his own tablet or smartphone as a personal driving assistant. The mobile application is deployed on a tablet equipped with dual back-facing cameras. The visual information from the two cameras, along with the data received from the Controller Area Network bus of the vehicle enable a thorough understanding of the 3D environment. First, we develop the sparse 3D reconstruction algorithm. Then, using monocular vision we perform lane markings detection. Obstacle detection is done by combining the superpixel segmentation with 3D information and the tracking algorithm is based on the Kalman Filter. Since the processing capabilities of the mobile platforms are limited, different optimizations are carried out in order to obtain a real-time implementation. The Android application may be used in urban traffic that is characterized by low-speed and short-medium distances to obstacles.

Few published articles on curvilinear structures exist compared with works on detecting lines or corners with high accuracy. In medical ultrasound imaging, the structures that need to be detected appear as a collection of microstructures correlated along a path. In this paper, we investigated techniques that extract meaningful low-level information for curvilinear structures, using techniques based on structure tensor. We proposed a novel structure tensor enhancement inspired by bilateral filtering. We compared the proposed approach with five state-of-the-art curvilinear structure detectors. We tested the algorithms against simulated images with known ground truth and real images from three different domains (medical ultrasound, scanning electron microscope, and astronomy). For the real images, we employed experts to delineate the ground truth for each domain. Techniques borrowed from machine learning robustly assessed the performance of the methods (area under curve and cross validation). As a practical application, we used the proposed method to label a set of 5000 ultrasound images. We conclude that the proposed tensor-based approach outperforms the state-of-the-art methods in providing magnitude and orientation information for curvilinear structures. The evaluation methodology ensures that the employed feature-detection method will yield reproducible performance on new, unseen images. We published all the implemented methods as open-source software.

The images captured in fog conditions have degraded contrast, which makes current image processing applications sensitive and error prone. We propose in this paper an efficient single-image enhancement algorithm suitable for daytime fog conditions and, based on an original mathematical model, for computing the atmospheric veil, taking into account the variation in fog density to the distance. This model is inspired by the functions that appear in partition of unity in the differential geometry field. When observing images captured in fog conditions, usually the fog has a very low density in front of the camera and this density has a nonlinear increase with the distance, such that objects are no longer visible at greater distances. By using our mathematical model, we are able to obtain superior reconstructions of the original fog-free image when compared with traditional methods. Another advantage of our method is the ability to adapt the model in accordance to the density of the fog. A quantitative and qualitative evaluation is performed on both synthetic and real camera images. This evaluation proves that our mathematical model is more suitable for contrast restoration in both homogeneous and heterogeneous fog conditions. Our algorithm is able to perform contrast restoration in real time for both color and grayscale images.

In this paper we present a stereovision-based approach for tracking multiple objects in crowded environments where, typically, the road lane markings are not visible and the surrounding infrastructure is not known. The proposed technique relies on measurement data provided by an intermediate occupancy grid derived from processing a stereovision-based elevation map and on free-form object delimiters extracted from this grid. Unlike other existing methods that track rigid objects using also rigid representations, we present a particle filter-based solution for tracking visual appearance-based free-form obstacle representations. At each step, the particle state is described by two components, i.e., the object's dynamic parameters and its estimated geometry. In order to solve the high-dimensionality state-space problem, a Rao-Blackwellized particle filter is used. By accurately modeling the object geometry using the polygonal lines instead of a 3-D box and, at the same time, separating the position and speed tracking from the geometry tracking at the estimator level, the proposed solution combines the efficiency of the rigid model with the benefits of a flexible object model.

Most pedestrian detection approaches that achieve high accuracy and precision rate and that can be used for realtime applications are based on histograms of gradient orientations. Usually multiscale detection is attained by resizing the image several times and by recomputing the image features or using multiple classifiers for different scales. In this paper we present a pedestrian detection approach that uses the same classifier for all pedestrian scales based on image features computed for a single scale. We go beyond the low level pixel-wise gradient orientation bins and use higher level visual words organized into Word Channels. Boosting is used to learn classification features from the integral Word Channels. The proposed approach is evaluated on multiple datasets and achieves outstanding results on the INRIA and Caltech-USA benchmarks. By using a GPU implementation we achieve a classification rate of over 10 million bounding boxes per second and a 16 FPS rate for multiscale detection in a 640×480 image.

Most of computation time when dealing with a pedestrian detector is spent in the feature computation and then in the multi-scale classification. This second step consists of applying scanning windows at multiple scales. Depending on the number of scales and on the image dimension, this step is slow because a large number of windows is generated. An efficient pruning algorithm able to remove most of the scan windows brings a significant contribution on the overall execution time. We propose a scan window pruning algorithm based on a combination of several filters: (1) remove windows based on the relation between the dimension and the position in the scene; (2) remove uniform regions from the image such as the sky or the road; (3) remove regions with high density of horizontal edges such as vegetation parts; (4) keep local maxima windows having a high density of connected vertical edges. The combination of these four filters eliminates more than 90% of the scanning windows in a given image and maintains the windows that are overlapped on regions with a high probability of representing a pedestrian. We have integrated our method in a generic framework for pedestrian detection [1] and we have studied two aspects: (1)how the performance of the algorithms varies with respect to the filters proposed by our pruning strategy and (2) what is the speed gain when the quality loss is negligible. The proposed filters have a negligible loss in performance and even improve it in some cases while the execution time is improved.

Multi-class segmentation assigns a class label to each pixel in an image. It represents a significant task for the semantic understanding of images and has received plentiful attention over the last years. The current state of art is dominated by conditional random field based approaches, defined over pixels or image segments. However, high accuracy segmentation comes at a high computational cost. The best performing methods can barely run at few frames per second and are far from real-time applications. Our goal is to bridge the gap between current state of the art segmentation approaches and real-time applications. In this paper we propose an efficient approach for individual pixel classification. Multiple local descriptors are computed densely and then quantized using visual codebooks. Joint boosting is used to classify each pixel based on the quantized local descriptors. We show that using careful design choices and GPU optimization we can achieve sate of the art segmentation results at over 50 FPS. We also propose a Conditional Random Field (CRF) model defined over superpixels that uses the proposed pixel classifier for the estimation of unary potentials. The CRF based multi-class segmentation can run at over 30 FPS. The proposed approach is validated on the MSRC21 and CamVid multi-class segmentation benchmarks, the former one consisting of urban traffic sequences.

Digital elevation maps are simple yet powerful representations of complex 3-D environments. These maps can be built and updated using various sensors and sensorial data processing algorithms. This paper describes a novel approach for modeling the dynamic 3-D driving environment, the particle-based dynamic elevation map, each cell in this map having, in addition to height, a probability distribution of speed in order to correctly describe moving obstacles. The dynamic elevation map is represented by a population of particles, each particle having a position, a height, and a speed. Particles move from one cell to another based on their speed vectors, and they are created, multiplied, or destroyed using an importance resampling mechanism. The importance resampling mechanism is driven by the measurement data provided by a stereovision sensor. The proposed model is highly descriptive for the driving environment, as it can easily provide an estimation of the height, speed, and occupancy of each cell in the grid. The system was proven robust and accurate in real driving scenarios, by comparison with ground truth data.

Modeling dynamic environments is an essential research topic in any driving assistance system. The complexity of the surrounding world, the measurement uncertainties or the unpredictable behavior of the traffic participants are the main factors that influence the detection and tracking process. In this paper we present a vision-based method for modeling and tracking unstructured dynamic environments. The proposed solution relies on raw information provided by a classified grid computed from a digital elevation map and employs two separate representation levels: a local dynamic persistence grid (DyPerGrid) that is generated as an intermediate representation level and a map of delimiters as a higher level obstacle description. A fast tracking solution is proposed by using the two models. The result is a geometrically consistent and accurate representation of the dynamic environment.

The zero-mean normalized cross-correlation is shown to improve the accuracy of optical flow, but its analytical form is quite complicated for the variational framework. This paper addresses this issue and presents a new direct approach to this matching measure. Our approach uses the correlation transform to define very discriminative descriptors that are precomputed and that have to be matched in the target frame. It is equivalent to the computation of the optical flow for the correlation transforms of the images. The smoothness energy is non-local and uses a robust penalty in order to preserve motion discontinuities. The model is associated with a fast and parallelizable minimization procedure based on the projected-proximal point algorithm. The experiments confirm the strength of this model and implicitly demonstrate the correctness of our solution. The results demonstrate that the involved data term is very robust with respect to changes in illumination, especially where large illumination exists.

This paper presents a new automatic image annotation algorithm. First, we introduce a new similarity measure between images: compactness. This uses low level visual descriptors for determining the similarity between two images. Compactness shows how close test image features lie to training image feature cluster centers. The measure provides the core for a k-nearest neighbor type image annotation method. Afterward, a formalism for defining different transfer techniques is devised and several label transfer techniques are provided. The method as whole is evaluated on four image annotation benchmarks. The results on these sets validate the accuracy of the approach, which outperforms many state-of-the-art annotation methods. The method presented here requires a simple training process, efficiently combines different feature types and performs better than complex learning algorithms, even in this incipient form. The main contributions of this paper are the usage of compactness as a similarity measure that enables efficient low level feature comparison and an annotation algorithm based on label transfer.

This paper proposes a method for achieving improved ego-vehicle global localization with respect to an approaching intersection, which is based on the alignment of visual landmarks perceived by the on-board visual system, with the information from a proposed extended digital map (EDM). The visual system relies on a stereovision system that provides a detailed 3-D description of the environment, including road landmark information (lateral lane delimiters, painted traffic signs, curbs, and stop lines) and dynamic environment information (other vehicles). An EDM is proposed, which enriches the standard map information with a detailed description of the intersection required for current lane identification, landmark alignment, and ego-vehicle accurate global localization. A novel approach for lane-delimiter classification, which is necessary for the lane identification, is also presented. An original solution for identifying the current lane, combining visual and map information with the help of a Bayesian network (BN), is proposed. Extensive experiments have been performed, and the results are evaluated with a Global Navigation Satellite System of high accuracy (2 cm). The achieved global localization accuracy is of submeter level, depending on the performance of the stereovision system.

Reliable vehicle ego motion estimation based on visual information is an important research goal because it has applications like accurate long term localization by fusion with other sensors, temporal fusion between frames, moving obstacles detection and tracking, path planning etc. This paper evaluates and significantly improves some steps of existing visual odometry methods. The main contribution is related to accuracy improvements in case of illumination changes by using the rank transform. Additionally we propose a new consistency check, based on image deformations, for subsets of features considered during the RANSAC iterations of the algorithm. Performance of GPU execution and results in various traffic scenarios are presented in order to show the advantages and the robustness of the method.

This paper addresses the problem of finding the host vehicle's lateral position on a multi-lane road, using information obtained by processing video sequences. A very important cue for lane identification is the class of the boundaries of the current lane. This paper presents a reliable solution for lane boundary type identification, based on frequency analysis of the gray level profile of these boundaries, assuming that the current lane is already detected. The lane boundary information is combined with the obstacle information, through a Bayesian Network which will output, frame by frame, the probability of the vehicle to be positioned on each lane of the road. The probability result will be propagated throughout the sequence by a Particle Filter.

The environment representation is one of the main challenges of autonomous navigation. In the case of complex driving environments such as crowded city traffic scenarios, achieving satisfactory results becomes even more difficult. In this paper we propose a real-time solution for two main issues of advanced driver assistance systems: unstructured environment representation and extraction of dynamic properties of traffic participants. For the real-time environment representation we propose a solution to extract object delimiters from the traffic scenes and represent them as polygonal models. In order to track dynamic entities, an intermediate evidence map named “Stereo Temporal Difference Map” is proposed. This difference map is computed by comparing the occupancy of a cell between two consecutive frames. Based on the Stereo Temporal Difference Map information, difference fronts are extracted and are subjected to a particle based filtering mechanism. Finally, the provided dynamic features are associated to the extracted polygonal models. The result is a more compact representation of the dynamic environment.

Traditionally, subpixel interpolation in stereo-vision systems was designed for the block-matching algorithm. During the evaluation of different interpolation strategies, a strong correlation was observed between the type of the stereo algorithm and the subpixel accuracy of the different solutions. Subpixel interpolation should be adapted to each stereo algorithm to achieve maximum accuracy. In consequence, it is more important to propose methodologies for interpolation function generation than specific function shapes. We propose two such methodologies based on data generated by the stereo algorithms. The first proposal uses a histogram to model the environment and applies histogram equalization to an existing solution adapting it to the data. The second proposal employs synthetic images of a known environment and applies function fitting to the resulted data. The resulting function matches the algorithm and the data as best as possible. An extensive evaluation set is used to validate the findings. Both real and synthetic test cases were employed in different scenarios. The test results are consistent and show significant improvements compared with traditional solutions.

The increase in computational power of consumer graphic cards has successfully motivated adaptation of stereo algorithms to this kind of hardware. In order to solve the stereo correspondence problem efficiently, the images need to be rectified and lens distortions need to be removed. This paper presents an efficient two step solution for rectifying and correcting lens distortions in images captured using a pair of stereo cameras. The first step consists of a one-time, off-line calculation of a look-up table, based on the calibration parameters, for each of the two cameras. The second step computes the final pixel intensities based on the pre-calculated mappings stored in the look-up table. The GPU implementation proposed makes use of the inherent parallelism in a cost-effective manner, making the method suitable for rectifying high resolution images in real-time. Results are compared against an optimized CPU-based implementation, written in assembly language using MMX instructions, for reference. The complete stereo reconstruction system was implemented and evaluated on a current generation GPU and offers a running time of 11ms for images with resolution 512×383.

In this paper, we describe our approach for intersection safety developed in the scope of the European project INTERSAFE-2. A complete solution for the safety problem including the tasks of perception and risk assessment using on-board lidar and stereo-vision sensors will be presented and interesting results are shown.

Robust and versatile lane tracking solutions can be developed using particle filter based tracking, stereo data and grayscale image processing. This paper presents multiple improvements for our previously published solutions. The lane model is extended by parameters such as width variation, vertical offset, and widths of lane markings. The increased flexibility in lane modeling is matched by a better use of measurement data, and the particle weighting process will benefit from two additional cues. The extensions in lane model and in particle weighting provide a more powerful description of the lane, but will also increase the stability and accuracy for the estimation of all lane parameters.

Visual odometry has been an important research activity in the last two years and it has lead to numerous papers being published. Few surveys and comparative studies between approaches exist at the moment. This paper makes a comparative study between two visual odometry methods from accuracy, speed, sensitiveness to noise, and degree of parallelization points of view. The comparison is performed strictly from the perspective of minimizing the cost function, since this is one of the most critical steps in motion estimation from visual data. We also proposed a method based on Kalman filtering to achieve better accuracy in the presence of illumination changes based on correlating the measurement model noise with the intensity variations in time.

In this paper we present a real-time algorithm that detects curbs using a cubic spline model. A Digital Elevation Map (DEM) is used to represent the dense stereovision data. Curb measurements (cells) are detected on the current frame DEM. In order to compensate the small number of curb measurements for each frame we perform temporal integration. The result is a rich set of curb measurements that provides a good support for the least square cubic spline fitting. Thus, the curb cubic spline approximation is more stable and available on a much larger area, around the ego car. This compensates the limited field of view of typical stereo sensors. The detected curbs enrich the description of the ego car's surrounding 3D environment and can be used for driving assistance applications.

This paper proposes a method for achieving accurate ego-vehicle global localization with respect to an approaching intersection; the method is based on the data alignment of the information from two input systems: a Sensorial Perception system, on-board of the ego-vehicle, and an a priori digital map. For this purpose an Extended Digital Map is proposed that contains the detailed information about the intersection infrastructure: detailed landmarks accurately measured and positioned on the map. The data alignment mechanism is thus based on superimposing the sensorial detected landmarks with the corresponding, correctly positioned map landmarks stored in the new Extended Digital Map. The data Alignment Algorithm requires as input, beside the information from the two input systems, the ego-vehicle driving lane. This information is inferred by using a probabilistic approach in the form of a Bayesian Network; the uncertain and noisy character of the sensorial data require such a probabilistic approach in the quest of the ego-lane.

Modeling and tracking the driving environment is a complex problem due to the heterogeneous nature of the real world. In many situations, modeling the obstacles and the driving surfaces can be achieved by the use of geometrical objects, and tracking becomes the problem of estimating the parameters of these objects. In the more complex cases, the scene can be modeled and tracked as an occupancy grid. This paper presents a novel occupancy grid tracking solution based on particles for tracking the dynamic driving environment. The particles will have a dual nature-they will denote hypotheses, as in the particle filtering algorithms, but they will also be the building blocks of our modeled world. The particles have position and speed, and they can migrate in the grid from cell to cell, depending on their motion model and motion parameters, but they will be also created and destroyed using a weighting-resampling mechanism that is specific to particle filtering algorithms. The tracking algorithm will be centered on particles, instead of cells. An obstacle grid derived from processing a stereovision-generated elevation map is used as measurement information, and the measurement model takes into account the uncertainties of the stereo reconstruction. The resulting system is a flexible real-time tracking solution for dynamic unstructured driving environments.

This paper presents an occupancy grid tracking solution based on particles. The particles will have a dual nature - they will denote hypotheses, as in the particle filtering algorithms, but they will also be the building blocks of our modeled world. The particles have position and speed, and they can migrate in the grid from cell to cell depending on their motion model and motion parameters, but they will also be created and destroyed using a weighting-resampling mechanism specific to particle filter algorithms. An obstacle grid derived from processing a stereovision-generated elevation map is used as measurement information, and the measurement model takes into account the uncertainties of the stereo reconstruction. The resulted system is a flexible, real-time tracking solution for dynamic unstructured driving environments.

In this work we focus on creating a real-time dense stereo reconstruction system with accurate sub-pixel estimation. We selected the Semi-Global Matching method as the basis of our system due to its high quality and possible real-time implementations. In our solution we use the Census transform as the matching metric because our results show that it can reduce the matching errors for traffic images compared to classical solutions. We also propose several modifications to the original Semi-Global algorithm to improve the sub-pixel accuracy and the execution time. One of these proposals is the reduction in the number of optimization directions without affecting the results. The second modification is a correction of the energy function to reduce the spread of depth values. Besides these improvements, the paper also introduces a new aggregation method used to reduce the spread of sub-pixel values. Finally we propose a new method to generate sub-pixel interpolation functions based on real-world data. The result of these enhancements is a significant improvement in sub-pixel accuracy. The system was implemented and evaluated on a current generation GPU with a running time of 19 ms for image having the resolution 512 × 383.

In this study we examine the history and growth of ICTD since the 1990s. We underline the trends defining this research field and examine the progress in research areas that have come to dominate discussion in ICTD through a thorough literature review of the last decade of ICTD work. In order to answer questions pertaining to the rigor, impact and significance of ICTD, and to compare the expectations and perceived achievements with respect to different development goals, we interview 50 expert ICTD researchers and practitioners. We analyze these results to understand stakeholders' opinions on the past performance of ICTD, both as an academic field and as an area of development practice, and identify defining ideas on the potential directions for the future of ICTD. This study is work in progress and we have continuing research in this area, the subset presented here is rigorous and ready for wider discussion.

Accurate and robust lane results are of great significance in any driving-assistance system. To achieve robustness and accuracy in difficult scenarios, probabilistic estimation techniques are needed to compensate for the errors in the detection of lane-delimiting features. This paper presents a solution for lane estimation in difficult scenarios based on the particle-filtering framework. The solution employs a novel technique for pitch detection based on the fusion of two stereovision-based cues, a novel method for particle measurement and weighing using multiple lane-delimiting cues extracted by grayscale and stereo data processing, and a novel method for deciding upon the validity of the lane-estimation results. Initialization samples are used for uniform handling of the road discontinuities, eliminating the need for explicit track initialization. The resulting solution has proven to be a reliable and fast lane detector for difficult scenarios.

Pedestrians are the most vulnerable participants in urban traffic. The first step toward protecting pedestrians is to reliably detect them. We present a new approach for standing- and walking-pedestrian detection, in urban traffic conditions, using grayscale stereo cameras mounted on board a vehicle. Our system uses pattern matching and motion for pedestrian detection. Both 2-D image intensity information and 3-D dense stereo information are used for classification. The 3-D data are used for effective pedestrian hypothesis generation, scale and depth estimation, and 2-D model selection. The scaled models are matched against the selected hypothesis using high-performance matching, based on the Chamfer distance. Kalman filtering is used to track detected pedestrians. A subsequent validation, based on the motion field's variance and periodicity of tracked walking pedestrians, is used to eliminate false positives.

We consider the problem of efficientMAC design for long-distance WiFi-based mesh networks. In such networks it is common to find long propagation delays, the use of directional antennas, and the presence of inter-link interference. Prior work has shown that these characteristics make traditional CSMA-based MACs a poor choice for long-distance mesh networks and this finding has led to several recent research efforts exploring the use of TDMA-based approaches to media access. In this paper we first identify, and then address, several shortcomings of current TDMA-based proposals. First, because they use fixed-length transmission slots, current TDMA-based solutions do not adapt to dynamic variations in traffic load leading to inefficiencies in both throughput and delay. As we show in this paper, the throughput achieved by existing solutions falls far short of the optimal achievable network throughput. Finally, due to the scheduling constraints imposed by inter-link interference, current TDMA-based solutions only apply to bipartite network topologies.

This paper examines low-cost computing projects for education in developing regions, and presents some of the common entrepreneurial and technical problems faced by past and current initiatives. In particular, we look at various models of computer usage, and evaluate their appropriateness according to their effectiveness in education, their socio-cultural suitability, and economic feasibility. Based on detailed field studies and interviews conducted in rural Indian classrooms and economic analysis, we show that shared rather than single-user devices constitute a more realistic and sustainable approach for low-cost computing projects targeting children's education.

Despite the increasing number of WiFi-based Long Distance (WiLD) network deployments, there is a lack of understanding of how WiLD networks perform in practice. In this paper, we perform a systematic study to investigate the commonly cited sources of packet loss induced by the wireless channel and by the 802.11 MAC protocol. The channel induced losses that we study are external WiFi, non-WiFi and multipath interference. The protocol induced losses that we study are protocol timeouts and the breakdown of CSMA over WiLD links. Our results are based on measurements performed on two real-world WiLD deployments and a wireless channel emulator. The two deployments allow us to compare measurements across rural and urban settings. The channel emulator allows us to study each source of packet loss in isolation in a controlled environment. Based on our experiments we observe that the presence of external WiFi interference leads to significant amount of packet loss in WiLD links. In addition to identifying the sources of packet loss, we analyze the loss variability across time. We also explore the solution space and propose a range of MAC and network layer adaptation algorithms to mitigate the channel and protocol induced losses. The key lessons from this study were also used in the design of a TDMA based MAC protocol for high performance long distance multihop wireless networks [12].

This paper presents a high-accuracy online calibration method for the absolute extrinsic parameters of a stereovision system that is suited for far-distance, vision-based vehicle applications. The method uses as prior knowledge the intrinsic parameters and the relative extrinsic parameters (relative position and orientation) of the two cameras, which are calibrated using offline procedures. These parameters remain unchanged if the two cameras are mounted on a rigid frame (stereo rig). The absolute extrinsic parameters define the position and orientation of the stereo system relative to a world coordinate system. They must be calibrated every time after mounting the stereo rig in the vehicle and are subject to changes due to static load factors for the used car setup. The proposed method is able to perform online the estimation of the absolute extrinsic parameters by driving the car on a flat and straight road, parallel with the longitudinal lane markers. The edge points of the longitudinal lane markers are extracted after a 2-D image classification process and reconstructed by stereovision in the stereo-rig coordinate system. After filtering out the noisy 3-D points, the normal vectors of the world coordinate system axes are estimated in the stereo-rig coordinate system by 3-D data fitting. The output of the method is the height and the orientation of the stereo cameras that are relative to the world coordinate system.

Understanding technology adoption in emerging regions is challenging given the complex interrelations among socioeconomic factors that affect it directly and indirectly. The issue of impact assessment of technology adoption projects, especially the kind implemented in areas where prior technology has been very limited, is highly problematic and open to many methodological difficulties. Ethnographic evaluations have provided insight into the quality of interactions and into conceptions of technology and its adoption, whereas some quantitative analysis has been useful for high-level abstraction. In this paper, we examine the use of Bayesian networks as tools that can be used in revealing the structure of the relationships between demographic, social, and economic factors, and penetration for various technologies. Our hypothesis is that technology adoption cases in emerging regions display unique aggregated characteristics that make Bayesian network-based analysis a useful starting point in defining relationships between variables in project analysis. We compare the usability of Bayesian networks in analyzing two data sets: (1) a detailed survey focusing on 500 respondents across 14 favelas in Rio de Janeiro; and (2) a comprehensive survey of 998 users of the Akshaya tele-kiosk initiative in Kerala, India. Our illustrations show how Bayesian networks can be useful as statistical analysis tools that reveal new hypotheses, suggest unintended correlations in data, and confirm standing hypotheses

This paper presents a new vision sensor-based road-following method for unmanned vehicles. Usually, the performance obtained with such methods is limited by several factors like image quality according to camera types (charge-coupled device CCD/CMOS), mounted camera position, stereo-/single-vision sensors, structured/unstructured environment, and image noise (illumination and shadow). Existing road-following algorithms for unmanned vehicles perform well, given a certain number of satisfied constraints, so there is a lack of flexibility in their use in real-world situations. The currently proposed local-difference-probability-based method overcomes most of these constraints assuring flexibility in real-world environments

We propose a system architecture for real-time hardware speech recognition on low-cost, power-constrained devices. The system is intended to support real-time speech-based user interfaces as part of an effort to bring Information and Communication Technologies (ICTs) to underdeveloped regions of the world.Our system architecture exploits a shared infrastructure model. The computationally intensive task of speech model training and retraining is performed offline by shared servers, while the actual recognition of speech is conducted on low-cost hand-held devices using custom hardware.The recognizer is extremely flexible and can support multiple languages or dialects with speaker-independent recognition. Dynamic loading of speech models is used for changing language grammar and retraining, while reprogramming is used to support evolution of recognition algorithms. The focus on small sets of words (at one time) reduces the complexity, cost and power consumption. We design the speech decoder, the central component of the recognizer, and we validate it via a prototype FPGA implementation. We then use ASIC synthesis to estimate power and size for the design.Our evaluations demonstrate an order of magnitude improvement in power compared with optimized recognition software running on a low-power embedded general-purpose processor of the same technology and of similar capabilities. The synthesis also estimates the area of the design to be about 2.5mm2, showing potential for lower cost. In designing and testing our recognizer we use datasets in both English and Tamil languages.

The aim of this paper is to develop a method for low-cost and accurate classification of highways and rural ways image pixels for lane detection. The method uses three main components: adaptive/predefined image splitting, subimage level classification and class merging based on homogeneity checking conditions. In the first step, a preclassification in road and nonroad pixels is carried out, on the resized input image, using the decision tree method. As a result of this first step we obtain the road reference feature value, and the lane-markings positions in case of highways. For the rural ways image splitting we use a predefined division method, and for the highways we use an adaptive division method based on the detected lane-markings. The proposed classification is carried out on the subimages using the K-mean classifier on a composed gray and texture based feature vector. The gray feature vector is fixed in the preclassification phase, and the texture feature vector is only updated during the classification is performed. This way the convergence is much faster and the classification accuracy is better. The resulting road and nonroad classes of subimages are merged into a road and a nonroad class using a homogeneity criterion based on the road reference feature value. Next, a forward and backward method is used to detect borders of the road region. Finally, we use the Kalman filter and the Bresenhem line drawing to connect the border pixels.

There are several different approaches to defining reconfigurable systems. Reconfigurable computing is also often called Custom or Adaptive. The significant potential for the acceleration of computing in general-purpose applications has been demonstrated (S. Hauck, 1998; J. Villasnor and W.H. Mangionesmith, 1997). Reconfigurable systems are those computing platforms whose architecture is modified by the software to suit the application at hand. J.M. Maciejowski (1997) gave another definition for reconfigurable systems. According to him, the reconfigurable (control) systems are important when a major failure occurs. The paper combines both approaches of reconfiguration in trying to give a solution implementing a reconfigurable embedded controller for the vector control of an AC drive.

This paper presents a robust and effective method for bidimensional recognition of 2D or 3D objects, from intensity images based on: matching of the models with symbolic structures of the scene, using of the inexact matching, intensive using of techniques for search space reduction.>