Semantic segmentation is fundamental for autonomous driving, requiring dense scene understanding across diverse and continuously changing environments. Real-world deployment must overcome two major challenges: (1) dynamic environmental shifts due to weather, lighting, and geography, and (2) the inability to retain labeled source data for continual adaptation. To address these issues, we propose a novel source-free adaptive and forgetting-resilient continual unsupervised domain adaptation for semantic segmentation (FORECASS). We introduce a teacher–student based framework, using EMA (Exponential Moving Average) updating technique, to produce stable pseudo-labels during continual adaptation. Central to our method is a refiner-based error estimation model that predicts pixel-wise pseudo-label reliability during adaptation. By leveraging the error map, the model selectively focuses learning on uncertain and challenging regions, playing a critical role in mitigating catastrophic forgetting. Complementarily, a structure-aware consistency mechanism enforces semantic coherence across views, further enhancing stability during sequential adaptation. Lightweight knowledge distillation is also incorporated to smooth alignment between the pseudo-label generator and the adaptation model. We validate our framework on the challenging continual adaptation sequence GTA → Cityscapes → IDD → Mapillary, achieving state-of-the-art results over both source-dependent and source-free baselines. The code is available at: https://github.com/baqar61/FORECASS .

Human action recognition is a prominent area of research in computer vision due to its wide-ranging applications, including surveillance, human–computer interaction, and autonomous systems. Although recent 3D CNN approaches have shown promising results by capturing both spatial and temporal information, they often struggle to incorporate the environmental context in which actions occur, limiting their ability to discriminate between similar actions and accurately recognize complex scenarios. To overcome these challenges, a novel and effective approach called Context-aware 3D CNN for Action Recognition based on Semantic segmentation (CARS) is presented in this paper. The CARS approach consists of an intermediate scene recognition module that uses a semantic segmentation model to capture contextual cues from video sequences. This information is then encoded and linked to the features captured by the 3D CNN model, resulting in a comprehensive global feature map. CARS integrates a Convolutional Block Attention Module (CBAM) that utilizes channel and spatial attention mechanisms to focus on the most relevant parts of the relevant 3D CNN feature map. We also replace the traditional cross-entropy loss with a focal loss that can better deal with underrepresented and hard- to-classify human actions. Extensive experiments on well-known benchmark datasets, including HMD51 and UCF101, show that the proposed CARS approach outperforms current 3D CNN-based state-of-the-art approaches. Moreover, the context extraction module in CARS is a generic plug-and-play network that can improve the classification performance of any 3D CNN architecture. • An effective scene action recognition based on semantic contextual information. • Integration of a reliable attention mechanism to enhance feature learning. • Improving the recognition of underrepresented action classes via focal loss. • Experiments on challenging benchmarks show higher performance over SOTA techniques.

Environment analysis is a critical part of autonomous vehicle for transport applications and for passenger safety. The solutions demonstrating the greatest robustness have been integrating multiple sensors used for redundancy and refinement purposes. Vision applications have proven to offer a high degree of flexibility and performance. One particular instance of this is higlighted in vehicle localisation, which predominantly relies on GNSS-based systems for positioning calculation using propagation time measurements. However, this signal may be degraded through the environment around the vehicle, worst case being urban canyons leading to Non Line Of Sight(NLOS) scenarios or multipaths issues due to reflecting obstacles. Previous work have shown vision-based algorithms can be used to mitigate these effects. One widely studied approach relies on the segmentation of an acquired wide-angle image installed on the roof of the vehicle and oriented toward the sky.Because the sky processing module is binary, the pipeline lack any way to express its uncertainty when applying weighting policies to the detected satellite state, which can be detrimental to the resulting positioning. In this paper, we propose a novel way of analysing wide-angle camera images, also known as fisheye images, dividing the image into patches to output the corresponding situation of each region of interest. Additionally we propose a new class to the previous sky versus non-sky segmentation, designated as mixed class and designed to serve as a fuzzy answer by the deep learning model to improve confidence to other scenarios as well as allow for new analysis policies of satellites signals. The data-driven algorithm is designed and tested on a publicly available dataset, composed of a large number of finely labelled images provided by ISAE-SUPAERO reaching a 94% accuracy.

Abstract Generative Adversarial Networks (GANs) have gained prominence in computer vision, with applications that extend to image fusion. Existing fusion methods often require extensive labeled data and task-specific training, limiting their generalizability. To address these limitations, this paper presents the Reconstruction-Guided Generative Adversarial Network (R2GAN), a generic GAN-based approach designed for generic image fusion, including visible-infrared, medical, and multi-focus image fusion. The proposed R2GAN architecture consists of a primary generator to improve fusion capabilities and auxiliary generators to ensure accurate reconstruction of source image features. To optimize the model, we propose a reconstruction-guided loss function to preserve the feature distribution of the source images and improve the consistency between the fused and source images. Additionally, we introduce a semantic segmentation-guided approach to generate a comprehensive and realistic Paired Multi-Focus image dataset (PMF) to train the R2GAN model. Experimental results in multiple fusion tasks demonstrate that R2GAN delivers superior performance, outperforming state-of-the-art image fusion methods. The R2GAN framework source code is available for access on GitHub at https://github.com/CHAHI24680/R2GAN .

Autonomous vehicles require online learning capabilities to enable long-term, unattended operation. However, long-term online learning is accompanied by the problem of forgetting previously learned knowledge. This paper introduces an online learning framework that includes a catastrophic forgetting prevention mechanism, named Long-Short-Term Online Learning (LSTOL). The framework consists of a set of shortterm learners and a long-term controller, where the former is based on the concept of ensemble learning and aims to achieve rapid learning iterations, while the latter contains a simple yet efficient probabilistic decision-making mechanism combined with four control primitives to achieve effective knowledge maintenance. A novel feature of the proposed LSTOL is that it avoids forgetting while learning autonomously. In addition, LSTOL makes no assumptions about the model type of short-term learners and the continuity of the data. The effectiveness of the proposed framework is demonstrated through experiments across well-known datasets in autonomous driving, including KITTI and Waymo. The source code for the method implementation is publicly available at https://github.com/epan-utbm/lstol.

Semantic segmentation is a well-studied topic and one of the most challenging tasks in computer vision applications, such as autonomous driving. Deep learning approaches based on convolutional neural networks (CNN) have demonstrated exceptional success on this task in recent years. Despite this success, existing approaches are plagued by higher-order inconsistencies between the ground truth images and the ones predicted by the segmentation model. This paper proposes a novel post-processing scheme based on adversarial learning to counter these inconsistencies. Such a scheme can be combined with a variety of existing CNN-based semantic segmentation networks to improve their segmentation performances. The proposed scheme is a Two-Stream Conditional Generative Adversarial Network (TScGAN), with one stream having initial semantic segmentation masks predicted by an existing CNN, while the other stream utilizes scene images to retain high-level information under a supervised residual network structure. In addition, TScGAN incorporates a novel dynamic weighting mechanism, which leads to significant and consistent gains in segmentation performance. Several comparative tests on public benchmark driving databases, including Cityscapes, Mapillary, and Berkeley DeepDrive100K, demonstrate the effectiveness of the proposed method when used with state-of-the-art CNN-based semantic segmentation models. Furthermore, the ablation experiment proved the structural rationality of our two-stream structure. The code for TScGAN could be found at https://github.com/epan-utbm/TScGAN-for-Improving-Semantic-Predictions

Haze is one of the atmospheric image degradations that causes severe distortions to outdoor images such as low contrast, color shift, and structure damage. Due to the unique physical characteristics of haze, the quality of hazy images is not accurately assessed using general-purpose image quality assessment (IQA) approaches. Therefore, several haze-aware IQA approaches have been proposed to provide more efficient dehazing quality evaluation. These approaches extract several haze-aware features to be either combined to form a single IQA metric or fed to a regression model that predicts the dehazing quality. However, these haze-relevant features are extracted using pixel intensity , in which luminance and structure information are inseparable, leading to less correlation between such features and the type of degradation they are supposed to represent. To address this issue, we propose a singular value decomposition (SVD) based IQA metric that can effectively separate the luminance component of an image from structure. This separation offers the ability to accurately evaluate the degradation at two different levels i.e. luminance and structure. The experimental results show that our proposed SVD-based dehazing quality evaluator (SDQE) outperforms the existing state-of-the-art non-reference IQA metrics in terms of accuracy and processing time.

Generative adversarial networks (GANs) represent a significant advance in the field of deep learning for image generation problems. With their ability to generate highly realistic and diverse images, GANs are quickly becoming the preferred technique for a wide range of applications. However, GANs come with several limitations and challenges, chief among which are their collapse mode and instability during training. In this paper, we propose a novel generator GAN model that incorporates the singular value decomposition (SVD) process into the decoder module. The SVD integration allows the generator to recognize the underlying structures in the feature space, resulting in more robust and effective image generation. This is achieved by adjusting the singular values during the training process, effectively optimizing the SVD-based generator to produce images that match the ground truth. Another advantage of SVD integration is its ability to perform discriminative spatial decomposition that clearly reflects the differences between the generated features and the target features. By including SVD in the generator model, the resulting loss values are higher and less prone to gradient vanishing than conventional GANs, such as Pix2Pix. Our proposed SVD-based generator model can be integrated into any auto-encoder architecture, making it as a generic and versatile solution for various image-generation tasks. The effectiveness of the proposed SVD-based GAN in image generation has been validated in three challenging benchmarks for image restoration and visible-to-infrared image translation. Extensive experiments demonstrate the significant quantitative and qualitative improvements achieved by our SVD-based GAN compared to baseline GAN architectures. The overall system outperforms the current state of the art in all benchmarks tested.

Monocular cues are one of the key elements considered by humans to appreciate the depth in their environment, even using only one eye. The idea of this paper is to help a deep neural network to perform the task of monocular depth estimation by integrating the so-called monocular cues into the learning process. To this end, we first propose an ontology model to define several human knowledge, mainly contextual, semantic, and geometric. Thus, thanks to ontology rules that imitate human-like reasoning, monocular cues maps are extracted and integrated into the deep neural network. The proposed approach can be applied in several domains, mainly autonomous driving, either in the urban environment or the industrial one. Our method was validated on the urban driving CityScapes dataset and have shown promising results.

Eye blink detection is a challenging problem that many researchers are working on because it has the potential to solve many facial analysis tasks, such as face anti-spoofing, driver drowsiness detection, and some health disorders. There have been few attempts to detect blinking in the wild scenario, while most of the work has been done under controlled conditions. Moreover, current learning approaches are designed to process sequences that contain only a single blink ignoring the case of the presence of multiple eye blinks. In this work, we propose a fast framework for eye blink detection and eye blink verification that can effectively extract multiple blinks from image sequences considering several challenges such as lighting changes, variety of poses, and change in appearance. The proposed framework employs fast landmarks detector to extract multiple facial key points including the ones that identify the eye regions. Then, an SVD-based method is proposed to extract the potential eye blinks in a moving time window that is updated with new images every second. Finally, the detected blink candidates are verified using a 2D Pyramidal Bottleneck Block Network (PBBN). We also propose an alternative approach that uses a sequence of frames instead of an image as input and employs a continuous 3D PBBN that follows most of the state-of-the-art approaches schemes. Experimental results show the better performance of the proposed approach compared to the state-of-the-art approaches.

The ability to sense and understanding the driving environment is a key technology for ADAS and autonomous driving. Human drivers have to pay more visual attention to important or target elements and ignore unnecessary ones present in their field of sight. A model that computes this visual attention of targets in a specific driving environment is essential and useful in supporting autonomous driving, object-specific tracking & detection, driving training, car collision warning, traffic sign detection, etc. In this paper, we propose a new framework of visual attention that can predict important objects in the driving scene using a conditional generative adversarial network. A large scale Visual Attention Driving Database (VADD) of saliency heat-maps is built from existing driving datasets using a saliency mechanism. The proposed framework model takes its strength from these saliency heat-maps as conditioning label variables. The results show that the proposed approach makes us able to predict heat-maps of most important objects in a driving environment.

As a critical sensor for high-level autonomous vehicles, LiDAR’s limitations in adverse weather (e.g. rain, fog, snow, etc.) impede the deployment of self-driving cars in all weather conditions. However, studies in literature on LiDAR’s performance in harsh conditions are insufficient. In this paper, based on a dataset collected with a popular Near-InfraRed (NIR) ToF LiDAR in a well-controlled artificial fog chamber, we statistically model the LiDAR ranging process in fog conditions through a data-driven approach. Specifically, giving an object at a known distance, our model is able to predict LiDAR measures (range and intensity) under various fog conditions. For a transmitted laser under fog, we first model and predict the minimum visibility required to detect its true range or not. Then, the noisy range and intensity measures are sampled from the probabilistic measurement distributions inferred from the dataset. The performance of the proposed method has been quantitatively and qualitatively evaluated. Experimental results show that our approach can provide a promising performance prediction of the utilized NIR ToF LiDAR under fog, which opens a new gate to the quantitative assessment of adverse weather and contributes to the specification of relevant Operational Domain Designs (ODDs). The developed ROS package is available at:https://github.com/cavayangtao/lanoising.

In this paper, we study the problem of autonomous and reliable positioning of a camera w.r.t. an object when only this latter is known but not the rest of the scene. We propose to combine the advantages and efficiency of a visual servoing scheme and the generalization ability of a generative adversarial network. The paper describes how to efficiently create a synthetic dataset in order to train a network that predicts an intermediate visual keyframe between two images. Subsequent predictions are used as visual features to autonomously converge towards the desired pose even for large displacements. We show that the proposed method can be used without any prior knowledge on the scene appearance except for the object itself, while being robust to various lighting conditions and specular surfaces. We provide experimental results, both in simulation and using a real service robot platform to validate and evaluate the effectiveness, robustness, and accuracy of our approach.

Detecting moving objects from a moving vehicle is a challenging problem and crucial for autonomous driving, especially in urban scenarios. The current literature has focused on this task as many approaches have been dedicated to moving object detection. These approaches consist of multistage pipelines, including semantic segmentation and optical flow estimation, and require multiple sources of information from active and passive sensors. However, they fail to accurately segment moving objects due to the large ego- camera motion, in addition to the high processing time. In this work, we propose a novel approach to moving object detection by processing information only from a camera. Our approach is based on integrating an encoder-decoder network (EDNet) with a semantic segmentation model (Mask R-CNN), where Mask R-CNN detects the objects of interest and the EDNet classifies their motion (moving/static) over two consecutive frames. We compare the results of our proposed model with existing MOD models on three SOTA benchmarks. We achieved SOTA performance in terms of visual quality and accuracy with competitive speed.

Classifying and modeling texture images, especially those with significant rotation, illumination, scale, and view-point variations, is a hot topic in the computer vision field. Inspired by local graph structure (LGS), local ternary patterns (LTP), and their variants, this paper proposes a novel image feature descriptor for texture and material classification, which we call Petersen Graph Multi-Orientation based Multi-Scale Ternary Pattern (PGMO-MSTP). PGMO-MSTP is a histogram representation that efficiently encodes the joint information within an image across feature and scale spaces, exploiting the concepts of both LTP-like and LGS-like descriptors, in order to overcome the shortcomings of these approaches. We first designed two single-scale horizontal and vertical Petersen Graph-based Ternary Pattern descriptors ($PGTP_{h}$and$PGTP_{v}$). The essence of$PGTP_{h}$and$PGTP_{v}$is to encode each$5\times 5$image patch, extending the ideas of the LTP and LGS concepts, according to relationships between pixels sampled in a variety of spatial arrangements (i.e., up, down, left, and right) of Petersen graph-shaped oriented sampling structures. The histograms obtained from the single-scale descriptors$PGTP_{h}$and$PGTP_{v}$are then combined, in order to build the effective multi-scale PGMO-MSTP model. Extensive experiments are conducted on sixteen challenging texture data sets, demonstrating that PGMO-MSTP can outperform state-of-the-art handcrafted texture descriptors and deep learning-based feature extraction approaches. Moreover, a statistical comparison based on the Wilcoxon signed rank test demonstrates that PGMO-MSTP performed the best over all tested data sets.

The complexity of a scene in addition to the need for real-time processing are the main challenges that face any background/foreground separation approach for maritime environment. Recent studies onLow-rank and Sparse Separation(LSS) achieved good performance when compared to traditional background subtraction techniques in segregating the foreground from a complex background. However, the issue of maintaining this type of a separation via an updating mechanism is not well addressed by the majority of LSS approaches. The study presents a tensor based singular value decomposition approach for background/foreground separation. The approach is uniquely designed to deal with most challenges related to a maritime environment such as sea dynamics, boat wakes, variety of foreground objects, and camera jitter. Furthermore, the proposed approach operates incrementally via updating the separation components as opposed to reperforming the decomposition on the entire video sequence when a set of frames arrives. Additionally, a forgetting mechanism is employed in the proposed approach to efficiently handle challenges such asStationary Foreground Objects(SFOs) and ghost effects. The performance of the proposed method with several state-of-the-art LSS and Non-LSS techniques on videos with complex maritime scenarios are evaluated. The results exhibit better performance over most of the tested challenges and also demonstrate the capability of the proposed method to perform the separation in less computational time.

Kinship verification aims to determine whether two persons are kin related or not. This is an emerging topic in computer vision due to its practical potential applications such as family album management. Most of previous works are based on checking kinship from face patterns and more recently from voices. We provide in this paper the first investigation in the literature on kinship verification from gait. The main purpose is to study whether family members do share some gait patterns. As this is a new topic, we started by collecting a new dataset for kinship verification from human gait containing several pairs of video sequences of celebrities and their relatives. The database will be released to the research community for research purposes. Along with the database, we provide results using baseline methods using silhouette and video based analysis. Moreover, we also propose a two-stream 3DCNN to tackle the problem. The preliminary experimental results point out the potential usefulness of gait information for kinship verification.

The field of autonomous driving has grown tremendously over the past few years, along with the rapid progress in sensor technology. One of the major purposes of using sensors is to provide environment perception for vehicle understanding, learning and reasoning, and ultimately interacting with the environment. In this paper, we first introduce a multisensor platform allowing vehicle to perceive its surroundings and locate itself in a more efficient and accurate way. The platform integrates eleven heterogeneous sensors including various cameras and lidars, a radar, an IMU (Inertial Measurement Unit), and a GPS-RTK (Global Positioning System / Real-Time Kinematic), while exploits a ROS (Robot Operating System) based software to process the sensory data. Then, we present a new dataset (https://epan-utbm.github.io/utbm_robocar_dataset/) for autonomous driving captured many new research challenges (e.g. highly dynamic environment), and especially for long-term autonomy (e.g. creating and maintaining maps), collected with our instrumented vehicle, publicly available to the community.

As a critical sensor for high-level autonomous vehicles, LiDAR's limitations in adverse weather (e.g. rain, fog, snow, etc.) impede the deployment of self-driving cars in all weather conditions. In this paper, we model the performance of a popular 903nm ToF LiDAR under various fog conditions based on a LiDAR dataset collected in a well-controlled artificial fog chamber. Specifically, a two-stage data-driven method, called LaNoising (la for laser), is proposed for generating LiDAR measurements under fog conditions. In the first stage, the Gaussian Process Regression (GPR) model is established to predict whether a laser can successfully output a true detection range or not, given certain fog visibility values. If not, then in the second stage, the Mixture Density Network (MDN) is used to provide a probability prediction of the noisy measurement range. The performance of the proposed method has been quantitatively and qualitatively evaluated. Experimental results show that our approach can provide a promising description of 903nm ToF LiDAR performance under fog.

Writer identification is experiencing a revival of activity in recent years and continues to attract great deal of attention as a challenging and important area of research in the field of forensic and authentication. In this work, we introduce a reliable off-line system for text-independent writer identification of handwritten documents. Feature engineering is an essential component of a pattern recognition system, which can enhance or decrease the classification performance. A well-designed and defined feature extraction method improves the classification task. This paper proposes, for feature extraction, an effective, yet high-quality and conceptually simple feature image descriptor referred to as Cross multi-scale Locally encoded Gradient Patterns (CLGP). The proposed CLGP feature extraction method, which is expected to better represent salient local writing structure, operates at small observation regions (i.e., connected component sub-images) of the writing sample. CLGP histogram feature vectors computed from all these observation regions in all writing samples are considered as classification inputs to identify query writers using the Nearest Neighbor Classifier (1-NN). Our system is evaluated on six standard databases (IFN/ENIT, AHTID/MW, CVL, IAM, Firemaker, and ICDAR2011) including handwritten samples in Arabic, English, French, Greek, German, and Dutch languages. Comparing the identification performance with old and recent state-of-the-art methods, the proposed system achieves the highest performance on IFN/ENIT, AHTID/MW, and ICDAR2011 databases, and demonstrates competitive performance on IAM, CVL, and Firemaker databases.

The background Initialization (BI) problem has attracted the attention of researchers in different image/video processing fields. Recently, a tensor-based technique called spatiotemporal slice-based singular value decomposition (SS-SVD) has been proposed for background initialization. SS-SVD applies the SVD on the tensor slices and estimates the background from low-rank information. Despite its efficiency in background initialization, the performance of SS-SVD requires further improvement in the case of complex sequences with challenges such as stationary foreground objects (SFOs), illumination changes, low frame-rate, and clutter. In this paper, a self-motion-assisted tensor completion method is proposed to overcome the limitations of SS-SVD in complex video sequences and enhance the visual appearance of the initialized background. With the proposed method, the motion information, extracted from the sparse portion of the tensor slices, is incorporated with the low-rank information of SS-SVD to eliminate existing artifacts in the initiated background. Efficient blending schemes between the low-rank (background) and sparse (foreground) information of the tensor slices is developed for scenarios such as SFO removal, lighting variation processing, low frame-rate processing, crowdedness estimation, and best frame selection. The performance of the proposed method on video sequences with complex scenarios is compared with the top-ranked state-of-the-art techniques in the field of background initialization. The results not only validate the improved performance over the majority of the tested challenges but also demonstrate the capability of the proposed method to initialize the background in less computational time.

Traffic signs play an important role for Advanced Driver Assistance Systems (ADAS) as well as for autonomous driving vehicles. Most of the works done focus on recognizing symbol based signs leaving apart important information provided by other type of signs like complementary panels or text based signs. In this paper, we include detection and classification of both symbol and text based signs focusing on the most common ones found in European urban environments. The system consists of three stages, traffic sign detection, refinement and classification. The detection and refinement is performed using Mask R-CNN while the classification is achieved with a proposed Convolutional Neural Network (CNN) architecture. We introduced the extended version of the German Traffic Sign Detection Benchmark (GTSDB), labeled in a pixel manner (masks) with 164 classes grouped into 8 categories. It is used for the detection and classification steps. Experimental results on German environments show that our proposed system is capable of detecting all categories of traffic signs while at the same time recognizing them with high accuracy achieving comparable performance with the state of the art.

This paper presents an efficient handcrafted texture operator for texture modeling and classification. The proposed descriptor, referred to as local directional decoded ternary pattern (LDDTP), consists in encoding both directional pattern features and contrast information in a compact way based on local derivative variations. The proposed operator first computes for each pixel within its 3×3 overlapping square neighborhood, on the one hand, central edge response through the 2ndderivative of Gaussian filter, and on the other hand, eight directional edge responses using the eight Frei-Chen masks to capture more detailed information. Then, spatial relationships among the neighboring pixels through the generated edge responses are exploited independently with the help of the concepts of LTP and LDP operators to enhance the discriminative power. Finally, the produced LDDTP pattern is splitted into two distinct parts: local directional decoded ternary pattern lower ( LDDTPL) and local directional decoded ternary pattern upper ( LDDTPU), which are combined into hybrid distributions to form the final LDDTP feature descriptor. Experimental results on eight publicly available texture datasets showed that the proposed LDDTP descriptor achieves classification performance, which is competitive or better than several old and recent state-of-the-art LBP variants.

The graph-based embedding is an effective and useful method in reducing the dimension and extracting relevant data. This paper introduces a framework for classifying high dimensional data via a joint graph-based embedding and weighting method which could be used in semi-supervised or supervised learning. We design on effective optimization algorithm to solve the objective function. Experiments on image classification show that our proposed method can have a performance that is better than that of many state-of-the-art methods including linear and nonlinear methods.

Facial expression conveys important signs about the human affective state, cognitive activity, intention and personality. In fact, the automatic facial expression recognition systems are getting more interest year after year due to its wide range of applications in several interesting fields such as human computer/robot interaction, medical applications, animation and video gaming. In this study, the authors propose to combine between different descriptors features (histogram of oriented gradients, local phase quantisation and binarised statistical image features) after applying principal component analysis on each of them to recognise the six basic expressions and the neutral face from the static images. Their proposed fusion method has been tested on four popular databases which are: JAFFE, MMI, CASIA and CK+, using two different cross‐validation schemes: subject independent and leave‐one–subject‐out. The obtained results show that their method outperforms both the raw features concatenation and state‐of‐the‐art methods.

In tasks such as abandoned luggage detection and stopped car detection, stationary foreground objects (SFOs) need to be detected and properly classified in real time. Different methods have been proposed to detect SFOs, but they are mainly focused on certain types of objects. In this paper, an incremental singular value decomposition-based method is presented to detect all types of SFOs such as abandoned objects and removed objects. The proposed method decomposes the video tensor spatiotemporally and divides it into background and foreground components. An appropriate analysis is applied to the foreground tensor to define a pixel time series of each stationary foreground category. Such analysis leads to the fact that SFOs can be detected easily owing to their continuous persistence in the decomposed foreground tensor. Furthermore, the unique structure of the pixel time series of each category allows identifying the category of the detected objects, whether they are abandoned or removed, and detecting the exact time of the start and end of each event. The results demonstrate that the proposed method achieves a superior performance in detecting SFOs at both object and pixel levels. In addition, the proposed method is computationally simple, and its complexity is lower compared to other approaches; hence, it can adequately satisfy real-time requirements.

This paper proposes a deep background subtraction method based on conditional Generative Adversarial Network (cGAN). The proposed model consists of two successive networks: generator and discriminator. The generator learns the mapping from the observing input (i.e., image and background), to the output (i.e., foreground mask). Then, the discriminator learns a loss function to train this mapping by comparing real foreground (i.e., ground-truth) and fake foreground (i.e., predicted output) with observing the input image and background. Evaluating the model performance with two public datasets, CDnet 2014 and BMC, shows that the proposed model outperforms the state-of-the-art methods.

In the past years, various graph-based data embedding algorithms were proposed and used in machine learning and pattern recognition fields. This paper introduces a graph-based non-linear embedding learning algorithm for image classification and recognition. The proposed embedding method can be used for supervised and semi-supervised learning settings. The proposed criterion allows the simultaneous estimation of a linear and a non-linear embedding. It integrates manifold smoothness, Sparse Regression and Margin Discriminant Embedding. The deployed sparse regression implicitly performs feature selection on the original features of the data matrix and of the linear transform. The proposed method is applied to four image datasets: 8 Sports Event Categories dataset, Scene 15 dataset, ORL Face dataset and COIL-20 Object dataset. The experiments demonstrate the effectiveness of the proposed embedding method.

Extracting the background from a video in the presence of various moving patterns is the focus of several background-initialization approaches. To model the scene background using rank-one matrices, this paper proposes a background-initialization technique that relies on the singular-value decomposition (SVD) of spatiotemporally extracted slices from the video tensor. The proposed method is referred to as spatiotemporal slice-based SVD (SS-SVD). To determine the SVD components that best model the background, a depth analysis of the computation of the left/right singular vectors and singular values is performed, and the relationship with tensor-tube fibers is determined. The analysis proves that a rank-1 matrix extracted from the first left and right singular vectors and singular value represents an efficient model of the scene background. The performance of the proposed SS-SVD method is evaluated using 93 complex video sequences of different challenges, and the method is compared with state-of-the-art tensor/matrix completion-based methods, statistical-based methods, search-based methods, and labeling-based methods. The results not only show better performance over most of the tested challenges, but also demonstrate the capability of the proposed technique to solve the background-initialization problem in a less computational time and with fewer frames.

Recently, Convolutional Network (ConvNet) features permit to achieve state-of-the-art performance in robotic fields such as visual navigation and SLAM. In this paper, a visual localization technique was proposed based on ConvNet networks by combining the powerful ConvNet features and image sequence matching. The pre-trained networks provided by MatConvNet are used to extract the features and then a sequence search technique is applied for visual recognition. Compared with the traditional approaches based on handcraft features and single image matching, the proposed method shows good performances even in presence of appearance and illumination changes. We present extensive experiments on five real world datasets to evaluate each of the specific challenges in visual recognition. We also conduct a comprehensive performance comparison of different ConvNet layers (each defining a level of features) considering both appearance and illumination changes.

Safety and security are the most discussed topics in the road and railway transportation field. Latest security initiatives in the field of railway transportation propose to implement video surveillance at level crossing (LC) environments. In this paper we explore the possibility of implementing a smart video surveillance security system that is tuned toward detecting and evaluating abnormal situations induced by users (pedestrians, vehicle drivers, and unattended objects) in LCs. This intelligent security system starts by detecting, separating, and tracking moving objects shot in the LC. Then, a hidden Markov model is developed to estimate ideal trajectories, allowing the detected targets to discard dangerous situations. After that, the level of risk of each target is instantly estimated by using the Dempster-Shafer data fusion technique. The proposed analysis allows for also recognizing hazard scenarios. The video surveillance system is connected to a communication system (the Wireless Access for Vehicular Environment), which takes the information on the dynamic status of the LC (safe or presence of a dangerous situation) and sends it to users approaching the LC. Four hazard scenarios are tested and evaluated with different real video image sequences: presence of the obstacle in the LC, presence of the stopped vehicles line, vehicle zigzagging between two closed half barriers, and pedestrian crossing the LC area.

The goal of structured mesh is to generate a compressed representation of the 3D surface, where near objects are provided with more details than objects far from the camera, according to the disparity map. The solution is based on the Kohonens Self-Organizing Map algorithm for the benefits of its ability to generate a topological map according to a probability distribution and its potential to be a natural massive parallel algorithm. The disparity map, which stands for a density distribution that reflects the proximity of objects to the camera, is partitioned into an appropriate number of cell units, in such a way that each cell is associated to a processing unit and responsible of a certain area of the plane. The advantage of the proposed model is that it is decentralized and based on data decomposition. The required processing units and memory are with linearly increasing relationship to the problem size. Experimental results show that our GPU implementation is able to provide near real-time performance with small size disparity maps and the running time increases in a linear way with a very weak increasing coefficient. The proposed method is suitable to deal with large scale problems in a massively parallel way.

In this paper, we develop a new efficient graph construction algorithm that is useful for many learning tasks. Unlike the main stream for graph construction, our proposed data self-representativeness approach simultaneously estimates the graph structure and its edge weights through sample coding. Compared with the recent l1 graph that is based on sparse coding, our proposed objective function has an analytical solution (based on self-representativeness of data) and thus is more efficient. This paper has two main contributions. Firstly, we introduce the Two Phase Weighted Regularized Least Square (TPWRLS) graph construction. Secondly, the obtained data graph is used, in a semi-supervised context, in order to categorize detected objects in driving/urban scenes using Local Binary Patterns as image descriptors. The experiments show that the proposed method can outperform competing methods.

This paper presents a cellular GPU model for structured mesh generation according to an input stereo-matching disparity map. Here, the disparity map stands for a density distribution that reflects the proximity of objects to the camera in 3D space. The meshing process consists in covering such data density distribution with a topological structured hexagonal grid that adapts itself and deforms according to the density values. The goal is to generate a compressed mesh where the nearest objects are provided with more details than objects which are far from the camera. The solution we propose is based on the Kohonen's Self-Organizing Map learning algorithm for the benefit of its ability to generate a topological map according to a probability distribution and its ability to be a natural massive parallel algorithm. We propose a GPU parallel model and its implantation of the SOM standard algorithm, and present experiments on a set of standard stereo-matching disparity map benchmarks.

In intelligent vehicle field, occupancy grid maps are popular tools for representing the environment. Usually, occupancy grids, mapping the environment as a field of uniformly distributed binary/ternary variables, are generated by various kinds of sensors (e.g. lidar, radar, monocular/binocular vision system). In literature, most of proposed occupancy grid mapping methods create array-based fixed-resolution maps in either cartesian, polar, or column/disparity spaces. The problems of such maps are accuracy deficiency and prohibitive memory cost when trying to increase the resolution. This paper addresses these issues by presenting a novel variableresolution occupancy grid map based on quadtree structure from stereovision measurements. In the proposed method, a quadtree-based grid map with a settled resolution is calculated from a stereoscopic system at first. Then, the previously created map adapts its resolution to actual stereo measurements by merging or splitting nodes in the quadtree structure. The principal advantage of the proposed method is the ability to improve map's accuracy. Meanwhile, compared with some existing methods, the stereo-vision based occupancy grid mapping algorithm is improved. Experimental results with real datasets demonstrate that the accuracy of grid map created by our method is promoted from decimeter to centimeter.

Moving objects detection and recognition around an intelligent vehicle are active research fields. A great number of approaches have been proposed in recent decades. This paper proposes a novel approach based solely on spatial information to solve this problem. Moving objects detection is achieved in conjunction with an egomotion estimation by sparse matched feature points. For objects recognition, we firstly present a method to boost simple spatial information by Kernel Principal Component Analysis (KPCA). Then, two kinds of classifiers (Random Forest and Gradient Boosting Trees) are trained offline to recognize several common categories of moving objects in urban scenarios (vehicle, pedestrian, cyclist, ...). Experiments are implemented and the results confirm the effectiveness of the proposed algorithm. Furthermore, a comparison to a previous similar method is performed to verify the enhancement of classification by the advanced spatial features.

3D scene based objects detection and tracking is a central problem in many intelligent transportation applications. Dynamic stereo vision is the known approach to solve this problem. It consists in detecting and tracking objects from their reconstructed features using stereo images. This paper proposes a new method for detecting and tracking objects using stereo vision with linear cameras. Edge points extracted from the stereo linear images are first matched to reconstruct points that represent the objects in the scene. To detect the objects, a clustering process based on a spectral analysis is then applied to the reconstructed points. The obtained clusters are finally tracked throughout their center of gravity using Kalman filtering and a Nearest Neighbour based data association algorithm. Experimental results using real stereo linear images are shown to demonstrate the effectiveness of the proposed methods for obstacle detection and tracking in front of a vehicle.

This paper is focused on the characterization of GNSS signals reception environment by estimating the percentage of visible sky. A new segmentation technique based on a color watershed using an adaptive combination of color and texture information is proposed. This information is represented by two morphological gradients, a classical color gradient and a texture gradient based on co-occurrence matrices. The segmented images are then used as input for a k-means classifier in order to determine the percentage of visible sky in fish-eye images. The obtained classification results are evaluated to demonstrate the effectiveness and the reliability of the proposed approach.

This paper presents a fusion method for objects tracking using laser sensory data and stereovision. Based on the extended Kalman filter, the tracking uses an oriented bounding box (OBB) representation for tracked objects. The representation model takes into account an inter-rays (IR) uncertainty concept, which is related to the fact that the laser raw data points representing the extremities of an extracted OBB do not coincide with the real objects extremities. To improve the objects state estimation, the tracking process integrates a fixed size (FS) assumption. The FS assumption allows to exploit the most precise object's size estimation, memorised during the tracking. To achieve data association, a threshold based laser points clustering provides satisfying results. However, there are many cases where, without additional information, it is impossible to cluster laser raw data points correctly. To discard clustering ambiguities, a fusion method combining laser sensory data and stereovision information is proposed. The stereovision information is extracted only within regions of interest, defined from laser points. The fusion method takes place in the early stage of the measurement extraction from laser raw data points. The proposed approach is tested and evaluated to demonstrate its reliability.

The focus of this paper is on real-time obstacle detection using linear stereo vision. This paper presents a multilevel neural method for matching edges extracted from stereo linear images. The method described performs edge stereo matching at different levels with a neural-network-based procedure. At each level, the process starts by selecting, in the left and right linear images, the most significant edges, i.e., those with the largest gradient magnitudes. The selected edges are then matched and the obtained pairs are used as reference pairs for matching less significant edges in the next level. In each level, the matching problem is formulated as an optimization task in which an objective function, representing the constraints on the solution, is minimized thanks to a Hopfield neural network.

This work is concerned with the stereo matching problem for real-time obstacle detection. The correspondence problem is viewed as an optimization task where the objective is to find a solution for which the matches are as compatible as possible with respect to specific constraints. The optimization process is performed by means of a genetic algorithm with a new encoding scheme. For an effective exploitation of the genetic algorithm for real-time obstacle detection, a multilevel searching strategy is proposed in order to speed-up the stereo matching process. The multilevel searching strategy consists of matching the edges at different levels by considering their gradient magnitudes. The performance of the proposed multilevel genetic stereo matching procedure is evaluated for real-time obstacle detection in front of a moving vehicle using linear stereo vision.

Depth from stereo is one of the most active research areas in the computer vision field. The heavily investigated problem in stereo approaches is the matching between two or more images of a scene observed, by two or more video cameras, from different viewpoints. It consists of identifying features in the left and right images that are projections of the same physical feature in the three-dimensional world. This paper presents a real-time stereo matching method using a voting schema. The correspondence problem is first mapped onto a two-dimensional matrix, called matching matrix, where each element represents a possible match between two features extracted from the left and right images. Local and global constraints are then used to search the true elements of the matching matrix, which represent compatible matches. The valid elements are determined by applying the local constraints. Global constraints are used to define the voting rules between the valid elements. The voting based-method is evaluated for real-time obstacle detection in front of a moving car using linear stereo vision.

Stereo vision is a popular approach allowing to compute 3D structure of a scene seen by two or more video cameras from different viewpoints. The heavily investigated problem in this approach is the stereo matching problem. We present a new genetic scheme to the correspondence problem where a new solution encoding is proposed. To evaluate a solution, the fitness function is defined from three competing constraints, such that best matches correspond to its minima. Experimental results are presented to demonstrate the effectiveness of the proposed approach for extracting depth information from stereo linear images.