This paper proposes an algorithm to model and process streams of LiDAR data under an autonomous vehicle framework. LiDAR is assumed to be an exteroceptive sensor that allows the vehicle to have dynamic 3D scene perception of its surroundings. We employ an encoder-decoder architecture based on 3D-Convolutional layers called 3D Convolution Encoder-Decoder (3D-CED), together with a transfer learning strategy to extract a set of features from point clouds, which are relevant in the context of autonomous driving. The resulting features allow to make inferences of the future point cloud data and detect multiple abstraction level anomalies in controlled scenarios by utilizing a probabilistic switching dynamic model called High Dimensional Markov Jump Particle Filter (HD-MJPF). Moreover, a comparison is provided between piecewise linear, piecewise nonlinear, and nonlinear predictive models for anomaly detection at multiple abstraction levels. Our approach is evaluated with data collected from the LiDAR sensors of the autonomous vehicle while performing certain tasks in a controlled environment.

Anomaly detection constitutes a fundamental step in developing self-aware autonomous agents capable of continuously learning from new situations, as it enables to distinguish novel experiences from already encountered ones. This paper combines Dynamic Bayesian Networks (DBNs) and Neural Networks (NNs) and proposes a method for detecting anomalies in video data at different abstraction levels. We use a Variational Autoencoder (VAE) to reduce the dimensionality of video frames, and Optical Flows between subsequent images, generating a latent space that captures both visual and dynamical information and that is comparable to low-dimensional sensory data (e.g., positioning, steering angle). An Adapted Markov Jump Particle Filter is employed to predict the following frames and detect anomalies in video data. Our method’s evaluation is executed using different video data from a semi-autonomous vehicle performing different tasks in a closed environment. Tests on benchmark anomaly detection datasets have additionally been conducted.

The technology for autonomous vehicles is close to replacing human drivers by artificial systems endowed with high-level decision-making capabilities. In this regard, systems must learn about the usual vehicle's behavior to predict imminent difficulties before they happen. An autonomous agent should be capable of continuously interacting with multi-modal dynamic environments while learning unseen novel concepts. Such environments are not often available to train the agent on it, so the agent should have an understanding of its own capacities and limitations. This understanding is usually called self-awareness. This paper proposes a multi-modal self-awareness modeling of signals coming from different sources. This paper shows how different machine learning techniques can be used under a generic framework to learn single modality models by using Dynamic Bayesian Networks. In the presented case, a probabilistic switching model and a bank of generative adversarial networks are employed to model a vehicle's positional and visual information respectively. Our results include experiments performed on a real vehicle, highlighting the potentiality of the proposed approach at detecting abnormalities in real scenarios.

This paper proposes a method for performing continual learning of predictive models that facilitate the inference of future frames in video sequences. For a first given experience, an initial Variational Autoencoder, together with a set of fully connected neural networks are utilized to respectively learn the appearance of video frames and their dynamics at the latent space level. By employing an adapted Markov Jump Particle Filter, the proposed method recognizes new situations and integrates them as predictive models avoiding catastrophic forgetting of previously learned tasks. For evaluating the proposed method, this article uses video sequences from a vehicle that performs different tasks in a controlled environment.

In a computational context, self-awareness (SA) is a capability of an autonomous system to describe the acquired experience about itself and its surrounding environment with appropriate models and correlate them incrementally with the currently perceived situation to expand its knowledge continuously. This article introduces a bio-inspired framework for generative and descriptive dynamic models that support SA computationally and efficiently. Generative models facilitate predicting future states, while descriptive models enable the selection of the representation that best fits the current observation. Our framework is founded on the analysis and extension of three bio-inspired theories that have studied SA from different viewpoints, and we demonstrate how probabilistic techniques, such as cognitive dynamic Bayesian networks and generalized filtering paradigms, can learn appropriate models from multidimensional proprioceptive and exteroceptive signals acquired by the autonomous system. We discuss essential capabilities for SA and show how our modeling framework supports these capabilities in theory and through a case study where a mobile robot uses multisensorial data to determine its internal and environmental state as well as distinguishing among normal and abnormal behaviors.

This paper proposes a method to detect abnormal motions in real vehicle situations based on trajectory data. Our approach uses a Gaussian process (GP) regression that facilitates to approximate expected vehicle's movements over a whole environment based on sparse observed data. The main contribution of this paper consists in decomposing the GP regression into spatial zones, where quasi-constant velocity models are valid. Such obtained models are employed to build a set of Kalman filters that encode observed vehicle's dynamics. This paper shows how proposed filters enable the online identification of abnormal motions. Detected abnormalities can be modeled and learned incrementally, automatically by intelligent systems. The proposed methodology is tested on real data produced by a vehicle that interacts with pedestrians in a closed environment. Automatic detection of abnormal motions benefits the traffic scene understanding and facilitates to close the gap between human driving and autonomous vehicle awareness.

In autonomous systems, self-awareness capabilities are useful to allow artificial agents to detect abnormal situations based on previous experiences. This paper presents a method that facilitates the incremental learning of new models by an agent. Available learned models can dynamically generate probabilistic predictions as well as evaluate their mismatch from current observations. Observed mismatches are grouped through an unsupervised learning strategy into different classes, each of them corresponding to a dynamic model in a given region of the state space. Such clusters define switching Dynamic Bayesian Networks (DBNs) employed for predicting future instances and detect anomalies. Inferences generated by several DBNs that use different sensorial data are compared quantitatively. For testing the proposed approach, it is considered the multi-sensorial data generated by a robot performing various tasks in a controlled environment and a real autonomous vehicle moving at a University Campus.

We propose a probabilistic method to track and interpret the interactions of moving objects. The proposed method is based on the analysis of location data from different moving objects that modify their dynamics according to rules of interactions, namely attractive and repulsive forces governing objects' motions in a scene. Our method uses a Bayesian structure to identify key elements of the interplay rules and facilitates the prediction of objects' dynamics as an interacting system. Such a prediction facilitates the detection of abnormalities by identifying unseen interaction effects in the scene.

We present an approach to learn a model to estimate the dynamical states at continuous and discrete inference levels when trajectory information is available. We learn from sparse data a probabilistic switching model that generates trajectories associated with a stationary plan of an agent. The learned generative model is used within a Markov Jump Linear System (MJLSs) to switch among set of space dependent linear filters that analyze new trajectories and detect deviations from the learned model based on internal innovation measurements. We show examples of application of the proposed approach to learn filters for evaluating deviations from a reference human driving task execution that includes static and dynamic obstacle avoidance.

This paper presents a novel approach for learning self-awareness models for autonomous vehicles. Proposed technique is based on the availability of synchronized multi-sensor dynamic data related to different maneuvering tasks performed by a human operator. It is shown that different machine learning approaches can be used to first learn single modality models using coupled Dynamic Bayesian Networks; such models are then correlated at event level to discover contextual multimodal concepts. In the presented case, visual perception and localization are used as modalities. Cross-correlations among modalities in time is discovered from data and are described as probabilistic links connecting shared and private multi-modal DBNs at the event (discrete) level. Results are presented on experiments performed on an autonomous vehicle, highlighting potentiality of the proposed approach to allow anomaly detection and autonomous decision making based on learned self-awareness models.

This paper focuses on multi-sensor anomaly detection for moving cognitive agents using both external and private first-person visual observations. Both observation types are used to characterize agents motion in a given environment. The proposed method generates locally uniform motion models by dividing a Gaussian process that approximates agents displacements on the scene and provides a Shared Level (SL) self-awareness based on Environment Centered (EC) models. Such models are then used to train in a semi-unsupervised way a set of Generative Adversarial Networks (GANs) that produce an estimation of external and internal parameters of moving agents. Obtained results exemplify the feasibility of using multi-perspective data for predicting and analyzing trajectory information.

This paper focuses on modeling and classifying trajectories from video sequences. Location, velocity and time of appearance are considered as features for recognizing and modeling motions of objects. In a training phase, a discretization of the proposed features is performed by using a self-organizing map approach such that a set of clusters (feature vocabulary) is created for describing trajectories. A cluster dissimilarity measure based on a weighted fusion of features facilitates the recognition of trajectory classes in an incremental way. As a result, an unsupervised method for encoding observed motion information and identifying trajectory patterns is proposed in this article. The method is evaluated with real and simulated data. Additionally' comparisons with previous works show the benefits of our method when encoding and identifying motion patterns in video sequences.

In recent years several architectures have been proposed to learn embodied agents complex self-awareness models. In this paper, dynamic incremental self-awareness (SA) models are proposed that allow experiences done by an agent to be modeled in a hierarchical fashion, starting from more simple situations to more structured ones. Each situation is learned from subsets of private agent perception data as a model capable to predict normal behaviors and detect abnormalities. Hierarchical SA models have been already proposed using low dimensional sensorial inputs. In this work, a hierarchical model is introduced by means of a cross-modal Generative Adversarial Networks (GANs) processing high dimensional visual data. Different levels of the GANs are detected in a self-supervised manner using GANs discriminators decision boundaries. Real experiments on semi-autonomous ground vehicles are presented.

We present a method to model and classify trajectory data that come from surveillance videos. Observations of the locations of moving entities are used to estimate their expected velocity in the scene. Such estimation is performed by a Gaussian process regression that enables to approximate probabilistically the expected velocity of entities given some observed evidence in the scene. Subsequently, regions where estimations have high certainty are decomposed into zones by superpixel segmentation. Each zone represents a region where motions of entities can be explained by a quasilinear dynamical model. We evaluated the proposed method with two datasets and confirmed its reliability for characterizing and classifying trajectories.

To understand the behavior of moving entities in a given environment, one should be capable of predicting their motion, that is, to model their dynamics. In a setting where different behaviors can arise, one can assume that each of them corresponds to different motivational states of observed entities. Here, those motivations are understood as goal positions or spots where entities seek to arrive. To build prediction models based on that idea, we present an unsupervised method to estimate motivational spots actively. Additionally, we use the output of such process to refine an adaptive system modeling the dynamics of inferred hidden causes of observed data. The whole method uses deep variational methods, and particularly, the network estimating motivations is trained through dynamic programming. Results show that modeling the dynamics of entities can be better achieved by integrating information about motivational spots. Notably, a network modeling the dynamics converges faster through the incorporation of information about motivations.

This paper proposes a method for detecting zones of visual attention based on the motion of agents in a video analytics context. By considering a Hough transform approach, linear flow motions are grouped based on attractive salient zones where they converge. Each group of linear flows is generalized through the whole environment by using a non-parametric stochastic approach that can be used to generate a map that illustrates the effects that each zone exerts on the dynamics of agents. A dataset of walking pedestrians and trajectories generated by a robot that executes a single task in a close environment are used to validate the proposed method.

Cognitive radio (CR) is a promising technology for future wireless spectrum allocation to improve the use of licensed bands. However, security challenges faced by cognitive radio technology are still a hot research topic. One of prevailing challenges is the radio frequency jamming attack, where adversaries are able to exploit on-the-fly reconfigurability potentials and learning mechanism of cognitive radios in order to devise and deploy advanced jamming tactics. Jamming attacks can significantly impact the performance of wireless communication systems and lead to significant overheads in terms of retransmission and increment of power consumption. In this context, a novel jammer detection algorithm is proposed using cyclic spectral analysis and artificial neural networks (ANN) for wide-band (WB) cognitive radios. The proposed approach assumes a WB spectrum occupied by various narrow-band (NB) signals, which can be either legitimate or jamming signals. The second order statistics, namely, the spectral correlation function (SCF) and ANN are used to classify each NB signal as a legitimate or jamming signal. The algorithm performance is shown with the help of simulations.

A method for online identification of group of moving objects in the video is proposed in this paper. This method at each frame identifies group of tracked objects with similar local instantaneous motion pattern using spectral clustering on motion similarity graph. Then, the output of the algorithm is used to detect the event of more than two object moving together as required by PETS2015 challenge. The performance of the algorithm is evaluated on the PETS2015 dataset.