Anomaly detection constitutes a fundamental step in developing self-aware autonomous agents capable of continuously learning from new situations, as it enables to distinguish novel experiences from already encountered ones. This paper combines Dynamic Bayesian Networks (DBNs) and Neural Networks (NNs) and proposes a method for detecting anomalies in video data at different abstraction levels. We use a Variational Autoencoder (VAE) to reduce the dimensionality of video frames, and Optical Flows between subsequent images, generating a latent space that captures both visual and dynamical information and that is comparable to low-dimensional sensory data (e.g., positioning, steering angle). An Adapted Markov Jump Particle Filter is employed to predict the following frames and detect anomalies in video data. Our method’s evaluation is executed using different video data from a semi-autonomous vehicle performing different tasks in a closed environment. Tests on benchmark anomaly detection datasets have additionally been conducted.

The technology for autonomous vehicles is close to replacing human drivers by artificial systems endowed with high-level decision-making capabilities. In this regard, systems must learn about the usual vehicle's behavior to predict imminent difficulties before they happen. An autonomous agent should be capable of continuously interacting with multi-modal dynamic environments while learning unseen novel concepts. Such environments are not often available to train the agent on it, so the agent should have an understanding of its own capacities and limitations. This understanding is usually called self-awareness. This paper proposes a multi-modal self-awareness modeling of signals coming from different sources. This paper shows how different machine learning techniques can be used under a generic framework to learn single modality models by using Dynamic Bayesian Networks. In the presented case, a probabilistic switching model and a bank of generative adversarial networks are employed to model a vehicle's positional and visual information respectively. Our results include experiments performed on a real vehicle, highlighting the potentiality of the proposed approach at detecting abnormalities in real scenarios.

Unmanned Aerial Vehicles (UAVs) attracted both industry and research community owing to their fascinating features like mobility, deployment flexibility and strong Line of Sight (LoS) links. The integration of Cognitive Radio (CR) can greatly help UAVs to overcome several issues especially spectrum scarcity. However, the dynamic radio environment in CR and the strong dependence of safe communications from LoS channels integrity in UAV communications make the Cognitive- UAV-Radio vulnerable to jamming attacks. This work aims to study the integration of CR and UAVs introducing a Self- Awareness (SA) framework from the physical layer security perspective. Under the SA framework, a Dynamic Bayesian Network (DBN) model is proposed as a representation of the radio environment and a modified Markov Jump Particle Filter (MJPF) is employed for prediction and state estimation purposes. A novel jammer detection framework is proposed that allows the UAV to perform abnormality evaluation at different hierarchical levels. The jammer is shown to be located effectively in both time and frequency domains. Experimental results show the effectiveness of the proposed framework in terms of detection probability and accuracy.

This paper proposes a method for performing continual learning of predictive models that facilitate the inference of future frames in video sequences. For a first given experience, an initial Variational Autoencoder, together with a set of fully connected neural networks are utilized to respectively learn the appearance of video frames and their dynamics at the latent space level. By employing an adapted Markov Jump Particle Filter, the proposed method recognizes new situations and integrates them as predictive models avoiding catastrophic forgetting of previously learned tasks. For evaluating the proposed method, this article uses video sequences from a vehicle that performs different tasks in a controlled environment.

Cellular connectivity for a massive number of Unmanned Aerial Vehicles (UAVs) will overcrowd the radio spectrum and cause spectrum scarcity. Incorporating Cognitive Radio (CR) with UAVs (Cognitive-UAV-Radios) has been proposed to overcome such an issue. However, the broadcasting nature of CR and the dominant line-of-sight links of UAV makes the Cognitive-UAV-Radios susceptible to jamming attacks. In this paper, we propose a framework to detect smart jammer, which locates and attacks the UAV commands with low Jamming-to-Signal-Power-Ratio (JSR). Smart jammer is more challenging than the types of jammers that always require high power values. Our work focuses on learning a Dynamic Bayesian Network (DBN) to model and analyze the signals' behaviour statistically. A Markov Jump Particle Filter (MJPF) is employed to perform predictions and consequently detect jamming signals. The results are satisfactory in terms of detection probability and false alarm rate that outperform the conventional Energy Detector approach.

This paper proposes a method to detect abnormal motions in real vehicle situations based on trajectory data. Our approach uses a Gaussian process (GP) regression that facilitates to approximate expected vehicle's movements over a whole environment based on sparse observed data. The main contribution of this paper consists in decomposing the GP regression into spatial zones, where quasi-constant velocity models are valid. Such obtained models are employed to build a set of Kalman filters that encode observed vehicle's dynamics. This paper shows how proposed filters enable the online identification of abnormal motions. Detected abnormalities can be modeled and learned incrementally, automatically by intelligent systems. The proposed methodology is tested on real data produced by a vehicle that interacts with pedestrians in a closed environment. Automatic detection of abnormal motions benefits the traffic scene understanding and facilitates to close the gap between human driving and autonomous vehicle awareness.

Self-learning abilities in autonomous systems are essential to improve their situational awareness and detection of normal/abnormal situations. In this work, we propose a graph matching technique for activity detection in autonomous agents by using the Gromov-Wasserstein framework. A clustering approach is used to discretise continuous agents' states related to a specific task into a set of nodes with similar objectives. Additionally, a probabilistic transition matrix between nodes is used as edges weights to build a graph. In this paper, we extract an abnormal area based on a sub-graph that encodes the differences between coupled of activities. Such sub-graph is obtained by applying a threshold on the optimal transport matrix, which is obtained through the graph matching procedure. The obtained results are evaluated through experiments performed by a robot in a simulated environment and by a real autonomous vehicle moving within a University Campus.

We propose a probabilistic method to track and interpret the interactions of moving objects. The proposed method is based on the analysis of location data from different moving objects that modify their dynamics according to rules of interactions, namely attractive and repulsive forces governing objects' motions in a scene. Our method uses a Bayesian structure to identify key elements of the interplay rules and facilitates the prediction of objects' dynamics as an interacting system. Such a prediction facilitates the detection of abnormalities by identifying unseen interaction effects in the scene.

We present an approach to learn a model to estimate the dynamical states at continuous and discrete inference levels when trajectory information is available. We learn from sparse data a probabilistic switching model that generates trajectories associated with a stationary plan of an agent. The learned generative model is used within a Markov Jump Linear System (MJLSs) to switch among set of space dependent linear filters that analyze new trajectories and detect deviations from the learned model based on internal innovation measurements. We show examples of application of the proposed approach to learn filters for evaluating deviations from a reference human driving task execution that includes static and dynamic obstacle avoidance.

This paper presents a novel approach for learning self-awareness models for autonomous vehicles. Proposed technique is based on the availability of synchronized multi-sensor dynamic data related to different maneuvering tasks performed by a human operator. It is shown that different machine learning approaches can be used to first learn single modality models using coupled Dynamic Bayesian Networks; such models are then correlated at event level to discover contextual multimodal concepts. In the presented case, visual perception and localization are used as modalities. Cross-correlations among modalities in time is discovered from data and are described as probabilistic links connecting shared and private multi-modal DBNs at the event (discrete) level. Results are presented on experiments performed on an autonomous vehicle, highlighting potentiality of the proposed approach to allow anomaly detection and autonomous decision making based on learned self-awareness models.

This paper focuses on multi-sensor anomaly detection for moving cognitive agents using both external and private first-person visual observations. Both observation types are used to characterize agents motion in a given environment. The proposed method generates locally uniform motion models by dividing a Gaussian process that approximates agents displacements on the scene and provides a Shared Level (SL) self-awareness based on Environment Centered (EC) models. Such models are then used to train in a semi-unsupervised way a set of Generative Adversarial Networks (GANs) that produce an estimation of external and internal parameters of moving agents. Obtained results exemplify the feasibility of using multi-perspective data for predicting and analyzing trajectory information.

This paper focuses on modeling and classifying trajectories from video sequences. Location, velocity and time of appearance are considered as features for recognizing and modeling motions of objects. In a training phase, a discretization of the proposed features is performed by using a self-organizing map approach such that a set of clusters (feature vocabulary) is created for describing trajectories. A cluster dissimilarity measure based on a weighted fusion of features facilitates the recognition of trajectory classes in an incremental way. As a result, an unsupervised method for encoding observed motion information and identifying trajectory patterns is proposed in this article. The method is evaluated with real and simulated data. Additionally' comparisons with previous works show the benefits of our method when encoding and identifying motion patterns in video sequences.

In recent years several architectures have been proposed to learn embodied agents complex self-awareness models. In this paper, dynamic incremental self-awareness (SA) models are proposed that allow experiences done by an agent to be modeled in a hierarchical fashion, starting from more simple situations to more structured ones. Each situation is learned from subsets of private agent perception data as a model capable to predict normal behaviors and detect abnormalities. Hierarchical SA models have been already proposed using low dimensional sensorial inputs. In this work, a hierarchical model is introduced by means of a cross-modal Generative Adversarial Networks (GANs) processing high dimensional visual data. Different levels of the GANs are detected in a self-supervised manner using GANs discriminators decision boundaries. Real experiments on semi-autonomous ground vehicles are presented.

We present a method to model and classify trajectory data that come from surveillance videos. Observations of the locations of moving entities are used to estimate their expected velocity in the scene. Such estimation is performed by a Gaussian process regression that enables to approximate probabilistically the expected velocity of entities given some observed evidence in the scene. Subsequently, regions where estimations have high certainty are decomposed into zones by superpixel segmentation. Each zone represents a region where motions of entities can be explained by a quasilinear dynamical model. We evaluated the proposed method with two datasets and confirmed its reliability for characterizing and classifying trajectories.

With the growing availability of wearable technology, video recording devices have become so intimately tied to individuals, that they are able to record the movements of users' hands, making hand-based applications one the most explored area in First Person Vision (FPV). In particular, hand pose recognition plays a fundamental role in tasks such as gesture and activity recognition, which in turn represent the base for developing human-machine interfaces or augmented reality applications. In this work we propose a graph-based representation of hands seen from the point of view of the user, obtained through the shape-fitting capability of a modified Instantaneous Topological Map. Spectral analysis of the graph Laplacian allows to arrange eigenvalues in vectors of features, which prove to be discriminative in classifying the considered hand poses.

This paper proposes a method for detecting zones of visual attention based on the motion of agents in a video analytics context. By considering a Hough transform approach, linear flow motions are grouped based on attractive salient zones where they converge. Each group of linear flows is generalized through the whole environment by using a non-parametric stochastic approach that can be used to generate a map that illustrates the effects that each zone exerts on the dynamics of agents. A dataset of walking pedestrians and trajectories generated by a robot that executes a single task in a close environment are used to validate the proposed method.