This paper proposes a hierarchical trajectory planning framework for UAVs operating under adversarial jamming conditions. Leveraging Bayesian Active Inference, the approach combines expert-generated demonstrations with probabilistic generative modeling to encode high-level symbolic planning, low-level motion policies, and wireless signal feedback. During deployment, the UAV performs online inference to anticipate interference, localize jammers, and adapt its trajectory accordingly—without prior knowledge of jammer locations. Simulation results demonstrate that the proposed method achieves near-expert performance, significantly reducing communication interference and mission cost compared to model-free reinforcement learning baselines, while maintaining robust generalization in dynamic environments.

In this paper, we propose a novel method for abnormality detection in Unmanned Surface Vehicles (USVs) based on a Multi-Modal Bayesian generative model to enhance safety and monitoring.During the training phase, we use a Null Force Filter and an unsupervised clustering algorithm on multimodal data collected from Global Positioning System (GPS) and motor current sensors.In the testing phase, we use a Coupled Modified Markov Jump Particle Filter (CM-MJPF) to infer the GPS position and motor current of the USV, as well as to detect abnormalities in both modalities.Due to the coupled methodology, the system is able to learn the statistical similarity between the evolving GPS and motor current data.As a result, the causality of defects is inherently captured within the dynamical inference, making the proposed approach explainable.

Unmanned aerial vehicles (UAVs) used as aerial base stations show significant promise for future wireless communication systems. This paper explores using a UAV as an autonomous agent, navigating over multiple hotspots to serve ground users (GUs) and maximize data transmission rates through strategic trajectory design. Existing interpretability methods often prove insufficient in providing comprehensive insights and generating logical, sequential decisions. In this paper, we propose an explainable reinforcement learning framework designed to produce interpretable and verifiable agent policies. Our method starts with an expert optimizer to solve training examples, enabling the learning agent (UAV) to analyse the solutions. Furthermore, we employ inverse reinforcement learning for data-driven reward function estimation. Additionally, we use a probabilistic Q-table function to understand and explain the actions executed by the expert across diverse environmental contexts, allowing the learning agent to produce interpretable policies that meet reasonable performance goals and easily transferred to unseen environments. Preliminary results indicate that the proposed approach is promising in achieving explainability in RL agents.

Autonomous vehicles (AVs) must be able to locate themselves accurately in dynamic environments in order to be able to navigate safely. In this paper, we present a robust LiDAR-based framework to improve the localization of AVs based on the classification of static and dynamic tracks. The approach leverages static tracks previously classified as reliable landmarks. Using Growing Neural Gas, Joint Probabilistic Data Association, and Unmotivated Kalman Filter algorithms, interaction dictionaries and vocabularies are generated. These serve as inputs for a Markov Jump Particle Filter (MJPF), which enables the accurate estimation of the ego-vehicle trajectory. Testing of the proposed framework with real-world LiDAR data demonstrates that it is capable of providing highly accurate localization results without the use of odometry, making it adaptable for use in environments without GPS. AV situational awareness and navigation performance are enhanced by using static tracks as robust references.

Experience sharing is a fundamental learning capability of human nature. Imagine that you need to pass through a congested road in the shortest time possible. You would try to remember how you previously handle such situations and optimize the route by fusing this knowledge with previous similar experiences. In addition, one can integrate these experiences with other information sources that suggest solutions. Transferring this type of intelligence to autonomous navigation systems implies the ability to first represent experiences through discriminative and generative models, and then create new optimal models, by fusing new and existing scenarios that the agent encounters. We propose a meta-level experience-sharing and learning method for a Bayesian autonomous agent. We introduce a joint state estimation method for discrete and continuous random variables, using Hierarchical Dynamic Bayesian Networks. In the proposed system, meta-level knowledge is learned and represented from lower dimensional odometry data, by observing behaviors from a third-person viewpoint. The learned knowledge is then transformed into a first-person agent viewpoint to perform such tasks online. Multiple learned predictive models and new experiences are fused into a single and more robust model during the online stage. Through the principle of free energy minimization, the learned Hierarchical Dynamic Bayesian Networks are used to coherently estimate discrete and continuous state variables for hierarchical-level anomaly detection and inference problems. Our results demonstrate that the proposed method optimizes the number of models by fusing similar experiences managed by the agent, while making anomaly detection and incremental learning steps more robust.

Learning from experience is a fundamental capability of intelligent agents. Autonomous systems rely on sensors that provide data about the environment and internal situations to their perception systems for learning and inference mechanisms. These systems can also learn Self-Aware and Situation-Aware generative modules from these data to localize themselves and interact with the environment. In this paper, we propose a self-aware cognitive architecture capable to perform tasks where the interactions between the self-state of an agent and the surrounding environment are explicitly and dynamically represented. We specifically develop a Deep Learning (DL) based Self-Aware interaction model, empowered by learning from Multi-Modal Perception (MMP) and World Models using multi-sensory data in a novel Multi-Agent Self-Awareness Architecture (MASAA). Two sub-modules are developed, the Situation Model (SM) and the First-Person model (FPM), that address different and interrelated aspects of the World Model (WM). The MMP model, instead, aims at learning the mapping of different sensory perceptions into Exteroceptive (EI) and Proprioceptive (PI) latent information. The WM then uses the learned MMP model as experience to predict dynamic self-behaviors and interaction patterns within the experienced environment. WM and MMP Models are learned in a data-driven way, starting from the lower-dimensional odometry data used to guide the learning of higher-dimensional video data, thus generating coupled Generalized State Hierarchical Dynamic Bayesian Networks (GS-HDBNs). We test our model on KITTI, CARLA, and iCab datasets, achieving high performance and a low average localization error (RMSE) of 2.897%, when considering two interacting agents.

Generalizability and interpretability are common terminologies that can be found in today’s machine learning algorithm design. Generalizability requires a clear understanding of one’s own action (self-awareness) and a robust interaction with the environment (situation awareness). Many current studies are devoted in developing an algorithm that is more robust in generalizing unseen situations while explaining self-action. However, such algorithms are complex and are not yet fully developed to be used in production. Intelligent transportation systems like self-driving cars are one of the emerging technologies that need generalizability and explainability in anomalous conditions. We propose to enhance generalizability and interpretability of a self-driving car model by introducing a novel methodology that fuses multi-sensorial data from proprioceptive and exteroceptive sensors of an agent, coupled in a Hierarchical Dynamic Bayesian Network model, in an Active Inference framework. The developed model has three stages: 1) a lower dimensional unsupervised learning stage, considering odometry and action modalities, carried out by first applying Null Force Filtering and then by applying modified GNG clustering algorithms; 2) a self-supervised higher-dimensional video modality learning stage assisted by the learned odometry vocabularies; and 3) an online model-based active learning in continuous and discrete state spaces, and action spaces, in the Active Inference framework. The developed system is tested using the CARLA simulator environment for localizing interacting agents, and exhibits low error compared to state-of-the-art methods.

Intelligent transportation systems (ITSs) provide a paradigm change in perceiving and interacting with transportation networks, leading to enhanced levels of safety, sustainability, and efficiency. Vehicular-to-everything (V2X) communication is the core component in the ITSs. The proprioceptive and exteroceptive sensors allow these vehicles to be aware of the surrounding environment and respond to emergencies by utilizing their abilities to reach a high level of self-awareness. In this paper, we propose a self-awareness approach to learn a generative dynamic Bayesian network (G-DBN) from the real-time LiDAR perception. Without reducing the dimensionality, we perform offline training and online testing phases on the three-dimensional (3D) point clouds. In the offline training phase, initially, the raw point clouds are preprocessed using a joint probabilistic data association filter (JPDAF) to obtain the 3D tracks of the multiple vehicles in space. Then, we perform an unsupervised clustering on all the generalized states (GSs) containing positions and velocities (a 6D vector) by considering the growing neural gas (GNG) technique, thus achieving a trained model from the 3D LiDAR point clouds. In the online testing phase, the high-dimensional Markov jump particle filter (HD-MJPF) utilizes the G-DBN’s probabilistic information to predict the positions of multiple vehicles and to detect the abnormalities at the discrete and continuous levels in normal and abnormal scenarios. Our proposed approach is useful for learning high-dimensional generative models and provides a way to meet the current curse of dimensionality challenges, that machine learning models are suffering.

This paper presents a novel self-supervised path-planning method for UAV-aided networks. First, we employed an optimizer to solve training examples offline and then used the resulting solutions as demonstrations from which the UAV can learn the world model to understand the environment and implicitly discover the optimizer’s policy. UAV equipped with the world model can make real-time autonomous decisions and engage in online planning using active inference. During planning, UAV can score different policies based on the expected surprise, allowing it to choose among alternative futures. Additionally, UAV can anticipate the outcomes of its actions using the world model and assess the expected surprise in a self-supervised manner. Our method enables quicker adaptation to new situations and better performance than traditional RL, leading to broader generalizability.

The following paper proposes a novel Vehicle-to-Everything (V2X) network abnormality detection scheme based on Bayesian generative models for enhanced network self-awareness functionality at the Base station (BS). In the learning phase, multimodal data signals contrived by the vehicles' integrated and sensing module are imbued into data-driven Gen-eralized Dynamic Bayesian network (GDBN) models. Following that, during the testing phase, an Interactive Modified Markov Jump Particle filter (IM-MJPF) is utilized to forecast forthcoming network states and vehicle trajectories by leveraging the assimilated semantics embedded in the coupled multi-GDBNs. This approach involves learning statistically correlated association between evolving trajectories and network communication links. Security and surveillance of Internet of Vehicles (IOVs) links are performed online with high detection probabilities by matching predicted with observed network connectivity maps (graphs).

Self-aware computing systems enable computing systems to reflect on their actions and behavior. This becomes even more relevant in Cyber-Physical Systems where computing systems have to control and interact with elements in the real world. This paper reports on recent advances made in computational self-awareness for cyber-physical systems.

Active inference is a probabilistic framework for modeling intelligent agent behaviours, which drives by the principle of minimizing free energy. In this paper, we integrate the imitation learning method with active inference to minimize the expected free energy under the supervision of an expert model. The proposed approach affords explainable decision-making as a combination of self-information and novelty-seeking or exploratory behavior in a hierarchical generative model. A lane-changing driving scenario is demonstrated to verify the efficiency of the proposed framework that outperforms conventional Reinforcement learning methods.

Vehicle-to-everything (V2X) communication is expected to be a prominent component of the sixth generation (6G) to accomplish intelligent transportation systems (ITS). Autonomous vehicles relying only on onboard sensors cannot bypass the limitations of safety and reliability. Thus, integrated sensing and communication is proposed as an effective way to achieve high situational- and self-awareness levels, enabling V2X to perceive the physical world and adjust its behaviour to emergencies. Secure navigation through the Global Positioning System (GPS) is essential in ITS for safe operation. Nevertheless, due to the lack of encryption and authentication mechanisms of civil GPS receivers, spoofers can easily replicate satellite signals by launching GPS spoofing attacks to deceive the vehicle and manipulate navigation data. In addition, due to its shared nature, V2X links are prone to jamming attacks which might endanger vehicular safety. This paper proposes a method to jointly detect GPS spoofing and jamming attacks in a V2X network. Simulation results demonstrate that the proposed method can detect spoofers and jammers with high detection probabilities.

The combination of different sensory information to predict upcoming situations is an innate capability of intelligent beings. Consequently, various studies in the Artificial Intelligence field are currently being conducted to transfer this ability to artificial systems. Autonomous vehicles can particularly benefit from the combination of multi-modal information from the different sensors of the agent. This paper proposes a method for video-frame prediction that leverages odometric data. It can then serve as a basis for anomaly detection. A Dynamic Bayesian Network framework is adopted, combined with the use of Deep Learning methods to learn an appropriate latent space. First, a Markov Jump Particle Filter is built over the odometric data. This odometry model comprises a set of clusters. As a second step, the video model is learned. It is composed of a Kalman Variational Autoencoder modified to leverage the odometry clusters for focusing its learning attention on features related to the dynamic tasks that the vehicle is performing. We call the obtained overall model Cluster-Guided Kalman Variational Autoencoder. Evaluation is conducted using data from a car moving in a closed environment and leveraging a part of the University of Alcalá DriveSet dataset, where several drivers move in a normal and drowsy way along a secondary road.

This paper proposes a multi-modal situational inter-action model for collaborative agents by fusing multi-sensorial information in a Multi-Agent Hierarchical Dynamic Bayesian Network (MAH-DBN) framework. The proposed model is learned in a data-driven methodology to estimate the states of interacting agents only from video sequences. This can be regarded as a two-fold methodology for improving visual-based localization and interaction between autonomous agents. In the learning stage, the odometry model is used to drive the video learning model for a robust localization and interaction modeling. During the testing phase, the learned Multi-Agent Hierarchical DBN (MAH-DBN) model is used for the localization of collaborative agents only from video sequences by proposing an inference method called Multi-Agent Coupled Markov Jump Particle Filter (MAC-MJPF).

This paper proposes a method to localize a moving agent - such as an autonomous surveillance vehicle - inside a known environment using First Person Viewpoint video data. Anomalies w.r.t. expected vehicle motion and image content are extracted to guide the localization, signal when the localization results are not trustworthy and explain the reason for the failure. During the training phase, a Dynamic Bayesian Network model is learned, which couples positional and video data. To learn it, clustering is performed on the odometry data, and a modified Kalman Variational Autoencoder is built over the video data. During the testing phase, a Coupled Markov Jump Particle Filter leverages the learned Dynamic Bayesian Network to extract anomalies and to estimate the vehicle’s position, given only camera data. The proposed method is evaluated on two real-world datasets of a vehicle performing perimeter monitoring of a closed environment and of a shopping cart moving in a supermarket.

In the research area of human-robot interactions, the automatic estimation of the mass of a container manipulated by a person leveraging only visual information is a challenging task. The main challenges consist of occlusions, different filling materials and lighting conditions. The mass of an object constitutes key information for the robot to correctly regulate the force required to grasp the container. We propose a single RGB-D camera-based method to locate a manipulated container and estimate its empty mass i.e., independently of the presence of the content. The method first automatically selects a number of candidate containers based on the distance with the fixed frontal view, then averages the mass predictions of a lightweight model to provide the final estimation. Results on the CORSMAL Containers Manipulation dataset show that the proposed method estimates empty container mass obtaining a score of 71.08% under different lighting or filling conditions.

AI-empowered Cognitive Radio (i.e., AI-enabled radios) is a paradigm shift to achieve the highest level of Self-Awareness in future wireless communications. This work proposes a joint automatic modulation conversion and classification (AMCC) framework, which allows an AI-enabled wireless node to predict signals' dynamics of different modulation schemes and explain how it can be transported (converted) with minimal effort and forwarded with higher spectral efficiency. To achieve this goal, we propose a Generalized Filtering framework integrated by Transport Planning to learn the way of converting low-order modulations to high-order modulations, which has also been validated by performing the automatic modulation classification. Simulation results demonstrate the effective performance of our novel framework on converting and classifying multiple modulation formats.

This paper proposes an algorithm to model and process streams of LiDAR data under an autonomous vehicle framework. LiDAR is assumed to be an exteroceptive sensor that allows the vehicle to have dynamic 3D scene perception of its surroundings. We employ an encoder-decoder architecture based on 3D-Convolutional layers called 3D Convolution Encoder-Decoder (3D-CED), together with a transfer learning strategy to extract a set of features from point clouds, which are relevant in the context of autonomous driving. The resulting features allow to make inferences of the future point cloud data and detect multiple abstraction level anomalies in controlled scenarios by utilizing a probabilistic switching dynamic model called High Dimensional Markov Jump Particle Filter (HD-MJPF). Moreover, a comparison is provided between piecewise linear, piecewise nonlinear, and nonlinear predictive models for anomaly detection at multiple abstraction levels. Our approach is evaluated with data collected from the LiDAR sensors of the autonomous vehicle while performing certain tasks in a controlled environment.

Anomaly detection constitutes a fundamental step in developing self-aware autonomous agents capable of continuously learning from new situations, as it enables to distinguish novel experiences from already encountered ones. This paper combines Dynamic Bayesian Networks (DBNs) and Neural Networks (NNs) and proposes a method for detecting anomalies in video data at different abstraction levels. We use a Variational Autoencoder (VAE) to reduce the dimensionality of video frames, and Optical Flows between subsequent images, generating a latent space that captures both visual and dynamical information and that is comparable to low-dimensional sensory data (e.g., positioning, steering angle). An Adapted Markov Jump Particle Filter is employed to predict the following frames and detect anomalies in video data. Our method’s evaluation is executed using different video data from a semi-autonomous vehicle performing different tasks in a closed environment. Tests on benchmark anomaly detection datasets have additionally been conducted.

This paper presents the outcomes of the PETS2021 challenge held in conjunction with AVSS2021 and sponsored by the EU FOLDOUT project. The challenge comprises the publication of a novel video surveillance dataset on through-foliage detection, the defined challenges addressing person detection and tracking in fragmented occlusion scenarios, and quantitative and qualitative performance evaluation of challenge results submitted by six worldwide participants. The results show that while several detection and tracking methods achieve overall good results, through-foliage detection and tracking remains a challenging task for surveillance systems especially as it serves as the input to behaviour (threat) recognition.

This paper proposes a method for performing future-frame prediction and anomaly detection on video data in a multi-modal framework based on Dynamic Bayesian Networks (DBNs). In particular, odometry data and video data from a moving vehicle are fused. A Markov Jump Particle Filter (MJPF) is learned on odometry data, and its features are used to aid the learning of a Kalman Variational Autoencoder (KVAE) on video data. Consequently, anomaly detection can be performed on video data using the learned model. We evaluate the proposed method using multi-modal data from a vehicle performing different tasks in a closed environment.

A novel approach is proposed for multimodal collective awareness (CA) of multiple networked intelligent agents. Each agent is here considered as an Internet-of-Things (IoT) node equipped with machine learning capabilities; CA aims to provide the network with updated causal knowledge of the state of execution of actions of each node performing a joint task, with particular attention to anomalies that can arise. Data-driven dynamic Bayesian models learned from multisensory data recorded during the normal realization of a joint task (agent network experience) are used for distributed state estimation of agents and detection of abnormalities. A set of switching dynamic Bayesian network (DBN) models collectively learned in a training phase, each related to particular sensorial modality, is used to allow each agent in the network to perform synchronous estimation of possible abnormalities occurring when a new task of the same type is jointly performed. Collective DBN (CDBN) learning is performed by unsupervised clustering of generalized errors (GEs) obtained from a starting generalized model. A growing neural gas (GNG) algorithm is used as a basis to learn the discrete switching variables at the semantic level. Conditional probabilities linking nodes in the CDBN models are estimated using obtained clusters. CDBN models are associated with a Bayesian inference method, namely, distributed Markov jump particle filter (D-MJPF), employed for joint state estimation and abnormality detection. The effects of networking protocols and of communications in the estimation of state and abnormalities are analyzed. Performance is evaluated by using a small network of two autonomous vehicles performing joint navigation tasks in a controlled environment. In the proposed method, first the sharing of observations is considered in ideal condition, and then the effects of a wireless communication channel have been analyzed for the collective abnormality estimation of the agents. Rician wireless channel and the usage of two protocols (i.e., IEEE 802.11p and IEEE 802.15.4) along with different channel conditions are considered as well.

The technology for autonomous vehicles is close to replacing human drivers by artificial systems endowed with high-level decision-making capabilities. In this regard, systems must learn about the usual vehicle's behavior to predict imminent difficulties before they happen. An autonomous agent should be capable of continuously interacting with multi-modal dynamic environments while learning unseen novel concepts. Such environments are not often available to train the agent on it, so the agent should have an understanding of its own capacities and limitations. This understanding is usually called self-awareness. This paper proposes a multi-modal self-awareness modeling of signals coming from different sources. This paper shows how different machine learning techniques can be used under a generic framework to learn single modality models by using Dynamic Bayesian Networks. In the presented case, a probabilistic switching model and a bank of generative adversarial networks are employed to model a vehicle's positional and visual information respectively. Our results include experiments performed on a real vehicle, highlighting the potentiality of the proposed approach at detecting abnormalities in real scenarios.

Unmanned Aerial Vehicles (UAVs) attracted both industry and research community owing to their fascinating features like mobility, deployment flexibility and strong Line of Sight (LoS) links. The integration of Cognitive Radio (CR) can greatly help UAVs to overcome several issues especially spectrum scarcity. However, the dynamic radio environment in CR and the strong dependence of safe communications from LoS channels integrity in UAV communications make the Cognitive- UAV-Radio vulnerable to jamming attacks. This work aims to study the integration of CR and UAVs introducing a Self- Awareness (SA) framework from the physical layer security perspective. Under the SA framework, a Dynamic Bayesian Network (DBN) model is proposed as a representation of the radio environment and a modified Markov Jump Particle Filter (MJPF) is employed for prediction and state estimation purposes. A novel jammer detection framework is proposed that allows the UAV to perform abnormality evaluation at different hierarchical levels. The jammer is shown to be located effectively in both time and frequency domains. Experimental results show the effectiveness of the proposed framework in terms of detection probability and accuracy.

This paper proposes a method for performing continual learning of predictive models that facilitate the inference of future frames in video sequences. For a first given experience, an initial Variational Autoencoder, together with a set of fully connected neural networks are utilized to respectively learn the appearance of video frames and their dynamics at the latent space level. By employing an adapted Markov Jump Particle Filter, the proposed method recognizes new situations and integrates them as predictive models avoiding catastrophic forgetting of previously learned tasks. For evaluating the proposed method, this article uses video sequences from a vehicle that performs different tasks in a controlled environment.

Cellular connectivity for a massive number of Unmanned Aerial Vehicles (UAVs) will overcrowd the radio spectrum and cause spectrum scarcity. Incorporating Cognitive Radio (CR) with UAVs (Cognitive-UAV-Radios) has been proposed to overcome such an issue. However, the broadcasting nature of CR and the dominant line-of-sight links of UAV makes the Cognitive-UAV-Radios susceptible to jamming attacks. In this paper, we propose a framework to detect smart jammer, which locates and attacks the UAV commands with low Jamming-to-Signal-Power-Ratio (JSR). Smart jammer is more challenging than the types of jammers that always require high power values. Our work focuses on learning a Dynamic Bayesian Network (DBN) to model and analyze the signals' behaviour statistically. A Markov Jump Particle Filter (MJPF) is employed to perform predictions and consequently detect jamming signals. The results are satisfactory in terms of detection probability and false alarm rate that outperform the conventional Energy Detector approach.

Millimeter Wave (mmWave) band can be a solution to serve the vast number of Internet of Things (IoT) and Vehicle to Everything (V2X) devices. In this context, Cognitive Radio (CR) is capable of managing the mmWave spectrum sharing efficiently. However, Cognitive mmWave Radios are vulnerable to malicious users due to the complex dynamic radio environment and the shared access medium. This indicates the necessity to implement techniques able to detect precisely any anomalous behaviour in the spectrum to build secure and efficient radios. In this work, we propose a comparison framework between deep generative models: Conditional Generative Adversarial Network (C-GAN), Auxiliary Classifier Generative Adversarial Network (AC-GAN), and Variational Auto Encoder (VAE) used to detect anomalies inside the dynamic radio spectrum. For the sake of the evaluation, a real mmWave dataset is used, and results show that all of the models achieve high probability in detecting spectrum anomalies. Especially, AC-GAN that outperforms C-GAN and VAE in terms of accuracy and probability of detection.

The advancements in connected and autonomous vehicles in these times demand the availability of tools providing the agents with the capability to be aware and predict their own states and context dynamics. This article presents a novel approach to develop an initial level of collective awareness (CA) in a network of intelligent agents. A specific collective self-awareness functionality is considered, namely, agent-centered detection of abnormal situations present in the environment around any agent in the network. Moreover, the agent should be capable of analyzing how such abnormalities can influence the future actions of each agent. Data-driven dynamic Bayesian network (DBN) models learned from time series of sensory data recorded during the realization of tasks (agent network experiences) are here used for abnormality detection and prediction. A set of DBNs, each related to an agent, is used to allow the agents in the network to reach synchronously aware possible abnormalities occurring when available models are used on a new instance of the task for which DBNs have been learned. A growing neural gas (GNG) algorithm is used to learn the node variables and conditional probabilities linking nodes in the DBN models; a Markov jump particle filter (MJPF) is employed for state estimation and abnormality detection in each agent using learned DBNs as filter parameters. Performance metrics are discussed to asses the algorithm's reliability and accuracy. The impact is also evaluated by the communication channel used by the network to share the data sensed in a distributed way by each agent of the network. The IEEE 802.11p protocol standard has been considered for communication among agents. Performances of the DBN-based abnormality detection models under different channel and source conditions are discussed. The effects of distances among agents and of the delays and packet losses are analyzed in different scenario categories (urban, suburban, and rural). Real data sets are also used acquired by autonomous vehicles performing different tasks in a controlled environment.

In a computational context, self-awareness (SA) is a capability of an autonomous system to describe the acquired experience about itself and its surrounding environment with appropriate models and correlate them incrementally with the currently perceived situation to expand its knowledge continuously. This article introduces a bio-inspired framework for generative and descriptive dynamic models that support SA computationally and efficiently. Generative models facilitate predicting future states, while descriptive models enable the selection of the representation that best fits the current observation. Our framework is founded on the analysis and extension of three bio-inspired theories that have studied SA from different viewpoints, and we demonstrate how probabilistic techniques, such as cognitive dynamic Bayesian networks and generalized filtering paradigms, can learn appropriate models from multidimensional proprioceptive and exteroceptive signals acquired by the autonomous system. We discuss essential capabilities for SA and show how our modeling framework supports these capabilities in theory and through a case study where a mobile robot uses multisensorial data to determine its internal and environmental state as well as distinguishing among normal and abnormal behaviors.

This paper proposes a method to detect abnormal motions in real vehicle situations based on trajectory data. Our approach uses a Gaussian process (GP) regression that facilitates to approximate expected vehicle's movements over a whole environment based on sparse observed data. The main contribution of this paper consists in decomposing the GP regression into spatial zones, where quasi-constant velocity models are valid. Such obtained models are employed to build a set of Kalman filters that encode observed vehicle's dynamics. This paper shows how proposed filters enable the online identification of abnormal motions. Detected abnormalities can be modeled and learned incrementally, automatically by intelligent systems. The proposed methodology is tested on real data produced by a vehicle that interacts with pedestrians in a closed environment. Automatic detection of abnormal motions benefits the traffic scene understanding and facilitates to close the gap between human driving and autonomous vehicle awareness.

Self-learning abilities in autonomous systems are essential to improve their situational awareness and detection of normal/abnormal situations. In this work, we propose a graph matching technique for activity detection in autonomous agents by using the Gromov-Wasserstein framework. A clustering approach is used to discretise continuous agents' states related to a specific task into a set of nodes with similar objectives. Additionally, a probabilistic transition matrix between nodes is used as edges weights to build a graph. In this paper, we extract an abnormal area based on a sub-graph that encodes the differences between coupled of activities. Such sub-graph is obtained by applying a threshold on the optimal transport matrix, which is obtained through the graph matching procedure. The obtained results are evaluated through experiments performed by a robot in a simulated environment and by a real autonomous vehicle moving within a University Campus.

In autonomous systems, self-awareness capabilities are useful to allow artificial agents to detect abnormal situations based on previous experiences. This paper presents a method that facilitates the incremental learning of new models by an agent. Available learned models can dynamically generate probabilistic predictions as well as evaluate their mismatch from current observations. Observed mismatches are grouped through an unsupervised learning strategy into different classes, each of them corresponding to a dynamic model in a given region of the state space. Such clusters define switching Dynamic Bayesian Networks (DBNs) employed for predicting future instances and detect anomalies. Inferences generated by several DBNs that use different sensorial data are compared quantitatively. For testing the proposed approach, it is considered the multi-sensorial data generated by a robot performing various tasks in a controlled environment and a real autonomous vehicle moving at a University Campus.

The barrier to entry for the development of independent, browser-based multiplayer games is high for two reasons: complexity and cost. In this work, we introduce and evaluate a method and library that aims to make this barrier as small as possible, by utilising appropriate development abstractions and peer-to-peer communication between players. Our preliminary evaluation shows that we can lower both the technical development overhead, as well as minimise server costs, at no loss to performance.

We propose a probabilistic method to track and interpret the interactions of moving objects. The proposed method is based on the analysis of location data from different moving objects that modify their dynamics according to rules of interactions, namely attractive and repulsive forces governing objects' motions in a scene. Our method uses a Bayesian structure to identify key elements of the interplay rules and facilitates the prediction of objects' dynamics as an interacting system. Such a prediction facilitates the detection of abnormalities by identifying unseen interaction effects in the scene.

Technologies such as cognitive radio and dynamic spectrum access rely on spectrum sensing which provides wireless devices with information about the radio spectrum in the surrounding environment. One of the main challenges in wireless communications is the interference caused by malicious users on the shared spectrum. In this manuscript, an artificial intelligence enabled cognitive radio framework is proposed at system‐level as part of a cyclic spectrum intelligence algorithm for interference mitigation in wideband radios. It exploits the cyclostationary feature of signals to differentiate users with different modulation schemes and an artificial neural network as classifier to detect potential malicious users. A dataset consisting of experimental modulated and dynamic signals is recorded by spectrum measurements with an in‐house software defined radio testbed and then processed. Cyclostationary features are extracted for each detected signal and fed to a neural network classifier as training and testing data in a complex and dynamic scenario. Results highlight a classification rate of in most of cases, even at low transmission power. A comparison with two previous works with hand‐crafted features, which employ an energy detector‐based classifier and a naive Bayes‐based classifier, respectively, is discussed.

This paper presents, a system capable of detecting unusual activities in crowds from real-world data captured from multiple sensors. The detection is achieved by classifying the distinct movements of people in crowds, and those patterns can be different and can be classified as normal and abnormal activities. Statistical features are extracted from the dataset collected by applying sliding time window operations. A model for classifying movements is trained by using Random Forest technique. The system was tested by using two datasets collected from mobile phones during social events gathering. Results show that mobile data can be used to detect anomalies in crowds as an alternative to video sensors with significant performances. Our approach is the first to detect any unusual behaviour in crowd with non-visual data, which is simple to train and easy to deploy. We also present our dataset for public research as there is no such dataset available to perform experiments on crowds for detecting unusual behaviours.

In this paper, an efficient method for crowd abnormal behavior detection and localization is introduced. Despite the significant improvements of deep-learning-based methods in this field, but still, they are not fully applicable for the real-time applications. We propose a simple yet effective descriptor based on binary tracklets, containing both orientation and magnitude information in a single feature. The results of the proposed method are comparable with deep-based methods while it performs more efficiently. The evaluation of our descriptors on three different datasets yields a promising result in abnormality detection, which is competitive with the state-of-the-art methods.

We present an approach to learn a model to estimate the dynamical states at continuous and discrete inference levels when trajectory information is available. We learn from sparse data a probabilistic switching model that generates trajectories associated with a stationary plan of an agent. The learned generative model is used within a Markov Jump Linear System (MJLSs) to switch among set of space dependent linear filters that analyze new trajectories and detect deviations from the learned model based on internal innovation measurements. We show examples of application of the proposed approach to learn filters for evaluating deviations from a reference human driving task execution that includes static and dynamic obstacle avoidance.

This paper presents a novel approach for learning self-awareness models for autonomous vehicles. Proposed technique is based on the availability of synchronized multi-sensor dynamic data related to different maneuvering tasks performed by a human operator. It is shown that different machine learning approaches can be used to first learn single modality models using coupled Dynamic Bayesian Networks; such models are then correlated at event level to discover contextual multimodal concepts. In the presented case, visual perception and localization are used as modalities. Cross-correlations among modalities in time is discovered from data and are described as probabilistic links connecting shared and private multi-modal DBNs at the event (discrete) level. Results are presented on experiments performed on an autonomous vehicle, highlighting potentiality of the proposed approach to allow anomaly detection and autonomous decision making based on learned self-awareness models.

This paper focuses on multi-sensor anomaly detection for moving cognitive agents using both external and private first-person visual observations. Both observation types are used to characterize agents motion in a given environment. The proposed method generates locally uniform motion models by dividing a Gaussian process that approximates agents displacements on the scene and provides a Shared Level (SL) self-awareness based on Environment Centered (EC) models. Such models are then used to train in a semi-unsupervised way a set of Generative Adversarial Networks (GANs) that produce an estimation of external and internal parameters of moving agents. Obtained results exemplify the feasibility of using multi-perspective data for predicting and analyzing trajectory information.

This paper focuses on modeling and classifying trajectories from video sequences. Location, velocity and time of appearance are considered as features for recognizing and modeling motions of objects. In a training phase, a discretization of the proposed features is performed by using a self-organizing map approach such that a set of clusters (feature vocabulary) is created for describing trajectories. A cluster dissimilarity measure based on a weighted fusion of features facilitates the recognition of trajectory classes in an incremental way. As a result, an unsupervised method for encoding observed motion information and identifying trajectory patterns is proposed in this article. The method is evaluated with real and simulated data. Additionally' comparisons with previous works show the benefits of our method when encoding and identifying motion patterns in video sequences.

In recent years several architectures have been proposed to learn embodied agents complex self-awareness models. In this paper, dynamic incremental self-awareness (SA) models are proposed that allow experiences done by an agent to be modeled in a hierarchical fashion, starting from more simple situations to more structured ones. Each situation is learned from subsets of private agent perception data as a model capable to predict normal behaviors and detect abnormalities. Hierarchical SA models have been already proposed using low dimensional sensorial inputs. In this work, a hierarchical model is introduced by means of a cross-modal Generative Adversarial Networks (GANs) processing high dimensional visual data. Different levels of the GANs are detected in a self-supervised manner using GANs discriminators decision boundaries. Real experiments on semi-autonomous ground vehicles are presented.

We present a method to model and classify trajectory data that come from surveillance videos. Observations of the locations of moving entities are used to estimate their expected velocity in the scene. Such estimation is performed by a Gaussian process regression that enables to approximate probabilistically the expected velocity of entities given some observed evidence in the scene. Subsequently, regions where estimations have high certainty are decomposed into zones by superpixel segmentation. Each zone represents a region where motions of entities can be explained by a quasilinear dynamical model. We evaluated the proposed method with two datasets and confirmed its reliability for characterizing and classifying trajectories.

To understand the behavior of moving entities in a given environment, one should be capable of predicting their motion, that is, to model their dynamics. In a setting where different behaviors can arise, one can assume that each of them corresponds to different motivational states of observed entities. Here, those motivations are understood as goal positions or spots where entities seek to arrive. To build prediction models based on that idea, we present an unsupervised method to estimate motivational spots actively. Additionally, we use the output of such process to refine an adaptive system modeling the dynamics of inferred hidden causes of observed data. The whole method uses deep variational methods, and particularly, the network estimating motivations is trained through dynamic programming. Results show that modeling the dynamics of entities can be better achieved by integrating information about motivational spots. Notably, a network modeling the dynamics converges faster through the incorporation of information about motivations.

This paper presents a method for observational learning in autonomous agents. A formalism based on deep learning implementations of variational methods and Bayesian filtering theory is presented. It is explained how the proposed method is capable of modeling the environment to mimic behaviors in an observed interaction by building internal representations and discovering temporal and causal relations. The method is evaluated in a typical surveillance scenario, i.e., perimeter monitoring. It is shown that the vehicle learns how to drive itself by simultaneously observing its surroundings and the actions taken by a human driver for a given task. That is achieved by embedding knowledge regarding perception-action couplings in dynamic representational states used to produce action flows. Thereby, representations link sensory data to control signals. In particular, the representational states associate visual features to stable action concepts such as turning or going straight.

With the growing availability of wearable technology, video recording devices have become so intimately tied to individuals, that they are able to record the movements of users' hands, making hand-based applications one the most explored area in First Person Vision (FPV). In particular, hand pose recognition plays a fundamental role in tasks such as gesture and activity recognition, which in turn represent the base for developing human-machine interfaces or augmented reality applications. In this work we propose a graph-based representation of hands seen from the point of view of the user, obtained through the shape-fitting capability of a modified Instantaneous Topological Map. Spectral analysis of the graph Laplacian allows to arrange eigenvalues in vectors of features, which prove to be discriminative in classifying the considered hand poses.

This paper proposes a method for detecting zones of visual attention based on the motion of agents in a video analytics context. By considering a Hough transform approach, linear flow motions are grouped based on attractive salient zones where they converge. Each group of linear flows is generalized through the whole environment by using a non-parametric stochastic approach that can be used to generate a map that illustrates the effects that each zone exerts on the dynamics of agents. A dataset of walking pedestrians and trajectories generated by a robot that executes a single task in a close environment are used to validate the proposed method.

In this paper we address the abnormality detection problem in crowded scenes. We propose to use Generative Adversarial Nets (GANs), which are trained using normal frames and corresponding optical-flow images in order to learn an internal representation of the scene normality. Since our GANs are trained with only normal data, they are not able to generate abnormal events. At testing time the real data are compared with both the appearance and the motion representations reconstructed by our GANs and abnormal areas are detected by computing local differences. Experimental results on challenging abnormality detection datasets show the superiority of the proposed method compared to the state of the art in both frame-level and pixel-level abnormality detection tasks.

Cognitive radio (CR) is a promising technology for future wireless spectrum allocation to improve the use of licensed bands. However, security challenges faced by cognitive radio technology are still a hot research topic. One of prevailing challenges is the radio frequency jamming attack, where adversaries are able to exploit on-the-fly reconfigurability potentials and learning mechanism of cognitive radios in order to devise and deploy advanced jamming tactics. Jamming attacks can significantly impact the performance of wireless communication systems and lead to significant overheads in terms of retransmission and increment of power consumption. In this context, a novel jammer detection algorithm is proposed using cyclic spectral analysis and artificial neural networks (ANN) for wide-band (WB) cognitive radios. The proposed approach assumes a WB spectrum occupied by various narrow-band (NB) signals, which can be either legitimate or jamming signals. The second order statistics, namely, the spectral correlation function (SCF) and ANN are used to classify each NB signal as a legitimate or jamming signal. The algorithm performance is shown with the help of simulations.

The introduction of cognitive radio enables dynamic spectrum access for higher spectrum utilization, due to its ability of awareness of their environment. However, the introduction of cognitive radio technology brings new challenges to wireless networks security. Due to intelligent nature of the attackers, many of the radio frequency jamming attacks can be stealthy by nature. The stealthy jamming attacks can significantly impact the performance of the wireless communication system and can lead to significant overhead in terms of retransmission and increment of power consumption. This paper presents a new physical layer approach for stealthy jammer detection in wide-band (WB) cognitive radio networks. The proposed algorithm consider a WB consists of multiple narrow-band sub-bands (SB), which can be occupied by licit or jamming signals . The cyclostationary spectral analysis is performed on this WB signal to compute spectral correlation function (SCF). The alpha profile is extracted from the SCF and used as input features to artificial neural network (ANN), which classify each NB signal as a licit signal or a jamming signal. In the end, the performance of the proposed approach is shown with the help of Monte-Carlo simulations under different empirical setups.

A method for online incremental mining of activity patterns from the surveillance video stream is presented in this paper. The framework consists of a learning block in which Dirichlet process mixture model is employed for the incremental clustering of trajectories. Stochastic trajectory pattern models are formed using the Gaussian process regression of the corresponding flow functions. Moreover, a sequential Monte Carlo method based on Rao-Blackwellized particle filter is proposed for tracking and online classification as well as the detection of abnormality during the observation of an object. Experimental results on real surveillance video data are provided to show the performance of the proposed algorithm in different tasks of trajectory clustering, classification, and abnormality detection.

Under a tracking framework, the definition of the target state is the basic step for automatic understanding of dynamic scenes. More specifically, far object tracking raises challenges related to the potentially abrupt size changes of the targets as they approach the sensor. If not handled, size changes can introduce heavy issues in data association and position estimation. This is why adaptability and self-awareness of a tracking module are desirable features. The paradigm of cognitive dynamic systems (CDSs) can provide a framework under which a continuously learning cognitive module can be designed. In particular, CDS theory describes a basic vocabulary of components that can be used as the founding blocks of a module capable to learn behavioral rules from continuous active interactions with the environment. This quality is the fundamental to deal with dynamic situations. In this paper we propose a general CDS-based approach to tracking. We show that such a CDS-inspired design can lead to the self-adaptability of a Bayesian tracker in fusing heterogeneous object features, overcoming size change issues. The experimental results on infrared sequences show how the proposed framework is able to outperform other existing far object tracking methods.

A method for online identification of group of moving objects in the video is proposed in this paper. This method at each frame identifies group of tracked objects with similar local instantaneous motion pattern using spectral clustering on motion similarity graph. Then, the output of the algorithm is used to detect the event of more than two object moving together as required by PETS2015 challenge. The performance of the algorithm is evaluated on the PETS2015 dataset.

It is well known from physiological studies that the level of human attention for adult individuals rapidly decreases after five to twenty minutes [1]. Attention retention for a surveillance operator represents a crucial aspect in Video Surveillance applications and could have a significant impact in identifying relevance, especially in crowded situations. In this field, advanced mechanisms for selection and extraction of saliency information can improve the performances of autonomous video surveillance systems and increase the effectiveness of human operator support. In particular, crowd monitoring represents a central aspect in many practical applications for managing and preventing emergencies due to panic and overcrowding.

Classification of moving objects for video surveillance applications still remains a challenging problem due to the video inherently changing conditions such as lighting or resolution. This paper proposes a new approach for vehicle/pedestrian object classification based on the learning of a static kNN classifier, a dynamic Hidden Markov Model (HMM)-based classifier, and the definition of a fusion rule that combines the two outputs. The main novelty consists in the study of the dynamic aspects of the moving objects by analysing the trajectories of the features followed in the HOG-PCA feature space, instead of the classical trajectory study based on the frame coordinates. The complete hybrid system was tested on the VIRAT database and worked in real time, yielding up to 100% peak accuracy rate in the tested video sequences.

The problem of behavior assessment in video surveillance is approached using trajectory classification. Lagrangian state dynamic is used for probabilistic modeling of trajectory patterns and an off-line parameter learning method for the model is proposed. For classification purpose, an on-line sequential maximum a posterior trajectory classifier is introduced based on particle filter. Finally, the performance of this method is evaluated using a traffic video data set.

Classifying frames, or parts of them, is a common way of carrying out detection tasks in computer vision. However, frame by frame classification suffers from sudden significant variations in image texture, colour and luminosity, resulting in noise in the extracted features and consequently in the decisions taken. Support Vector Machines have been widely validated as powerful tools for frame by frame detection of non-separable datasets, but are extremely sensitive to these variations between adjacent frames, creating as consequence sudden flickering in the classification results. This work proposes a Dynamic Bayesian Network to smooth the classification results of Support Vector Machines (SVM) in detection tasks. The method is evaluated in First Person Vision (FPV) videos, where a SVM is used to decide whether or not the user's hands are in his field of view.

In this work, we propose a strategy for optimizing a superpixel algorithm for video signals, in order to get closer to real time performances which are on the one hand needed for egocentric vision applications and on the other must be bearable by wearable technologies. Instead of applying the algorithm frame by frame, we propose a technique inspired to Bayesian filtering and to video coding which allows to re-initialize superpixels using the information from the previous frame. This results in faster convergence and demonstrates how performances improve with respect to the standard application of the algorithm from scratch at each frame.

This article evaluates convergence rates of binary majority consensus algorithms in networks with different types of disturbances and studies the potential capacity of randomization to foster convergence. Simulation results show that (a) additive noise, topology randomness, and stochastic message loss may improve the convergence rate; (b) presence of faulty nodes degrades the convergence rate; and (c) explicit randomization of consensus algorithms can be exploited to improve the convergence rate. Watts-Strogatz and Waxman graphs are used as underlying network topologies. A consensus algorithm is proposed that exchanges state information with dynamically randomly selected neighbors and, through this randomization, achieves almost sure convergence in some scenarios.

Interaction analysis of ships mooring and maneuvering in harbors is pursued in this paper by using Bayesian probabilistic models. A number of ship-to-ship interactions are deduced from the navigation rules in port areas, and then used to train different Event-based Dynamic Bayesian Networks (E-DBNs). When data of two interacting ships are injected into the network, inference is performed in order to verify if the interaction between the vessels is known or not, and in the latter case actions to preserve the port safety can be taken. Results are drawn in the final part of the paper by including into the networks data provided by a simulator of realistic trajectories relative to an existing port.

Extraction and representation of relevant information from large-scale surveillance systems constitute fundamental processes for allowing automatic interpretation of complex scenes. In particular, when the amount of information increases (i.e., due to a larger number of monitored areas), attention focusing techniques are needed to highlight most relevant parts within the overall acquired data. When wide area surveillance systems are considered, one of the major problems in event detections is the reconstruction of the scene as a whole, from spatially limited observations. In this paper, a novel representation technique for sparse information, based on information theory, is presented. Self Organizing Maps (SOMs) have been used for classifying and correlating observed sparse data time series. By means ofInformation Bottlenecktheory, it is possible to determine the optimal data representation in the SOM-space as a tradeoff between the signal reconstruction capabilities and the original data statistical similarities preservation. Proposed experiments show how the so calledinformation bottleneck-basedSOMselectionfor knowledge modelling, can be applied to the field of crowd monitoring for people density map estimation and event detection. Results are presented on synthetic and real video sequences.

On-the-fly reconfigurability capabilities and learning prospectives of Cognitive Radios inherently bring a set of new security issues. One of them is intelligent radio frequency jamming, where adversary is able to deploy advanced jamming strategies to degrade performance of the communication system. In this paper, we observe the jamming/antijamming problem from a game-theoretical perspective. A game with incomplete information on opponent's payoff and strategy is modelled as a Markov Decision Process (MDP). A variant of fictitious play learning algorithm is deployed to find optimal strategies in terms of combination of channel hopping and power alteration anti-jamming schemes.

Recently, a Bayesian estimator with a hybrid update was developed [1], based on a mathematical formulation of sampling. Such an Event Based State Estimator (EBSE) allows for a stable synchronous state estimate, relying on asynchronous measurements. Usefulness of such a filter comes with its approximate analytic formulation, which is attainable given a send-on-delta sampling strategy. We argue that such a formulation can be extended to cope with a failing detector in case the filter is used for tracking. The basic idea is to approach the issue as a package loss problem, where a missed target is assimilated to a lost package. More in detail, we propose that this approach can be exploited in video tracking, where faulty detectors are commonplace. We show how tracking performance with a poor pedestrian detector, failing to recognize its target, can improve with respect to standard Kalman filter.

Detection of dim moving point targets in cluttered background can have a great impact on the tracking performances. This may become a crucial problem, especially in low-SNR environments, where target characteristics are highly susceptible to corruption. In this paper, an extended target model, namely Interacting Multiple Model (IMM), applied to Track-Before-Detect (TBD) based detection algorithm, for far objects, in infrared (IR) sequences is presented. The approach can automatically adapts the kinematic parameter estimations, such as position and velocity, in accordance with the predictions as dimensions of the target change. A sub-par sensor can cause tracking problems. In particular, for a single object, noisy observations (i.e. fragmented measures) could be associated to different tracks. In order to avoid this problem, presented framework introduces a cooperative mechanism between Joint Probabilistic Data Association Filter (JPDAF) and IMM. The experimental results on real and simulated sequences demonstrate effectiveness of the proposed approach.

Detection of emotions of a crowd is a new research area, which has never, to our knowledge, been accounted for research in previous literature. A bio-inspired model for representation of emotional patterns in crowds has been demonstrated. Emotions have been defined as evolving patterns as part of a dynamic pattern of events. This model has been developed to detect emotions of a crowd based on the knowledge from a learned context, psychology and experience of people in crowd management. The emotions of multiple people making a crowd in any surveillance environment are estimated by sensors signals such as a camera and are being tracked and their behavior is modeled using bio-inspired dynamic model. The behavior changes correspond to changes in emotions. The proposed algorithm involves the probabilistic signal processing modelling techniques for analysis of different types of behavior, interaction detection and estimation of emotions. The emotions are recognized by the expectation of temporal occurrences of causal events modeled by Gaussian mixture model. The model has been evaluated using the simulated behavioral model of a crowd.

This paper investigates distributed consensus for density classification in asynchronous random networks with faulty nodes. We compare four different models of faulty behavior under randomized topology. Using computer simulations, we show that (a) faulty nodes' impact depends on their location and (b) faulty nodes with persistent failures inhibit consensus stronger than commonly-used Byzantine faulty nodes with random failures. We also show that (c) randomization by Byzantine faulty nodes can be strongly beneficial for binary consensus and (d) topology randomization can increase robustness towards faulty node behavior.

In most recent Intelligent Video Surveillance systems, mechanisms used to support human decisions are integrated in cognitive artificial processes. Large scale video surveillance networks must be able to analyse a huge amount of information. In this context, a cognitive perception mechanism integrate in an intelligent system could help an operator for focusing his attention on relevant aspects of the environment ignoring other parts. This paper presents a bio-inspired algorithm called Selective Attention Automatic Focus (S2AF), as a part of more complex Cognitive Dynamic Surveillance System (CDSS) for crowd monitoring. The main objective of the proposed method is to extract relevant information needed for crowd monitoring directly from the environmental observations. Experimental results are provided by means of a 3D crowd simulator; they show how by the proposed attention focus method is able to detect densely populated areas.

Main drawbacks in single-camera multi-target visual tracking can be partially removed by increasing the amount of information gathered on the scene, i.e. by adding cameras. By adopting such a multi-camera approach, multiple sensors cooperate for overall scene understanding. However, new issues arise such as data association and data fusion. This work addresses the issue of evaluating the performance of a multi-camera tracking algorithm based on Rao-Blackwellized Monte Carlo data association (RBMCDA) on real data. For this purpose, a new metric based on three performance indexes is developed.

This work addresses the problem of people counting in crowded situations, such as urban environments, in computer vision. As crowding density increases in a scene, it might become impossible to count people as single individuals: a global group-based approach is then preferable and in fact often necessary. A simple method for estimating the count of people in such tight crowds is here proposed, relying on accurate camera calibration. A training phase is also needed by the algorithm in order to learn the parameters needed for estimation.

This work addresses the issue of fire and smoke detection in a scene within a video surveillance framework. Detection of fire and smoke pixels is at first achieved by means of a motion detection algorithm. In addition, separation of smoke and fire pixels using colour information (within appropriate spaces, specifically chosen in order to enhance specific chromatic features) is performed. In parallel, a pixel selection based on the dynamics of the area is carried out in order to reduce false detection. The output of the three parallel algorithms are eventually fused by means of a MLP.

Summary form only given. TENCON delegates are invited to attend the following tutorial sessions, free of charge, which will be held at the USC Engineering Audio Visual Room, University of San Carlos - Talamban Campus on November 19, 2012. Tutorial 1: Recent Advances in Robotics and Emerging Opportunities. This tutorial reviews the recent exciting developments in robotics centering on humans and operating in unstructured environments. It covers emerging trends in manufacturing, autonomous mobile robots, social robotics, and security and service applications. Current state of the art techniques to realize fundamental capabilities of robotic systems are concisely explained and challenges for research discussed. Tutorial 2: The Next Generation MIMO-OFDM Systems. This tutorial describes the VLSI implementation of our proposed 4x4 MIMO-OFDM (2.6G bps with 160MHz BW) and SxS MIMO-OFDM (3.0G bps with SOMHz BW) systems. A low-latency and a full-pipelined architecture are employed for all processing blocks to provide the real-time operations on OFDM modulation and MIMO detection. The designed transceiver has been evaluated in the circuit size and power dissipation by using a 90-nm CMOS process. In an FPGA board, the proposed total system has been implemented. For the designed system, the circuit behavior on gate size and power consumption is verified. The communication performance is also evaluated.

In this paper a system is presented able to reproduce the actions of multiple moving objects into a 3D model. A multi-camera surveillance system is used for automatically detect, track and classify the objects. Data fusion from multiple sensors allows to get a more precise estimation of the position of detected moving objects and to solve occlusions problem. These data are then used to automatically place and animate objects avatars in a 3D virtual model of the scene, thus allowing a human operator to remotely visualize the dynamic 3D reconstruction by selecting a arbitrary point of view.

This paper describes automatic video sequences processing techniques for detecting suspect and dangerous situations within public transportations. Proposed surveillance system is able to raise different kind of warnings and alarms on the basis of the particular detected situation. Algorithms used for objects detection and tracking will be described in details and performances will be discussed in relation with alarm conditions that are showed in the sequences that have been made available for this conference. An empty reference image is used for object extraction through image difference. In order to perform background updating a high level module is implemented taking into account the detected objects and their classification tags. The system has been tested on several sequences showing dangerous events due to human behaviors in an underground station.

In video surveillance projects, automatic and real-time event detection solutions are required to guarantee an efficient and cost-effective use of the infrastructure. Many solutions have been proposed to automatically detect a variety of events of interest. However, not all solutions and technologies may satisfy all the requirements of the surveillance scenario. For this reason, performance evaluation of existing event detection solutions becomes an important step in the deployment of video surveillance projects. In this paper, we propose a practical approach that aims at minimizing the ground truth generation problem and the expertise required to evaluate and compare the results by introducing specific requirements of specific event detection scenarios. This approach is believed to be applicable for an initial evaluation of candidate solutions to a specific surveillance scenario before more exhaustive tests in an integrated environment. The proposed method is under evaluation in the framework of the challenge of real-time event detection solutions (CREDS).

In automatic video surveillance systems, full time monitoring represents an important goal to achieve. The aim of the paper is to find a new methodology for video surveillance adaptive parameter regulation. The method presented is based on self organizing maps (SOM) which are used for parameter regulation purposes. Results shown prove how this method permits overall good performances to be achieved for all environments.

This paper exploits an innovative technique to improve performances related to localization, tracking and classification of objects in a video surveillance system. The developed strategy has been applied to the problem of interaction between objects, i.e. well tuned traditional algorithms are able to track and classify objects whenever they enter the scene well-isolated from the other moving objects, but the state-of-the-art techniques fail when an occlusion situation is verified from the beginning. The performances of the developed algorithms have been evaluated on sequences of real images and experimental results have shown the validity of the approach.

A system for face detection in outdoor environments for multisensor video surveillance applications is presented. The system is characterized by a combination of two pan-tilt video cameras, which cooperate in order to track and to characterize moving objects with positioning and biometric informations. The final result of the action of the system is the collection of small video shots regarding the face of humans detected in outdoor environments with a robust behavior.

This paper describes a technique for tracking single objects moving within the guarded scene during dynamic occlusion situations. The processing modules used for object detection and tracking will be shown in detail and the performances of the algorithm discussed. The proposed approach uses an empty reference image for object extraction through image difference; the reference frame is updated continuously by a background updating module taking into account the detected objects. The tracking module is responsible for objects labeling being able to preserve objects identity even when an overlapping occurs on the image plane between different objects. A shape matching technique is used that is based on a linear Kalman filter. The system has been tested on several outdoor sequences showing dynamic occlusions among objects in order to show the validity of the approach.

This paper presents a system for automatic video surveillance applications. The system has been designed to monitor outdoor environments such as car parks or streets, providing the human operator with a symbolic description of the scene. The final task of the architecture is to automatically provide alarms when specific events of interest are detected. In this way the level of automation of the system is increased as well as overall performances. One of the main drawbacks of traditional video surveillance systems lies in the alarm generation. This task has to be visually performed by the human operators with intrinsic limitation. The possibility of having this process automated is here described within the design of an architecture capable of acquiring, processing and successfully storing data coming from one or more sensors.

Increased communication capabilities and automatic scene understanding allow human operators to simultaneously monitor multiple environments. Due to the amount of data to be processed in new surveillance systems, the human operator must be helped by automatic processing tools in the work of inspecting video sequences. In this paper, a novel approach allowing layered content-based retrieval of video-event shots referring to potentially interesting situations is presented. Interpretation of events is used for defining new video-event shot detection and indexing criteria. Interesting events refer to potentially dangerous situations: abandoned objects and predefined human events are considered in this paper. Video-event shot detection and indexing capabilities are used for online and offline content-based retrieval of scenes to be detected.

An image stabilization algorithm is presented that is specifically oriented toward video-surveillance applications. The proposed approach is based on a novel motion-compensation method that is an adaptation of a well-known image-stabilization algorithm for visualization in video-surveillance applications. In particular, the illustrated methods take into account the specificity of typical video-surveillance applications, where objects moving in a scene often cover a large part of an image thus causing the failure of classic image-stabilization techniques. Evaluation methods for image stabilization algorithms are discussed.

In the past few years, the development of complex surveillance systems has captured the interest of both the research and industrial worlds. Strong and challenging requirements of modern society are involved in this problem, which aims to increase safety and security in several application domains such as transport, tourism, home and bank security, military applications, etc. At the same time, fast improvements in microelectronics, telecommunications, and computer science make it necessary to consider new perspectives in this field. The main objective of this paper is to investigate, discuss, and evaluate the impact of distributed processing and new communication techniques on multimedia surveillance systems, which represent the so-called third-generation surveillance systems (3 GSSs). In particular, aspects related to the distribution of intelligence among multiple-processing and wide-bandwidth resources are discussed in detail. It is shown how distribution of intelligence can be obtained by a hierarchical architecture that partitions, in a dynamic way, the main logical processing tasks (i.e., representation, recognition, and communication) performed in a 3 GSS physical architecture made up of intelligent cameras, hubs, and central control rooms. The advantages of this solution are pointed out in terms of 1) increased flexibility and reconfigurability and 2) optimal allocation of available processing and bandwidth resources. Finally, a case study is analyzed that allows one to gain a deeper insight into a distributed surveillance system.

This paper proposes a video-surveillance system based on a mobile camera. In particular the developed system creates (during the off-line phase) a panoramic multilayer background image allowing one to use common change detection algorithms to search for a change detection binary image. Different approaches to get the change detection images are presented. The performances of the implemented algorithms are presented by using ROC curves.

A new method for modeling non-rigid objects in a multiple-camera guarded environment is proposed. The statistical model of the shape of a non-rigid object takes into account the correlations between different points of view. The model is automatically generated by processing a typical training set for the considered shape by a principal component analysis algorithm. Snakes and dynamic contours are used to describe the shape of the non-rigid object.