This paper proposes a hierarchical trajectory planning framework for UAVs operating under adversarial jamming conditions. Leveraging Bayesian Active Inference, the approach combines expert-generated demonstrations with probabilistic generative modeling to encode high-level symbolic planning, low-level motion policies, and wireless signal feedback. During deployment, the UAV performs online inference to anticipate interference, localize jammers, and adapt its trajectory accordingly—without prior knowledge of jammer locations. Simulation results demonstrate that the proposed method achieves near-expert performance, significantly reducing communication interference and mission cost compared to model-free reinforcement learning baselines, while maintaining robust generalization in dynamic environments.

In this paper, we propose a novel method for abnormality detection in Unmanned Surface Vehicles (USVs) based on a Multi-Modal Bayesian generative model to enhance safety and monitoring.During the training phase, we use a Null Force Filter and an unsupervised clustering algorithm on multimodal data collected from Global Positioning System (GPS) and motor current sensors.In the testing phase, we use a Coupled Modified Markov Jump Particle Filter (CM-MJPF) to infer the GPS position and motor current of the USV, as well as to detect abnormalities in both modalities.Due to the coupled methodology, the system is able to learn the statistical similarity between the evolving GPS and motor current data.As a result, the causality of defects is inherently captured within the dynamical inference, making the proposed approach explainable.

Unmanned aerial vehicles (UAVs) used as aerial base stations show significant promise for future wireless communication systems. This paper explores using a UAV as an autonomous agent, navigating over multiple hotspots to serve ground users (GUs) and maximize data transmission rates through strategic trajectory design. Existing interpretability methods often prove insufficient in providing comprehensive insights and generating logical, sequential decisions. In this paper, we propose an explainable reinforcement learning framework designed to produce interpretable and verifiable agent policies. Our method starts with an expert optimizer to solve training examples, enabling the learning agent (UAV) to analyse the solutions. Furthermore, we employ inverse reinforcement learning for data-driven reward function estimation. Additionally, we use a probabilistic Q-table function to understand and explain the actions executed by the expert across diverse environmental contexts, allowing the learning agent to produce interpretable policies that meet reasonable performance goals and easily transferred to unseen environments. Preliminary results indicate that the proposed approach is promising in achieving explainability in RL agents.

Autonomous vehicles (AVs) must be able to locate themselves accurately in dynamic environments in order to be able to navigate safely. In this paper, we present a robust LiDAR-based framework to improve the localization of AVs based on the classification of static and dynamic tracks. The approach leverages static tracks previously classified as reliable landmarks. Using Growing Neural Gas, Joint Probabilistic Data Association, and Unmotivated Kalman Filter algorithms, interaction dictionaries and vocabularies are generated. These serve as inputs for a Markov Jump Particle Filter (MJPF), which enables the accurate estimation of the ego-vehicle trajectory. Testing of the proposed framework with real-world LiDAR data demonstrates that it is capable of providing highly accurate localization results without the use of odometry, making it adaptable for use in environments without GPS. AV situational awareness and navigation performance are enhanced by using static tracks as robust references.

Experience sharing is a fundamental learning capability of human nature. Imagine that you need to pass through a congested road in the shortest time possible. You would try to remember how you previously handle such situations and optimize the route by fusing this knowledge with previous similar experiences. In addition, one can integrate these experiences with other information sources that suggest solutions. Transferring this type of intelligence to autonomous navigation systems implies the ability to first represent experiences through discriminative and generative models, and then create new optimal models, by fusing new and existing scenarios that the agent encounters. We propose a meta-level experience-sharing and learning method for a Bayesian autonomous agent. We introduce a joint state estimation method for discrete and continuous random variables, using Hierarchical Dynamic Bayesian Networks. In the proposed system, meta-level knowledge is learned and represented from lower dimensional odometry data, by observing behaviors from a third-person viewpoint. The learned knowledge is then transformed into a first-person agent viewpoint to perform such tasks online. Multiple learned predictive models and new experiences are fused into a single and more robust model during the online stage. Through the principle of free energy minimization, the learned Hierarchical Dynamic Bayesian Networks are used to coherently estimate discrete and continuous state variables for hierarchical-level anomaly detection and inference problems. Our results demonstrate that the proposed method optimizes the number of models by fusing similar experiences managed by the agent, while making anomaly detection and incremental learning steps more robust.

Learning from experience is a fundamental capability of intelligent agents. Autonomous systems rely on sensors that provide data about the environment and internal situations to their perception systems for learning and inference mechanisms. These systems can also learn Self-Aware and Situation-Aware generative modules from these data to localize themselves and interact with the environment. In this paper, we propose a self-aware cognitive architecture capable to perform tasks where the interactions between the self-state of an agent and the surrounding environment are explicitly and dynamically represented. We specifically develop a Deep Learning (DL) based Self-Aware interaction model, empowered by learning from Multi-Modal Perception (MMP) and World Models using multi-sensory data in a novel Multi-Agent Self-Awareness Architecture (MASAA). Two sub-modules are developed, the Situation Model (SM) and the First-Person model (FPM), that address different and interrelated aspects of the World Model (WM). The MMP model, instead, aims at learning the mapping of different sensory perceptions into Exteroceptive (EI) and Proprioceptive (PI) latent information. The WM then uses the learned MMP model as experience to predict dynamic self-behaviors and interaction patterns within the experienced environment. WM and MMP Models are learned in a data-driven way, starting from the lower-dimensional odometry data used to guide the learning of higher-dimensional video data, thus generating coupled Generalized State Hierarchical Dynamic Bayesian Networks (GS-HDBNs). We test our model on KITTI, CARLA, and iCab datasets, achieving high performance and a low average localization error (RMSE) of 2.897%, when considering two interacting agents.

Generalizability and interpretability are common terminologies that can be found in today’s machine learning algorithm design. Generalizability requires a clear understanding of one’s own action (self-awareness) and a robust interaction with the environment (situation awareness). Many current studies are devoted in developing an algorithm that is more robust in generalizing unseen situations while explaining self-action. However, such algorithms are complex and are not yet fully developed to be used in production. Intelligent transportation systems like self-driving cars are one of the emerging technologies that need generalizability and explainability in anomalous conditions. We propose to enhance generalizability and interpretability of a self-driving car model by introducing a novel methodology that fuses multi-sensorial data from proprioceptive and exteroceptive sensors of an agent, coupled in a Hierarchical Dynamic Bayesian Network model, in an Active Inference framework. The developed model has three stages: 1) a lower dimensional unsupervised learning stage, considering odometry and action modalities, carried out by first applying Null Force Filtering and then by applying modified GNG clustering algorithms; 2) a self-supervised higher-dimensional video modality learning stage assisted by the learned odometry vocabularies; and 3) an online model-based active learning in continuous and discrete state spaces, and action spaces, in the Active Inference framework. The developed system is tested using the CARLA simulator environment for localizing interacting agents, and exhibits low error compared to state-of-the-art methods.

Intelligent transportation systems (ITSs) provide a paradigm change in perceiving and interacting with transportation networks, leading to enhanced levels of safety, sustainability, and efficiency. Vehicular-to-everything (V2X) communication is the core component in the ITSs. The proprioceptive and exteroceptive sensors allow these vehicles to be aware of the surrounding environment and respond to emergencies by utilizing their abilities to reach a high level of self-awareness. In this paper, we propose a self-awareness approach to learn a generative dynamic Bayesian network (G-DBN) from the real-time LiDAR perception. Without reducing the dimensionality, we perform offline training and online testing phases on the three-dimensional (3D) point clouds. In the offline training phase, initially, the raw point clouds are preprocessed using a joint probabilistic data association filter (JPDAF) to obtain the 3D tracks of the multiple vehicles in space. Then, we perform an unsupervised clustering on all the generalized states (GSs) containing positions and velocities (a 6D vector) by considering the growing neural gas (GNG) technique, thus achieving a trained model from the 3D LiDAR point clouds. In the online testing phase, the high-dimensional Markov jump particle filter (HD-MJPF) utilizes the G-DBN’s probabilistic information to predict the positions of multiple vehicles and to detect the abnormalities at the discrete and continuous levels in normal and abnormal scenarios. Our proposed approach is useful for learning high-dimensional generative models and provides a way to meet the current curse of dimensionality challenges, that machine learning models are suffering.

This paper presents a novel self-supervised path-planning method for UAV-aided networks. First, we employed an optimizer to solve training examples offline and then used the resulting solutions as demonstrations from which the UAV can learn the world model to understand the environment and implicitly discover the optimizer’s policy. UAV equipped with the world model can make real-time autonomous decisions and engage in online planning using active inference. During planning, UAV can score different policies based on the expected surprise, allowing it to choose among alternative futures. Additionally, UAV can anticipate the outcomes of its actions using the world model and assess the expected surprise in a self-supervised manner. Our method enables quicker adaptation to new situations and better performance than traditional RL, leading to broader generalizability.

The following paper proposes a novel Vehicle-to-Everything (V2X) network abnormality detection scheme based on Bayesian generative models for enhanced network self-awareness functionality at the Base station (BS). In the learning phase, multimodal data signals contrived by the vehicles' integrated and sensing module are imbued into data-driven Gen-eralized Dynamic Bayesian network (GDBN) models. Following that, during the testing phase, an Interactive Modified Markov Jump Particle filter (IM-MJPF) is utilized to forecast forthcoming network states and vehicle trajectories by leveraging the assimilated semantics embedded in the coupled multi-GDBNs. This approach involves learning statistically correlated association between evolving trajectories and network communication links. Security and surveillance of Internet of Vehicles (IOVs) links are performed online with high detection probabilities by matching predicted with observed network connectivity maps (graphs).

Self-aware computing systems enable computing systems to reflect on their actions and behavior. This becomes even more relevant in Cyber-Physical Systems where computing systems have to control and interact with elements in the real world. This paper reports on recent advances made in computational self-awareness for cyber-physical systems.

Active inference is a probabilistic framework for modeling intelligent agent behaviours, which drives by the principle of minimizing free energy. In this paper, we integrate the imitation learning method with active inference to minimize the expected free energy under the supervision of an expert model. The proposed approach affords explainable decision-making as a combination of self-information and novelty-seeking or exploratory behavior in a hierarchical generative model. A lane-changing driving scenario is demonstrated to verify the efficiency of the proposed framework that outperforms conventional Reinforcement learning methods.

Vehicle-to-everything (V2X) communication is expected to be a prominent component of the sixth generation (6G) to accomplish intelligent transportation systems (ITS). Autonomous vehicles relying only on onboard sensors cannot bypass the limitations of safety and reliability. Thus, integrated sensing and communication is proposed as an effective way to achieve high situational- and self-awareness levels, enabling V2X to perceive the physical world and adjust its behaviour to emergencies. Secure navigation through the Global Positioning System (GPS) is essential in ITS for safe operation. Nevertheless, due to the lack of encryption and authentication mechanisms of civil GPS receivers, spoofers can easily replicate satellite signals by launching GPS spoofing attacks to deceive the vehicle and manipulate navigation data. In addition, due to its shared nature, V2X links are prone to jamming attacks which might endanger vehicular safety. This paper proposes a method to jointly detect GPS spoofing and jamming attacks in a V2X network. Simulation results demonstrate that the proposed method can detect spoofers and jammers with high detection probabilities.

The combination of different sensory information to predict upcoming situations is an innate capability of intelligent beings. Consequently, various studies in the Artificial Intelligence field are currently being conducted to transfer this ability to artificial systems. Autonomous vehicles can particularly benefit from the combination of multi-modal information from the different sensors of the agent. This paper proposes a method for video-frame prediction that leverages odometric data. It can then serve as a basis for anomaly detection. A Dynamic Bayesian Network framework is adopted, combined with the use of Deep Learning methods to learn an appropriate latent space. First, a Markov Jump Particle Filter is built over the odometric data. This odometry model comprises a set of clusters. As a second step, the video model is learned. It is composed of a Kalman Variational Autoencoder modified to leverage the odometry clusters for focusing its learning attention on features related to the dynamic tasks that the vehicle is performing. We call the obtained overall model Cluster-Guided Kalman Variational Autoencoder. Evaluation is conducted using data from a car moving in a closed environment and leveraging a part of the University of Alcalá DriveSet dataset, where several drivers move in a normal and drowsy way along a secondary road.

This paper proposes a multi-modal situational inter-action model for collaborative agents by fusing multi-sensorial information in a Multi-Agent Hierarchical Dynamic Bayesian Network (MAH-DBN) framework. The proposed model is learned in a data-driven methodology to estimate the states of interacting agents only from video sequences. This can be regarded as a two-fold methodology for improving visual-based localization and interaction between autonomous agents. In the learning stage, the odometry model is used to drive the video learning model for a robust localization and interaction modeling. During the testing phase, the learned Multi-Agent Hierarchical DBN (MAH-DBN) model is used for the localization of collaborative agents only from video sequences by proposing an inference method called Multi-Agent Coupled Markov Jump Particle Filter (MAC-MJPF).

This paper proposes a method to localize a moving agent - such as an autonomous surveillance vehicle - inside a known environment using First Person Viewpoint video data. Anomalies w.r.t. expected vehicle motion and image content are extracted to guide the localization, signal when the localization results are not trustworthy and explain the reason for the failure. During the training phase, a Dynamic Bayesian Network model is learned, which couples positional and video data. To learn it, clustering is performed on the odometry data, and a modified Kalman Variational Autoencoder is built over the video data. During the testing phase, a Coupled Markov Jump Particle Filter leverages the learned Dynamic Bayesian Network to extract anomalies and to estimate the vehicle’s position, given only camera data. The proposed method is evaluated on two real-world datasets of a vehicle performing perimeter monitoring of a closed environment and of a shopping cart moving in a supermarket.

AI-empowered Cognitive Radio (i.e., AI-enabled radios) is a paradigm shift to achieve the highest level of Self-Awareness in future wireless communications. This work proposes a joint automatic modulation conversion and classification (AMCC) framework, which allows an AI-enabled wireless node to predict signals' dynamics of different modulation schemes and explain how it can be transported (converted) with minimal effort and forwarded with higher spectral efficiency. To achieve this goal, we propose a Generalized Filtering framework integrated by Transport Planning to learn the way of converting low-order modulations to high-order modulations, which has also been validated by performing the automatic modulation classification. Simulation results demonstrate the effective performance of our novel framework on converting and classifying multiple modulation formats.

This paper proposes an algorithm to model and process streams of LiDAR data under an autonomous vehicle framework. LiDAR is assumed to be an exteroceptive sensor that allows the vehicle to have dynamic 3D scene perception of its surroundings. We employ an encoder-decoder architecture based on 3D-Convolutional layers called 3D Convolution Encoder-Decoder (3D-CED), together with a transfer learning strategy to extract a set of features from point clouds, which are relevant in the context of autonomous driving. The resulting features allow to make inferences of the future point cloud data and detect multiple abstraction level anomalies in controlled scenarios by utilizing a probabilistic switching dynamic model called High Dimensional Markov Jump Particle Filter (HD-MJPF). Moreover, a comparison is provided between piecewise linear, piecewise nonlinear, and nonlinear predictive models for anomaly detection at multiple abstraction levels. Our approach is evaluated with data collected from the LiDAR sensors of the autonomous vehicle while performing certain tasks in a controlled environment.

Anomaly detection constitutes a fundamental step in developing self-aware autonomous agents capable of continuously learning from new situations, as it enables to distinguish novel experiences from already encountered ones. This paper combines Dynamic Bayesian Networks (DBNs) and Neural Networks (NNs) and proposes a method for detecting anomalies in video data at different abstraction levels. We use a Variational Autoencoder (VAE) to reduce the dimensionality of video frames, and Optical Flows between subsequent images, generating a latent space that captures both visual and dynamical information and that is comparable to low-dimensional sensory data (e.g., positioning, steering angle). An Adapted Markov Jump Particle Filter is employed to predict the following frames and detect anomalies in video data. Our method’s evaluation is executed using different video data from a semi-autonomous vehicle performing different tasks in a closed environment. Tests on benchmark anomaly detection datasets have additionally been conducted.

This paper proposes a method for performing future-frame prediction and anomaly detection on video data in a multi-modal framework based on Dynamic Bayesian Networks (DBNs). In particular, odometry data and video data from a moving vehicle are fused. A Markov Jump Particle Filter (MJPF) is learned on odometry data, and its features are used to aid the learning of a Kalman Variational Autoencoder (KVAE) on video data. Consequently, anomaly detection can be performed on video data using the learned model. We evaluate the proposed method using multi-modal data from a vehicle performing different tasks in a closed environment.

A novel approach is proposed for multimodal collective awareness (CA) of multiple networked intelligent agents. Each agent is here considered as an Internet-of-Things (IoT) node equipped with machine learning capabilities; CA aims to provide the network with updated causal knowledge of the state of execution of actions of each node performing a joint task, with particular attention to anomalies that can arise. Data-driven dynamic Bayesian models learned from multisensory data recorded during the normal realization of a joint task (agent network experience) are used for distributed state estimation of agents and detection of abnormalities. A set of switching dynamic Bayesian network (DBN) models collectively learned in a training phase, each related to particular sensorial modality, is used to allow each agent in the network to perform synchronous estimation of possible abnormalities occurring when a new task of the same type is jointly performed. Collective DBN (CDBN) learning is performed by unsupervised clustering of generalized errors (GEs) obtained from a starting generalized model. A growing neural gas (GNG) algorithm is used as a basis to learn the discrete switching variables at the semantic level. Conditional probabilities linking nodes in the CDBN models are estimated using obtained clusters. CDBN models are associated with a Bayesian inference method, namely, distributed Markov jump particle filter (D-MJPF), employed for joint state estimation and abnormality detection. The effects of networking protocols and of communications in the estimation of state and abnormalities are analyzed. Performance is evaluated by using a small network of two autonomous vehicles performing joint navigation tasks in a controlled environment. In the proposed method, first the sharing of observations is considered in ideal condition, and then the effects of a wireless communication channel have been analyzed for the collective abnormality estimation of the agents. Rician wireless channel and the usage of two protocols (i.e., IEEE 802.11p and IEEE 802.15.4) along with different channel conditions are considered as well.

The technology for autonomous vehicles is close to replacing human drivers by artificial systems endowed with high-level decision-making capabilities. In this regard, systems must learn about the usual vehicle's behavior to predict imminent difficulties before they happen. An autonomous agent should be capable of continuously interacting with multi-modal dynamic environments while learning unseen novel concepts. Such environments are not often available to train the agent on it, so the agent should have an understanding of its own capacities and limitations. This understanding is usually called self-awareness. This paper proposes a multi-modal self-awareness modeling of signals coming from different sources. This paper shows how different machine learning techniques can be used under a generic framework to learn single modality models by using Dynamic Bayesian Networks. In the presented case, a probabilistic switching model and a bank of generative adversarial networks are employed to model a vehicle's positional and visual information respectively. Our results include experiments performed on a real vehicle, highlighting the potentiality of the proposed approach at detecting abnormalities in real scenarios.

Unmanned Aerial Vehicles (UAVs) attracted both industry and research community owing to their fascinating features like mobility, deployment flexibility and strong Line of Sight (LoS) links. The integration of Cognitive Radio (CR) can greatly help UAVs to overcome several issues especially spectrum scarcity. However, the dynamic radio environment in CR and the strong dependence of safe communications from LoS channels integrity in UAV communications make the Cognitive- UAV-Radio vulnerable to jamming attacks. This work aims to study the integration of CR and UAVs introducing a Self- Awareness (SA) framework from the physical layer security perspective. Under the SA framework, a Dynamic Bayesian Network (DBN) model is proposed as a representation of the radio environment and a modified Markov Jump Particle Filter (MJPF) is employed for prediction and state estimation purposes. A novel jammer detection framework is proposed that allows the UAV to perform abnormality evaluation at different hierarchical levels. The jammer is shown to be located effectively in both time and frequency domains. Experimental results show the effectiveness of the proposed framework in terms of detection probability and accuracy.

This paper proposes a method for performing continual learning of predictive models that facilitate the inference of future frames in video sequences. For a first given experience, an initial Variational Autoencoder, together with a set of fully connected neural networks are utilized to respectively learn the appearance of video frames and their dynamics at the latent space level. By employing an adapted Markov Jump Particle Filter, the proposed method recognizes new situations and integrates them as predictive models avoiding catastrophic forgetting of previously learned tasks. For evaluating the proposed method, this article uses video sequences from a vehicle that performs different tasks in a controlled environment.

Cellular connectivity for a massive number of Unmanned Aerial Vehicles (UAVs) will overcrowd the radio spectrum and cause spectrum scarcity. Incorporating Cognitive Radio (CR) with UAVs (Cognitive-UAV-Radios) has been proposed to overcome such an issue. However, the broadcasting nature of CR and the dominant line-of-sight links of UAV makes the Cognitive-UAV-Radios susceptible to jamming attacks. In this paper, we propose a framework to detect smart jammer, which locates and attacks the UAV commands with low Jamming-to-Signal-Power-Ratio (JSR). Smart jammer is more challenging than the types of jammers that always require high power values. Our work focuses on learning a Dynamic Bayesian Network (DBN) to model and analyze the signals' behaviour statistically. A Markov Jump Particle Filter (MJPF) is employed to perform predictions and consequently detect jamming signals. The results are satisfactory in terms of detection probability and false alarm rate that outperform the conventional Energy Detector approach.

Millimeter Wave (mmWave) band can be a solution to serve the vast number of Internet of Things (IoT) and Vehicle to Everything (V2X) devices. In this context, Cognitive Radio (CR) is capable of managing the mmWave spectrum sharing efficiently. However, Cognitive mmWave Radios are vulnerable to malicious users due to the complex dynamic radio environment and the shared access medium. This indicates the necessity to implement techniques able to detect precisely any anomalous behaviour in the spectrum to build secure and efficient radios. In this work, we propose a comparison framework between deep generative models: Conditional Generative Adversarial Network (C-GAN), Auxiliary Classifier Generative Adversarial Network (AC-GAN), and Variational Auto Encoder (VAE) used to detect anomalies inside the dynamic radio spectrum. For the sake of the evaluation, a real mmWave dataset is used, and results show that all of the models achieve high probability in detecting spectrum anomalies. Especially, AC-GAN that outperforms C-GAN and VAE in terms of accuracy and probability of detection.

The advancements in connected and autonomous vehicles in these times demand the availability of tools providing the agents with the capability to be aware and predict their own states and context dynamics. This article presents a novel approach to develop an initial level of collective awareness (CA) in a network of intelligent agents. A specific collective self-awareness functionality is considered, namely, agent-centered detection of abnormal situations present in the environment around any agent in the network. Moreover, the agent should be capable of analyzing how such abnormalities can influence the future actions of each agent. Data-driven dynamic Bayesian network (DBN) models learned from time series of sensory data recorded during the realization of tasks (agent network experiences) are here used for abnormality detection and prediction. A set of DBNs, each related to an agent, is used to allow the agents in the network to reach synchronously aware possible abnormalities occurring when available models are used on a new instance of the task for which DBNs have been learned. A growing neural gas (GNG) algorithm is used to learn the node variables and conditional probabilities linking nodes in the DBN models; a Markov jump particle filter (MJPF) is employed for state estimation and abnormality detection in each agent using learned DBNs as filter parameters. Performance metrics are discussed to asses the algorithm's reliability and accuracy. The impact is also evaluated by the communication channel used by the network to share the data sensed in a distributed way by each agent of the network. The IEEE 802.11p protocol standard has been considered for communication among agents. Performances of the DBN-based abnormality detection models under different channel and source conditions are discussed. The effects of distances among agents and of the delays and packet losses are analyzed in different scenario categories (urban, suburban, and rural). Real data sets are also used acquired by autonomous vehicles performing different tasks in a controlled environment.

The articles in this month’s special issue cover concepts and fundamentals, architectures and techniques, and applications and case studies in the exciting area of self-awareness in autonomous systems.

In a computational context, self-awareness (SA) is a capability of an autonomous system to describe the acquired experience about itself and its surrounding environment with appropriate models and correlate them incrementally with the currently perceived situation to expand its knowledge continuously. This article introduces a bio-inspired framework for generative and descriptive dynamic models that support SA computationally and efficiently. Generative models facilitate predicting future states, while descriptive models enable the selection of the representation that best fits the current observation. Our framework is founded on the analysis and extension of three bio-inspired theories that have studied SA from different viewpoints, and we demonstrate how probabilistic techniques, such as cognitive dynamic Bayesian networks and generalized filtering paradigms, can learn appropriate models from multidimensional proprioceptive and exteroceptive signals acquired by the autonomous system. We discuss essential capabilities for SA and show how our modeling framework supports these capabilities in theory and through a case study where a mobile robot uses multisensorial data to determine its internal and environmental state as well as distinguishing among normal and abnormal behaviors.

This paper proposes a method to detect abnormal motions in real vehicle situations based on trajectory data. Our approach uses a Gaussian process (GP) regression that facilitates to approximate expected vehicle's movements over a whole environment based on sparse observed data. The main contribution of this paper consists in decomposing the GP regression into spatial zones, where quasi-constant velocity models are valid. Such obtained models are employed to build a set of Kalman filters that encode observed vehicle's dynamics. This paper shows how proposed filters enable the online identification of abnormal motions. Detected abnormalities can be modeled and learned incrementally, automatically by intelligent systems. The proposed methodology is tested on real data produced by a vehicle that interacts with pedestrians in a closed environment. Automatic detection of abnormal motions benefits the traffic scene understanding and facilitates to close the gap between human driving and autonomous vehicle awareness.

Self-learning abilities in autonomous systems are essential to improve their situational awareness and detection of normal/abnormal situations. In this work, we propose a graph matching technique for activity detection in autonomous agents by using the Gromov-Wasserstein framework. A clustering approach is used to discretise continuous agents' states related to a specific task into a set of nodes with similar objectives. Additionally, a probabilistic transition matrix between nodes is used as edges weights to build a graph. In this paper, we extract an abnormal area based on a sub-graph that encodes the differences between coupled of activities. Such sub-graph is obtained by applying a threshold on the optimal transport matrix, which is obtained through the graph matching procedure. The obtained results are evaluated through experiments performed by a robot in a simulated environment and by a real autonomous vehicle moving within a University Campus.

In autonomous systems, self-awareness capabilities are useful to allow artificial agents to detect abnormal situations based on previous experiences. This paper presents a method that facilitates the incremental learning of new models by an agent. Available learned models can dynamically generate probabilistic predictions as well as evaluate their mismatch from current observations. Observed mismatches are grouped through an unsupervised learning strategy into different classes, each of them corresponding to a dynamic model in a given region of the state space. Such clusters define switching Dynamic Bayesian Networks (DBNs) employed for predicting future instances and detect anomalies. Inferences generated by several DBNs that use different sensorial data are compared quantitatively. For testing the proposed approach, it is considered the multi-sensorial data generated by a robot performing various tasks in a controlled environment and a real autonomous vehicle moving at a University Campus.

We propose a probabilistic method to track and interpret the interactions of moving objects. The proposed method is based on the analysis of location data from different moving objects that modify their dynamics according to rules of interactions, namely attractive and repulsive forces governing objects' motions in a scene. Our method uses a Bayesian structure to identify key elements of the interplay rules and facilitates the prediction of objects' dynamics as an interacting system. Such a prediction facilitates the detection of abnormalities by identifying unseen interaction effects in the scene.

Technologies such as cognitive radio and dynamic spectrum access rely on spectrum sensing which provides wireless devices with information about the radio spectrum in the surrounding environment. One of the main challenges in wireless communications is the interference caused by malicious users on the shared spectrum. In this manuscript, an artificial intelligence enabled cognitive radio framework is proposed at system‐level as part of a cyclic spectrum intelligence algorithm for interference mitigation in wideband radios. It exploits the cyclostationary feature of signals to differentiate users with different modulation schemes and an artificial neural network as classifier to detect potential malicious users. A dataset consisting of experimental modulated and dynamic signals is recorded by spectrum measurements with an in‐house software defined radio testbed and then processed. Cyclostationary features are extracted for each detected signal and fed to a neural network classifier as training and testing data in a complex and dynamic scenario. Results highlight a classification rate of in most of cases, even at low transmission power. A comparison with two previous works with hand‐crafted features, which employ an energy detector‐based classifier and a naive Bayes‐based classifier, respectively, is discussed.

We are now living in an era of visual information where data is unceasingly generated and pushed into consumption at astounding rates. A remarkable portion of this sensory input comes in the form of videos streaming from large-scale surveillance infrastructures as well as consumer-grade monitoring systems. The sheer amount of ground-based, aerial and mobile video surveillance data demands fittingly competent, accurate, effective techniques to extract useful cues and provide assistance for detection, prevention, and intervention tasks in traffic, safety, security, defense, forensic, health, biology, ethology, and retail space management applications.

This paper presents, a system capable of detecting unusual activities in crowds from real-world data captured from multiple sensors. The detection is achieved by classifying the distinct movements of people in crowds, and those patterns can be different and can be classified as normal and abnormal activities. Statistical features are extracted from the dataset collected by applying sliding time window operations. A model for classifying movements is trained by using Random Forest technique. The system was tested by using two datasets collected from mobile phones during social events gathering. Results show that mobile data can be used to detect anomalies in crowds as an alternative to video sensors with significant performances. Our approach is the first to detect any unusual behaviour in crowd with non-visual data, which is simple to train and easy to deploy. We also present our dataset for public research as there is no such dataset available to perform experiments on crowds for detecting unusual behaviours.

In this paper, an efficient method for crowd abnormal behavior detection and localization is introduced. Despite the significant improvements of deep-learning-based methods in this field, but still, they are not fully applicable for the real-time applications. We propose a simple yet effective descriptor based on binary tracklets, containing both orientation and magnitude information in a single feature. The results of the proposed method are comparable with deep-based methods while it performs more efficiently. The evaluation of our descriptors on three different datasets yields a promising result in abnormality detection, which is competitive with the state-of-the-art methods.

We present an approach to learn a model to estimate the dynamical states at continuous and discrete inference levels when trajectory information is available. We learn from sparse data a probabilistic switching model that generates trajectories associated with a stationary plan of an agent. The learned generative model is used within a Markov Jump Linear System (MJLSs) to switch among set of space dependent linear filters that analyze new trajectories and detect deviations from the learned model based on internal innovation measurements. We show examples of application of the proposed approach to learn filters for evaluating deviations from a reference human driving task execution that includes static and dynamic obstacle avoidance.

This paper presents a novel approach for learning self-awareness models for autonomous vehicles. Proposed technique is based on the availability of synchronized multi-sensor dynamic data related to different maneuvering tasks performed by a human operator. It is shown that different machine learning approaches can be used to first learn single modality models using coupled Dynamic Bayesian Networks; such models are then correlated at event level to discover contextual multimodal concepts. In the presented case, visual perception and localization are used as modalities. Cross-correlations among modalities in time is discovered from data and are described as probabilistic links connecting shared and private multi-modal DBNs at the event (discrete) level. Results are presented on experiments performed on an autonomous vehicle, highlighting potentiality of the proposed approach to allow anomaly detection and autonomous decision making based on learned self-awareness models.

This paper focuses on multi-sensor anomaly detection for moving cognitive agents using both external and private first-person visual observations. Both observation types are used to characterize agents motion in a given environment. The proposed method generates locally uniform motion models by dividing a Gaussian process that approximates agents displacements on the scene and provides a Shared Level (SL) self-awareness based on Environment Centered (EC) models. Such models are then used to train in a semi-unsupervised way a set of Generative Adversarial Networks (GANs) that produce an estimation of external and internal parameters of moving agents. Obtained results exemplify the feasibility of using multi-perspective data for predicting and analyzing trajectory information.

This paper focuses on modeling and classifying trajectories from video sequences. Location, velocity and time of appearance are considered as features for recognizing and modeling motions of objects. In a training phase, a discretization of the proposed features is performed by using a self-organizing map approach such that a set of clusters (feature vocabulary) is created for describing trajectories. A cluster dissimilarity measure based on a weighted fusion of features facilitates the recognition of trajectory classes in an incremental way. As a result, an unsupervised method for encoding observed motion information and identifying trajectory patterns is proposed in this article. The method is evaluated with real and simulated data. Additionally' comparisons with previous works show the benefits of our method when encoding and identifying motion patterns in video sequences.

In recent years several architectures have been proposed to learn embodied agents complex self-awareness models. In this paper, dynamic incremental self-awareness (SA) models are proposed that allow experiences done by an agent to be modeled in a hierarchical fashion, starting from more simple situations to more structured ones. Each situation is learned from subsets of private agent perception data as a model capable to predict normal behaviors and detect abnormalities. Hierarchical SA models have been already proposed using low dimensional sensorial inputs. In this work, a hierarchical model is introduced by means of a cross-modal Generative Adversarial Networks (GANs) processing high dimensional visual data. Different levels of the GANs are detected in a self-supervised manner using GANs discriminators decision boundaries. Real experiments on semi-autonomous ground vehicles are presented.

We present a method to model and classify trajectory data that come from surveillance videos. Observations of the locations of moving entities are used to estimate their expected velocity in the scene. Such estimation is performed by a Gaussian process regression that enables to approximate probabilistically the expected velocity of entities given some observed evidence in the scene. Subsequently, regions where estimations have high certainty are decomposed into zones by superpixel segmentation. Each zone represents a region where motions of entities can be explained by a quasilinear dynamical model. We evaluated the proposed method with two datasets and confirmed its reliability for characterizing and classifying trajectories.

To understand the behavior of moving entities in a given environment, one should be capable of predicting their motion, that is, to model their dynamics. In a setting where different behaviors can arise, one can assume that each of them corresponds to different motivational states of observed entities. Here, those motivations are understood as goal positions or spots where entities seek to arrive. To build prediction models based on that idea, we present an unsupervised method to estimate motivational spots actively. Additionally, we use the output of such process to refine an adaptive system modeling the dynamics of inferred hidden causes of observed data. The whole method uses deep variational methods, and particularly, the network estimating motivations is trained through dynamic programming. Results show that modeling the dynamics of entities can be better achieved by integrating information about motivational spots. Notably, a network modeling the dynamics converges faster through the incorporation of information about motivations.

This paper presents a method for observational learning in autonomous agents. A formalism based on deep learning implementations of variational methods and Bayesian filtering theory is presented. It is explained how the proposed method is capable of modeling the environment to mimic behaviors in an observed interaction by building internal representations and discovering temporal and causal relations. The method is evaluated in a typical surveillance scenario, i.e., perimeter monitoring. It is shown that the vehicle learns how to drive itself by simultaneously observing its surroundings and the actions taken by a human driver for a given task. That is achieved by embedding knowledge regarding perception-action couplings in dynamic representational states used to produce action flows. Thereby, representations link sensory data to control signals. In particular, the representational states associate visual features to stable action concepts such as turning or going straight.

With the growing availability of wearable technology, video recording devices have become so intimately tied to individuals, that they are able to record the movements of users' hands, making hand-based applications one the most explored area in First Person Vision (FPV). In particular, hand pose recognition plays a fundamental role in tasks such as gesture and activity recognition, which in turn represent the base for developing human-machine interfaces or augmented reality applications. In this work we propose a graph-based representation of hands seen from the point of view of the user, obtained through the shape-fitting capability of a modified Instantaneous Topological Map. Spectral analysis of the graph Laplacian allows to arrange eigenvalues in vectors of features, which prove to be discriminative in classifying the considered hand poses.

Target tracking in a network of wireless cameras may fail if data are captured or exchanged asynchronously. Unlike traditional sensor networks, video processing may generate significant delays that also vary from camera to camera. Moreover, the continuous and rapid change of the dynamics of the consensus variable (the target state) makes tracking even more challenging under these conditions. To address this problem, we propose a consensus approach that enables each camera to predict information of other cameras with respect to its own capturing time-stamp based on the received information. This prediction is key to compensate for asynchronous data exchanges. Simulations show the performance improvement with the proposed approach compared to the state of the art in the presence of asynchronous frame captures and random processing delays.

This paper proposes a method for detecting zones of visual attention based on the motion of agents in a video analytics context. By considering a Hough transform approach, linear flow motions are grouped based on attractive salient zones where they converge. Each group of linear flows is generalized through the whole environment by using a non-parametric stochastic approach that can be used to generate a map that illustrates the effects that each zone exerts on the dynamics of agents. A dataset of walking pedestrians and trajectories generated by a robot that executes a single task in a close environment are used to validate the proposed method.

In this paper we address the abnormality detection problem in crowded scenes. We propose to use Generative Adversarial Nets (GANs), which are trained using normal frames and corresponding optical-flow images in order to learn an internal representation of the scene normality. Since our GANs are trained with only normal data, they are not able to generate abnormal events. At testing time the real data are compared with both the appearance and the motion representations reconstructed by our GANs and abnormal areas are detected by computing local differences. Experimental results on challenging abnormality detection datasets show the superiority of the proposed method compared to the state of the art in both frame-level and pixel-level abnormality detection tasks.

Cognitive radio (CR) is a promising technology for future wireless spectrum allocation to improve the use of licensed bands. However, security challenges faced by cognitive radio technology are still a hot research topic. One of prevailing challenges is the radio frequency jamming attack, where adversaries are able to exploit on-the-fly reconfigurability potentials and learning mechanism of cognitive radios in order to devise and deploy advanced jamming tactics. Jamming attacks can significantly impact the performance of wireless communication systems and lead to significant overheads in terms of retransmission and increment of power consumption. In this context, a novel jammer detection algorithm is proposed using cyclic spectral analysis and artificial neural networks (ANN) for wide-band (WB) cognitive radios. The proposed approach assumes a WB spectrum occupied by various narrow-band (NB) signals, which can be either legitimate or jamming signals. The second order statistics, namely, the spectral correlation function (SCF) and ANN are used to classify each NB signal as a legitimate or jamming signal. The algorithm performance is shown with the help of simulations.

The introduction of cognitive radio enables dynamic spectrum access for higher spectrum utilization, due to its ability of awareness of their environment. However, the introduction of cognitive radio technology brings new challenges to wireless networks security. Due to intelligent nature of the attackers, many of the radio frequency jamming attacks can be stealthy by nature. The stealthy jamming attacks can significantly impact the performance of the wireless communication system and can lead to significant overhead in terms of retransmission and increment of power consumption. This paper presents a new physical layer approach for stealthy jammer detection in wide-band (WB) cognitive radio networks. The proposed algorithm consider a WB consists of multiple narrow-band sub-bands (SB), which can be occupied by licit or jamming signals . The cyclostationary spectral analysis is performed on this WB signal to compute spectral correlation function (SCF). The alpha profile is extracted from the SCF and used as input features to artificial neural network (ANN), which classify each NB signal as a licit signal or a jamming signal. In the end, the performance of the proposed approach is shown with the help of Monte-Carlo simulations under different empirical setups.

In this paper we introduce a novel method for general semantic segmentation that can benefit from general semantics of Convolutional Neural Network (CNN). Our segmentation proposes visually and semantically coherent image segments. We use binary encoding of CNN features to overcome the difficulty of the clustering on the high-dimensional CNN feature space. These binary codes are very robust against noise and non-semantic changes in the image. These binary encoding can be embedded into the CNN as an extra layer at the end of the network. This results in real-time segmentation. To the best of our knowledge our method is the first attempt on general semantic image segmentation using CNN. All the previous papers were limited to few number of category of the images (e.g. PASCAL VOC). Experiments show that our segmentation algorithm outperform the state-of-the-art non-semantic segmentation methods by large margin.

A method for online incremental mining of activity patterns from the surveillance video stream is presented in this paper. The framework consists of a learning block in which Dirichlet process mixture model is employed for the incremental clustering of trajectories. Stochastic trajectory pattern models are formed using the Gaussian process regression of the corresponding flow functions. Moreover, a sequential Monte Carlo method based on Rao-Blackwellized particle filter is proposed for tracking and online classification as well as the detection of abnormality during the observation of an object. Experimental results on real surveillance video data are provided to show the performance of the proposed algorithm in different tasks of trajectory clustering, classification, and abnormality detection.

Under a tracking framework, the definition of the target state is the basic step for automatic understanding of dynamic scenes. More specifically, far object tracking raises challenges related to the potentially abrupt size changes of the targets as they approach the sensor. If not handled, size changes can introduce heavy issues in data association and position estimation. This is why adaptability and self-awareness of a tracking module are desirable features. The paradigm of cognitive dynamic systems (CDSs) can provide a framework under which a continuously learning cognitive module can be designed. In particular, CDS theory describes a basic vocabulary of components that can be used as the founding blocks of a module capable to learn behavioral rules from continuous active interactions with the environment. This quality is the fundamental to deal with dynamic situations. In this paper we propose a general CDS-based approach to tracking. We show that such a CDS-inspired design can lead to the self-adaptability of a Bayesian tracker in fusing heterogeneous object features, overcoming size change issues. The experimental results on infrared sequences show how the proposed framework is able to outperform other existing far object tracking methods.

In large and crowded transportation sites such as ports, parking lots, or busy streets, the unsafe actions of a moving object (such as a pedestrian, a ship, or a car) may compromise the safety and security of the entire infrastructure. It follows that a full comprehension of the situations taking place in the area could significantly decrease the amount of work performed by human administrators and operators in charge of the zone, reducing also the impact of human errors. The idea behind this paper is to use trajectory data to analyze behaviors (atomic actions without any external influence) and interactions (actions inducted by the presence of another entity) of each target in the scene. Probabilistic techniques applied on a topological map of the scene allow tackling the problem in a robust way, handling the uncertainties arising in such environments. The system is tested in a real maritime scenario, where trajectory data of ships and vessels are used to determine normal and abnormal situations occurring in a canal.

A method for online identification of group of moving objects in the video is proposed in this paper. This method at each frame identifies group of tracked objects with similar local instantaneous motion pattern using spectral clustering on motion similarity graph. Then, the output of the algorithm is used to detect the event of more than two object moving together as required by PETS2015 challenge. The performance of the algorithm is evaluated on the PETS2015 dataset.

We present a generic formulation of self- and cross-correcting Bayesian trackers using a Dynamic Bayesian Network. Correction operations in a tracker such as parameter tuning, model updates and re-initialization are represented using hidden variables together with the target state and measurement variables in the Dynamic Bayesian network model. The representation allows one to model different self- and cross-correcting tracking frameworks under the same formulation and facilitates comparison and the design of new trackers. The proposed model is demonstrated with three state-of-the-art trackers that are based on different principles to implement online correction of target tracking.

It is well known from physiological studies that the level of human attention for adult individuals rapidly decreases after five to twenty minutes [1]. Attention retention for a surveillance operator represents a crucial aspect in Video Surveillance applications and could have a significant impact in identifying relevance, especially in crowded situations. In this field, advanced mechanisms for selection and extraction of saliency information can improve the performances of autonomous video surveillance systems and increase the effectiveness of human operator support. In particular, crowd monitoring represents a central aspect in many practical applications for managing and preventing emergencies due to panic and overcrowding.

Classification of moving objects for video surveillance applications still remains a challenging problem due to the video inherently changing conditions such as lighting or resolution. This paper proposes a new approach for vehicle/pedestrian object classification based on the learning of a static kNN classifier, a dynamic Hidden Markov Model (HMM)-based classifier, and the definition of a fusion rule that combines the two outputs. The main novelty consists in the study of the dynamic aspects of the moving objects by analysing the trajectories of the features followed in the HOG-PCA feature space, instead of the classical trajectory study based on the frame coordinates. The complete hybrid system was tested on the VIRAT database and worked in real time, yielding up to 100% peak accuracy rate in the tested video sequences.

The problem of behavior assessment in video surveillance is approached using trajectory classification. Lagrangian state dynamic is used for probabilistic modeling of trajectory patterns and an off-line parameter learning method for the model is proposed. For classification purpose, an on-line sequential maximum a posterior trajectory classifier is introduced based on particle filter. Finally, the performance of this method is evaluated using a traffic video data set.

Classifying frames, or parts of them, is a common way of carrying out detection tasks in computer vision. However, frame by frame classification suffers from sudden significant variations in image texture, colour and luminosity, resulting in noise in the extracted features and consequently in the decisions taken. Support Vector Machines have been widely validated as powerful tools for frame by frame detection of non-separable datasets, but are extremely sensitive to these variations between adjacent frames, creating as consequence sudden flickering in the classification results. This work proposes a Dynamic Bayesian Network to smooth the classification results of Support Vector Machines (SVM) in detection tasks. The method is evaluated in First Person Vision (FPV) videos, where a SVM is used to decide whether or not the user's hands are in his field of view.

In this work, we propose a strategy for optimizing a superpixel algorithm for video signals, in order to get closer to real time performances which are on the one hand needed for egocentric vision applications and on the other must be bearable by wearable technologies. Instead of applying the algorithm frame by frame, we propose a technique inspired to Bayesian filtering and to video coding which allows to re-initialize superpixels using the information from the previous frame. This results in faster convergence and demonstrates how performances improve with respect to the standard application of the algorithm from scratch at each frame.

The emergence of new wearable technologies, such as action cameras and smart glasses, has increased the interest of computer vision scientists in the first person perspective. Nowadays, this field is attracting attention and investments of companies aiming to develop commercial devices with first person vision (FPV) recording capabilities. Due to this interest, an increasing demand of methods to process these videos, possibly in real time, is expected. The current approaches present a particular combinations of different image features and quantitative methods to accomplish specific objectives like object detection, activity recognition, user-machine interaction, and so on. This paper summarizes the evolution of the state of the art in FPV video analysis between 1997 and 2014, highlighting, among others, the most commonly used features, methods, challenges, and opportunities within the field.

We propose a tracker-level fusion framework for robust visual tracking. The framework combines trackers addressing different tracking challenges to improve the overall performance. A novelty of the proposed framework is the inclusion of an online performance measure to identify the track quality level of each tracker so as to guide the fusion. The fusion is then based on appropriately mixing the prior state of the trackers. Moreover, the track-quality level is used to update the target appearance model. We demonstrate the framework with two Bayesian trackers on video sequences with various challenges and show its robustness compared with the independent use of the two individual trackers, and also compared with state-of-the-art trackers that use tracker-level fusion.

Interaction analysis of ships mooring and maneuvering in harbors is pursued in this paper by using Bayesian probabilistic models. A number of ship-to-ship interactions are deduced from the navigation rules in port areas, and then used to train different Event-based Dynamic Bayesian Networks (E-DBNs). When data of two interacting ships are injected into the network, inference is performed in order to verify if the interaction between the vessels is known or not, and in the latter case actions to preserve the port safety can be taken. Results are drawn in the final part of the paper by including into the networks data provided by a simulator of realistic trajectories relative to an existing port.

Extraction and representation of relevant information from large-scale surveillance systems constitute fundamental processes for allowing automatic interpretation of complex scenes. In particular, when the amount of information increases (i.e., due to a larger number of monitored areas), attention focusing techniques are needed to highlight most relevant parts within the overall acquired data. When wide area surveillance systems are considered, one of the major problems in event detections is the reconstruction of the scene as a whole, from spatially limited observations. In this paper, a novel representation technique for sparse information, based on information theory, is presented. Self Organizing Maps (SOMs) have been used for classifying and correlating observed sparse data time series. By means ofInformation Bottlenecktheory, it is possible to determine the optimal data representation in the SOM-space as a tradeoff between the signal reconstruction capabilities and the original data statistical similarities preservation. Proposed experiments show how the so calledinformation bottleneck-basedSOMselectionfor knowledge modelling, can be applied to the field of crowd monitoring for people density map estimation and event detection. Results are presented on synthetic and real video sequences.

On-the-fly reconfigurability capabilities and learning prospectives of Cognitive Radios inherently bring a set of new security issues. One of them is intelligent radio frequency jamming, where adversary is able to deploy advanced jamming strategies to degrade performance of the communication system. In this paper, we observe the jamming/antijamming problem from a game-theoretical perspective. A game with incomplete information on opponent's payoff and strategy is modelled as a Markov Decision Process (MDP). A variant of fictitious play learning algorithm is deployed to find optimal strategies in terms of combination of channel hopping and power alteration anti-jamming schemes.

Recently, a Bayesian estimator with a hybrid update was developed [1], based on a mathematical formulation of sampling. Such an Event Based State Estimator (EBSE) allows for a stable synchronous state estimate, relying on asynchronous measurements. Usefulness of such a filter comes with its approximate analytic formulation, which is attainable given a send-on-delta sampling strategy. We argue that such a formulation can be extended to cope with a failing detector in case the filter is used for tracking. The basic idea is to approach the issue as a package loss problem, where a missed target is assimilated to a lost package. More in detail, we propose that this approach can be exploited in video tracking, where faulty detectors are commonplace. We show how tracking performance with a poor pedestrian detector, failing to recognize its target, can improve with respect to standard Kalman filter.

Detection of dim moving point targets in cluttered background can have a great impact on the tracking performances. This may become a crucial problem, especially in low-SNR environments, where target characteristics are highly susceptible to corruption. In this paper, an extended target model, namely Interacting Multiple Model (IMM), applied to Track-Before-Detect (TBD) based detection algorithm, for far objects, in infrared (IR) sequences is presented. The approach can automatically adapts the kinematic parameter estimations, such as position and velocity, in accordance with the predictions as dimensions of the target change. A sub-par sensor can cause tracking problems. In particular, for a single object, noisy observations (i.e. fragmented measures) could be associated to different tracks. In order to avoid this problem, presented framework introduces a cooperative mechanism between Joint Probabilistic Data Association Filter (JPDAF) and IMM. The experimental results on real and simulated sequences demonstrate effectiveness of the proposed approach.

Detection of emotions of a crowd is a new research area, which has never, to our knowledge, been accounted for research in previous literature. A bio-inspired model for representation of emotional patterns in crowds has been demonstrated. Emotions have been defined as evolving patterns as part of a dynamic pattern of events. This model has been developed to detect emotions of a crowd based on the knowledge from a learned context, psychology and experience of people in crowd management. The emotions of multiple people making a crowd in any surveillance environment are estimated by sensors signals such as a camera and are being tracked and their behavior is modeled using bio-inspired dynamic model. The behavior changes correspond to changes in emotions. The proposed algorithm involves the probabilistic signal processing modelling techniques for analysis of different types of behavior, interaction detection and estimation of emotions. The emotions are recognized by the expectation of temporal occurrences of causal events modeled by Gaussian mixture model. The model has been evaluated using the simulated behavioral model of a crowd.

In most recent Intelligent Video Surveillance systems, mechanisms used to support human decisions are integrated in cognitive artificial processes. Large scale video surveillance networks must be able to analyse a huge amount of information. In this context, a cognitive perception mechanism integrate in an intelligent system could help an operator for focusing his attention on relevant aspects of the environment ignoring other parts. This paper presents a bio-inspired algorithm called Selective Attention Automatic Focus (S2AF), as a part of more complex Cognitive Dynamic Surveillance System (CDSS) for crowd monitoring. The main objective of the proposed method is to extract relevant information needed for crowd monitoring directly from the environmental observations. Experimental results are provided by means of a 3D crowd simulator; they show how by the proposed attention focus method is able to detect densely populated areas.

Main drawbacks in single-camera multi-target visual tracking can be partially removed by increasing the amount of information gathered on the scene, i.e. by adding cameras. By adopting such a multi-camera approach, multiple sensors cooperate for overall scene understanding. However, new issues arise such as data association and data fusion. This work addresses the issue of evaluating the performance of a multi-camera tracking algorithm based on Rao-Blackwellized Monte Carlo data association (RBMCDA) on real data. For this purpose, a new metric based on three performance indexes is developed.

This work addresses the problem of people counting in crowded situations, such as urban environments, in computer vision. As crowding density increases in a scene, it might become impossible to count people as single individuals: a global group-based approach is then preferable and in fact often necessary. A simple method for estimating the count of people in such tight crowds is here proposed, relying on accurate camera calibration. A training phase is also needed by the algorithm in order to learn the parameters needed for estimation.

Online evaluation of tracking algorithms has received attentions in computer vision community to detect failures and apply correction methods for achieving better performances. In this paper, a novel online evaluation framework is proposed for a multitarget feature points based object tracking. An online partial least square regression and correlation model is constructed from short trajectory histories for the tracks. The model allows to estimate the state of one track from the other track states. The core idea for the method is creating a virtual reference data for evaluation from the learned model. The proposed self-evaluation mechanism is presented as a Dynamic Bayesian Network. The method is evaluated on a simulation data for tracking feature points from a pedestrian.

This work addresses the issue of fire and smoke detection in a scene within a video surveillance framework. Detection of fire and smoke pixels is at first achieved by means of a motion detection algorithm. In addition, separation of smoke and fire pixels using colour information (within appropriate spaces, specifically chosen in order to enhance specific chromatic features) is performed. In parallel, a pixel selection based on the dynamics of the area is carried out in order to reduce false detection. The output of the three parallel algorithms are eventually fused by means of a MLP.

This paper proposes and algorithm for extended object tracking using sparse feature points. The described technique is based on the Rao-Blackwellized Particle Filter. In particular, two different data association techniques that take into consideration clutter and missed detections, are coupled and tested in order to provide a comparison of their performance for the problem of extended object tracking.

Online evaluation of tracking algorithms is an important task in real time tracking systems to detect failures. In visual object tracking based on sparse features, detecting the failure of one of the feature points (corners) and correcting it will improve the performance of the tracker as a whole. In this paper a time reversed Markov chain is applied as evaluation technique to identify the failed trackers and Partial Least Square regression is used for learning the correlation between feature points from training data set. The detected feature point trackers are recovered from the knowledge of the learned correlation model. The results are explained on a Bayesian algorithm for rigid/nonrigid 2D visual object tracking. The experimental outcomes show a global performance improvement of the tracking algorithm even in the presence of clutter.

This paper describes a Bayesian algorithm for rigid/non-rigid 2D visual object tracking based on sparse image features. The algorithm is inspired by the way human visual cortex segments and tracks different moving objects within its FOV by constructing dynamical nonretinotopic layers. The method is explained as a recursive algorithm between time slices (intra-slice) and as a forward-backward message passing within every time slice (inter-slice) under the Probabilistic Graphical Model (PGM) framework. Finally, an observation model function that resembles the Generalized Hough Trans form and allows exploiting internal structure of the problem is employed in order to increase the robustness and accuracy of the algorithm against clutter and missed detections.

Real-time automatic human behavior recognition is one of the most challenging tasks for intelligent surveillance systems. Its importance lies in the possibility of robust detection of suspicious behaviors in order to prevent possible threats. The widespread integration of tracking algorithms into modern surveillance systems makes it possible to acquire descriptive motion patterns of different human activities. In this work, a statistical framework for human interaction recognition based on Dynamic Bayesian Networks (DBNs) is presented: the environment is partitioned by a topological algorithm into a set of zones that are used to define the state of the DBNs. Interactive and non-interactive behaviors are described in terms of sequences of significant motion events in the topological map of the environment. Finally, by means of an incremental classification measure, a scenario can be classified while it is currently evolving. In this way an autonomous surveillance system can detect and cope with potential threats in real-time.

In this work, the problem of the estimation of parameters in case of mixtures of models composed of the sum of multiple Gaussians is considered. It will be shown how this estimation can be performed efficiently by using the Generalized Hough Transform (GHT). The theoretical results will be applied to a corner-based object tracking application considering, in particular, the case of two or more objects that come into proximity and occlude each other. Quantitative results show the performances of the derived algorithm both on synthetically generated data and real tracking sequences.

This paper proposes a computationally efficient approach for estimating camera motion based on a voting mechanism that neither relies on explicit feature matching nor RANSAC. In the presented method, correspondences arise as a consequence of finding the camera movement in a voting space and not as a prerequisite. Experiments show promising results compared with (KLT+RANSAC) while taking less time in computation.

The process of finding representative shape patterns from sparse datasets is a challenging task: especially for non-rigid objects, shape deformations through time can produce very different sets of corners from frame to frame and a proper comparison of point features can be very difficult. Evaluating a multi-objective fitness function in a discrete voting space, partial similarities between deformable objects can be found and a correct data association can be performed. A genomic encoding of corner-based shapes is introduced and, taking advantage of a robust genetic-based search algorithm, sets of corners pertaining to objects of interest are mapped into common models. The most representative features are detected and used to evolve shape prototypes.

Even if the problem of human action categorization from videos has received a lot of attention during the past decade, it remains a challenging problem in operative conditions due to camera motion, occlusion, moving background, illumination changes and the variations of human appearance and postures. In this paper a new motion descriptor, based on a sparse optical flow computed by interest point tracking is presented. This motion descriptor is by design invariant to scale, camera motion and is not affected by non stationary background. The results of the recognition method are computed using a standard database and are compared to other approaches in literature.

In this paper two different cognitive radio architectures, i.e. stand-alone and distributed, are proposed for spectrum sensing purposes. In particular, both architectures implement a fast and reliable algorithm based on cyclic features extraction which allows to identify spectrum holes. The performances of such systems are compared in detecting primary users' presence in a monitored area classifying the used transmission standards, IEEE 802.11a and IEEE 802.16e. The considered scenario is challenging since both standards use the OFDM transmission technique, are designed to have the same bandwidth and use the same frequency band. A set of numerical simulations have been carried out to compare the performances of the proposed systems in a heavy multipath scenario and their advantages and disadvantages are discussed.

In Smart Spaces (SSs), the capability of learning from experience is fundamental for autonomous adaptation to environmental changes and for proactive interaction with users. New research trends for reaching this goal are based on neurophysiological observations of human brain structure and functioning. A learning technique that is used to provide the SS with the so-called Autobiographical Memory is presented here by drawing inspiration from a bio-inspired model of the interactions occurring between the system and the user. Starting from the hypothesis that user's actions have a direct influence on the internal system state variables and vice versa, a statistical voting algorithm is proposed for inferring the cause/effect relationships among users and the system. The main contribution of this paper lies in proposing a general framework that is able to allow the SS to be aware of its present state as well as of the behavior of its users and to be able to predict the expected consequences of user actions.

The application of intelligent systems composed by smart cameras is continuously spreading in a wide range of applications, playing a key role in public, military and commercial scenarios. As well, in the last years, the capability of wireless sensor networks to collect information from the environment in a distributed manner has been successfully applied in both civilian and military applications. In this paper, basing on recent studies on autonomous cognitive systems, we explore the concepts for designing interactive, adaptable and intelligent multi-sensor surveillance systems able to react to situations in a preventive way by using actuators placed in the monitored environment. To this end, taking inspiration from ambient intelligence (AmI) and cognitive radio (CR) paradigms, fusion of information provided by heterogeneous sensors is used to improve awareness regarding surrounding environment.

Automatic understanding of human behavior is an important and challenging objective in several surveillance applications. One of the main problems of this task consists in accurately defining models able to characterize in a discriminative but, at the same time, enough general way people actions. In this work a bio-inspired model is proposed to represent people interactions in a Bayesian framework using their patterns of movement. Couples of observed interacting trajectories are encoded into a Dynamic Bayesian Network (DBN) model where states and conditional probability densities are learned in an online manner in order to statistically describe interactions. Observed trajectories are processed by the Instantaneous Topological Map (ITM) algorithm that automatically creates a topological map used to define the states of the DBN. The transition probabilities are estimated by combining states frequency of occurrence, evaluated by a voting-based approach, and their temporal occurrence represented by Gaussian Mixture Models. The discriminative capabilities of this model to detect interactions are shown both in a simulated and in a real-world environment.

In the near future automatic systems able to detect the traffic situation and to understand driver behavior and intent will probably become vehicle tools important for improving driver safety. Therefore, robust video processing techniques able to cope with difficult environmental road condition such as luminosity changes, dynamic and cluttered background, etc. are necessary for these applications. In this work, lanes detection, vehicle position and traffic analysis are the information extracted to characterize the driving situation and the proposed techniques try to cope with the above mentioned issues. The presented framework is tested using an on-board camera in real-world scenario respecting the real-time constraint and showing good performances in highways and urban roads.

A correct video segmentation, namely the detection of moving objects within a scene plays a very important role in many application in safety, surveillance, traffic monitoring and object detection. The main objective of this paper is to implement an effective background segmentation algorithm for corner sets extracted from video sequences. A dynamic prototype of the structure of background corners is produced and incoming corners are classified using a fuzzy ARTMAP neural network and labeled as pertaining to the background or foreground using a spatial clustering method. Finally the accuracy of the proposed algorithm is evaluated using PETS2006 benchmark data.

In this paper the problem of detecting the presence of similar OFDM signals, i.e. WLAN and WiMAX signals, in an Open Spectrum scenario is faced. The identification of the channel occupancy and the signal classification are performed by using a fast detector based on a single spectral correlation function estimator and a multi-class support vector machine classifier which are designed and tested in a multipath environment. Finally, the obtained numerical results and the amount of processing necessary to perform the considered operations are reported and discussed.

The paper presents a probabilistic Bayesian framework for object tracking using a combination of a corner-based model and coefficients of Undecimated wavelet packet transform (UWPT) inside a patch around each corner. This combination uses the UWPT coefficients patch helps to enrich the global representation of the object shape model by local descriptors. The goal is to maximize the posterior of the object global position. To this end, a voting mechanism is used based on the coherency among the model corners. The role of the local wavelet-based descriptors is to filter out some irrelevant observation before the voting process. Experimental results indicate good performances of the algorithm in crowd scenes and partial occlusions.

The topic on which this paper is focused is transfer process of tracking of moving objects between sensors in a distributed surveillance system.

This paper presents a tracking algorithm based on a sequential importance sampling (SIS) particle filter scheme followed by a resampling strategy where shape and color cues are exploited to handle deformable objects. The state vector is composed by a set of corners and it enables to jointly describe position and shape of the target. Mean Shift trackers, applied to color cues associated to state subspaces, are employed to predict the target global motion. An adaptive system noise is defined based on this information to cope with local deformations. The updating procedure is accomplished by a shape matching technique. Experimental results prove the effectiveness of the proposed approach with respect to simple deformations, partial occlusions and moving camera.

The paper presents a new corner-model based learning method able to track non-rigid objects in the presence of occlusion. A voting mechanism followed by a probability density analysis of the voting space histogram is used to estimate new position of the target. The model is updated at any frame. The problem rises in the occlusion events where the occluder corners affect the model and the tracker may follow the occluder. The key point of the method toward success is automatically deciding on the corners to classify them into two classes, good and malicious corners. Good corners are used to update the model in a conservative way removing the corners that are voting to the highly voted wrong positions due to the occluder. This leads to a continuous model learning during occlusion. Experimental results show a successful tracking along with a more precise estimation of shape and motion during occlusion

One of the main issues for Ambient Intelligence (AmI) systems is to continuously localize the user and to detect his/her identity in order to provide dedicated services. A video-radio fusion methodology, relying on the Particle Filter algorithm, is here proposed to track objects in a complex extensive environment, exploiting the complementary benefits provided by both systems. Visual tracking commonly outperforms radio localization in terms of precision but it is inefficient because of occlusions and illumination changes. Instead, radio measurements, gathered by a user's radio device, are unambiguously associated to the respective target through the "virtual" identity (i.e. MAC/IP addresses). The joint usage of the two data typologies allows a more robust tracking and a major flexibility in the architectural setting up of the AmI system. The method has been extensively tested in a simulated and off-line framework and on real world data proving its effectiveness.

Learning from experience is a basic task of human brain that is not yet fulfilled satisfactorily by computers. Therefore, in recent years to cope with this issue, bio-inspired approaches has gathered the attention of several researchers. In this work a learning method is proposed based on a model derived from neurophysiological observations of the generation of the sense of self which is connected to the memorization of the interaction with external entity. The domain of application where this algorithm is employed, is a Cognitive Surveillance system which aims at detecting intruders and communicate guidance messages to a user (a guard) provided with a mobile device in order to chase him. The proposed method intends to allow the system to establish an efficient interaction with the user by sending messages only when necessary. To this end the zones of the monitored area are classified according to the probability that a change in pursuit strategy will occur by learning online the motion of the user and the intruder. The proposed algorithm has been tested on real world data demonstrating the capacity of learning this information to be used to tag the zones of the area under exam.

Smart Spaces are an emerging technology which is gathering interest in several domains of application since they allow to supply services and to interact with users in a pervasive way. One of their basic tasks regards the localization of the users in order to provide services in a personalized and location- based way. However since the guarded area is usually complex (e.g. with occlusions) and extent several sensors must be used. In this work video data acquired by video-cameras and radio signals of the WLAN, by which user can access to services, are jointly employed to improve the association between video track and radio identifier, and, through a two step temporal filtering, it is possible to enhance the system reliability. Results are presented in a simulated environment showing the effectiveness of the proposed approach.

We propose a multi-target tracking algorithm based on the probability hypothesis density (PHD) filter and data association using graph matching. The PHD filter is used to compensate for miss-detections and to remove noise and clutter. This filter propagates the first order moment of the multi-target posterior (instead of the full posterior) to reduce the growth in complexity with the number of targets from exponential to linear. Next the filtered states are associated using graph matching. Experimental results on face, people and vehicle tracking show that the proposed multi-target tracking algorithm improves the accuracy of the tracker, especially in cluttered scenes.

This paper presents a corner based voting method for estimating the object shift in video image frames. Information about the corners distribution around a reference point is used to represent the object shape and then to find the most probable target position in the next frame. Tracking is done through using a voting space obtained by matching corners information. A motion vector for the reference point is nonlinearly estimated with three different strategies by using the global information of the matched corners. The results show a comparison between three considered strategies for estimating the object shift.

The Bayesian filtering for recursive state estimation and the shape-based matching methods are two of the most commonly used approaches for target tracking. The multiple hypothesis shape-based tracking (MHST) algorithm, proposed by the authors in a previous work, combines these two techniques using the particle filter algorithm. The state of the object is represented by a vector of the target corners (i.e. points in the image with high curvature) and the multiple state configurations (particles) are propagated in time with a weight associated to their probability. In this paper we demonstrate that, in the MHST, the likelihood probability used to update the weights is equivalent to the voting mechanism for generalized Hough transform (GHT)-based tracking. This statement gives an evident explanation about the suitability of a MAP (maximum a posteriori) estimate from the posterior probability obtained using MHST. The validity of the assertion is verified on real sequences showing the differences between the MAP and the MMSE estimate.

In the last years, the Particle Filter algorithm has been extensively proposed and employed for handling the problem of visual tracking of multiple moving objects under different assumptions. This wide usage is due to the capability of performing a recursive multiple hypothesis state estimation for non-linear non-Gaussian motion and observation models. In this paper a method, based on the Particle Filter framework, is proposed for multiple objects tracking, exploiting a target representation consisting of position and shape described as a fixed dimensionality vector composed by a fixed number of grouped target corners. However, usually, application domains of visual tracking algorithms are characterized by non-rigid objects and high occlusions rate entailing new corners to appear and others to disappear at each frame. In order to cope with this problem, a voting method (i.e. the Generalized Hough Transform) is employed to estimate the likelihood function to weight different propagated particles (i.e. multiple corners configurations describing shapes) by means of the corners extracted from the currently observed image. This method, in addition to the high dimensionality of the state representation, depicts the two main particularities of the presented Particle Filter. The proposed algorithm has been tested in a real-world domain and experiments indicate good results in tracking both rigid and non-rigid objects.

Cognitive radios is emerging in research laboratories as a promising wireless paradigm, which will integrate benefits of software defined radio with a complete aware communication behavior. To reach this goal many issues remain still open, such as powerful algorithms for sensing the external environment. This paper presents a further step in the direction of allowing cooperative spectrum sensing in peer-to-peer cognitive networks by using distributed detection theory. The approach aims at improving the radio awareness with respect to stand alone scenario as it is shown with theoretical and experimental results

In this paper a system is presented able to reproduce the actions of multiple moving objects into a 3D model. A multi-camera surveillance system is used for automatically detect, track and classify the objects. Data fusion from multiple sensors allows to get a more precise estimation of the position of detected moving objects and to solve occlusions problem. These data are then used to automatically place and animate objects avatars in a 3D virtual model of the scene, thus allowing a human operator to remotely visualize the dynamic 3D reconstruction by selecting a arbitrary point of view.

This paper describes a video surveillance system aimed at the automatic identification of events of interest, especially of abandoned and stolen objects in a guarded indoor environment. In particular the implemented system combines three phases of data processing: object extraction, object recognition and tracking, and decision about actions. Extracted objects are classified as "human" or "non-human" and static or dynamic, an event of interest following from a split between a "human" and a static "non-human" object, finally static "nonhuman" is analyzed to discriminate between abandoned or stolen object.

This paper presents a method based on skin colormodel face detection and human body proportion constraints to estimate the number and position of people entering the monitored scene as a compact group. This helps to split the group into individual persons and solve a common lack in traditional background subtraction based detection and tracking methods: despite common systems, this algorithm can recognize the number and position of the people in a single change detection blob. The hypotheses are: standing subjects, face visibility from the point of view of the camera, a calibrated map to estimate objects' distance from the sensor and to estimate expected people's height on the image plane. These estimations allow dynamic thresholding of several shape parameters and lead to very interesting results.

In this paper, a surveillance system able at detecting abandoned objects automatically is proposed. In particular the implemented system aims at detecting the presence of abandoned objects or in a guarded indoor environment and at tracking the owner while he is in the scene and advise if another person take the abandoned object instead of the owner.

In recent decades vandal acts and graffiti drawing problem have increased and have required a lot of public funding. To face this problem the communal administrations have invested in automatic video surveillance systems. To deal with this problem through image processing techniques, this paper presents a method for graffiti detection based on change detection algorithm and motion vector. The aim of this system is to detect the graffiti painting act while people are going to draw, identify them and distinguish the drawer.

This work describes a data fusion technique to improve performances in objects localization and tracking for automatic video surveillance systems. The developed strategy is designed to perform well in case of interaction among objects, i.e. when the moving objects to track, and whose position we want to locate on the common map reference system, result superimposed in the image plane. In order to solve such complex situations, different kind of techniques have been integrated but the focus of the paper is on the data association step in the fusion chain. As discussed in the text, failing in the association phase means computing wrong position during fusion process. The performances of the developed technique has been evaluated on sequences of real images and experimental results show the validity of the approach in the reduction of association errors during occlusion phases.

This paper describes automatic video sequences processing techniques for detecting suspect and dangerous situations within public transportations. Proposed surveillance system is able to raise different kind of warnings and alarms on the basis of the particular detected situation. Algorithms used for objects detection and tracking will be described in details and performances will be discussed in relation with alarm conditions that are showed in the sequences that have been made available for this conference. An empty reference image is used for object extraction through image difference. In order to perform background updating a high level module is implemented taking into account the detected objects and their classification tags. The system has been tested on several sequences showing dangerous events due to human behaviors in an underground station.

In this paper, techniques and related issues for the definition of a contextual knowledge in ambient-intelligence systems are explored. A logical structure for this kind of system, inspired by a neurobiological brain model, is proposed. Through these considerations, the role and the importance of context awareness in the definition of an artificial organism showing adaptability, pervasiveness, and scalability features are described. Techniques for the definition of a multilayer context representation are explained and practically demonstrated with a test-bed. In the proposed system, a complex event classification is obtained through the fusion of heterogeneous data coming from a set of sensors thanks to the design of a self-organizing map (SOM). The SOM represents the core of the system and testing proofs show good results in the classification of the events taking place in the monitored environment.

In this work, a mode identification system for superimposed signals in the same band is presented. More precisely, a pattern recognition approach, based on the Wigner distribution for features extraction and non parametric classifiers (k-nearest neighbors and Parzen) is proposed for identifying the transmission modes in an indoor wireless environment. A reconfigurable terminal, based on software defined radio technology, is considered aimed at the identification of the presence of two co-existent communication modes such as Bluetooth, based on frequency hopping - code division multiple access, and IEEE WLAN 802.11b, based on direct sequence - code division multiple access. Results in terms of error classification probability, expressed as relative error frequency, are provided with a comparison between the use of the two classifiers.

The aim of this paper is to present a multi camera system for location estimation inspired to a model inherited from the Data Fusion domain: the Joint Directorate of Laboratories (JDL) model (E. Waltz et al., 1990). The problem specifically faced is the tracking of objects in two complementary applications: intelligent monitoring (video surveillance) and sport viewing (football players tracking), where multiple occluding objects have to be successfully segmented and located using different features such as color, position and dynamics.

In this paper, an algorithm for tracking multiple rigid and non-rigid objects in conditions of occlusion is presented. The proposed method is based on a scalable and adaptive model based on joint information of color and shape. Through a GHT (generalized Hough transform) based voting method the center of mass of each object can be determined in real time with a good degree of precision. Quantitative and qualitative results are presented to validate the efficiency of the method

This paper describes a positioning algorithm that uses the signal strength, received by a wireless local area network, to determine users position. The system improves positioning accuracy by mitigating the multipath and noise, through an empirical analysis of environment and a prepost curser mitigator. Results, in terms of relative error, are presented for indoor environment.

This paper is concerned with a particular application of image stabilization. Image stabilization is a necessary step to reduce the effect of camera motion when, as in this case, image sequences are acquired from a mobile platform. In this work, in order to find an efficient motion estimator, two one-dimensional characteristic curves are extracted from each video frame. Such curves are then compared in consequent frames to estimate image displacement. The proposed algorithm is suitable for real time processing and provides good performance; in order to verify its validity, it has been tested on a variety of color video-sequences taken from the point of view of trains moving along railways tracks. Results are provided in order to compare the proposed approach with a feature-based stabilization method.

In automatic video surveillance systems, full time monitoring represents an important goal to achieve. The aim of the paper is to find a new methodology for video surveillance adaptive parameter regulation. The method presented is based on self organizing maps (SOM) which are used for parameter regulation purposes. Results shown prove how this method permits overall good performances to be achieved for all environments.

This paper exploits an innovative technique to improve performances related to localization, tracking and classification of objects in a video surveillance system. The developed strategy has been applied to the problem of interaction between objects, i.e. well tuned traditional algorithms are able to track and classify objects whenever they enter the scene well-isolated from the other moving objects, but the state-of-the-art techniques fail when an occlusion situation is verified from the beginning. The performances of the developed algorithms have been evaluated on sequences of real images and experimental results have shown the validity of the approach.

A system for face detection in outdoor environments for multisensor video surveillance applications is presented. The system is characterized by a combination of two pan-tilt video cameras, which cooperate in order to track and to characterize moving objects with positioning and biometric informations. The final result of the action of the system is the collection of small video shots regarding the face of humans detected in outdoor environments with a robust behavior.

This paper proposes a novel architecture for multisensor data fusion in the context of ambient intelligence (Ami). The proposed system integrates a heterogeneous network of sensors with CCD cameras and computational units working together in a LAN. Activities of humans interacting in the monitored area are detected and classified by combining sensors data output with a neural method.

One of the most critical problems in asynchronous DS/CDMA reception consists of the effective acquisition of the initial pseudo-noise (PN) synchronism. In particular the multi-user interference (MUI) contrasts the operation to synchronise correctly the system. In this paper, an adaptive PN acquisition scheme based on a statistical hypothesis testing and able to reduce the detrimental effects on the acquisition process of multi-access interference present on the channel is presented. The application considered for the proposed algorithm concerns transmission over an AWGN channel, performed by a DS/CDMA system. The results shown can prove the robustness of the proposed method in presence of different levels of MUI, provided there is a reliable estimation of the channel conditions.

The paper presents a system for video object generation and selective encoding with applications in surveillance, mobile videophones, and the automotive industry. Object tracking and MPEG-4 compression are performed in real-time. The system belongs to a new generation of intelligent vision sensors called smart cameras, which execute autonomous vision tasks and report events and data to a remote base-station. A detection module signals the presence of an object of interest within the camera field of view, while the tracking part follows the target to generate temporal trajectories. The compression is MPEG-4 compliant and implements the simple profile of the standard, which is capable of encoding up to four video objects. At the same time, the compression is selective, maintaining a higher quality for foreground objects and a lower quality for background representation. This property contributes to bandwidth reduction while preserving the essential information of foreground objects. The system performance is demonstrated in experiments that involve objects representing faces and vehicles seen from both static and moving cameras.

This paper describes a technique for tracking single objects moving within the guarded scene during dynamic occlusion situations. The processing modules used for object detection and tracking will be shown in detail and the performances of the algorithm discussed. The proposed approach uses an empty reference image for object extraction through image difference; the reference frame is updated continuously by a background updating module taking into account the detected objects. The tracking module is responsible for objects labeling being able to preserve objects identity even when an overlapping occurs on the image plane between different objects. A shape matching technique is used that is based on a linear Kalman filter. The system has been tested on several outdoor sequences showing dynamic occlusions among objects in order to show the validity of the approach.

This paper presents a system for automatic video surveillance applications. The system has been designed to monitor outdoor environments such as car parks or streets, providing the human operator with a symbolic description of the scene. The final task of the architecture is to automatically provide alarms when specific events of interest are detected. In this way the level of automation of the system is increased as well as overall performances. One of the main drawbacks of traditional video surveillance systems lies in the alarm generation. This task has to be visually performed by the human operators with intrinsic limitation. The possibility of having this process automated is here described within the design of an architecture capable of acquiring, processing and successfully storing data coming from one or more sensors.

An algorithm for tracking multiple non-rigid objects in cluttered scenes is presented. The proposed approach models the shape of the objects by using corners. In particular, a learning algorithm is introduced in order to extract an adaptive model of the object automatically. The obtained adaptive model is used to individuate the object position and scale when occlusions are present. The method is used on an existing video-surveillance system in order to track moving objects in cluttered scenes. Results show that the proposed approach provides good performances with low processing times.

In this paper, a brief state of the art in the field of advanced surveillance systems is given. The goal is to provide a framework under which to describe the different contributions to research presented in the special session on multisensor surveillance systems based on image and video data. Some problems of interest considered by current research are highlighted starting from the necessity of augmented perceptual capabilities, to the use of video object based coding techniques, going to the capability of extending current video representation techniques in order to make it possible to easily correlate, recognize, index and retrieve multisensor video observations.

Increased communication capabilities and automatic scene understanding allow human operators to simultaneously monitor multiple environments. Due to the amount of data to be processed in new surveillance systems, the human operator must be helped by automatic processing tools in the work of inspecting video sequences. In this paper, a novel approach allowing layered content-based retrieval of video-event shots referring to potentially interesting situations is presented. Interpretation of events is used for defining new video-event shot detection and indexing criteria. Interesting events refer to potentially dangerous situations: abandoned objects and predefined human events are considered in this paper. Video-event shot detection and indexing capabilities are used for online and offline content-based retrieval of scenes to be detected.

This work is aimed at investigating the use of multi-carrier CDMA (MC-CDMA) techniques in variable bit-rate transmission over low-earth-orbit (LEO) satellite channels by means of realistic simulations. It is known from the literature that MC-CDMA techniques are much more resilient with respect to multi-user interference effects in multipath fading channels than single-carrier DS/CDMA ones. Moreover, MC-CDMA exhibits a natural capability to deliver multirate services simply by assigning to each user a variable-cardinality set of subcarriers. The achieved simulation results clearly confirmed the expected improved robustness of MC-CDMA techniques transmitting multirate data streams in frequency selective LEO satellite channels, with respect to state-of-the-art DS/CDMA transceivers.

An image stabilization algorithm is presented that is specifically oriented toward video-surveillance applications. The proposed approach is based on a novel motion-compensation method that is an adaptation of a well-known image-stabilization algorithm for visualization in video-surveillance applications. In particular, the illustrated methods take into account the specificity of typical video-surveillance applications, where objects moving in a scene often cover a large part of an image thus causing the failure of classic image-stabilization techniques. Evaluation methods for image stabilization algorithms are discussed.

This paper discusses a typical architecture of a third-generation surveillance system (3GSS). In particular a method for choosing the optimal distribution of intelligence required by 3GSS is presented. Experimental results over a simulated system illustrate the presented approach.

Interest for advanced video-based surveillance applications has been growing rapidly. This is especially true in the field of railway urban transport where video-based surveillance can be exploited to face many relevant security aspects (e.g. vandal acts, overcrowding situations, abandoned object detection, etc.). This paper investigates an open problem in the implementation of video-based surveillance systems for transport applications, i.e.: the implementation of reliable image understanding modules in order to recognize dangerous situations with reduced false alarm and misdetection rates. We consider the use of a neural network-based classifier for detecting the behavior of vandals in metro stations. The achieved results show that the classifier choice mentioned above allows one to achieve very good performances also in the presence of high scene complexity.

In the past few years, the development of complex surveillance systems has captured the interest of both the research and industrial worlds. Strong and challenging requirements of modern society are involved in this problem, which aims to increase safety and security in several application domains such as transport, tourism, home and bank security, military applications, etc. At the same time, fast improvements in microelectronics, telecommunications, and computer science make it necessary to consider new perspectives in this field. The main objective of this paper is to investigate, discuss, and evaluate the impact of distributed processing and new communication techniques on multimedia surveillance systems, which represent the so-called third-generation surveillance systems (3 GSSs). In particular, aspects related to the distribution of intelligence among multiple-processing and wide-bandwidth resources are discussed in detail. It is shown how distribution of intelligence can be obtained by a hierarchical architecture that partitions, in a dynamic way, the main logical processing tasks (i.e., representation, recognition, and communication) performed in a 3 GSS physical architecture made up of intelligent cameras, hubs, and central control rooms. The advantages of this solution are pointed out in terms of 1) increased flexibility and reconfigurability and 2) optimal allocation of available processing and bandwidth resources. Finally, a case study is analyzed that allows one to gain a deeper insight into a distributed surveillance system.

An effective real-time post-processing algorithm for error recovery in noise corrupted JPEG bit streams integrated into an existing remote video-surveillance system is presented. The algorithm exploits information extracted by the video-surveillance system in order to detect corrupted frames and to recover them, enhancing the performances of the system, without compromising the real-time behavior of the application. Results show the validity of the presented approach.

This paper proposes a video-surveillance system based on a mobile camera. In particular the developed system creates (during the off-line phase) a panoramic multilayer background image allowing one to use common change detection algorithms to search for a change detection binary image. Different approaches to get the change detection images are presented. The performances of the implemented algorithms are presented by using ROC curves.

A new method for modeling non-rigid objects in a multiple-camera guarded environment is proposed. The statistical model of the shape of a non-rigid object takes into account the correlations between different points of view. The model is automatically generated by processing a typical training set for the considered shape by a principal component analysis algorithm. Snakes and dynamic contours are used to describe the shape of the non-rigid object.

Tracking of non-rigid objects (e.g. humans) is a crucial application for understanding the behavior of objects. Different methods have been presented in literature, whose main drawback is low robustness or high computational load in analysis of cluttered scenes. In the paper a low computational algorithm for tracking non-rigid objects in cluttered scenes is presented. The proposed approach models the shape of the objects by using corners. A learning algorithm is introduced in order to automatically extract the model of the object from a short video sequence acquired immediately before merging of more objects in the scene. The adaptive model extraction mechanism strongly improves method robustness. The method is tested on an existing video-surveillance system in order to track moving objects in cluttered scenes. Results show that the proposed approach gives good performances with low-processing times.

A camera calibration method for video-surveillance applications is presented. The proposed method works on the hypothesis of a fixed TV camera and it is developed in order to minimize the human intervention during the calibration process. For the application, the proposed algorithm needs the 3D measure of only one point in the scene. Other measures are simulated by using a moving object whose geometry is known and by estimating the 3D position of the object by means of an extended Kalman filter. Experimental results show that the proposed algorithm, other than simplify the installation step of video-surveillance systems, considerably improves the accuracy of the calibration with respect to similar algorithms.

In this paper, the problem of extracting and grouping image features from complex scenes is solved by a hierarchical approach based on two main processes: voting and clustering. Voting is performed for assigning a score to both global and local features. The score represents the evidential support provided by input data for the presence of a feature. Clustering aims at individuating a minimal set of significant local features by grouping together simpler correlated observations. It is based on a spatial relation between simple observations on a fixed level, i.e., the definition of a distance in an appropriate space. As the multilevel structure of the system implies that input data for an intermediate level are outputs of the lower level, voting can be seen as a functional representation of the "part-of" relation between features at different abstraction levels. The proposed approach has been tested on both synthetic and real images and compared with other existing feature grouping methods.

In this paper, a surveillance system with automatic video-shot detection and indexing capabilities is presented. The proposed system aims at detecting the presence of abandoned objects in a guarded environment and at automatically performing online semantic video segmentation in order to facilitate the human operator's task of retrieving the cause of an alarm. The former task is performed by operating image segmentation based on temporal rank-order filtering, followed by classification in order to reduce false alarms. The latter task is performed by operating temporal video segmentation when an alarm is detected. In the clips of interest, the key frame is the one depicting a person leaving a dangerous object, and is determined on the basis of a feature indicating the movement around the dangerous region. Experimental results are reported in terms of static region detection, classification, clip and key-frame detection errors versus different levels of complexity of the guarded environment, in order to establish the performance that can be expected from the system in different situations.

Performance evaluation of image processing intermediate results in video based surveillance systems is extremely important due to the variety of approaches to this task. An approach based on the use of receiver operating characteristics (ROC) curves in order to evaluate the performance of a vision complex system for surveillance purposes is presented. The ROC curves have already been used in other research fields such as in the comparison of edge detection algorithms or in the evaluation of artificial neural networks: in this case they are used in order to compare different parameters selections within a system for the localization of moving objects. The presented results show the possibility of using ROC curves as a means for evaluation and comparison of video based surveillance systems.

In this paper, a joint video-shot and layer indexing technique is presented with applications to automatic surveillance of indoor environments. A video-based surveillance system has been developed that simultaneously tracks moving objects and detects the presence of abandoned objects. Whenever an abandoned abject is detected, the system is able to determine the video-shot in which a particular object (layer) appears in the guarded environment, from the first frame in which that object enters in the scene to the frame in which the object has been left. The semantic information related on both the dangerous object and the person who left it, allows the system to perform the video-shot detection and indexing tasks. What is important in a video-shot is the information related to the dangerous object present in it. For this reason a video-shot has a two level indexing: the first one is related to the characteristics of the video-shot and the second one is related to the characteristics of a particular layer.

This work presents a real-time post-processing algorithm developed for enhancing the performances of remote JPEG-based video surveillance applications, seriously degraded by transmission over noisy channels. The aim of the algorithm is to distinguish between blocks changes due to variations in the observed scene and noise-altered blocks, which contain errors due to channel noise and can be corrected exploiting the strong spatio-temporal redundancy of the encoded digital source without any side information. Experimental results, obtained through JPEG transmission simulations performed in the context of a remote video-surveillance system devoted to detection of abandoned objects, show a good improvement both in terms of perceptive quality and of the performance of the overall video-surveillance system.

Describes a probabilistic technique for the coupled reconstruction and restoration of underwater acoustic images. The technique is founded on the physics of the image-formation process. Beamforming, a method widely applied in acoustic imaging, is used to build a range image from backscattered echoes, associated point by point with another type of information representing the reliability (or confidence) of such an image. Unfortunately, this kind of images is plagued by problems due to the nature of the signal and to the related sensing system. In the proposed algorithm, the range and confidence images are modeled as Markov random fields whose associated probability distributions are specified by a single energy function. This function has been designed to fully embed the physics of the acoustic image-formation process by modeling a priori knowledge of the acoustic system, the considered scene, and the noise-affecting measures and also by integrating reliability information to allow the coupled and simultaneous reconstruction and restoration of both images. Optimal (in the maximum a posteriori probability sense) estimates of the reconstructed range image map and the restored confidence image are obtained by minimizing the energy function using simulated annealing. Experimental results show the improvement of the processed images over those obtained by other methods performing separate reconstruction and restoration processes that disregard reliability information.

A probabilistic approach to edge reinforcement is proposed that is based on Bayesian networks of two-dimensional (2-D) fields of variables. The proposed net is composed of three nodes, each devoted to estimating a field of variables. The first node contains available observations. The second node is associated with a coupled random field representing the estimates of the actual values of observed data and of their discontinuities. At the third node, a field of variables is used to represent parameters describing the membership of a discontinuity into a group. The edge reinforcement problem is stated in terms of minimization of local functionals, each associated with a different node, and made up of terms that can be computed locally. It is shown that a distributed minimization is equivalent to the minimization of a global reinforcement criterion. Results concerning the reinforcement of straight lines in synthetic and real images are reported, and applications to synthetic aperture radar (SAR) images are described.

A new set of non-linear signal and image processing operators is presented. Their definition is based on the introduction of the statistical properties of Bayesian reconstruction in soft morphological operators. Statistical soft operators represent a trade-off between the noise cleaning properties of statistical morphology and the shape preservation properties of soft morphology. The main characteristic of these operators is the individualization of two parts within each structuring element (SE) according to soft morphology (i.e. "hard" and "soft" SEs), and to define on this basis a probabilistic estimation model which is a generalization of the statistical morphology model. Results are presented to show that the statistical soft morphological operators can be considered robust to structured noise, i.e. noise showing both statistical (e.g. additive Gaussian noise) and morphological (e.g. noise with a particular shape) structure.

A method for color image restoration based on the concept of Markov random fields and space-filling curves is presented. This work is a vectorial extension of a scalar deterministic solution for Markov random fields (MRFs). The proposed method represents an efficient alternative to the use of the vectorial deterministic solution for MRFs. The application of the space filling curve transformation allows one to apply the MRF algorithm to a scalar image with N/sup 3/ grey levels (typically N=256). The scalar MRF approach is based on expressing the energy function by means of the Euclidean norm in the vectorial space. This approach implies a high computational load. The new method involves a computational load lower than the vectorial case because the energy function is presented in the scalar space obtained after space filling curve based transformation.

This paper proposes a new approach to the problem of intelligently regulating image-processing parameters of a distributed network. The proposed approach is based on two-step probabilistic process: (a) belief updating, which consists in computing a functional cost at each node of the network and, (b) belief maximization, which depends on maximizing this functional cost by using a stochastic optimization algorithm. The architecture of an image processing system, consisting of three modules connected in a chain-like structure, is presented as an example showing the capabilities of the proposed approach. Each module is provided with a priori information about the set of parameters that manage a particular data transformation, and with evaluation criteria to judge data quality and to decide on the parameters to be adjusted. Experimental results obtained by using a digitally controlled camera and lens objective, are presented to show the validity of the proposed approach.

This letter presents a new method for real-time determination of the three-dimensional (3-D) orientation, i.e., the rotation angle with respect to one of the principal object axes, of a moving known object from a monocular image sequence. The method is composed by three steps: 1) extraction of the morphological skeleton from binary images, 2) projection of the skeleton function on two planes and its analytical approximation by nonuniform rational B-splines, and 3) comparison with a set of data stored into a model database. Several experiments performed on real images prove the method's validity.

The availability of a wide set of multidimensional information sources in different application fields (e.g., color cameras, multispectral remote sensing imagery devices, etc.) is the basis for the interest of image processing research on extensions of scalar nonlinear filtering approaches to multidimensional data filtering. A new approach to multidimensional median filtering is presented. The method is structured into two steps. Absolute sorting of the vectorial space based on Peano space filling curves is proposed as a preliminary step in order to map vectorial data onto an appropriate one-dimensional (1-D) space. Then, a scalar median filtering operation is applied. The main advantage of the proposed approach is the computational efficiency of the absolute sorting step, which makes the method globally faster than existing median filtering techniques. This is particularly important when dealing with a large amount of data (e.g., image sequences). Presented results also show that the filtering performances of the proposed approach are comparable with those of vector median filters presented in the literature.

The authors discuss the need to provide a realistic model of a generic noise probability density function (PDF), in order to optimize the signal detection in non-Gaussian environments. The target is to obtain a model depending on a few parameters (that are quick and easy to estimate), and is so general that it is able to describe many kinds of noise (e.g., symmetric or asymmetric, with variable sharpness). To this end, a new HOS-based model is introduced, which is derived from the generalized Gaussian function, and depends on three parameters: kurtosis, for representing variable sharpness, and left and right variances (whose combination provides the same information of skewness) for describing the deviation from symmetry. This model is applied to the design of a locally optimum detection (LOD) test. Promising experimental results are presented which are derived from the application of the test to detecting signals corrupted by real underwater acoustic noise.

The objective of the PASSWORDS Project is to design and develop a prototype of an intelligent video image analysis system for video surveillance and security applications, based on concrete needs expressed by potential users. The goal of the system developed within PASSWORDS is to detect certain dangerous situations in some scenes (e.g. vandalism in a metro station), providing for example a remote operator with an alarm signal. This paper illustrates the different steps which compose the PASSWORDS system, focusing the attention on the image processing module.

A non-linear digital filter is presented in this paper: this filter aims at extending the concept of scalar rank-ordering in the case of multichannel images. The filter is based on two steps: (1) a transformation from a p-dimensional steps to a one-dimensional space by means of a space filling curve; (2) a scalar median filtering step. Results which demonstrate the advantages and the good restoration computational performances of the filter are shown.

The properties and applications of a class of statistical morphological operators, i.e. binary statistical morphology (BSM) operators, for binary image processing are described. The proposed operators are based on quantization of the output of a statistical morphological operator, modeled as a binary probabilistic hypothesis-testing step. The operator obtained is shown to be equivalent to a rank-order filter. Relationships are established between the quantization threshold, rank of the equivalent rank-order filter and parameters of the model. It is also shown that basic BSM operators, i.e. binary statistical dilation and binary statistical erosion can be used as the basis for defining more complex filters. In this paper, attention is paid to describe specific properties of BSM operators which are useful for different applications, e.g. shape description.

The paper describes a method for detecting 2D straight segments and their correspondences in successive frames of an image sequence by means of a Hough-based matching approach. The main advantage of this method is the possibility of extracting and matching 2D straight segments directly in the feature space, without the need for complex matching operations and time-consuming inverse transformations. An additional advantage is that only four attributes of 2D straight segments are required to perform an efficient matching process: position, orientation, length, and midpoint. Tests were performed on both synthetic and real images containing complex man-made objects moving in a scene. A comparison with a well-known 2D line matching algorithm is also made.

A distributed optimization framework and its application to the regulation of the behavior of a network of interacting image processing algorithms are presented. The algorithm parameters used to regulate information extraction are explicitly represented as state variables associated with all network nodes. Nodes are also provided with message-passing procedures to represent dependences between parameter settings at adjacent levels. The regulation problem is defined as a joint-probability maximization of a conditional probabilistic measure evaluated over the space of possible configurations of the whole set of state variables (i.e., parameters). The global optimization problem is partitioned and solved in a distributed way, by considering local probabilistic measures for selecting and estimating the parameters related to specific algorithms used within the network. The problem representation allows a spatially varying tuning of parameters, depending on the different informative contents of the subareas of an image. An application of the proposed approach to an image processing problem is described. The processing chain chosen as an example consists of four modules. The first three algorithms correspond to network nodes. The topmost node is devoted to integrating information derived from applying different parameter settings to the algorithms of the chain. The nodes associated with data-transformation processes to be regulated are represented by an optical sensor and two filtering units (for edge-preserving and edge-extracting filterings), and a straight-segment detection module is used as an integration site.

Recognition-based tracking and image coding methods are often based on different techniques. However, these techniques share the necessity of reliable and fast methods for the representation of the information content of the scene. In this paper, a method based on a lossy shape descriptor is presented which can be used for both recognition and coding purposes. The statistical morphological skeleton provides a noise-robust shape descriptor on which a further approximation phase is performed in order to improve the compression ratio. Then, it is possible to estimate the object's pose through a comparison of the shape descriptor with a set of object models stored in a database. An application to surveillance is presented where the obtained description is used to transmit shape information to a remote control center.

The task of object identification is fundamental to the operations of an autonomous vehicle. It can be accomplished by using techniques based on a Multisensor Fusion framework, which allows the integration of data coming from different sensors. In this paper, an approach to the synergic interpretation of data provided by thermal and visual sensors is proposed. Such integration is justified by the necessity for solving the ambiguities that may arise from separate data interpretations. The architecture of a distributed Knowledge-Based system is described. It performs an Intelligent Data Fusion process by integrating, in an opportunistic way, data acquired with a thermal and a video (b/w) camera. Data integration is performed at various architecture levels in order to increase the robustness of the whole recognition process. A priori models allow the system to obtain interesting data from both sensors; to transform such data into intermediate symbolic objects; and, finally, to recognize environmental situations on which to perform further processing. Some results are reported for different environmental conditions (i.e. a road scene by day and by night, with and without the presence of obstacles).

An acoustic underwater communication problem is considered at low frequency ranges (up to 1 kHz), where propagation performances are improved. This range is dominated by ship noise components and is expected to be non-Gaussian. Classical detector performances may decay in the presence of non-Gaussianity. Spectrum-based and bispectrum-based methods have been used and compared to statistically characterize shipping noise and then to design non-conventional detectors; the main process properties are stationarity, frequency composition and coupling, Gaussianity. Analysis results are reported and refer to real acoustic data recording background traffic and a known-target-ship passage.>

A new class of statistical morphological operators for binary image processing is introduced. These operators are based on a digitized version of the mean field approximation. The main advantage of the new operators is provided by the capability of taking into account both noise and shape information. Binary statistical dilation (BSD) and binary statistical erosion (BSE) are considered as a case study. Extensivity properties of BSD and BSE are also discussed.>

A real-time visual surveillance system is based on three main image processing phases, devoted to extract information about the observed scene: change detection, focus of attention, feature-extraction. In this paper attention is paid to a theory (i.e., binary statistical morphology) which provides a common framework for designing fast and noise-robust methods for the three tasks of interest. The main theoretical novelty is to establish a link between binary statistical morphology and voting methods. An application is presented which deals with intruder detection in a railway-crossing area.>

The architecture of a distributed vision system (DVS) based on a combination of multiple modules of standard and extended Kalman filters is presented. It exploits a representation of static and dynamic knowledge for estimation purposes. Spatial constraints describe how observed image features lead to estimate parameters (i.e., in the present application, the density and position of monitored people in the monitored scene); time constraints are used to describe knowledge on dynamic evolution of the mentioned estimated variables. Using dynamic knowledge allows the system to track groups of people, dynamically interacting each others, on the image plane over time. Experimental results, deriving from an extensive test phase carried out on real-life images of an underground station, confirm that integration of different spatial and temporal constraints is an efficient approach for optimizing parameter estimation in DVSs.>

A probabilistic method for obtaining a complete image representation on the basis of spatial-temporal knowledge is presented. The main goal of the algorithm is to obtain a consistent segmentation of a noisy image sequence. Consistent means that the same region must maintain the same label in all consequent images of the sequence where it appears. To this end, a processing scheme is presented which extends Bayesian networks of Gibbs-Markov random fields (GMRF) to segmentation of dynamic scenes.>

The problem of analyzing a sequence of images by taking into account the symbolic and numerical content of the signal is considered. An algorithm for segmentation and tracking of regions among images of a sequence is presented. The method is based on a Gibbs Markov random field (GMRF) model which couples the image process to a spatial-temporal region process. Optical flow field is used to adaptively decide the temporal clique to be used during annealing of energy. Displacement vectors of pixels belonging to recognized regions are predicted by using prior knowledge about object behavior.>

A knowledge-based vision system called DOORS (Distributed Object-Oriented Recognition System), for obstacle detection and tracking, is presented. It integrates multisensory information sources by adaptively selecting appropriate fusion strategies at the abstraction levels of both physical and virtual sensors. The basic methodology adopted for data representation and processing control has several points in common with object-oriented programming techniques and with blackboard approaches. Some preliminary results are presented for the case of an obstacle moving on a country-road scene.>