Autonomous vehicles (AVs) must be able to locate themselves accurately in dynamic environments in order to be able to navigate safely. In this paper, we present a robust LiDAR-based framework to improve the localization of AVs based on the classification of static and dynamic tracks. The approach leverages static tracks previously classified as reliable landmarks. Using Growing Neural Gas, Joint Probabilistic Data Association, and Unmotivated Kalman Filter algorithms, interaction dictionaries and vocabularies are generated. These serve as inputs for a Markov Jump Particle Filter (MJPF), which enables the accurate estimation of the ego-vehicle trajectory. Testing of the proposed framework with real-world LiDAR data demonstrates that it is capable of providing highly accurate localization results without the use of odometry, making it adaptable for use in environments without GPS. AV situational awareness and navigation performance are enhanced by using static tracks as robust references.

The integration of drones in health informatics is transforming real-time health data collection and surveillance, particularly in high-risk construction environments. Construction workers are frequently exposed to hazardous conditions, including heat stress, airborne pollutants, and physical fatigue, necessitating continuous health monitoring to prevent accidents and long-term health complications. This paper explores the deployment of drone-based systems equipped with advanced sensors, including thermal imaging, gas detectors, and real-time physiological monitoring, to assess workers’ health conditions remotely. By leveraging AI-driven analytics and cloud-based data processing, these drones provide instant alerts for abnormal health indicators, enabling timely intervention. Unlike conventional monitoring methods that rely on periodic manual checks, drones offer a scalable and autonomous solution, enhancing both worker safety and operational efficiency. The study presents a framework for integrating drones with wearable health sensors, outlining their potential to revolutionize workplace health surveillance in construction fields. Key challenges, including data privacy, regulatory compliance, and system reliability, are also discussed, along with future directions for improving drone-assisted health monitoring.

Learning from experience is a fundamental capability of intelligent agents. Autonomous systems rely on sensors that provide data about the environment and internal situations to their perception systems for learning and inference mechanisms. These systems can also learn Self-Aware and Situation-Aware generative modules from these data to localize themselves and interact with the environment. In this paper, we propose a self-aware cognitive architecture capable to perform tasks where the interactions between the self-state of an agent and the surrounding environment are explicitly and dynamically represented. We specifically develop a Deep Learning (DL) based Self-Aware interaction model, empowered by learning from Multi-Modal Perception (MMP) and World Models using multi-sensory data in a novel Multi-Agent Self-Awareness Architecture (MASAA). Two sub-modules are developed, the Situation Model (SM) and the First-Person model (FPM), that address different and interrelated aspects of the World Model (WM). The MMP model, instead, aims at learning the mapping of different sensory perceptions into Exteroceptive (EI) and Proprioceptive (PI) latent information. The WM then uses the learned MMP model as experience to predict dynamic self-behaviors and interaction patterns within the experienced environment. WM and MMP Models are learned in a data-driven way, starting from the lower-dimensional odometry data used to guide the learning of higher-dimensional video data, thus generating coupled Generalized State Hierarchical Dynamic Bayesian Networks (GS-HDBNs). We test our model on KITTI, CARLA, and iCab datasets, achieving high performance and a low average localization error (RMSE) of 2.897%, when considering two interacting agents.

Generalizability and interpretability are common terminologies that can be found in today’s machine learning algorithm design. Generalizability requires a clear understanding of one’s own action (self-awareness) and a robust interaction with the environment (situation awareness). Many current studies are devoted in developing an algorithm that is more robust in generalizing unseen situations while explaining self-action. However, such algorithms are complex and are not yet fully developed to be used in production. Intelligent transportation systems like self-driving cars are one of the emerging technologies that need generalizability and explainability in anomalous conditions. We propose to enhance generalizability and interpretability of a self-driving car model by introducing a novel methodology that fuses multi-sensorial data from proprioceptive and exteroceptive sensors of an agent, coupled in a Hierarchical Dynamic Bayesian Network model, in an Active Inference framework. The developed model has three stages: 1) a lower dimensional unsupervised learning stage, considering odometry and action modalities, carried out by first applying Null Force Filtering and then by applying modified GNG clustering algorithms; 2) a self-supervised higher-dimensional video modality learning stage assisted by the learned odometry vocabularies; and 3) an online model-based active learning in continuous and discrete state spaces, and action spaces, in the Active Inference framework. The developed system is tested using the CARLA simulator environment for localizing interacting agents, and exhibits low error compared to state-of-the-art methods.

The evolution of autonomous vehicles (AVs) requires a paradigm shift towards the integration of human factors to improve safety and efficiency at levels 2,3 and 4 of automation. This paper presents a comparison of three different fusion technologies (Low-Level fusion, Medium-Level fusion, and a hybrid fusion), highlighting the critical role of multimodal data integration and semi-supervised learning in predicting and adapting to levels of driver awareness. Our approach uses semi-supervised learning to deal with the data labelling problem, using unlabelled data to train an autoencoder and sparsely labelled data to train a 4-state classifier. Our model facilitates the fusion of data from different physiological signals, including skin electrodermal activity, heart rate, body temperature and acceleration. Using real driving data, the Medium-Level fusion approach gives the best performance, achieving 84% accuracy in predicting situations where the user may not be aware enough to take control of the vehicle. This research highlights the essential nature of fusion technologies to create adaptive and user-centred AV systems.

Intelligent transportation systems (ITSs) provide a paradigm change in perceiving and interacting with transportation networks, leading to enhanced levels of safety, sustainability, and efficiency. Vehicular-to-everything (V2X) communication is the core component in the ITSs. The proprioceptive and exteroceptive sensors allow these vehicles to be aware of the surrounding environment and respond to emergencies by utilizing their abilities to reach a high level of self-awareness. In this paper, we propose a self-awareness approach to learn a generative dynamic Bayesian network (G-DBN) from the real-time LiDAR perception. Without reducing the dimensionality, we perform offline training and online testing phases on the three-dimensional (3D) point clouds. In the offline training phase, initially, the raw point clouds are preprocessed using a joint probabilistic data association filter (JPDAF) to obtain the 3D tracks of the multiple vehicles in space. Then, we perform an unsupervised clustering on all the generalized states (GSs) containing positions and velocities (a 6D vector) by considering the growing neural gas (GNG) technique, thus achieving a trained model from the 3D LiDAR point clouds. In the online testing phase, the high-dimensional Markov jump particle filter (HD-MJPF) utilizes the G-DBN’s probabilistic information to predict the positions of multiple vehicles and to detect the abnormalities at the discrete and continuous levels in normal and abnormal scenarios. Our proposed approach is useful for learning high-dimensional generative models and provides a way to meet the current curse of dimensionality challenges, that machine learning models are suffering.

Active inference is a probabilistic framework for modeling intelligent agent behaviours, which drives by the principle of minimizing free energy. In this paper, we integrate the imitation learning method with active inference to minimize the expected free energy under the supervision of an expert model. The proposed approach affords explainable decision-making as a combination of self-information and novelty-seeking or exploratory behavior in a hierarchical generative model. A lane-changing driving scenario is demonstrated to verify the efficiency of the proposed framework that outperforms conventional Reinforcement learning methods.

The combination of different sensory information to predict upcoming situations is an innate capability of intelligent beings. Consequently, various studies in the Artificial Intelligence field are currently being conducted to transfer this ability to artificial systems. Autonomous vehicles can particularly benefit from the combination of multi-modal information from the different sensors of the agent. This paper proposes a method for video-frame prediction that leverages odometric data. It can then serve as a basis for anomaly detection. A Dynamic Bayesian Network framework is adopted, combined with the use of Deep Learning methods to learn an appropriate latent space. First, a Markov Jump Particle Filter is built over the odometric data. This odometry model comprises a set of clusters. As a second step, the video model is learned. It is composed of a Kalman Variational Autoencoder modified to leverage the odometry clusters for focusing its learning attention on features related to the dynamic tasks that the vehicle is performing. We call the obtained overall model Cluster-Guided Kalman Variational Autoencoder. Evaluation is conducted using data from a car moving in a closed environment and leveraging a part of the University of Alcalá DriveSet dataset, where several drivers move in a normal and drowsy way along a secondary road.

This paper proposes a multi-modal situational inter-action model for collaborative agents by fusing multi-sensorial information in a Multi-Agent Hierarchical Dynamic Bayesian Network (MAH-DBN) framework. The proposed model is learned in a data-driven methodology to estimate the states of interacting agents only from video sequences. This can be regarded as a two-fold methodology for improving visual-based localization and interaction between autonomous agents. In the learning stage, the odometry model is used to drive the video learning model for a robust localization and interaction modeling. During the testing phase, the learned Multi-Agent Hierarchical DBN (MAH-DBN) model is used for the localization of collaborative agents only from video sequences by proposing an inference method called Multi-Agent Coupled Markov Jump Particle Filter (MAC-MJPF).

This paper proposes a method to localize a moving agent - such as an autonomous surveillance vehicle - inside a known environment using First Person Viewpoint video data. Anomalies w.r.t. expected vehicle motion and image content are extracted to guide the localization, signal when the localization results are not trustworthy and explain the reason for the failure. During the training phase, a Dynamic Bayesian Network model is learned, which couples positional and video data. To learn it, clustering is performed on the odometry data, and a modified Kalman Variational Autoencoder is built over the video data. During the testing phase, a Coupled Markov Jump Particle Filter leverages the learned Dynamic Bayesian Network to extract anomalies and to estimate the vehicle’s position, given only camera data. The proposed method is evaluated on two real-world datasets of a vehicle performing perimeter monitoring of a closed environment and of a shopping cart moving in a supermarket.

This paper proposes an algorithm to model and process streams of LiDAR data under an autonomous vehicle framework. LiDAR is assumed to be an exteroceptive sensor that allows the vehicle to have dynamic 3D scene perception of its surroundings. We employ an encoder-decoder architecture based on 3D-Convolutional layers called 3D Convolution Encoder-Decoder (3D-CED), together with a transfer learning strategy to extract a set of features from point clouds, which are relevant in the context of autonomous driving. The resulting features allow to make inferences of the future point cloud data and detect multiple abstraction level anomalies in controlled scenarios by utilizing a probabilistic switching dynamic model called High Dimensional Markov Jump Particle Filter (HD-MJPF). Moreover, a comparison is provided between piecewise linear, piecewise nonlinear, and nonlinear predictive models for anomaly detection at multiple abstraction levels. Our approach is evaluated with data collected from the LiDAR sensors of the autonomous vehicle while performing certain tasks in a controlled environment.

A novel approach is proposed for multimodal collective awareness (CA) of multiple networked intelligent agents. Each agent is here considered as an Internet-of-Things (IoT) node equipped with machine learning capabilities; CA aims to provide the network with updated causal knowledge of the state of execution of actions of each node performing a joint task, with particular attention to anomalies that can arise. Data-driven dynamic Bayesian models learned from multisensory data recorded during the normal realization of a joint task (agent network experience) are used for distributed state estimation of agents and detection of abnormalities. A set of switching dynamic Bayesian network (DBN) models collectively learned in a training phase, each related to particular sensorial modality, is used to allow each agent in the network to perform synchronous estimation of possible abnormalities occurring when a new task of the same type is jointly performed. Collective DBN (CDBN) learning is performed by unsupervised clustering of generalized errors (GEs) obtained from a starting generalized model. A growing neural gas (GNG) algorithm is used as a basis to learn the discrete switching variables at the semantic level. Conditional probabilities linking nodes in the CDBN models are estimated using obtained clusters. CDBN models are associated with a Bayesian inference method, namely, distributed Markov jump particle filter (D-MJPF), employed for joint state estimation and abnormality detection. The effects of networking protocols and of communications in the estimation of state and abnormalities are analyzed. Performance is evaluated by using a small network of two autonomous vehicles performing joint navigation tasks in a controlled environment. In the proposed method, first the sharing of observations is considered in ideal condition, and then the effects of a wireless communication channel have been analyzed for the collective abnormality estimation of the agents. Rician wireless channel and the usage of two protocols (i.e., IEEE 802.11p and IEEE 802.15.4) along with different channel conditions are considered as well.

The technology for autonomous vehicles is close to replacing human drivers by artificial systems endowed with high-level decision-making capabilities. In this regard, systems must learn about the usual vehicle's behavior to predict imminent difficulties before they happen. An autonomous agent should be capable of continuously interacting with multi-modal dynamic environments while learning unseen novel concepts. Such environments are not often available to train the agent on it, so the agent should have an understanding of its own capacities and limitations. This understanding is usually called self-awareness. This paper proposes a multi-modal self-awareness modeling of signals coming from different sources. This paper shows how different machine learning techniques can be used under a generic framework to learn single modality models by using Dynamic Bayesian Networks. In the presented case, a probabilistic switching model and a bank of generative adversarial networks are employed to model a vehicle's positional and visual information respectively. Our results include experiments performed on a real vehicle, highlighting the potentiality of the proposed approach at detecting abnormalities in real scenarios.

The advancements in connected and autonomous vehicles in these times demand the availability of tools providing the agents with the capability to be aware and predict their own states and context dynamics. This article presents a novel approach to develop an initial level of collective awareness (CA) in a network of intelligent agents. A specific collective self-awareness functionality is considered, namely, agent-centered detection of abnormal situations present in the environment around any agent in the network. Moreover, the agent should be capable of analyzing how such abnormalities can influence the future actions of each agent. Data-driven dynamic Bayesian network (DBN) models learned from time series of sensory data recorded during the realization of tasks (agent network experiences) are here used for abnormality detection and prediction. A set of DBNs, each related to an agent, is used to allow the agents in the network to reach synchronously aware possible abnormalities occurring when available models are used on a new instance of the task for which DBNs have been learned. A growing neural gas (GNG) algorithm is used to learn the node variables and conditional probabilities linking nodes in the DBN models; a Markov jump particle filter (MJPF) is employed for state estimation and abnormality detection in each agent using learned DBNs as filter parameters. Performance metrics are discussed to asses the algorithm's reliability and accuracy. The impact is also evaluated by the communication channel used by the network to share the data sensed in a distributed way by each agent of the network. The IEEE 802.11p protocol standard has been considered for communication among agents. Performances of the DBN-based abnormality detection models under different channel and source conditions are discussed. The effects of distances among agents and of the delays and packet losses are analyzed in different scenario categories (urban, suburban, and rural). Real data sets are also used acquired by autonomous vehicles performing different tasks in a controlled environment.

This paper proposes a method to detect abnormal motions in real vehicle situations based on trajectory data. Our approach uses a Gaussian process (GP) regression that facilitates to approximate expected vehicle's movements over a whole environment based on sparse observed data. The main contribution of this paper consists in decomposing the GP regression into spatial zones, where quasi-constant velocity models are valid. Such obtained models are employed to build a set of Kalman filters that encode observed vehicle's dynamics. This paper shows how proposed filters enable the online identification of abnormal motions. Detected abnormalities can be modeled and learned incrementally, automatically by intelligent systems. The proposed methodology is tested on real data produced by a vehicle that interacts with pedestrians in a closed environment. Automatic detection of abnormal motions benefits the traffic scene understanding and facilitates to close the gap between human driving and autonomous vehicle awareness.

This paper presents a novel approach for learning self-awareness models for autonomous vehicles. Proposed technique is based on the availability of synchronized multi-sensor dynamic data related to different maneuvering tasks performed by a human operator. It is shown that different machine learning approaches can be used to first learn single modality models using coupled Dynamic Bayesian Networks; such models are then correlated at event level to discover contextual multimodal concepts. In the presented case, visual perception and localization are used as modalities. Cross-correlations among modalities in time is discovered from data and are described as probabilistic links connecting shared and private multi-modal DBNs at the event (discrete) level. Results are presented on experiments performed on an autonomous vehicle, highlighting potentiality of the proposed approach to allow anomaly detection and autonomous decision making based on learned self-awareness models.

This paper focuses on multi-sensor anomaly detection for moving cognitive agents using both external and private first-person visual observations. Both observation types are used to characterize agents motion in a given environment. The proposed method generates locally uniform motion models by dividing a Gaussian process that approximates agents displacements on the scene and provides a Shared Level (SL) self-awareness based on Environment Centered (EC) models. Such models are then used to train in a semi-unsupervised way a set of Generative Adversarial Networks (GANs) that produce an estimation of external and internal parameters of moving agents. Obtained results exemplify the feasibility of using multi-perspective data for predicting and analyzing trajectory information.

In recent years several architectures have been proposed to learn embodied agents complex self-awareness models. In this paper, dynamic incremental self-awareness (SA) models are proposed that allow experiences done by an agent to be modeled in a hierarchical fashion, starting from more simple situations to more structured ones. Each situation is learned from subsets of private agent perception data as a model capable to predict normal behaviors and detect abnormalities. Hierarchical SA models have been already proposed using low dimensional sensorial inputs. In this work, a hierarchical model is introduced by means of a cross-modal Generative Adversarial Networks (GANs) processing high dimensional visual data. Different levels of the GANs are detected in a self-supervised manner using GANs discriminators decision boundaries. Real experiments on semi-autonomous ground vehicles are presented.

This paper presents a method for observational learning in autonomous agents. A formalism based on deep learning implementations of variational methods and Bayesian filtering theory is presented. It is explained how the proposed method is capable of modeling the environment to mimic behaviors in an observed interaction by building internal representations and discovering temporal and causal relations. The method is evaluated in a typical surveillance scenario, i.e., perimeter monitoring. It is shown that the vehicle learns how to drive itself by simultaneously observing its surroundings and the actions taken by a human driver for a given task. That is achieved by embedding knowledge regarding perception-action couplings in dynamic representational states used to produce action flows. Thereby, representations link sensory data to control signals. In particular, the representational states associate visual features to stable action concepts such as turning or going straight.

This paper presents an innovative smartphone application which takes advantage of the availability of smartphone technologies to develop a highly accurate pedestrian detection and tracking application, based on monocular camera and embedded sensors. Monocular pedestrian detection is performed based on well-known vision approaches, later pose estimation is used to correct the location of the pedestrian, based on the information provided by the mobile device, and finally movement is estimated by means of a Kalman Filter. The information from the internal sensors corrects the distance given by the pin-hole model. Test performed proved the viability of the system as well as its accuracy.

Robust real-time obstacle detection/avoidance is a challenging problem especially for micro and small aerial vehicles due to the limited number of the on-board sensors due to the battery constraint and low payload. Usually lightweight sensors such as CMOS camera are the best choice comparing with laser or radar sensors. For real-time applications, most studies focus on using stereo cameras to reconstruct a 3D model of the obstacles or to estimate their depth. Instead, in this paper, a method that mimics the human behavior of detecting the state of the approaching obstacles using single camera is proposed. During the flight, this method is able to detect the changes of the size area of the obstacles. First, the method detects the feature points of the obstacles, and then extracts the obstacles that has probability of getting close. In addition, by comparing the changes in the area ratios of the obstacle in the image sequence, the method can decide if it is obstacle or not. Finally, by estimating the obstacle 2D position in the image and combining with the tracked waypoints, the UAV can take the action of avoidance.

The work presented describes a tool for driver behavior analysis. The proposal offers a contribution based on Gaussian Mixture Model (GMM) modeling technique. GMM is a powerful tool for the statistical modeling, which allows to identify specific patterns of behavior in comparison with available data. Data are obtained by the in-vehicle sensors using Controller Area Network bus (CAN bus), an Inertial Measurement Unit (IMU) and a GPS. These data allow to provide driver behavior analysis and aggressive behavior identification. This development has been tested in real-traffic situations with different drivers.

During the last decade, ground mobile robots that are able to drive autonomously in off-road environments have received a great deal of attention. Autonomous navigation in unstructured environments faces many new challenges compared to structured urban environments, these challenges increase the complexity of the localization, obstacle detection, path planning and navigation commands. Accordingly this paper presents a fusion system for stereo-vision and laser-rangefinder outdoor obstacle detection, which is implemented as an application for autonomous off-road navigation. The test platform is an electric golf-cart that is modified mechanically and electrically to operate in driver-less mode. This vehicle is equipped with binocular camera, laser-rangefinder, electronic compass and on-board embedded computer, which operates using Robotic Operating System (ROS) architecture. The proposed architecture gathers the data from all different sensors, in order to make navigation decisions from one point to another, avoiding obstacles in the path. Experimental results indicate the high performance of the proposed approaches, they show that the perception from the stereo-vision detection enhances the laser-rangefinder detection, which consequently makes a better decision in maneuvering the obstacle and returns back to the original path.

This paper presents an autocalibration method to determine the pose of a stereo vision system based on knowing the geometry of the ground in front of the cameras. This pose changes considerably while the vehicle is driven, therefore it is good to know constantly the pose of the camera for several applications based on computer vision, such as advanced driver assistance systems, autonomous vehicles or robotics. These constant changes of the pose make interesting to be able to detect constantly the variations in its extrinsic parameters (height, pitch, roll). The validation of the autocalibration method is accomplished by a visual odometry implementation. A study of the improvement of the results of the visual odometry estimation taking into account the changes of the camera pose is presented, demonstrating the advantages of the autocalibration method.

The movement of the vehicle provides useful information for different applications, such as driver assistant systems or autonomous vehicles. This information can be known by means of a GPS, but there are some areas in urban environments where the signal is not available, as tunnels or streets with high buildings. A new method for 2D visual ego motion estimation in urban environments is presented in this paper. This method is based on a stereo-vision system where the feature road points are tracked frame to frame in order to estimate the movement of the vehicle, avoiding outliers from dynamic obstacles. The road profile is used to obtain the world coordinates of the feature points as a unique function of its left image coordinates. For these reasons it is only necessary to search feature points in the lower third of the left images. Moreover, the Kalman filter is used as a solution for filtering problem. That is, in some cases, it is necessary to filter raw data due to noise acquisition of time series. The results of the visual ego motion are compared with raw data from a GPS.

Abstract:This work presents an approach to dynamic surface inspection in laminated materials based on the configuration of a visual system to obtain good quality control of the manufacturing surface. It aims to overcome some of the limitations of the single‐use visual inspection systems by integrating and differentiating knowledge. The configuration task for surface inspection is solved as a Configuration‐Design task according with the CommonKADS methodology. This task is analysed at the knowledge level and is decomposed into simple subtasks to reach the inference level. The generic domain knowledge involved in the configuration and revision tasks is differentiated in six types: initial, global, environment, real‐time, image quality and computer vision techniques. The main goal of the proposed architecture is to reuse this partitioned knowledge for the configuration and revision of different visual surface inspection systems. The architecture has been implemented in a platform that distinguishes several operation modes: visualization of specific surface inspection results, system configuration and knowledge injection. Results obtained in the configuration and revision tasks for two surface inspection systems, stainless‐steel and wood, are presented.