Abstract Causality has become a fundamental approach for explaining the relationships between events, phenomena, and outcomes in various fields of study. It has invaded various fields and applications, such as medicine, healthcare, economics, finance, fraud detection, cybersecurity, education, public policy, recommender systems, anomaly detection, robotics, control, sociology, marketing, and advertising. In this paper, we survey its development over the past five decades, shedding light on the differences between causality and other approaches, as well as the preconditions for using it. Furthermore, the paper illustrates how causality interacts with new approaches such as Artificial Intelligence (AI), Generative AI (GAI), Machine and Deep Learning, Reinforcement Learning (RL), and Fuzzy Logic. We study the impact of causality on various fields, its contribution, and its interaction with state-of-the-art approaches. Additionally, the paper exemplifies the trustworthiness and explainability of causality models. We offer several ways to evaluate causality models and discuss future directions.

This article introduces a novel method for distilling knowledge from complex models using fuzzy systems. The complex knowledge comes from a proposed hybrid NFN-LSTM model (teacher) composed of a long shor-term memory (LSTM) coupled to a neo-fuzzy neuron (NFN) structure. The proposed student model, the NFN-MOD, is an explainable Takagi–Sugeno fuzzy model that resembles modular characteristics to mimic the temporal memory of the LSTM part in the teacher model. The NFN-MOD is adaptable across many scenarios, including solo learning (without a teacher), with the estimation of a previously trained teacher, or training in parallel with the teacher. The complexity reduction of the student model is achieved through the pruning of its consequent parameters with the lowest L1-norm. Application of NFN-MOD in industrial case studies (sulfur recovery unit and cement manufacturing process) demonstrates the efficiency of NFN-MOD in distilling complex knowledge from the teacher model NFN-LSTM, with emphasis on parallel training and parameter pruning. In addition, a novel explainability analysis is introduced, which evaluates the influence of antecedent parameters of the student model in relation to the expected real system output.

3D LiDAR-based place recognition has been extensively researched in urban environments, yet it remains underexplored in agricultural settings. Unlike urban contexts, horticultural environments, characterized by their permeability to laser beams, result in sparse and overlapping LiDAR scans with suboptimal geometries. This phenomenon leads to intra-and inter-row descriptor ambiguity. In this work, we address this challenge by introducing SPVSoAP3D, a novel modeling approach that combines a voxel-based feature extraction network with an aggregation technique based on a second-order average pooling operator, complemented by a descriptor enhancement stage. Furthermore, we augment the existing HORTO-3DLM dataset by introducing two new sequences derived from horticultural environments. We evaluate the performance of SPVSoAP3D against state-of-the-art (SOTA) models, including OverlapTransformer, PointNetVLAD, and LOGG3D-Net, utilizing a cross-validation protocol on both the newly introduced sequences and the existing HORTO-3DLM dataset. The findings indicate that the average operator is more suitable for horticultural environments compared to the max operator and other first-order pooling techniques. Additionally, the results highlight the improvements brought by the descriptor enhancement stage. The code is publicly available at https://github.com/Cybonic/SPVSoAP3D.git

Abstract Machine Learning (ML) has attracted great interest in the modeling of systems using computational learning methods, being utilized in a wide range of advanced fields due to its ability and efficiency to process large amounts of data and to make predictions or decisions with a high degree of accuracy. However, with the increase in the complexity of the models, ML's methods have presented complex structures that are not always transparent to the users. In this sense, it is important to study how to counteract this trend and explore ways to increase the interpretability of these models, precisely where decision‐making plays a central role. This work addresses this challenge by assessing the interpretability and explainability of fuzzy‐based models. The structural and semantic factors that impact the interpretability of fuzzy systems are examined. Various metrics have been studied to address this topic, such as the Co‐firing Based Comprehensibility Index (COFCI), Nauck Index, Similarity Index, and Membership Function Center Index. These metrics were assessed across different datasets on three fuzzy‐based models: (i) a model designed with Fuzzy c‐Means and Least Squares Method, (ii) Adaptive‐Network‐based Fuzzy Inference System (ANFIS), and (iii) Generalized Additive Model Zero‐Order Takagi‐Sugeno (GAM‐ZOTS). The study conducted in this work culminates in a new comprehensive interpretability metric that covers different domains associated with interpretability in fuzzy‐based models. When addressing interpretability, one of the challenges lies in balancing high accuracy with interpretability, as these two goals often conflict. In this context, experimental evaluations were performed in many scenarios using 4 datasets varying the model parameters in order to find a compromise between interpretability and accuracy.

Multispectral images and leaf-clip measurements were used for evaluating the Chlorophyll (Chl) content on grapevine leaves through the use of vegetation indices. The multispectral leaf images were taken by a sensor onboard an UAS, which was placed over a table at a height of 70 cm. Images were radiometrically and geometrically processed to obtain accurate coregistered five band image stacks. Using a Kmeans segmentation, each leaf imaged in a multispectral image was automatically detected and the mean leaf reflectance values were used to compute three vegetation indices: NDVI, NDRE, and GLI. When compared to leaf-clip measurements, the results indicated that the NDRE index was the best predictor of Chl leaf content (R2=0.81). The obtained NDRE regression model can be used in UAS-based multispectral orthomosaics to generate canopy Chl maps at vine-row scale, which can assist vine growers in monitoring the spatial and temporal variability of grapevine vigor.

Intention prediction (IP) is a challenging task for intelligent vehicle’s perception systems. IP provides the likelihood, or probability, of a target vehicle to perform a maneuver subjected to a finite set of possibilities. There are many factors that influence the decision-making process of a driver, which should be considered in a prediction framework. In addition, the lack of labeled large-scale dataset with maneuver annotation imposes another challenge to the task. This paper proposes an Interaction-aware Maneuver Prediction framework, called IAMP, using interaction graphs to extract complex interaction features from traffic scenes. In addition, we explored a semi-supervised approach called Noisy Student to take advantage of unlabeled data in the training step. Experimental results show relevant improvement when using unlabeled data, increasing the average performance of a classifier by 7.17% of accuracy. Moreover, this approach also made it possible to obtain an intention predictor with similar results to a classifier., even when using a shorter observation horizon.

Object detection in autonomous driving applications implies the detection and tracking of semantic objects that are commonly native to urban driving environments, as pedestrians and vehicles. One of the major challenges in state-of-the-art deep-learning based object detection are false positives which occur with overconfident scores. This is highly undesirable in autonomous driving and other critical robotic-perception domains because of safety concerns. This paper proposes an approach to alleviate the problem of overconfident predictions by introducing a novel probabilistic layer to deep object detection networks in testing. The suggested approach avoids the traditional Sigmoid or Softmax prediction layer which often produces overconfident predictions. It is demonstrated that the proposed technique reduces overconfidence in the false positives without degrading the performance on the true positives. The approach is validated on the 2D-KITTI objection detection through the YOLOV4 and SECOND (Lidar-based detector). The proposed approach enables interpretable probabilistic predictions without the requirement of re-training the network and therefore is very practical.

This paper presents a novel feature selection method based on the conditional mutual information (CMI). The proposed High Order Conditional Mutual Information Maximization (HOCMIM) method incorporates high order dependencies into the feature selection procedure and has a straightforward interpretation due to its bottom-up derivation. The HOCMIM is derived from the CMI’s chain expansion and expressed as a maximization optimization problem. The maximization problem is solved using a greedy search procedure, which speeds up the entire feature selection process. The experiments are run on a set of benchmark datasets (20 in total). The HOCMIM is compared with eighteen state-of-the-art feature selection algorithms, from the results of two supervised learning classifiers (Support Vector Machine and K-Nearest Neighbor). The HOCMIM achieves the best results in terms of accuracy and shows to be faster than high order feature selection counterparts.

Deep learning on 3D point clouds has drawn much attention, due to its large variety of applications in intelligent perception for automated and robotic systems. Unlike structured 2D images, it is challenging to extract features and implement convolutional networks over these unordered points. Although a number of previous works achieved high accuracies for point cloud recognition, they tend to process local point information in such a way that semantic information is not fully encoded. In this paper, we propose a deep neural network for 3D point cloud processing that utilizes effective feature aggregation methods emphasizing both generalizability and relevance. In particular, our method uses fixed-radius grouping for pooling layers and spherical kernel convolution for semantics mining. To address the issue of gradient degradation and memory consumption of a deep network, a parallel feature feed-forward mechanism and bottleneck layers are implemented to reduce the number of parameters. Experiments show that our algorithm achieves state-of-the-art results and competitive accuracy in both classification and part segmentation while maintaining an efficient architecture.

This paper proposes a methodology for iterative learning of multi-input multi-output (MIMO) fuzzy models focusing on dynamic system identification. The first step of the proposed method is the learning of the antecedent part of the fuzzy system, which is learned iteratively, where fuzzy rules can be added or merged based on the presented novelty detection and similarity criteria defined by a recursive extension of the Gath-Geva clustering algorithm. Then, the consequent part consists in the direct implementation of a non-recursive fuzzy approach that uses global least squares, Observer Kalman Filter Identification (OKID) and the Eigensystem Realization Algorithm (ERA). The proposed method is validated using experimental data from a real quadrotor aerial robot, a nonlinear dynamic system. Using quantitative performance metrics, the proposed method is compared with Hammerstein-Wiener models (H.-W.), nonlinear autoregressive models with exogenous input (NARX), and state-space models using subspace method with time-domain data (N4SID), other MIMO system identification techniques. The proposed method achieved better results compared to other techniques, showing the importance and versatility of learning based on novelty detection for MIMO problems.

The novelty of this study consists in a multi-modality approach to scene classification, where image and audio complement each other in a process of deep late fusion. The approach is demonstrated on a difficult classification problem, consisting of two synchronised and balanced datasets of 16,000 data objects, encompassing 4.4 hours of video of 8 environments with varying degrees of similarity. We first extract video frames and accompanying audio at one second intervals. The image and the audio datasets are first classified independently, using a fine-tuned VGG16 and an evolutionary optimised deep neural network, with accuracies of 89.27% and 93.72%, respectively. This is followed by late fusion of the two neural networks to enable a higher order function, leading to accuracy of 96.81% in this multi-modality classifier with synchronised video frames and audio clips. The tertiary neural network implemented for late fusion outperforms classical state-of-the-art classifiers by around 3% when the two primary networks are considered as feature generators. We show that situations where a single-modality may be confused by anomalous data points are now corrected through an emerging higher order integration. Prominent examples include a water feature in a city misclassified as a river by the audio classifier alone and a densely crowded street misclassified as a forest by the image classifier alone. Both are examples which are correctly classified by our multi-modality approach.

This paper introduces a new mobile robot localization solution consisting of two main modules: a Particle-Filter based Localization (PFL) and a Reinforcement-Learning based map updating, integrating relative measurements and absolute indoor positioning sensor (A-IPS) data. Concerning localization using 2D-LiDARs, featureless areas are known to be problematic. To solve this problem a classic PFL approach was modified to incorporate A-IPS position measurements in the prediction and update stages. The localization approach has the particularity of including the possibility of updating the map whenever major modifications are detected in the environment in relation to the current localization map. Due to the random sampling-based nature of the PFL, an associated map update solution is not trivial since small inconsistencies in the estimated pose can lead to erroneous map associations. The proposed method learns to decide by assigning higher rewards the greater is the overlap between the map and the 2DLIDAR scans, via RL, and then a proper update of the map is achieved. Validation of the proposed pipeline was carried out in a differential drive platform with algorithms developed in ROS. Tests were performed in two scenarios in order to assess the performance of both the localization module and the map update stage. The results show that the proposed localization method offers improvements in relation to known approaches, and consequently suggest promising perspectives for the proposed map update decision framework.

Connected and autonomous/automated vehicle (CAV) technologies are shaping the design and the new developments in the automotive industry and, in a wider perspective, in the mobility sector as well. Despite the recent advances and on-going developments, and the enthusiasm around autonomous mobility systems, real-world testing of CAVs is a crucial element to allow the next generation of intelligent vehicles to come to our daily-life. The importance of realistic testing is recognized by academia, industry, public sector and stakeholders, and is reflected in all projects involving pilots and advanced prototyping. AUTOCITS* is one of the projects where CAVs and interoperability tests have been conducted. This paper concentrates on the assessment and performance evaluation of tests carried out during the AUTOCITS's Lisbon Pilot, in realworld conditions, involving CAVs and C-ITS technologies. New specific quantitative indicators (key performance indicators - KPls) are proposed to back the assessment and evaluation criteria presented in this work. The KPIs' expressions are provided, which demonstrated to be very difficult to find in the literature. Results are reported and discussed according to the scenarios and field-data recorded during the Pilot.

This paper presents a hybrid 3D-like grid-based mapping approach, that we called HMAP, used as a reliable and efficient 3D representation of the environment surrounding a mobile robot. Considering 3D point-clouds as input data, the proposed mapping approach addresses the representation of height-voxel (HVoxel) elements inside the HMAP, where free and occupied space is modeled through HVoxels, resulting in a reliable method for 3D representation. The proposed method corrects some of the problems inherent to the representation of complex environments based on 2D and 2.5D representations, while keeping an updated grid representation. Additionally, we also propose a complete pipeline for SLAM based on HMAPs. Indoor and outdoor experiments were carried out to validate the proposed representation using data from a Microsoft Kinect One (indoor) and a Velodyne VLP-16 LiDAR (outdoor). The obtained results show that HMAPs can provide a more detailed view of complex elements in a scene when compared to a classic 2.5D representation. Moreover, validation of the proposed SLAM approach was carried out in an outdoor dataset with promising results, which lay a foundation for further research in the topic.

Cooperative, connected and automated Intelligent Transportation System (ITS) aims to improve road traffic safety, comfort, security, and traffic management by sharing data/information among vehicles, infrastructure and road users. Cooperative ITS (C-ITS) also targets the reduction of environmental impact by the road transportation systems. Regarding self-driving vehicles, C-ITS plays a complementary role for enhancing on-board vehicle sensory data and thus making autonomous vehicles more robust and safe. In this paper we present the C-ITS platform, the test-cases and scenarios to be performed during the Lisbon Pilot of the AUTOCITS project. We also provide a description of the C-ITS Day-1 services, as simulated and also real-world events, that will be deployed and evaluated during the test-cases. Conventional, instrumented and autonomous vehicles will take part on the Lisbon Pilot, all equipped with on-board connected vehicular technologies. Two scenarios will be considered, motorway and urban-node scenarios, having distinct vehicles and test-cases, but all sharing the same C-ITS technology framework.

Affective facial expression is a key feature of nonverbal behaviour and is considered as a symptom of an internal emotional state. Emotion recognition plays an important role in social communication: human-to-human and also for human-to-robot. Taking this as inspiration, this work aims at the development of a framework able to recognise human emotions through facial expression for human-robot interaction. Features based on facial landmarks distances and angles are extracted to feed a dynamic probabilistic classification framework. The public online dataset Karolinska Directed Emotional Faces (KDEF) [1] is used to learn seven different emotions (e.g. angry, fearful, disgusted, happy, sad, surprised, and neutral) performed by seventy subjects. A new dataset was created in order to record stimulated affect while participants watched video sessions to awaken their emotions, different of the KDEF dataset where participants are actors (i.e. performing expressions when asked to). Offline and on-the-fly tests were carried out: leave-one-out cross validation tests on datasets and on-the-fly tests with human-robot interactions. Results show that the proposed framework can correctly recognise human facial expressions with potential to be used in human-robot interaction scenarios.

In this paper we propose a gait pattern analysis system that uses stereo vision and machine learning techniques for robotic walker applications. This work contributes with a user monitoring system, that allows the development of more user-centered approaches, such as safer and adaptive HMIs. It also provides a tool to help healthcare personnel in medical assessments. The gait analysis system presented in this paper takes advantage of a stereo vision-based sensor, mounted onboard a robotic walker, to model the user's gait pattern by applying a weighted kernel-density estimator to the captured data. Features are then extracted using a sliding temporal window and classified into one of the trained gait patterns. We have performed experiments both to validate the proposed gait pattern classification system and also to validate its usability. The results obtained from the different experiments evidenced a satisfactory system's performance.

In this paper a study is made of the problem of classifying scenarios, in terms of semantic categories, based on data gathered from sensors mounted on-board mobile robots operating indoors. Once the data are transformed to feature space, supervised classification is performed by a probabilistic approach called Dynamic Bayesian Mixture Models (DBMM). This approach combines class-conditional probabilities from supervised learning models and incorporates past inferences. In this work, several experiments on multi-class semantic place classification are reported based on publicly available datasets. Such experiments were conducted in a such way that generalization aspects are emphasized, which is particularly important in real-world applications. Benchmark results show the effectiveness and competitive performance of the DBMM method, in terms of classification rates, using features extracted from 2D range data and from a RGB-D (Kinect) sensor.

In this work, we present a human-centered robot application in the scope of daily activity recognition towards robot-assisted living. Our approach consists of a probabilistic ensemble of classifiers as a dynamic mixture model considering the Bayesian probability, where each base classifier contributes to the inference in proportion to its posterior belief. The classification model relies on the confidence obtained from an uncertainty measure that assigns a weight for each base classifier to counterbalance the joint posterior probability. Spatio-temporal 3D skeleton-based features extracted from RGB-D sensor data are modeled in order to characterize daily activities, including risk situations (e.g.: falling down, running or jumping in a room). To assess our proposed approach, challenging public datasets such as MSR-Action3D and MSR-Activity3D [1] [2] were used to compare the results with other recent methods. Reported results show that our proposed approach outperforms state-of-the-art methods in terms of overall accuracy. Moreover, we implemented our approach using Robot Operating System (ROS) environment to validate the DBMM running on-the-fly in a mobile robot with an RGB-D sensor onboard to identify daily activities for a robot-assisted living application.

Advances in autonomous navigation, safety, and natural-landmark based localization, are among the key objectives in the development of the next generation of autonomous vehicles, to be deployed in manufacturing and semi-structured environments. In this paper, autonomous navigation and collision detection will be focused, where it is proposed a novel navigation approach that incorporates a RRT-based dynamic path planning and a path-following controller. Safety issues are taken into account in the form of a laser-based object detection and tracking. Experimental results obtained in a virtual environment provide evidence that our proposed navigation method is promising for real-world applications.

Why is pedestrian detection still very challenging in realistic scenes? How much would a successful solution to monocular depth inference aid pedestrian detection? In order to answer these questions we trained a state-of-the-art deformable parts detector using different configurations of optical images and their associated 3D point clouds, in conjunction and independently, leveraging upon the recently released KITTI dataset. We propose novel strategies for depth upsampling and contextual fusion that together lead to detection performance which exceeds that of the RGB-only systems. Our results suggest depth cues as a very promising mid-level target for future pedestrian detection approaches.

In this work, we propose an approach that relies on cues from depth perception from RGB-D images, where features related to human body motion (3D skeleton features) are used on multiple learning classifiers in order to recognize human activities on a benchmark dataset. A Dynamic Bayesian Mixture Model (DBMM) is designed to combine multiple classifier likelihoods into a single form, assigning weights (by an uncertainty measure) to counterbalance the likelihoods as a posterior probability. Temporal information is incorporated in the DBMM by means of prior probabilities, taking into consideration previous probabilistic inference to reinforce current-frame classification. The publicly available Cornell Activity Dataset [1] with 12 different human activities was used to evaluate the proposed approach. Reported results on testing dataset show that our approach overcomes state of the art methods in terms of precision, recall and overall accuracy. The developed work allows the use of activities classification for applications where the human behaviour recognition is important, such as human-robot interaction, assisted living for elderly care, among others.

The cascade classifier is a usual approach in object detection based on vision, since it successively rejects negative occurrences, e.g., background images, in a cascade structure, keeping the processing time suitable for on-the-fly applications. On the other hand, similar to other classifier ensembles, cascade classifiers are likely to have high Vapnik-Chervonenkis (VC) dimension, which may lead to overfitting the training data. Therefore, this work aims at improving the generalization capacity of the cascade classifier by controlling its complexity, which depends on the model of their classifier stages, the number of stages, and the feature space dimension of each stage, which can be controlled by integrating the parameter setting of the feature extractor (in our case an image descriptor) into the maximum-margin framework of support vector machine training, as will be shown in this paper. Moreover, to set the number of cascade stages, bounds on the false positive rate (FP) and on the true positive rate (TP) of cascade classifiers are derived based on a VC-style analysis. These bounds are applied to compose an enveloping receiver operating curve (EROC), i.e., a new curve in the TP–FP space in which each point is an ordered pair of upper bound on the FP and lower bound on the TP. The optimal number of cascade stages is forecasted by comparing EROCs of cascades with different numbers of stages.

Many robotic systems combine cameras with Laser Rangefinders (LRF) for simultaneously achieving multi-purpose visual sensing and accurate depth recovery. Employing a single sensor modality for accomplishing both goals is an appealing proposition because it enables substantial savings in equipment, and tends to decrease the overall complexity of the system. This article explores the possibility of replacing LRF by passive stereo vision for reconstructing the scene along a 2D scan plane. We present a new stereo algorithm that is specifically tailored for the purpose. The algorithm recovers the depth along the scan plane using a symmetry-based matching cost (SymStereo), and refines the raw estimates by applying dynamic programming, followed by a Markov Random Field (MRF) that decides if the reconstructed contour is a line or not. We report for the first time comparative experiments between Stereo Rangefinding (SRF) and LRF. The results are encouraging by showing that SRF can be a plausible alternative to LRF in several application scenarios. Moreover, since SRF also enables independent depth estimates along multiple scan planes with arbitrary orientation, being the only constraint that the scan plane intersects the stereo baseline, it is an important benefit that can be decisive for many robotic applications.