Brain-Computer Interfaces (BCIs) are one of the few remaining communication options for individuals in a Completely Locked-In State (CLIS), where all voluntary motor functions are lost. However, decoding electroencephalographic (EEG) signals in CLIS is particularly challenging due to low signal-to-noise ratios, high intra- and inter-session variability, and cognitive fluctuations. In this study, we systematically evaluate classical and deep learning-based (DL) classification methods on a longitudinal P300-based BCI dataset acquired from a CLIS patient over ten months, comprising seven different stimulation paradigms.A systematic approach is followed to assess model generalization across BCI sessions and paradigms. Overall, more than 40 approaches are compared, including spatial filters for feature extraction with standard classifiers, as well as DL methods based on CNNs and Attention-based architectures. All methods are evaluated with raw input data and three different normalization strategies. Additionally, SMOTE data augmentation is applied to upsample the minority class. The results show high generalization performance across sessions and paradigms, with some approaches achieving nearly 100% performance. Normalization strategies significantly influence performance, while SMOTE often leads to performance degradation. These findings offer valuable insights for designing more robust BCI systems tailored to CLIS users, showing that collecting data across sessions and multiple BCI paradigms can improve BCI performance, while reducing or eliminating the need for per session calibration. Despite the very promising results, they are based on offline analysis. Thus, the best-performing approaches now require online validation for deployment in real-world CLIS scenarios.

Error-related potentials (ErrPs) extracted from electroencephalographic signals hold potential for application in Brain-Machine Interfaces, in contexts such as robot teleoperation or shared control in assistive platforms. Due to difficulties in signal classification, in part caused by its non-stationary and noisy nature, their use has not been fully realized yet.This work proposes a new approach to ErrP classification based on a two-stream deep learning architecture with three training stages. Its first stage is a self-supervised autoencoder architecture with a multi-head attention layer providing relevant latent features. The second stage comprises a supervised contrastive learning approach considering two backbone networks, where one inherits weights from the first stage and the other is updated by considering the feature embeddings distribution. The final stage comprises supervised classification, where the two backbones are fused and used to classify the input EEG signal. At the end of the three stages, a data-driven two-stream ErrP model is obtained.Twenty-five variants of the proposed approach using the Deep Convolutional Network, Shallow Convolutional Network and EEGNet backbones were tested in an ablation study and benchmarked against a large number of classical classification methods, using data from the BNCI dataset intended to assess cross subject generalization capabilities. The proposed approach obtained the best results overall, highlighting the approach’s capabilities in capturing relevant representations of the EEG signal.

This work proposes a novel approach that uses a semantic segmentation mask to obtain a 2D spatial layout of the segmentation-categories across the scene, designated by segmentation-based semantic features (SSFs). These features represent, per segmentation-category, the pixel count, as well as the 2D average position and respective standard deviation values. Moreover, a two-branch network, GS2F2App, that exploits CNN-based global features extracted from RGB images and the segmentation-based features extracted from the proposed SSFs, is also proposed. GS2F2App was evaluated in two indoor scene benchmark datasets: the SUN RGB-D and the NYU Depth V2, achieving state-of-the-art results on both datasets.

Local motion planning is an essential component of autonomous robot navigation systems as it involves generating collision-free trajectories for the robot in real-time, given its current position, the map of the environment and a goal. Considering an a priori goal path, computed by a global planner or as the output of a mission planning approach, this paper proposes a Two-Stream Deep Reinforcement Learning strategy for local motion planning that takes as inputs a local costmap representing the robot’s surrounding obstacles and a local costmap representing the nearest goal path. The proposed approach uses a Double Dueling Deep Q-Network and a new reward model to avoid obstacles while trying to maintain the lateral error between the robot and the goal path close to zero. Our approach enables the robot to navigate through complex environments, including cluttered spaces and narrow passages, while avoiding collisions with obstacles. Evaluation of the proposed approach was carried out in an in-house simulation environment, in five scenarios. Double and Double Dueling architectures were evaluated; the presented results show that the proposed strategy can correctly follow the desired goal path and, when needed, avoid obstacles ahead and recover back to following the goal path.

Visual user interface paradigms are one of the key modules for brain-actuated wheelchairs, which are a class of promising assistive devices that can increase the autonomy and mobility of people suffering from severe motor impairments. This work proposes a Dynamic Environment-based Visual Interface System (DEVIS) for intuitive navigation target selection for brain-actuated wheelchairs. It is composed of a novel Dynamic Visual Interface (DVI), an RGB image-based perception module (indoor scene classification, object detection and classification, and object tracking), and a P300-based BCI. The DVI displays potential navigation goals in three forms of visual cues: an RGB camera image streaming with object bounding boxes overlaid on objects detected and tracked, three global points of interest (indoor places), and two static commands. Hence, the DVI allows a user to select, through the P300-based BCI, navigation targets/commands that are flashing independently and randomly to create an oddball paradigm. Experimental evaluations were carried out in a dynamic setting with 5 participants, who were asked to select targets displayed in the DVI. The dynamic setting was obtained using RGB image sequences of the ISR-RGB-D Dataset, which represents a mission performed by a mobile robotic platform. The target selection was performed through a P300-based BCI, using non-self-paced and self-paced modalities. The obtained results show that the proposed DEVIS can run in real-time and that users are able to correctly select targets with high BCI accuracy rates.

Convolutional Neural Networks (CNNs) have been increasingly applied in visual classification tasks by replacing hand-crafted features with deep features. However, problems such as inter-class similarity and intra-class variation led to the need of obtaining more descriptive features. To accomplish this, a new semantic inter-object relationship approach is proposed, which is based on the distance relationships between recognized objects. This new source of information represents how close or apart objects belonging to two object classes are, which, together with the number of object occurrences, allows to develop a more descriptive semantic feature representation of the scene. To exploit such semantic features, a two-branch CNN architecture based on 1D and 2D convolutional layers, is proposed. Also, an enhancement version, GSF2AppV2, of the Global and Semantic Feature Fusion described in [1] is proposed by integrating the new semantic inter-object relationship approach, as well as the aforementioned two-branch CNN architecture. The GSF2AppV2 is also composed of a CNN-based global feature branch, where five different CNN-based feature extraction approaches were assessed as global feature extraction modules. Moreover, to combine global and semantic features, two feature fusion approaches are proposed and evaluated: correlation and triple concatenation. GSF2AppV2 was evaluated in two benchmark datasets: the SUN RGB-D and NYU Depth V2. State-of-the-art results were achieved on both datasets, showing the effectiveness of the proposed semantic feature approach on the pipeline.

This paper presents a dynamic gesture recognition approach using a novel spatiotemporal 2D skeleton image representation that can be fed to computationally efficient deep convolutional neural networks, for applications on human-robot interaction. Gestures are a seamless modality of human interaction and represent a potentially natural way to interact with the smart devices around us, like robots. The contribution of this paper is the proposal of a visually interpretable representation of dynamic gestures, which has a two-fold advantage: (i) conveys both spatial and temporal characteristics relying on a technique inspired in computer graphics, (ii) and can be used with simple and efficient architectures of convolutional neural networks. In our representation, a 3D skeleton model is projected to a 2D camera’s point-of-view, preserving spatial relations, and through a sliding window the temporal domain is encoded in a fused image of consecutive frames, through a shading motion effect achieved by manipulating a transparency coefficient. The result is a 2D image that when fed to simple custom-designed convolutional neural networks, it is achieved accurate classification of dynamic gestures. Experimmental reuslts obtained with a purposely captured 6 gesture dataset of 11 subjects, and also 2 public datasets, give evidence of a strong performance of our approach, when compared to other methods.

This paper presents a study in terms of accuracy and inference speed using RGB-D object detection and classification for mobile platform applications. The study is divided in three stages. In the first, eight state-of-the-art CNN-based object classifiers (AlexNet, VGG16-19, ResNet1850-101, DenseNet, and MobileNetV2) are used to compare the attained performances with the corresponding inference speeds in object classification tasks. The second stage consists in exploiting YOLOv3/YOLOv3-tiny networks to be used as Region of Interest generator method. In order to obtain a real-time object recognition pipeline, the final stage unifies the YOLOv3/YOLOv3-tiny with a CNN-based object classifier. The pipeline evaluates each object classifier with each Region of Interest generator method in terms of their accuracy and frame rate. For the evaluation of the proposed study under the conditions in which real robotic platforms navigate, a nonobject centric RGB-D dataset was recorded in Institute of Systems and Robotics facilities using a camera on-board the ISR-InterBot mobile platform. Experimental evaluations were also carried out in Washington and COCO datasets. Promising performances were achieved by the combination of YOLOv3tiny and ResNet18 networks on the embedded hardware Nvidia Jetson TX2.

Robot localization in indoor environments is crucial for achieving flexible automated navigation. In this paper, we propose a novel multi-stage localization approach for mobile robots which combines a commercial beacon-based absolute indoor positioning system with laser scan data. This configuration of sensors can be deployed in a non-intrusive way in most robotic platforms without the use of proprioceptive sensors (e.g. wheel encoders) which often introduce non-negligible maintenance and downtime costs. The data fusion is performed by a particle filter aided by a refinement stage. The proposed approach was evaluated in two different indoor scenarios with a mobile platform equipped with a mobile Marvelmind beacon and a Hokuyo UTM-30LX scanning laser rangefinder. The proposed localization framework, purposely without proprioceptive data, is compared with an AMCL approach having as inputs odometry, calculated from wheel encoders' data, and laser scan data. Preliminary results show that the proposed approach can provide an accurate localization estimate and that using the refinement stage improves localization.

This paper introduces a new mobile robot localization solution consisting of two main modules: a Particle-Filter based Localization (PFL) and a Reinforcement-Learning based map updating, integrating relative measurements and absolute indoor positioning sensor (A-IPS) data. Concerning localization using 2D-LiDARs, featureless areas are known to be problematic. To solve this problem a classic PFL approach was modified to incorporate A-IPS position measurements in the prediction and update stages. The localization approach has the particularity of including the possibility of updating the map whenever major modifications are detected in the environment in relation to the current localization map. Due to the random sampling-based nature of the PFL, an associated map update solution is not trivial since small inconsistencies in the estimated pose can lead to erroneous map associations. The proposed method learns to decide by assigning higher rewards the greater is the overlap between the map and the 2DLIDAR scans, via RL, and then a proper update of the map is achieved. Validation of the proposed pipeline was carried out in a differential drive platform with algorithms developed in ROS. Tests were performed in two scenarios in order to assess the performance of both the localization module and the map update stage. The results show that the proposed localization method offers improvements in relation to known approaches, and consequently suggest promising perspectives for the proposed map update decision framework.

Connected and autonomous/automated vehicle (CAV) technologies are shaping the design and the new developments in the automotive industry and, in a wider perspective, in the mobility sector as well. Despite the recent advances and on-going developments, and the enthusiasm around autonomous mobility systems, real-world testing of CAVs is a crucial element to allow the next generation of intelligent vehicles to come to our daily-life. The importance of realistic testing is recognized by academia, industry, public sector and stakeholders, and is reflected in all projects involving pilots and advanced prototyping. AUTOCITS* is one of the projects where CAVs and interoperability tests have been conducted. This paper concentrates on the assessment and performance evaluation of tests carried out during the AUTOCITS's Lisbon Pilot, in realworld conditions, involving CAVs and C-ITS technologies. New specific quantitative indicators (key performance indicators - KPls) are proposed to back the assessment and evaluation criteria presented in this work. The KPIs' expressions are provided, which demonstrated to be very difficult to find in the literature. Results are reported and discussed according to the scenarios and field-data recorded during the Pilot.

This paper presents a new collaborative approach for robot motion planning of an assistive robotic platform that takes into account the intentions of the user provided through Electrooculographic (EOG) signals, as well as obstacles surrounding the robotic platform. In order to increase human confidence in the operation of robotic platforms with some degree of navigational autonomy, the intent of the user must be included in the decision process. In our system, the humanrobot interface works through ocular movements (saccades and blinks), which are acquired as EOG signals and classified using a Convolutional Neural Network. In our proposed approach, a model-free Reinforcement Learning (RL) layer is used to provide commands to a virtual robotic platform. The RL layer is constantly being updated with the inputs from the user's intent, environment perception and previous machine-based decisions. In order to prevent collisions, machine-based perception using the proposed RL motion planning approach will assist the user by selecting suitable actions while learning from prior driving behaviors. The approach was validated by a set of tests that consisted of driving a robotic platform in an in-house 3D virtual model of our Research Center (ISR-UC). The experimental results show a better performance of the proposed approach with RL when compared to the version without the RL-based motion planning component. Results show that the approach is a promising step in the concept put forward for collaborative Human-Robotic Interaction (HRI), and opens a path for future research.

In this paper, we propose a markerless multi-view-based multi-user head tracking system for VR/AR/MR applications. In the Virtual Reality domain, the user typically interacts with the virtual world through his movements. The point-of-view of the user's head mounted display (HMD) is based on the user's head pose. Most systems track the head of the user based on the tracking of the HMD itself and are typically constrained to a single user. Aiming to improve the user's experience in complex dynamic environments, we propose a head tracking method based on a multi-camera scenario, able to deal with the interaction of multiple users while coping with the existence of dynamic objects. The method relies on the voxelization of dense point cloud data, which is segmented to detect elements in the scene. Each element is holistically tracked along time and each detected user has his head's position tracked using a dedicated Particle Filter. The system was evaluated using an annotated ground-truth, according to several metrics, and demonstrated a mean 3D positioning error below 18 mm. Furthermore, tests with multiple users were performed that successfully validate the proposed head tracking approach. The aforementioned tests also showed that the proposed approach is computationally efficient (running time below 20 ms for 4 users).

This paper presents a hybrid 3D-like grid-based mapping approach, that we called HMAP, used as a reliable and efficient 3D representation of the environment surrounding a mobile robot. Considering 3D point-clouds as input data, the proposed mapping approach addresses the representation of height-voxel (HVoxel) elements inside the HMAP, where free and occupied space is modeled through HVoxels, resulting in a reliable method for 3D representation. The proposed method corrects some of the problems inherent to the representation of complex environments based on 2D and 2.5D representations, while keeping an updated grid representation. Additionally, we also propose a complete pipeline for SLAM based on HMAPs. Indoor and outdoor experiments were carried out to validate the proposed representation using data from a Microsoft Kinect One (indoor) and a Velodyne VLP-16 LiDAR (outdoor). The obtained results show that HMAPs can provide a more detailed view of complex elements in a scene when compared to a classic 2.5D representation. Moreover, validation of the proposed SLAM approach was carried out in an outdoor dataset with promising results, which lay a foundation for further research in the topic.

This paper presents an approach to robot-assisted navigation on a mobility assistance context, by learning from the user while helping him navigate efficiently and safely in complex environments. Assistive robots such as robotic walkers provide the ability to support a user's body weight on the upper limbs while walking. However, walkers can add an extra layer of distress due to their specific manipulation constraints. For the users of such devices, lack of dexterous upper limb control can be a considerable problem as it means they may be unable to operate these devices efficiently; users may also have visual impairments that reduce their navigational efficiency. The proposed approach uses a Reinforcement Learning (RL) model and a dynamic window-based local motion planning algorithm. It aims to learn needed corrections in the motion command based on the surrounding environment and the user's intent. The proposed solution handles corrections and guides the user through the environment without collisions while learning and aiding the user when he is unable to operate the device efficiently. The RL based approach was tested in indoor scenarios with a robotic walker platform showing preliminary promising results.

In this paper we propose a gait pattern analysis system that uses stereo vision and machine learning techniques for robotic walker applications. This work contributes with a user monitoring system, that allows the development of more user-centered approaches, such as safer and adaptive HMIs. It also provides a tool to help healthcare personnel in medical assessments. The gait analysis system presented in this paper takes advantage of a stereo vision-based sensor, mounted onboard a robotic walker, to model the user's gait pattern by applying a weighted kernel-density estimator to the captured data. Features are then extracted using a sliding temporal window and classified into one of the trained gait patterns. We have performed experiments both to validate the proposed gait pattern classification system and also to validate its usability. The results obtained from the different experiments evidenced a satisfactory system's performance.

Advances in autonomous navigation, safety, and natural-landmark based localization, are among the key objectives in the development of the next generation of autonomous vehicles, to be deployed in manufacturing and semi-structured environments. In this paper, autonomous navigation and collision detection will be focused, where it is proposed a novel navigation approach that incorporates a RRT-based dynamic path planning and a path-following controller. Safety issues are taken into account in the form of a laser-based object detection and tracking. Experimental results obtained in a virtual environment provide evidence that our proposed navigation method is promising for real-world applications.

The video shows an overview of the ISRobotCar, an experimental autonomous electric vehicle that integrates the robot operating system (ROS.org) and several sensors such as an IBEO laser scanner, inertial measurement units (IMU), RTK-GPS, vision cameras, and magnet detectors for magnetic guidance. The main purpose of this vehicle is to serve as a platform for experimental testing of algorithms required for autonomous navigation and cooperative navigation in urban environments. In the video, we present the ISRobotCar moving autonomously in a parking-like area as well as its main hardware and control modules.