Sleep is fundamental to health, and society is more and more aware of the impact and relevance of sleep disorders. Traditional diagnostic methods, like polysomnography, are intrusive and resource-intensive. Instead, research is focusing on developing novel, less intrusive or portable methods that combine intelligent sensors with activity recognition for diagnosis support and scoring. Event cameras offer a promising alternative for automated, in-home sleep activity recognition due to their excellent low-light performance and low power consumption. This work introduces EventSleep2-data , a significant extension to the EventSleep dataset, featuring 10 complete night recordings (around 7 h each) of volunteers sleeping in their homes. Unlike the original short and controlled recordings, this new dataset captures natural, full-night sleep sessions under realistic conditions. This new data incorporates challenging real-world scene variations, an efficient movement-triggered sparse data recording pipeline, and synchronized 2-channel EEG data for a subset of recordings. We also present EventSleep2-net , a novel event-based sleep activity recognition approach with a dual-head architecture to simultaneously analyze motion classes and static poses. The model is specifically designed to handle the motion-triggered, sparse nature of complete night recordings. Unlike the original EventSleep architecture, EventSleep2-net can predict both movement and static poses even during long periods with no events. We demonstrate state-of-the-art performance on both EventSleep1-data, the original dataset, and EventSleep2-data, with comprehensive ablation studies validating our design decisions. Together, EventSleep2-data and EventSleep2-net overcome the limitations of the previous setup and enable continuous, full-night analysis for real-world sleep monitoring, significantly advancing the potential of event-based vision for sleep disorder studies.

Event cameras record sparse illumination changes with high temporal resolution and high dynamic range. Thanks to their sparse recording and low consumption, they are increasingly used in applications such as AR/VR and autonomous driving. Current top-performing methods often ignore specific event-data properties, leading to the development of generic but computationally expensive algorithms, while event-aware methods do not perform as well. We proposeEvent Transformer$^+$+, that improves our seminal workEvTwith a refined patch-based event representation and a more robust backbone to achieve more accurate results, while still benefiting from event-data sparsity to increase its efficiency. Additionally, we show how our system can work with different data modalities and propose specific output heads, for event-stream classification (i.e. action recognition) and per-pixel predictions (dense depth estimation). Evaluation results show better performance to the state-of-the-art while requiring minimal computation resources, both on GPU and CPU.

Electric vehicles are seen as a sustainable alternative in the current fight against increasing pollution worldwide. However, they are still seeing a limited spread among drivers and one of the challenges is to deepen the knowledge about the users' perceptions of this innovative means of transport. The current paper presents the work done in this framework by creating two Discrete Choice Experiments. The first one investigates aspects related to car ownership characteristics and performances, whereas the second one focuses on the features of the charging infrastructure. We describe the whole process that guided us to the current final version of these experiments. This includes the preliminary stage of a literature review and a productive discussion with stakeholders in the project framework arriving at the spread of questionnaires in two different Pilots. The current work stresses the need for all these passages to propose reliable experiments that could get new insights on the users' perception of electric vehicles.

This work presents a novel approach for semi-supervised semantic segmentation. The key element of this approach is our contrastive learning module that enforces the segmentation network to yield similar pixel-level feature representations for same-class samples across the whole dataset. To achieve this, we maintain a memory bank which is continuously updated with relevant and high-quality feature vectors from labeled data. In an end-to-end training, the features from both labeled and unlabeled data are optimized to be similar to same-class samples from the memory bank. Our approach not only outperforms the current state-of-the-art for semi-supervised semantic segmentation but also for semi-supervised domain adaptation on well-known public benchmarks, with larger improvements on the most challenging scenarios, i.e., less available labeled data. Code is available at https://github.com/Shathe/SemiSeg-Contrastive

LiDAR semantic segmentation provides 3D semantic information about the environment, an essential cue for intelligent systems, such as autonomous vehicles, during their decision making processes. Unfortunately, the annotation process for this task is very expensive. To overcome this, it is key to find models that generalize well or adapt to additional domains where labeled data is limited. This work addresses the problem of unsupervised domain adaptation for LiDAR semantic segmentation models. We propose simple but effective strategies to reduce the domain shift by aligning the data distribution on the input space. Besides, we present a learning-based module to align the distribution of the semantic classes of the target domain to the source domain. Our approach achieves new state-of-the-art results on three different public datasets, which showcase adaptation to three different domains.

Object recognition in video is an important task for plenty of applications, including autonomous driving perception, surveillance tasks, wearable devices or IoT networks. Object recognition using video data is more challenging than using still images due to blur, occlusions or rare object poses. Specific video detectors with high computational cost or standard image detectors together with a fast post-processing algorithm achieve the current state-of-the-art. This work introduces a novel post-processing pipeline that overcomes some of the limitations of previous post-processing methods by introducing a learning-based similarity evaluation between detections across frames. Our method improves the results of stat-of-the-art specific video detectors, specially regarding fast moving objects, and presents low resource requirements. And applied to efficient still image detectors, such as YOLO, provides comparable results to much more computationally intensive detectors.

Single-view depth estimation suffers from the problem that a network trained on images from one camera does not generalize to images taken with a different camera model. Thus, changing the camera model requires collecting an entirely new training dataset. In this work, we propose a new type of convolution that can take the camera parameters into account, thus allowing neural networks to learn calibration-aware patterns. Experiments confirm that this improves the generalization capabilities of depth prediction networks considerably, and clearly outperforms the state of the art when the train and test images are acquired with different cameras.

Wearable technologies are enabling plenty of new applications of computer vision, from life logging to health assistance. Many of them are required to recognize the elements of interest in the scene captured by the camera This work studies the problem of object detection and localization on videos captured by this type of camera. Wearable videos are a much more challenging scenario for object detection than standard images or even another type of videos, due to lower quality images (e.g. poor focus) or high clutter and occlusion common in wearable recordings. Existing work typically focuses on detecting the objects of focus or those being manipulated by the user wearing the camera. We perform a more general evaluation of the task of object detection in this type of video, because numerous applications, such as marketing studies, also need detecting objects which are not in focus by the user. This work presents a thorough study of the well known YOLO architecture, that offers an excellent trade-off between accuracy and speed, for the particular case of object detection in wearable video. We focus our study on the public ADL Dataset, but we also use additional public data for complementary evaluations. We run an exhaustive set of experiments with different variations of the original architecture and its training strategy. Our experiments drive to several conclusions about the most promising directions for our goal and point us to further research steps to improve detection in wearable videos.

Motor rehabilitation based on the association of electroencephalographic (EEG) activity and proprioceptive feedback has been demonstrated as a feasible therapy for patients with paralysis. To promote long-lasting motor recovery, these interventions have to be carried out across several weeks or even months. The success of these therapies partly relies on the performance of the system decoding movement intentions, which normally has to be recalibrated to deal with the nonstationarities of the cortical activity. Minimizing the recalibration times is important to reduce the setup preparation and maximize the effective therapy time. To date, a systematic analysis of the effect of recalibration strategies in EEG-driven interfaces for motor rehabilitation has not yet been performed. Data from patients with stroke (4 patients, 8 sessions) and spinal cord injury (SCI) (4 patients, 5 sessions) undergoing two different paradigms (self-paced and cue-guided, respectively) are used to study the performance of the EEG-based classification of motor intentions. Four calibration schemes are compared, considering different combinations of training datasets from previous and/or the validated session. The results show significant differences in classifier performances in terms of the true and false positives (TPs) and (FPs). Combining training data from previous sessions with data from the validation session provides the best compromise between the amount of data needed for calibration and the classifier performance. With this scheme, the average true (false) positive rates obtained are 85.3% (17.3%) and 72.9% (30.3%) for the self-paced and the cue-guided protocols, respectively. These results suggest that the use of optimal recalibration schemes for EEG-based classifiers of motor intentions leads to enhanced performances of these technologies, while not requiring long calibration phases prior to starting the intervention.

Learning models of behaviours has many applications in robotics spanning both control, e.g. learning from demonstration and perception, e.g. monitoring and surveillance. Inverse reinforcement learning encodes behaviours as a reward function learned from a set of demonstrations. This paper addresses the problem of learning from unlabelled datasets containing an unknown number of behaviours in continuous action-state spaces. The proposed method uses a hierarchical clustering approach to directly group trajectories that share a common reward function. The similarity metric is based on the distribution of maximum entropy of the feature counts computed using path integrals. We evaluated the method in three different tasks: navigation on a set of synthetic maps, human driving styles on a simulator and human reaching. Results show that clustering in the reward space is able to discover the latent reward structure resulting in compact models that can generate all the observed behaviours.

Brain-machine interfaces (BMIs) have demonstrated how they can be used for reaching tasks with both invasive and non-invasive signal recording methods. Despite the constant improvements in this field, there still exist diverse factors to overcome before achieving a natural control. In particular, the high variability of the brain signals often leads to the incorrect decoding of the subject intentions, producing unreliable behaviors in the controlled device. A possible solution to this problem would be that of correcting this erroneous decoding using a feedback signal from the user. In this work, we evaluate the possibility of decoding neural signals associated to performance monitoring (EEG-recorded error-related potentials) during a reaching task. Compared to previous works where these error potentials were recorded under scenarios with discrete movements performed by the cursor, under real conditions the cursor is moving continuously and thus the system is required to asynchronously detect any possible error. To this end, we simulated two different erroneous events during the monitoring of a reaching task: errors at the beginning of the movement, and errors happening in the middle of the trajectory being executed. Through the analysis of the recorded EEG of three subjects, we demonstrate the existence of neural correlates for the two types of elicited error potentials, and we are able to asynchronously detect them with high accuracies.

Recent works have explored the use of brain signals to directly control virtual and robotic agents in sequential tasks. So far in such brain-computer interfaces (BCI), an explicit calibration phase was required to build a decoder that translates raw electroencephalography (EEG) signals from the brain of each user into meaningful instructions. This paper proposes a method that removes the calibration phase, and allows a user to control an agent to solve a sequential task. The proposed method assumes a distribution of possible tasks, and infers the interpretation of EEG signals and the task by selecting the hypothesis which best explains the history of interaction. We introduce a measure of uncertainty on the task and on the EEG signal interpretation to act as an exploratory bonus for a planning strategy. This speeds up learning by guiding the system to regions that better disambiguate among task hypotheses. We report experiments where four users use BCI to control an agent on a virtual world to reach a target without any previous calibration process.

In this paper we introduce a method to learn multiple behaviors in the form of motor primitives from an unlabeled dataset. One of the difficulties of this problem is that in the measurement space, behaviors can be very mixed, despite existing a latent representation where they can be easily separated. We propose a mixture model based on Dirichlet Process (DP) to simultaneously cluster the observed time-series and recover a sparse representation of the behaviors using a Laplacian prior as the base measure of the DP. We show that for linear models, e.g potential functions generated by linear combinations of a large number of features, it is possible to compute analytically the marginal of the observations and derive an efficient sampler. The method is evaluated using robot behaviors and real data from human motion and compared to other techniques.

We address the problem of bootstrapping language acquisition for an artificial system similarly to what is observed in experiments with human infants. Our method works by associating meanings to words in manipulation tasks, as a robot interacts with objects and listens to verbal descriptions of the interactions. The model is based on an affordance network, i.e., a mapping between robot actions, robot perceptions, and the perceived effects of these actions upon objects. We extend the affordance model to incorporate spoken words, which allows us to ground the verbal symbols to the execution of actions and the perception of the environment. The model takes verbal descriptions of a task as the input and uses temporal co-occurrence to create links between speech utterances and the involved objects, actions, and effects. We show that the robot is able form useful word-to-meaning associations, even without considering grammatical structure in the learning process and in the presence of recognition errors. These word-to-meaning associations are embedded in the robot's own understanding of its actions. Thus, they can be directly used to instruct the robot to perform tasks and also allow to incorporate context in the speech recognition task. We believe that the encouraging results with our approach may afford robots with a capacity to acquire language descriptors in their operation's environment as well as to shed some light as to how this challenging process develops with human infants.

Ego-motion estimation and 3D scene reconstruction from image data has been a long term aim both in the Robotics and Computer Vision communities. Nevertheless, while both visual SLAM and Structure from Motion already provide an accurate ego-motion estimation, visual scene estimation does not offer yet such a satisfactory result; being in most cases limited to a sparse set of salient points. In this paper we propose an algorithm to densify a sparse point-based reconstruction into a dense multi-plane based one, from the only input of a set of sparse images.

Scan matching techniques have been widely used to compute the displacement of robots. This estimate is part of many algorithms addressing navigation and mapping. This paper addresses the scan matching problem in three dimensional workspaces. We propose an generalization of the Metric based Iterative Closest Point (MbICP) to the 3D case. The main contribution is the development of all the mathematical tools required to formulate the ICP with this new metric, including the derivation of point to plane distances based on the new metric. We also provide experimental results to evaluate the algorithms and different combinations of ICP and MbICP to illustrate the advantages of the metric based approach.

Reinforcement learning algorithms have been successfully applied in robotics to learn how to solve tasks based on reward signals obtained during task execution. These reward signals are usually modeled by the programmer or provided by supervision. However, there are situations in which this reward is hard to encode, and so would require a supervised approach of reinforcement learning, where a user directly types the reward on each trial. This paper proposes to use brain activity recorded by an EEG-based BCI system as reward signals. The idea is to obtain the reward from the activity generated while observing the robot solving the task. This process does not require an explicit model of the reward signal. Moreover, it is possible to capture subjective aspects which are specific to each user. To achieve this, we designed a new protocol to use brain activity related to the correct or wrong execution of the task. We showed that it is possible to detect and classify different levels of error in single trials. We also showed that it is possible to apply reinforcement learning algorithms to learn new similar tasks using the rewards obtained from brain activity.

We present an active learning algorithm for the problem of body schema learning, i.e. estimating a kinematic model of a serial robot. The learning process is done online using Recursive Least Squares (RLS) estimation, which outperforms gradient methods usually applied in the literature. In addiction, the method provides the required information to apply an active learning algorithm to find the optimal set of robot configurations and observations to improve the learning process. By selecting the most informative observations, the proposed method minimizes the required amount of data. We have developed an efficient version of the active learning algorithm to select the points in real-time. The algorithms have been tested and compared using both simulated environments and a real humanoid robot.

This paper presents a method to associate meanings to words in manipulation tasks. We base our model on an affordance network, i.e., a mapping between robot actions, robot perceptions and the perceived effects of these actions upon objects. We extend the affordance model to incorporate words. Using verbal descriptions of a task, the model uses temporal co-occurrence to create links between speech utterances and the involved objects, actions and effects. We show that the robot is able form useful word-to-meaning associations, even without considering grammatical structure in the learning process and in the presence of recognition errors. These word-to-meaning associations are embedded in the robot's own understanding of its actions. Thus they can be directly used to instruct the robot to perform tasks and also allow to incorporate context in the speech recognition task.

Affordances encode relationships between actions, objects, and effects. They play an important role on basic cognitive capabilities such as prediction and planning. We address the problem of learning affordances through the interaction of a robot with the environment, a key step to understand the world properties and develop social skills. We present a general model for learning object affordances using Bayesian networks integrated within a general developmental architecture for social robots. Since learning is based on a probabilistic model, the approach is able to deal with uncertainty, redundancy, and irrelevant information. We demonstrate successful learning in the real world by having an humanoid robot interacting with objects. We illustrate the benefits of the acquired knowledge in imitation games.

In this paper we build an imitation learning algorithm for a humanoid robot on top of a general world model provided by learned object affordances. We consider that the robot has previously learned a task independent affordance-based model of its interaction with the world. This model is used to recognize the demonstration by another agent (a human) and infer the task to be learned. We discuss several important problems that arise in this combined framework, such as the influence of an inaccurate model in the recognition of the demonstration. We illustrate the ideas in the paper with some experimental results obtained with a real robot.

Affordances represent the behavior of objects in terms of the robot's motor and perceptual skills. This type of knowledge plays a crucial role in developmental robotic systems, since it is at the core of many higher level skills such as imitation. In this paper, we propose a general affordance model based on Bayesian networks linking actions, object features and action effects. The network is learnt by the robot through interaction with the surrounding objects. The resulting probabilistic model is able to deal with uncertainty, redundancy and irrelevant information. We evaluate the approach using a real humanoid robot that interacts with objects.

Continuous-time marked point processes appear in many areas of science and engineering including queuing theory, seismology, neuroscience and finance. In numerous applications, these point processes are unobserved but actually drive an observation process. Here, we are interested in optimal sequential Bayesian estimation of such partially observed point processes. This class of filtering problems is non-standard as there is typically no underlying Markov structure and the likelihood function relating the observations to the point process has a complex form. Hence, except in very specific cases it is impossible to solve them in closed-form. We develop an original trans-dimensional Sequential Monte Carlo method to address this class of problems. An application to partially observed queues is presented.

In this paper, we describe the adaptation of an autonomous robotic wheelchair for cognitive disabled children. The constraints imposed by these users require developing specific human-machine interfaces adapted to their limitations. In most cases it is necessary to develop additional tools to teach the children the spatial relations between the wheelchair, its motion and the environment. In addition to this, it is important to interact closely with the children and their educators. The paper describes the whole process followed to make the children use the autonomous wheelchair and the lessons learnt during the validation phase with the wheelchair and the children

This paper addresses the scan matching problem for mobile robot displacement estimation. The contribution is a new metric distance and all the tools necessary to be used within the iterative closest point framework. The metric distance is defined in the configuration space of the sensor, and takes into account both translation and rotation error of the sensor. The new scan matching technique ameliorates previous methods in terms of robustness, precision, convergence, and computational load. Furthermore, it has been extensively tested to validate and compare this technique with existing methods

This paper presents a metric-based matching algorithm to estimate the robot planar displacement by matching dense two-dimensional range scans. The contribution is a geometric distance that takes into account the translation and orientation of the sensor at the same time. This result is used in the two steps of the matching - estimation process. The correspondences between scans are established with this measure and the minimization of the error is also carried out in terms of this distance. As a result, the translation and rotation are compensated in this framework simultaneously. In fact, this is the contribution with respect to previous work that addressed only translation or translation and rotation but separately. The new technique has been implemented and tested on a real vehicle. The experiments illustrate how it is more robust and accurate than prior techniques. At the end of the paper, we give an extension of our distance measure to 3D range-data matching problems.

This paper addresses the modeling of the static and dynamic parts of the scenario and how to use this information within a real sensor-based navigation system. The contribution in the modeling aspect is a formulation of the Detection and Tracking of Mobile Objects and the Simultaneous Localization and Map Building in such a way that the nature (static/dynamic) of the observations is included in the estimation process. This is achieved by a set of filters tracking the moving objects and a map of the static structure constructed on line. In addition, this paper discusses how this modeling module is integrated in a real sensor-based navigation system taking advantage selectively of the dynamic and static information. The experimental results confirm that the complete navigation system is able to move a vehicle in unknown and dynamic scenarios. Furthermore, the system overcomes many of the limitations of previous systems associated to the ability to distinguish the nature of the parts of the scenario.

This paper presents a method to cooperatively localize pairs of robots fusing bearing-only information provided by cameras and the motion of the vehicles. The algorithm uses the robots as landmarks to estimate their relative location. Bearings are the simplest measurements directly obtained from the cameras, as opposed to measuring depths which would require knowledge or reconstruction of the world structure. We present the general recursive Bayes estimator and three different implementations based on an extended Kalman filter, a particle filter and a combination of both techniques. We have compared the performance of the different implementations using real data acquired with two platforms equipped with omnidirectional cameras and simulated data.

This paper presents a probabilistic scan matching algorithm to estimate the robot planar displacement by matching dense two-dimensional range scans. The general framework follows an iterative process of two steps: (i) computation of correspondences between scans, and (ii) estimation of the relative displacement. The contribution is a probabilistic modelling of this process that takes into account all the uncertainties involved: the uncertainty of the displacement of the sensor and the measurement noises. Furthermore, it also considers all the possible correspondences resulting from these uncertainties. This technique has been implemented and tested on a real vehicle. The experiments illustrate how the performances of this method are better than previous geometric ones in terms of robustness, accuracy and convergence.

We address here a sensor-based navigation system to safely drive vehicles in realistic scenarios. The system is composed of three modules with the following functionalities: model builder, planning and reactive motion. These modules are integrated within a planner-reactor architecture that supervises and coordinates them in order to carry out the motion task. The advantage of our system is to achieve a robust and trustworthy navigation in difficult scenarios, which remain troublesome for many of the existing systems. In order to validate the system, we present experiments with a wheelchair vehicle transporting a human among locations in an office type scenario.

This paper presents a new method for relative localization of a pair of robots based on the trajectories described by unidentifiable moving objects. Our approach uses a Rao-Blackwellized particle filter to estimate both the relative location of the robots and the data associations between the moving objects around the robots. We describe our implementation on real robots and present experiments illustrating the robustness of our algorithm.

This paper describes a method for the identification of moving objects by a team of robots based on kinematic information. The objective is to be able to identify moving objects observed by different robots without using specific landmarks. Our method uses a Bayesian approach and is based on the matching of maps of dynamic objects built by the members of the team. These maps contain the relative position of moving objects and their velocity at a given time. Experimental results using data from a real environment carried out to validate the method are presented and discussed.