Aiming at developing a safe, intuitive, and collaborative steerable drilling robotic system for pedicle screw fixation procedures, in this paper, we leverage our recently developed steerable drilling robotic framework, and developed a collaborative drilling mode to control this system. In this control mode, first a user positions a concentric tube steerable drilling robot (CT-SDR) in the workspace and aligns it based on a preplanned trajectory. Next, the CT-SDR is directly controlled by the user through an admittance mode to perform a drilling procedure and creating a J-shape tunnel. To evaluate the user comfort and intuitiveness of the drilling procedure using this system and the proposed control interface, we performed a user study with 11 subjects, who had no prior experience in using this system. The results of this study were analyzed using various qualitative and quantitative metrics.

Brain-controlled mobile robotics can provide restoration of mobility for individuals with severe physical disabilities and empower healthy people with a broader reachable range in particular environments, which have been flourished over the past twenty years. This paper conducts a systematic state-of-the-art overview of noninvasive EEG-based intelligent mobile robots. We first present the general architecture and basic concepts, typical system types, and main research efforts on the whole-system design. Then, relevant key techniques associated with the brain-machine interfaces (BMIs), control strategies, and robot intelligence are reviewed to elucidate the research progress of the overall system. System performance evaluation is critical and complicated, here we summarize the conditions of the recruited participants, the experimental protocol, tasks and environments, with an emphasis on evaluation metrics regarding BMI performance, navigation performance, system robustness, and the user. We further highlight the remaining challenges and the potential research directions of future work. This study with informative outline is envisioned to enhance current understanding and suggest the future perspectives on EEG-based mobile robotic devices. Note to Practitioners—The brain-computer interfaces (BCIs) can provide a novel and feasible way to convey the users’ intentions to the terminal devices through the brain signals, which has exhibited emerging prospects in both civil and military applications thus having attracted increasing interests in researchers, industries, and government. The integration of noninvasive electroencephalogram (EEG)-based BCIs and expanded wheeled robotics have potentials to improve the human daily life with enhanced level of mobility, exhibiting significant socioeconomic impacts. This paper provides a systematic review of the EEG-based intelligent mobile robots from the aspects of key milestones, key techniques, and performance evaluation. To bring such the next generation assistive products from the laboratory to real applications, this paper further highlights the remaining challenges and shed light on the potential future work.

Communication latency in any delicate telerobotic operation (such as remote surgery over distance) would impose a significant challenge due to the temporal degradation of visual perception and can substantially affect the outcomes. Less is known, however, about the neurophysiological basis of how operators adapt/react to delayed visual feedback. Identification of such neural markers might provide novel ways for future applications to monitor the mental workload (MW). In this study, we recorded electroencephalography (EEG) data from nine users while performing a peg transfer task using the da Vinci Research Kit with three levels of induced visual delay in the video feedback. Our results suggest that spectral EEG-based features can provide markers of the operator's MW modulated by arbitrary visual delay. We also show that the exposure to different visual delays could be successfully classified/detected solely from EEG data, using a Riemannian geometry-based classifier, which highlights the utility of EEG signals for detecting the effect of visual delay on brain activity.

Two distinct technologies have gained attention lately due to their prospects for motor rehabilitation: robotics and brain-machine interfaces (BMIs). Harnessing their combined efforts is a largely uncharted and promising direction that has immense clinical potential. However, a significant challenge is whether motor intentions from the user can be accurately detected using non-invasive BMIs in the presence of instrumental noise and passive movements induced by the rehabilitation exoskeleton. As an alternative to the straight-forward continuous control approach, this study instead aims to characterize the onset and offset of motor imagery during passive arm movements induced by an upper-body exoskeleton to allow for the natural control (initiation and termination) of functional movements. Ten participants were recruited to perform kinesthetic motor imagery (MI) of the right arm while attached to the robot, simultaneously cued with LEDs indicating the initiation and termination of a goal-oriented reaching task. Using electroencephalogram signals, we built a decoder to detect the transition between i) rest and beginning MI and ii) maintaining and ending MI. Offline decoder evaluation achieved group average onset accuracy of 60.7% and 66.6% for offset accuracy, revealing that the start and stop of MI could be identified while attached to the robot. Furthermore, pseudo-online evaluation could replicate this performance, forecasting reliable online exoskeleton control in the future. Our approach showed that participants could produce quality and reliable sensorimotor rhythms regardless of noise or passive arm movements induced by wearing the exoskeleton, which opens new possibilities for BMI control of assistive devices.

Adaptively increasing the difficulty level in learning was shown to be beneficial than increasing the level after some fixed time intervals. To efficiently adapt the level, we aimed at decoding the subjective difficulty level based on Electroencephalography (EEG) signals. We designed a visuomotor learning task that one needed to pilot a simulated drone through a series of waypoints of different sizes, to investigate the effectiveness of the EEG decoder. The EEG decoder was compared with another condition that the subjects decided when to increase the difficulty level. We examined the decoding performance together with behavioral outcomes. The online accuracies were higher than the chance level for 16 out of 26 cases, and the behavioral results, such as task scores, skill curves, and learning patterns, of EEG condition were similar to the condition based on manual regulation of difficulty.

This article proposes a novelshared intelligencesystem for brain-machine interface (BMI) teleoperated mobile robots where user’s intention and robot’s intelligence are concurrent elements equally participating in the decision process. We designed the system to rely onpoliciesguiding the robot’s behavior according to the current situation. We hypothesized that the fusion of thesepolicieswould lead to the identification of the next, most probable, location of the robot in accordance with the user’s expectations. We asked 13 healthy subjects to evaluate the system during teleoperated navigation tasks in a crowded office environment with a keyboard (reliable interface) and with 2-class motor imagery (MI) BMI (uncertain control channel). Experimental results show that ourshared intelligencesystem 1) allows users to efficiently teleoperate the robot in both control modalities; 2) it ensures a level of BMI navigation performances comparable to the keyboard control; 3) it actively assists BMI users in accomplishing the tasks. These results highlight the importance of investigating advanced human-machine interaction (HMI) strategies and introducing robotic intelligence to improve the performances of BMI actuated devices.

In search and rescue missions, drone operations are challenging and cognitively demanding. High levels of cognitive workload can affect rescuers' performance, leading to failure with catastrophic outcomes. To face this problem, we propose a machine learning algorithm for real-time cognitive workload monitoring to understand if a search and rescue operator has to be replaced or if more resources are required. Our multimodal cognitive workload monitoring model combines the information of 25 features extracted from physiological signals, such as respiration, electrocardiogram, photoplethysmogram, and skin temperature, acquired in a noninvasive way. To reduce both subject and day inter-variability of the signals, we explore different feature normalization techniques, and introduce a novel weighted-learning method based on support vector machines suitable for subject-specific optimizations. On an unseen test set acquired from 34 volunteers, our proposed subject-specific model is able to distinguish between low and high cognitive workloads with an average accuracy of 87.3% and 91.2% while controlling a drone simulator using both a traditional controller and a new-generation controller, respectively.

Decoding the subjective perception of task difficulty may help improve operator performance, i.e., automatically optimize the task difficulty level. Here, we aim to decode a compound of cognitive states that covaries with the task difficulty level. We designed a protocol composed of two different subtasks, flying and visual recognition, to induce different difficulty levels. We first showed that electroencephalography (EEG) signals can be a reliable source for discriminating different compound states. To gain insight into the underlying components in the compound states, we examined the attentional index and engagement index as in our previous study. We showed that, first, attention and engagement are essential components but fail to provide the best accuracy, and, second, our model is consistent with our previous study, which means that lateralized modulations in the α bands are representative of the flying task. We also analyzed a practical issue in the design of adaptive human-machine interaction (HMI) systems, namely, the latency of changes in the user's compound state. We hypothesized that the EEG correlates of the task difficulty level do not instantaneously reflect the changes in the task difficulty. We validated the hypothesis by measuring the time required for our decoders to provide stable accuracy after the task changed. This amount of time, or latency, could be as high as ten seconds. The results suggest that the latency of changes in the user's compound state between different tasks is a factor that should be taken into account when building adaptive HMI systems.

This work studies the class of algorithms for learning with side-information that emerges by extending generative models with embedded context-related variables. Using finite mixture models (FMMs) as the prototypical Bayesian network, we show that maximum-likelihood estimation (MLE) of parameters through expectation-maximization (EM) improves over the regular unsupervised case and can approach the performances of supervised learning, despite the absence of any explicit ground-truth data labeling. By direct application of the missing information principle (MIP), the algorithms' performances are proven to range between the conventional supervised and unsupervised MLE extremities proportionally to the information content of the contextual assistance provided. The acquired benefits regard higher estimation precision, smaller standard errors, faster convergence rates, and improved classification accuracy or regression fitness shown in various scenarios while also highlighting important properties and differences among the outlined situations. Applicability is showcased with three real-world unsupervised classification scenarios employing Gaussian mixture models. Importantly, we exemplify the natural extension of this methodology to any type of generative model by deriving an equivalent context-aware algorithm for variational autoencoders (VAs), thus broadening the spectrum of applicability to unsupervised deep learning with artificial neural networks. The latter is contrasted with a neural-symbolic algorithm exploiting side information.

Despite the growing interest in brain-machine interface (BMI)-driven neuroprostheses, the translation of the BMI output into a suitable control signal for the robotic device is often neglected. In this article, we propose a novel control approach based on dynamical systems that was explicitly designed to take into account the nature of the BMI output that actively supports the user in delivering real-valued commands to the device and, at the same time, reduces the false positive rate. We hypothesize that such a control framework would allow users to continuously drive a mobile robot and it would enhance the navigation performance. 13 healthy users evaluated the system during three experimental sessions. Users exploit a 2-class motor imagery BMI to drive the robot to five targets in two experimental conditions: with a discrete control strategy, traditionally exploited in the BMI field, and with the novel continuous control framework developed herein. Experimental results show that the new approach: 1) allows users to continuously drive the mobile robot via BMI; 2) leads to significant improvements in the navigation performance; and 3) promotes a better coupling between user and robot. These results highlight the importance of designing a suitable control framework to improve the performance and the reliability of BMI-driven neurorobotic devices.

Recent advances in the Brain-Machine Interface (BMI) and in the robotic fields allowed researchers to design a new generation of brain-actuated neuroprostheses to control a variety of assistive devices-ranging from powered wheelchairs to telepresence robots and robotic arms. However, the current interaction between BMI and robotic applications is still at its infancy. One of the reasons for the limited integration between these two technologies might be identified in the lack of a common research framework. In this scenario, ROS-Neuro may represent a solution to this challenge by providing a common middleware between BMI and robotics.In this work, we propose two packages for the ROS-Neuro middleware to acquire neural signals from different commercial acquisition devices and to store them into common data formats. We describe the usage and the main functionalities of both packages such as the provided nodes, the published and subscribed topics and the available services. Furthermore, the packages are available online together with source codes, tutorials and documentations. We evaluated the performances of the two packages to ensure the absence of delays or missing data with three commercial acquisition systems and in different operating configurations. Results demonstrated the correct behavior of the provided packages. The acquisition and recorder packages represent the first releases in the ROS-Neuro ecosystem and they aim at boosting both the shared development and the research activity in the multidisciplinary field of brain-actuated robots.

Real-time workload estimation can be an important tool to improve human-machine interaction. Despite multiple efforts in this sense, most studies focused on tasks designed to have extreme conditions of workload (high vs low), and these levels remain rather constant throughout task execution. However, some applications may induce changing levels of workload and it is not clear how decoders trained in the extreme conditions will perform. In this study we study this scenario in a simulated drone task where participants had to fly through several waypoints. The task difficulty was modulated by the size of waypoints; in one of the studied conditions, the size was modulated in real time by the EEG-based decoding of the perceived difficulty. We show that this protocol can effectively induce different levels of workload and perceived difficulty. Furthermore, post-experiment analysis using a sparse-regularized technique supports the feasibility of decoding perceived difficulty in this dynamically changing task above chance level (average class-balanced accuracy across subjects 67% ± 7%). Interestingly, an analysis of the selected features in independent recording sessions showed that roughly 20% of the features were stable across multiple sessions.

Estimation of intracranial sources, using inverse solutions methods, has been proposed as a mean to improve performance in non-invasive brain-computer interfaces. These methods estimate the activity of a large number of neural sources from a smaller number of scalp electroencephalography (EEG) channels. This is a highly undetermined problem and regularisation constraints need to be applied. In this paper we compared the effect of several regularisation constraints and parameters in the localisation error and classification performance. Results on three event-related potential protocols-rapid serial visual processing, P300-speller and error-related potentials-showed no significant difference in the maximum performance between minimum norm or weighted minimum norm regularisation constraints. Standardised methods despite yielding lower localisation error resulted in decreased classification performance. Noteworthy, testing on data acquired in different days than the training suggests that discriminant features extracted from intracranial sources are stable across sessions.

Steady State Visually Evoked Potentials (SSVEP) are signals produced in the occipital part of the brain when someone gaze a light flickering at a fixed frequency. These signals have been used for Brain Machine Interfacing (BMI), where one or more stimuli are presented and the system has to detect what is the stimulus the user is attending to. It has been proposed that the SSVEP signal is produced by superposition of Visually Evoked Potentials (VEP) but there is not a model that shows that. We propose a model for a SSVEP signal that is a superposition of the response due to the rising and falling edges of the stimulus and that can be calculated for different frequencies.

Error-related potentials (ErrP) have been increasingly studied in psychophysical experiments as well as for brain-machine interfacing. In the latter case, the generalisation capabilities of ErrP decoders is a crucial element to avoid frequent recalibration processes, thus increasing their usability. Previous studies have suggested that ErrP signals are rather stable across recording sessions. Also, studies using protocols of serial stimuli presentation show that these potentials do not change significantly with the presentation rate. Here we complement these studies by analysing the decoding generalisation capabilities. Using data from monitoring experiments, we evaluate how much the performance degrades when tested in a condition different than the one the decoder was trained with. Moreover, we compare different spatial filtering techniques to see which preprocessing steps yield less-sensitive features for ErrP decoding.

In recent years, several studies have shown that there is a correlation between electroencephalographic (EEG) signals and hand-reaching kinematic parameters after applying linear decoders. These studies have been generally conducted using actual upper limb movements, but so far there has been little discussion about the possibility of applying these decoders to motor imagery tasks. Moreover, the use of these decoders is rather controversial and there is no general agreement about the metrics used to compare decoded and real kinematics. In this paper, we have applied this methodology to upper limb imagined movements using a center-out protocol. Our results show that, although decoding performance is poor, there are significant components, particularly in horizontal imagined movements, that could be translated into reliable output commands. For this purpose, we have proposed a discrete classification of reached targets showing significant classification rates when the number of classified targets decreases.

Psychophysical studies have reported correlation between neural activity in frontal and parietal areas and subject's reaction time in simple tasks. Here we study whether similar correlates can also be identified in driver's electroencephalography (EEG) activity when they perform steering actions triggered by exogenous stimuli (e.g. Obstacles along the road). We report analysis of the EEG signals of fifteen subjects while they drive in a realistic car simulator. We found that the peak latency of the event-related potentials in frontal and parietal areas significantly correlates with the onset of the steering behavior. Similarly, modulations of the power in the theta band (4-8 Hz) prior to the action also correlates with the reaction times. These results provide evidence of reliable neural markers of the driver's response variability.

This paper presents an important step forward towards increasing the independence of people with severe motor disabilities, by using brain-computer interfaces to harness the power of the Internet of Things. We analyze the stability of brain signals as end-users with motor disabilities progress from performing simple standard on-screen training tasks to interacting with real devices in the real world. Furthermore, we demonstrate how the concept of shared control-which interprets the user's commands in context-empowers users to perform rather complex tasks without a high workload. We present the results of nine end-users with motor disabilities who were able to complete navigation tasks with a telepresence robot successfully in a remote environment (in some cases in a different country) that they had never previously visited. Moreover, these end-users achieved similar levels of performance to a control group of 10 healthy users who were already familiar with the environment.

In their early days, brain-computer interfaces (BCIs) were only considered as control channel for end users with severe motor impairments such as people in the locked-in state. But, thanks to the multidisciplinary progress achieved over the last decade, the range of BCI applications has been substantially enlarged. Indeed, today BCI technology cannot only translate brain signals directly into control signals, but also can combine such kind of artificial output with a natural muscle-based output. Thus, the integration of multiple biological signals for real-time interaction holds the promise to enhance a much larger population than originally thought end users with preserved residual functions who could benefit from new generations of assistive technologies. A BCI system that combines a BCI with other physiological or technical signals is known as hybrid BCI (hBCI). In this work, we review the work of a large scale integrated project funded by the European commission which was dedicated to develop practical hybrid BCIs and introduce them in various fields of applications. This article presents an hBCI framework, which was used in studies with nonimpaired as well as end users with motor impairments.

The articles in this special issue focus on brain-computer interfacing. The papers are dedicated to this growing and diversifying research enterprise, and features important review articles as well as some important current examples of research in this area. The field of brain-computer interface (BCI) research began to develop about 25 years ago and transformed from initially isolated demonstrations by a few groups into a large scientific enterprise that is currently producing hundreds of peer-reviewed articles and several dedicated conferences and workshops each year. This level of productivity is reflective of the large and continually growing enthusiasm by the scientific community, funding agencies, and the public.

Operating a brain-actuated vehicle in real-world environments requires much of our visual attention. However, a typical brain-computer interface (BCI) sends the feedback information about the current status of the user's brain also via the visual channel. As a result, users have to split their visual attention into two: one for the surroundings and the other for the visual BCI feedback. Therefore, we recently developed a tactile stimulation system that successfully replaced the conventional visual feedback. Here we employ the multiple object tracking experiments to quantify the visual load added by the visual feedback. The result show that the additional visual load is almost eliminated, and the true negative rate of the BCI operation (intentional non-control) is improved when the visual feedback is replaced by the tactile feedback.

Brain responses to visual stimuli can provide information about visual recognition processes. Several studies have shown stimulus-dependent modulation of the evoked neural responses after gaze shifts (i.e. eye fixation related potentials, EFRP) depending on the relevance of the fixated object. However these studies are typically performed on still images under constrained conditions. Here we extend this approach to study overt visual attention during a simulated driving task. Simultaneous analysis of eye-tracking and electroencephalography data revealed similar patterns than those previously reported. However, natural visual exploration yielded shorter fixations, which imposes constraints in the analysis of the elicited brain responses. Nevertheless, we found significant differences between EFRPs corresponding to target objects or non-object stimuli. These results suggest the possibility of decoding such information during driving, allowing better understanding of how drivers process the environmental information.

One of the challenges in using brain computer interfaces over extended periods of time is the uncertainty in the system. This uncertainty can be due to the user's internal states, the non stationarity of the brain signals, or the variation of the class discriminative information over time. Therefore, the users are often unable to maintain the same accuracy and time efficiency in delivering BCI commands. In this paper, we tackle the issue of variation in BCI command delivery time for a motor imagery task with the aim of providing assistance through adaptive shared control. This is important mainly because having long delivery of mental commands leads to uncertainty in the user's intent classification and limits the responsiveness of the system. In order to address this issue, we separate the trials into long and short groups so that we have the same number of trials in each group. We demonstrate that using only a few samples at the beginning of the trial, we are able to predict whether the current trial will be short or long with high accuracies (70% - 86%). Eventually, this prediction enables us to tune the shared control parameters to overcome the issue of uncertainty.

Activity recognition systems based on body-worn motion sensors suffer from a decrease in performance during the deployment and run-time phases, because of probable changes in the sensors (e.g. displacement or rotatation), which is the case in many real-life scenarios (e.g. mobile phone in a pocket). Existing approaches to achieve robustness tend to sacrifice information (e.g. by rotation-invariant features) or reduce the weight of the anomalous sensors at the classifier fusion stage (adaptive fusion), ignoring data which might still be perfectly meaningful, although different from the training data. We propose to use adaptation to rebuild the classifier models of the sensors which have changed position by a two-step approach: in the first step, we run an anomaly detection algorithm to automatically detect which sensors are delivering unexpected data; subsequently, we trigger a system self-training process, so that the remaining classifiers retrain the “anomalous” sensors. We show the benefit of this approach in a real activity recognition dataset comprising data from 8 sensors to recognize locomotion. The approach achieves similar accuracy compared to the upper baseline, obtained by retraining the anomalous classifiers on the new data.

The prospect of controlling devices merely by the power of one's thoughts is compelling, especially for assistive technology applications. In the accompanying video, we show how we have strived to push brain-computer interface (BCI) technology out of the lab and into the real world, while simultaneously moving away from testing solely with healthy subjects to undertaking trials with patients and potential end-users. We describe the evolution of the motor imagery based BCI, which has resulted in a major milestone: the first patient trial of a motor imagery based BCI controlled wheelchair.

In this paper we propose a method to modulate the level of assistance provided by a shared controller, not only given the environmental context, but also according to the context of the user's current behaviour. We show that the enhanced situational context can be adequately captured by using online performance metrics (such as those more usually found in the evaluation of shared control systems). The resultant controller not only allows the user to perform better in the primary task (like many shared control systems), but has also has increased the level of user acceptance, due to the personalised dynamics of the control policy.

Providing adaptive shared control for Brain-Computer Interfaces (BCIs) can result in better performance while reducing the user's mental workload. In this respect, online estimation of accuracy and speed of command delivery are important factors. This study aims at real-time differentiation between fast and slow trials in a motor imagery BCI. In our experiments, we refer to trials shorter than the median of trial lengths as “fast” trials and to those longer than the median as “slow” trials. We propose a classifier for real-time distinction between fast and slow trials based on estimates of the entropy rates for the first 2-3 s of the electroencephalogram (EEG). Results suggest that it can be predicted whether a trial is slow or fast well before a cutoff time. This is important for adaptive shared control especially because 55% to 75% of trials (for the five subjects in this study) are longer than that cutoff time.

Activity recognition using on body sensors are prone to degradation due to changes on sensor readings. The changes can occur because of degradation or alteration in the behaviour of the sensor with respect to the others. In this paper we propose a method which detects anomalous nodes in the network and takes compensatory actions to keep the performance of the system as high as possible while the system is running. We show on two activity datasets with different configurations of on body sensors that detection and compensation of anomalies make the system more robust against the changes.

The use of brain-generated signals for human-robot interaction has gained increasing attention in the last years. Indeed brain-controlled robots can potentially be employed to substitute motor capabilities (e.g. brain-controlled prosthetics for amputees or patients with spinal cord injuries); to help in the restoration of such functions (e.g. as a tool for stroke rehabilitation) as well as non-clinical applications like telepresence or entertainment. This half-day tutorial gives an introduction to the field of brain-computer interfaces and presents several design principles required to successfully employ them for robot control.

For people with severe physical disabilities, low resolution input devices, such as buttons, sip and puff switches and brain-computer interfaces provide an opportunity to interact with the world. However, it can be difficult to control assistive technology, such as wheelchairs, tele-presence robots and robotic arms, when you have only a limited number of commands available and/or a lack of temporal precision in issuing such commands. These limitations can be overcome by employing shared control techniques, whereby the system assists the user in performing the desired task. In this study we compare the use of a simple discrete shared control policy with a more dynamic proportional shared control policy. We evaluate both approaches on a wheelchair that is only operated by two temporally-constrained discrete buttons. The experiments were performed in two different realistic indoor scenarios: an open-plan, spacious environment and a smaller, more cluttered office environment. A total of 10 healthy participants took part in this study.

Advances in sensing, portable computing devices, and wireless communication has lead to an increase in the number and variety of sensing enabled devices (e.g. smartphones or sensing garments). Pervasive computing and activity recognition systems should be able to take advantage of these sensors, even if they are not always available or appear in runtime. These sensors can be integrated into an ensemble that fuse their information to obtain the final decision. There is therefore a need for mechanisms to select which sensors should compose the ensemble, as well as techniques for dynamically reconfigure the ensemble so as to integrate new sensors. Sensors can be integrated into an ensemble where information from each of them is fused to obtain the final decisions. From the machine learning point of view, this corresponds to the combination of classifiers where measures of the accuracy and diversity of the ensemble are used to select the elements that may lead to the highest performance. Recent works propose measures of accuracy and diversity based on an information theoretical approach. In this paper we study the use of these measures for selecting ensembles in activity recognition based on body sensor networks. Besides a comparison with traditional diversity measures (e.g., Q-, κ-statistics), we also present mechanisms to exploit these measures for the dynamic reconfiguration of the ensemble and detection of changes in the network.

This paper presents a semi-autonomous navigation strategy aimed at the control of assistive devices (e.g. an intelligent wheelchair) using low throughput interfaces. A mobile robot proposes the most probable action, as analyzed from the environment, to a human user who can either accept or reject the proposition. In case of rejection, the robot will propose another action, until both entities agree on what needs to be done. In a known environment, the system infers the intended goal destination based on the first executed actions. Furthermore, we endowed the system with learning capabilities, so as to learn the user habits depending on contextual information (e.g. time of the day or if a phone rings). This additional knowledge allows the robot to anticipate the user intention and propose appropriate actions, or goal destinations.

Human activity recognition is a thriving research field. There are lots of studies in different sub-areas of activity recognition proposing different methods. However, unlike other applications, there is lack of established benchmarking problems for activity recognition. Typically, each research group tests and reports the performance of their algorithms on their own datasets using experimental setups specially conceived for that specific purpose. In this work, we introduce a versatile human activity dataset conceived to fill that void. We illustrate its use by presenting comparative results of different classification techniques, and discuss about several metrics that can be used to assess their performance. Being an initial benchmarking, we expect that the possibility to replicate and outperform the presented results will contribute to further advances in state-of-the-art methods.

Opportunistic activity and context recognition systems draw from the characteristic to use sensing devices that just happen to be available instead of pre-defining them at the design time of the system in order to achieve a recognition goal at runtime. Whenever a user and/or application states a recognition goal at runtime to the system, the available sensing devices configure an ensemble of the best available set of sensors for the specified recognition goal. This paper presents an approach to show how machine learning technologies (classification, fusion and anomaly detection) are integrated in a prototypical opportunistic activity and context recognition system (referred to as the OPPORTUNITY Framework). We define a metric that quantifies the ensemble's capabilities according to a recognition goal and evaluate the approach with respect to the requirements of an opportunistic system (e.g. to compute an ensemble's configuration and reconfiguration at runtime).

This paper discusses and evaluates the role of shared control approach in a BCI-based telepresence framework. Driving a mobile device by using human brain signals might improve the quality of life of people suffering from severely physical disabilities. By means of a bidirectional audio/video connection to a robot, the BCI user is able to interact actively with relatives and friends located in different rooms. However, the control of robots through an uncertain channel as a BCI may be complicated and exhaustive. Shared control can facilitate the operation of brain-controlled telepresence robots, as demonstrated by the experimental results reported here. In fact, it allows all subjects to complete a rather complex task, driving the robot in a natural environment along a path with several targets and obstacles, in shorter times and with less number of mental commands.

For people with very severe motor dysfunctions, Brain-Computer Interfaces (BCIs) may provide the solution to regain mobility and manipulation capabilities. Unfortunately, BCIs are characterized by a limited bandwidth and uncertainty on the BCI output. In the past, we have developed a Bayesian plan recognition framework that estimates from uncertain human-robot interface signals the task a robot should execute. This paper extends our plan recognition framework to incorporate uncertain BCI signals. A benchmark test is proposed and adopted to evaluate both the plan recognition framework and the performance of the BCI user, for the concrete application of wheelchair driving.

Opportunistic sensing allows to efficiently collect information about the physical world and the persons behaving in it. This may mainstream human context and activity recognition in wearable and pervasive computing by removing requirements for a specific deployed infrastructure. In this paper we introduce the newly started European research project OPPORTUNITY within which we develop mobile opportunistic activity and context recognition systems. We outline the project's objective, the approach we follow along opportunistic sensing, data processing and interpretation, and autonomous adaptation and evolution to environmental and user changes, and we outline preliminary results.

This paper presents a comparison between six different ways to convey navigational information provided by a robot to a human. Visual, auditory, and tactile feedback modalities were selected and designed to suggest a direction of travel to a human user, who can then decide if he agrees or not with the robot's proposition. This work builds upon a previous research on a novel semi-autonomous navigation system in which the human supervises an autonomous system, providing corrective monitoring signals whenever necessary.

The promise of Brain-Computer Interfaces (BCI) technology is to augment human capabilities by enabling interaction with computers through a conscious and spontaneous modulation of the brainwaves after a short training period. Indeed, by analyzing brain electrical activity online, several groups have designed brain-actuated devices that provide alternative channels for communication, entertainment and control. Thus, a person can write messages using a virtual keyboard on a computer screen and also browse the internet. Alternatively, subjects can operate simple computer games, or brain games, and interact with educational software. Work with humans has shown that it is possible for them to move a cursor and even to drive a wheelchair. This paper briefly reviews the field of BCI, with a focus on noninvasive systems based on electroencephalogram (EEG) signals. It also describes three brain-actuated devices we have developed: a virtual keyboard, a brain game, and a wheelchair. Finally, it shortly discusses current research directions we are pursuing in order to improve the performance and robustness of our BCI system, especially for real-time control of brain-actuated robots.

Brain-computer interfaces (BCIs), as any other interaction modality based on physiological signals and body channels (e.g., muscular activity, speech and gestures), are prone to errors in the recognition of subject's intent. An elegant approach to improve the accuracy of BCIs consists in a verification procedure directly based on the presence of error-related potentials (ErrP) in the EEG recorded right after the occurrence of an error. Six healthy volunteer subjects with no prior BCI experience participated in a new human-robot interaction experiment where they were asked to mentally move a cursor towards a target that can be reached within a few steps using motor imagination. This experiment confirms the previously reported presence of a new kind of ErrP. These Interaction ErrP" exhibit a first sharp negative peak followed by a positive peak and a second broader negative peak (~290, ~350 and ~470 ms after the feedback, respectively). But in order to exploit these ErrP we need to detect them in each single trial using a short window following the feedback associated to the response of the classifier embedded in the BCI. We have achieved an average recognition rate of correct and erroneous single trials of 81.8% and 76.2%, respectively. Furthermore, we have achieved an average recognition rate of the subject's intent while trying to mentally drive the cursor of 73.1%. These results show that it's possible to simultaneously extract useful information for mental control to operate a brain-actuated device as well as cognitive states such as error potentials to improve the quality of the brain-computer interaction. Finally, using a well-known inverse model (sLORETA), we show that the main focus of activity at the occurrence of the ErrP are, as expected, in the pre-supplementary motor area and in the anterior cingulate cortex."

In this paper, we investigate the use of brain activity for person authentication. It has been shown in previous studies that the brain-wave pattern of every individual is unique and that the electroencephalogram (EEG) can be used for biometric identification. EEG-based biometry is an emerging research topic and we believe that it may open new research directions and applications in the future. However, very little work has been done in this area and was focusing mainly on person identification but not on person authentication. Person authentication aims to accept or to reject a person claiming an identity, i.e., comparing a biometric data to one template, while the goal of person identification is to match the biometric data against all the records in a database. We propose the use of a statistical framework based on Gaussian Mixture Models and Maximum A Posteriori model adaptation, successfully applied to speaker and face authentication, which can deal with only one training session. We perform intensive experimental simulations using several strict train/test protocols to show the potential of our method. We also show that there are some mental tasks that are more appropriate for person authentication than others.

The promise of Brain-Computer Interfaces (BCI) technology is to augment human capabilities by enabling people to interact with a computer through a conscious and spontaneous modulation of their brainwaves after a short training period. Indeed, by analyzing brain electrical activity online, several groups have designed brain-actuated systems that provide alternative channels for communication, entertainment and control. Thus, a person can write messages using a virtual keyboard on a computer screen and also browse the internet. Alternatively, subjects can operate simple computer games, or brain games, and interact with educational software. Researchers have also been able to train monkeys to move a computer cursor to desired targets and also to control a robot arm. Work with humans has shown that it is possible for them to move a cursor and even to drive a mobile robot between rooms in a house model. In this talk I will review the field of BCI, with a focus on non-invasive systems based on electroencephalogram (EEG) signals. I will also describe three brain-actuated applications we have developed: a virtual keyboard, a brain game, and a mobile robot (emulating a motorized wheelchair). Finally, we discuss current research directions we are pursuing in order to improve the performance and robustness of our BCI system, especially for real-time control of brain-actuated robots.

We motivate the need for on-line learning in brain-computer interfaces (BCI) and illustrate its benefits with the simplest method, namely fixed learning rates. However, the use of this method is supported by the risk of hampering the user to acquire suitable control of the BCI if the embedded classifier changes too rapidly. We report the results with 3 beginner subjects in a series of consecutive recordings, where the classifiers are iteratively trained with the data of a given session and tested on the next session. Interestingly, performance improved over sessions significantly for 2 of the subjects. These results show that on-line learning improves systematically the performance of the subjects. Moreover, performance with online learning is statistically similar to that obtained training the classifier off-line with the same amount of data.

This paper proposes a novel and simple local neural classifier for the recognition of mental tasks from on-line spontaneous EEG signals. The proposed neural classifier recognizes three mental tasks from on-line spontaneous EEG signals. Correct recognition is around 70%. This modest rate is largely compensated by two properties, namely low percentage of wrong decisions (below 5%) and rapid responses (every 1/2 s). Interestingly, the neural classifier achieves this performance with a few units, normally just one per mental task. Also, since the subject and his/her personal interface learn simultaneously from each other, subjects master it rapidly (in a few days of moderate training). Finally, analysis of learned EEG patterns confirms that for a subject to operate satisfactorily a brain interface, the latter must fit the individual features of the former.

We present preliminary classification results for a real time brain-computer interface. Our approach seeks to build individual brain interfaces rather than universal ones. This means that the interface should adapt to its owner; as it will incorporate a neural classifier that learns user-specific features. Three feature sets extracted with Fourier transform, autoregressive models and wavelets were evaluated with early stopping MLP committee. The goal was to classify EEG patterns related to imagined hand movements and relax. The best results were obtained with the autoregressive special features. The results so far are not satisfactory for their intended use as basis for robust EEG classification but they give us valuable basis for future work.

This paper presents an adaptive method that allows mobile robots to learn cognitive maps of indoor environments incrementally and online. Our approach models the environment. By means of a variable-resolution partitioning that discretizes the world in perceptually homogeneous regions. The resulting model incorporates both a compact geometrical representation of the environment and a topological map of the spatial relationships between its obstacle-free areas. The efficiency of the learning process is based on the use of local memory-based techniques for partitioning and of active learning techniques for selecting the most appropriate region to be explored next. In addition, a feedforward neural network is used to interpret sensor readings. We present experimental results obtained with two different mobile robots, the Nomad 200 and Khepera. The current implementation of the method relies on the assumption that obstacles are parallel or perpendicular to each other. This results in variable-resolution partitioning consisting of simple rectangular partitions and reduces the complexity of treating the underlying geometrical properties.

This paper presents a modular neural controller that learns goal-oriented obstacle-avoiding motion strategies for a sensor-based three-link planar robot arm. It acquires these strategies through reinforcement learning from local sensory data. The controller has two reinforcement-based modules: a module for negotiating obstacles and a module for moving to the goal. Both modules generate actions that are interpreted with regard to a goal vector in the robot joint space. A differential inverse kinematics (DIV) module is used to obtain such a goal vector. The DIV module is based on the inversion of a neural network that has been previously trained to approximate the manipulator forward kinematics in polar coordinates. The controller achieves a satisfactory performance quite rapidly and shows good generalization capabilities in the face of new environments.

This paper presents a neural controller that learns goal-oriented obstacle-avoiding reaction strategies for a multilink robot arm. It acquires these strategies through reinforcement learning from local sensory data. The robot arm has rings of range sensors placed along its links. The neural controller achieves a good performance quite rapidly and shows good generalization abilities in the face of new environments. Suitable input and output codification schemes help greatly to attain these aims. The input codification exploits the inherent symmetry of the robot kinematics and the action given by the controller is interpreted with regard to the shortest path vector (SPV) to the closest goal in the configuration space. In order to avoid the SPV computation for multilink manipulators, we put forward the use of a module for differential inverse kinematics based on the inversion of a neural network that has been previously trained to approximate the manipulator forward kinematics. The use of this module does not only get round the SPV calculation, but also speeds up the learning process.

In this paper we propose a reinforcement connectionist learning architecture that allows an autonomous robot to acquire efficient navigation strategies in a few trials. Besides rapid learning, the architecture has three further appealing features. First, the robot improves its performance incrementally as it interacts with an initially unknown environment, and it ends up learning to avoid collisions even in those situations in which its sensors cannot detect the obstacles. This is a definite advantage over nonlearning reactive robots. Second, since it learns from basic reflexes, the robot is operational from the very beginning and the learning process is safe. Third, the robot exhibits high tolerance to noisy sensory data and good generalization abilities. All these features make this learning robot's architecture very well suited to real-world applications. We report experimental results obtained with a real mobile robot in an indoor environment that demonstrate the appropriateness of our approach to real autonomous robot control.

Proposes a reinforcement learning architecture that allows an autonomous robot to acquire efficient navigation strategies in a few trials. Besides fast learning, the architecture has 3 further appealing features. (1) Since it learns from built-in reflexes, the robot is operational from the very beginning. (2) The robot improves its performance incrementally as it interacts with an initially unknown environment, and it ends up learning to avoid collisions even if its sensors cannot detect the obstacles. This is a definite advantage over non-learning reactive robots. (3) The robot exhibits high tolerance to noisy sensory data and good generalization abilities. All these features make this learning robot's architecture very well suited to real-world applications. The authors report experimental results obtained with a real mobile robot in an indoor environment that demonstrate the feasibility of this approach.>