Rehabilitation following total knee replacement (TKR) principally aims to reduce pain and restore knee muscle strength and function, which necessitates objective monitoring. Given the growing interest in wearable technologies for rehabilitation, this paper investigates the feasibility of using electrical impedance tomography (EIT) to monitor and estimate muscle activity, specifically focusing on resultant knee angle (KA) and muscle strength (MS). A data collection system was designed to simultaneously capture EIT data, kinematic and physiological reference data, namely KA and MS, followed by two experiments. Data processing methods, such as region of interest (ROI) selection and modelling of the correlation between EIT and reference data, were used to assess feasibility. The results indicate that the long short-term memory (LSTM) model achieved an R2 value of 0.97, suggesting a strong correlation between the EIT data and KA. While EIT demonstrated potential in detecting KA, it did not show effectiveness in capturing changes in muscle activity and MS during isometric contractions.

Speech impairments caused by severe diseases and injuries are serious health problems. The deprivation of communication ability can significantly decrease the quality of daily life. Inner speech is a natural mental activity used by many healthy and disabled people with intact cognitive ability. With the help of Brain-Computer Interfaces (BCIs), imagined speech can be decoded as semantic output or downstream commands for external devices, showing great potential for intuitive neural interface control. As many works have demonstrated promising results with invasive brain recordings, it remains a challenge for non-invasive BCIs to realize reliable speech decoding due to the trade-off between safety and signal quality. In this study, we explored the feasibility and neural mechanism behind a non-invasive brain recording technique based on electroencephalogram (EEG) during speech imagery with 6 English phonemes. We found significant time-frequency representation differences between /b/ and /u:/ and showed feasibility for pair-wise imagined phoneme classification with Filter Bank Common Spatial Pattern. We also demonstrated the EEG neural representation of phonemes in the latent space and how they are separated. Our results suggested that for naïve BCI users, subject-specific phoneme pairs yielded the best performance. Finally, we discussed the challenges and potential optimization directions for future EEG-based speech BCIs.

Smart robotic wheelchairs, as well as other assistive robotic devices, can provide an effective form of independent mobility for those who suffer with motor disabilities. Although many control interfaces exist to operate these devices, brain computer interfaces (BCI) offer a control modality for those who have little to no motor function, as well as being able to re-associate movement with brain functionality. Although BCIs have been designed for robotic wheelchairs, more research and development is required before they can be adopted for use in the ‘real world’. One key challenge on that journey is the user training required to achieve an acceptable accuracy of the control. In this paper, we aim to identify the best training method by comparing users trained on a simple task, in a simulated environment on a 2D display (VR-2DD) and in a virtual environment using a virtual reality headset (VR-HMD). We trained 15 participants in mix of high and low noise virtual environments or on a simple training task, and found a significant improvement in the classification accuracies of the participants who trained using the VR-2DD task compared with those who were trained with the simple task. We also carried out active (online) tests across all participants in the same virtual training environment, with a varying level of external stimuli, and found a significant improvement in the performance of participants in both VR groups compared to participants in the simple task group.

Many studies have shown that a smart wheelchair could improve the quality of life of people with restricted mobility by providing them with more freedom in the daily activities they can undertake independently. In addition to enhancing independent mobility, it is important to ensure safety for wheelchair users and those around them. To date, previous studies have mostly focused on (semi-)autonomous navigation or obstacle avoidance. By contrast, in this study, we tackle the challenging, but important problem of safely docking to tables. We propose a robotic navigation assistance, applied to electric powered wheelchairs using Time-of-Flight (ToF) sensors to facilitate table-docking for users. To meet this objective, we designed a low-cost sensor system that was integrated into our smart wheelchair prototype, which can detect a table and accurately estimate its height. We then developed a robust algorithm to deliver the manoeuvring assistance. First, we simulated the smart wheelchair system within Unity3D to find the best positions for the ToF sensors and evaluate the accuracy of the docking system, employing different table styles. Then, we experimentally validate the system on our physical wheelchair, using varying angles of approach, which demonstrate its feasibility.

Shared control is a mode where the user input is combined with a planned motion to achieve a common goal. In navigation, a shared control approach could provide a potential mobility solution for people who have a mobility impairment and find traditional powered wheelchairs unsuitable. While state-of-the-art work in shared control has demonstrated its capability in improving safety, human-machine interaction and reduce confusion, it is still challenging to use shared control in dynamic, crowded scenarios, in a way that is acceptable to users. Learning from recent advances in robot navigation, we present a reinforcement learning based framework, which allows navigation to be achieved in a user-specific shared controlled way. Our approach was trained and tested in a Unity3D based simulator. It achieved 33% fewer collisions, similar high user agreement (≤ 85%) and 27% less completion time when compared with our previous model-based method.

Shared control wheelchairs can help users to navigate through crowds by enabling the person to drive the wheelchair while receiving support in avoiding pedestrians. To date, research into shared control has largely overlooked the perspectives of wheelchair users. In this paper, we present two studies that aim to address this gap. The first study involved a series of semi-structured interviews with wheelchair users which highlighted the presence of two different interaction loops, one between the user and the wheelchair and a second one between the user and the crowd. In the second study we engaged with wheelchair users and designers to co-design appropriate feedback loops for future shared control interaction interfaces. Based on the results of the co-design session, we present design implications for shared control wheelchair around the need for empathy, embodiment and social awareness; situational awareness and adaptability; and selective information management.

An assist-as-needed semi-autonomous control algorithm is designed to address the problem of safely driving a vehicle (a power wheelchair) in an environment with static obstacles. The main idea is to maximize the human driver’s control experience while allowing them to navigate safely (the inputs do not lead to collisions). The proposed physically-inspired model-based obstacle avoidance algorithm relies on optimal maps of the expected time for executing a safe stop manoeuvre. These maps are pre-computed using policy iteration in the case of an experienced driver stochastic model. As the burden of complex calculations is handled offline, the online implementation of the algorithm requires little computing resources. Its efficiency is tested experimentally in a study with healthy participants: a statistically significant result confirmed that the proposed algorithm outperforms a baseline rule-based control. A discussion with pros and cons ends this paper.

Shared control, where a human user cooperates with an algorithm to operate a device, has the potential to greatly expand access to powered mobility, but also raises unique ethical challenges. A shared-control wheelchair may perform actions that do not reflect its user's intent in order to protect their safety, causing frustration or distrust in the process. Unlike physical accidents there is currently no frame-work for investigating or adjudicating these events, leading to a reduced capability to improve the shared control algorithm's user experience. In this paper we suggest a system based on the idea of an ‘ethical black box' that records the sensor context of sub-critical disagreements and collision risks in order to allow human investigators to examine them in retrospect and assess whether the algorithm has taken control from the user without justification.

For people who have a mobility impairment and find standard wheelchairs unsuitable, a shared-controlled approach could provide a potential mobility solution. However, state-of-the-art research on shared control wheelchairs mainly focus on static environments. In this paper, we present a hierarchical design for our shared-controlled wheelchair using a velocity-based approach together with probabilistic shared control (PSC). By modifying the collision avoidance element and model the robot-pedestrian interaction based on their physical distance, we extended the implementation of PSC to dynamic environments. Our approach was tested in a Unity3D based simulator with human participants. It achieved least number of collisions while obtaining relatively low computational cost and high user agreement comparing with other state-of-the-art methods.

Power wheelchairs allow people with motor disabilities to have more mobility and independence. However, driving safely such a vehicle is a daily challenge particularly in urban environments while navigating on sidewalks, negotiating curbs or dealing with uneven grounds. Indeed, differences of elevation have been reported to be one of the most challenging environmental barrier to negotiate, with tipping and falling being the most common accidents power wheelchair users encounter. It is thus our challenge to design assistive solutions for power wheelchair navigation in order to improve safety while navigating in such environments. To this aim, we propose a shared-control algorithm which provides assistance while navigating with a wheelchair in an environment consisting of negative obstacles. We designed a dedicated sensor-based control law allowing trajectory correction while approaching negative obstacles e.g. steps, curbs, descending slopes. This shared control proposed method takes into account the human-in-the loop factor. In this study, our solution the ability of our system to ensure a safe trajectory while navigating on a sidewalk is demonstrated through simulation, thus providing a proof-of-concept of our method.

This article presents progress made towards implementing a shared control framework for a smart wheelchair based upon stochastic dynamic programming (a model-based control design). First, we describe the mechanical, electrical and software design process of our instrumented wheelchair platform. Then, we detail a deterministic control-oriented model of the wheelchair motion dynamics using Euler-Lagrange equations. Finally, we discuss the development of a stochastic model of the human driver’s intention in view of using Markov chain.

Measuring manual wheelchair activity by using wearable sensors is becoming increasingly common for rehabilitation and monitoring purposes. Until recently most research has focused on the identification of activities of daily living or on counting the number of strokes. However, how a person pushes their wheelchair - their stroke pattern - is an important descriptor of the wheelchair user's quality of movement. This paper evaluates the capability of inertial sensors located at different upper limb locations plus the wheel of the wheelchair, to classify two types of stroke pattern for manual wheelchairs: semicircle and arc. Data was collected using bespoke inertial sensors with a wheelchair fixed to a treadmill. Classification was completed with a linear SVM algorithm, and classification performance was computed for each sensor location in the upper limb, and then in combination with wheel sensor. For single sensors, forearm location had the highest accuracy (96%) followed by hand (93%) and arm (90%). For combined sensor location with wheel, best accuracy came in combination with forearm. These results set the direction towards a wearable wheelchair monitor that can measure the quality as well as the quantity of movement and which offers multiple on-body locations for increased usability.

Shared control is an increasingly popular approach to facilitate control and communication between humans and intelligent machines. However, there is little consensus in guidelines for design and evaluation of shared control, or even in a definition of what constitutes shared control. This lack of consensus complicates cross fertilization of shared control research between different application domains. This paper provides a definition for shared control in context with previous definitions, and a set of general axioms for design and evaluation of shared control solutions. The utility of the definition and axioms are demonstrated by applying them to four application domains: automotive, robot-assisted surgery, brain-machine interfaces, and learning. Literature is discussed for each of these four domains in light of the proposed definition and axioms. Finally, to facilitate design choices for other applications, we propose a hierarchical framework for shared control that links the shared control literature with traded control, co-operative control, and other human-automation interaction methods. Future work should reveal the generalizability and utility of the proposed shared control framework in designing useful, safe, and comfortable interaction between humans and intelligent machines.

Children with motor or cognitive impairments who require powered mobility at a very young age will face social and environmental barriers that make learning how to use the mobility device a challenging task. We present a first approach of a framework to help therapists and service providers to assess and monitor how children use their mobility device, which results from the combination of a plug and play inertial sensor, and the support of the Assessment Learning tool (ALP) from Nilsson and Durkin. We performed a formative study on four able-bodied children using an electric wheelchair. Results suggest it is possible to measure children's driving skills with this approach, and that results can be mapped to the validated ALP tool. We present the limitations of our study and the direction of future work.

Independent mobility is important for the self-esteem and well-being of people with mobility impairments. For people with severe disabilities, there is a body of research investigating how best to share control of motion between a person with disabilities and a "smart wheelchair". Traditionally in "shared control", the control law is a linear combination of the human's intended velocity and the path planner's velocity. However, this formulation of sharing control between a human and a machine does not guarantee safety on a theoretical level. To guarantee safety in formulating the blending of the human's input velocity and planner's velocity, we implement a practical form of probabilistic shared control formulated by Trautman. We tested this shared control by conducting experiments in a simulation where participants drive a wheelchair. The results of the experiment suggest probabilistic shared control has similar performance to linear blending in terms of significant reduction in number of collisions. However, for the sip-puff switch (a particularly difficult interface to use), probabilistic shared control yielded a greater reduction in collisions than linear blending.

This paper presents an important step forward towards increasing the independence of people with severe motor disabilities, by using brain-computer interfaces to harness the power of the Internet of Things. We analyze the stability of brain signals as end-users with motor disabilities progress from performing simple standard on-screen training tasks to interacting with real devices in the real world. Furthermore, we demonstrate how the concept of shared control-which interprets the user's commands in context-empowers users to perform rather complex tasks without a high workload. We present the results of nine end-users with motor disabilities who were able to complete navigation tasks with a telepresence robot successfully in a remote environment (in some cases in a different country) that they had never previously visited. Moreover, these end-users achieved similar levels of performance to a control group of 10 healthy users who were already familiar with the environment.

Shared control is an exciting up-and-coming engineering field that is blending the boundaries of control, where humans interact with robots or vehicles that are partly automated. The main challenges for robotics and automation today are posed by the less structured, unpredictable environments, in which humans operate naturally, especially when they need to interact with other humans. As a result, many semi-automated systems today need to be supervised by human operators, but as fast as the levels of automation of many systems are increasing, we also need to speed up how we think about what this implies for human-machine interaction. The widely applied paradigm of human-centered automation has been popular and useful for the past two decades, but it requires an update with the current trend in automation becoming more ubiquitous in human environments and less dependent on explicit input from the human operator. In contrast to the supervisory control paradigm---where control is traded between human and machine---the shared control paradigm implicitly assumes the interaction between two or more independent agents that together perform a task to achieve a common goal. This implies that the design of the shared control system is not necessarily only human centered. In shared control systems, all the acting agents need to be aware of the others' capabilities, weaknesses, and authority. Hence, reciprocal communication of each agent's operational boundaries, whether human or machine, is essential.

One of the challenges in using brain computer interfaces over extended periods of time is the uncertainty in the system. This uncertainty can be due to the user's internal states, the non stationarity of the brain signals, or the variation of the class discriminative information over time. Therefore, the users are often unable to maintain the same accuracy and time efficiency in delivering BCI commands. In this paper, we tackle the issue of variation in BCI command delivery time for a motor imagery task with the aim of providing assistance through adaptive shared control. This is important mainly because having long delivery of mental commands leads to uncertainty in the user's intent classification and limits the responsiveness of the system. In order to address this issue, we separate the trials into long and short groups so that we have the same number of trials in each group. We demonstrate that using only a few samples at the beginning of the trial, we are able to predict whether the current trial will be short or long with high accuracies (70% - 86%). Eventually, this prediction enables us to tune the shared control parameters to overcome the issue of uncertainty.

The prospect of controlling devices merely by the power of one's thoughts is compelling, especially for assistive technology applications. In the accompanying video, we show how we have strived to push brain-computer interface (BCI) technology out of the lab and into the real world, while simultaneously moving away from testing solely with healthy subjects to undertaking trials with patients and potential end-users. We describe the evolution of the motor imagery based BCI, which has resulted in a major milestone: the first patient trial of a motor imagery based BCI controlled wheelchair.

In this paper we propose a method to modulate the level of assistance provided by a shared controller, not only given the environmental context, but also according to the context of the user's current behaviour. We show that the enhanced situational context can be adequately captured by using online performance metrics (such as those more usually found in the evaluation of shared control systems). The resultant controller not only allows the user to perform better in the primary task (like many shared control systems), but has also has increased the level of user acceptance, due to the personalised dynamics of the control policy.

Powered wheelchair users often struggle to drive safely and effectively and, in more critical cases, can only get around when accompanied by an assistant. To address these issues, we propose a collaborative control mechanism that assists users as and when they require help. The system uses a multiple-hypothesis method to predict the driver's intentions and, if necessary, adjusts the control signals to achieve the desired goal safely. The main emphasis of this paper is on a comprehensive evaluation, where we not only look at the system performance but also, perhaps more importantly, characterize the user performance in an experiment that combines eye tracking with a secondary task. Without assistance, participants experienced multiple collisions while driving around the predefined route. Conversely, when they were assisted by the collaborative controller, not only did they drive more safely but also they were able to pay less attention to their driving, resulting in a reduced cognitive workload. We discuss the importance of these results and their implications for other applications of shared control, such as brain-machine interfaces, where it could be used to compensate for both the low frequency and the low resolution of the user input.

For people with severe physical disabilities, low resolution input devices, such as buttons, sip and puff switches and brain-computer interfaces provide an opportunity to interact with the world. However, it can be difficult to control assistive technology, such as wheelchairs, tele-presence robots and robotic arms, when you have only a limited number of commands available and/or a lack of temporal precision in issuing such commands. These limitations can be overcome by employing shared control techniques, whereby the system assists the user in performing the desired task. In this study we compare the use of a simple discrete shared control policy with a more dynamic proportional shared control policy. We evaluate both approaches on a wheelchair that is only operated by two temporally-constrained discrete buttons. The experiments were performed in two different realistic indoor scenarios: an open-plan, spacious environment and a smaller, more cluttered office environment. A total of 10 healthy participants took part in this study.

Powered wheelchairs play a vital role in bringing independence to the severely mobility-impaired. Our robotic wheelchair aims to assist users in driving safely, without undermining their capabilities or curtailing the natural development of their skills. An important research question is to determine the conditions under which shared control is most beneficial. In this paper, we describe an experiment, where a distracting secondary task caused the majority of participants to crash the wheelchair when driving without assistance. However, when they were assisted by our collaborative controller, not only did they drive safely, but they also increased their performance in the secondary task. We demonstrate that a degree of shared control is beneficial even to proficient drivers under certain circumstances, for instance when they are under a heightened workload.

Powered wheelchair users want to be active drivers, not just passengers. However, in some situations (varying from person to person), they may require assistance; hence, research is being carried out into the development of 'smart' wheelchairs. Predominantly, this research has been derived from the field of mobile robotics, focussing on creating autonomous systems, which unfortunately tend to treat the human as little more than a precious piece of cargo. Instead, the design should be based around each individual user's abilities and desires, maximising the amount of control they are given. In this paper, we look at how collaborative control techniques can be used to achieve this, offering the user help, as and when it is required. We then evaluate the effects of this collaboration, which is built by predicting user intentions and responding to these predictions with adaptable levels of assistance.