Personalized robotic exoskeleton control is essential in assisting individuals with motor deficits. However, current research still lacks a solution from the end of a practical need of the problem to the end of its successful demonstration in physical environments, namely an end-to-end solution, that enables stable and continuous walking across different tasks. This study addresses this challenge by introducing a hierarchical control framework for the purpose. At the low level, impedance control ensures joint compliance without causing injury to users. At the high level, a reinforcement learning (RL)-based optimal adaptive controller automatically personalizes assistance to both hip extension and flexion (namely, bi-directional) to reach a target range of motion (ROM) under multiple walking conditions. As the first potentially feasible approach to this challenging problem and to meet practical use requirements, we developed a least-square policy iteration-based solution to configure the intrinsic parameters within the well-established finite state machine impedance control (FSM-IC). We successfully tested the control solution on eight young unimpaired participants and one participant post-stroke wearing a hip exoskeleton while walking on an instrumented treadmill. The proposed method can be applied to solving for optimal impedance parameters for individual users and different task scenarios to increase joint ROM. Our next step is to further evaluate this solution framework on additional people with hemiparesis who may benefit from hip joint assistance in therapy or daily activities to restore normative or improve gait patterns.

Human-in-the-loop (HIL) optimization usually optimizes assistive torque of exoskeletons to minimize the human's energetic expenditure in walking, quantified by metabolic cost. This formulation can, however, result in altered gait pattern of the human joint from the natural pattern, which is undesired. In this paper, we proposed a novel concept of HIL optimization of a hip exoskeleton. The optimization goal was to maintain the hip kinematics while providing optimal mechanical energy from the exoskeleton by modulating the admittance control. Policy iteration was used to optimize the switching time within the gait phase, at which a single parameter of the admittance controller was altered to provide assistance. The stiffness and equilibrium angle were considered as the two parameters for altering at the switching time, resulting in three possible modes of operation for the algorithm: (i) switching the equilibrium point, (ii) switching stiffness while equilibrium point is set at maximum extension and, (iii) maximum flexion. The optimization algorithm was found to converge for all three modes, with the equilibrium mode resulting in multiple solutions. Further analysis of power injected by the exoskeleton in the three modes showed that the first and third mode reduced human energetic exertion while the second mode increased human exertion. Implications of the results as well as the observed muscle activation patterns in response to assistance are discussed.

We are motivated by the real challenges presented in a human-robot system to develop new designs that are efficient at data level and with performance guarantees, such as stability and optimality at system level. Existing approximate/adaptive dynamic programming (ADP) results that consider system performance theoretically are not readily providing practically useful learning control algorithms for this problem, and reinforcement learning (RL) algorithms that address the issue of data efficiency usually do not have performance guarantees for the controlled system. This study fills these important voids by introducing innovative features to the policy iteration algorithm. We introduce flexible policy iteration (FPI), which can flexibly and organically integrate experience replay and supplemental values from prior experience into the RL controller. We show system-level performances, including convergence of the approximate value function, (sub)optimality of the solution, and stability of the system. We demonstrate the effectiveness of the FPI via realistic simulations of the human-robot system. It is noted that the problem we face in this study may be difficult to address by design methods based on classical control theory as it is nearly impossible to obtain a customized mathematical model of a human-robot system either online or offline. The results we have obtained also indicate the great potential of RL control to solving realistic and challenging problems with high-dimensional control inputs.

Personalizing medical devices such as lower limb wearable robots is challenging. While the initial feasibility of automating the process of knee prosthesis control parameter tuning has been demonstrated in a principled way, the next critical issue is to improve tuning efficiency and speed it up for the human user, in clinic settings, while maintaining human safety. We, therefore, propose a policy iteration with constraint embedded (PICE) method as an innovative solution to the problem under the framework of reinforcement learning. Central to PICE is the use of a projected Bellman equation with a constraint of assuring positive semidefiniteness of performance values during policy evaluation. Additionally, we developed both online and offline PICE implementations that provide additional flexibility for the designer to fully utilize measurement data, either from on-policy or off-policy, to further improve PICE tuning efficiency. Our human subject testing showed that the PICE provided effective policies with significantly reduced tuning time. For the first time, we also experimentally evaluated and demonstrated the robustness of the deployed policies by applying them to different tasks and users. Putting it together, our new way of problem solving has been effective as PICE has demonstrated its potential toward truly automating the process of control parameter tuning for robotic knee prosthesis users.

Robotic exoskeletons are exciting technologies for augmenting human mobility. However, designing such a device for seamless integration with the human user and to assist human movement still is a major challenge. This paper aims at developing a novel data-driven solution framework based on reinforcement learning (RL), without first modeling the human-robot dynamics, to provide optimal and adaptive personalized torque assistance for reducing human efforts during walking. Our automatic personalization solution framework includes the assistive torque profile with two control timing parameters (peak and offset timings), the least square policy iteration (LSPI) for learning the parameter tuning policy, and a cost function based on a transferred work ratio. The proposed controller was successfully validated on a healthy human subject to assist unilateral hip extension in walking. The results showed that the optimal and adaptive RL controller as a new approach was feasible for tuning assistive torque profile of the hip exoskeleton that coordinated with human actions and reduced activation level of hip extensor muscle in human.

The tuning process for a robotic prosthesis is a challenging and time-consuming task both for users and clinicians. An automatic tuning approach using reinforcement learning (RL) has been developed for a knee prosthesis to address the challenges of manual tuning methods. The algorithm tunes the optimal control parameters based on the provided knee joint profile that the prosthesis is expected to replicate during gait safely. This paper presents an intuitive interface designed for the prosthesis users and clinicians to choose the preferred knee joint profile during gait and use the autotuner to replicate in the prosthesis. The interface-based approach is validated by observing the ability of the tuning algorithm to successfully converge to various alternate knee profiles by testing on two able-bodied subjects walking with a robotic knee prosthesis. The algorithm was found to converge successfully in an average duration of 1.15 min for the first subject and 2.31 min for the second subject. Further, the subjects displayed different preferences for optimal profiles reinforcing the need to tune alternate profiles. The implications of the results in the tuning of robotic prosthetic devices are discussed.

Reliable environmental context prediction is critical for wearable robots (e.g., prostheses and exoskeletons) to assist terrain-adaptive locomotion. This article proposed a novel vision-based context prediction framework for lower limb prostheses to simultaneously predict human's environmental context for multiple forecast windows. By leveraging the Bayesian neural networks (BNNs), our framework can quantify the uncertainty caused by different factors (e.g., observation noise, and insufficient or biased training) and produce a calibrated predicted probability for online decision-making. We compared two wearable camera locations (a pair of glasses and a lower limb device), independently and conjointly. We utilized the calibrated predicted probability for online decision-making and fusion. We demonstrated how to interpret deep neural networks with uncertainty measures and how to improve the algorithms based on the uncertainty analysis. The inference time of our framework on a portable embedded system was less than 80 ms/frame. The results in this study may lead to novel context recognition strategies in reliable decision-making, efficient sensor fusion, and improved intelligent system design in various applications.

Robotic prostheses provide new opportunities to better restore lost functions than passive prostheses for trans-femoral amputees. But controlling a prosthesis device automatically for individual users in different task environments is an unsolved problem. Reinforcement learning (RL) is a naturally promising tool. For prosthesis control with a user in the loop, it is desirable that the controlled prosthesis can adapt to different task environments as quickly and smoothly as possible. However, most RL agents learn or relearn from scratch when the environment changes. To address this issue, we propose the knowledge-guided Q-learning (KG-QL) control method as a principled way for the problem. In this report, we collected and used data from two able-bodied (AB) subjects wearing a RL controlled robotic prosthetic limb walking on level ground. Our ultimate goal is to build an efficient RL controller with reduced time and data requirements and transfer knowledge from AB subjects to amputee subjects. Toward this goal, we demonstrate its feasibility by employing OpenSim, a well-established human locomotion simulator. Our results show the OpenSim simulated amputee subject improved control tuning performance over learning from scratch by utilizing knowledge transfer from AB subjects. Also in this paper, we will explore the possibility of information transfer from AB subjects to help tuning for the amputee subjects.

This paper aims to develop an optimal controller that can automatically provide personalized control of robotic knee prosthesis in order to best support gait of individual prosthesis wearers. We introduced a new reinforcement learning (RL) controller for this purpose based on the promising ability of RL controllers to solve optimal control problems through interactions with the environment without requiring an explicit system model. However, collecting data from a human-prosthesis system is expensive and thus the design of a RL controller has to take into account data and time efficiency. We therefore propose an offline policy iteration based reinforcement learning approach. Our solution is built on the finite state machine (FSM) impedance control framework, which is the most used prosthesis control method in commercial and prototypic robotic prosthesis. Under such a framework, we designed an approximate policy iteration algorithm to devise impedance parameter update rules for 12 prosthesis control parameters in order to meet individual users' needs. The goal of the reinforcement learning-based control was to reproduce near-normal knee kinematics during gait. We tested the RL controller obtained from offline learning in real time experiment involving the same able-bodied human subject wearing a robotic lower limb prosthesis. Our results showed that the RL control resulted in good convergent behavior in kinematic states, and the offline learning control policy successfully adjusted the prosthesis control parameters to produce near-normal knee kinematics in 10 updates of the impedance control parameters.