We develop a task-independent predictive framework that estimates hip, knee and ankle future behavior from sonomyographic sensing of quadriceps musculature. Two regression models, support vector regression and Gaussian process regression, were trained and tested such that no ambulation mode recognition was required. Sonomyography features of the anterior thigh musculature were extracted during the swing phase of level, incline and stair ambulation tasks as inputs to the two models for continuous prediction of the future stance phase hip, knee and ankle moments. Next, sonomyography features of the anterior thigh musculature were extracted during the stance phase and used to predict the following swing phase hip, knee and ankle angles. Leave-one-stride-out cross-validation is used to evaluate this continuous prediction framework. Additionally, initial, peak and terminal joint moment and angle parameters are extracted from trajectories and evaluated. Both regression models were able to accurately predict continuous future joint moments and angles, as well as initial, peak and terminal value parameters of future joint moments and angles. However, the support vector regression model required relatively lower computational cost. Thus, we recommend the support vector regression model as an optimal model for forward prediction of joint mechanics from sonomyographic sensing during ambulation.

Natural control of limb motion is continuous and progressively adaptive to individual intent. While intuitive interfaces have the potential to rely on the neuromuscular input by the user for continuous adaptation, continuous volitional control of assistive devices that can generalize across various tasks has not been addressed. In this study, we propose a method to use spatiotemporal ultrasound features of the rectus femoris and vastus intermedius muscles of able-bodied individuals for task-invariant learning of continuous knee kinematics during steady-state and transient ambulation. The task-invariant learning paradigm was statistically evaluated against a task-specific paradigm for the steady-state (1) level-walk, (2) incline, (3) decline, (4) stair ascent, and (5) stair descent ambulation tasks. The transitions between steady-state stair ambulation and level-ground walking were also investigated. It was observed that the continuous knee kinematics can be learned using a task-invariant learning paradigm with statistically comparable accuracy to a task-specific paradigm. Statistical analysis further revealed that incorporating the temporal ultrasound features significantly improves the accuracy of continuous estimations (p < 0.05). The average root mean square errors (RMSEs) of knee angle and angular velocity estimation were 7.06° and 53.1°/sec, respectively, for the task-invariant learning compared to 6.00° and 51.8°/sec for the task-specific models. High accuracy of continuous task-invariant paradigms overcome the barrier of task-specific control schemes and motivate the implementation of direct volitional control of lower-limb assistive devices using ultrasound sensing, which may eventually enhance the intuitiveness and functionality of these devices towards a "free form" control approach.

An important problem in designing human-robot systems is the integration of human intent and performance in the robotic control loop, especially during complex tasks. Bimanual coordination is a complex human behavior that is critical in many fine motor tasks, including robot-assisted surgery. To fully leverage the capabilities of the robot as an intelligent and assistive agent, online recognition of bimanual coordination could be important. Robotic assistance for a suturing task, for example, will be fundamentally different during phases when the suture is wrapped around the instrument (i.e., making a c-loop), than when the ends of the suture are pulled apart. In this study, we develop an online recognition method of bimanual coordination modes (i.e., the directions and symmetries of right and left hand movements) using geometric descriptors of hand motion. We (1) develop this framework based on ideal trajectories obtained during virtual 2D bimanual path following tasks performed by human subjects operating Geomagic Touch haptic devices, (2) test the offline recognition accuracy of bimanual direction and symmetry from human subject movement trials, and (3) evalaute how the framework can be used to characterize 3D trajectories of the da Vinci Surgical System’s surgeon-side manipulators during bimanual surgical training tasks. In the human subject trials, our geometric bimanual movement classification accuracy was 92.3% for movement direction (i.e., hands moving together, parallel, or away) and 86.0% for symmetry (e.g., mirror or point symmetry). We also show that this approach can be used for online classification of different bimanual coordination modes during needle transfer, making a C loop, and suture pulling gestures on the da Vinci system, with results matching the expected modes. Finally, we discuss how these online estimates are sensitive to task environment factors and surgeon expertise, and thus inspire future work that could leverage adaptive control strategies to enhance user skill during robot-assisted surgery.

Bimanual coordination is critical in many robotic and haptic systems, such as surgical robots and rehabilitation robots. While these systems often incorporate two robotic manipulators for each limb, there may be a missed opportunity to leverage overarching models of human bimanual coordination to improve the way in which the robotic manipulators are controlled and respond to the dynamic human operator. In this paper, we study the influences of several bimanual motion factors (e.g., symmetry and direction) on kinematic human joint-space features and performance outcome task-space features in a user study with eleven subjects and two haptic devices. Additionally, we evaluated the ability to use joint-space features to classify types of bimanual movement, showing the potential for a robotic system to predict how users coordinate their limbs. Three classifiers: (1) likelihood ratio, (2) k-nearest neighbor, and (3) support vector machine, were evaluated for classification accuracy in regards to the factor of number of targets. Likelihood ratio resulted in an accuracy of 79.6% with the majority of correct predictions occurring immediately at the start of movement. The task-space performance results reveal that despite the relative direction of both hands, reaching two targets results in lower performance than a single target, and symmetry alone does not contribute to performance disparity. Also, dimensionless integrated absolute jerk (DIAJ) is an indicator of superior performance for this particular task. Furthermore, these results align with current bimanual coordination theory by showing manual performance disparities are a consequence of task constraints and conceptualization.

OBJECTIVE: Powered assistive devices need improved control intuitiveness to enhance their clinical adoption. Therefore, the intent of individuals should be identified and the device movement should adhere to it. Skeletal muscles contract synergistically to produce defined lower limb movements, so unique contraction patterns in lower extremity musculature may provide a means of device joint control. Ultrasound (US) imaging enables direct measurement of the local deformation of muscle segments. Hence, the objective of this study was to assess the feasibility of using US to estimate human lower limb movements. METHODS: A novel algorithm was developed to calculate US features of the rectus femoris muscle during a non-weight-bearing knee flexion/extension experiment by nine able-bodied subjects. Five US features of the skeletal muscle tissue were studied, namely thickness, angle between aponeuroses, pennation angle, fascicle length, and echogenicity. A multiscale ridge filter was utilized to extract the structures in the image and a random sample consensus (RANSAC) model was used to segment muscle aponeuroses and fascicles. A localization scheme further guided RANSAC to enable tracking in a US image sequence. Gaussian process regression models were trained using segmented features to estimate both knee joint angle and angular velocity. RESULTS: The proposed segmentation-estimation approach could estimate knee joint angle and angular velocity with an average root mean square error value of 7.45° and 0.262 rad/s, respectively. The average processing rate was 3-6 frames/s that is promising toward real-time implementation. CONCLUSION: Experimental results demonstrate the feasibility of using US to estimate human lower extremity motion. The ability of the algorithm to work in real time may enable the use of US as a neural interface for lower limb applications. SIGNIFICANCE: Intuitive intent recognition of human lower extremity movements using wearable US imaging may enable volitional assistive device control and enhance locomotor outcomes for those with mobility impairments.