Grasping assistance is essential for restoring autonomy in individuals with motor impairments, particularly in unstructured environments where object categories and user intentions are diverse and unpredictable. We present OVGrasp , a hierarchical control framework for grasp assistance that integrates RGB-D vision, open vocabulary prompts, and voice commands to enable robust multimodal interaction. To enhance generalisation in open environments, OVGrasp incorporates a vision language foundation model with an open vocabulary mechanism, which enables zero-shot detection of previously unseen objects without retraining. A multimodal decision maker further fuses spatial and linguistic cues to infer user intent, such as grasp or release, in situations involving multiple objects. We deploy the complete framework on a custom egocentric view wearable exoskeleton and conduct systematic evaluations on fifteen objects across three grasp types. Experimental results with ten participants show that OVGrasp achieves a grasping ability score (GAS) of 87.00%, surpassing existing baselines and providing improved kinematic alignment with natural hand movement. • OVGrasp: a hierarchical framework for grasp assistance. • Open-vocabulary detection enables zero-shot generalisation to unseen objects. • Multimodal decision-making fuses vision, depth, and speech for intent detection. • Integrated in a soft hand exoskeleton with egocentric RGB-D sensing. • Achieves superior grasping ability score and improved joint kinematics in tests.

Recent advancements in bionic prosthetic technology offer transformative opportunities to restore mobility and functionality for individuals with missing limbs. Users of bionic limbs, or bionic humans, learn to seamlessly integrate prosthetic extensions into their motor repertoire, regaining critical motor abilities. The remarkable movement generalization and environmental adaptability demonstrated by these individuals highlight motor intelligence capabilities unmatched by current artificial intelligence systems. Addressing these limitations, MyoChallenge '24 at NeurIPS 2024 established a benchmark for human-robot coordination with an emphasis on joint control of both biological and mechanical limbs. The competition featured two distinct tracks: a manipulation task utilizing the myoMPL model, integrating a virtual biological arm and the Modular Prosthetic Limb (MPL) for a passover task; and a locomotion task using the novel myoOSL model, combining a bilateral virtual biological leg with a trans-femoral amputation and the Open Source Leg (OSL) to navigate varied terrains. Marking the third iteration of the MyoChallenge, the event attracted over 50 teams with more than 290 submissions all around the globe, with diverse participants ranging from independent researchers to high school students. The competition facilitated the development of several state-of-the-art control algorithms for bionic musculoskeletal systems, leveraging techniques such as imitation learning, muscle synergy, and model-based reinforcement learning that significantly surpassed our proposed baseline performance by a factor of 10. By providing the open-source simulation framework of MyoSuite, standardized tasks, and physiologically realistic models, MyoChallenge serves as a reproducible testbed and benchmark for bridging ML and biomechanics. The competition website is featured here: https://sites.google.com/view/myosuite/myochallenge/myochallenge-2024.

Human beings adapt their motor patterns in response to their surroundings, utilizing sensory modalities such as visual inputs. This context-informed adaptive motor behavior has increased interest in integrating computer vision algorithms into robotic assistive technologies, marking a shift towardscontext aware control. However, such integration has rarely been achieved so far, with current methods mostly relying on data-driven approaches. In this study, we introduce a novel control framework for a soft hip exosuit, employing instead a physics-informed computer vision method grounded on geometric modeling of the captured scene for assistance tuning during stairs and level walking. This approach promises to provide a viable solution that is more computationally efficient and does not depend on training examples. Evaluating the controller with six subjects on a path comprising level walking and stairs, we achieved an overall detection accuracy of$93.0\pm 1.1\%$. Computer vision-based assistance provided significantly greater metabolic benefits compared to non-vision-based assistance, with larger energy reductions relative to being unassisted during stair ascent ($-18.9 \pm 4.1\%$vs.$-5.2 \pm 4.1\%$) and descent ($-10.1 \pm 3.6\%$vs.$-4.7 \pm 4.8\%$). Such a result is a consequence of the adaptive nature of the device, enabled by the context aware controller, that allowed for more effective walking support: i.e. the assistive torque showed a significant increase while ascending stairs ($+33.9\pm 8.8\%$) and decrease while descending stairs ($-17.4\pm 6.0\%$) compared to a condition without assistance modulation enabled by vision. These results highlight the potential of the approach, promoting effective real-time embedded applications in assistive robotics.