Autonomous on-ramp merging requires a safe and efficient decision and planning framework to navigate dynamic and complex traffic scenarios. Reinforcement Learning (RL) offers adaptability in such environments but struggles to ensure safety in unseen situations, whereas Model Predictive Control (MPC) can enforce safety constraints but relies on accurate models. To leverage the strengths of both, we propose a hierarchical Safety-Augmented RL and MPC integration approach, i.e., SARMI, for decision and planning in the on-ramp merging scenario. The high-level decision-making layer employs a discrete Soft Actor-Critic (SAC-Discrete) algorithm enhanced with the Augmented Lagrangian method to generate actions. MPC generates reference trajectories based on RL actions and feeds predicted states back to RL for reward and cost design, ensuring consistency between RL and MPC. Our key innovations include: i) incorporating a Gaussian-based risk field model into the cost design of constrained RL, which quantifies collision risk based on MPC-predicted states, enabling agents to make proactive decisions; ii) an Augmented Lagrangian SAC-Discrete method with barrier-like quadratic penalties to promote compliance with safety constraints and alleviate oscillations in dual gradient descent; iii) theoretical analysis proving the equivalence between the optimal solutions of the primal and dual problems in Augmented Lagrangian SAC-Discrete and iv) a dual safety mechanism combining action masking (to filter invalid actions) and action shielding (to replace unsafe actions in RL), enhancing safety during the exploration and execution stages, respectively. Experiments demonstrate the superiority of our method over the baseline SAC-Discrete, showing improved safety and efficiency.

Four-wheel independent steering (4WIS) vehicles have attracted increasing attention for their superior maneuverability. Human drivers typically choose to cross or drive over low-profile obstacles (e.g., plastic bags) to efficiently navigate through narrow spaces, while existing planners neglect obstacle attributes, leading to suboptimal efficiency or planning failures. To address this issue, we propose a novel multimodal trajectory planning framework that employs a neural network for scene perception, integrates 4WIS hybrid A* search to generate a warm start, and formulates an optimal control problem (OCP) for trajectory optimization. Specifically, a multimodal perception network fusing visual information and vehicle states is employed to capture semantic and contextual scene information, enabling the planner to adapt the strategy according to scene complexity (hard or easy planning task). For hard tasks, guided points are introduced to decompose complex tasks into local subtasks, improving search efficiency. The multiple steering modes of 4WIS vehicles—Ackermann, diagonal, and zero-turn—are also incorporated as kinematically feasible motion primitives. Moreover, a hierarchical obstacle handling strategy, which categorizes obstacles as “non-traversable”, “crossable”, and “drive-over”, is incorporated into the node expansion process, explicitly linking obstacle attributes to planning actions to enable efficient decisionmaking. Furthermore, to address dynamic obstacles with motion uncertainty, we introduce a probabilistic risk field model, constructing risk-aware driving corridors that serve as linear collision constraints in the OCP. Experimental results demonstrate the proposed framework’s effectiveness in generating safe, efficient, and smooth trajectories for 4WIS vehicles, especially in constrained environments.

Robotic ultrasound (US) systems represent an emerging frontier in medical imaging. A fundamental component of these systems is the rigid body transformation between the robot flange and the attached US probe, which enables mapping of visual data from image space to the robot's reference frame. Traditionally, calibrating this transformation has been a tedious process, complicated by equipment demands and operational constraints arising from the probe's narrow field of view. This work presents a novel calibration strategy based on conic sections geometry, which offers several key simplifications over existing approaches: 1). It requires no external equipment beyond a single cone phantom; 2). It operates with a small input size and imposes no strict alignment or motion constraints on the US scan plane during calibration; and 3). It employs a straightforward pattern analysis pipeline to process images acquired from phantom scans. Experimental validation results show that the proposed method achieves accuracy comparable to existing state-of-the-art approaches while delivering superior precision, thereby demonstrating enhanced calibration reproducibility enabled by its streamlined workflow. These advantages make this method particularly suitable for application in clinical scenarios that require frequent and efficient calibration.

In modern healthcare, the demand for autonomous robotic assistants has grown significantly, particularly in the operating room, where surgical tasks require precision and reliability. Robotic scrub nurses have emerged as a promising solution to improve efficiency and reduce human error during surgery. However, challenges remain in terms of accurately grasping and handing over surgical instruments, especially when dealing with complex objects in dynamic environments. In this work, we introduce RoboNurse-VLA, a novel robotic scrub nurse system based on a Vision-Language-Action (VLA) model. RoboNurse-VLA integrates Segment Anything Model 2 (SAM 2) and Llama 2, leveraging an LLM head to enhance reasoning capabilities. By combining SAM 2’s mask generation with Llama 2’s advanced reasoning, RoboNurse-VLA can accurately interpret task requirements, identify optimal grasping points, and determine appropriate handover poses. Designed for real-time operation, RoboNurse-VLA enables precise grasping and seamless handover of surgical instruments based on voice commands from the surgeon. Utilizing state-of-the-art vision and language models, it effectively addresses challenges related to object detection, pose optimization, and handling difficult-to-grasp instruments. Extensive evaluations demonstrate that RoboNurse-VLA outperforms existing models, achieving high success rates in surgical instrument handovers, even for previously unseen tools and complex objects. This work represents a significant advancement in autonomous surgical assistance, highlighting the potential of VLA models for real-world medical applications. More details can be found at https:// robonurse-vla.github.io.

The remote center of motion (RCM) constraint is a vital requirement in the design of robotic systems for transrectal ultrasound (TRUS) probe-scanning. This paper presents the design and development of a novel RCM-constrained manipulator specifically tailored for TRUS probe-scanning applications. The proposed system features a six-degree-of-freedom (6-DoF) parallel-serial hybrid mechanism that enables the TRUS probe to perform pivot and spin rotations while maintaining the RCM constraint. Subsequently, the kinematic model incorporating the RCM constraint is derived. Additionally, a geometry-aware path planning method is then introduced, considering variations in the desired rotation targets. This method parameterizes distance metrics on SO(3) (a Lie group) using coordinate-free Riemannian geometry, enabling the dynamic optimization of rotation orders to minimize the calculated Riemannian metrics. Furthermore, a smooth rotational trajectory generation method is proposed, constructing rotation curves between the ordered matrices on SO(3) while minimizing angular acceleration. Both simulations and experimental results validate the effectiveness and practicality of the proposed manipulator and its path planning method.

Despite advantages from neurosurgical systems, achieving intuitive and safe collaboration with robot during the coarse positioning of instrument insertion end effector (IIEE) remains a critical issue. In this paper, we propose a novel non-contact hand-guided method for such advancement based on magnetic sensing. First, a wearable magnet band and a magnetic sensor are designed, based on which the magnetic localization is achieved for surgeon’s hand location detection. Second, a quadratic programming-based control is implemented, to guarantee the pose-based servo performance, higher rotational manipulability for IIEE fine alignment, and joint position&velocity limits avoidance. For evaluation, two experiments are designated and conducted. Results show that the magnetic localization algorithm can achieve < 4.7 mm and 2.6° errors in a dynamic path tracking test, which can provide an accurate magnet location for hand guidance. Moveover, workflow of the proposed solution in a brain biopsy scenario demonstrates its enhancement of IIEE rotational manipulability (11.6% increase at final configuration), and safety improvement of collision avoidance when other surgeon approaches for cannula delivery. This research contributes to enhanced intuitiveness and safety for surgeon-robot collaborative coarse positioning of IIEE in neurosurgery.

Proactive caching scheme proactively caches data before the request comes, which is essential for optimizing of the resource utilization and ensuring the stability of cached contents. However, current proactive caching methods suffer from high computational complexity and often fall into suboptimal solutions. This study proposes a hierarchical reinforcement learning (HRL)-based proactive caching scheme to determine whether to update the caching queue and how to update the caching queue. Specifically, we build a bi-level optimization model to maximize the content cache hit ratio while minimizing the content cache cost. We formulate the upper-level and lower-level optimization problems as Markov decision process (MDP) models and employ double deep Q-learning methods to solve the optimal caching strategy. In particular, the content history request features and the user preference generated by the collaborative filtering-based recommendation system are used as the state inputs of caching decisions. In addition, we design a two-stage curriculum learning paradigm for training the agents from easy tasks to hard tasks. At the first stage, a greedy decision tree with a posterior knowledge is designed to provide the upper-level decisions for training the lower-level agent in the easy task. Then, we train all agents together in the hard task while freezing the trained lower-level agent. We conduct both the simulation and on-road experiments to validate the effectiveness of our method, and results indicate that our method outperforms the baseline methods by 2.2%–37.2% in terms of cache hit ratio and 75.4%–77.6% in terms of cache cost.

In minimally invasive surgery (MIS), the field of view (FOV) control is crucial. Autonomous endoscope robots have been developed to facilitate MIS procedures by enabling autonomous surgical target tracking, thus reducing the workload on surgeons. However, existing visual servoing-based target tracking methods for autonomous endoscopes often overlook the insecurity stemming from restricted workspace conditions. Instances, such as collisions between the endoscope robot’s tip and the patient’s chest or abdominal wall pose risks to patient tissue, while extensive motion of the endoscope shaft may damage incision port tissue. Addressing these security concerns, this article proposes a novel approach called virtual fixture-based restricted workspace constraint (RWSC) to reconstruct the endoscope robot’s movement range. A quadratic programming (QP) optimization framework is employed to govern the robot’s motion, ensuring autonomous target tracking while adhering to RWSCs. To solve the QP problem, we propose an adaptive zeroing neural network (ZNN) featuring a newly designed activation function (AF). This AF enhances the ZNN with predefined-time convergence and noise rejection capabilities, making it especially suitable for time-sensitive and noise-prone surgical applications. Theoretical analysis and experimental results demonstrate that our adaptive ZNN achieves shorter convergence times than existing neural dynamic-based QP solvers. Physical validations show the efficacy of the proposed RWSCs in limiting the workspace of the endoscope robot, while the FOV control strategy enables autonomous target tracking of flexible endoscopes under diverse constraints and objectives.

This work presents a novel method for point registration in 3D space. The proposed algorithm utilizes transformation-invariant geometry information to estimate the pose of objects based on correspondences between points in two sets. Conventional methods use geometry descriptors to find these correspondences, which can result in a large number of outliers. Most existing algorithms are error-prone when outliers are present. Instead of formulating point registration as a non-convex optimization problem, we propose an intuitive method that filters out spurious correspondences. This is achieved by evaluating three different geometry-based transformation-invariant descriptors for outlier removal. We construct fully connected graphs with the proposed descriptors on correspondences, and convert the outlier removal problem into a subgraph isomorphism problem that is solved using a binary clustering approach. The resulting inlier clustering is used to estimate the transformation between the two point sets. The effectiveness of the proposed approach is evaluated on standard 3D data and the 3DMatch scan matching dataset, and compared against existing state-of-the-art methods. Results show that our method effectively reduces outliers and performs similarly to these methods.

In this paper, an integrated framework consisting of the sensing, navigation and control is proposed for an autonomous mobile manipulator driven by series elastic actuators (SEAs) to preform mobile manipulation tasks in unknown environments. First, ORB-SLAM2 technique is combined into the environment sensing by extracting the ORB features, automatic initialization, repositioning and loop detection for real-time posture estimation. Then, the navigation function is designed for generating collision-free trajectory in an environment with obstacles. To realize kinematic and dynamic control of the mobile manipulator with the developed SEA joints, the whole body dynamics is considered and described. And to handle dynamic uncertainties and the SEA inherent saturation limits, a novel adaptive neural network control considering the whole body dynamics is proposed. Without knowing the exact parameters of the whole body model, the designed controller merely requires the position and velocity of the actuators and links, which can make the tracking errors converge to zero and keep all signals uniformly bounded in the closed-loop system. The performance and efficiency of the proposed method are verified by extensive experiments. Note to Practitioners—This paper is motivated by issues of manipulation control of autonomous unmanned system. Traditional manipulation frameworks focus either on sensing or control by assuming that the environment is known, which would result in lacking of autonomy for a specified task. Since mobile manipulation tasks often consist of nonholonomic and holonomic constraints for wheeled mobile manipulators with differential steering. In addition, most current works for whole body control are based on the condition that robot dynamic parameters are known beforehand. Therefore, it is necessary to establish an enhanced framework to simultaneously deal with these problems. In this paper, an integrated navigation and control framework is proposed. To make the mobile manipulator work in the unknown environment, task sensing and whole body control for mobile manipulators are also developed. The framework is partitioned into the task sensing, navigation and control, where the mobile manipulator can fulfill the mobile manipulation tasks in the unstructured environments.

Robot learning through kinesthetic teaching is a promising way of cloning human behaviors, but it has its limits in the performance of complex tasks with small amounts of data, due to compounding errors. In order to improve the robustness and adaptability of imitation learning, a hierarchical learning strategy is proposed: low-level learning comprises only behavioral cloning with supervised learning, and high-level learning constitutes policy improvement. First, the Gaussian mixture model (GMM)-based dynamical system is formulated to encode a motion from the demonstration. We then derive the sufficient conditions of the GMM parameters that guarantee the global stability of the dynamical system from any initial state, using the Lyapunov stability theorem. Generally, imitation learning should reason about the motion well into the future for a wide range of tasks; it is significant to improve the adaptability of the learning method by policy improvement. Finally, a method based on exponential natural evolution strategies is proposed to optimize the parameters of the dynamical system associated with the stiffness of variable impedance control, in which the exploration noise is subject to stability conditions of the dynamical system in the exploration space, thus guaranteeing the global stability. Empirical evaluations are conducted on manipulators for different scenarios, including motion planning with obstacle avoidance and stiffness learning.

This article presents a framework to enable a human and robot to perform collaborative tasks safely and efficiently. It consists of three functions. First, human motion is predicted by utilizing Gaussian mixture regression. Second, the motion planning of an online robot is performed such that the robot can appropriately react to a human coworker while executing a task. In our proposed framework, the predicted human motion is transferred to a virtual force acting on a robot's end effector. Its initial trajectory is modified so as to avoid any collisions with the human. To obtain a smooth, collision-free, and energy-minimized trajectory, a constrained optimization problem is formulated. A neural dynamics optimization algorithm is then adopted to solve it. Third, an adaptive fuzzy controller is proposed to track the robot's desired trajectory with uncertain dynamics parameters. We provide the rigorous proof of stability for the proposed methods. The physical experiments are conducted to demonstrate the effectiveness of the proposed collaborative strategy.

Robots for high-speed manipulation require to produce motions beyond the performance limitations set by the traditional approaches. Recent results integrate the properties associated with dynamic coupling driving and structural mechanics to compute optimal smooth arm motions; however, when accelerating the convergence speed of potential solutions, those approaches cannot avoid premature convergence. In this article, we propose an autonomous motion planning method at the torque level for a class of robots considering multiple conflicting performance metrics. Specifically, we focus on the hyper dynamic manipulation of a golf swing robot using a knee-guided multiobjective optimization algorithm. Compared with traditional planning methods in position or velocity level, it can study motor performance limitations with strong nonlinear dynamic coupling beyond the motion limits designed by the manufacturers. First, the robot’s joint torque is approximated by the B-spline method using the solution at each iteration. Then, we transform the motion planning problem into a multiobjective optimization problem with soft constraints of torque limits and hard constraints of joint stops, and develop a knee-guided evolutionary algorithm to find the optimization solution with the quality tradeoffs between the scale of parameters and metrics. Finally, we conduct the simulation to demonstrate the dynamics performance of the golf swing robot. The results indicate that our approach can generate superior dynamics performance beyond limits with low energy consumption and high precision.

Recently, the human-like behavior on the anthropomorphic robot manipulator is increasingly accomplished by the kinematic model establishing the relationship of an anthropomorphic manipulator and human arm motions. Notably, the growth and broad availability of advanced data science techniques facilitate the imitation learning process in anthropomorphic robotics. However, the enormous dataset causes the labeling and prediction burden. In this article, the swivel motion reconstruction approach was applied to imitate human-like behavior using the kinematic mapping in robot redundancy. For the sake of efficient computing, a novel incremental learning framework that combines an incremental learning approach with a deep convolutional neural network is proposed for fast and efficient learning. The algorithm exploits a novel approach to detect changes from human motion data streaming and then evolve its hierarchical representation of features. The incremental learning process can fine-tune the deep network only when model drifts detection mechanisms are triggered. Finally, we experimentally demonstrated this neural network's learning procedure and translated the trained human-like model to manage the redundancy optimization control of an anthropomorphic robot manipulator (LWR4+, KUKA, Germany). This approach can hold the anthropomorphic kinematic structure-based redundant robots. The experimental results showed that our architecture could not only enhance the regression accuracy but also significantly reduce the processing time of learning human motion data.

Accurate path tracking and stability are the main challenges of lateral motion control in mobile robots, especially under the situation with complex road conditions. The interaction force between robots and the external environment may cause interference, which should be considered to guarantee its path tracking performance in dynamic and uncertain environments. In this article, a flexible lateral control scheme is considered for the developed wheel-legged robot, which consists of a cubature Kalman algorithm to evaluate the centroid slip angle and the yaw rate. Furthermore, a fuzzy compensation and preview angle-enhanced sliding model controller to improve the tracking accuracy and robustness. Finally, some simulations and experimental demonstrations using the four-wheel-legged robot (BIT-NAZA) are carried out to illustrate the effectiveness and robustness, and the proposed method has achieved satisfactory results in high-precision trajectory tracking and stability control of the mobile robot.

Under-actuated bionic hands have achieved tremendous popularity in many fields because of their advantages of lightweight, budget-friendly, satisfactory flexibility, and adaptability. Except for the bionic mechanical design, various anthropomorphic control strategies have been proposed and investigated in the last decades. However, due to its under-actuated characteristic, there are still many challenges for anthropomorphic control of all the degrees of freedom (DOFs) using less input. It is challenging to map the human hand kinematic synergies on robotic hands, particularly for a dynamic environment. Therefore, it is worth studying how to control the under-actuated bionic hand effectively in a dynamic environment. In this paper, an anthropomorphic control method is proposed using sensor fusion of hand kinematic inputs to control the under-actuated bionic hand. In order to map the kinematics of human fingers to the bionic hand, a novel finger bending angle is defined to represent the posture of human fingers. Multiple Leap Motion Controllers (LMC) are fused to estimate the stable and accurate finger bending angles to avoid the occlusion problem. Finally, experiments with real-time control of the under-actuated bionic hand are implemented to demonstrate the proposed approach’s effectiveness.

In recent decades, there has been an increasing interest in the use of robotic powered exoskeletons to assist patients with movement disorders in rehabilitation and daily life. Providing assistive torque that compensates for the user's remaining muscle contributions is a growing and challenging field within exoskeleton control. In this article, ankle joint torques were estimated using electromyography (EMG)-driven neuromusculoskeletal (NMS) model and an artificial neural network (ANN) model in seven movement tasks, including fast walking, slow walking, self-selected speed walking, and isokinetic dorsi/plantar flexion at 60°/s and 90°/s. In each method, EMG signals and ankle joint angles were used as input, the models were trained with data from 3-D motion analysis, and ankle joint torques were predicted. Six cases using different motion trials as calibration (for the NMS model)/training (for the ANN) were devised, and the agreement between the predicted and measured ankle joint torques was computed. We found that the NMS model could overall better predict ankle joint torques from EMG and angle data than the ANN model with some exceptions; the ANN predicted ankle joint torques with better agreement when trained with data from the same movement. The NMS model predicted ankle joint torque best when calibrated with trials during which EMG reached maximum levels, whereas the ANN predicted well when trained with many trials and types of movements. In addition, the ANN prediction may become less reliable when predicting unseen movements. Detailed comparative studies of methods to predict ankle joint torque are crucial for determining strategies for exoskeleton control.

The complexity of surgical operation can be released significantly if surgical robots can learn the manipulation skills by imitation from complex tasks demonstrations such as puncture, suturing, and knotting, etc.. This paper proposes a reinforcement learning algorithm based manipulation skill transferring technique for robot-assisted Minimally Invasive Surgery by Teaching by Demonstration. It employed Gaussian mixture model and Gaussian mixture Regression based dynamic movement primitive to model the high-dimensional human-like manipulation skill after multiple demonstrations. Furthermore, this approach fascinates the learning and trial phase performed offline, which reduces the risks and cost for the practical surgical operation. Finally, it is demonstrated by transferring manipulation skills for reaching and puncture using a KUKA LWR4+ robot in a lab setup environment. The results show the effectiveness of the proposed approach for modelling and learning of human manipulation skill.

In this paper, an Internet of Things-based human-robot collaborative control scheme is developed in Robot-assisted Minimally Invasive Surgery scenario. A hierarchical operational space formulation is designed to exploit the redundancies of the 7-DoFs redundant manipulator to handle multiple operational tasks based on their priority levels, such as guaranteeing a remote center of motion constraint and avoiding collision with a swivel motion without influencing the undergoing surgical operation. Furthermore, the concept of the Internet of Robotic Things is exploited to facilitate the best action of the robot in human-robot interaction. Instead of utilizing compliant swivel motion, HTC VIVE PRO controllers, used as the Internet of Things technology, is adopted to detect the collision. A virtual force is applied to the robot elbow, enabling a smooth swivel motion for human-robot interaction. The effectiveness of the proposed strategy is validated using experiments performed on a patient phantom in a lab setup environment, with a KUKA LWR4+ slave robot and a SIGMA 7 master manipulator. By comparison with previous works, the results show improved performances in terms of the accuracy of the RCM constraint and surgical tip.

In this paper, an improved motion planning scheme is proposed for surgical robot control with multiple active constraints, including joint constraints, joint velocity constraints and remote center of motion constraints. It introduces an improved recurrent neural network (RNN) to optimize the online motion planning respect to multiple constraints. The demonstrated surgical operation trajectory is derived using teaching by demonstration. An improved motion planning scheme using the novel recurrent neural network is then designed to achieve the accurate task tracking under the multiple constraints. The general quadratic performance index is adopted to represent the constraints. Finally, the effectiveness of the proposed algorithm is demonstrated using KUKA LWR4+ robot in a lab setup environment.

For the time varying optimization problem, the tracking error cannot converge to zero at the finite time because of the optimal solution changing over time. This paper proposes a novel varying parameter recurrent neural network (VPRNN) based hierarchical optimization of a 7-DoF surgical manipulator for Robot-Assisted Minimally Invasive Surgery (RAMIS), which guarantees task tracking, Remote Center of Motion (RCM) and manipulability index optimization. A theoretically grounded hierarchical optimization framework based is introduced to control multiple tasks based on their priority. Finally, the effectiveness of the proposed control strategy is demonstrated with both simulation and experimental results. The results show that the proposed VPRNN-based method can optimal three tasks at the same time and have better performance than previous work.

A series of previous work has found that the environmental constraint (EC), which is the natural result of the contact of the robot and the interacting objects, is immensely helpful for the realization of high-precision robotic tasks. However, due to the existence of multifarious errors, such as mechanical error, modeling error and sensing error, there would be discrepancy between the actual constraints and the ideal models. In such case, it is hard to realize manipulation with EC-based strategies. Inspired by human, a preliminary framework which aims at the integration of the coarse sensing information and the environmental constraints is proposed for robotic manipulation. By mapping the sensing information into the new space, where the environmental constraint can be formally described, the region that integrates the sensing information and the environmental constraint is constructed and the conditions to achieve high-precision manipulation are derived. Based on the conditions, a motion planning strategy is proposed to achieve the required task. The effectiveness of this strategy is verified by case studies.

Learning by imitation is a powerful way that can reduce the complexly in searching space. It could help the mobile robot to acquire new skills from interaction with a human-being in natural way. In this paper, the dynamic movement primitives (DMPs) is utilized to imitate the trajectory from human walking. DMPs is a modified formulation of virtual spring-dampers (VSD) system that enjoys better fitting performance in learning. Further, while dealing with the trajectory tracking problem of mobile robots, a novel nonlinear model predictive control (MPC) approach is proposed for motion control. The nonlinear MPC scheme applies a new neural network named Varying-parameter Lagrangian Neural Network (VP-LNN) to solve a Quadratic Programming (QP) problem by iterating over a finite receding horizon. The new network of VP-LNN can converge to the global optimal solution. Thus, a new human-robot interaction (HRI) scheme for mobile robot is proposed, which can reduce the complexity in motion planning in various applications.

In this paper, a sensory-motor fusion-based manipulation and grasping control strategy has been developed for a robotic hand-eye system. The proposed hierarchical control architecture has three modules: 1) vision servoing; 2) surface electromyography (sEMG)-based movement recognition; and 3) hybrid force and motion optimization for manipulation and grasping. A stereo camera is used to obtain the 3-D point cloud of a target object and provides the desired operational position. The AdaBoost-based motion recognition is employed to discriminate different movements based on sEMG of human upper limbs. The operational space motion planning for bionic arm and force planning for multifingered robotic hand can be both transformed as a convex optimization problem with various constraints. A neural dynamics optimization solution is proposed and implemented online. The proposed formulation can achieve a substantial reduction of computational load. The actual implementation includes a bionic arm with dextrous hand, high-speed active vision, and an EMG sensors. A series of manipulation tasks consisting of tracking/recogniting/grasping of an object are implemented, and experiment results exhibit the responsiveness and flexibility of the proposed sensory motion fusion approach.