Self-balancing lower limb exoskeletons (SBLLE) hold the promise of helping individuals with diverse mobility impairments regain their ability to walk. Equipping exoskeletons with stable turning capabilities is crucial for their real-world application. This paper presents an effective control framework that enables stable turning motions in SBLLE without relying on external assistive devices. A turn-specific gait planner has been designed to generate stable reference gaits. Moreover, an adaptive variable impedance controller (AVIC) is introduced, which adjusts the center of mass (CoM) motion in real time to maintain balance during walking, even when carrying different patients. The proposed approach is validated through simulations, comparative studies, and multi-stage human-subject experiments with ten participants, ranging from healthy users to individuals with severe lower-limb motor impairment. Results consistently demonstrate stable turning performance, with the Zero Moment Point (ZMP) remaining within the support polygon during the entire walking process.

Expressing and controlling fine-grained spatial attributes of objects in large-scale models presents significant challenges, as these spatial attributes are often difficult to describe textually and exhaustive enumeration is impractical. This hinders effective alignment with user preferences regarding spatial attribute-object relationships in fine-grained synthesis tasks. To tackle this problem, we propose AttrObjDiff, a novel framework built on the pre-trained Stable Diffusion model to integrate spatial attribute maps. Firstly, AttrObjDiff constrains the denoising step using trainable cross-attention fusion modules, attribute-enhancing cross-attention and LoRAs. The fusion modules take layout features extracted by a frozen ControlNet and corresponding fine-grained attribute maps as inputs to generate joint constraint features of spatial attribute-object relationships. We leverage attribute-enhancing cross-attention within the U-Net to further refine these spatial attributes. Finally, LoRAs are employed to align with these joint constraint features of finegrained relationships. Secondly, AttrObjDiff enhances the reverse process with lightweight noise reranking models to improve spatial object-attribute alignment. The reranking models select semantic noises related to fine-grained relationships, improving synthesis quality without significantly increasing computational costs. Experimental results demonstrate that our method can generate high-quality images guided by fine-grained spatial object-attribute relationships, improving synthesis controllability and semantic consistency.

This study proposes a novel Transformer-based framework for identifying terrain transition states and recognizing steady-state terrains using data from inertial measurement units. Compared to traditional time series classification methods for transition states, our approach reframes the problem as a time series fitting and terrain change-point detection task, capturing the dynamic nature of human locomotion across varying terrains. Outdoor experiments demonstrate the model’s superior performance in both steady-state and transition detection, with enhanced interpretability. Specifically, steady-state identification achieves accuracies of 99.63% on normal terrain and 98.06% on complex terrain. Compared to traditional convolutional neural network-based approaches, our method improves terrain classification accuracy by 12.30% –37.67% under normal conditions and 12.34% –39.90% under complex conditions. Moreover, the normalized root mean square error for transition curve fitting is significantly reduced to 0.016 and 0.032 for normal and complex terrains, outperforming other models.

Self-balancing lower limb rehabilitation exoskeletons (SLLREs) allow individuals with lower limb dysfunction to walk without the use of crutches. Stable and human-like walking motions are crucial for SLLREs because achieving a close imitation of healthy human walking is a key goal in rehabilitation therapy. Existing SLLREs can realize stable walking but lack human-like features such as knee-stretched, heel-strike and toe-off. This paper designs a walking motion generator based on hierarchical optimization to generate a human-like walking motion with variable hip height, heelstrike, toe-off, and knee-stretched features. This generator consists of a knee-stretched optimizer and a stabilizing filter. Specifically, the knee-stretched optimizer realizes the stretched knee feature by optimizing the hip trajectory with varying heights. And the stabilizing filter realizes stable walking by optimizing the hip trajectory in the sagittal plane direction. To validate the effectiveness of the proposed human-like walking motion generator, walking experiments were conducted on SLLRE AutoLEE-G3 both in a simulation environment and the real world. The experimental results show that the humanlike walking motions look more natural and reduce the required torque for the knee joint compared with knee-bent walking.

Crouch gait is one of the key characteristics of children with cerebral palsy, and early detection of gait changes is crucial for subsequent exoskeleton-assisted therapy. This study uses the Vicon 3D motion capture system to collect experimental data on four gait phases of children with cerebral palsy and introduces a CNN-LSTM hybrid model. The model combines the spatial feature extraction strengths of CNN with the temporal sequence modeling capabilities of LSTM, enabling it to effectively identify the complex dynamic changes in gait specific to children with cerebral palsy. By integrating these two components, the model not only accurately extracts key gait features but also captures the temporal dependencies within the gait cycle, allowing for precise recognition of crouch gait. Experimental results demonstrate that the proposed model exhibits good robustness and achieves high accuracy in both overall gait recognition and distinguishing the four individual gait phases. It significantly outperforms traditional machine learning architectures.

By accurately recognizing the wearer’s motion, the underwater exoskeleton enables more efficient human-machine collaboration and provides enhanced assistance in complex and dynamic underwater environments. In this study, we propose a soft underwater exosuit motion mode recognizer based on a long short-term memory network and convolutional neural networks, referred to as LSTM-CNN. This model is designed to perform two tasks: motion mode classification and state transition label recognition. First, the LSTM network extracts features from the time-series data, followed by further feature extraction and classification using the convolutional and fully connected networks. The recognition of motion modes relies on three IMU sensors placed on the left and right legs and the back of the torso of the soft underwater exosuit. On the dataset containing four classes, including non-assist, breaststroke, flutter kick, and underwater walking, LSTM-CNN achieved an overall accuracy of 99.943±0.006% in motion mode classification and 92.101±0.054% in state transition label recognition. The experimental results indicate that the LSTM-CNN achieves better accuracy and performs optimally across various evaluation metrics compared to the other methods.

Underwater assistance is crucial for individuals who depend on diving for their livelihood. In this paper, we propose a novel underwater exosuit actuator designed to assist with flutter kicking during diving, thereby decreasing the effort the diver has to exert. The actuator can provide bidirectional assistance to the up and downbeats when the diver kicks underwater, and has no restriction on leg movements when it is deactivated. Both the benchtop experiment and human subject tests were conducted to verify its performance. The benchtop experiment verified its kinematic features, while tests with five participants validated its assistive performance. The results indicate that the actuator delivers a peak torque of 0.0947 Nm/kg and a peak force of 100 N in both directions, while allowing free leg movement during walking or kicking when not powered, thus ensuring safety during diving.

Magnetically actuated robots have recently shown great capabilities for remote applications in medical procedures. However, the efficient actuation of magnetic robots with dexterous field and gradient generation in large workspaces remains challenging. To overcome the critical challenges, we report a reconfigurable electromagnetic actuation system (REMA) for regulating magnetic fields (maximum: 17 mT) and gradients (maximum: 120 mT/m) in large workspaces. Reconfigurable coil configurations are achieved by employing three mobile electromagnetic coils mounted on three independent 6-DOF robotic arms. Furthermore, the field characteristics generated by a single coil and three coils were modeled via Finite-element method (FEM) and measurements from experiments, respectively. Since there are non-linearities between desired field generation and coil configuration, we propose a multi-objective optimization (MOO) method for generating the Pareto-optimized coil configuration to achieve field and force control in large workspaces. Finally, extensive experiments were conducted to demonstrate the capability and dexterity of our system for autonomous magnetic manipulation in large workspaces, thus showing its potential for clinical applications. Note to Practitioners—This paper aims to address the dexterous generation of magnetic fields and gradients in large workspaces, aiming to realize accurate, efficient, and automated control of different magnetic robots. This paper introduces a reconfigurable electromagnetic actuation system based on three independent robotic arms with three electromagnetic coils. Subsequently, we propose a multi-objective optimization (MOO) method to regulate the coil configuration for generating different fields and gradients. This approach facilitates the application of magnetically driven helical robots, catheters, and capsule robots in various medical scenarios. The results demonstrate that our proposed platform and optimization strategy can effectively implement magnetic manipulations across diverse application scenarios. Looking ahead, we anticipate integrating our work with medical imaging devices to furnish doctors with enhanced tools for medical applications.

This paper proposes a specific and effective real-time sequence planning method using robot manipulators to complete complex assembly tasks. Many previous studies developed different traversal methods to obtain the optimal assembly sequence. Besides, a number of algorithms were proposed to enhance flexibility when the conditions or rules were changed in various sequence optimization problems. However, these state-of-the-art (STOA) methods necessarily require modifications when task details are changed. Consequently, to further improve the generalization ability and improve the performance of the sequence optimization, a neural combinatorial optimization algorithm combined with a self-learning strategy is proposed for assembly sequence planning. In addition, obstacle avoidance and the non-collision constraints between workpieces in the assembly process are considered. According to the experiment results, the new method is superior to the STOA methods in terms of optimization efficiency. More importantly, the proposed method has satisfactory generalization ability for different assembly tasks.Note to Practitioners—This paper studies assembly sequence planning problems for different real-world applications in industrial and home service fields. Many assembly sequence planning solutions have been widely utilized before. However, the generalization ability of the previous methods is not satisfactory since the re-adjust process is required when the workpiece number or collision condition changes in different tasks.Motivated by the above reasons, this paper develops a learning-based assembly sequence planning solution to resolve complex assembly problems without parameter re-adjustment processes. Users can directly apply the developed workpiece identification and localization method to obtain the sensing information. Then, the newly designed collision-free cost function should be programmed as the core of the assembly sequence optimization. Next, the proposed neural combinatorial optimization (NCO) with the sensing information and target configuration as inputs can provide the optimal assembly sequence by self-learning. The learned NCO-based method can be directly applied to diverse planning tasks, even with different workpiece numbers. Users can also refer to the experimental examples in this paper for the extension of the proposed method to their own applications.

Gaussian process (GP) offers a robust solution for modeling the dynamics of unmanned surface vehicles (USV) in model-based reinforcement learning (MBRL). However, the rapidly increasing computational complexity with a large sample capacity of GP limits its application in complex scenarios that require substantial samples to cover the state space. In this article, a novel probabilistic MBRL approach, probabilistic neural networks model predictive control (PNMPC) is proposed to tackle this issue. With an iterative learning framework, PNMPC properly models the USV dynamics using neural networks from a probabilistic perspective to avoid the computational complexity associated with sample capacity. Employing this model to effectively propagate system uncertainties, a model predictive control (MPC) policy is developed to robustly control the USV against external disturbances. Evaluated by position-keeping and multiple targets-tracking scenarios on a real USV data-driven simulation, the proposed method consistently demonstrates its significant superiority in both model accuracy and control performance compared to not only GP model-based approaches but also the probabilistic neural networks-based MBRL baselines, across various scales of external disturbances.Note to Practitioners—Modelling the system dynamics and maintaining computational efficiency with a large sample set has been challenging for MBRL in the USV domain. We propose a novel neural network modeling method to capture the dynamic features of USV within an RL loop and develop a robust MPC policy based on its uncertainty propagation. Our method achieves computational complexity independent of the sample capacity and outperforms related baselines in model accuracy and control performance.

In this paper, we propose practical model-based policy optimization (PMBPO) to address the time efficiency issue caused by overly frequent model updates in recent probabilistic model-based reinforcement learning (MBRL) methods that accelerate learning by generating samples from the model. PMBPO enhances the reliability of the generated samples by introducing an expressive probabilistic model that focuses on the system’s dynamic features over continuous time steps. A time-efficient learning framework is proposed by offline updating and interacting with the model at the end of each epoch. One policy fallback mechanism is further designed to mitigate the negative impact of model bias on the learned policy. Evaluated on five Mujoco control benchmarks and one quadruped robot control scenario, PMBPO reduces the one-step computation time by 90% while achieving 70% more cumulative rewards compared to the state-of-the-art MBRL approaches. It extends the feasibility of MBRL in practical control scenarios. The code of PMBPO is available at https: //github.com/mrjun123/PMBPO.

Untethered magnetically actuated soft microrobots are promising in micromanipulation applications. Autonomous navigation and micromanipulation in dynamic environments are challenging because uncertain dynamic obstacles lead to increased planning time and reduced real-time performance. Here, we developed a novel H-shape soft magnetic microrobot with finger behaviors, including gripping and releasing. Without additional stimuli (e.g., light, temperature, etc.), the finger behaviors and movements of the microrobot can be controlled simultaneously by adjusting the strength and direction of the magnetic fields. We proposed a sliding-window-based local dynamic path planning method in dynamic environments to address the drawbacks of inefficiency and the high computational cost of global search. An enhanced optimized rapidly-exploring random tree algorithm is developed for planning avoidance paths at a safe distance from the boundary of obstacles. In order to automatically manipulate multiple objects, the manipulation process is quantitatively divided into several subtasks, and a finite state machine (FSM) based task planner is proposed to control the switching between subtasks. Experiments demonstrate the autonomous obstacle avoidance and navigation of the microrobot in dynamic environments with obstacles moving 2.5 times faster than the robot. Eight spherical objects of 2 mm diameter were automatically transported to the corresponding target position with an average error of 0.29 mm and an average transport time of 17.25 seconds. In the future, this work promises to enable automated cell sorting by microrobots. Note to Practitioners—This article was motivated by the recent interest in utilizing the small-scale microrobots to perform micromanipulation tasks. For automated manipulation of objects, effective Untethered actuation, manipulation behavior control, and navigation are required. Besides, obstacle avoidance and dynamic obstacles in the environment need to be considered. The strategy proposed here is developed for automated multi-object manipulation via a novel soft magnetic microrobot. The developed task planner quantitatively divides the operation process into several subtasks and controls the switching between them. The dynamic path planning algorithm is used to efficiently search for collision avoidance paths in dynamic obstacle environments. The motion-behavior coordination controller is used to control the robot to perform manipulation subtasks. Experimental results have validated the proposed strategy in actuation, navigation, and multiple cargoes manipulation in dynamic obstacle environments.

Currently, deep neural networks can be extremely effective in robotic tactile perception. However, a major challenge is to solve the problem of continual learning of robotic tactile perception in an open and dynamic environment. In this paper, we propose a novel continual learning method for the domian incremental learning task in the field of tactile perception. To be specific, we introduce a morphology-specific variational autoencoders which can mitigate catastrophic forgetting by generating pseudo-samples for training in the continual learning process. We integrate the generative model and the discriminative model into one model, which reduces the size of model and improves the continual learning ability. In addition, considering the ordinal information between the hardness levels, we propose to add conditional information to the model and introduce a modified loss function to combine the latent value with the hardness information, which improves the continual learning performance by controlling the distribution and quality of pseudo-sample generation. Following this, we designed a tactile robot experiment, collected hardness data, and tested our model on this object hardness recognition task. We show experimentally that, after training, the model can still maintain the accuracy of more than 94% after learning three tasks in terms. Note to Practitioners—In the field of robotics tactile perception, the issue of continual learning in robots is a crucial problem that urgently requires resolution. We hope robots to effectively engage in continual learning across multiple tasks, ensuring the acquisition of new knowledge while mitigating the risk of forgetting previously acquired knowledge. In this paper, we propose a novel continual learning method for the domian incremental learning task. we introduce a morphology-specific variational autoencoders based on replaying pseudo-samples during continual learning process which reduces the size of model and improves the continual learning ability. We enhance model performance by integrating generative and discriminative models, incorporating conditional information to control the distribution of replayed sample types, and leveraging sequential relationships among samples. It is proved that the proposed method is able to effectively improve the accuracy in a tactile domian incremental learning task.

This paper focused on developing an effective Multi-Agent Reinforcement Learning (MARL) approach that quickly explores optimal control policies of multiple agents through interactions with unknown environments. Multi-Agent Continuous Dynamic Policy Gradient (MACDPP) was proposed to tackle the issues of limited capability and sample efficiency in the current MARL approaches. It alleviates the inconsistency of multiple agents’ policy updates by introducing the relative entropy regularization to the Centralized Training with Decentralized Execution (CTDE) framework with the Actor-Critic (AC) structure. Evaluated by multi-agent cooperation and competition tasks and traditional control tasks including OpenAI benchmarks and robot arm manipulation, MACDPP demonstrates its significant superiority in learning capability and sample efficiency compared with both related multi-agent and widely implemented signal-agent baselines. It converges to$62\%$higher average return and uses$38\%$fewer samples compared with the suboptimal baseline over all tasks, indicating the potential of MARL in challenging control scenarios, especially when the number of interactions is limited. The open source code of MACDPP is available at https://github.com/AdrienLin1/MACDPP.Note to Practitioners—Learning proper cooperation strategy over multiple agents in complicated systems has been a challenge in the domain of Reinforcement Learning. Our work extends the traditional MARL approach FKDPP that has been successfully implemented in the real-world chemical plant by Yokogawa to the CTDE framework and AC structure that supports continuous actions. This extension significantly expands its range of applications from cooperative/competitive tasks to the joint control of one complex system while maintaining its effectiveness.

Self-balancing walking exoskeletons (SBWEs), which enable paralyzed individuals to walk without assistive devices, have been increasingly employed in rehabilitation training. This paper proposes a Kelvin-Voigt viscoelastic model-based bioinspired viscoelastic compliance controller (BVCC) for a novel SBWE named AutoLEE-II, which features high structural rigidity, low leg inertia relative to center of mass (CoM), and small hip joint axis misalignment between the user and SBWE. First, a novel series-parallel hybrid mechanism is designed for AutoLEE-II. This mechanism, inspired by the lower limbs of humans, reduces hip axis misalignment between the user and SBWE, decreases leg inertia relative to CoM, and improves structural stiffness. Second, a BVCC mimicking biological muscle is proposed to introduce viscoelastic compliance to SBWE to maintain locomotion stability of the SBWE during standing and walking. The BVCC is robust to the variable physical parameters of different users. Finally, self-balancing walking experiments are conducted with AutoLEE-II with empty load, manikin load and human subject load to validate the performance of AutoLEE-II and the proposed compliance controller BVCC. Note to Practitioners—This paper aims to design a self-balancing walking exoskeleton (SBWE) that provides rehabilitation training exercise and walking assistance services for individuals with hemiplegia, paraplegia, and quadriplegia. First, we biomimetically designed the mechanical structure of the SBWE, named AutoLEE-II based on the distribution of human joints and connecting links. The bionic mechanism reduces axis misalignment between the SBWE and users, improves stiffness and reduces the inertia of the legs relative to center of mass. We then designed a bioinspired viscoelastic compliance controller (BVCC) based on the centroid dynamics model, which is robust to the physical properties of the user and introduces the SBWE with active compliance. Finally, self-balancing walking experiments with an empty load, a manikin load and human subject loads are performed to validate mechanical structure of the proposed AutoLEE-II and the locomotion stability of the physical parameter robust BVCC.

Exosuits play an essential role in facilitating rehabilitation training for patients with functional impairments. However, the safety of human-exosuit interactions and their adaptability to varied environments are crucial challenges that hinder their transition from laboratory prototypes to practical real-world applications. This study presents an innovative active exosuit designed to correct foot drop, characterized by its safe interactive performance and terrain daptability. Regarding safe human-exosuit interaction, meticulous design of the dimensions of hinge module ensures that, upon contact between the hinges, the driving rope reaches its maximum contraction length, thereby mechanically safeguarding against excessive dorsiflexion of the ankle joint by preventing further stretch. In terms of adaptability to various terrains, a visual system capable of terrain recognition has been integrated, enabling a seamless transition between control modes based on the detected terrain, achieving a recognition accuracy of 99.1% through the Transformer in Transformer (TNT) algorithm. To quantify the effect of the exosuit in correcting drop-foot, five healthy participants with artificially induced impairment were recruited to participate in the experiment. The experimental results revealed that the peak of the plantarflexion angle was decreased by 21.3%, 26.0%, and 46.5%, respectively, with the assistance of an exosuit when walking at the terrains of level ground (LG), upramps (UR), and upstairs (US). The root mean square (RMS) of the electromyography (EMG) signals of the tibialis anterior (TA) muscle was reduced by 26.9%, 32.5%, and 20.2%, respectively. These initial results demonstrate that the exosuit can effectively assist dorsiflexion at different terrains.Note to Practitioners—This paper has presented the design and evaluation of a soft exosuit equipped with multiple hinges and a vision system, which offers dorsiflexion assistance to improve ground clearance across various terrains. The multi-hinge structure facilitates effective assistance force transmission and acts as a mechanical limiter, preventing damage to the ankle joint from overstretching the Bowden cable due to control errors. The vision subsystem enables precise terrain recognition, facilitating tailored assistance based on the current terrain conditions. Moreover, a position-based control strategy is adopted based on the reference trajectory measured from the healthy side, enabling the exosuit to adapt to three terrains: LG, UR, and US. This work has the potential to advance the development of exosuits for practical applications significantly.

How to optimally synchronize exoskeleton powered assistance remains a problem that limits the broad application of such devices. Kinematic change during frequent switching between go and stop, a common and representative activity of daily living (ADL), makes it challenging to predict the gait phase and deliver assistance due to unpredictable movements. Conventional adaptive oscillators (AO) have been verified to be effective in gait phase prediction in steady-state walking. When walking cadences are changed, it usually requires multiple walking strides to synchronize the assistance, causing inaccurate or even unwanted force disturbances that can be dangerous in some cases. To solve this problem, a gait-based AO is proposed in this paper. It has two-level AO systems. The high-level AO learns from last stride and updates the low-level AO, which is designed to estimate the gait phase of the current stride in real time. This approach significantly decreases the time required to learn walking kinematics, while simultaneously improving the accuracy of predictions. Experiments were conducted on seven participants, and the results showed that the proposed gait-based AO can predict the gait phase faster, more accurate, and more stable than the conventional one during non-steady-state walking. Note to Practitioners—This article developed a new gait phase prediction method called gait-based adaptive oscillator. It aims to provide fast and reliable gait phase prediction in situations characterized by frequent transitioning between stop and go (non-steady-state walking). This method has the potential to be an alternative to existing phase prediction methods for robotic exoskeleton assisted walking in daily life, as it requires no model training before use while can synchronize exoskeleton assistance within two strides.

This article addresses the robustness of unmanned surface vehicles (USV) using model-based reinforcement learning (MBRL). A novel MBRL approach, Lyapunov probabilistic model predictive control (LPMPC) is proposed to simultaneously learn both the probabilistic model of a USV and its corresponding estimated Lyapunov region of attraction (ROA) under one reinforcement learning framework. Unlike the existing MBRL USV systems with less consideration of robustness and safety, our method naturally learns a general indicator of system stability based on the probabilistic model’s belief and employs it to guide its policy. Evaluated by different navigation tasks in a simulation driven by real boat data, LPMPC demonstrated significant advantages in both control robustness and task completion against various levels of environmental disturbances compared with the baseline approach without Lyapunov ROA’s guidance. Note to Practitioners—Modelling the system stability without human prior knowledge is challenging in the domain of USV. This work proposed a data-driven method to iteratively learn a task-relevant stability model of USV in a probabilistic view. Based on the evaluation of a real boat data-driven simulation, the learned stability model contributed to superior driving skills in different USV scenarios by properly indicating and avoiding potentially risky states. In future research, we plan to expand the definition of risks in different tasks, such as loss of control, overlarge sway, and excessive energy consumption and investigate the proposed approach in real-world USV.

The design and development of new exoskeleton robots can help patients with lower limb paralysis realize autonomous walking. During the motion of an exoskeleton robot carrying patients, the inertia of the hip joint will have time-varying characteristics, which will cause fluctuations in the rotation angle and affect the walking stability of the exoskeleton robot. In this paper, we present the rotation angle control strategy for an exoskeleton robot assisted by paraplegic patients with the BP neural network tuning control strategy. First, based on the skeleton structure of human lower limbs, an exoskeleton robot with 12 degrees of freedom is designed and manufactured to help patients autonomously walk. Next, the dynamic model of the exoskeleton robot hip joint is established, which takes into account nonlinear factors such as transfer flexibility, friction torque, and time-varying load inertia. Then, BP neural networks are used to adjust the parameters of the position loop PID controllers in the hip joint, and the tracking error is reduced by adjusting the controller parameters in real time. Finally, walking experiments of the physical prototype of the exoskeleton robot show that the exoskeleton robot designed in this paper can help patients with lower limb paralysis walk autonomously, and the proposed control strategy can reduce the rotation angle tracking error of the hip joint.Note to Practitioners—This paper addresses the importance of dynamic modeling and control for the hip joint in the motion accuracy of exoskeleton robots. The mechanical structure of the proposed exoskeleton robot can realize the autonomous walking of patients with lower limb paralysis. The proposed dynamic modeling method is suitable for the servo system time-varying model of split-limb robots. In addition, the proposed BP neural network control strategy can improve the position control accuracy of time-varying systems, which is suitable for the real-time control of robots. Numerical simulation and physical experiments demonstrate the effectiveness of the proposed control strategy.

The operation accuracy of humanoid robot hands driven by the tendon sheath will be reduced due to the influence of friction torque during rotation, which is not conducive to the dexterous operations of humanoid robot hands. In order to improve the control accuracy of humanoid robot hands, this paper proposes a control strategy based on the disturbance observer compensation, which eliminates the external disturbance torque by compensating the friction torque. Firstly, this article proposes the mechanical structure of humanoid robot hands driven by the tendon sheath with 19 degrees of freedom (DOF). This humanoid robot hands can grasp most irregular objects. Next, the dynamic model of humanoid robot hands’ drive systems is established based on the tendon sheath transmission theory. The driving system’s dynamic model reveals the influence of friction torque on the motion accuracy of the humanoid robot hand. Then, the disturbance observer (DOB) is designed based on the robust stability theorem. The DOB is used to improve the control accuracy of the driving system, thereby improving the operational accuracy of humanoid robot hands. Finally, this article conducts simulation rotation tracking control and prototype grasping control experiments on humanoid robot hands. The experimental results show that the proposed control strategy based on disturbance compensation can effectively improve the operational accuracy of humanoid robot hands. Note to Practitioners—This paper proposes a humanoid hand with 19 degrees of freedom based on the tendon-driven theory and applies the cable theory for its dynamic modeling. To address the issue of decreased precision caused by friction in practical operations, a friction compensation control strategy based on disturbance observer is proposed in this study. This control strategy improves the motion accuracy and stability of the mechanical hand. Finally, the effectiveness of the proposed control strategy is demonstrated through numerical simulation and experimental validation.

A framework is proposed in this paper for learning variable impedance in percutaneous puncture surgery, with the aim of simplifying the robotic puncture of soft tissues. The framework involves simulating the dynamic changes that occur when the human arm interacts with human tissues and transferring the resulting adaptive capabilities to the robot through learning movement trends and stiffness changes. To enhance performance during task execution, we integrate the variable impedance control framework with the interactive operation and feedback controllers. To provide flexibility for trajectory modification during operation, derivative Gaussian processes are introduced to identify the target position and obtain a model of motion trends. This control law is combined with virtual dynamics that describe puncture dynamics, enabling the robot to regulate interactions and plan its trajectory. We present experiments involving tissue puncturing tasks performed by the Franka-Emika Panda robot with varying degrees of hardness. The results demonstrate that our framework is capable of learning manipulation skills for physical interaction with humans, thereby reducing application complexity in tasks involving complex force interactions for robots. Compared to using fixed or variable impedance gain controllers, our approach effectively improves the success rate, stability, and efficiency of percutaneous puncture. Note to Practitioners—This paper is motivated by the limitations encountered by robots when handling deformed objects. In traditional robot control processes, the assumption of a fixed and unchanging contact object poses a significant challenge in applying robot control to the medical industry. Consequently, it becomes imperative for robot control systems to develop stable intelligent approaches capable of interacting with deformed objects. In this paper, we propose a framework for robot-assisted puncture that combines robotic impedance control techniques with sensing mechanisms. By integrating these approaches, our framework demonstrates effectiveness in performing tasks involving soft tissues with varying levels of hardness. Our proposed method encompasses three main ideas: 1) Sensing muscle activity during task execution enables the acquisition of task parameters from the human arm. 2) The utilization of a robot control method enhances the stability of the robot’s execution process. 3) The proposed method shows potential for application in processing and treating objects with low stiffness, deformed objects, and thin-walled parts. Experimental results validate the effectiveness of the developed method. In future work, it is important for the robot-assisted puncture system to consider recognizing and localizing more diverse targets to enhance its generalization capabilities.

Existing research suggests that lower limb exoskeleton robots, when used for rehabilitation training based on the pre-stroke gait trajectories of stroke patients, may be more beneficial for gait rehabilitation. However, it’s challenging to obtain such personalized trajectories for specific patients. Therefore, this hypothesis is difficult to be verified. This paper introduces an Individualized Gait Trajectory Generation (IGTG) method based on Fast Fourier Transform (FFT) to approximate and regress pre-stroke gaits, along with conducting clinical rehabilitation validation trials. Initially, human gait trajectories are described using Fourier coefficients to construct gait features. Subsequently, a probabilistic mapping between these gait features and physical body parameters is established. Then, personalized gait trajectories are obtained by applying the inverse Fourier transform to the predicted gait features. The application of fast Fourier transform can reduce the number of the regression data points needed, decrease dependency on large datasets, and enhance the systematic robustness. This algorithm is trained using body parameters and gait trajectories collected from 128 healthy subjects. The algorithm is further applied to generate specific personalized trajectories for the 9 stroke patients. Clinical trial results indicate that rehabilitation training using these individualized gait trajectories reduces blood oxygen saturation (SpO2) and heart rate (HR) by up to 66.67% and 69.23% respectively compared to training with fixed trajectories. Note to Practitioners—The main purpose of this paper is to solve gait trajectories mismatch problem when different stroke patients use lower limb exoskeleton robot for rehabilitation training. Variations in body factors among individuals lead to different gait trajectories including walking speed, gender, age, and other anthropometric parameters. Therefore, this paper introduces a novel Individualized Gait Trajectory Generation (IGTG) method to generate suitable gait trajectories for stroke patients with different body characteristic parameters when taking gait rehabilitation training with a lower limb exoskeleton robot. The detailed methodology introduction and a full analysis of experimental results are also given. Finally, clinical experiments involving stroke patients were conducted to demonstrate the feasibility and effectiveness of the presented method.

Lateral walking gait phase recognition and prediction are the premise of hip exoskeleton application in lateral resistance walk exercise. We presented a fusion network with stacked denoise autoencoder and meta learning (SDA-NN-ML) to recognize gait phase and predict gait percentage from IMU signals. Experiments were conducted to detect the four lateral walking gait phases and predict their percentage across different speeds. The performance of SDA-NN-ML and Support Vector Machine (SVM), Adaptive Boosting (AdaBoost) and Long Short Term Memory (LSTM) were evaluated. The cross-subject recognition accuracy of SDA-NN-ML (89.94%) decreased by 4.62% compared to the training accuracy, which outperformed SVM (8.60%), AdaBoost (5.61%), and LSTM (7.12%). For real-time and cross-subject prediction of gait phase percentage, the RMSE of SDA-NN-ML (0.2043) outperformed that of a single regression network (0.2426). With a signal noise ratio of 100:30, the cross-subject recognition accuracy decreased by a mere 5.70%, while the prediction result (RMSE) of SDA-NN-ML increased by 0.0167 when compared to the noise-free results. SDA-NN-ML demonstrates a stable multi-step-ahead prediction ability with an accuracy higher than 82.50% and an RMSE of less than 0.23 when the ahead time is less than 200 ms. The results demonstrated that the proposed method has high accuracy and robust performance in lateral walking gait recognition and prediction.

Target localization technology is widely applied in various applications, such as rescue missions, robot navigation, and the Internet of Things. However, in some scenarios, the positions of sensors are unknown due to the load limitation of the sensor carriers and environmental interferences, resulting in the instability of the global navigation satellite system (GNSS). This paper focuses on optimal angle-of-arrival (AOA) sensor placement using multiple position-unknown sensors for target localization accuracy improvement. To guarantee the uniqueness of the target coordinate, at least two anchors are needed. The anchors are some static benchmark objects in the environment with priori known positions. Firstly, a new optimization problem for AOA target localization accuracy improvement incorporating position-unknown sensors and anchors is formulated. Secondly, the optimal theoretical localization accuracies of the unknown sensors and target are derived by minimizing the trace of the Cramér-Rao lower bounds (CRLBs). Thirdly, a mixture optimization method, including a geometrical initialization and the new proposed simultaneous perturbation stochastic approximation and adaptive momentum estimation (SPSA-Adam) algebraic algorithm, is developed. Then, the correctness of the new theoretical findings and the effectiveness of the proposed sensor placement optimization method are verified by simulation examples.

This article addresses the prediction stability, prediction accuracy, and control capability of the current probabilistic model-based reinforcement learning (MBRL) built on neural networks. A novel approach to dropout-based probabilistic ensembles with trajectory sampling (DPETS) is proposed, where the system uncertainty is stably predicted by combining the Monte Carlo dropout (MC Dropout) and trajectory sampling in one framework. Its loss function is designed to correct the fitting error of neural networks for more accurate prediction of probabilistic models. The state propagation in its policy is extended to filter the aleatoric uncertainty for superior control capability. Evaluated by several Mujoco benchmark control tasks under additional disturbances and one practical robot arm manipulation task, DPETS outperforms related MBRL approaches in both average return and convergence velocity while achieving superior performance than well-known model-free baselines with significant sample efficiency. The open-source code of DPETS is available at https://github.com/mrjun123/DPETS.

Magnetic microrobots have garnered significant attention and hold great potential for biomedical research applications. However, achieving precise manipulation in vivo poses significant challenges, particularly in medical image-based real-time feedback control, because it is difficult for a visual camera to track the motion of magnetic microrobots inside the body in biomedical applications. To realize the precise control of magnetic microrobots, it is also necessary to design and implement a simple and powerful control method. This approach allows for avoiding resource-intensive and complex control strategies. In this article, we present a learning-based real-time control method utilizing ultrasound images. Inspired by the ADboost concept, we use a reinforcement learning approach to integrate two simple control methods: a proportional-integral-derivative controller and a guiding vector field controller. We develop a novel$Q$-learning method called average$Q$-learning that incorporates average operation and$n$-step bootstraps. Its primary objective is to dynamically adjust the outputs of the different simple controllers. While each controller individually offers a straightforward solution, their integration contributes to a powerful control approach. To demonstrate its scalability, a nonsmooth path is utilized to investigate the integration performance of three simple controllers. In addition, we enhance a classic segmentation module, U-net, by incorporating an atrous spatial pyramid pooling module. To validate the effectiveness of the proposed control method, we conduct simulations and experiments using various planar paths. The quantitative analysis of the results demonstrates the efficacy of our approach in achieving precise manipulation, leveraging real-time control based on medical images for magnetic microrobots. Overall, this study provides a preliminary investigation into the field of medical image-based precise manipulation of magnetic microrobots in vivo applications.

Assisting underwater movements improves divers' efficiency and reduces the risk of decompression sickness from physical activity. Although exoskeletons have been developed for numerous land-based scenarios, their application in underwater diving remains unexplored. This article proposes a soft underwater lower-limb exosuit designed to assist three aquatic movements: flutter kick, breaststroke kick, and underwater walk. We presented the mechanical design of the exosuit that is capable of assisting bidirectional leg movements in full kicking/gait cycle, while ensuring natural leg mobility without impeding normal leg function. A cascade force integral controller is also designed to resolve issues related to uncontrollable states and stiffness variations within the system. To verify the assistive performance of the system, experiments were conducted with nine participants to assess how the proposed exosuit aids in reducing metabolic cost across various motion patterns and frequencies. The findings indicate that the underwater exosuit effectively reduces the air consumption rate by$29.77\pm 7.68$% during flutter kick,$25.70\pm 5.99$% during breaststroke kick, and$18.35\pm 4.53$% during underwater walk.

The sense of touch is essential for robots to perform various daily tasks. Artificial Neural Networks have shown significant promise in advancing robotic tactile learning. However, due to the changing of tactile data distribution as robots encounter new tasks, ANN-based robotic tactile learning suffers from catastrophic forgetting. To solve this problem, we introduce a novel continual learning (CL) framework called the Probabilistic Spiking Neural Network with Variational Continual Learning (PSNN-VCL). In this framework, PSNN introduces uncertainty during spike emission and can apply fast Variational Inference by optimizing the uncertainty through backpropagation, which significantly reduces the required model parameters for VCL. We establish a robotic tactile CL benchmark using publicly available datasets to evaluate our method. Experimental results demonstrated that, compared to other CL methods, PSNN-VCL not only achieves superior performance in terms of widely used CL metrics but also achieves at least a 50% reduction in model parameters on the robotic tactile CL benchmark.

In this paper, considering overall deformations of the exoskeleton, we couple deformations relationship network (DRN) with fractional order viscoelastic (FOV) controller, proposing a novel DRN-FOV closed-loop control method, endowing exoskeleton with stable dynamic walking ability. Simply by utilizing only the data from the 6-axis force/torque sensors, the DRN can directly capture the mapping relationship between the foot reaction force/torque of the exoskeleton and its overall deformations. We introduce the FOV to eliminate disturbances and stabilize during walking tasks. The closed-loop control method directly compensates for the overall deformations of the exoskeleton and enables the wearer to walk stably wearing the exoskeleton. To assess the effectiveness of the proposed control method, walking tasks were effectively carried out on subjects with varying body parameters using the developed exoskeleton. The experimental results show that the DRN-FOV closed-loop control method accurately estimates and compensates for deformations, resulting in an improved dynamic walking ability of the exoskeleton with wearers.

Multi-agent pathfinding (MAPF) is a well-studied abstract model for navigation in a multi-robot system, where every robot finds the path to its goal position without any collision. Due to its numerous practical applications of multi-robot systems, MAPF has steadily emerged as a research hotspot. The optimal solution for MAPF is NP-hard. In this paper, we offer a comprehensive analysis of different MAPF solvers. First, we review the cutting-edge solvers of classical MAPF, including optimal, bounded sub-optimal, and unbounded sub-optimal. The performance of some representative classical MAPF solvers is quantitatively compared. In the next part, we summarize the beyond classical MAPF solvers, which try to use the classical MAPF solvers in real-world scenarios. Last, we conclude some challenges that MAPF is experiencing in detail, review recent research on these issues, and make some suggestions for further work.

Multi-sensor based motion tracking is of great interest to the robotics community as it may lessen the need for expensive optical motion capture equipment. However, the traditional convolution algorithms have difficulty adapting to the data due to the changes of joints’ relative position during motion. The time-series networks often used in the past ignore the spatial characteristics of sensors. We tackle this challenge by combining long short-term memory (LSTM) with graph convolution network (GCN), adding the prior knowledge of sensor distribution, and integrating it into the motion law through the adjacency matrix. This article proposes a novel shortcut enhanced LSTM-GCN network (SE-LSTM-GCN). It connects LSTM and GCN in sequence and extracts temporal and spatial features of data. At the same time, the shortcut is used in the network to enhance the output of two middle layers and to restore the filtered information. Our experimental results on two different motion tracking datasets show that the proposed network is able to learn the mapping relationship with better universality, less tracking error, and without increasing much training time, and can better perform human motion tracking tasks.Note to Practitioners—Accurate and real-time multiple soft sensors motion tracking suits are more accepted for their low cost. However, the soft-sensor based motion tracking is not comparable to the traditional optical equipment in prediction error. To this end, we present a novel network shortcut enhanced LSTM-GCN (SE-LSTM-GCN), consisting of shortcuts, long short-term memory (LSTM), and graph convolution network (GCN). The LSTM solves the non-linear and hysteresis of soft strain sensors, and GCN is integrated into the network since the knowledge of sensor location can be put into the adjacency matrix generated by the k-nearest neighbor (KNN). While shortcuts are used to enhance the output of middle layers to form combined features. Experimental results on two public datasets show that the proposed network is superior to competing algorithms in terms of prediction error. The network can be deployed in embedded devices, such as VR gloves to provide a better gaming experience. The current algorithm is based on the relationship between sensor data and distance. In future research, we will focus on adding other human kinematics laws to the network.

Magnetic micro/mini-swimmers have a great potential application in biomedical research and have gained broad attention. Recent research studies focus on the automatic control methods of magnetic micro/mini-swimmers, such as artificial intelligence methods. However, their autonomous manipulation remains a challenge since they are subject to various disturbances from the external environment and model uncertainties. The current methods employ a status observer to estimate these disturbances and uncertainties. In this paper, we apply a data-driven technique that utilizes the nonlinear approximation ability of neural networks (NNs). To be specific, a flexible structure of NNs, Broad Learning System (BLS), is employed to model the input-output mapping relationship between the direction of the rotating magnetic field and the swimming direction of the helical miniature swimmer. Then, according to the dual-variable decoupling control, a levitation controller is formulated and a path following controller is proposed based on a planar three-degree-of-freedom model of magnetic helical swimmers, which is inspired by the model of wheeled mobile robots. Simulations and experiments are conducted to quantitatively validate the proposed control method using different planar paths. The experiment results show that the mean absolute error of the path following control is about 3% of the body length which is less than$0.5mm$. Our proposed control method provides a preliminary study to alleviate the impact of disturbances and uncertainties on the control performance of magnetic micro/mini-swimmers. Note to Practitioners—This paper is mainly motivated by the potential application of artificial intelligence methods in magnetic micro/mini-robot community, especially for the applications that reject disturbances and uncertainties. In practice, a path planner is employed to compute a pre-defined reference path that connects the start and the targeted locations. Then our control method is applied to the magnetic helical swimmer and it is guided to follow the reference path. Tackling external disturbances and model uncertainties is still challenging during the control progress. Neural network-based technique is an intuitive approach since their nonlinear modeling ability and generalization are suitable for estimating these disturbances and uncertainties. As for training the neural-network model, the training data about the angle parameters should be recorded by manual control of the helical swimmer. These angle parameters define the rotating axis of the uniform rotating magnetic field and the self-rotating axis of the magnetic helical swimmer. According to the planar motion model, the optimal controller is formulated using the feedback distance error and angle error, and the sum of the control signal and the prediction of the compensating model is used as the final control input. Our electromagnetic coil system features easy operation and configuration of cameras or other sensors. Simulations and experiments validate the performance of the neural network-based compensating method and the proposed optimal control method using magnetic micro/mini-swimmers.

Animals, including humans that have muscles with viscoelastic compliance can achieve improved stability. Hence, we investigate a biomimetic control framework with viscoelastic compliance and subsequently apply it to a self-balancing lower limb exoskeleton robot (SBLLER), to ensure stability during locomotion. This article presents a novel biomimetic viscoelastic compliance control framework (VCCF) for an SBLLER that enables wearers to walk without the need for crutches or other external stabilization tools during self-balancing locomotion and rehabilitation training tasks. First, we devised a biomimetic viscoelastic mathematical model (BVMM) and subsequently analyzed its viscoelastic properties. Second, the VCCF, which incorporates the desired and real center of mass (CoM) and a BVMM that connects the desired and real CoMs, is designed to introduce active viscoelastic compliance for an SBLLER so that the SBLLER can absorb the early landing impact forces and stabilize itself. In addition, the VCCF exhibits robustness to the physical parameters of diverse wearers and can effectively accommodate a wide range of subjects, which is essential for promoting the application of exoskeletons. Finally, the validity of the proposed VCCF is confirmed through disturbance and walking experiments.

Remote-actuated magnetic robots, relying solely on the magnetic torque stemming from rotating magnetic fields, hold immense promise in biomedical applications. However, to precisely actuate magnetic robots in large workspaces, the efficient generation of isotropic rotating fields using electromagnetic actuation (EMA) systems presents an enduring challenge. This is because the choice of configuration of the EMA system is a major concern, particularly when considering collision avoidance between coils and the human body while ensuring isotropic actuation. In this study, we presented an analysis of the characteristics of various three-coil configurations by quantitatively evaluating field isotropy. Furthermore, we introduced a performance-guided optimization method to adjust coil configurations by optimizing designed evaluation metrics, aiming to generate rotating fields with isotropic characteristics in a target local region. Finally, we implemented a reconfigurable EMA and conducted extensive experiments to demonstrate the capability of our method and platform. The experimental results showcase the potential of our approach for advanced clinical applications.

Untethered microrobots possess a promising perspective for micromanipulation applications. With specifically designed morphologies and structures, microrobots are able to perform controllable delivery of target objects. However, the manipulation process still lacks autonomy, to achieve which the mechanism of picking, transporting, and releasing behaviors needs further investigation. In this article, we propose to achieve automated microrobotic manipulation using magnetic microswarms with multimodal morphology. The microswarm is composed of around 11–21 million$\text{Fe}_{3}\mathrm{O}_{4}$nanoparticles (1.0$\text{--}1.8\,\mu$L particle suspension). When exposed to different dynamic magnetic fields, the swarm could exhibit corresponding forms. We realize precise and controllable cargo picking and releasing by exploiting the fluid fields of different swarm forms. In order to quantitatively describe these behaviors, we design a finite-state machine. A super-twisting sliding-mode controller has been formulated for the motion control of swarms. The disturbances are compensated via a disturbance observer. To enable automated micromanipulation in obstructed scenarios, a path planner inspired by rapidly exploring random tree algorithm is designed for path planning when obstacles exist. We also propose an enhanced-genetic algorithm to optimally transport multiple objects to the target position. Experiments demonstrate that our method could effectively transport micro-objects with different sizes and shapes. The precise selectivity of the method is validated when multiple objects exist in the working environment. Finally, the long-distance delivery ability and adaptivity to various friction situations of our strategy are demonstrated. This work explores a concise, untethered, and automated micromanipulation strategy, provides a new automatic tool for micromanipulation tasks, and extends the application potential of swarm microrobotics.

Accurately estimating uncertain body disturbances is critical for the effective integration of wearable exoskeletons for active human users. In this article, considering nonlinear time-varying human disturbances, we propose a TDE-BVC feedback control method that performs biomimetic viscoelastic compliance (BVC) with transformer-based deformation estimator (TDE). The method provides human-exoskeleton with stable and dynamic walking capabilities. We developed a transformer-based end-to-end deformation estimation sequence network that simultaneously captures the mapping relationship between foot force/torque and exoskeleton deformation. Moreover, we integrated the BVC to eliminate the impact and external disturbances experienced by the human-exoskeleton, enabling it to closely follow a reference gait, and utilized Lyapunov’s theorem to prove its stability. The control strategy is independent of the parameters of the human exoskeleton. To evaluate the effectiveness of the proposed method, walking experiments were conducted on different subjects. Our results indicate that with only 6-axis force/torque sensors, the TDE-BVC controller could accurately estimate and compensate for the deformation of different human-exoskeletons in each control cycle$(p\lt 0.001)$, with robustly stable and adaptive dynamic walking within a bounded control error.

Hardness is one of the most critical tactile properties for robots to recognize objects. Machine learning methods have shown superior performance in object hardness classification. However, existing machine learning methods for tactile hardness classification cannot use the ordinal information between hardness classes because the one-hot encoding only cares about the correct class and ignores the inter-class relationship. To solve this problem, we propose to generalize the one-hot encoding using unimodal distributions including the Poisson and binomial distributions for tactile ordinal classification problems, resulting in two tactile ordinal networks (TacONet): TacONet-p and TacONet-b. Furthermore, we collect a tactile hardness dataset on the silicone samples with three different shapes (Shapes A, B, C), and each shape samples have thirteen hardness classes ranging from 0A (Shore A scale) to 60A at 5A intervals. We validate the resulting method for tactile hardness classification using a real robot. Experimental results demonstrate that compared with state-of-the-art methods, the proposed method achieves better classification performance in terms of accuracy and quadratic weighted kappa (QWK) on the tactile hardness dataset, reaching a classification accuracy up to 99.5% and a QWK up to 99.9% on Shape C. Note to Practitioners—In the field of robotics tactile recognition, hardness classification is one of the most important and common tasks for robots to accurately recognize objects, particularly when the environment is dark or visual sensors are not working. In this paper, we propose a novel tactile ordinal network for tactile hardness classification tasks. The existing machine learning models for tactile hardness classification are trained by minimizing the cross-entropy loss between predicted vectors and one-hot encoding vectors of true classes, which makes the models only care about the correct classes and ignores the inter-class relationship of hardness classes. In other words, these models have the same probability to misclassify a hardness class with any other hardness class. To tackle this problem, we propose to generalize the one-hot encoding method using a unimodal distribution method to encode the true classes. The unimodal distribution encoding vectors can make the model learn the ordinal information between classes. It is proved that the proposed method is able to effectively improve the classification accuracy and QWK in a tactile hardness classification task.

Multisource biosignals are being increasingly used in human–machine interaction applications. In particular, a critical problem in exoskeleton-assisted rehabilitation for patients with hemiplegia is the generation of rhythmic and symmetrical locomotion. To support lower limb rehabilitation, we propose a model for predicting contralateral joint angles using multisource biosignals. First, a vibroarthrography (VAG) sensor is attached to the affected leg, and surface electromyography sensors are attached to the sound leg. The corresponding signals are used to estimate the hip, knee, and ankle joint angles of the affected leg. Second, an algorithm based on a temporal convolution network (TCN) is introduced to predict the contralateral lower-limb joint angles during human locomotion. The TCN is compared with a long short-term memory (LSTM) network and a convolutional neural network. Experiments were conducted by 10 healthy participants. The results of the proposed model were compared with measurements from encoders in three joints mounted on an exoskeleton to verify the applicability of the proposed model. In addition, by using the outputs of a motion capture system as the ground truth, the experimental results validated the model prediction performance. The prediction root mean square error (RMSE) of the TCN was 52%–70% lower than that of the LSTM network and CNN at different paces. The prediction RMSE using VAG was 20%–24% lower than that without using VAG. Note to Practitioners—We propose a model for predicting the contralateral lower-limb joint angles of an exoskeleton. The model uses measurements from multiple biosignal sensors to support exoskeleton-assisted hemiplegia rehabilitation. A sensing system based on acoustic signals and bioelectric signals is developed to improve the joint angle prediction in the affected leg of patients with hemiplegia. A temporal convolution network is implemented to estimate joint angles, and training considering the root mean square error and Pearson correlation coefficient as evaluation indicators is performed. Unlike similar methods that include no sensory feedback from the affected leg, the proposed method incorporates VAG signals from the affected leg and sEMG signals from the sound leg to handle different degrees of hemiplegia and support the generation of rhythmic and symmetrical locomotion for gait rehabilitation using an exoskeleton.

Soft magnetic miniature robots (SMMRs) have potential biomedical applications due to their flexible size and mobility to access confined environments. However, navigating the robot to a goal site with precise control performance and high repeatability in unstructured environments, especially in flow rate conditions, still remains a challenge. In this study, drawing inspiration from the control requirements of drug delivery and release to the goal lesion site in the presence of dynamic biofluids, we propose a flow rate rejection control strategy based on a deep reinforcement learning (DRL) framework to actuate an SMMR to achieve goal-reaching and hovering in fluidic tubes. To this end, an SMMR is first fabricated, which can be operated by an external magnetic field to realize its desired functionalities. Subsequently, a simulator is constructed based on neural networks to map the relationship between the applied magnetic field and robot locomotion states. With minimal prior knowledge about the environment and dynamics, a gated recurrent unit (GRU)-based DRL algorithm is formulated by considering the designed history state-action and estimated flow rates. In addition, the randomization technique is applied during training to distill the general control policy for the physical SMMR. The results of numerical simulations and experiments are illustrated to demonstrate the robustness and efficacy of the presented control framework. Finally, in-depth analyses and discussions indicate the potentiality of DRL for soft magnetic robots in biomedical applications.

The widespread application of learning-based methods in robotics has allowed significant simplifications to controller design and parameter adjustment. In this article, robot motion is controlled with learning-based methods. A control policy using a broad learning system (BLS) for robot point-reaching motion is developed. A sample application based on a magnetic small-scale robotic system is designed without detailed mathematical modeling of the dynamic systems. The parameter constraints of the nodes in the BLS-based controller are derived based on Lyapunov theory. The design and control training processes for a small-scale magnetic fish motion are presented. Finally, the effectiveness of the proposed method is demonstrated by convergence of the artificial magnetic fish motion to the targeted area with the BLS trajectory, successfully avoiding obstacles.

The lower limb power-assist exoskeletons are expected to help paraplegic people to walk again in daily life. However, most of these exoskeletons deal with walking in the scene that has been seen or has an external vision sensor, rather than in the unknown environment. It is a great challenge to understand the wear’s intention and plan the footstep sequence in an unknown scene. Moreover, the traditional visual footstep planning is dominated by the robot, which can lead to an awkward trajectory plan. Therefore, we construct a visual footstep planning system and propose an onboard vision planning algorithm based on the Bezier curve to address the previous two problems. Specially, our human–computer interaction system understands the environment and the wearer’s behavior intention by integrating Hololens and Realsense. Then, we apply the Bezier curve to plan footsteps for the first time in the field of the exoskeleton and define two parameters of the Bezier curve, which are more suitable for our exoskeleton system and could increase the planning speed. Finally, we add the tracking feature cost in the cost function, which could better fit the planned footprints to the planned path and make the gait smoother. Extended experimental results show that the average planning time is 67.46% less than that of the traditional search algorithm. Moreover, the effectiveness of our system is also verified on the visual interaction platform.

Aiming at the problem that it is difficult to calculate the force of permanent magnets in the magnetic field, this paper proposes a nonlinear mechanical model of linear array magnetic field based on radial basis function neural network (RBFNN). Combined with the linear Halbach array adsorption module of the wall-climbing robot, the three-dimensional geometric magnetic fields of four typical linear array permanent magnets were constructed, and the theoretical models of the interaction between the magnetic fields were given respectively. Further, the finite element simulation calculation of the magnetic force was carried out using COMSOL Multiphysics. According to the parametric scanning results of the orthogonal test, a nonlinear intelligent prediction model of the force between magnetic fields with local loss sensitivity is established by using the RBFNN numerical fitting method. The average deviation of the network test set is 1.19, and the standard deviation is 0.80. The intelligent prediction model has strong generalization performance, faster convergence speed and stronger flexibility, which provides a theoretical basis for the interaction and control of array magnetic fields.

Microswimmer and miniswimmer toward precision-targeted medicine have attracted extensive attention recently. We have developed an autonomous manipulation approach for magnetic-driven helical miniswimmer at low Reynolds number in the horizontal plane ($H$-plane). Different from our previous work which just makes the barycenter of miniswimmer on the reference path as well as takes the swimming direction not into consideration in planar path following, our control policy in this article can make the miniswimmer to follow the reference path and, at the same time, its swimming direction is also along the reference path. A robust tracking method is employed to locate the helical miniswimmer in real time. Due to different external disturbances, an angle compensating model in the global coordinate frame is developed by radial basis function (RBF) networks trained by backpropagation algorithms, which is used to express the swimming model of the helical miniswimmer facing the gravity and lateral disturbances. A discrete-time optimal controller is formulated based on the linear-quadratic feedback control. Simulations and experiments are conducted to quantitatively validate the autonomous manipulation, and the results show the control performance with submillimeter accuracy in the$H$-plane.Note to Practitioners:This article is motivated by the potential application of precision-targeted medicine using magnetic-driven microswimmer/miniswimmer. The formulated controller employs the error model in the horizontal plane to design the control law. Simulations and experiments validate the effectiveness of the proposed discrete-time optimal control scheme using magnetic-driven miniswimmers.

Touch modality identification has attracted increasing attention due to its importance in human–robot interactions. There are three issues involved in the tactile perception for the touch modality identification, including the high dimensionality of tactile signals, complex tensor morphology of tactile sensing units, and the misalignment among different tactile time-series samples. In this article, we propose a novel kernel-based approach to deal with these three issues in a unified framework. Specifically, the techniques, including sparse principal component analysis and subsampling, are employed to reduce the feature dimension. Then, a singular value decomposition (SVD)-based kernel is proposed to preserve the spatial information of the tactile sensing elements. The sample misalignment issue is addressed via the employment of a global alignment kernel. Moreover, the merits of these two kernels are fused through an ideal regularized composite kernel, which simultaneously takes the label information of the training set into consideration. The effectiveness of the proposed kernel-based approach is verified on a public touch modality data set with a comprehensive comparison with the competing methods.Note to Practitioners—In a wealth of tactile recognition tasks, we are in the face of various challenges. For instance, tactile measurements are commonly tensorial and high-dimensional. The misalignments among tactile measurements prevail, such as different durations of tactile measurements and the misaligned starting time point of tactile measurements. This article presents a kernel-based method using an ideal regularized composite kernel to deal with all challenges in a unified framework. The kernel-based method consists of two key components including the SVD-based kernel and the global alignment kernel. The proposed method may shed new insights on new advances in tactile signal processing particularly in human–robot interactions.

As the controller parameter adjustment process is simplified significantly by using learning algorithms, the studies about learning-based control attract a lot of interest in recent years. This article focuses on the intelligent servo control problem using learning from desired demonstrations. Compared with the previous studies about the learning-based servo control, a control policy using the broad learning system (BLS) is developed and first applied to a microrobotic system, since the advantages of the BLS, such as simple structure and no-requirement for retraining when new demos' data is provided. Then, the Lyapunov theory is skillfully combined with the complex learning algorithm to derive the controller parameters' constraints. Thus, the final control policy not only can obtain the movement skills of the desired demonstrations but also have the strong ability of generalization and error convergence. Finally, simulation and experimental examples verify the effectiveness of the proposed strategy using MATLAB and a microswimmer trajectory tracking system.

Road network graphs provide critical information for autonomous-vehicle applications, such as drivable areas that can be used for motion planning algorithms. To find road network graphs, manual annotation is usually inefficient and labor-intensive. Automatically detecting road network graphs could alleviate this issue, but existing works still have some limitations. For example, segmentation-based approaches could not ensure satisfactory topology correctness, and graph-based approaches could not present precise enough detection results. To provide a solution to these problems, we propose a novel approach based on transformer and imitation learning in this article. In view of that high-resolution aerial images could be easily accessed all over the world nowadays, we make use of aerial images in our approach. Taken as input an aerial image, our approach iteratively generates road network graphs vertex-by-vertex. Our approach can handle complicated intersection points with various numbers of incident road segments. We evaluate our approach on a publicly available dataset. The superiority of our approach is demonstrated through comparative experiments. Our work is accompanied by a demonstration video which is available athttps://tonyxuqaq.github.io/projects/RNGDet/.

Magnetically actuated small-scale robots have great potential for numerous applications in remote, confined, or enclosed environments. Multiple small-scale robots enable cooperation and increase the operating efficiency. However, independent control of multiple magnetic small-scale robots is a great challenge, because the robots receive identical control inputs from the same external magnetic field. In this article, we propose a novel strategy of completely decoupled independent control of magnetically actuated flexible swimming millirobots. A flexible millirobot shows a crawling motion on a flat plane within an oscillating magnetic field. Millirobots with different magnetization directions have the same velocity response curve to the oscillating magnetic field but with a difference of phase. We designed and fabricated a group of up to four heterogeneous millirobots with identical geometries and different magnetization directions. According to their velocity response curves, an optimal direction of oscillating magnetic field is calculated to induce a desired velocity vector for the millirobot group, one of which is nonzero and the others are approximately zero. The strategy is verified by experiments of independent position control of up to four millirobots and independent path following control of up to three millirobots with small errors. We further expect that with this independent control strategy, the millirobots will be able to cooperate to finish complicated tasks.

Sensor drift is one of the severe issues that gas sensors suffer from. To alleviate the sensor drift problem, a gas sensor drift compensation approach is proposed based on local discriminant subspace projection (LDSP). The proposed approach aims to find a subspace to reduce the distribution difference between two domains, i.e., the source and target domain. Similar to domain regularized component analysis (DRCA) which is a recently proposed sensor drift correction method, the mean distribution discrepancy is minimized in the common subspace in our approach. LDSP extends DRCA in two aspects, i.e., it not only takes the label information of the source data into consideration to reduce the possibility of the case that samples in the subspace with different class labels stay close to each other, but also borrows the idea of locality-preserving projection to deal with multimodal data. Specifically, inspired by local Fisher discriminant analysis (LFDA), the label information is utilized to maximize the local between-class variance of source data in the latent common subspace and simultaneously minimize the local within-class variance. The formulation of LDSP is a generalized eigenvalue problem that can be readily solved. The experimental results have shown the proposed method outperforms other gas sensor drift compensation methods in terms of classification accuracy on two public gas sensor drift datasets.

This article presents a novel discriminative subspace-learning-based unsupervised domain adaptation (DA) method for the gas sensor drift problem. Many existing subspace learning approaches assume that the gas sensor data follow a certain distribution such as Gaussian, which often does not exist in real-world applications. In this article, we address this issue by proposing a novel discriminative subspace learning method for DA with neighborhood preserving (DANP). We introduce two novel terms, including the intraclass graph term and the interclass graph term, to embed the graphs into DA. Besides, most existing methods ignore the influence of the subspace learning on the classifier design. To tackle this issue, we present a novel classifier design method (DANP+) that incorporates the DA ability of the subspace into the learning of the classifier. The weighting function is introduced to assign different weights to different dimensions of the subspace. We have verified the effectiveness of the proposed methods by conducting experiments on two public gas sensor datasets in comparison with the state-of-the-art DA methods.

In the context of robotic high-precision soldering, we propose an image-based pin alignment control method based on active plastic deformation. The plastic deformation is a well-known failure mechanism in most situations, which includes a phenomenon that the objects do not return original state. Here, in contrast to this convention, we utilize the plastic deformation of the metal pin to do pin alignment for improving the quality of the solder joint. To address this, we embed the springback compensation into the image-based pin alignment controller. Lastly, the proposed strategy is successfully demonstrated and evaluated in a practical modified robotic manipulation system. The result shows that the alignment error is less than 20µm, which is far less than pin alignment without considering plastic deformation and elastic recovery. This work considers active plastic deformation and spontaneous elastic recovery of soft object, which would greatly promote the use of robotics in micromanufacturing and microfabrication in lad and industry, especially for soft objects.

6D pose estimation for objects is an important technology in human-computer interaction. Previous works trained one or more complicated networks to predict 6D poses. Although complex models have nice performance generally, the high storage and computation cost make it difficult to be applied on hardware platforms with limited computing ability such as the low-cost mobile terminal. Hence, how to reduce the complexity of the model while maintaining accuracy remains a challenge. In this paper, we present a lightweight generic architect that processes the color and depth images respectively by employing two efficient backbone networks, then use a fusion network to realize pose regression. Furthermore, an iterative refinement network compressed is implemented by using the Filter Pruning via Geometric Median (FPGM) algorithm to refine the poses while improving real-time performance. Comprehensive experiments conducted on two benchmark datasets, LineMOD and YCB-Video, confirm that the proposed model is more than twice as fast as the state-of-the-art (SOTA) DenseFusion. For main metrics, the BFLOPs (Billion FLoat OPerations) are reduced by 97.0%, and the parameter size declines by 87.4%. The average distance (ADD) for LineMOD increases by 2.6%. The overall performance of the new model is proven outperforming SOTA methods both in efficiency and accuracy.

Understanding the effects of different hip assistance modes is a fundamental step in the process of designing hip assistance devices and controllers that can provide better performance in terms of metabolic cost. We have developed and tested a soft exoskeleton for hip assistance, which includes three assistance modes: hip extension assistance (HEA), hip flexion assistance (HFA), and hip extension and flexion assistance (HEFA). A proportional derivative (PD) iterative learning controller based on the feedforward model was proposed to control the assistive force accurately. The three hip assistance modes were evaluated on seven male subjects walking on a treadmill at a speed of 5 km/h in two scenarios-first with a 15-kg backpack and then without any backpack. The net metabolic costs could be reduced during the loaded condition, compared with those under no exoskeleton condition, by 9.95%, 6.25%, and 15.28% for HEA, HFA, and HEFA, respectively. The reductions were found significant in HEA ( p=0.048) and HEFA ( p=0.005) modes, while the HFA mode ( p=0.202) was not found statistically significant. It indicates that the HEA and HEFA modes with the soft exoskeleton provide more benefit to the net metabolic cost compared with the HFA mode. The net metabolic costs reduced during the unloaded condition were 9.21%, 2.58%, and 13.05% for HEA, HFA, and HEFA, respectively. The improvements in the walking efficiency during both the conditions with the developed soft exoskeleton are demonstrated. Note to Practitioners-This article was motivated by the problem that how to reduce the metabolic cost most appropriately of walking by hip assistance of soft exoskeleton. In this article, we conduct a comparison of three hip assistance modes to discuss the balance of system weight and assistance efficiency. We then propose a PD iterative learning controller based on the feedforward model to track the desired assistive force accurately. Preliminary experiments suggest that the hip extension assistance (HEA) is more suitable than hip flexion assistance (HFA) for hip assistance during single motion assistance. Multiple motion assistance is more beneficial for metabolic cost reduction when the weight of the soft exoskeleton is the same. The experimental tests show that the proposed soft exoskeleton and the control algorithm are effective for walking assistance during the loaded condition. However, the performance of the hip assistance device is tested based on the treadmill walking only. In future research, we will conduct a performance evaluation of the soft exoskeleton on a complex road environment.

Steering microswimmers toward 3-D autonomous manipulation tasks has received extensive attention. Our previous works have accomplished autonomously manipulating microswimmers in the 2-D space. This article aims to extend the 2-D autonomous manipulation to 3-D autonomous manipulation. Specifically, this article addresses the problem of an autonomous system that consists of 3-D path planning and 3-D path following for magnetically driven helical microswimmers. The path-planning algorithm called optimal Bidirectional RRT* is formulated to explore the shortest route in the confined 3-D space. A proxy-based sliding mode control (PSMC) approach is developed to design stable controllers based on the error model in the Serret–Frenet frame. We transport the swimming model trained by a kind of neural network to another new helical microswimmer according to an online updating scheme. The updating scheme can identify and refine compensating angles between the swimming direction of the microswimmer and the magnetic direction in the 3-D space facing the weight disturbances of the swimmer and lateral disturbances. The experiments are conducted to quantitatively validate the 3-D autonomous manipulation system. Experimental results show the effectiveness of path planning and path following with submillimeter accuracy in a 3-D space. Future works will focus on autonomous manipulations in dynamic environments.Note to Practitioners—This article is motivated by the issue of 3-D autonomous manipulation tasks for magnetically driven helical microswimmers. The formulated path planning is responsible for finding the shortest route in the 3-D confined space. The closed-loop controller is charge of steering the helical microswimmers on a reference path based on an online updating model trained by neural networks. It is demonstrated that the helical microswimmer can find the shortest path and follow it in a 3-D space with submillimeter accuracy.

People who suffer from paraplegia completely lose sensory and locomotor functions; there are no known treatment methods for their recovery at this time. Exoskeleton robots have the potential to dramatically improve the locomotor ability of these individuals. Although some exoskeleton robots for paraplegic patients have been commercialized and are able to restore walking motion at present, the pilot must acquire the ability to maintain their balance and shift their weight using forearm crutches, which is very challenging for paraplegics. To make this easier, we propose a new automated intelligent gait planning method that integrates a finite-state machine (FSM) model as an underlying foundation and a gait generation model in addition to the exoskeleton system. The underlying FSM model is defined using an inverted pendulum model and a minimum jerk algorithm. To compare the planning gait, 33 volunteers provide normal walking gaits; there are two more volunteers (paraplegic and nonparaplegic) wearing the Shenzhen Institute of Advanced Technology (SIAT) exoskeleton robot to validate the effects of the proposed gait and offer the groups of surface electromyogram (sEMG) data for analysis. As a result, the input of the proposed gait planning method is simplified to two parameters. The proposed walking gait significantly reduces the arm muscle output. Note to Practitioners-This article was motivated by the problem that the four-degree of freedom (DoF) underactuated paraplegic rehabilitation lower limb exoskeleton robot lacks of the center of gravity (COG) transfer process when coordinating with paraplegia patients during the training process for beginner. The existing approach to deal with this problem generally is to train the pilot for obtaining the COG transfer ability by using crutches. This article suggests a gait planning method for the four-DOF underactuated rehabilitation lower limb exoskeleton robot considering the COG transfer process to make the exoskeleton robot coordinate with a pilot and ensure safety. The gait planning method is based on the inverted pendulum model and simplified to several parameters. By adjusting these parameters, the step length, step height, walking speed, and the shape of gait can be adjusted according to the requirements of the exoskeleton robot and pilot. In this article, we mathematically characterize a gait planning method for the exoskeleton control strategy. Preliminary online experiments suggest that this approach is feasible and can significantly reduce the arm muscle output of pilot. In future research, we will adjust the gait by estimating the velocity of center of mass (COM) of the pilot to make the exoskeleton robot coordinate with pilot actively.

Tactile sensing with spiking neural networks (SNNs) has attracted increasing attention in the past decades. In this article, a novel SNN framework is proposed for the tactile surface roughness categorization task. In contrast to supervised SNN methods such as ReSuMe and Tempotron that require prespecifying target spike trains, the presented method performs the classification through directly comparing the distance between multineuron spike trains. Unlike simple spike train fusion methods using average pairwise spike train distance or pooled spike train distance, the proposed method merges spike trains from different neurons with the multineuron spike train distance, which can capture the complex correlation of multiple spike trains. Specifically, the spike trains are generated via the Izhikevich neurons from tactile signals. The similarity of the multineuron spike trains is computed using the multineuron Victor–Purpura spike train distance, which can be efficiently implemented in an inductive manner. The classification can be performed by incorporating$k$-nearest neighbors and the multineuron spike train distance as a similarity metric. The proposed framework is quite general, i.e., other multineuron spike train distances and spike train kernel-based methods can be readily incorporated. The effectiveness of the proposed method has been demonstrated on a tactile data set by comparing it with various feature- and spike-based methods.Note to Practitioners—In the soft neuromorphic implementation of biomimetic tactile sensing and the development of the tactile sensing capability in neurobotic systems, the processing and analysis of spike-like tactile signals are quite common. Inspired by human tactile perception, this article proposes a novel supervised spiking neural network method for tactile sensing tasks. The traditional methods have to prespecify target spike trains, which is still an open question. In addition, the current ways to fuse spike trains from multiple neurons are far from mature. This article tackles these two problems using spike train similarity comparison with multineuron spike train distance. The direct spike train similarity comparison avoids the need to prespecify target spike trains. The multineuron spike train distance can inherently fuse spike trains from different neurons. It is demonstrated that the proposed method is able to effectively perform classification in a tactile roughness discrimination task.

Deep learning approaches have significantly contributed to recent progress in stereo matching. These deep stereo matching methods are usually based on supervised training, which requires a large amount of high-quality ground-truth depth map annotations that are expensive to collect. Furthermore, only a limited quantity of stereo vision training data are currently available, obtained either by active sensors (Lidar and ToF cameras) or through computer graphics simulations and not meeting requirements for deep supervised training. Here, we propose a novel deep stereo approach called the "self-supervised multiscale adversarial regression network (SMAR-Net)," which relaxes the need for ground-truth depth maps for training. Specifically, we design a two-stage network. The first stage is a disparity regressor, in which a regression network estimates disparity values from stacked stereo image pairs. Stereo image stacking method is a novel contribution as it not only contains the spatial appearances of stereo images but also implies matching correspondences with different disparity values. In the second stage, a synthetic left image is generated based on the left-right consistency assumption. Our network is trained by minimizing a hybrid loss function composed of a content loss and an adversarial loss. The content loss minimizes the average warping error between the synthetic images and the real ones. In contrast to the generative adversarial loss, our proposed adversarial loss penalizes mismatches using multiscale features. This constrains the synthetic image and real image as being pixelwise identical instead of just belonging to the same distribution. Furthermore, the combined utilization of multiscale feature extraction in both the content loss and adversarial loss further improves the adaptability of SMAR-Net in ill-posed regions. Experiments on multiple benchmark datasets show that SMAR-Net outperforms the current state-of-the-art self-supervised methods and achieves comparable outcomes to supervised methods. The source code can be accessed at: https://github.com/Dawnstar8411/SMAR-Net.

Lower-limb exoskeleton robots can effectively help patients with lower-limb disabilities caused by stroke or spinal cord injury to walk again. However, when faced with complex ground surfaces, patients find it difficult to quickly and accurately transmit the motion intention to the robot, resulting in falls or errors. In this article, we develop the vision-assisted autonomous lower-limb exoskeleton robot (VALOR). Based on the principles of human visual feedback and motion decision-making, a vision-assisted autonomous gait pattern planning method is proposed to improve the adaptability of the robot to the environment. The robot obtains environmental information via an RGB-D camera and extracts the ground object features that might affect gait. Then, the robot makes an autonomous decision according to the environmental features, robot state, and safety constraints. Lastly, a suitable step length and height are given to the parameterized gait pattern planning model of robot to assist with walking. The feasibility of the proposed method is verified on VALOR in a controlled indoor environment with limited obstacles, and our results demonstrate that the method can significantly improve robot adaptability to complex walking environments.

The important purpose of soft exosuit is to decrease the energy consumption of users by providing assistance. If there is something wrong with the judgment of the human gait during the assisting process, the assisting effect will be poor and even people's walking will be affected. A novel method is designed to predict human gait information in the paper, which utilizes the curve shape and mathematical characteristics of the Rayleigh oscillator equation in limit cycle to fit gait information. Only 1/4 gait cycle data is needed to input into the trained neural network to output an Rayleigh oscillator equation that can better predict the remaining gait cycle information. Data of four subjects on different terrains which include flat ground and upstairs are collected. Experiment results showed that the trained ANN model costs about 0.006s in CPU, it has a prediction speed similar to traditional prediction methods. Simultaneously the Rayleigh oscillator equation has good performance in predicting gait information, and it can show a higher stability and accuracy compared with traditional gait prediction methods.

Microrobots capable of performing minimally invasive surgery, targeted drug delivery, and manipulation of micro-objects have shown great potentials in multiple application areas. However, microrobots assembled by nanoparticles with stable configuration and good performance on closed-loop control are still to be further studied. In this paper, ferromagnetic nanoparticles are used as experimental materials, rather than paramagnetic nanoparticles that require complex synthesis processes, and the advantage is demonstrated, for example, high pattern stability. The locomotion velocity as a function of the magnetic field frequency is modeled, analyzed and verified by experiment. Moreover, a path following experiment based on the arbitrary planar path following algorithm is performed. The nanoparticle microrobots are of great significance for biomedical applications. In the future works, experiments in bio-fluids, multimodal locomotion, and targeted drug delivery tasks will be investigated.

Hand gesture recognition are widely applied in game industry and rehabilitation engineering. Some recognition methods based on physiological data are attracting attention. In this paper, a novel method for dynamic hand gesture recognition based on sounds of wrist tendon is proposed. Bone conduction sensors which automatically filter the ambient noise is utilized to collect the wrist tendon sounds of three types of dynamic hand gestures. Continuous Wavelet Transform(CWT) is utilized extract the features of sounds and Support Vector Machine(SVM) is utilized to train the classifier model. Five healthy subjects participated the experiment and the average test accuracy of three classes achieves 92.33%. The proposed method provides an effective perspective for hand gesture recognition.

The advent of myoelectric control schemes provides promising chances for locomotion empowerment and restoration of those with disabilities. Despite substantial efforts have been made into advancing sEMG-based motion recognition, it may be a little tricky to determine appropriate muscles and features for people with muscle disorders or different muscle use preferences. To mitigate it, an advantageous sEMG channel and feature selection method based on ReliefF algorithm was proposed. Related experiments were conducted on a eight able-bodied subject database to showcase the feasibility and efficiency of the proposed approach, that is, considerably high classification performance was maintained with the original feature set reduced by more than half. Ulteriorly, we also investigated the influences of different number of neighbors or features on classification accuracy for ascertaining the optimal values. The strengths of our proposed method lie in not only customizing channel and feature selection for individual users, but also offering preliminary insight for a general mapping mechanism between human muscles and corresponding motions.

In this paper, we propose a novel method for multimodal material classification based on ensemble learning and optimized features. The proposed method consists of three key steps. Firstly, we extract a set of features for each modality. Compared to existing methods, the extracted features are relatively simple but more effective when they are incorporated into the classifiers. Then, the feature selection algorithms, including Multi-Cluster Feature Selection (MCFS) and Laplacian Score (LS) are employed to reduce the feature dimension due to the curse of dimensionality. Finally, an ensemble learning method is proposed to integrate the merits of different feature selection methods. The effectiveness of the proposed method is demonstrated on the LMT-108 surface material dataset which includes multiple modalities such as sound, acceleration, and image. The experimental results have shown that our approach performs better than the competing methods.

Soft Exosuit is a kind of Lower-limb wearable robots to augment and assist the wearer's performance. The wearer need different assistance modes to reduce the metabolic rate when walking on different terrains. Therefore, assistance modes need to be selected according to different terrains for the wearer of soft Exosuit. Recently, our team has designed a stable terrain classification and recognition system (TCRS) for the soft Exosuit to discriminate terrains and estimating environmental features. Through this system, soft Exosuit can perceive the environment to auxiliary control of the locomotion modes. A depth sensor with an inertial measurement unit(IMU) can acquire to stabilize the point cloud of environments. Subsequently, the 2D point cloud is extracted from the origin 3D point cloud. Then, they are classified to estimate terrain environmental features, including the incline angle of the slope, the width, and the stairs' height. Finally, the TCRS was evaluated by classifying and recognizing five basic terrains in three different experimental scenarios while the subject was wearing the soft Exosuit with the TCRS module. The results show that the terrain classification accuracy rate reaches 97.74 %, and the environmental features estimation error is less than 15 %. The promising results indicate the robustness and the potential application of the presented TCRS to provide proper auxiliary force to reduce the metabolic rate of wearers on different terrains.

The problem of human-exoskeleton interaction with uncertain dynamical parameters remains an open-ended research area. It requires an elaborate control strategy design of the exoskeleton to accommodate complex and unpredictable human body movements. In this paper, we proposed a novel control approach for the lower limb exoskeleton to realize its task of assisting the human operator walking. The main challenge of this study was to determine the human lower extremity dynamics, such as the joint torque. For this purpose, we developed a neural network-based torque estimation method. It can predict the joint torques of humans with surface electromyogram signals (sEMG). Then an radial basis function neural network (RBF NN) enhanced adaptive impedance controller is employed to ensure exoskeleton track desired motion trajectory of a human operator. Algorithm performance is evaluated with two healthy subjects and the rehabilitation lower-limb exoskeleton developed by Shenzhen Institutes of Advanced Technology (SIAT).

High flexible and high precise manipulation is one of the most critical technique for complex microsystem’s measurement, manufacture, and assembly. Although recent advances in microrobotics have successfully realized the automatic manipulation and positioning of tiny objects, their flexible manipulation in 3-D free space remains a challenge, such as the wide-angle rotation manipulation of microsize sphere, due to the complicate surface forces. Herein, this article proposed a feedforward model and realized the precise rotation for microsized sphere by two cooperative flexible micropipettes. Firstly, a microrobotic manipulation system with six degrees-of-freedom (DOFs) was developed and integrated with the microscope. Then, a feedforward compensation control strategy involving dual rotation was proposed for the precise manipulation of microsized sphere (${\sim }90~\mu \text {m}$) based on the analysis of contact forces. As a result, the rotation of the microsized sphere in two different planes was realized and the microsized sphere release procedure was also accomplished after rotation. Compared with existing techniques only allowing limited amplitudes rotation, this article realizes wide-angle rotation manipulation of microsized sphere in 3-D free space. This research opens new prospects for the microsized object accurate manipulation, which is expected to give a long-term impact for complex microsystem’s manufacture and assembly.Note to Practitioners—This article is motivated by the problem of the flexible manipulation of tiny object in 3-D space. The proposed nanorobotic manipulation system, two micropipettes and feedforward compensation model control strategy could realize the precise translational and rotational manipulation of microbeads in 3-D space, which offers obvious advantages of existing techniques. The proposed system and method could be a general solution for precise and flexible micromanipulation. Thus, it could find wide applications ranging from fundamental research to industrial applications, such as biological cell positioning, characterization of a particular micro/nanoregion, microassembly, and manufacturing.

Controlling magnetic microswimmers toward 3-D manipulation tasks has received considerable attention. Although related studies on manipulating helical microswimmers have been developed, stable closed-loop controls and accuracy swimming models should be still investigated. This article addresses the problem of 3-D path following for magnetically driven helical microswimmers with an adaptive-compensation scheme. The orientation-compensation model in the global coordinate frame is learned by radial basis function (RBF) networks trained with backpropagation algorithms, which is used to express the motion of the helical microswimmer in the presence of the weight of the swimmer and lateral disturbances from the boundary effects. A proxy-based sliding-mode control (PSMC) approach is developed to design stable controllers based on the kinematic error model. The effects of variable parameters and boundary effects are also considered. Experimental results including different paths in 3-D space validated the path following with submillimeter accuracy using the helical microswimmer. Note to Practitioners-This article is motivated by the issue of the following predefined paths for magnetically driven helical microswimmers in 3-D space. The proposed closed-loop controller employs the error model in 3-D space to formulate the control law according to an orientation-compensation model learned by neural networks. It is demonstrated that the helical microswimmer is able to follow different paths in 3-D space with submillimeter accuracy using the proposed control scheme.

Magnetically actuated microswimmers have attracted researchers to investigate their swimming characteristics and controlled actuation. Although plenty of studies on actuating helical microswimmers have been carried out, robust closed-loop controls should be still explored for practical applications. In this paper, we proposed a data-driven model-free method using Image-Based Visual Servoing (IBVS), which uses features directly extracted in the image space as feedbacks. The IBVS method can eliminate camera calibration errors. We have demonstrated with experiments that the proposed IBVS method can enable velocity-independent path following of an arbitrarily given path on the plane, which permits a better experience of user interaction. The proposed control method is successfully applied to obstacle avoidance tasks and has the potential for the application in complex circumstances. This approach is promising for biomedical applications.

In this paper, controlling miniature swimmers via electromagnetic actuation has received considerable attention due to their minor invasive trait in various biomedical applications and ease of passing through the complex environments. Studying the navigation and control system is an essential step towards such applications. Currently, navigation and control for magnetically driven miniature swimmers are still challenging research issues. This paper aims to formulate a navigation and control system of magnetically driven helical miniature swimmers. First, a global planning algorithm named informed optimal random exploring tree (Informed RRT*) is applied to compute the feasible path in cluttered environments. Second, a closed-loop control algorithm is presented to follow various of reference paths using visual feedback on a planar substrate. In particular, a single-hidden layer feedforward neural networks is employed to approximate the mapping relationship between the magnetic self-rotation direction and the actual moving direction of helical miniature swimmers. The neural network is first implemented to control the magnetically driven miniature swimmers in this paper. Experiments are conducted to verify the ability of navigation and visual feedback control tasks.

Building multiple hash tables serves as a very successful technique for gigantic data indexing, which can simultaneously guarantee both the search accuracy and efficiency. However, most of existing multitable indexing solutions, without informative hash codes and strong table complementarity, largely suffer from the table redundancy. To address the problem, we propose a complementary binary quantization (CBQ) method for jointly learning multiple tables and the corresponding informative hash functions in a centralized way. Based on CBQ, we further design a distributed learning algorithm (D-CBQ) to accelerate the training over the large-scale distributed data set. The proposed (D-)CBQ exploits the power of prototype-based incomplete binary coding to well align the data distributions in the original space and the Hamming space and further utilizes the nature of multi-index search to jointly reduce the quantization loss. (D-)CBQ possesses several attractive properties, including the extensibility for generating long hash codes in the product space and the scalability with linear training time. Extensive experiments on two popular large-scale tasks, including the Euclidean and semantic nearest neighbor search, demonstrate that the proposed (D-)CBQ enjoys efficient computation, informative binary quantization, and strong table complementarity, which together help significantly outperform the state of the arts, with up to 57.76% performance gains relatively.

Today, exoskeletons are widely applied to provide walking assistance for patients with lower limb motor incapacity. Most existing exoskeletons are under-actuated, resulting in a series of problems, e.g., interference and unnatural gait during walking. In this study, we propose a novel intelligent autonomous lower extremity exoskeleton (Auto-LEE), aiming at improving the user experience of wearable walking aids and extending their application range. Unlike traditional exoskeletons, Auto-LEE has 10 degrees of freedom, and all the joints are actuated independently by direct current motors, which allows the robot to maintain balance in aiding walking without extra support. The new exoskeleton is designed and developed with a modular structure concept and multi-modal human-robot interfaces are considered in the control system. To validate the ability of self-balancing bipedal walking, three general algorithms for generating walking patterns are researched, and a preliminary experiment is implemented.

This paper has developed a coordination control method for a dual-arm exoskeleton robot based on human impedance transfer skills, where the left (master) robot arm extracts the human limb impedance stiffness and position profiles, and then transfers the information to the right (slave) arm of the exoskeleton. A computationally efficient model of the arm endpoint stiffness behavior is developed and a co-contraction index is defined using muscular activities of a dominant antagonistic muscle pair. A reference command consisting of the stiffness and position profiles of the operator is computed and realized by one robot in real-time. Considering the dynamics uncertainties of the robotic exoskeleton, an adaptive-robust impedance controller in task space is proposed to drive the slave arm tracking the desired trajectories with convergent errors. To verify the robustness of the developed approach, a study of combining adaptive control and human impedance transfer control under the presence of unknown interactive forces is conducted. The experimental results of this paper suggest that the proposed control method enables the subjects to execute a coordination control task on a dual-arm exoskeleton robot by transferring the stiffness from the human arm to the slave robot arm, which turns out to be effective.

The strategies to adopt brain-computer interfaces (BCIs) to drive assisted devices are proved to be feasible in many studies. Although several studies focus on detecting the initiation of normal walking by BCIs, few consider how to distinguish the change of gait pattern for different terrains in a realistic walking situation. Therefore, this paper proposes an innovative experimental paradigm for robust control of exoskeleton based on a BCI system. Several pseudo online trials are conducted to prove the feasibility of the proposed paradigm. Firstly, a labeled windows generator (LWG) is built to produce electroencephalogram (EEG) windows based on acquired gait data and EEG data. Then the common spatial pattern (CSP) is used to extract features from the labeled EEG windows. Finally, a support vector machine (SVM) classifier is trained to predict the intention of the subject. The experimental results corroborate the feasibility of obtaining the intention of stepping over obstacles from normal walking through the proposed BCI-controlled exoskeleton system.

Though the rapidly development in exoskeleton robot area, there are still some challenges, such as the safe and friendly human-machine interaction. In this paper, we proposed a novel knee exoskeleton driven by Series Elastic Actuator (SEA) which is a compliant actuator has the characteristic low output impedance, low friction, high quality force control and back-drivability. As for walking assistance, the force control is vital. Here, we mainly apply two control method to achieve the robust force control, one is the feedback control with enchaned distrubance observer, another is the Integral Sliding Mode Control (ISM). We test both the stability and force tracking performance of the two controler. The result showed that both algorithms can achieve the satisfactory performance and the ISM algorithm obtained the slightly better result.

Untethered and wirelessly-controlled microrobots have many applications in the field of biomedicine. Therefore, many laboratories and scientists have invested more scientific research into magnetic microrobots which can make more contributions to medical care. Many magnetic field devices and microrobots are manufactured. In the development of micro-robots, helical microrobots have been well developed. Rigid-body robots account for the majority of these, but they may cause damage to human organs during treatment. However, soft and deformable robots can relieve more medical restrictions. In general, helical microrobots are driven by uniform fields which have their own limitations while the gradient magnetic field can relieve more restrictions and have more functions. This paper presents a flexible deformable helical swimmer controlled in a rotating gradient magnetic field. Helical swimmers are covered with magnetic nano-particles and the helical structure is derived from the inner fiber structure of the lotus root. The soft helical swimmers are controlled to swim several special trajectories in the rotating gradient magnetic field and we analyze the frequency and other factors for velocity or other effects.

The development of sharing technology makes it possible for expensive lower limb exoskeleton robots to be extensively employed. However, due to the uniqueness of gait pattern, it is challenging for lower limb exoskeleton robot to adapt to different wearers' gait patterns. Studies have shown that the gait pattern is affected by many physical factors. This paper proposes an individualized gait pattern generation (IGPG) method for sharing lower limb exoskeleton (SLEX) robot. First, the gait sequences are parameterized to extract gait features. Then, the Gaussian process regression with automatic relevance determination is used to establish the mapping relationships between the body parameters and the gait features, and the weights of each body parameters on gait pattern are also given. The gait features of an unknown subject can be predicted based on the training set. Finally, the individualized gait pattern is reconstructed by autoencoder neural network and scaling process based on predicted gait features. The experimental results show that the gait pattern predicted by IGPG is very similar to the subject's actual trajectory and has been successfully applied on the SLEX robot. With the help of sharing technology, the training set will be increased, and the prediction accuracy of individualized gait pattern will also be improved.

In this paper, a method is proposed to search for spatio-temporal path for action localization in unconstrained videos. We mainly focus on two requirements, i.e., accurate human extraction and speeding generation of action proposal. The approach first generates human proposals at the frame level, then scores them based on two complementary parts, i.e., posteriori probability evaluated via a fine-tuned Faster-RCNN and template-matching similarity based on the spatiotemporal continuity. Finally, the generation of action proposal is formulated as a Max-Path discovery problem, coupled with dynamic programming to find an optimal path with maximum score. Experiments on UCF-Sports are performed to verify that the proposed method can achieve fast high-quality action proposal and link the missed-detection proposals in successive frames together to form a complete action.

Magnetically actuated microswimmers have shown great potentials in multiple application scenarios, attracting researchers to investigate their swimming characteristics and controlling methods. However, among those studies, only a small number of closed-loop control schemes have been applied, which is crucial for the accuracy and repeatability in applications. In this paper, we proposed an Image-Based Visual Servoing(IBVS) method for arbitrary planar path following using features directly presented in image space as feedbacks. During experiments we found that IBVS guarantees convergence while not requiring camera calibrations. Furthermore, the proposed path following method is intuitive and flexible, and provides great potentials in various applications.

Microrobots show great potential to realize a wide range of medical applications especially in a hard-to-reach region within human body. Among a diversity of medical applications, the microrobots are expected to efficiently perform tasks within the human body, but the existing electromagnetic coil systems such as Helmholtz coil, Maxwell coil, etc. do not compromise on the insertion of such a large volume of human parts. We aim to magnetically manipulate diverse microrobots in medical applications with respect to surgical or biopsy tasks within insiders of the human head such as to perform brachytherapy with the helical microswimmer, etc. For this reason, we proposed a novel, a magnetically actuated system called the HyBrid system with the conceptual design in capability of generating both uniform and gradient magnetic fields. The system consists of three different coil setups orthogonally arranged over a large workspace which is available to let the human head inserted into. The simulation results of the magnetic field generation expressed that its performance is suitable and feasible towards magnetically manipulation within the human head, and more efficient than 3D-Helmholtz coil system under the same conditions.

This paper propose a practical and effective method for identifying the complete inertia parameters of a space robotic system. The key is the following steps: equivalent single-body identification, and equivalent two-body identification. For the former, all joints are locked into a normal configuration, and the thrusters are used for orbital maneuvering. The object function is defined from the view of acceleration and velocity. For the latter, only one joint is unlocked and driven to move along the exciting trajectory under the free-floating mode. The linear and angular momentum equations are used to define the object function. Hence, the parameter identification problem is transformed into a non-linear optimization problem, and the PSO algorithm is used to determine the optimal parameters. By sequentially unlocking the 1st to nth joint, the mass properties of body 0 to n are completely identified. The proposed method is general and can be easily extended for many cases.

In this paper, we propose a new approach for anomaly detection in video surveillance. This approach is based on a nonparametric Bayesian regression model built upon Gaussian process priors. It establishes a set of basic vectors describing motion patterns from low-level features via online clustering, and then constructs a Gaussian process regression model to approximate the distribution of motion patterns in kernel space. We analyze different anomaly measure criterions derived from Gaussian process regression model and compare their performances. To reduce false detections caused by crowd occlusion, we utilize supplement information from previous frames to assist in anomaly detection for current frame. In addition, we address the problem of hyperparameter tuning and discuss the method of efficient calculation to reduce computation overhead. The approach is verified on published anomaly detection datasets and compared with other existing methods. The experiment results demonstrate that it can detect various anomalies efficiently and accurately.

In this paper, we present a novel approach for video anomaly detection in crowded scenes. The proposed approach detects anomalies based on the contextual information analysis within spatio-temporal video volume. Around each pixel, spatio-temporal volumes are built and clustered to construct the activity pattern codebook. Then, the composition information of the volumes within a large spatiotemporal window is described via a dictionary learned by sparse representation. Furthermore, multi-scale analysis is employed to adapt the size change of abnormal events. Finally, the sparse reconstruction cost is designed to evaluate the abnormal level of an input motion pattern. We demonstrate the efficiency of the proposed method on the existing public available anomaly-detection datasets and the performance comparasion with three existing methods validates that the proposed method detects anomalies more accurately.

In this paper, an indoor navigation algorithm is proposed for the purpose of robot autonomous path planning. Due to the complex situation in indoor environments, it can cause a serious trouble for robot to identify the route during patrolling, especially for corner and door detection, which is the key step for intelligent navigation. To solve this problem, a kinect sensor is used for the door detection and corner location via depth images. The continuously varied ratios and depth difference in the images have been analyzed for the corner and door identification. Furthermore, the precise position of the doors and corners can be localized via the 3-dimensional characteristics of the depth images. Experiments in different scenarios have been performed to verify the efficacy of the algorithm for robot indoor autonomous navigation.

We develop a localization method to enable a team of mobile robots to search for multiple unknown transient radio sources. Because of signal source anonymity, short transmission durations, and dynamic transmission patterns, robots cannot treat the radio sources as continuous radio beacons. Moreover, robots do not know the source transmission power and have limited sensing ranges. To cope with these challenges, we pair up robots and develop a cooperative sensing model using signal strength ratios from the paired robots. We formally prove that the joint conditional posterior probability of source locations for the m-robot team can be obtained by combining the pairwise joint posterior probabilities, which can be derived from signal strength ratios. Moreover, we propose a pairwise ridge walking algorithm (PRWA) to coordinate the robot pairs based on the clustering of high-probability regions and the minimization of local Shannon entropy. We have implemented and validated the algorithm under both the hardware-driven simulation and physical experiments. Experimental results show that the PRWA-based localization scheme consistently outperforms the other four heuristics.

This paper focuses mainly on adaptive dictionary updating and abnormality detection via weighted space coding in video surveillance. Generally, abnormality analysis conducted on a large amount of video data is very complicated, time-consuming and time-variant. However, our dictionary is very efficient at following up on shifted contents in video and abandoning old inactive information in time. The adaptability characteristic also helps reduce the dictionary's size to a small scale, since it only needs to keep recent or active information. We also introduce a simple, but effective, judgement criterion for abnormal detection based on sparse coding over weighted bases. Because of the condensed dictionary and the simplified judgment criterion, our algorithm performs online learning and online detection with a high speed and a high accuracy in various scenes.

In this paper, we present a novel approach for video anomaly detection in crowded and complicated scenes. The proposed approach detects anomalies based on a hierarchical activity pattern discovery framework comprehensively considering both global and local spatio-temporal contexts. The discovery is a coarse-to-fine learning process with unsupervised ways for automatically constructing normal activity patterns at different levels. An unified anomaly energy function is designed based on these discovered activity patterns to identify the abnormal level of an input motion pattern. We demonstrate the efficiency of the proposed method on the UCSD anomaly detection datasets (Ped1 and Ped2) and compare the performance with existing work.

This paper presents a newly-designed robot named “Rubbot” dedicated to climbing on soft flexible clothes. Equipped with novel grippers which grip and rub on clothes, Rubbot is able to climb on flexible clothes and control how much fabric to grasp by feedback from infrared sensor. Rubbot also has a frame which has three passive folders which adjust the climbing posture of Rubbot. This not only makes Rubbot quite functional with clothes of different thicknesses and curved surfaces, but also makes Rubbot's motion more flexible. A theory of the deformation of cloth is then presented based on an analysis of creases created while Rubbot is climbing, this leads to a more reliable method to climb flexible surfaces. Finally experiments have verified that Rubbot is effective on flexible surfaces, as it can climb on 95% of the surfaces human clothes and still perform well on non-rigidly backed cloth.

Hand posture is a natural and effective human robot interaction way. In this paper, an user-independent hand posture recognition system using depth and color images captured from an RGB-D camera is presented. To recognize hand posture against complicated background conditions, we propose a novel method for automatic and accurate hand posture segmentation which detects the hand with Chamfer matching, tracks the hand with Kalman filter and segments the hand with region growing algorithm only in the depth space. A new hand posture descriptor invariant to scale, shift and in-plane rotation is constructed with the combination of local contour Fourier descriptor and global Bag-of-Features (BoF) descriptor based on Scale Invariance Feature Transform (SIFT). The sparse representation-based classification (SRC) is applied to perform the hand posture recognition task in the system. Experiments with a self-built large scale hand posture database collected online show the robustness and effectiveness of the proposed system.

This paper presents a novel climbing robot called Clothbot which has high maneuverability on flexible clothes. It has a novel gripper consisting of two parallel wheels that can grip continuously and stably on various kinds of clothes. Clothbot also has an omni-directional tail of two DOFs so that it can change its center of gravity to control the moving direction on complex and undeterminate clothes. Consequently, Clothbot is able to access most positions of the clothes by moving straight and turning around with only four motors. It is compact, small and light-weighted but has a load capacity six times its own weight. A series of experiments validate its high performance on flexible clothes.

This paper proposes a novel path planning method for a robot to climb on the deformable clothes surface. Based on the deformable characteristic of the clothes, the tension force of clothes is analyzed and the model of tension degree is established. A clothes climbing robot called Clothbot is composed of a two-wheeled gripper and a 2 Degrees of Freedom (DOF) tail. Based on the locomotion of this robot, the weights of tension degree and the locomotion characteristic are added into the A* algorithm. Combined with the two weights applied, the optimal path to the target for the Clothbot is obtained. The Clothbot has been developed to evaluate the algorithm. The simulation and the experiments have verified the feasibility of this method. In addition, The error state of the movement of the robot which is called side tumbling has been corrected by the motion of the 2-DOF tail.

This paper proposed a novel Tri-star wheeled robot called “Tribot”, which targets on high obstacle performance in unstructured environments, especially at the performance for climbing vertical obstacles. Tribot equips with six Tri-star wheels and each wheel can be driven independently. The chassis of the Tribot is divided into two parts which are connected by an articulated mechanism, making the Tribot has a remarkable obstacle performance to adapt changing environments mechanically, without any interpolate complex control. Numerous experiments have been conducted for vertical obstacle performance tests. Although the diameter of the wheel of the Tribot is only 220 mm, the robot can climb over vertical obstacle of 450 mm high, twice more of the wheel diameter. All results show that Tribot has excellent vertical climbing performance in unstructured environments.

In this paper, we study the stability region for a set of intelligent controllers developed by learning human expert control skills using support vector machines (SVMs). Based on the discrete-time system Lyapunov theory, a Chebychev points based estimation approach is proposed to evaluate the stability region, a key property of this set of SVM-based human learning controllers. One of such learning controllers has been implemented in vertical balance control of a dynamically stable, statically unstable single wheel mobile robot - Gyrover. The experimental results validate the proposed scheme for estimation of the stability region.

Gender is an important cue in social activities. In this correspondence, we present a study and analysis of gender classification based on human gait. Psychological experiments were carried out. These experiments showed that humans can recognize gender based on gait information, and that contributions of different body components vary. The prior knowledge extracted from the psychological experiments can be combined with an automatic method to further improve classification accuracy. The proposed method which combines human knowledge achieves higher performance than some other methods, and is even more accurate than human observers. We also present a numerical analysis of the contributions of different human components, which shows that head and hair, back, chest and thigh are more discriminative than other components. We also did challenging cross-race experiments that used Asian gait data to classify the gender of Europeans, and vice versa. Encouraging results were obtained. All the above prove that gait-based gender classification is feasible in controlled environments. In real applications, it still suffers from many difficulties, such as view variation, clothing and shoes changes, or carrying objects. We analyze the difficulties and suggest some possible solutions.

In this paper, we present an innovative design of a solder paste inspection device which can be practically integrated into existing solder paste printing machines. Since solder paste inspection systems usually occupy a large space in vertical direction, we designed a mirror box that can re-direct the transmission of fringe pattern. In this way, a new parallel solder paste inspection device with a significant reduction in the vertical constraint is developed. We also developed a hybrid weighting algorithm that applied the distance and fringe contrast to acquire the height of solder pastes. Furthermore, we developed an algorithm that generates the 2-D image from the fringe pattern images during the 4-steps algorithm. It gives benefit (time for solder paste inspection) to traditional approach that uses some special lighting systems to create the 2-D image. Experimental results show our device can inspect the 20mm times 20mm PCB area within 2 seconds and the maximum standard deviation for the average height is 3 mum.

This paper introduces a real-time video surveillance system which detects human abnormal behaviors. We present two approaches to such a problem. The first one employs principal component analysis for feature selection and support vector machine for classification of human behaviors. The proposed feature selection method is based on the border information of four consecutive blobs. The second approach computes optical flow to obtain the velocity of each pixel for determining whether a human behavior is normal or not. Both algorithms are successfully implemented in crowded environments for detecting the human abnormal behaviors, such as (1) running people in a crowded environment, (2) bending down movement while most are walking or standing, (3) a person carrying a long bar and (4) a person waving hand in the crowd. Experimental results demonstrate the two methods proposed are robust and efficient in detecting human abnormal behaviors.