Accurate vessel segmentation in X-ray angiograms is crucial for numerous clinical applications. However, the scarcity of annotated data presents a significant challenge, which has driven the adoption of self-supervised learning (SSL) methods such as masked image modeling (MIM) to leverage large-scale unlabeled data for learning transferable representations. Unfortunately, conventional MIM often fails to capture vascular anatomy because of the severe class imbalance between vessel and background pixels, leading to weak vascular representations. To address this, we introduce Vascular anatomy-aware Masked Image Modeling (VasoMIM), a novel MIM framework tailored for X-ray angiograms that explicitly integrates anatomical knowledge into the pre-training process. Specifically, it comprises two complementary components: anatomy-guided masking strategy and anatomical consistency loss. The former preferentially masks vessel-containing patches to focus the model on reconstructing vessel-relevant regions. The latter enforces consistency in vascular semantics between the original and reconstructed images, thereby improving the discriminability of vascular representations. Empirically, VasoMIM achieves state-of-the-art performance across three datasets. These findings highlight its potential to facilitate X-ray angiogram analysis.

Fluid–structure interaction (FSI) systems involve distinct physical domains, fluid and solid, governed by different partial differential equations and coupled at a dynamic interface. While learning-based solvers offer a promising alternative to costly numerical simulations, existing methods struggle to capture the heterogeneous dynamics of FSI within a unified framework. This challenge is further exacerbated by inconsistencies in response across domains due to interface coupling and by disparities in learning difficulty across fluid and solid regions, leading to instability during prediction. To address these challenges, we propose the Heterogeneous Graph Attention Solver (HGATSolver). HGATSolver encodes the system as a heterogeneous graph, embedding physical structure directly into the model via distinct node and edge types for fluid, solid, and interface regions. This enables specialized message-passing mechanisms tailored to each physical domain. To stabilize explicit time stepping, we introduce a novel physics-conditioned gating mechanism that serves as a learnable, adaptive relaxation factor. Furthermore, an Inter-domain Gradient-Balancing Loss dynamically balances the optimization objectives across domains based on predictive uncertainty. Extensive experiments on two constructed FSI benchmarks and a public dataset demonstrate that HGATSolver achieves state-of-the-art performance, establishing an effective framework for surrogate modeling of coupled multi-physics systems.

Dexterous manipulation, which refers to the ability of a robotic hand or multi-fingered end-effector to skillfully control, reorient, and manipulate objects through precise, coordinated finger movements and adaptive force modulation, enables complex interactions similar to human hand dexterity. With recent advances in robotics and machine learning, there is a growing demand for these systems to operate in complex and unstructured environments. Traditional model-based approaches struggle to generalize across tasks and object variations due to the high dimensionality and complex contact dynamics of dexterous manipulation. Although model-free methods such as reinforcement learning (RL) show promise, they require extensive training, large-scale interaction data, and carefully designed rewards for stability and effectiveness. Imitation learning (IL) offers an alternative by allowing robots to acquire dexterous manipulation skills directly from expert demonstrations, capturing fine-grained coordination and contact dynamics while bypassing the need for explicit modeling and large-scale trial-and-error. This survey provides an overview of dexterous manipulation methods based on imitation learning, details recent advances, and addresses key challenges in the field. Additionally, it explores potential research directions to enhance IL-driven dexterous manipulation. Our goal is to offer researchers and practitioners a comprehensive introduction to this rapidly evolving domain.

Lower limb exoskeletons have been used in clinic to alleviate the drop foot symptom, where the joint torques for exoskeleton control significantly affect the systematic performance. However, how to design suitable assistive torques have not been well addressed. In this study, methods for joint torque generation and real-time regulation have been proposed. Firstly, a collaborative optimization method was proposed to generate torque curves. By optimizing simultaneously the muscle activations of the designed patient model and output torques of the exoskeleton model, the patient model can walk naturally with assistance. The optimized torques were used as the initial curve for further optimization during real implements. Secondly, comprehensive gait symmetry indices were designed, and a human-in-the-loop optimization algorithm was developed to online regulate the torque curve, by which individual assistance can be realized and optimized torque curves can be obtained rapidly. Thirdly, simulation and actual experiments were implemented utilizing an ankle exoskeleton. Seven hemiplegic patients were recruited in the actual experiment to sequentially execute walking in no exoskeleton, default torque, optimized torque, and zero torque modes. Experiment results showed that, gait symmetry and muscle activations can be significantly improved using optimized torque curves, and more symmetrical and coordinated walking can be achieved in five minutes.

Accurate segmentation of multi-branched blood vessels from Digital Subtraction Angiography (DSA) images is essential to improve efficiency and safety of vascular interventional procedures. However, the high-speed flow of contrast agents may lead to incomplete visualization of multiple blood vessel branches and unclear boundary contours, resulting in the so-called partial labeling issue. This significantly undermines the network’s ability to extract and understand the features of multi-branched vascular structures with uncertain region, thereby severely impairing the accuracy of the recognition results. In this paper, we introduce a novel pseudo-label guided multi-task learning strategy, capable of effectively learning feature representation completion under partial label supervision. Specifically, a pretext task branch that generates boundary pseudo-label signals extracts absence structural information and transfers it to the target task for multi-branch vascular segmentation. To achieve more precise semantic-supplementing between tasks, an affinity-based criss-cross feature propagation (CCFP) module is designed to dynamically fill semantic and structural voids caused by missing categories. Furthermore, to mitigate performance degradation caused by unreliable pseudo-labels, a unique loss function is proposed to constrain redundant information at both the pixel and structural levels. We validate our approach through the creation of an in-house DSA dataset composed of six sub-datasets, each containing different vascular branches. Extensive experimental results demonstrate that our method not only addresses the challenge of partial labeling but also strikes a balance between pixel-wise accuracy and the preservation of structural integrity, offering potential value in the field of clinical applications.

Offline preference-based reinforcement learning (PbRL) offers an effective approach to addressing the challenges of designing rewards and mitigating the high costs associated with online interaction. However, since labeling preference needs real-time human feedback, acquiring sufficient preference labels is challenging. To solve this, this article proposes an offline PbRL with a high sample efficiency (LEASE) algorithm, where a learned transition model is leveraged to generate unlabeled preference data. Considering the pretrained reward model may generate incorrect labels for unlabeled data, we design an uncertainty-aware mechanism to ensure the performance of the reward model, where only high-confidence and low-variance data are selected. Moreover, the generalization bound of the reward model is provided to analyze the factors influencing reward accuracy, and the policy learned byLEASEhas a theoretical improvement guarantee. The above developed theory is based on a state-action pair, which can be easily combined with other offline algorithms. The experimental results show thatLEASEcan achieve comparable performance to the baseline under fewer preference data without online interaction.

Biomedical multimodal large language models (MLLMs) outperform general-domain models on complex biomedical images, yet they still face significant modality disparities. Current methods mainly fine-tune general MLLMs with specialized data, lacking efficient architectural innovation. Tailored frameworks often underperform across multiple modalities, reducing visual comprehension and relying on unsustainable data scaling. To address this, we propose Multi-level Multi-branch Collaborative LLaVA (MMC-LLaVA), a new framework that enhances multi-modal visual information capture in biomedical images. Based on LLaVA-Med, MMC-LLaVA adds two visual branches: one for global structure and another for text-guided fine-grained details, complemented by a gated dynamic fusion module. Experiments on Med-VQA show that MMC-LLaVA outperforms previous state-of-the-art models across multiple metrics, with notable gains in detailed description scores.

Transcatheter tricuspid valve replacement (TTVR) has emerged as a promising minimally invasive procedure for treating severe tricuspid regurgitation (TR). However, accurate catheter delivery remains a significant challenge, primarily due to the reliance on 2D vision feedback, complex catheter kinematics, camera-to-robot pose calibration, which are difficult to generalize across patients. To address these issues, this paper presents a model-free robotic catheter delivery strategy for TTVR using Data-Enabled Predictive Control (DeePC). This approach leverages data-driven control to optimize catheter positioning without the need for prior knowledge of the system’s dynamics, eliminating the need for complex kinematic models or camera calibration. The proposed method incorporates environmental constraints to ensure the safety of the procedure, delivering the catheter to the desired location with high accuracy across varying catheters and camera poses. Experimental results demonstrate the effectiveness and versatility of the approach, suggesting its potential for broader applications in robotic-assisted surgeries. This work presents a new perspective for vision based robotic TTVR, as well as other clinical interventions involving robotic catheter control.

Registration of pre-operative 3D volumes and intra-operative 2D images is critical for neurological interventions. In various 2D/3D registration tasks, deep learning-based approaches have become popular and achieved tremendous success. However, due to vast space of transformation parameters, estimation errors are significant in these approaches. To tackle above issues, a novel learning-based framework for 2D/3D registration is proposed, consisting of CNN regression and centroid alignment. The former introduces a residual regression network (Res-RegNet) to preliminarily estimate transformation parameters. To further reduce estimation errors, the latter utilizes target vessel centroids to refine projected images. The proposed framework is individually trained and evaluated on three patients, reaching mean Dice of 76.69%, 78.51%, and 85.39%, respectively, all outperforming baseline methods. Extensive ablation studies demonstrate centroid alignment can significantly improve registration performance. As a normalization layer in Res-RegNet, SPADE can modulate activations using binarized inputs through a spatially-adaptive, learned transformation. Semantic information of inputs is preserved to learn better representations for parameter estimation. Moreover, the inference rate of our framework is about 21 FPS combined with the state-of-the-art segmentation model, significantly surpassing real-time requirements (6$\sim$12 FPS) in clinical practice. These promising results indicate the potential of the framework to facilitate various 2D/3D registration tasks.Note to Practitioners—This paper was motivated by the problem of image-guided neurological interventions. Existing 2D/3D registration methods suffer from 1) long iteration times, which are difficult to meet real-time clinical necessities, or 2) significant parameter estimation errors, leading to poor registration accuracies. Therefore, this paper suggests a new registration framework, combining with CNN regression to give predictions of transformation parameters via a single forward propagation, and centroid alignment to reduce estimation errors by translation transformation. The framework is trained and tested on three patients separately and achieves state-of-the-art performance, demonstrating its superiority. Furthermore, the proposed framework is a learning-based method that is adaptable to various image modalities. Therefore, it has latent capacities to be integrated into surgical navigation systems.

Fast and accurate segmentation of three-dimensional vasculature significantly enhances the precision and safety of endovascular surgeries. However, current 3D segmentation methods suffer from low accuracy due to a lack of global information and severe time consumption, making them impractical for clinical scenarios. In this paper, we propose a novel Dynamic Anatomy-aware Inference (DAI) framework, which leverages the anatomical prior of vascular structures to facilitate segmentation performance. In this framework, a Target Window Search (TWS) method is proposed to sample potential patches containing target vasculature, which leads to less computational burden and more consistent input data distribution for the embedded segmentation models. Then, a Spatial Fuse Module (SFM) is designed to encode the features of sampled patches based on their topological relationships, so that the long-range dependencies of target vasculature are effectively captured. Furthermore, comprehensive experiments are conducted to validate the effectiveness of proposed methods. Compared with the prevalent sliding window inference framework, a variety of models embedded in DAI achieve significant improvements in terms of both efficiency and accuracy: 13-147 × reductions in inference time (↓), 2-9% increases in Dice (↑), 4-16% increases in mIoU (↑), 57-83% decreases in HD (↓), 21-56% increases in clDice (↑).

Cryo-electron microscopy single particle analysis (cryo-EM SPA) is the most powerful technique for biomacromolecule structure determination. However, many factors such as complicated noise and radiation damage make the quality of cryo-EM images extremely poor, where high-frequency structure details are submerged, limiting the application of deep learning and suppressing the resolution of reconstruction. Thus, image restoration is of vital importance. Some related works explore micrograph restoration, but the particles in restored micrographs are still of poor quality. Moreover, the training approach of existing methods uses noisy observations or simulated data as supervision, leading to reduced performance on real cryo-EM data. In this paper, we define the task of particle restoration and propose a novel 4-step framework to this end. Labels are created for each particle image and paired data is collected within our framework, compensating for the absence of ground truth. A deep neural network with encoder-decoder architecture is designed to learn the mapping from degraded particles to high-quality ones, while other networks can also be employed as a plug-and-play module. Three datasets are constructed from real cryo-EM data and extensive experiments are carried out. Both quantitative metrics and qualitative visualization indicate that our framework is effective for cryo-EM particle restoration. It becomes easier to extract particle features after restoration, aiding in SPA and the effective application of deep learning on cryo-EM images. The downstream task experiments of cryo-EM SPA are also conducted, showing that the proposed framework has the potential to improve cryo-EM SPA performance.

Accurate recognition of human motion intention (HMI) is beneficial for exoskeleton robots to improve the wearing comfort level and achieve natural human-robot interaction. A classifier trained on labeled source subjects (domains) performs poorly on unlabeled target subject since the difference in individual motor characteristics. The unsupervised domain adaptation (UDA) method has become an effective way to this problem. However, the labeled data are collected from multiple source subjects that might be different not only from the target subject but also from each other. The current UDA methods for HMI recognition ignore the difference between each source subject, which reduces the classification accuracy. Therefore, this article considers the differences between source subjects and develops a novel theory and algorithm for UDA to recognize HMI, where the margin disparity discrepancy (MDD) is extended to multisource UDA theory and a novel weight-aware-based multisource UDA algorithm (WMDD) is proposed. The source domain weight, which can be adjusted adaptively by the MDD between each source subject and target subject, is incorporated into UDA to measure the differences between source subjects. The developed multisource UDA theory is theoretical and the generalization error on target subject is guaranteed. The theory can be transformed into an optimization problem for UDA, successfully bridging the gap between theory and algorithm. Moreover, a lightweight network is employed to guarantee the real-time of classification and the adversarial learning between feature generator and ensemble classifiers is utilized to further improve the generalization ability. The extensive experiments verify theoretical analysis and show that WMDD outperforms previous UDA methods on HMI recognition tasks.

Current studies for brain-muscle modulation often analyze selected properties in electrophysiological signals, leading to a partial understanding. This article proposes a cross-modal generative model that converts brain activities measured by electroencephalography (EEG) to corresponding muscular responses recorded by electromyography (EMG). Examining the generation process in the model highlights how the motor cue, representing implicit motor information hidden within brain activities, modulates the interaction between brain and muscle systems. The proposed model employs a two-stage generation process to bridge the semantic gap in cross-modal signals. Initially, the shared movement-related information between EEG and EMG signals is extracted using a contrastive learning framework. These shared representations act as conditional vectors in the subsequent EMG generation stage based on generative adversarial networks (GANs). Experiments on a self-collected multimodal electrophysiological signal data set show the algorithm's superiority over existing time series generative methods in cross-modal EMG generation. Further insights derived from the model's inference process underscore the brain's strategy for muscle control during movements. This research provides a data-driven approach for the neuroscience community, offering a comprehensive perspective of brain-muscular modulation.

Motor skills are performed through sequential movements rather than isolated actions. Yet, decoding these sequences from biosignals poses a significant challenge. To address this gap, this study transitions motor decoding from classifying movements in isolated time windows to segmenting sequential movements. The proposed algorithm segments the electromyography (EMG) sequence in a coarse-to-fine manner. It begins with frame-level segmentation and locating the approximate boundaries at the movement-level. A region-growing-inspired fusion strategy is then designed to incorporate the coarse segmentation and localization results for the fined output. Experiments on a self-collected EMG dataset demonstrate impressive results in segmenting movements for participant-dependent/independent setups (accuracy:$94.2\hbox{\%}/74.7\hbox{\%}$; dice coefficient:$92.5\hbox{\%}/61.7\hbox{\%}$; mean Intersection over Union:$80.9\hbox{\%}/51.9\hbox{\%}$). Further analysis shows the algorithm's ability to capture the natural rhythm in participants' movement sequences. This research paves the way for a deep understanding of motor sequences, which benefits various applications, such as rehabilitation engineering.

Model-based reinforcement learning (RL), which learns an environment model from the offline dataset and generates more out-of-distribution model data, has become an effective approach to the problem of distribution shift in offline RL. Due to the gap between the learned and actual environment, conservatism should be incorporated into the algorithm to balance accurate offline data and imprecise model data. The conservatism of current algorithms mostly relies on model uncertainty estimation. However, uncertainty estimation is unreliable and leads to poor performance in certain scenarios, and the previous methods ignore differences between the model data, which brings great conservatism. To address the above issues, this article proposes a mildly conservative model-based offline RL algorithm (DOMAIN) without estimating model uncertainty, and designs the adaptive sampling distribution of model samples, which can adaptively adjust the model data penalty. In this article, we theoretically demonstrate that theQvalue learned by the DOMAIN outside the region is a lower bound of the trueQvalue, the DOMAIN is less conservative than previous model-based offline RL algorithms, and has the guarantee of safety policy improvement. The results of extensive experiments show that DOMAIN outperforms prior RL algorithms and the average performance has improved by 1.8% on the D4RL benchmark.

Underwater collision-free navigation and dense reconstruction are essential for marine refined exploration. However, existing passive vision-based methods are difficult to apply in low-light and weak-feature underwater environments. In this article, a more adaptable three-dimensional (3-D) dense mapping robotic system based on self-designed scanning binocular structured light (BSL), named ROV-Scanner, is developed to address this challenge. First, the measurement principle based on the refraction model ensures its high accuracy. Second, an underwater 3-D dense mapping algorithm fusing the Doppler velocity log (DVL), inertial measurement unit (IMU), and pressure sensor multifrequency information is proposed to realize dense mapping during robot motion. Then, an air–water two-stage extrinsic calibration algorithm is proposed. In particular, the extrinsic parameters between DVL and camera are innovatively calibrated using BSL, enhancing robustness. Furthermore, for the first time, a framework of BSL-based collision-free navigation is presented to guarantee the safe movement of the system in unknown environments. Experimental results show that our system can simultaneously achieve autonomous collision-free navigation and dense mapping in dark underwater environments, which has great potential for application in marine refined exploration.

Diabetic peripheral neuropathy (DPN) causes tactile impairment and leads to foot ulcers and lower limb disabilities. Reducing vibration perception threshold (VPT) and preventing foot ulceration are recommended goals of tactile enhancement for DPN. In this study, a new-design vibrational foot orthosis (VFO) with a novel vibratory stimulus, a square wave pulse integrated with pseudorandom noise (PRN) via a stochastic resonance (SR) technique, was developed to improve tactile sensitivity. A pilot study was conducted to investigate the effectiveness of using the VFO for tactile stimulation in twenty DPN patients with two distinct interventions: 1) a 100 Hz square wave pulse stimulus combined with PRN by a SR technique; and 2) a random 0–100 Hz square wave pulse stimulus combined with PRN by a SR technique. VPT values at the 1st metatarsophalangeal joint (MTP) and 5th MTP were assessed in a pretest and posttest as the outcome. The results indicated that both interventions could significantly decrease VPT values (P<.001); however, intervention 2 might effectively lower VPT values more than intervention 1 by around threefold. In conclusion, the VFO can improve tactile sensitivity, particularly the random stimulus mixed with PRN by a SR technique, which may provide a greater advantage than no random stimulus. The VFO can address the challenges of a total contact design, appropriate manufacturing for diabetic feet, and fabrication with medical-grade materials. Contributions: The VFO may be useful for neurorehabilitation, avoiding foot ulcers and their recurrence, as well as preventing lower limb amputation.

Learning motor representations in brains presents a challenge due to the entanglement of motor-related and unrelated information within neural imaging data. This study introduces a cross-modal learning algorithm that utilizes electromyogram (EMG) muscle cues to refine the learning of electroencephalogram (EEG) motor representations. The algorithm begins with original EEG representations from a baseline motor classification model. Subsequently, EMG muscle cues are learned to decompose the original EEG representations into motor-related and unrelated components. The decomposition process is achieved by aligning the EMG representations more closely with motor-related components and less with unrelated ones. Experimental results on a self-collected multi-modal dataset show the proposed algorithm leads to a performance enhancement of approximately 4% across various algorithms compared with the original EEG representations in motor classification. This advancement demonstrates the algorithm’s effectiveness in isolating motor-related information from complex brain activities. The innovative use of muscle cues for EEG motor characteristic learning opens new possibilities for incorporating cross-modal learning in creating more accurate brain-computer interfaces.

Ultrasound (US) imaging of the carotid artery (CA) is a non-invasive diagnostic tool widely used in the medical field to assess the condition of the carotid artery, thereby predicting the risk of cardiovascular and cerebrovascular diseases. However, implementing this method in primary healthcare can be challenging due to the requirement for professionally trained sonographers. With the adoption of US robotic devices, the probe pose can be acquired while scanning, offering the possibility for 3D reconstruction and providing analyses that are not dependent on operator experience. This article introduces a method to semi-automatically acquire serialized US images of the common carotid artery (CCA). The method involves a specially designed robotic device built with a 6-RSU parallel mechanism, which is controlled according to robot pose, force sensor data and synchronous US images. To validate the images acquired, a method is proposed to segment the intima-media of CCA and calculate the intima-media thickness (IMT), which is a key indicator for cerebrovascular events prediction. After that, we propose an algorithm to reconstruct CCA into 3D voxel data with patient movement and cardiac cycle compensated, and a longitudinal view US image of CCA can be resliced from the voxel. The methods are tested on human subjects and the results indicate that the system and workflow can provide both quantitative and qualitative information of CCA for further diagnosis.

The development of robotic hand that can imitate human movements has always been an important research topic. In this paper, a linkage-driven underactuated three-finger hand is proposed to imitate the flexion/extension (f/e) and abduction/adduction (a/a) motions of human hand. The robotic hand has three identical underactuated fingers, each of which contains an underactuated planar linkage, a spherical four-bar mechanism, and a set of bevel gears. The spherical four-bar mechanism is designed to provide 2-degree-of-freedom actuation, driving the f/e and a/a motions of the proximal joint simultaneously. Based on screw theory, the kinematic model of the spherical mechanism is established, and the maximum available workspace index (MAW) of the spherical mechanism is proposed to evaluate the workspace with the same adduction and abduction angle ranges. The effects of the parameters of the spherical mechanism on the MAW and the transmission efficiency are obtained, and the parameters of the spherical mechanism are optimized. The optimization results show that the MAW of the spherical mechanism can be increased by up to 3.5 times. Finally, experiments are carried out to show the proposed robotic hand can perform simultaneous adaptive grasping and in-hand manipulation.Note to Practitioners—The robotic hand is of great significance for replacing workers in heavy, repetitive, and unsafe working environments. Uncertain environments in non-manufacturing fields, such as industries for pick-and-place, sorting, palletizing, assembly and other operations, marine development, and medical service, require that robotic hands can operate tasks like human hands. Inspired by the human hand with the f/e and a/a motions, a linkage-driven underactuated three-finger hand is proposed in this paper to imitate the functions of the human hand. Due to its underactuated characteristics and a/a function, the designed robotic hand is more dexterous and economical. The effects of the design parameters of the spherical mechanism on the MAW and the transmission efficiency are analyzed, and the parameters of the spherical mechanism are optimized to improve the performance of the robotic hand. This approach can be used in other industrial applications related to the robotic hand or the spherical mechanism.

In computer-assisted intravascular interventional surgery, extracting detailed information of target vessels from X-ray angiographic images can be meaningful in improving safety and effectiveness. However, large amounts of effort have been dedicated to segmenting the whole blood vessels from the background while ignoring the internal structure, which is limited in clinical application. In this paper, we propose a flexible and universal endpoint-based framework for vessel structural information extraction. The framework first localizes all the endpoints of target vessel segments through a Coarse-to-Fine Keypoint Detection Network (CFKD-Net), in which the designed Multi-branch Feature Aggregation (MFA) module captures both in-patch and cross-patch information to help recognize the points of interest based on global structure. A novel MaskMSELoss is also proposed to disambiguate those irrelevant responses. Then a designed VEssel Segmentation and Analysis (VESA) algorithm will generate the segmentation mask and morphological analysis for each vessel segment simply based on the endpoints. It can also be flexibly applied to analyze variant blood vessels which are not pre-defined before. Extensive experiments on two different coronary artery datasets consistently demonstrate that this framework can achieve state-of-the-art detection performance and successfully extract and analyze target vessel segments. Since the framework shows excellent performance on the coronary arteries with severe deformation and strong noise, it is highly promising for analyzing other vascular images.

This paper presents a novel learning-based algorithm to investigate the high-level shared semantic information between electroencephalography (EEG) and electromyography (EMG) signals, for understanding brain-muscle modulation during movement execution. The proposed algorithm incorporates a spatial encoder that condenses spatial information obtained from EEG/EMG signals into unified temporal tokens using a learnable correlation matrix. These tokens are then encoded and decoded via a siamese temporal encoder and classification head to extract joint semantic information presented in cross-modal signals. Additionally, an analysis pipeline is designed to examine brain-muscle modulation based on the proposed algorithm. Experimental results from a self-collected multimodal bio-signals dataset validate the efficacy of the proposed algorithm in extracting and analyzing high-level latent semantic information shared in EEG and EMG signals, outperforming the state-of-the-art model by 5.35% in accuracy, 4.69% in precision, and 8.65% in recall. Notably, the designed analysis pipeline can also reveal low-level relationships, such as those related to time and space, between multimodal bio-signals. This research provides neuroscientists with a valuable tool for obtaining enhanced insights into brain-muscle modulation.

Hospital patients can have catheters and lines inserted during the course of their admission to give medicines for the treatment of medical issues, especially the central venous catheter (CVC). However, malposition of CVC will lead to many complications, even death. Clinicians always detect the malposition based on position detection of CVC tip via X-ray images. To reduce the workload of the clinicians and the percentage of malposition occurrence, we propose an automatic catheter tip detection framework based on a convolutional neural network (CNN). The proposed framework contains three essential components which are modified HRNet, segmentation supervision module, and deconvolution module. The modified HRNet can retain high-resolution features from start to end, ensuring the maintenance of precise information from the X-ray images. The segmentation supervision module can alleviate the presence of other line-like structures such as the skeleton as well as other tubes and catheters used for treatment. In addition, the deconvolution module can further increase the feature resolution on the top of the highest-resolution feature maps in the modified HRNet to get a higher-resolution heatmap of the catheter tip. A public CVC Dataset is utilized to evaluate the performance of the proposed framework. The results show that the proposed algorithm offering a mean Pixel Error of 4.11 outperforms three comparative methods (Ma's method, SRPE method, and LCM method). It is demonstrated to be a promising solution to precisely detect the tip position of the catheter in X-ray images.

This article addresses the issue of output-feedback consensus control of multiagent systems under the directed topology and subject to bounded external disturbances. By employing a smooth time-varying function, a distributed practical predefined-time (PPT) observer is developed to estimate the reference trajectory for the entire team (i.e., the leader's state) and a practical preset-time extended-state observer is also proposed to estimate bounded disturbances and unmeasurable system states. Next, a novel continuous and nonsingular PPT consensus control law is designed on the basis of the observers. Furthermore, the designed control protocol can achieve PPT stability, that is, consensus tracking errors are enforced to a neighborhood around zero within a predetermined time, which can be specified a priori, independent of initial states of agents and/or any other design parameters. Finally, illustrative numerical examples, including a comparative one, are provided to demonstrate the performance of the present predefined-time control approach.

During intravascular interventional surgery, the 3D surgical navigation system can provide doctors with 3D spatial information of the vascular lumen, reducing the impact of missing dimension caused by digital subtraction angiography (DSA) guidance and further improving the success rate of surgeries. Nevertheless, this task often comes with the challenge of complex registration problems due to vessel deformation caused by respiratory motion and high requirements for the surgical environment because of the dependence on external electromagnetic sensors. This article proposes a novel 3D spatial predictive positioning navigation (SPPN) technique to predict the real-time tip position of surgical instruments. In the first stage, we propose a trajectory prediction algorithm integrated with instrumental morphological constraints to generate the initial trajectory. Then, a novel hybrid physical model is designed to estimate the trajectory's energy and mechanics. In the second stage, a point cloud clustering algorithm applies multi-information fusion to generate the maximum probability endpoint cloud. Then, an energy-weighted probability density function is introduced using statistical analysis to achieve the prediction of the 3D spatial location of instrument endpoints. Extensive experiments are conducted on 3D-printed human artery and vein models based on a high-precision electromagnetic tracking system. Experimental results demonstrate the outstanding performance of our method, reaching 98.2% of the achievement ratio and less than 3 mm of the average positioning accuracy. This work is the first 3D surgical navigation algorithm that entirely relies on vascular interventional robot sensors, effectively improving the accuracy of interventional surgery and making it more accessible for primary surgeons.

Percutaneous coronary intervention (PCI) has increasingly become the main treatment for coronary artery disease. The procedure requires high experienced skills and dexterous manipulations. However, there are few techniques to model PCI skill so far. In this study, a learning framework with local and ensemble learning is proposed to learn skill characteristics of different skill-level subjects from their PCI manipulations. Ten interventional cardiologists (four experts and six novices) were recruited to deliver a medical guidewire to two target arteries on a porcine model for in vivo studies. Simultaneously, translation and twist manipulations of thumb, forefinger, and wrist are acquired with electromagnetic (EM) and fiber-optic bend (FOB) sensors, respectively. These behavior data are then processed with wavelet packet decomposition (WPD) under 1-10 levels for feature extraction. The feature vectors are further fed into three candidate individual classifiers in the local learning layer. Furthermore, the local learning results from different manipulation behaviors are fused in the ensemble learning layer with three rule-based ensemble learning algorithms. In subject-dependent skill characteristics learning, the ensemble learning can achieve 100% accuracy, significantly outperforming the best local result (90%). Furthermore, ensemble learning can also maintain 73% accuracy in subject-independent schemes. These promising results demonstrate the great potential of the proposed method to facilitate skill learning in surgical robotics and skill assessment in clinical practice.

Physical human–robot interaction performance of present rehabilitation robots are still not satisfactory in the clinical practice. Especially, the work space where the robot can be driven smoothly by users is still very limited, which prevents rehabilitation robots from being applied successfully. In this study, a new concept of drivable space is proposed to evaluate the work spaces of rehabilitation robots, and a method for expanding the drivable space is designed based on the dynamics of the coupled human–robot system and human joint characteristics. First, the definition of drivable space is presented based on comparison of human joint torques, and the minimal torques necessary to drive robot joints, which is mainly determined by the torque estimation errors for general rehabilitation robots driven smoothly by motors. Therefore, a method for improving torque estimation accuracies based on dynamics modeling is then designed. A data-driven error prediction method based on Gaussian process regression is proposed to adaptively compensate the model errors, by which the most accurate dynamic model so far for the coupled system can be obtained, and a method for generation of the training dataset, which is used in error prediction, is designed as well. Moreover, the torque–angle relationship of human joints is modeled and used to optimize the torque error distribution, by which it can be proven that the drivable space can be further expanded. Finally, performance of the proposed methods are demonstrated and validated by experiments carried out on a lower limb rehabilitation robot.

Real-time and precise morphological analysis of intraoperative AAA is a significant pre-imperative for robot-assisted minimally invasive surgery (RMIS). However, this task is frequently accompanied by the difficulties of ambiguous boundaries and obscured surfaces of aneurysms. To remedy these problems, we propose a Light-Weight Dual-Stream Boundary-Aware Network (DSB-Net) and a novel diagnosis algorithm for real-time morphological analysis of AAA. In the network, the features at the boundaries are preserved by incorporating a boundary localization stream, while the interior segmentation accuracy is guaranteed with a mask prediction stream. Moreover, the diagnosis algorithm is developed to measure the exact size of AAA. Quantitative and qualitative assessments on two different types of datasets illustrate that (1) The presented DSB-Net remarkably outperforms the other previously proposed medical networks with the inference rate of 10.8 FPS, which meets the real-time clinical necessities. (2) The developed algorithm provides accurate size measurements for AAA, which indicates the proposed approach can be integrated into the robotic navigation framework for RMIS.

This article investigates the tracking control for input and full-state-constrained nonlinear time-delay systems with unknown time-varying powers, whose nonlinearities do not impose any growth assumption. By utilizing the auxiliary control signal and nonlinear state-dependent transformation (NSDT) to counteract the effect of input saturation and cope with full-state constraints, respectively, and then introducing lower and higher powers and Lyapunov-Krasovskii (L-K) functionals in control design together with the adaptive neural-networks (NNs) method, an adaptive neural tracking control design is provided without feasibility conditions. It is proved that NNs approximation is valid, all the closed-loop signals are semiglobally bounded, and input and full-state constraints are not violated.

This article discusses the adaptive fuzzy asymptotic tracking control for high-order nonlinear time-delay systems with full-state constraints. Fuzzy-logic systems and a separation principle are utilized to relax growth assumptions imposed on unknown nonlinearities. The adverse effect caused by unknown time delays is eliminated by choosing appropriate Lyapunov-Krasovskii functionals. By integrating nonlinear-transformed functions with a key coordinate transformation into the control design and constructing a specific compact set on the initial values of system states, the desired trajectory and parameter estimates, it is rigorously proved that all closed-loop signals are semiglobally bounded, the fuzzy approximation is valid, the full-state constraints are not violated without feasibility conditions on virtual controllers, and asymptotic tracking is achieved. The effectiveness and advantages of this control scheme are confirmed by two examples including a single-link robotic system.

In this article, an adaptive practical tracking control scheme is presented for full-state constrained high-order nonlinear systems. By skillfully introducing the adaptive gain, nonlinear transformed functions and sign functions into control design, a novel continuous state-feedback controller is constructed without imposing restrictive approximation techniques and feasibility conditions. Under mild assumptions, the boundedness of all the closed-loop signals can be guaranteed, full-state constraints are not transgressed for all time, and the tracking error tends to an arbitrarily small region of zero in a finite time.

The clinical success of the percutaneous coronary intervention (PCI) is highly dependent on endovascular manipulation skills and dexterous manipulation strategies of interventionalists. However, the analysis of endovascular manipulations and related discussion for technical skill assessment are limited. In this study, a multilayer and multimodal-fusion architecture is proposed to recognize six typical endovascular manipulations. The synchronously acquired multimodal motion signals from ten subjects are used as the inputs of the architecture independently. Six classification-based and two rule-based fusion algorithms are evaluated for performance comparisons. The recognition metrics under the determined architecture are further used to assess technical skills. The experimental results indicate that the proposed architecture can achieve the overall accuracy of 96.41%, much higher than that of a single-layer recognition architecture (92.85%). In addition, the multimodal fusion brings significant performance improvement in comparison with single-modal schemes. Furthermore, the K -means-based skill assessment can obtain an accuracy of 95% to cluster the attempts made by different skill-level groups. These hopeful results indicate the great possibility of the architecture to facilitate clinical skill assessment and skill learning.

Human–robot interaction has been widely studied, where compliant and safe human–robot interaction of continuum manipulators in the constrained environment is one of the key issues that have not been well addressed. In this study, finite-time observer-based variable impedance control of cable-driven continuum manipulators (CDCM) is proposed to overcome the limitation of existing methods. First, the pseudo-rigid modeling method is utilized to establish the kinematics and dynamics of the CDCM. Then, a variable impedance controller with selected controller parameters for the CDCM is designed to realize compliant and safe human–robot interaction operations with force–position coupling constraint, which is rarely studied in the literature. In order to realize the closed-loop variable impedance controller, a finite-time observer is designed to estimate acceleration feedbacks, which can avoid the difficulty of directly sensing the interaction forces and shows excellent robust stability in noisy environments. On this basis, by combining the advantages of the variable impedance controller and finite-time observer, the finite-time observer-based variable impedance controller is proposed, and the stabilities of the proposed method are analyzed. Finally, the feasibility of the proposed control scheme for the CDCM is demonstrated by some numerical simulations.

Surgical image segmentation is critical for surgical robot control and computer-assisted surgery. In the surgical scene, the local features of objects are highly similar, and the illumination interference is strong, which makes surgical image segmentation challenging. To address the above issues, a bilinear squeeze reasoning network is proposed for surgical image segmentation. In it, the space squeeze reasoning module is proposed, which adopts height pooling and width pooling to squeeze global contexts in the vertical and horizontal directions, respectively. The similarity between each horizontal position and each vertical position is calculated to encode long-range semantic dependencies and establish the affinity matrix. The feature maps are also squeezed from both the vertical and horizontal directions to model channel relations. Guided by channel relations, the affinity matrix is expanded to the same size as the input features. It captures long-range semantic dependencies from different directions, helping address the local similarity issue. Besides, a low-rank bilinear fusion module is proposed to enhance the model's ability to recognize similar features. This module is based on the low-rank bilinear model to capture the inter-layer feature relations. It integrates the location details from low-level features and semantic information from high-level features. Various semantics can be represented more accurately, which effectively improves feature representation. The proposed network achieves state-of-the-art performance on cataract image segmentation dataset CataSeg and robotic image segmentation dataset EndoVis 2018.

Magnetic Resonance Imaging (MRI) has been proven to be an efficient way to diagnose Alzheimer's disease (AD). Recent dramatic progress on deep learning greatly promotes the MRI analysis based on data-driven CNN methods using a large-scale longitudinal MRI dataset. However, most of the existing MRI datasets are fragmented due to unexpected quits of volunteers. To tackle this problem, we propose a novel Temporal Recurrent Generative Adversarial Network (TR-GAN) to complete missing sessions of MRI datasets. Unlike existing GAN-based methods, which either fail to generate future sessions or only generate fixed-length sessions, TR-GAN takes all past sessions to recurrently and smoothly generate future ones with variant length. Specifically, TR-GAN adopts recurrent connection to deal with variant input sequence length and flexibly generate future variant sessions. Besides, we also design a multiple scale & location (MSL) module and a SWAP module to encourage the model to better focus on detailed information, which helps to generate high-quality MRI data. Compared with other popular GAN architectures, TR-GAN achieved the best performance in all evaluation metrics of two datasets. After expanding the Whole MRI dataset, the balanced accuracy of AD vs. cognitively normal (CN) vs. mild cognitive impairment (MCI) and stable MCI vs. progressive MCI classification can be increased by 3.61% and 4.00%, respectively.

Hashing is a popular search algorithm for its compact binary representation and efficient Hamming distance calculation. Benefited from the advance of deep learning, deep hashing methods have achieved promising performance. However, those methods usually learn with expensive labeled data but fail to utilize unlabeled data. Furthermore, the traditional pairwise loss used by those methods cannot explicitly force similar/dissimilar pairs to small/large distances. Both weaknesses limit existing methods’ performance. To solve the first problem, we propose a novel semi-supervised deep hashing model named adversarial binary mutual learning (ABML). Specifically, our ABML consists of a generative model$G_{H}$and a discriminative model$D_{H}$, where$D_{H}$learns labeled data in a supervised way and$G_{H}$learns unlabeled data by synthesizing real images. We adopt an adversarial learning (AL) strategy to transfer the knowledge of unlabeled data to$D_{H}$by making$G_{H}$and$D_{H}$mutually learn from each other. To solve the second problem, we propose a novel Weibull cross-entropy loss (WCE) by using the Weibull distribution, which can distinguish tiny differences of distances and explicitly force similar/dissimilar distances as small/large as possible. Thus, the learned features are more discriminative. Finally, by incorporating ABML with WCE loss, our model can acquire more semantic and discriminative features. Extensive experiments on four common data sets (CIFAR-10, large database of handwritten digits (MNIST), ImageNet-10, and NUS-WIDE) and a large-scale data set ImageNet demonstrate that our approach successfully overcomes the two difficulties above and significantly outperforms state-of-the-art hashing methods.

In this article, the underwater target tracking control problem of a biomimetic underwater vehicle (BUV) is addressed. Since it is difficult to build an effective mathematic model of a BUV due to the uncertainty of hydrodynamics, target tracking control is converted into the Markov decision process and is further achieved via deep reinforcement learning. The system state and reward function of underwater target tracking control are described. Based on the actor-critic reinforcement learning framework, the deep deterministic policy gradient actor-critic algorithm with supervision controller is proposed. The training tricks, including prioritized experience replay, actor network indirect supervision training, target network updating with different periods, and expansion of exploration space by applying random noise, are presented. Indirect supervision training is designed to address the issues of low stability and slow convergence of reinforcement learning in the continuous state and action space. Comparative simulations are performed to show the effectiveness of the training tricks. Finally, the proposed actor-critic reinforcement learning algorithm with supervision controller is applied to the physical BUV. Swimming pool experiments of underwater object tracking of the BUV are conducted in multiple scenarios to verify the effectiveness and robustness of the proposed method.

Recently, single-particle cryo-electron microscopy (cryo-EM) has become an indispensable method for determining macromolecular structures at high resolution to deeply explore the relevant molecular mechanism. Its recent breakthrough is mainly because of the rapid advances in hardware and image processing algorithms, especially machine learning. As an essential support of single-particle cryo-EM, machine learning has powered many aspects of structure determination and greatly promoted its development. In this article, we provide a systematic review of the applications of machine learning in this field. Our review begins with a brief introduction of single-particle cryo-EM, followed by the specific tasks and challenges of its image processing. Then, focusing on the workflow of structure determination, we describe relevant machine learning algorithms and applications at different steps, including particle picking, 2-D clustering, 3-D reconstruction, and other steps. As different tasks exhibit distinct characteristics, we introduce the evaluation metrics for each task and summarize their dynamics of technology development. Finally, we discuss the open issues and potential trends in this promising field.

Robot-assisted rehabilitation has been a promising solution to improve motor learning of neurologically impaired patients. State-of-the-art control strategies are typically limited to the ignorance of heterogeneous motor capabilities of poststroke patients and therefore intervene suboptimally. In this article, we propose a control framework for robot-assisted motor learning, emphasizing the detection of human intention, generation of reference trajectories, and modification of robotic assistance. A real-time trajectory generation algorithm is presented to extract the high-level features in active arm movements using an adaptive frequency oscillator (AFO) and then integrate the movement rhythm with the minimum-jerk principle to generate an optimal reference trajectory, which synchronizes with the motion intention in the patient as well as the motion pattern in healthy humans. In addition, a subject-adaptive assistance modification algorithm is presented to model the patient’s residual motor capabilities employing spatially dependent radial basis function (RBF) networks and then combining the RBF-based feedforward controller with the impedance feedback controller to provide only necessary assistance while simultaneously regulating the maximum-tolerated error during trajectory tracking tasks. We conduct simulation and experimental studies based on an upper limb rehabilitation robot to evaluate the overall performance of the motor-learning framework. A series of results showed that the difficulty level of reference trajectories was modulated to meet the requirements of subjects’ intended motion, furthermore, the robotic assistance was compliantly optimized in response to the changing performance of subjects’ motor abilities, highlighting the potential of adopting our framework into clinical application to promote patient-led motor learning.

Medical diagnostic robot systems have been paid more and more attention due to its objectivity and accuracy. The diagnosis of mild cognitive impairment (MCI) is considered an effective means to prevent Alzheimer's disease (AD). Doctors diagnose MCI based on various clinical examinations, which are expensive and the diagnosis results rely on the knowledge of doctors. Therefore, it is necessary to develop a robot diagnostic system to eliminate the influence of human factors and obtain a higher accuracy rate. In this paper, we propose a novel Group Feature Domain Adversarial Neural Network (GF- DANN) for amnestic MCI (aMCI) diagnosis, which involves two important modules. A Group Feature Extraction (GFE) module is proposed to reduce individual differences by learning group- level features through adversarial learning. A Dual Branch Domain Adaptation (DBDA) module is carefully designed to reduce the distribution difference between the source and target domain in a domain adaption way. On three types of data set, GF-DANN achieves the best accuracy compared with classic machine learning and deep learning methods. On the DMS data set, GF-DANN has obtained an accuracy rate of 89.47%, and the sensitivity and specificity are 90% and 89%. In addition, by comparing three EEG data collection paradigms, our results demonstrate that the DMS paradigm has the potential to build an aMCI diagnose robot system.

Real-time guidewire segmentation and endpoint localization play a pivotal role in robot-assisted minimally invasive surgery, which is helpful to reduce radiation dose and procedure time. Nevertheless, the tasks often come with the challenge of limited computational resources. For this purpose, a real-time multi-task framework with two stages is developed. In the first stage, a Fast Attention-fused Network (FAD-Net) is proposed to obtain accurate guidewire segmentation masks. In the second stage, a lightweight localization network and a post-processing algorithm are designed to robustly predict the guidewire endpoint position. Quantitative and qualitative evaluations on intraoperative X-ray sequences from 30 patients demonstrate that the developed framework outperforms the previously-published results for the tasks, achieving state-of-the-art performance. Moreover, the inference rate of the developed framework is approximately 10.6 FPS, which meets the real-time requirement of X-ray fluoroscopy. These results indicate the proposed approach has the potential to be integrated into the robotic navigation framework for endovascular interventions, enabling robotic-assisted minimally invasive surgery.

Active rehabilitation training can promote the neural reorganization and facilitate the rehabilitation of paralyzed patients. To provide safe and efficient active training based on rehabilitation robots, human motion intention should be recognized firstly, which can be implemented by prediction of human joint angles using sEMG. In this study, a novel CNN-LSTM model using multi-band sEMG fused with historical angles is proposed to improve the angle prediction accuracy. Eight models using sEMG signals of different numbers of frequency bands (1, 3, 5, 7) and fused or not fused with historical angles are designed and tested based on 10 subjects. The results show that, sEMG signals of suitable number of frequency bands can efficiently raise the prediction accuracy, and adding historical angles to the inputs can effectively eliminate the fluctuation of angle prediction and significantly improve the prediction accuracy. In particular, the average prediction error for the model based on 5-band sEMG and historical angles on the test data set is 0.784 degrees, which is accurate enough for practical application for the robot assisted rehabilitation.

In computer-assisted vascular surgery, real-time multi-instrument segmentation serves as a pre-requisite step. However, a large amount of effort has been dedicated to single-instrument rather than multi-instrument in computer-assisted intervention research to this day. To fill the overlooked gap, this study introduces a Light-Weight Deep Feature Refinement Network (DFR-Net) based on multi-task learning for real-time multi-instrument segmentation. In this network, the proposed feature refinement module (FRM) can capture long-term dependencies while retaining precise positional information, which helps model locate the foreground objects of interest. The designed channel calibration module (CCM) can re-calibrate fusion weights of multi-level features, which helps model balance the importance of semantic information and appearance information. Besides, the connectivity loss function is developed to address fractures in the wire-like structure segmentation results. Extensive experiments on two different types of datasets consistently demonstrate that DFR-Net can achieve state-of-the-art segmentation performance while meeting the real-time requirements.

Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. While numerous methods for detecting, segmenting and tracking of medical instruments based on endoscopic video images have been proposed in the literature, key limitations remain to be addressed: Firstly, robustness, that is, the reliable performance of state-of-the-art methods when run on challenging images (e.g. in the presence of blood, smoke or motion artifacts). Secondly, generalization; algorithms trained for a specific intervention in a specific hospital should generalize to other interventions or institutions. In an effort to promote solutions for these limitations, we organized the Robust Medical Instrument Segmentation (ROBUST-MIS) challenge as an international benchmarking competition with a specific focus on the robustness and generalization capabilities of algorithms. For the first time in the field of endoscopic image processing, our challenge included a task on binary segmentation and also addressed multi-instance detection and segmentation. The challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures from three different types of surgery. The validation of the competing methods for the three tasks (binary segmentation, multi-instance detection and multi-instance segmentation) was performed in three different stages with an increasing domain gap between the training and the test data. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap. While the average detection and segmentation quality of the best-performing algorithms is high, future research should concentrate on detection and segmentation of small, crossing, moving and transparent instrument(s) (parts).

The real-time localization of the guidewire endpoints is a stepping stone to computer-assisted percutaneous coronary intervention (PCI). However, methods for multi-guidewire endpoint localization in fluoroscopy images are still scarce. In this paper, we introduce a framework for real-time multi-guidewire endpoint localization in fluoroscopy images. The framework consists of two stages, first detecting all guidewire instances in the fluoroscopy image, and then locating the endpoints of each single guidewire instance. In the first stage, a YOLOv3 detector is used for guidewire detection, and a post-processing algorithm is proposed to refine the guidewire detection results. In the second stage, a Segmentation Attention-hourglass (SA-hourglass) network is proposed to predict the endpoint locations of each single guidewire instance. The SA-hourglass network can be generalized to the keypoint localization of other surgical instruments. In our experiments, the SA-hourglass network is applied not only on a guidewire dataset but also on a retinal microsurgery dataset, reaching the mean pixel error (MPE) of 2.20 pixels on the guidewire dataset and the MPE of 5.30 pixels on the retinal microsurgery dataset, both achieving the state-of-the-art localization results. Besides, the inference rate of our framework is at least 20FPS, which meets the real-time requirement of fluoroscopy images (6-12FPS).

Variable impedance control has been considered as one of the most important compliant control approaches for its abilities in improving compliance, safety, and efficiency in robot-environment interaction. However, existing variable impedance controllers have deficits in stability guarantee. This article proposes a stability-guaranteed variable impedance control approach for robots with modeling uncertainties based on approximate dynamic inversion (ADI). Novel constraints on variable impedance profiles are given to guarantee the exponential stability of the desired variable impedance dynamics. An ADI-based impedance control law is designed to achieve the desired variable impedance dynamics through the convergence of a variable impedance error. Based on the extended Tikhonovs theorem, it is proven that the closed-loop control system has semiglobal practical exponential stability. The proposed impedance controller can be implemented in a PID form and is appealing for its simple structure, easy implementation, and control stability guarantee. The effectiveness of the proposed variable impedance controller is illustrated by an illustrative example taken on a five-bar parallel robot.

Semantic segmentation of surgical instruments plays a critical role in computer-assisted surgery. However, specular reflection and scale variation of instruments are likely to occur in the surgical environment, undesirably altering visual features of instruments, such as color and shape. These issues make semantic segmentation of surgical instruments more challenging. In this paper, a novel network, Pyramid Attention Aggregation Network, is proposed to aggregate multi-scale attentive features for surgical instruments. It contains two critical modules: Double Attention Module and Pyramid Upsampling Module. Specifically, the Double Attention Module includes two attention blocks (i.e., position attention block and channel attention block), which model semantic dependencies between positions and channels by capturing joint semantic information and global contexts, respectively. The attentive features generated by the Double Attention Module can distinguish target regions, contributing to solving the specular reflection issue. Moreover, the Pyramid Upsampling Module extracts local details and global contexts by aggregating multi-scale attentive features. It learns the shape and size features of surgical instruments in different receptive fields and thus addresses the scale variation issue. The proposed network achieves state-of-the-art performance on various datasets. It achieves a new record of 97.10% mean IOU on Cata7. Besides, it comes first in the MICCAI EndoVis Challenge 2017 with 9.90% increase on mean IOU.

RGB-Infrared (IR) person re-identification is very challenging due to the large cross-modality variations between RGB and IR images. The key solution is to learn aligned features to the bridge RGB and IR modalities. However, due to the lack of correspondence labels between every pair of RGB and IR images, most methods try to alleviate the variations with set-level alignment by reducing the distance between the entire RGB and IR sets. However, this set-level alignment may lead to misalignment of some instances, which limits the performance for RGB-IR Re-ID. Different from existing methods, in this paper, we propose to generate cross-modality paired-images and perform both global set-level and fine-grained instance-level alignments. Our proposed method enjoys several merits. First, our method can perform set-level alignment by disentangling modality-specific and modality-invariant features. Compared with conventional methods, ours can explicitly remove the modality-specific features and the modality variation can be better reduced. Second, given cross-modality unpaired-images of a person, our method can generate cross-modality paired images from exchanged images. With them, we can directly perform instance-level alignment by minimizing distances of every pair of images. Extensive experimental results on two standard benchmarks demonstrate that the proposed model favourably against state-of-the-art methods. Especially, on SYSU-MM01 dataset, our model can achieve a gain of 9.2% and 7.7% in terms of Rank-1 and mAP. Code is available at https://github.com/wangguanan/JSIA-ReID.

The real-time segmentation of surgical instruments plays a crucial role in robot-assisted surgery. However, it is still a challenging task to implement deep learning models to do real-time segmentation for surgical instruments due to their high computational costs and slow inference speed. In this paper, we propose an attention-guided lightweight network (LWANet), which can segment surgical instruments in real-time. LWANet adopts encoder-decoder architecture, where the encoder is the lightweight network MobileNetV2, and the decoder consists of depthwise separable convolution, attention fusion block, and transposed convolution. Depthwise separable convolution is used as the basic unit to construct the decoder, which can reduce the model size and computational costs. Attention fusion block captures global contexts and encodes semantic dependencies between channels to emphasize target regions, contributing to locating the surgical instrument. Transposed convolution is performed to upsample feature maps for acquiring refined edges. LWANet can segment surgical instruments in real-time while takes little computational costs. Based on 960x544 inputs, its inference speed can reach 39 fps with only 3.39 GFLOPs. Also, it has a small model size and the number of parameters is only 2.06 M. The proposed network is evaluated on two datasets. It achieves state-of-the- art performance 94.10% mean IOU on Cata7 and obtains a new record on EndoVis 2017 with a 4.10% increase on mean IOU.

The increasingly-used robotic systems can provide precise delivery and reduce X-ray radiation to medical staff in percutaneous coronary interventions (PCI), but natural manipulations of interventionalists are forgone in most robot-assisted procedures. Therefore, it is necessary to explore natural manipulations to design more advanced human-robot interfaces (HRI). In this study, a multilayer-multimodal fusion architecture is proposed to recognize six typical subpatterns of guidewire manipulations in conventional PCI. The synchronously acquired multimodal behaviors from ten subjects are used as the inputs of the fusion architecture. Six classification-based and two rule-based fusion algorithms are evaluated for performance comparisons. Experimental results indicate that the multimodal fusion brings significant accuracy improvement in comparison with single-modal schemes. Furthermore, the proposed architecture can achieve the overall accuracy of 96.90%, much higher than that of a singlelayer recognition architecture (92.56%). These results have indicated the potential of the proposed method for facilitating the development of HRI for robot-assisted PCI.

Recent deep learning-based Brain-Computer Interface (BCI) decoding algorithms mainly focus on spatial-temporal features, while failing to explicitly explore spectral information which is one of the most important cues for BCI. In this paper, we propose a novel regional attention convolutional neural network (RACNN) to take full advantage of spectral-spatial-temporal features for EEG motion intention recognition. Time-frequency based analysis is adopted to reveal spectral-temporal features in terms of neural oscillations of primary sensorimotor. The basic idea of RACNN is to identify the activated area of the primary sensorimotor adaptively. The RACNN aggregates a varied number of spectral-temporal features produced by a backbone convolutional neural network into a compact fixed-length representation. Inspired by the neuroscience findings that functional asymmetry of the cerebral hemisphere, we propose a region biased loss to encourage high attention weights for the most critical regions. Extensive evaluations on two benchmark datasets and real-world BCI dataset show that our approach significantly outperforms previous methods.

Surgical instrument segmentation is crucial for computer-assisted surgery. Different from common object segmentation, it is more challenging due to the large illumination variation and scale variation in the surgical scenes. In this paper, we propose a bilinear attention network with adaptive receptive fields to address these two issues. To deal with the illumination variation, the bilinear attention module models global contexts and semantic dependencies between pixels by capturing second-order statistics. With them, semantic features in challenging areas can be inferred from their neighbors, and the distinction of various semantics can be boosted. To adapt to the scale variation, our adaptive receptive field module aggregates multi-scale features and selects receptive fields adaptively. Specifically, it models the semantic relationships between channels to choose feature maps with appropriate scales, changing the receptive field of subsequent convolutions. The proposed network achieves the best performance 97.47% mean IoU on Cata7. It also takes the first place on EndoVis 2017, exceeding the second place by 10.10% mean IoU.

The desired impedance dynamics can be achieved for a robot if and only if an impedance error converges to zero or a small neighborhood of zero. Although the convergence of impedance errors is important, it is seldom obtained in the existing impedance controllers due to robots modeling uncertainties and external disturbances. This brief proposes two composite learning impedance controllers (CLICs) for robots with parameter uncertainties based on whether a factorization assumption is satisfied or not. In the proposed control designs, the convergence of impedance errors, reflected by the convergence of parameter estimation errors and some auxiliary errors, is achieved by using composite learning laws under a relaxed excitation condition. The theoretical results are proven based on the Lyapunov theory. The effectiveness and advantages of the proposed CLICs are validated by simulations on a parallel robot in three cases.

The application of the fast sliding-mode control technique on solving consensus problems of fractional-order multiagent systems is investigated. The design and analysis are based on a combination of the distributed coordination theory and the knowledge of fractional-order dynamics. First, a sliding-mode manifold (surface) vector is defined, and then the fractional-order multiagent system is transformed into an integer-order (namely, first-order) multiagent system. Second, based on the fast sliding-mode control technique, a protocol is proposed for the obtained first-order multiagent system. Third, a new Lyapunov function is presented. By suitably estimating the derivative of the Lyapunov function, the reachability of the sliding-mode manifold is derived. It is proved that the exponential finite-time consensus can be achieved if the communication network has a directed spanning tree. Finally, the effectiveness of the proposed algorithms is demonstrated by some examples.

Guidewire tracking is a clinical challenge in percutaneous coronary intervention (PCI). The current practice of image-based and sensor-based tracking techniques is still limited by radiation exposure, contrast injection, device sterilization, and procedure safety. In this paper, an interventionalist-behavior-based data fusion framework is developed to provide a novel strategy for tracking guidewire motions in PCI. Four types of natural behavior were acquired from ten interventionalists while performing guidewire translation and rotation based on a simulation platform. Different numbers of behaviors are fused by a hierarchical framework with six local tracking models and three ensemble algorithms. After Gaussian mixture regression-based ensemble fusion, a three-behavior scheme can achieve average tracking errors of 1.07 ± 0.17 mm for guidewire translation, and 20.05 ± 3.36° for guidewire rotation. Relevant statistical analysis further reveals that this scheme outperforms the cases using fewer behaviors, and ensemble fusion brings significant error reduction compared with only local fusion. These meaningful results indicate the great potential of the proposed framework for promoting the improvement of guidewire tracking in PCI.

RGB-Infrared (IR) person re-identification is an important and challenging task due to large cross-modality variations between RGB and IR images. Most conventional approaches aim to bridge the cross-modality gap with feature alignment by feature representation learning. Different from existing methods, in this paper, we propose a novel and end-to-end Alignment Generative Adversarial Network (AlignGAN) for the RGB-IR RE-ID task. The proposed model enjoys several merits. First, it can exploit pixel alignment and feature alignment jointly. To the best of our knowledge, this is the first work to model the two alignment strategies jointly for the RGB-IR RE-ID problem. Second, the proposed model consists of a pixel generator, a feature generator and a joint discriminator. By playing a min-max game among the three components, our model is able to not only alleviate the cross-modality and intra-modality variations, but also learn identity-consistent features. Extensive experimental results on two standard benchmarks demonstrate that the proposed model performs favourably against state-of-the-art methods. Especially, on SYSU-MM01 dataset, our model can achieve an absolute gain of 15.4% and 12.9% in terms of Rank-1 and mAP.

Recent deep Re-ID models mainly focus on learning high-level semantic features, while failing to explicitly explore color information which is one of the most important cues for person Re-ID. In this paper, we propose a novel Color-Sensitive Re-ID to take full advantage of color information. On one hand, we train our model with real and fake images. By using the extra fake images, more color information can be exploited and it can avoid overfitting during training. On the other hand, we also train our model with images of the same person with different colors. By doing so, features can be forced to focus on the color difference in regions. To generate fake images with specified colors, we propose a novel Color Translation GAN (CTGAN) to learn mappings between different clothing colors and preserve identity consistency among the same clothing color. Extensive evaluations on two benchmark datasets show that our approach significantly outperforms state-of-the-art Re-ID models.

Both motor and cognitive function rehabilitation benefits can be improved significantly by patients' active participation. However, post-stroke patients, especially with attention-deficit disorders, can hardly engage in training for a longer time. In order to improve patients' attention focused on the training, an attention regulation system based on the brain-machine interface (BCI) and multimodal feedback is proposed for post-stroke lower limb rehabilitation. First, an interactive speed-tracking riding game is designed to increase the training challenge and patients' neural engagement. The character's riding speed, which is synchronized with patients' actual cycling speed, is displayed on the screen in real time. And patients' attention can further be enhanced when they try their best to track the reference speed curve. Second, an attention classifier is designed and trained by using subjects' EEG signals, which are acquired if they are tracking the reference speed curve or not. This classifier is finally applied to monitor subject's attention. If the subject is recognized with inadequate attention, sharp voice (auditory feedback) and red screen (visual feedback) will be given by the designed game to remind the subject to focus on the training. The contrast experiment results show that subjects' performance indicated by speed tracking accuracy and muscle activation can be improved significantly by using the attention regulation system. Moreover, the phenomenon of prominent decrease in theta rhythm and increase in beta rhythm can be found, which is consistent with previous research and further validates the feasibility of the proposed system in attention enhancement.

Interventional therapy for coronary bifurcation lesion has always been an intractable problem in percutaneous coronary intervention (PCI). Dual-guidewire detection can greatly assist physicians in interventional therapy of bifurcated lesions. Nevertheless, this task often comes with the challenges of X-ray images with low signal noise ratio (SNR) as well as the thinner structure of the guidewire compared to other interventional tools. In this paper, a fully automatic detection method based on an improved U-Net and the modified focal loss is proposed for dual-guidewire segmentation in 2D X-ray fluoroscopy, which accomplishes accurate and robust segmentation. The main contributions of this paper are twofold: (1) the proposed method not only addresses the extreme foreground-background class imbalance generated by the slender guidewire structure, but also solve the problem of misclassified examples caused by the guidewire-like structures and contrast agents; (2) the running speed is about 8 frames per second, which reaches near-real-time processing speed. Furthermore, data augmentation algorithm and transfer learning are used to further improve the performance. The proposed method was verified on clinical 2D X-ray image sequences of 30 patients, in which F1-score reached 0.932. The experiment results indicated that our approach is promising for assisting bifurcation lesion surgery.

Solving a thorny issue of real-time path planning for surgery robot in uncertain environments, a novel algorithm named bidirectional continuous tree search (BCTS) is proposed. Most partially observable markov decision process (POMDP) planners address challenges of unknown environments with discrete states, observations and actions, which are fail to automate the operative procedure. However, the BCTS method addresses the issue by handling POMDPs in continuous state, observation and action spaces. The proposed approach has a bidirectional search structure with the intent of greatly improving the calculation efficiency. Meanwhile, Bayesian optimization (BO) algorithm is considered to dynamically sample promising actions while we construct a belief tree. In view of the speed of BO process, the upper and lower bounds of the optimal action values given by fast informed bound (FIB) and point-based value iteration (PBVI) limit the search scope, so we can improve the speed of BO. In addition, we apply an optimal path planning generator, radial basis function neural network (RBFNN), to obtain a smoother trajectory. Finally, simulation of glaucoma surgery has been carried out to explore the best surgical approach. The results show that the introduced structure can effectively guide the surgery robot to perform surgical procedures and receive a real-time as well as smooth path.

Intervention surgery strongly requires information on the guide‐wire position under the monitoring of X‐ray video. Hence, the related researches such as guide‐wire detecting or tracking have become widespread. However, most of the existing methods require a lot of resources for computing or large data for training since the X‐ray videos have internal physicals such as anatomical skeleton contours and organs that are quite similar to a guide‐wire. This work presents a practical method that only requires a moderate number of training data for detecting a guide‐wire tip in an X‐ray video sequence during the percutaneous coronary intervention surgery. The method applies maximally stable extremal regions (MSER) combine with modified multi‐filters (region area range filter and stroke width variation filter) for region detection and local binary patterns (LBP) for guide‐wire recognition. The motivation for applying MSER and LBP are the robust efficacy and the low requirement of resources. The approach evaluated 20 different sequences of X‐ray videos, a total of 1295 frames. 50 selected frames were used as training templates and others to experiment. The method was successfully performed to the detecting guide‐wires with p‐value < 0.01 compared with conventional MSER methods, 93.7% average detection accuracy, and 21 fps average speed.

Previous studies on robotic rehabilitation have shown that subjects' active participation and effort involved in rehabilitation training can promote the performance of therapies. In order to improve the voluntary effort of participants during the rehabilitation training, assist-as-needed (AAN) control strategies regulating the robotic assistance according to subjects' performance and conditions have been developed. Unfortunately, the heterogeneity of patients' motor function capability in task space is not taken into account during the implementation of these controllers. In this paper, a new scheme called greedy AAN (GAAN) controller is designed for the upper limb rehabilitation training of neurologically impaired subjects. The proposed GAAN control paradigm includes a baseline controller and a Gaussian RBF network that is utilized to model the functional capability of subjects and to provide corresponding a task challenge for them. In order to avoid subjects' slacking and encourage their active engagement, the weight vectors of RBF networks evaluating subjects' impairment level are updated based on a greedy strategy that makes the networks progressively learn the maximum forces over time provided by subjects. Simultaneously, a challenge level modification algorithm is employed to adjust the task challenge according to the task performance of subjects. Experiments on 12 subjects with neurological impairment are conducted to validate the performance and feasibility of the GAAN controller. The results show that the proposed GAAN controller has significant potential to promote the subjects' voluntary engagement during training exercises.

This paper is devoted to modeling and controlling a new upper-limb rehabilitation robot which has a parallel structure. Genetic algorithm (GA) is successfully applied in parameter identification based on dynamic analysis of the parallel robot. For accurate identification, joint velocities and accelerations are computed by the Kalman filter. By taking the non-linear characteristics of frictions into account, the unknown friction parameters differ depending on directions of motion. Compared with the traditional least square estimation (LSE) based method, the proposed identification method improves performance. Further, a model-based PD computed-torque controller is designed, and the feasibility of the estimated dynamic model and controller is validated by passive training task along a circular trajectory.

The rehabilitation training with active involvement and efforts of impaired subjects can enhance the outcome of rehabilitation therapies. To promote patients' voluntary engagement, a variety of assist as needed (AAN) controllers have been proposed for robot-aided therapy. However, patients' impairment level is not taken into account in the implementation of those control schemes. In this study, a novel AAN controller is developed for upper limb rehabilitation therapy. The control paradigm uses Gaussian radial basis function (RBF) network to learn the model of subjects' motor capability in workspace. The update of weight vectors of RBF network is based on a greedy strategy, which has the potential to promote subjects' voluntary engagement by providing task challenge for them. Considering the difference in the impairment degree of patients, the assistance and impedance level of robot control is regulated based on the task performance of patients. The results of experiments at four healthy subjects verify the feasibility and adaptability of the proposed AAN controller.

Guidewire tip detection in the percutaneous coronary intervention is important. It assists physicians in navigating and is a prerequisite for clinic applications such as surgical skill assessment and robot assisted surgery. Nevertheless, accurate detection is not a trivial task due to the noisy background of the 2D X-ray image and the thin, deformable structure of the tip. In this paper, an automatic method based on cascaded convolution neural networks is proposed to segment the tip in the 2D X-ray image. The main contribution of the method is to use a cascade detection-segmentation structure to overcome the noisy background and the large deformation of the tip, achieve robust, high-precision segmentation. On the other hand, sufficient annotated training samples are necessary for convolution neural network models, while pixel-level annotating is tedious and time consuming. Accordingly, a novel data augmentation algorithm is introduced to improve the model generalization and performance, reduce the cost of data annotation. Evaluations were conducted on a dataset consisting of 22 different sequences of 2D X-ray images, 15 sequences for training and 7 sequences for evaluation. The proposed approach obtained tip precision of 0.532 pixels, F1score of 0.939, false tracking rate of 0.800%, and missing tracking rate of 9.900% on the test set. And the running speed is 4-5 frames per second.

Detection of surgical devices, in particular of guide-wire detection, is prerequisite during image-guided navigation in percutaneous coronary intervention (PCI). Guide-wire detection is a challenging task for following reasons: (i) X-ray images have a low signal-to-noise rate (SNR); (ii) there is a high similarity between guide-wires and some other adjacent anatomical skeletons' contours; (iii) guide-wires have various shapes and their motion is complex and nonlinear. Traditionally, guide-wires are detected using curve fitting method, and third-order B-spline curve model is always used to fit guide-wires, while B-spline fitting method has some obvious shortcomings such as it is a semi-automatic method which needs manual initialization, and it is not a real-time method because of high computational complexity. Recently, with the availability of large annotated datasets and the accessibility of hardware resources with GPUs, it is succeeded in detecting general objects with convolutional neural networks (ConvNet). In this paper, we present a novel image-based fully-automatic and real-time approach with ConvNet for guide-wires detection. ConvNet method is robust to guide-wires' various poses and other structures' effects. We evaluate our method on 22 different sequences of X-ray images. The detection accuracy evaluated by average precision (AP) reaches 89.2% and the detection speed achieves 40fps. Our experiment result shows a promising for accurate and real-time guide-wires detection in PCI navigation with ConvNet model.

For the treatment of cardiovascular diseases, clinical success of percutaneous coronary intervention is highly dependent on natural technical skills and dexterous manipulation strategies of surgeons. However, the increasing used robotic surgical systems have been designed without considering manipulation techniques, especially surgical behaviors and motion patterns. This has driven research towards exploitation of natural manipulation skills in recent years. In this paper, natural guidewire manipulations are analyzed and predicted using an sEMG-based nonlinear autoregressive neural network with exogenous inputs. The relationship between natural endovascular manipulation and guidewire rotation is built through the network. Two experiments at different rotational speed were performed to verify the effectiveness and robustness of the applied model. The experimental results show that the average predictive root mean error of five subjects is 15.61° at the low speed and 21.85° at the high speed. These favorable results could be of interest to improve existing robotic surgical systems.

In this paper, we address the problem of online RGB-D tracking where the target object undergoes significant appearance changes. To sufficiently exploit the color and depth cues, we propose a novel RGB-D tracking framework (DLS) that simultaneously builds the target 2D appearance model and 3D distribution model. The framework decomposes the tracking task into detection, learning and segmentation. The detection and segmentation components locate the target collaboratively by using the two target models. An adaptive depth histogram is proposed in the segmentation component to efficiently locate the target in depth frames. The learning component estimates the detection and segmentation errors, updates the target models from the most confident frames by identifying two kinds of distractors: potential failure and occlusion. Extensive experimental results on a large-scale benchmark dataset show that the proposed method performs favourably against state-of-the-art RGB-D trackers in terms of efficiency, accuracy, and robustness.

This paper studies the containment control of multiagent systems (MASs) with multiple dynamic leaders in both continuous-time domain and discrete-time domain. The leaders' motions are described by the nth-order polynomial trajectories. This setting makes practical sense because given some critical points, the leaders' trajectories are usually planned by the polynomial interpolations. In order to drive all followers into the convex hull spanned by the leaders, a PIn-type containment algorithm is proposed (P and I are short for proportional and integral, respectively; Inimplies that the algorithm includes up to the n-thorder integral terms). It is theoretically proved that the PIn-type containment algorithm is able to solve the containment problem of MASs where the followers are described by any order integral dynamics. Compared to the previous results on the MASs with dynamic leaders, the distinguished features of this paper are that: 1) the containment problem is studied not only in the continuoustime domain but also in the discrete-time domain while most existing results only work in the continuous-time domain; 2) to deal with the leaders with the nth-order polynomial trajectories, existing results require the follower's dynamics to be the (n + 1)th-order integral while the followers considered in this paper can be described by any-order integral dynamics; 3) the “sign” function is not employed in the proposed algorithm, which avoids the chattering phenomenon; and 4) both disturbance and measurement noise are taken into account. Finally, some simulation examples are given to demonstrate the effectiveness of the proposed algorithm.

In this paper, a robot, namely iLeg, is designed for the purpose of rehabilitation of patients with hemiplegia or paraplegia. The iLeg is composed of one reclining seat and two leg orthoses, and each leg orthosis has three degrees of freedom, which correspond to the hip, knee, and ankle. Based on this robotic system, two controllers, i.e., passive training controller and active training controller, are proposed. The former takes advantage of the proportional-integral control method to solve the trajectory tracking problem, and the latter employs the surface electromyography signals to achieve active training. Two simplified impedance controllers, i.e., damping-type velocity controller and spring-type position controller, are designed for active training. A perceptron neural network detects movement intentions. The performance of the controllers was investigated with one able-bodied male. The results showed that the leg orthosis tracked the predefined trajectory based on the passive training controller, with the error rates of 0.45%, 0.44%, and 0.27%, respectively, for the hip, knee, and ankle. The active training controller whose loop rate is 6.67 Hz can move the leg orthosis smoothly, and the average recognition error of the perceptron neural network is less than 5%.

The optimal formation problem of multirobot systems is solved by a recurrent neural network in this paper. The desired formation is described by the shape theory. This theory can generate a set of feasible formations that share the same relative relation among robots. An optimal formation means that finding one formation from the feasible formation set, which has the minimum distance to the initial formation of the multirobot system. Then, the formation problem is transformed into an optimization problem. In addition, the orientation, scale, and admissible range of the formation can also be considered as the constraints in the optimization problem. Furthermore, if all robots are identical, their positions in the system are exchangeable. Then, each robot does not necessarily move to one specific position in the formation. In this case, the optimal formation problem becomes a combinational optimization problem, whose optimal solution is very hard to obtain. Inspired by the penalty method, this combinational optimization problem can be approximately transformed into a convex optimization problem. Due to the involvement of the Euclidean norm in the distance, the objective function of these optimization problems are nonsmooth. To solve these nonsmooth optimization problems efficiently, a recurrent neural network approach is employed, owing to its parallel computation ability. Finally, some simulations and experiments are given to validate the effectiveness and efficiency of the proposed optimal formation approach.

In order to implement model-based recognition of human motion intention, dynamics modeling and identification of a lower limb rehabilitation robot named iLeg is investigated. Due to the relatively strong motion constraints, the traditional identification methods become insufficient for iLeg in three aspects: (1) the coupling factors among joints have not been considered in the traditional joint friction models, which makes the structural error and the torque estimation errors relatively large; (2) because of the small and complicated feasible region caused by the motion constraints, the traditional initialization strategy, for searching the valid initial solutions of the optimization problem for the exciting trajectories, becomes very inefficient; and (3) the condition number of the observation matrix, calculated from the preliminary dynamic model and the associated optimized exciting trajectory, is too large for the identification, and, however, further reduction of the condition number has not been considered in the literature. Therefore, corresponding contributions are presented to overcome the limitation. First, the coupling factors among joints are considered in the joint friction model by using the Palmgren empirical formulation and a polynomial fitting method. Then, an indirectly generating strategy is designed, by which the valid initial solutions of the optimization problem can be found with good efficiency. Moreover, a recursive optimization method based on the optimization of the dynamic model and the exciting trajectories, is proposed to further reduce the condition number. Finally, the performance of the proposed methods is demonstrated by several experiments.

The percutaneous coronary interventions (PCI) require complex operating skills of the interventional devices and make the surgeons being exposed to heavy X-ray radiation. Accurate delivery of the interventional devices and avoiding the radiation are especially important for the surgeons. This paper presents a novel dedicated dual-finger robotic hand (DRH) and a console to assist the surgeons to deliver the interventional devices in PCIs. The system is designed in the master-slave way which helps the surgeons to reduce the exposure to radiation. The mechanism of the DRH is bio-inspired and motions are decoupled in kinematics. In PCI procedures, the accuracy of the guidewire delivery and the catheter tip placement have significant effects on the surgical results. The performances of the DRH in delivering the guidewire and the balloon/stent catheter were evaluated by three surgical manipulations. The results show that the DRH has the ability to deliver the guidewire and the balloon/stent catheter precisely.

sEMG (surface electromyography) signals have been used as human-machine interface to control robots or prostheses in recent years. sEMG-based torque estimation is a widely research methodology to obtain human motion intention. Most researches focus on improving the accuracy of sEMG-torque models, which often makes them complicated and confined in the laboratory research. However, an accurate estimation of muscle torque could be unnecessary to perform the robot-assisted rehabilitation training. This paper proposes a practical method to estimate the net muscle torques of lower limbs using sEMG, which can be used to implement a real-time coordinated active training with iLeg-a horizontal exoskeleton for lower limb rehabilitation developed at our laboratory. Two three-layer back propagation (BP) neural networks are built to estimate the net muscle torques at hip and knee joints respectively. Experimental results show that the well-trained neural networks estimate the user's motion intention in real-time, and can assist the user to perform an active training with iLeg.

The population of disabled stroke survivors is increasing sharply over the world, and robot-assisted training has been proved to be effective to help motor relearning and alleviate the shortage of physical therapists. This paper presents the design details of a novel upper limb rehabilitation robot named CASIA-ARM, which can assist poststroke patients to perform shoulder and elbow rehabilitation training in the horizontal plane, and provide both force and visual feedback to the patient: 1) A five-bar closed-chain structure is designed to realize a stiff and low-inertial mechanism. Workspace and singularity analysis is given, which guarantees that the workspace is large enough for upper limb moving and no singularity in the workspace. 2) Cable transmission, motor current controller and impedance control method guarantee the transparent and compliant interaction between the human and the robot. 3) Implementation examples of passive training based on position control and active training based on impedance control are presented in detail. 4) In order to evaluate the robot's applicability and performance, examples of trajectory tracking and active reaching tasks are used in this study, and the preliminary test results are also given (tracking error: 0.86±0:42 mm, force control error: 1.41±0.79 N in the X direction, and 1.22±0.91 N in the Y direction).

A lower limb rehabilitation robot, namely iLeg, has been developed recently. Since active exercises have been proven to be effective for neurorehabilitation and motor recovery, they are suggested to be implemented on iLeg. To this goal, patients' motion intention should be recognized. Therefore, a method based on the dynamic model of the human-robot interface (HRI) is designed to recognize the human motion intention. This paper is devoted to modeling and identifying the dynamics of the HRI. Firstly, the dynamic model of the HRI is designed by combining the dynamic models of the human leg and iLeg, where the human leg dynamic model (HLDM) is mainly concerned. By considering the motion trajectories during the rehabilitation exercises provided by iLeg, the human leg can be taken as a manipulator with two degrees of freedom; meanwhile, the joint angles and torques of the human leg can be measured indirectly by using the position and torque sensors mounted on the joints of iLeg. As a result, an 8-parameter HLDM can be designed by using the Lagrangian method. Then, the dynamic model of the HRI is identified by respectively and independently identifying the undetermined dynamic parameters of iLeg and the HLDM, where the dynamic parameters of the HLDM are mainly considered. Finally, the feasibility of the dynamic model of the HRI is validated by experiments.

The paper is a feasibility analysis of using the recently introduced by one of the authors spiking neural networks architecture NeuCube for modelling and recognition of complex EEG spatio-temporal data related to both physical and intentional (imagined) movements. The preliminary experiments reported in the paper suggest that NeuCube is much more efficient for the task than standard machine learning techniques, resulting in high recognition accuracy, a better adaptability to new data, a better interpretation of the models, leading to a better understanding of the brain data and the processes that generated it.

Visual tracking is a popular and challenging topic in computer vision and robotics. Owing to changes in the appearance of the target and complicated variations that may occur in various scenes, online learning scheme is necessary for advanced visual tracking framework to adopt. This paper briefly introduces the challenges and applications of visual tracking and focuses on discussing the state‐of‐the‐art online‐learning‐based tracking methods by category. We provide detail descriptions of representative methods in each category, and examine their pros and cons. Moreover, several most representative algorithms are implemented to provide quantitative reference. At last, we outline several trends for future visual tracking research.

Faces convey a wealth of social signals, including race, expression, identity, age and gender, all of which have attracted increasing attention from multi-disciplinary research, such as psychology, neuroscience, computer science, to name a few. Gleaned from recent advances in computer vision, computer graphics, and machine learning, computational intelligence based racial face analysis has been particularly popular due to its significant potential and broader impacts in extensive real-world applications, such as security and defense, surveillance, human computer interface (HCI), biometric-based identification, among others. These studies raise an important question: How implicit, non-declarative racial category can be conceptually modeled and quantitatively inferred from the face? Nevertheless, race classification is challenging due to its ambiguity and complexity depending on context and criteria. To address this challenge, recently, significant efforts have been reported toward race detection and categorization in the community. This survey provides a comprehensive and critical review of the state-of-the-art advances in face-race perception, principles, algorithms, and applications. We first discuss race perception problem formulation and motivation, while highlighting the conceptual potentials of racial face processing. Next, taxonomy of feature representational models, algorithms, performance and racial databases are presented with systematic discussions within the unified learning scenario. Finally, in order to stimulate future research in this field, we also highlight the major opportunities and challenges, as well as potentially important cross-cutting themes and research directions for the issue of learning race from face.

Errors in the above-named article [ibid., vol. 41, no. 4, pp. 950-963, Aug. 2011] are noted for pages 952, 955, and 956.

The trajectory tracking problem of a closed-chain five-bar robot is studied in this paper. Based on an error transformation function and the backstepping technique, an approximation-based tracking algorithm is proposed, which can guarantee the control performance of the robotic system in both the stable and transient phases. In particular, the overshoot, settling time, and final tracking error of the robotic system can be all adjusted by properly setting the parameters in the error transformation function. The radial basis function neural network (RBFNN) is used to compensate the complicated nonlinear terms in the closed-loop dynamics of the robotic system. The approximation error of the RBFNN is only required to be bounded, which simplifies the initial "trail-and-error" configuration of the neural network. Illustrative examples are given to verify the theoretical analysis and illustrate the effectiveness of the proposed algorithm. Finally, it is also shown that the proposed approximation-based controller can be simplified by a smart mechanical design of the closed-chain robot, which demonstrates the promise of the integrated design and control philosophy.

Ongoing efforts within neuroscience and intelligent system have been directed toward the building of artificial computational models using simulated neuron units as basic building blocks. Such efforts, inspired in the standard design of traditional neural networks, are limited by the difficulties arising from single functional performance and computational inconvenience, especially when modeling large scale, complex and dynamic processes such as cognitive recognition. Here, we show that there is a different form of implementing cortex-like mechanism, the motivation comes directly from recent pioneering works on detailed functional decomposition analysis of the visual cortex and developments on spiking neural networks (SNNs), a promising direction for neural networks, as they utilize information representation as trains of spikes, embedded with spatiotemporal characteristics. A practical implementation is presented, which can be simply described as cortical-like feed-forward hierarchy using biologically plausible neural system. As a proof of principle, a prototype model has been testified on the platform of several facial expression dataset. Of note, small structure modifications and different learning schemes allow for implementing more complicated decision system, showing great potential for discovering implicit pattern of interest and further analysis. Our results support the approach of using such hierarchical consortia as an efficient way of complex pattern analysis task not easily solvable using traditional, single functional way of implementations.

A recurrent neural network is proposed for solving the non-smooth convex optimization problem with the convex inequality and linear equality constraints. Since the objective function and inequality constraints may not be smooth, the Clarke's generalized gradients of the objective function and inequality constraints are employed to describe the dynamics of the proposed neural network. It is proved that the equilibrium point set of the proposed neural network is equivalent to the optimal solution of the original optimization problem by using the Lagrangian saddle-point theorem. Under weak conditions, the proposed neural network is proved to be stable, and the state of the neural network is convergent to one of its equilibrium points. Compared with the existing neural network models for non-smooth optimization problems, the proposed neural network can deal with a larger class of constraints and is not based on the penalty method. Finally, the proposed neural network is used to solve the identification problem of genetic regulatory networks, which can be transformed into a non-smooth convex optimization problem. The simulation results show the satisfactory identification accuracy, which demonstrates the effectiveness and efficiency of the proposed approach.

A finite-time attitude tracking control scheme is proposed for spacecraft using terminal sliding mode and Chebyshev neural network (NN) (CNN). The four-parameter representations (quaternion) are used to describe the spacecraft attitude for global representation without singularities. The attitude state (i.e., attitude and velocity) error dynamics is transformed to a double integrator dynamics with a constraint on the spacecraft attitude. With consideration of this constraint, a novel terminal sliding manifold is proposed for the spacecraft. In order to guarantee that the output of the NN used in the controller is bounded by the corresponding bound of the approximated unknown function, a switch function is applied to generate a switching between the adaptive NN control and the robust controller. Meanwhile, a CNN, whose basis functions are implemented using only desired signals, is introduced to approximate the desired nonlinear function and bounded external disturbances online, and the robust term based on the hyperbolic tangent function is applied to counteract NN approximation errors in the adaptive neural control scheme. Most importantly, the finite-time stability in both the reaching phase and the sliding phase can be guaranteed by a Lyapunov-based approach. Finally, numerical simulations on the attitude tracking control of spacecraft in the presence of an unknown mass moment of inertia matrix, bounded external disturbances, and control input constraints are presented to demonstrate the performance of the proposed controller.

In this paper, a motion controller for a pan-tilt camera mounted on an autonomous helicopter is presented. The motion planner is designed according to the kinematics of the pan-tilt camera. However, the solution of the inverse kinematics of the pan-tilt camera is not unique. To deal with this situation, a decision maker is designed. The decision maker makes use of a cost function to decide which solution should be chose. And the error signal is defined as the difference between pan-tilt joint angles and the desired joint angles. The dynamics of the error system is derived, and proportional controllers are designed according to the error dynamics to control the pan and tilt joints respectively. This system is validated in a virtual reality environment, and the simulation results show that this system can track the target successfully.

Local image features have been proven to be a powerful way to describe pattern of interest, both from single objects and complex scenes. While learning from images represented by local features is challenging, recent publications and developments in object recognition has shown that significant performance achievements can be achieved by carefully combining multi-level, coarse-to-fine, sparsely distributed feature encodings, and kernel based learning methods, which defines a generalized similarity measure among data using multiple kernel functions instead of a single one, also known as multiple kernel learning (MKL). In this paper we show that the Kernel ICA descriptors based MKL supervised learning approach perform better than other descriptors for object recognition, since the ICA-based representation is localized. In low-level feature extraction, ICA produces independent image bases that emphasize edge information in the image data. In high-level classification, MKL classifies the ICA features as discriminative components. We demonstrate our algorithm on different databases for recognition tasks, showing that the proposed method is accurate and more efficient than current approaches.

In this paper, the firing rate of the neuron based on temporal coincidence coding is estimated for correlation detection. Two cases are considered: the independent inputs are stochastic spike trains that are modeled by the homogeneous Poisson process or the renew process. The situation that the inputs are correlated is also considered and the conditions for the neuron to detect the correlation among inputs are discussed. The results are demonstrated by the simulations.

A neural-network-based adaptive approach is proposed for the leader-following control of multiagent systems. The neural network is used to approximate the agent's uncertain dynamics, and the approximation error and external disturbances are counteracted by employing the robust signal. When there is no control input constraint, it can be proved that all the following agents can track the leader's time-varying state with the tracking error as small as desired. Compared with the related work in the literature, the uncertainty in the agent's dynamics is taken into account; the leader's state could be time-varying; and the proposed algorithm for each following agent is only dependent on the information of its neighbor agents. Finally, the satisfactory performance of the proposed method is illustrated by simulation examples.

This paper investigates the problem of output feedback attitude control of an uncertain spacecraft. Two robust adaptive output feedback controllers based on Chebyshev neural networks (CNN) termed adaptive neural networks (NN) controller-I and adaptive NN controller-II are proposed for the attitude tracking control of spacecraft. The four-parameter representations (quaternion) are employed to describe the spacecraft attitude for global representation without singularities. The nonlinear reduced-order observer is used to estimate the derivative of the spacecraft output, and the CNN is introduced to further improve the control performance through approximating the spacecraft attitude motion. The implementation of the basis functions of the CNN used in the proposed controllers depends only on the desired signals, and the smooth robust compensator using the hyperbolic tangent function is employed to counteract the CNN approximation errors and external disturbances. The adaptive NN controller-II can efficiently avoid the over-estimation problem (i.e., the bound of the CNNs output is much larger than that of the approximated unknown function, and hence, the control input may be very large) existing in the adaptive NN controller-I. Both adaptive output feedback controllers using CNN can guarantee that all signals in the resulting closed-loop system are uniformly ultimately bounded. For performance comparisons, the standard adaptive controller using the linear parameterization of spacecraft attitude motion is also developed. Simulation studies are presented to show the advantages of the proposed CNN-based output feedback approach over the standard adaptive output feedback approach.

A dual neural-network method for the coordination of kinematically redundant robots is proposed in this paper. The performance criteria for single robots provided by Nedungadi and Kazerounian are generalized to a multicriteria form for the coordinated-manipulation system composed of multiple serial manipulators. By optimizing the local joint torques and generalized forces applied on the object/workpiece using a designed weighting matrix, the proposed method achieves the global stability during the coordinated-manipulation process. Moreover, the proposed algorithm has an explicit physical meaning, i.e., both the global kinetic energy of the coordination system and the two-norm of the generalized forces applied on the object are minimized simultaneously. In addition, the physical limits of both joint torques and the generalized forces applied on the object are considered, which makes the original coordination problem become a complicated optimization problem subject to both equality and inequality constraints. Compared with numerical optimization algorithms used in existing literatures, the dual neural-network method has better computational capability to deal with the complicated optimization problem. Finally, illustrative examples are given to show that the proposed method is effective and efficient for the multirobot coordinated-manipulation system.

This paper focuses on deriving a robust backstepping control approach to solve the active vibration isolation problem using a Stewart platform. The dynamics of the Stewart platform driven by the linear voice coil motors is developed with the Newton-Euler method. By fully considering the characteristics of vibration isolation, the properties of the dynamics of the Stewart platform are applied to transform the coupled dynamics into six independent single-input single-output (SISO) channels. Furthermore, in the procedure of controller design, the influence factors of vibration isolation are taken into account, such as the parameter perturbation and the unmodeled dynamics, etc. Meanwhile, high-gain design method is employed to deal with the problem introduced by input unmodeled dynamics of the system. It is demonstrated that a sufficiently small L2gain from disturbance to output can be obtained in Lyapunov synthesis. The simulation results show that the controller can effectively attenuate low frequency vibrations in six degrees of freedom (DOFs) and a satisfactory vibration isolation performance can be achieved.

A recurrent neural network is proposed to deal with the convex optimization problem. By employing a specific nonlinear unit, the proposed neural network is proved to be convergent to the optimal solution in finite time, which increases the computation efficiency dramatically. Compared with most of existing stability conditions, i.e., asymptotical stability and exponential stability, the obtained finite-time stability result is more attractive, and therefore could be considered as a useful supplement to the current literature. In addition, a switching structure is suggested to further speed up the neural network convergence. Moreover, by using the penalty function method, the proposed neural network can be extended straightforwardly to solving the constrained optimization problem. Finally, the satisfactory performance of the proposed approach is illustrated by two simulation examples.

The dual heuristic programming (DHP) approach has a superior ability for solving approximate dynamic programming problems in adaptive critic designs (ACD). The common approaches applied in the DHP are design the multilayer feedforward neural networks (MLFNN) as the differential model of the plant for training the critic and action networks. However, the problems of overfitting and premature convergence to local optima usually pose great challenges in the practice of MLFNNs during the training procedure. In this paper a least squares support vector machine (LS-SVM) regressor optimized by particle swarm algorithm (PSO) is proposed for generating the control actions and the learning rules for the critic and action networks. PSO is introduced to select the LS-SVM's hyper-parameters. The introduction of the SVM based training mechanism imparts the developed algorithm with inherent capacity for combating the overfitting problem as well as showing relatively high efficiency in converging to the optima. Simulation on the balancing of a cart pole plant shows that the proposed learning strategy is verified as faster convergence and higher efficiency as compared to traditional BP based adaptive dynamic programming approaches.

In this paper, a radial basis function disturbance observer (RBFDO) based controller is developed to solve the control problem of an electrically driven Stewart platform for multiple degree-of-freedom (DOF) active vibration isolation. The RBFDO's are employed to monitor the modeling errors and external disturbances, etc. And on-line tuning rules for updating the weights of the RBFDO's are designed based on the e1-modification algorithm. Meanwhile, by considering the dynamics of the Stewart platform and its voice coil actuators, the developed RBFDO's are integrated with the backstepping method to design the active vibration isolation controller. In the presence of external vibrations and model uncertainties, the uniformly ultimately boundedness of the stabilization errors and the weight estimation errors can be guaranteed by the Lyapunov theory. Finally, simulation results demonstrate the proposed controller can effectively attenuate external low-frequency vibrations in six DOFs.

In this paper, a delayed projection neural network is proposed for solving a class of linear variational inequality problems. The theoretical analysis shows that the proposed neural network is globally exponentially stable under different conditions. By the proposed linear matrix inequality (LMI) method, the monotonicity assumption on the linear variational inequality is no longer necessary. By employing Lagrange multipliers, the proposed method can resolve the constrained quadratic programming problems. Finally, simulation examples are given to demonstrate the satisfactory performance of the proposed neural network.

A robust adaptive control approach is proposed to solve the consensus problem of multiagent systems. Compared with the previous work, the agent's dynamics includes the uncertainties and external disturbances, which is more practical in real-world applications. Due to the approximation capability of neural networks, the uncertain dynamics is compensated by the adaptive neural network scheme. The effects of the approximation error and external disturbances are counteracted by employing the robustness signal. The proposed algorithm is decentralized because the controller for each agent only utilizes the information of its neighbor agents. By the theoretical analysis, it is proved that the consensus error can be reduced as small as desired. The proposed method is then extended to two cases: Agents form a prescribed formation, and agents have the higher order dynamics. Finally, simulation examples are given to demonstrate the satisfactory performance of the proposed method.

Recent publications and developments based on SVM have shown that using multiple kernels instead of a single one can enhance interpretability of the decision function and improve classifier performance, which motivates researchers to explore the use of homogeneous model obtained as linear combinations of kernels. However, the use of multiple kernels faces the challenge of choosing the kernel weights, and an increased number of parameters that may lead to overfitting. In this paper we show that MKL problem with a enhanced spatial pyramid match kernel can be solved efficiently using projected gradient method. Weights on each kernel matrix (level) are included in the standard SVM empirical risk minimization problem with a L2constraint to encourage sparsity. We demonstrate our algorithm on classification tasks, which is based on a linear combination of the proposed kernels computed at multiple pyramid levels of image encoding, and we show that the proposed method is accurate and significantly more efficient than current approaches.

An adaptive neural network controller is proposed to deal with the task-space tracking problem of manipulators with kinematic and dynamic uncertainties. The orientation of manipulator is represented by the unit quaternion, which avoids singularities associated with three-parameter representation. By employing the adaptive Jacobian scheme, neural networks, and backstepping technique, the torque controller is obtained which is demonstrated to be stable by the Lyapunov approach. The adaptive updating laws for controller parameters are derived by the projection method, and the tracking error can be reduced as small as desired. The favorable features of the proposed controller lie in that: (1) the uncertainty in manipulator kinematics is taken into account; (2) the unit quaternion is used to represent the end-effector orientation; (3) the “linearity-in-parameters” assumption for the uncertain terms in dynamics of manipulators is no longer necessary; (4) effects of external disturbances are also considered in the controller design. Finally, the satisfactory performance of the proposed approach is illustrated by simulation results on a PUMA 560 robot.

A recurrent neural network is proposed to deal with the nonlinear variational inequalities with linear equality and nonlinear inequality constraints. By exploiting the equality constraints, the original variational inequality problem can be transformed into a simplified one with only inequality constraints. Therefore, by solving this simplified problem, the neural network architecture complexity is reduced dramatically. In addition, the proposed neural network can also be applied to the constrained optimization problems, and it is proved that the convex condition on the objective function of the optimization problem can be relaxed. Finally, the satisfactory performance of the proposed approach is demonstrated by simulation examples.

Object recognition and categorization are considered as fundamental steps in the vision based navigation for inspection robot as it must plan its behaviors based on various kinds of obstacles detected from the complex background. However, current approaches typically require some amount of supervision, which is viewed as a expensive burden and restricted to relatively small number of applications in practice. For this purpose, we present an computationally efficient approach that does not need supervision and is capable of learning object categories automatically from unlabeled images which are represented by an set of local features, and all sets are clustered according to their partial-match feature correspondences, which is done by a enhanced Spatial Pyramid Match algorithm (E-SPK). Then a graph-theoretic clustering method is applied to seek the primary grouping among the images. The consistent subsets within the groups are identified by inferring category templates. Given the input, the output of the approach is a partition of the images into a set of learned categories. We demonstrate this approach on a field experiment for a powerline inspection robot.

In this paper, we propose a new computer aided diagnosis method of pulmonary nodules in X-ray CT images to reduce false positive (FP) rate under high true positive (TP) rate conditions. An essential core of the method is to extract and combine two novel and effective features from the raw CT images: One is orientation features of nodules in a region of interest (ROI) extracted by a Gabor filter, while the other is variation of CT values of the ROI in the direction along body axis. By using the extracted features, a principal component analysis technic and any pattern recognition technics such as neural network approaches can then used to discriminate between nodule and non-nodule images. Simulation results show that discrimination performance using the proposed features is extremely improved compared to that of the conventional method.

This paper proposes a new forward passageway (FP) based real-time collision-free target tracking approach for a mobile robot with local sensing. After the position of the target is estimated and localized in robot coordinate system through the combination of vision system and encoder, the sonar information and the target position are converted to a uniform environment model framework called decision-making space. Based on the space, a FP based decision-making is given to endow the robot with the ability to avoid possible obstacles and track the target in unknown environments. Experiment results show the validity of the proposed approach.

An adaptive control approach is proposed to deal with the multi-manipulator system consensus problem based on the multi-agent theory. In the current multi-agent literature, agents are assumed to have determined models. However, the real manipulator's dynamics contains uncertain parameters. According to the “linearity-in-parameters” property, the adaptive updating law for uncertain dynamics parameters is derived by the projection method. Then, a decentralized controller is designed based on the backstepping scheme, which only utilizes the information of connected manipulators. By the proposed controller, all the manipulators' joints move towards the same configuration to achieve certain coordination tasks. In addition, performance of the control system is analyzed by the Lyapunov method, and the consensus error is proved to approach zero. Finally, the effectiveness of the proposed scheme is illustrated by simulations on a multiple two-link manipulators system.

A controller is proposed for the robust backstepping control of a class of nonlinear pure-feedback systems using fuzzy logic. The proposed control scheme utilizes fuzzy logic systems to learn the behavior of the unknown plant dynamics. Filtered signals are employed to circumvent algebraic loop problems encountered in the implementation of the usual controllers, and the approximation errors can be efficiently counteracted by employing smooth robust compensators. Most importantly, the uniform ultimate boundedness of all signals in the closed-loop system can be guaranteed, anda prioriknowledge of the plant dynamics is no longer required. Furthermore, the proposed method can be used for adaptive control of a large class of single-input--single-output nonlinear systems in both strict-feedback and pure-feedback forms, and has great potential in many diverse applications. The performance of the proposed approach is demonstrated through three simulation examples, including one nonlinear pure-feedback and two nonlinear strict-feedback systems.

This paper presents an approach for sonar feature map building. The approach is composed of extracting features at the data-level fusion stage and fusing the extracted features with the registered features in the map at the feature-level fusion stage. A data-level fusion model, termed three measurements association model (TMAM), has been developed for associating three measurements with a line or a point feature. By use of TMAM, different sets of measurements obtained from a single sonar sensor at consecutive steps are associated with the line and point features. Subsequently, the parameters of the identified features are estimated by use of the iterated least square estimation method. Finally, when a feature is extracted, a simple feature-level fusion strategy is used to update the map. The proposed approach has been tested both in simulation and on real data.

A recurrent neural network is proposed for solving non-smooth nonlinear programming problems, which can be regarded as a generalization of the smooth nonlinear programming neural network used in (X.B. Gao, 2004). Based on the non-smooth analysis and the theory of differential inclusions, the proposed neural network is demonstrated to be globally convergent to the exact optimal solution of the original optimization problem. Compared with the existing neural networks, the proposed approach takes both equality and inequality constraints into account, and no penalty parameters have to be estimated beforehand. Therefore, it can solve a larger class of non-smooth programming problems. Finally, several illustrative examples are given to show the effectiveness of the proposed neural network.

A recurrent neural network for the optimal control of a group of interconnected dynamic systems is presented in this paper. On the basis of decomposition and coordination strategy for interconnected dynamic systems, the proposed neural network has a two-level hierarchical structure: several local optimization subnetworks at the lower level and one coordination subnetwork at the upper level. A goal-coordination method is used to coordinate the interactions between the subsystems. By nesting the dynamic equations of the subsystems into their corresponding local optimization subnetworks, the number of dimensions of the neural network can be reduced significantly. Furthermore, the subnetworks at both the lower and upper levels can work concurrently. Therefore, the computation efficiency, in comparison with the consecutive executions of numerical algorithms on digital computers, is increased dramatically. The proposed method is extended to the case where the control inputs of the subsystems are bounded. The stability analysis shows that the proposed neural network is asymptotically stable. Finally, an example is presented which demonstrates the satisfactory performance of the neural network.

High-voltage power transmission line inspection robot must plan its behavior to detect the obstacles from the complex background according to their types when it is crawling along the power transmission line in order to negotiate reliably. In most cases, robot fulfills the task by its vision system. However, motion blur due to camera motion caused by wind or other unknown causes can significantly degrade the quality of the image acquired. This is a typical kind of the so called image restoration problem, which is a hard problem since no prior knowledge of the motion is available. For this purpose, a novel approach for image restoration is proposed. The restoration procedure consists of two stages: estimation of blur function parameters and reconstruction of images. Image degradation model is proposed first to identify blur function parameters, then a recurrent neural network is used to restore the blurred image. Experiments on real blurred images on power transmission line prove the feasibility and reliability of this algorithm. Our experiments show that the restoration procedure consumes only small amount of computation time.

In this paper, we analyze neural spike dynamics of a double feedback neural unit (DFNU). An essential emphasis of the analysis is on use of the DFNU's simple formulations that can provide quantitative analytic results. Comparing dynamics of Hodgkin-Huxley model to that of the DFNU, it is shown that dynamics of the DFNU is also physiologically plausible under a condition. The results suggest that high-frequency firings are relatively appropriate for a neural informational carrier due to the reliability and robustness to noisy inputs. To realize such reliable spike communication, we improved the DFNU's performance by using extra noisy inputs with appropriate amplitudes. Simulation studies show that there is optimal region of the amplitude that makes the DFNU possess the noise-enhanced reliable communication ability as similar to stochastic resonance phenomena.

Real-time control of multi-robot coordination system has attracted a lot of attention in recent years. Traditional numerical algorithm is ineffective to perform this task. In this paper, a dual neural network approach is applied to resolve the coordination problem of two redundant robots. By this approach, the joint torque and distributed load can be obtained by optimizing a multiple criteria, and the physical limits of the joint torque and distributed load can be also incorporated into the control scheme. The dual neural network has a simple structure which is composed of only one layer of neuron array. The network configuration is updated by the command signals of desired acceleration of the grasped object, and the output of the network is the manipulator's joint torque. A simulation example is presented to demonstrate the effectiveness of the dual neural network method.

Inspection robot must plan its behavior to detect the obstacles from the complex background according to their types when it is crawling along the power transmission line in order to negotiate reliably. However, in most instances, detecting the obstacles from the complex background is a hard task. For this purpose, a novel and fast visual obstacle recognition algorithm is designed based on the structure of the 220 KV power transmission line. Basic principle and architecture of the algorithm are given. By this approach, three typical obstacles on the power transmission line such as insulator strings, counterweights and suspension clamps can be recognized with high accuracy. Experiments in the real power transmission line show its effectiveness. This method can contribute to the process of the mobile robot negotiating obstacles

In this paper, a new type of six-degrees of freedom (DOF) flexible parallel mechanism (FPM) is presented. This type of parallel mechanism possesses several favorable properties: (1) its number of DOFs is independent of the number of serial chains which make up the mechanism; (2) it has no kinematical singularities; (3) it is designed to move on rails, and therefore its workspace is much larger than that of a conventional parallel manipulator; and (4) without changing the number of DOFs and the kinematics of the mechanisms, the number of the serial chains can be reconfigured according to the needs of the tasks. These properties make the mechanism very preferable in practice, especially for such tasks as joining huge ship blocks, in which the manipulated objects vary dramatically both in weights and dimensions. Furthermore, the mechanism can be used as either a fully actuated system or an underactuated system. In the fully actuated case, the mechanism has six DOF motion capabilities and manipulation capabilities. However, in the underactuated case, the mechanism still has six DOF motion capabilities, but it has only five DOF manipulation capabilities. In this paper, both the inverse and forward kinematics are studied and expressed in a closed form. The workspace and singularity analysis of the mechanism are also presented. An example is presented to illustrate how to calculate the kinematics of the mechanism in both fully-actuated and underactuated cases. Finally, an application of such a mechanism to manufacturing industry is introduced.