In situ tissue biopsy with an endoluminal catheter is an efficient approach for disease diagnosis, featuring low invasiveness and few complications. However, the endoluminal catheter struggles to adjust the biopsy direction by distal endoscope bending or proximal twisting for tissue sampling within the tortuous luminal organs, due to friction-induced hysteresis and narrow spaces. Here, we propose a pneumatically-driven robotic catheter enabling the adjustment of the sampling direction without twisting the catheter for an accurate in situ omnidirectional biopsy. The distal end of the robotic catheter consists of a pneumatic bending actuator for the catheter’s deployment in torturous luminal organs and a pneumatic rotatable biopsy mechanism (PRBM). By hierarchical airflow control, the PRBM can adjust the biopsy direction under low airflow and deploy the biopsy needle with higher airflow, allowing for rapid omnidirectional sampling of tissue in situ. This paper describes the design, modeling, and characterization of the proposed robotic catheter, including repeated deployment assessments of the biopsy needle, puncture force measurement, and validation via phantom tests. The PRBM prototype has six sampling directions evenly distributed across 360 degrees when actuated by a positive pressure of 0.3 MPa. The pneumatically-driven robotic catheter provides a novel biopsy strategy, potentially facilitating in situ multidirectional biopsies in tortuous luminal organs with minimum invasiveness.

In the fields of health care and clinical monitoring, vision-based Facial Expression Recognition (FER) has achieved significantly progress, but it still faces the challenge of poor generalization ability under unconstrained conditions of occlusions and pose variation. Recently, Vision-Language Model (VLM) has greatly advanced the FER task. However, the existing VLM-based FER methods typically leverage a hard-crafted prompt (e.g., “a photo of [class]”) and only focus on the holistic semantic alignment, which may suffer from modal heterogeneity. In this work, we propose a fine-grained vision-language model via Prompt Masking for FER (PMFER). Specifically, for each expression, we first create fine-grained prompts using facial action units to guide the image encoder to learn discriminative representations. Further, to finely align text prompts and visual action units, we randomly drop a phrase description in the prompts and then predict the dropped phrase by conducting modal cross attention, implicitly promoting fine-grained vision-language alignment. In addition, we also design a modal-adversarial strategy to holistically eliminate the modal difference between visual and textual embeddings in a common latent space. Experimental results demonstrate that our PMFER model outperforms the state-of-the-art methods on several FER benchmarks, especially under the conditions of occlusions and pose variations. Note to Practitioners—Facial expression recognition is very important in health care and clinical monitoring, which provides an useful tool to assess the psychological and physiological conditions of patients. Although FER has made significant progress with the development of deep learning technologies, it still faces problems in the complex environments (e.g., occlusions and pose variations). To address the above issues, we propose a novel FER method in this work based on the recent vision-language model. It takes RGB image and text prompts as input and finally predicts the expression classification. Different from the existing methods, the proposed PMFER can enable fine-grained modal alignment for facial key units. Compared with the state-of-the-art methods on the public datasets, it can achieve better results, especially under the conditions of occlusions and pose variations. Also, we evaluate the proposed method on a real-world pain dataset, and the results demonstrate that PMFER has a good generalization and can be applied to health care.

A common limitation of autonomous tissue manipulation in robotic minimally invasive surgery (MIS) is the absence of force sensing and control at the tool level. Recently, our team has developed miniature force-sensing forceps that can simultaneously measure the grasping and pulling forces during tissue manipulation. Based on this design, here we further present a method to automate tissue traction that comprises grasping and pulling stages. During this process, the grasping and pulling forces can be controlled either separately or simultaneously through force decoupling. The force controller is built upon a static model of tissue manipulation, considering the interaction between the force-sensing forceps and soft tissue. The efficacy of this force control approach is validated through a series of experiments comparing targeted, estimated, and actual reference forces. To verify the feasibility of the proposed method in surgical applications, various tissue resections are conducted on ex vivo tissues employing a dual-arm robotic setup. Finally, we discuss the benefits of multiforce control in tissue traction, evidenced through comparative analyses of various ex vivo tissue resections with and without the proposed method, and the potential generalization with traction on different tissues. The results affirm the feasibility of implementing automatic tissue traction using miniature forceps with multiforce control, suggesting its potential to promote autonomous MIS.

This paper proposes a network called points-guided sampling net (PGSN) to guide the sampling process in sampling-based motion planner by utilizing the geometric information of obstacles. The geometric information is extracted from the point cloud of obstacles. By analyzing the properties of the point cloud, we propose a VAE feature extraction net that incorporates the variational autoencoder (VAE) framework with unique architectures designed for point clouds. Furthermore, we design a multi-modal sampling net to model the probability distribution of the states based on training trajectories taken from different environments. Based on PGSN, we propose a sampling-based motion planning algorithm called the point-guided rapidly-exploring random tree (PG-RRT). Three experiments are conducted to verify the proposed PGSN: Exp I shows the proposed VAE feature extraction net can successfully extract geometric features from the inputted point cloud; Exp II verifies the multi-modal sampling net successfully chooses corresponding mode with respect to extracted features; Exp III demonstrates the efficacy of our PG-RRT algorithm by showing PG-RRT outperforms other algorithms. Moreover, we provide theoretical analysis and insights towards understanding our model. Note to Practitioners—Obstacles cause lots of the sampling space invalid, thus the traditional sampling-based motion planning (SBMP) algorithm is usually unable to generate a trajectory within a reasonable short period of time. To improve the success rate and efficiency of SBMP, this paper proposes a novel deep neural network called points-guided sampling net (PGSN). PGSN is designed to exploit: (1) environmental point clouds and (2) training trajectories from multiple environments with different obstacles. In the first step, the point clouds include important geometric information. To utilize this information, we adopt a variational autoencoder approach which combines an encoder and a decoder together to extract geometric features more accurately from point clouds. In the second step, trajectories from multiple environments have a multi-modal property which can be represented by a truncated multivariate Gaussian mixture model. We propose a multi-modal sampling net to learn optimal parameters of this model from the training trajectories, and to select corresponding mode based on the extracted features. Experiments demonstrate that the proposed algorithm is feasible and can achieve higher success rate than the state-of-the-art methods. Our method uses a single frame of point cloud to improve efficiency, therefore multiple point clouds from different perspective maybe needed when objects occlude with each other.

Misalignment between the camera frame and the operator frame is commonly seen in a teleoperated system and usually degrades the operation performance. The effects of such misalignment have not been fully investigated foreye-in-handsystems–systems that have the camera (eye) mounted to the end-effector (hand) to gain compactness in confined spaces such as in endoscopic surgery. This article provides a systematic study on the effects of the camera frame misalignment in a teleoperatedeye-in-handrobot and proposes a simple correction method in the view display. A simulation is designed to compare the effects of the misalignment under different conditions. Users are asked to move a rigid body from its initial position to the specified target position via teleoperation, with different levels of misalignment simulated. It is found that misalignment between the input motion and the output view is much more difficult to compensate by the operators when it is in the orthogonal direction ($\sim$40 s) compared with the opposite direction ($\sim$20 s). An experiment on a real concentric tube robot with aneye-in-handconfiguration is also conducted. Users are asked to telemanipulate the robot to complete a pick-and-place task. Results show that with the correction enabled, there is a significant improvement in the operation performance in terms of completion time (mean 40.6%, median 38.6%), trajectory length (mean 34.3%, median 28.1%), difficulty (50.5%), unsteadiness (49.4%), and mental stress (60.9%).

This paper proposes a new network for reconstructing multi-object point cloud. Different from previous networks which reconstruct multi-object point cloud as a whole, our network iteratively reconstructs each individual object point cloud from a frame of multi-object point cloud. To achieve this goal, we have designed MO-Transformer, a transformer-based autoregressive network. During training, MO-Transformer takes a frame of multi-object point cloud and individual object point clouds as input. During testing, MO-Transformer iteratively reconstructs individual object point clouds only based on the input multi-object point cloud. To train the proposed MO-Transformer, we design a new loss function called separate Chamfer distance (SCD). In addition, we prove that SCD is an upper bound of the traditional Chamfer distance calculated based on the entire multi-object point cloud. The reconstruction experiment verifies the efficacy of our network in multi-object point cloud reconstruction. Furthermore, the reconstruction experiment also investigates the effect of different dimensions using a series of datasets. The ablation study experiment verifies the necessity of SCD in training MO-Transformer.

Magnetically-actuated flexible endoscopes (MAFE) have been well used in minimally-invasive surgery because they can be steered by a magnetic field thus more flexible than traditional endoscopes. Model-free and uncalibrated visual-feedback control makes it possible to manipulate MAFE with a magnetic field without external tracking systems. Because no extra sensor is required to obtain position and posture information, the size of MAFE can be made smaller. However, the traditional control method focuses on 2DoF control, which lacks control over the posture of the end of MAFE. This may result in unnecessary contact between MAFE and tissue and cause injury during the advancement of the endoscope. In this letter, we propose algorithms to enhance the pose control of MAFE to 4DoF and 5DoF based on model-free and uncalibrated visual-feedback control. Experiments in structured environments verify that the control algorithms are able to realize 4DoF manual navigation and 5DoF automatic navigation.

Collaborative robots are widely utilized in intelligent manufacturing to cooperate with the human to accomplish different assembly tasks. To improve the efficiency of human–robot cooperation, robots should be able to recognize human intentions and provide necessary assistance proactively. The major challenge for current human intention recognition methods is that they only deal with known human intentions of predefined tasks and lack of ability to learn unknown intentions corresponding to new tasks. This article introduces an evolving hidden Markov model (EHMM)-based approach to learn new human intentions incrementally by carrying out structure and parameter updating based on the observed sequence, in parallel with the recognition. The incremental learning ability makes it applicable in dynamic environments with changing tasks. A set of assistive execution policies has been developed for the robot to provide appropriate assistance to the human partner based on the intention recognition results in real time. Experiments have been carried out to verify the effectiveness of our approach in human–robot cooperative assembly tasks. The results show very high recognition accuracy (≥95.45%), and the human subjects show their high satisfaction with the intention learning ability of the proposed approach.Note to Practitioners—This article aims to effectively improve the productivity of human–robot cooperation by exploiting human adaptability and robot repeatability. Smooth cooperation requires the peer robot to provide proactive assistance to humans by inferring human intention after training. Moreover, the robot should also be able to learn untrained intentions online by human demonstrations. This is made possible by our proposed evolving hidden Markov model (EHMM) that unifies intention inference and incremental learning. Simplified cooperative assembly tasks have been designed to verify the proposed unified intention inference and learning model. A robotic assembly platform has been introduced to integrate the proposed EHMM with a perception module and a collaborative manipulation module. We have demonstrated, through experiments and surveys, that the proposed approach can promote efficacy and acceptance of human–robot cooperative assembly.

In medical image analysis (MIA) and computer-assisted surgery (CAS), aligning two multiple point sets (PSs) together is an essential but also a challenging problem. For example, rigidly aligning multiple point sets into one common coordinate frame is a prerequisite for statistical shape modelling (SSM). Accurately aligning the pre-operative space with the intra-operative space in CAS is very crucial to successful interventions. In this article, we formally formulate the multiple generalized point set registration problem (MGPSR) in a probabilistic manner, where both the positional and the normal vectors are used. The six-dimensional vectors consisting of both positional and normal vectors are called as generalized points. In the formulated model, all the generalized PSs to be registered are considered to be the realizations of underlying unknown hybrid mixture models (HMMs). By assuming the independence of the positional and orientational vectors (i.e., the normal vectors), the probability density function (PDF) of an observed generalized point is computed as the product of Gaussian and Fisher distributions. Furthermore, to consider the anisotropic noise in surgical navigation, the positional error is assumed to obey a multi-variate Gaussian distribution. Finally, registering PSs is formulated as a maximum likelihood (ML) problem, and solved under the expectation maximization (EM) technique. By using more enriched information (i.e., the normal vectors), our algorithm is more robust to outliers. By treating all PSs equally, our algorithm does not bias towards any PS. To validate the proposed approach, extensive experiments have been conducted on surface points extracted from CT images of (i) a human femur bone model; (ii) a human pelvis bone model. Results demonstrate our algorithm’s high accuracy, robustness to noise and outliers. Note to Practitioners—This paper was motivated by solving the problem of registering two or more PSs. Most existing registration approaches use only the positional information associated with each point, and thus lacks robustness to noise and outliers. Three significant improvements are brought by our proposed approach. First, the normal vectors that can be extracted from the point sets are utilized in the registration. Second, the positional error distribution is assumed to be anisotropic and inhomogeneous. Third, all the PSs to be registered are treated equally that means no PS is considered as the model one. The registration problem is cast into a maximum likelihood (ML) problem and solved under the expectation maximization (EM) framework. We have demonstrated through extensive experiments that the proposed registration approach achieves significantly improved accuracy, robustness to noise and outliers. The algorithm is particularly suitable for biomedical applications involving the registration procedures, such as image-guided surgery.

This paper presents a novel continuum robot sheath for use in single-port minimally invasive procedures such as neuroendoscopy in which the sheath is designed to deliver multiple robotic arms. Actuation of the sheath is achieved by using precurved superelastic tubes lining the working channels used for arm delivery. These tubes perform a similar role to push/pull tendons, but can accomplish shape change of the sheath via rotation. A kinematic model using Cosserat rod theory is derived which is based on modeling the system as a set of eccentrically aligned precurved tubes constrained along their length by an elastic backbone. The specific case of a two-arm sheath is considered in detail. Simulation and experiments are used to investigate the validate the concept and model.

Small-scale robots have great potential in minimally invasive surgery (MIS). In this paper, we propose an untethered magnetic gripper with small scale and build a double-magnet model for it. The gripper is 4.3mm long and its maximum width is 4mm. It contains a spindle and two magnets, which can achieve precise control of orientation, position and open angle with external magnetic driven field. As a result, it can perform operations such as transporting medicines in confined and constrained environments. Modeling and analysis of the magnetic gripper have been carried out. Relationship between the open angle and external magnetic field has been established. Kinematics model of the gripper has been built. A 3-axis Helmholtz-Maxwell coil system has been established to generate the magnetic field, in which orientation and open angle can be controlled with uniform magnetic field while position can be controlled with gradient field. The proposed gripper have been validated with phantom experiments. An opened angle control error of 0.63° and direction control error of 1.1° have been obtained.

Registering two point sets (PSs) is an essential problem in medical robotics and computer-assisted surgery (CAS). As one typical example, in computer-assisted orthopedic surgery (CAOS), the preoperative scan has to be aligned with the intraoperative scan accurately. In this article, we first formally formulate the generalized PS registration problem in a probabilistic manner. Especially, not only positional but also orientational information is incorporated into the registration. Notably, the positional error is assumed to obey a multivariate Gaussian distribution to accommodate the anisotropic noise. The expectation–maximization (EM) framework is utilized to solve the maximum likelihood (ML) problem. In the E-step, the correspondence probabilities between points in two generalized PSs are computed. In the M-step, the constrained optimization problem with respect to the rigid transformation matrix is reformulated as an unconstrained one. This is achieved by utilizing the Rodrigues parameterization to represent the rotation matrix. Both extensive simulated and real experiments are conducted to validate the proposed algorithm by comparing it with state-of-the-art registration methods.Note to Practitioners—This article was motivated by considering the anisotropic positional uncertainty into the rigid point set (PS) registration in the application of preoperative-to-intraoperative registration within image-guided surgery. We provide iterative solutions that compute the rotation and translation vector that aligns two 3-D PSs. The correspondences between points in two PSs are not known and regarded as hidden variables in the optimization process. Expectation–maximization technique is utilized to solve the maximum likelihood problem. We have demonstrated through experiments that our proposed approach can achieve lower registration error values than the compared state-of-the-art registration methods on various data sets. The readers should note that the proposed method is particularly suitable for cases that anisotropic noise is involved.

It is evident that corticomuscular coherence (CMC), representing the functional coupling between motor cortex and muscle tissues, plays a crucial role in neurophysiologic studies and applications. It is hypothesized that there is an unknown time delay comprising at least neural conduction time in the process of corticomuscular interaction. In this study, we developed a novel delay estimation method, defined as the rate of voxels change (RVC) for the estimation of time delay in two coupled physiological signals. The RVC is the dynamic variation of the local CMCs observed in different time offsets. Both simulation and physiological data confirm the capability of RVC in estimating cortical-muscular delay. The underlying mechanisms of individual discrepancy of the latency is also investigated via exploring the correlation between delays, brain activity and motor performance. Correlation analyses indicate an intrinsic link between the connectivity strength of the brain network and the length of time delay in cortical-muscular interactions.

Multi-modal sensory fusion is believed to obtain higher accuracy in gesture recognition. Its difficulty lies in mining discriminative features and fusing features from different modalities. Surface electromyography(sEMG) and ultrasound signals are typical signal modalities in gesture recognition. It is expected that the fusion of them can take advantage of the complementarity of electrophysiological information and muscle morphology information. This paper proposed two kinds of feature fusion method. The one is concatenating the manual designed sEMG and ultrasound features, and the other is a convolutional neural network (CNN) based feature exaction and fusion method for sEMG and ultrasound signals. Eight able-bodied subjects were involved to participate in the experiments. In the experiments, four channels of sEMG and A-mode ultrasound signals corresponding to 20 gestures were collected synchronously to evaluate the proposed method. The experimental results demonstrated that the fusion sEMG-ultrasound feature always outperformed the separate sEMG or ultrasound feature regardless of the feature extraction method, and as for fusion sEMG-ultrasound feature, the CNN based method achieve a high accuracy (97.38±1.49%) in 20 gestures, which surpassed the method of concatenating the manual designed features and applying machine learning algorithm (LDA, KNN, SVM).

This paper introduces a robust generalized point cloud (PC) registration method that utilizes not only the positional but also the orientation information associated with each point. The proposed method solves the rigid PC registration problem in a probabilistic manner, which casts the problem into a maximum likelihood (ML) framework. A hybrid mixture model (HMM) is utilized to represent one generalized PC. In the HMM, a von-Mises-Fisher mixture model (FMM) is adopted to model the orientational uncertainty, while a Gaussian mixture model (GMM) is used to represent the positional uncertainty. An expectation-maximization (EM) algorithm is adopted to solve the optimization problem in an iterative manner to find the optimal rotation matrix and the translation vector between two generalized PCs. In both expectation step (E step) and maximization step (M step), orientational information is utilized, which can potentially improve the algorithm's robustness to noise and outliers. In the E step, the posterior probabilities that represent the degree of point correspondences in two PCs are computed. In the M step, an efficient closed-form solution to a rigid transformation matrix is developed. E and M steps will iterate until certain convergence criteria are satisfied. Extensive experiments under different noise levels and outlier ratios have been carried out on a data set of femur bone computed tomography images. Experimental results show that the proposed method outperforms the state-of-the-art ones in terms of accuracy, robustness, and convergence speed significantly. Note to Practitioners-This paper was motivated by solving the problem of registering two PCs. Most existing approaches generally use only the positional information associated with each point and thus lack robustness to noise and outliers. This paper suggests a new robust method that also adopts the normal vectors associated with each point. The registration problem is cast into a maximum likelihood (ML) problem and solved under the expectation-maximization (EM) framework. Closed-form solutions for estimating parameters in both expectation and maximization steps are provided in this paper. We have demonstrated through extensive experiments that the proposed registration algorithm achieves improved accuracy, robustness to noise and outliers, and faster convergence speed.

Aligning different views or representations of anatomy is an essential task in both medical imaging computing (MIC) and computer-assisted interventions' (CAIs') communities. Motivated by simultaneously registering multiple point sets (PSs) and further improving the algorithm's robustness to outliers and noise, in this paper, we propose a novel probabilistic approach to jointly register multiple generalized PSs. A generalized PS includes high-dimensional points consisting of both positional vectors and orientational information (or normal vectors). Hybrid mixture models (HMMs) combining Gaussian and von Mises-Fisher (VMF) distributions are used to model positional and orientational components of the generalized PSs. All generalized PSs are jointly registered using the expectation-maximization (EM) technique. In the E-step, the posterior probabilities representing point correspondence confidences are computed. In the M-step, the rigid transformation matrices, positional variances, and orientational concentration parameters are updated for each generalized PS. E and M steps will iterate until some termination condition is satisfied. We validate our algorithm using the surface points extracted from the human femur CT model. The experimental results demonstrate that the proposed algorithm outperforms the state-of-the-art ones in terms of the accuracy, robustness, as well as convergence speed. In addition, our algorithm is able to recover a better central PS than the state-of-the-art one does in the case of registering multiple PSs. Our algorithm is very suitable for registering complex structures arising in medical imaging. This paper was motivated by solving the problem of registering two or multiple point sets. Most existing approaches generally use only the positional information associated with each point and thus lack robustness to noise and outliers. This paper suggests a new robust method that also adopts the normal vectors associated with each point. The registration problem is cast into a maximum-likelihood (ML) problem and solved under the expectation-maximization (EM) framework. Closed-form solutions to estimating parameters in both expectation and maximization steps are provided in this paper. We have demonstrated through extensive experiments that the proposed registration algorithm achieves improved accuracy, robustness to noise and outliers, and faster convergence speed.

This paper presents a novel continuum robot sheath for use in single-port minimally invasive procedures such as neuroendoscopy in which the sheath is designed to deliver multiple robotic arms. Articulation of the sheath is achieved by using precurved superelastic tubes lining the working channels used for arm delivery. These tubes perform a similar role to push/pull tendons, but can accomplish shape change of the sheath via rotation as well as translation. A kinematic model using Cosserat rod theory is derived which is based on modeling the system as a set of eccentrically aligned precurved tubes constrained along their length by an elastic backbone. The specific case of a two-arm sheath is considered in detail and its relationship to a concentric tube balanced pair is described. Simulation and experiment are used to investigate the concept, map its workspace and to evaluate the kinematic model.

This paper presents a sensor fusion-based surgical instrument tracking system which uses multiple monocular modules. The system is an optical tracking system, which has been widely utilized in the image-guide surgery because of its high accuracy and precision. However, the line-of-sight occlusion problem which remains unresolved in current systems frustrates surgeons during the operation. To address this challenge, we propose a surgical instrument tracking system based on multiple monocular modules. The rationale is to enable the system to track the surgical instruments inside the surgical site from different views. Three sensor fusion algorithms are proposed to integrate all sensor data from the multimodule system. In order to show the feasibility of the tracking system, simulations and comparison experiments have been carried out. The intensive investigation results give a practical instruction to the real implementation of the proposed system in image-guided interventions. Moreover, an image-guided surgical trial by using a cadaver head has been carried out to validate the feasibility of the proposed system and the tracking algorithms. The results from both the simulation and the cadaver trial have shown the effectiveness of the proposed robust fusion algorithm.

This paper introduces a robust point cloud registration method which utilizes not only positional but also the orientation information at each point. The proposed method takes a probabilistic approach which forms the problem as a hybrid mixture model, in which a Von-Mises-Fisher mixture model (FMM) is adopted to model the orientation part and a gaussian mixture model (GMM) is used to represent the position part. When two point clouds are optimally registered, the correspondence is the maximum of the posterior probability of the overall mixture model. Expectation-Maximization (EM) algorithm has been adopted to solve the optimization problem in an iterative manner to find the optimal rotation and translation between two point clouds. Extensive experiments under different noise levels and different outlier ratios have been carried out on a dataset of the femur CT images. Comparison results show that the proposed method outperforms the state-of-the-art methods under most of the experimental conditions, which indicates the validity of our method.

Alignment of two point clouds is an essential problem in medical robotics and computer-assisted surgery. In this paper, we first formally formulate the generalized point cloud registration problem in a probabilistic manner. Specifically, not only positional but also the orientational information are incorporated into registration. Notably, the positional error is assumed to obey a multivariate Gaussian distribution to accommodate anisotropic cases. Expectation conditional maximization framework is utilized to solve the problem. In E-step, the correspondence probabilities between points in two generalized point clouds are computed. In M -step, the constrained optimization problem with respect to the transformation matrix is re-formulated as an unconstrained one. Extensive experiments are conducted to compare the proposed algorithm with the state-of-the-art registration methods. The experimental results demonstrate the algorithm's robustness to noise and outliers, fast convergence speed.

In this paper, we propose a gait recognition method for service robots to conduct human following tasks. A walking sequence segmentation method is designed to extract the consecutive gait cycles from an arbitrary walking sequence. Based on the segmentation results, a novel hybrid gait feature is proposed to capture the static, dynamic, and trajectory features for each segmented key and supplementary gait cycles. A dataset of 25 human subjects is collected to evaluate the proposed method in three different walking paths with various walking directions. Experimental results show that the proposed method achieves satisfactory performance in terms of identification accuracy and Fcomb indexes on our dataset. Compared with five state-of-the-art gait recognition methods, the proposed method achieves the best performance on human gait recognition based on the walking sequences defined in our proposed dataset.

Multirobot comanipulation shows great potential in surpassing the limitations of single-robot manipulation in complicated tasks such as robotic surgeries. However, a dynamic multirobot setup in unstructured environments poses great uncertainties in robot configurations. Therefore, the coordination relationships between the end-effectors and other devices, such as cameras (hand–eye calibration) and tools (tool–flange calibration), as well as the relationships among the base frames (robot–robot calibration) have to be determined timely to enable accurate robotic cooperation for the constantly changing configuration of the systems. We formulated the problem of hand–eye, tool–flange, and robot–robot calibration to a matrix equation$\mathbf{AXB=YCZ}$. A series of generic geometric properties and lemmas were presented, leading to the derivation of the final simultaneous algorithm. In addition to the accurate iterative solution, a closed-form solution was also introduced based on quaternions to give an initial value. To show the feasibility and superiority of the simultaneous method, two nonsimultaneous methods were compared through thorough simulations under various robot movements and noise levels. Comprehensive experiments on real robots were also performed to further validate the proposed methods. The comparison results from both simulations and experiments demonstrated the superior accuracy and efficiency of the proposed simultaneous calibration method.

In this paper, two gesture design methods for the humanoid robot NAO are proposed and compared. The first method is mapping human motions to the robot by an RGB-D sensor and kinematic modeling. The second method is based on hand-puppeteering. Thirteen subjects are recruited to design a forearm waving gesture for a NAO robot by the two methods. The obtained two groups of forearm waving gestures are then compared by another sixteen subjects. Our experimental results indicate that the forearm waving gestures obtained from the hand-puppeteering method are slower and have smaller range of motion than those obtained from the motion mapping method. Besides, people tend to perceive the forearm waving gestures obtained from the hand-puppeteering method as more likeable and as conveying the greeting message better than those obtained from the motion mapping method. This work contributes to a better understanding of the nature of the two gesture design methods and offers instructive reference for robot behavior designers on design method choosing.

Optical means of instrument tracking has been widely used in image-guided interventions and considered the de facto standard for tracking rigid bodies with a direct line-of-sight. However, the occlusion problem which remains unresolved in current systems frustrates surgeons during the operation. To address this challenge, we propose a surgical instrument tracking system based on multiple reconfigurable monocular modules. The main approach is to enable the system to dynamically reconfigure the multiple monocular modules when occlusion occurs partially within the workspace. In this paper, we focus on the system architecture and an agile multicamera calibration method which only uses the customized tool for the surgical instrument tracking scenario. Additionally, two fast non-iterative algorithms are proposed and studied. In order to show the feasibility and superiority of the corresponding multicamera calibration algorithm, comparison experiments have carried out. The intensive investigation results give a practical instruction to the real implementation of the proposed system in image-guided interventions.

Ulcer is one of the most common symptoms of many serious diseases in the human digestive tract. Especially for the ulcers in the small bowel where other procedures cannot adequately visualize, wireless capsule endoscopy (WCE) is increasingly being used in the diagnosis and clinical management. Because WCE generates large amount of images from the whole process of inspection, computer-aided detection of ulcer is considered an indispensable relief to clinicians. In this paper, a two-staged fully automated computer-aided detection system is proposed to detect ulcer from WCE images. In the first stage, we propose an effective saliency detection method based on multi-level superpixel representation to outline the ulcer candidates. To find the perceptually and semantically meaningful salient regions, we first segment the image into multi-level superpixel segmentations. Each level corresponds to different initial region sizes of the superpixels. Then we evaluate the corresponding saliency according to the color and texture features in superpixel region of each level. In the end, we fuse the saliency maps from all levels together to obtain the final saliency map. In the second stage, we apply the obtained saliency map to better encode the image features for the ulcer image recognition tasks. Because the ulcer mainly corresponds to the saliency region, we propose a saliency max-pooling method integrated with the Locality-constrained Linear Coding (LLC) method to characterize the images. Experiment results achieve promising 92.65% accuracy and 94.12% sensitivity, validating the effectiveness of the proposed method. Moreover, the comparison results show that our detection system outperforms the state-of-the-art methods on the ulcer classification task.

Tasks that are too hard for single robot can be easily carried out by multiple robots in a cooperative manner. If some/all robots have mobile bases, the cooperation is subjected to great uncertainties in both the robotic system and environment. Therefore, the relationships among all the base frames (robot-robot calibration) and the relationships between the end-effectors and the other devices such as cameras and tools (hand-eye and tool-flange calibrations) have to be calculated to enable the robots to cooperate. To address these challenges, in this paper, we propose a simultaneous hand-eye, tool-flange and robot-robot calibration method. Thorough simulations are conducted to show the superiority of the proposed simultaneous method under different noise levels and various numbers of robot movements. Furthermore, the comparison to two non-simultaneous calibration methods has also been carried out to show the efficiency and robustness of the proposed simultaneous method.