In computer-assisted orthopedic surgery (CAOS), robust and accurate registration of the preoperative full bone model and the intraoperative partial point set is a prerequisite for reliable surgical navigation, yet remains highly challenging due to partial overlap, noise, and outliers. We propose the Overlap-Guided Bidirectional Hybrid Mixture Registration (OBHMR) framework for robust and accurate partial-to-full registration. First, geometric features (i.e., surface normals) extracted from raw point sets are incorporated in both correspondence estimation and transform computation. Meanwhile, we formulate a hybrid mixture model that jointly represents positions with Gaussian mixtures (GMMs) and normals with von Mises–Fisher (vMF) mixtures across the two generalized point sets. Second, a dual-branch overlap prediction network leverages feature similarity and geometric structure to provide accurate point-wise overlap scores that guide hybrid-mixture construction under partial overlap. Third, a correspondence module integrates rotation-invariant features, multi-level self-attention, and clustering-based refinement to enhance reliability under noise and misalignment. Finally, a bidirectional objective jointly aligns source-to-target and target-to-source mixtures, explicitly accounting for discrepancies induced by noise and outliers in both the preoperative and intraoperative point sets to achieve robust optimization. Extensive experiments on 1,399 femur and 1,301 hip models demonstrate superior performance over state-of-the-art methods across overlap ratios from 5% to 70%, under both isotropic and anisotropic noise and outlier ratios up to 100%, achieving errors as low as 1.27° rotation and 1.18 mm translation at 50% overlap with 2.5mm noise. Additional tests on liver and ModelNet40 confirm strong generalization across medical and non-medical data. Ablation studies further validate the contributions of normals, overlap estimation, and the bidirectional formulation.

This paper presents a novel robust and accurate normal-assisted learning-based rigid point set registration approach, i.e., Deep Bi-directional Hybrid Mixture Registration (DeepBHMR), where normal vectors are used in both correspondence and transformation computational stages while the bi-directional registration processes are considered. DeepBHMR consists of three components, (1) the correspondence estimation network that predicts the correspondence probabilities; (2) the posterior estimation module that computes the HMMs parameters; (3) the transformation estimation module that calculates the rigid transformation matrix by utilizing the bidirectional optimization mechanism. DeepBHMR has been extensively validated on various medical data sets, outperforming state-of-the-art registration methods. For femur bones, the mean rotation error value is approximately 1° (i.e, 1.01°) and the translation error is less than 1 mm (i.e., 0.30 mm) respectively, which meets the requirement of computer-assisted orthopedic surgery. Furthermore, even (1) trained with femur data and tested on distinct shapes and (2) under the large transformation, the mean RMSE values of registration are 2.60 mm and 3.05 mm respectively, demonstrating DeepBHMR’s favorable generalizability to different data shapes and great capability to handle global registration. Additionally, the individual significant contributions and computational efficiency of adopting normal vectors and utilizing the bidirectional mechanism have been validated in ablation studies. The results demonstrate the DeepBHMR’s favorable generalizability from femur bones to hip bones and that DeepBHMR can successfully handle the large transformation or partial-to-full registration simultaneously. The code implementation of DeepBHMR has been made publicly available at https://github.com/zzyrobot/DeepBiHMM.git.

Polyp segmentation is vital for early colorectal cancer detection, yet traditional fully supervised methods struggle with morphological variability and domain shifts, requiring frequent retraining. Additionally, reliance on large-scale annotations is a major bottleneck due to the time-consuming and error-prone nature of polyp boundary labeling. Recently, vision foundation models like Segment Anything Model (SAM) have demonstrated strong generalizability and fine-grained boundary detection with sparse prompts, effectively addressing key polyp segmentation challenges. However, SAM's prompt-dependent nature limits automation in medical applications, since manually inputting prompts for each image is labor-intensive and time-consuming. We propose OP-SAM, a One-shot Polyp segmentation framework based on SAM that automatically generates prompts from a single annotated image, ensuring accurate and generalizable segmentation without additional annotation burdens. Our method introduces Correlation-based Prior Generation (CPG) for semantic label transfer and Scale-cascaded Prior Fusion (SPF) to adapt to polyp size variations as well as filter out noisy transfers. Instead of dumping all prompts at once, we devise Euclidean Prompt Evolution (EPE) for iterative prompt refinement, progressively enhancing segmentation quality. Extensive evaluations across five datasets validate OP-SAM's effectiveness. Notably, on Kvasir, it achieves 76.93% IoU, surpassing the state-of-the-art by 11.44%.

Robust and accurate 2D/3D registration, which aligns preoperative models with intraoperative images of the same anatomy, is crucial for successful interventional navigation. To mitigate the challenge of a limited field of view in single-image intraoperative scenarios, multi-view 2D/3D registration is required by leveraging multiple intraoperative images. In this paper, we propose a novel multi-view 2D/3D rigid registration approach comprising two stages. In the first stage, a combined loss function is designed, incorporating both the differences between predicted and ground-truth poses and the dissimilarities (e.g., normalized cross-correlation) between simulated and observed intraoperative images. More importantly, additional cross-view training loss terms are introduced for both pose and image losses to explicitly enforce cross-view constraints. In the second stage, test-time optimization is performed to refine the estimated poses from the coarse stage. Our method exploits the mutual constraints of multi-view projection poses to enhance the robustness of the registration process. The proposed framework achieves a mean target registration error (mTRE) of$0.79+2.17\ \mathbf{mm}$on six specimens from the DeepFluoro dataset, demonstrating superior performance compared to state-of-the-art registration algorithms.

Human-robot interaction (HRI) encompasses a wide range of collaborative tasks, with handover being one of the most fundamental. As robots become more integrated into human environments, the potential for service robots to assist in handing objects to humans is increasingly promising. In robot-to-human (R2H) handover, selecting the optimal grasp is crucial for success, as it requires avoiding interference with the human's preferred grasp region and minimizing intrusion into their workspace. Existing methods either inadequately consider geometric information or rely on data-driven approaches, which often struggle to generalize across diverse objects. To address these limitations, we propose a novel zero-shot system that combines semantic and geometric information to generate optimal handover grasps. Our method first identifies grasp regions using semantic knowledge from vision-language models (VLMs) and, by incorporating customized visual prompts, achieves finer granularity in region grounding. A grasp is then selected based on grasp distance and approach angle to maximize human ease and avoid interference. We validate our approach through ablation studies and real-world comparison experiments. Results demonstrate that our system improves handover success rates and provides a more user-preferred interaction experience. Videos, appendixes and more are available at https://sites.google.com/view/vlm-handover.

In computer-assisted orthopedic surgery (CAOS), accurate point set registration is essential for enhancing surgical accuracy. However, the sparse and low-overlap nature of intraoperative point sets presents significant challenges for reliable registration. To deal with these challenges, we propose a novel registration-after-completion framework, where the intraoperative point set is first completed, after which the two full point sets are registered. Our main contributions include the follows. First, we propose a progressive two-stage strategy to progressively complete the sparse and partial intraoperative point set. Second, considering that 1) intra-operative point set contains noise 2) the point completion process is not perfect, and 3) the resolution of preoperative image is limited, we adopt the bidirectional hybrid mixture models (HMMs) to represent the point set pairs and formulate the probabilistic registration network. In the proposed novel correspondence network where a dual-path cross-attention mechanism is adopted for feature fusion and a clustering mechanism is leveraged for calculating point-to-mixture correspondences. Furthermore, the bidirectional registration mechanism is leveraged to compute the transformation based on estimated correspondences. Third, we have extensively validated the proposed approach on various datasets and bone phantoms. Our experiments on 1399 human femur and 1301 hip models demonstrate that our method achieves state-of-the-art performance across overlap rates from 15% to 35% and at various point counts (i.e., 25, 50, and 100 points) under conditions with less than 50% overlap. Additionally, real phantom experiments on femur and hip models validate the method’s performance in simulated surgical scenarios. Experiments on ModelNet40 further confirmed our method’s effectiveness and generalizability.

In this paper, we propose a novel unsupervised intraoperative liver deformation correction method, called Learning Coherent point drift Network (LCNet), for image-guided liver surgery (IGLS). We first estimate the correspondences between the preoperative and intraoperative point sets in the optimal transport (OT) module by leveraging both original points and extracted features. Afterwards, we compute the point-wise displacement vector by solving the involved matrix equation in the Transformation module, where the point localisation noise is explicitly considered and modeled. Additionally, we present three variants of the proposed approach, i.e., LCNet, LCNet-ED and LCNet-WD, where better registration performances of LCNet against the other two demonstrate the superiority of the utilised Chamfer loss. We have extensively evaluated LCNet on the MedShapeNet dataset consisting of 615 different liver shapes of real patients, and the 3Dircadb dataset comprising 20 liver models of real patients. Extensive experimental results under different deformation and noise magnitudes demonstrate that LCNet outperforms existing state-of-the-art registration algorithms and holds significant application potential in IGLS. For example, when the overlapping ratio between the preoperative and intraoperative point sets is 25%, the deformation magnitude is 8 mm, the maximum point localization noise magnitude is 2 mm and the rotation angle lies in the range of [−45°, 45°], LCNet achieves a root-mean-square error (RMSE) value being 3.21 mm on MedShapeNet dataset, significantly outperforming those of Lepard and RoITr being 5.41 mm (p < 0.001) and 4.90 mm (p < 0.001) respectively.

Long-horizon composite task planning for multi-robot systems in cross-regional complex scenarios faces dual challenges: spatial-semantic comprehension of natural language described tasks and collaborative optimization of subtask al-location. To address these challenges, this paper proposes a progressive three-stage task planning framework. First, an augmented scene graph is constructed to enable large language models (LLMs) to comprehend environmental structures, thereby generating simplified Linear Temporal Logic (LTL) task sequences. Subsequently, a novel heuristic function is employed to select optimal task allocation plans. Finally, LLMs are used to generate low-level executable robot instructions based on robotic system instruction templates. We establish a long-horizon composite task dataset for experimental validation on real-world quadrupedal multi-robot systems. Experimental results demonstrate the effectiveness of our approach in resolving cross-regional composite tasks.

In this paper, we present a novel curve-to-surface registration method, termed Bi-directional Hybrid Mixture Model Registration based on Dual-constrained Tangent and Normal Vectors (BiHMM-DTN), where two different tangent vectors at the intraoperative point are simultaneously used with the normal vector at the corresponding preoperative point to construct the geometric constraints. While hybrid registration models incorporating tangent and normal vectors (HMM-TN) demonstrate success, their geometric constraints prove inadequate or inappropriate for sparse intraoperative point sets, frequently yielding suboptimal optimization outcomes. By critically revisiting the geometric constraints of HMM-TN, we propose a dual-constraints-based hybrid mixture model registration framework with enhanced intraoperative point set acquisition protocols. To deal with noise and outliers in preoperative and intraoperative point sets—caused by reconstruction inaccuracies and tracking errors, respectively—our approach employs a bi-directional registration mechanism for curve-to-surface registration. We provide rigorous proofs validating the geometric completeness of the dual constraints within this mechanism. The BiHMM-DTN framework is formulated as a maximum likelihood estimation (MLE) problem and optimized using an expectation-maximization (EM) algorithm. Furthermore, to enhance convergence stability and accelerate optimization, the rotation matrix is updated iteratively through successive incremental steps. Extensive experiments on human femur and hip models demonstrate that our method outperforms state-of-the-art approaches, including both traditional optimization and deep learning methods, under various noise and outlier conditions. Furthermore, real-world phantom experiments highlight the potential clinical value of our method for surgical navigation applications. The codes and data are available at https://github.com/sam-zyzhang/BiHMM-DTN.git.

3D point cloud registration is an essential problem in computer vision, robotics, surgical navigation and augmented reality. Accurate registration of partially overlapped intraoperative point clouds (e.g., femoral reconstruction) remains critical yet challenging in orthopedic navigation due to incomplete overlap and dynamic noise. In this study, we propose a partial-to-partial point cloud registration framework based on directional spatial consistency. First, we extract overlapped areas from partially overlapping point clouds and leverage the point registration graph matching module to calculate the hard point matching matrix. Second, we sample nodes from the source point cloud and generate translation-invariant edge vectors (direction/scale-preserving) via their k-nearest neighbors, guided by predicted point correspondences. This bypasses translation ambiguities by encoding spatial consistency through edges, reducing pose estimation to 3DoF alignment (rotation). The loss explicitly couples point-level matches with edge-level geometric constraints for dual optimization. Building upon this framework, we extract reliable overlapping edge representations and prune their similarity matrix by thresholding low-confidence scores, effectively suppressing spurious matches. The proposed edge-aware matching mechanism further exploits the translation invariance of local structures to refine point correspondences with enhanced accuracy. Finally, we introduce a bidirectional registration mechanism to reinforce optimization stability, achieving state-of-the-art performance across benchmarks. Extensive experiments on ModelNet40, ShapeNet, and MedShapeNet validate our method under diverse scenarios: partial-to-partial, unseen categories, partial-to-full, and cross-dataset generalization, surpassing existing methods in registration accuracy. The codes are available at https://github.com/pidan0824/DSCGM.

Indoor location-based services are essential to daily life but lack a standard like outdoor Global Positioning System (GPS). WiFi received signal strength (RSS) is an optimal option thanks to its ubiquitous deployment, though existing research typically uses only a few constrained devices. We present a learning-based WiFi RSS indicator (RSSI) fingerprinting method designed for general environments. RSSI samples are collected in daily scenarios with hundreds of WiFi access points (APs) without disclosing their locations, and with randomly distributed reference points. The continuously measured RSSI values across multiple timestamps are treated as sequences, serving as fingerprints corresponding to location sequences. Instead of using all detected APs, we discard those sensed only in limited timestamps, i.e., APs with confined coverage and limited identification positions. We then employ various 1D feature extractors to estimate the location sequence from the refined RSSI indices. Our method outperforms state-of-the-art methods on open-access datasets from a small office with densely equipped WiFi APs and a larger university campus with sparse WiFi signals. Real-world experiments on the CUHK campus further demonstrate the statistical consistency of the proposed method. We share the data collection code and self-collected data to facilitate future studies.

Deep learning can predict depth maps and capsule ego-motion from capsule endoscopy videos, aiding in 3D scene reconstruction and lesion localization. However, the collisions of the capsule endoscopies within the gastrointestinal tract cause vibration perturbations in the training data. Existing solutions focus solely on vision-based processing, neglecting other auxiliary signals like vibrations that could reduce noise and improve performance. Therefore, we propose V2-SfMLearner, a multimodal approach integrating vibration signals into vision-based depth and capsule motion estimation for monocular capsule endoscopy. We construct a multimodal capsule endoscopy dataset containing vibration and visual signals, and our artificial intelligence solution develops an unsupervised method using vision-vibration signals, effectively eliminating vibration perturbations through multimodal learning. Specifically, we carefully design a vibration network branch and a Fourier fusion module, to detect and mitigate vibration noises. The fusion framework is compatible with popular vision-only algorithms. Extensive validation on the multimodal dataset demonstrates superior performance and robustness against vision-only algorithms. Without the need for large external equipment, our V2-SfMLearner has the potential for integration into clinical capsule robots, providing real-time and dependable digestive examination tools. The findings show promise for practical implementation in clinical settings, enhancing the diagnostic capabilities of doctors. Note to Practitioners—This paper is motivated by the problem of estimating the depth and ego-motion information for the wireless capsule endoscopy in the human gastrointestinal tract to realize accurate, efficient, robust, and real-time inspection. Our estimation method does not engage any external localization equipment. Instead, inspired by the existing research on integrating capsule endoscopy and inertial measurement units, we introduce vibration signals into vision-based depth and ego-motion estimation approaches, improving the accuracy and robustness of the estimation results based on multimodal learning methods. Research on capsule robots or computer vision can readily be combined with our framework for various clinical and industrial applications.

Registration of intraoperative ultrasound (iUS) with preoperative CT represents a significant yet challenging task in the context of radiation-free spinal surgical navigation. The presence of thickness response artifacts in US images poses a considerable obstacle to the accurate extraction of bone boundaries. Furthermore, US-CT registration typically necessitates the detection and correspondence of high-quality landmarks at the initial stage. This can be accomplished by surgeons who have undergone extensive training in the localization of standard spinal US views, enabling them to identify key vertebral landmarks for subsequent precise registration. In this paper, we propose a fully automated iUS registration method that employs a limited number of spinal US views as observation objects. Specifically, three-dimensional vertebral meshes segmented from the preoperative CT images are superimposed on the US images and then fed to the reinforcement learning (RL) agent for sequential decision-making. The proposed method achieves fully automatic US-CT registration without relying on prespecified initialization. This is achieved by training the agent to approach bone surfaces on several randomly selected 2D US views. The instability of RL-based iUS registration is primarily attributable to the difficulty of correlating long-range information within the neural network. To address this issue, we propose a Field of View Separation (FoVS) module. The proposed approach employs separate encoders for US and mesh images, followed by cross-attention aggregation, which facilitates information flow between non-adjacent pixels. This approach enables pretraining of feature extraction on distinct encoders and the application of supplementary loss for enhanced feature matching precision, thereby significantly improving the learning capability and stability of the network. Furthermore, a refinement module is introduced to correct the results of the RL registration, which improves the stability of the registration process. To ascertain the efficacy of each module, action, and auxiliary task, comprehensive experiments are conducted. The results demonstrate that the performance of the RL agent is enhanced by the associated modules and auxiliary tasks. The registration exhibited an angular error of$8.83 \; \pm \; 4.69$degrees and a translational error of$3.34 \; \pm \; 1.42$mm, achieving the state-of-the-art (SOTA) results. It is noteworthy that fine-tuning the model prior to the surgical phase can significantly reduce the registration error, which is a promising outcome for its clinical translation.Note to Practitioners—The objective of this study is to address the issue of image registration using iUS in conjunction with preoperative CT scans in the context of spine surgery. The current 2D/3D image registration methods are constrained by several limitations. Firstly, they often exhibit reduced accuracy, and require high-quality images in substantial quantities. Secondly, there is a lack of effective mechanisms to rectify errors identified after the registration process. This paper proposes a fully automated registration framework based on RL, which incorporates image rendering and mesh clipping to enable continuous adjustment of the pose of 3D data, thereby facilitating 2D/3D registration. The framework employs the distinctive attributes of iUS images and incorporates a refinement module to evaluate registration accuracy, thereby facilitating the rectification of any registration issues. The proposed framework was tested on both sheep lumbar subjects and human lumbar phantoms, demonstrating the highest level of performance to date and indicating its potential for integration into surgical navigation systems.

Using underwater robots instead of humans for the inspection of coastal piers can enhance efficiency while reducing risks. A key challenge in performing these tasks lies in achieving efficient and rapid path planning within complex environments. Sampling-based path planning methods, such as Rapidly-exploring Random Tree* (RRT*), have demonstrated notable performance in high-dimensional spaces. In recent years, researchers have begun designing various geometry-inspired heuristics and neural network-driven heuristics to further enhance the effectiveness of RRT*. However, the performance of these general path planning methods still requires improvement when applied to highly cluttered underwater environments. In this paper, we propose PierGuard, which combines the strengths of bidirectional search and neural network-driven heuristic regions. We design a specialized neural network to generate high-quality heuristic regions in cluttered maps, thereby improving the performance of the path planning. Through extensive simulation and real-world ocean field experiments, we demonstrate the effectiveness and efficiency of our proposed method compared with previous research. Our method achieves approximately 2.6 times the performance of the state-of-the-art geometric-based sampling method and nearly 4.9 times that of the state-of-the-art learning-based sampling method. Our results provide valuable insights for the automation of pier inspection and the enhancement of maritime safety. (Video1).

Trajectory planning for quadrotors in cluttered environments has been challenging in recent years. While many trajectory planning frameworks have been successful, there still exists potential for improvements, particularly in enhancing the speed of generating efficient trajectories. In this paper, we present a novel hierarchical trajectory planning framework to reduce computational time and memory usage called MINER-RRT*, which consists of two main components. First, we propose a sampling-based path planning method boosted by neural networks, where the predicted heuristic region accelerates the convergence of rapidly-exploring random trees. Second, we utilize the optimal conditions derived from the quadrotor’s differential flatness properties to construct polynomial trajectories that minimize control effort in multiple stages. Extensive simulation and real-world experimental results demonstrate that, compared to several state-of-the-art (SOTA) approaches, our method can generate high-quality trajectories with better performance in 3D cluttered environments (https://youtu.be/fXuuMRX19q0). Note to Practitioners—The motivation is the problem of planning trajectories for quadrotor autonomous flight in 3D cluttered and complex scenarios such as wild forest exploration and subterranean environment search-and-rescue. Sampling-based path planning methods are suitable for dealing with the complexity of the physical environment but are not convenient for computing dynamics and their differentials. Optimization-based trajectory generation methods are appropriate for handling various high-order constraints but rely on high-quality initial path solutions. Therefore, this paper combines the advantages of the two methods to propose a novel trajectory planning framework that can generate high-quality trajectories for quadrotors faster than many previous algorithms. We conduct numerous simulations and real-world experiments to verify that our method can be effectively deployed in real scenarios and empower quadrotors for complex autonomous tasks in the future.

Tracheotomy is commonly performed for patients needing prolonged intubation, airway obstruction, and neck injuries. Accurate placement of the incision and the tracheal window is paramount in order to avoid complications. Current surgical technique heavily relies on palpating cartilage landmarks on the neck to place the incision. In order to achieve the accelerated goals of the robot-assisted subtask in a tracheotomy, this paper proposes a novel autonomous palpation-based acquisition strategy - RASEC in the tracheal region, which can interactively predict the next acquisition point to maximize the expected information and minimize the costs of palpation procedure. We employ a Gaussian Process (GP) to model the distribution of hardness and utilize anatomical information as a priori input to guide the point of palpation for medical robots. The dynamic tactile sensor based on the resonant frequency is introduced to measure tissue hardness in the tracheal region by millimeter-scale contact to secure the interaction. We investigate the kernel fusion method to blend the Squared Exponential (SE) kernel with the Ornstein-Uhlenbeck (OU) kernel and optimize the Bayesian optimization search by leveraging the anatomical information of the larynx as a priori knowledge. Moreover, we further regularize the exploration and greed factors. The tactile sensor’s moving distance and the robotic base link’s rotation angle during the incision localization process are considered new factors in the acquisition strategy. Simulation and physical phantom experiments are conducted for comparison with state-of-the-art GP-based exploration approaches. The results show that the sensor’s moving distance was reduced by 53.1% and the rotation angle of the base was reduced by 75.2% of the previous values without sacrificing overall performance capabilities. The satisfying algorithmic index (average precision 0.932, average recall 0.973, average F1 score 0.952) with fewer central estimation distance errors (0.423 mm) and high resolution (1 mm) indicates the performance of the proposed RASEC in terms of exploration efficiency, cost awareness, and localization accuracy for incision localization and recommendation in real robot-assisted subtask in the tracheotomy procedure.Note to Practitioners—This work is well motivated to introduce the Level of Autonomy (LoA) 2 - task-level autonomy, specifically in the context of tracheotomy procedures. The incorporation of robotic palpation techniques aims to provide surgeons with enhanced capabilities for incision recommendations, which directly benefit surgeons to visualize hands-on information and localize the trachea regions more efficiently and further reduce cognitive load. To detect the trachea region for intubation incision without costly ergodic acquisition, this article suggests a highly efficient acquisition strategy utilizing the fusion kernel function and regularized impact factors, eliminating the time consumption for such localization task. The actual clinical value is that our proposed strategy can earn more time for further increasing the probability of patient resuscitation, to facilitate supervised autonomy in the real clinic scene.

Deep learning has demonstrated exceptional performance in medical image analysis, but its effectiveness degrades significantly when applied to different medical centers due to domain shifts. Lesion detection, a critical task in medical imaging, is particularly impacted by this challenge due to the diversity and complexity of lesions, which can arise from different organs, diseases, imaging devices, and other factors. While collecting data and labels from target domains is a feasible solution, annotating medical images is often tedious, expensive, and requires professionals. To address this problem, we combine active learning with domain-invariant feature learning. We propose a Domain-shift Active Learning (DistAL) framework, which includes a transferable feature learning algorithm and a hybrid sample selection strategy. Feature learning incorporates contrastive-consistency training to learn discriminative and domain-invariant features. The sample selection strategy is called RUDY, which jointly considers Representativeness, Uncertainty, and DiversitY. Its goal is to select samples from the unlabeled target domain for cost-effective annotation. It first selects representative samples to deal with domain shift, as well as uncertain ones to improve class separability, and then leverages K-means++ initialization to remove redundant candidates to achieve diversity. We evaluate our method for the task of lesion detection. By selecting only 1.7% samples from the target domain to annotate, DistAL achieves comparable performance to the method trained with all target labels. It outperforms other AL methods in five experiments on eight datasets collected from different hospitals, using different imaging protocols, annotation conventions, and etiologies.

In this paper, we present a simultaneous exploration and object search framework for the application of autonomous trolley collection. For environment representation, a task-oriented environment partitioning algorithm is presented to extract diverse information for each sub-task. First, LiDAR data is classified as potential objects, walls, and obstacles after outlier removal. Segmented point clouds are then transformed into a hybrid map with the following functional components: object proposals to avoid missing trolleys during exploration; room layouts for semantic space segmentation; and polygonal obstacles containing geometry information for efficient motion planning. For exploration and simultaneous trolley collection, we propose an efficient exploration-based object search method. First, a traveling salesman problem with precedence constraints (TSP-PC) is formulated by grouping frontiers and object proposals. The next target is selected by prioritizing object search while avoiding excessive robot backtracking. Then, feasible trajectories with adequate obstacle clearance are generated by topological graph search. We validate the proposed framework through simulations and demonstrate the system with real-world autonomous trolley collection tasks.

Continuum robots, characterized by their high flexibility and infinite degrees of freedom (DoFs), have gained prominence in applications such as minimally invasive surgery and hazardous environment exploration. However, the intrinsic complexity of continuum robots requires a significant amount of time for their motion planning, posing a hurdle to their practical implementation. To tackle these challenges, efficient motion planning methods such as Rapidly Exploring Random Trees (RRT) and its variant, RRT*, have been employed. This paper introduces a unique RRT*-based motion control method tailored for continuum robots. Our approach embeds safety constraints derived from the robots’ posture states, facilitating autonomous navigation and obstacle avoidance in rapidly changing environments. Simulation results show efficient trajectory planning amidst multiple dynamic obstacles and provide a robust performance evaluation based on the generated postures. Finally, preliminary tests were conducted on a two-segment cable-driven continuum robot prototype, confirming the effectiveness of the proposed planning approach. This method is versatile and can be adapted and deployed for various types of continuum robots through parameter adjustments.

Learning-based methods have improved locomotion skills of quadruped robots through deep reinforcement learning. However, the sim-to-real gap and low sample efficiency still limit the skill transfer. To address this issue, we propose an efficient model-based learning framework that combines a world model with a policy network. We train a differentiable world model to predict future states and use it to directly supervise a Variational Autoencoder (VAE)-based policy network to imitate real animal behaviors. This significantly reduces the need for real interaction data and allows for rapid policy updates. We also develop a high-level network to track diverse commands and trajectories. Our simulated results show a tenfold sample efficiency increase compared to reinforcement learning methods such as PPO. In real-world testing, our policy achieves proficient command-following performance with only a two-minute data collection period and generalizes well to new speeds and paths.

In this paper, we introduce a novel overlap-based bidirectional point set registration approach, i.e., Overlap-guided Bidirectional Hybrid Mixture Registration (OBHMR), which incorporates geometric information (i.e., normal vectors) in both the correspondence and transformation stages and formulates the optimization objective of registration in a bidirectional manner. More importantly, to address the issue of partial-to-full registration, OBHMR utilises the predicted point-wise overlap score using networks to formulate the overlap-guided Hybrid Mixture Model consisting of the Gaussian Mixture Model (GMM) and Fisher Mixture Model (FMM). OBHMR contains four components: (1) the overlap-guided correspondence network that estimates the correspondence probabilities and calculates the point-wise overlap score; (2) the learning posterior module that estimates the overlap-guided HMM parameters; (3) the transformation module that computes the rigid transformation by formulating the optimisation objective in a bidirectional registration way, given correspondences and overlap-guided HMM parameters. Experiments using 1457 human femur and 1301 human hip models demonstrate significant improvements in partial-to-full registration performance (p < 0.01) under different overlapping ratios, compared to state-of-the-art registration approaches. Furthermore, individual contributions of three modules (i.e., additional normal vectors, overlap score estimation module and the bidirectional mechanism) in OBHMR have been validated in ablation studies. The results demonstrate OBHMR’s capability of tackling the challenging partial-to-full registration problems in computer-assisted orthopedic surgery. The codes are available at https://github.com/Dxinz/DeepOBHMR.

In recent years, great progress has been made in forecasting human motion in crowded scenes. However, current methods are far from practical applications due to the unbearable high computation costs, especially for encoding scene context. In addition, neglecting the partially detected trajectories makes the predicted outcome deviate from the real trajectory distribution. To handle the aforementioned concerns, we propose to represent the scene context and partially observed trajectories with sparse graphs. Customized for this special data structure, we design a hierarchical Graph Transformer Network model SparseGTN to predict multiple possible future trajectories of the target pedestrian by digesting the sparsely represented inputs. Our approach exhibits superiority over the state-of-the-art (SOTA) methods, utilizing a mere 3.42% of the number of floating point operations (FLOPs) and 0.53% of the number of model parameters. The code will be available online⋆.

In this paper, we introduce a novel normal-assisted learning-based rigid registration approach, i.e., Deep Bi-directional Hybrid Mixture Registration (DeepBHMR). Our approach utilises helpful normal vectors explicitly in both correspondence and transformation stages and formulates the optimization objective of registration in a bi-directional way that considers noise in both point sets. DeepBHMR consists of three modules: (1) the correspondence network that estimates the correspondence probability relating points within one generalized point set (i.e., positional and normal vectors) with components of Hybrid Mixture Models (HMMs) representing the other generalized point set; (2) the posterior module that computes HMMs parameters; (3) the transformation module that computes the rotation matrix and the translation vector given the estimated generalized-point to hybrid-distribution correspondences and HMMs parameters. DeepBHMR has been validated on 291 human femur and 260 hip models, and extensive experimental results demonstrate that DeepBHMR outperforms the state-of-the-art registration methods (p-value < 0.01). In the circumstance of femur bones, the mean rotation and translation error values are around 1° (i.e., 1.01°) and less than 1 mm (i.e., 0.36mm), respectively. Furthermore, even under the large transformation (i.e., in the range of [0,180]° and [0, 100] mm), the mean RMSE values being 3.05 mm is still satisfactory. Additionally, the results demonstrate the DeepBHMR’s favorable generalizability from femur shapes to hip shapes. We have carefully validated the significant benefits of incorporating normal vectors and the bidirectional mechanism. DeepBHMR can successfully handle the challenging scenario of large transformation and partial registration. The codes are available at https://github.com/zzyrobot/DeepBHMR.git.

In this paper, we introduce Bidirectional Non-Overlapping Filtering Network (Bi-NOFNet), which registers the partial intraoperative point set with full preoperative point set for computer-assisted interventions (CAI). Our contributions are three-folds. First, Bi-NOFNet adopts customised feature extractor to extract distinctive features from both point sets, with which the per-point overlap mask is predicted and the overlapping region is segmented for the preoperative point set. Furthermore, we propose two methods to filter out the non-overlapping regions, at feature-level (i.e., Bi-NOFNet(Feature)) and point-level (i.e., Bi-NOFNet (Point)). For these two methods, we develop supervised registration strategy where the ground-truth overlap mask and displacement vectors are employed, and weakly-supervised registration strategies where only the ground-truth overlap mask is available. Additionally, to fully utilise the information in both space, we propose a bidirectional registration mechanism, which predicts the displacement vectors associated with the intraoperative point set (i.e., the forward way) and those warpping the preoperative point set (i.e., the backward way). Experiments have been conducted on the proposed DeformMedShapeNet dataset that contains 615 different liver shapes. Extensive results demonstrate that Bi-NOFNet performs well for partial-to-full registration tasks under various scenarios of noise, overlap ratios and deformation levels, outperforming existing non-rigid registration approaches.

An intelligent robot grasping system should be able to automatically grasp a variety of objects that have never been seen, which requires accurate and efficient grasp pose detection. To this end, we propose a deep grasp detector designed for the robot equipped with a parallel gripper. The deep model consumes RGB or depth data and extracts features via a feature pyramid network (FPN), followed by multiple grasp prediction units to output grasp parameters in a single stage without refining process. Attaching grasp prediction units to different FPN stages increases the model capability to predict different-size grasps. Furthermore, in each prediction unit, the grasp parameters are regressed with the horizontal anchor as a reference to overcome the challenges posed by the various shapes of the grasp regions. We improve the accuracy and efficiency of grasp rotation estimation by regressing the angle directly and encoding the angle with a continuous Gaussian-like curve during training. This encoded angle regression strategy provides distance information of different angle predictions without introducing additional computational costs. Evaluations on three datasets prove the superior performance of our method than state of the arts. The experiments in real scenarios further validate the effectiveness of our grasping system.Note to Practitioners—This paper proposes a robot system that can automatically grasp novel objects with a parallel gripper and RGB-D camera. We focus on generating accurate grasp configurations for various objects using the captured color or depth image, which is the cornerstone of a successful grasp. To obtain effective and efficient grasp pose detection, we present a deep model that generates robust grasp poses represented by rotated bounding boxes for multiple novel objects. The first step of the grasp detector is to capture the image features through a feature pyramid network (FPN). Then, we attach separate grasp prediction units to each layer of the FPN stage and adopt anchors as references to make the model robust to variable grasp rectangle sizes. In each grasp prediction unit, two separate subnetworks are used to directly output the grasp rectangles and their probabilities, without using an extra second stage to refine the predicted grasp areas. For the prediction of rotation angle, we encode the rectangle angles with a continuous Gaussian-like curve during training to improve the prediction accuracy. Our grasp detector is trained and tested on three datasets and validated on real-scene grasp experiments. Comparisons with state-of-the-art methods show that our model is more accurate while maintaining high efficiency. The proposed grasp detection model can be applied to generate stable grasps for novel objects with different shapes, colors, and materials. Our grasping system is capable of working in multiple scenarios, including homes, factories, and warehouses.

Cable-driven surgical instrument is a reusable and disposable component of Robot-assisted Minimally Invasive Surgery, and it is time-consuming to identify the motion hysteresis compensation model of each instrument in large-scale identification. Besides, the identification results failure caused by the loss of cable tension or lubrication conditions during repeated usage interferes with the accuracy of motion hysteresis compensation. To simplify the identification process and lengthen the instrument life, this paper leverages the relationship between actuate motor current and hysteresis phases, developing a novel motion hysteresis compensation method to compensate the motor reference trajectory. Firstly, prior knowledges including motor current and hysteresis curves of several instruments are collected to train the Hysteresis Identification model and Curve Generation model. Once the two models are well-trained in practical use, the only sensor data in need is the motor current for the Hysteresis Identification model, and the compensation curve will be generated pertinently for the instrument under current use. Finally, Feedforward Compensation scheme is conducted to compensate the reference trajectory of the actuate motor. Experiments study the range of hysteresis compensation errors when the method is applied to numerous surgical instruments with different motion hysteresis, and the feasibility and accuracy are verified. The method can be potentially applied to a wide range of cable-driven mechanisms facing the problem of large-scale identification or the identification results failure after repeated use. Note to Practitioners—This paper proposes a feasible motion hysteresis compensation method for cable-driven mechanism (CDM). For some CDMs like elongated cable-driven surgical instrument, their motion hysteresis characteristics will change during repeated use, and their distal angle and cable tension are not available in the environment of use. In these cases, the proposed method enables re-identification because the distal angle or cable tension are not needed during the practical use of our method. Besides, it is easy and low-cost for large-scale identification due to the simplicity of identification procedure. Once the Hysteresis Identification model and Curve Generation model are trained using the pre-collected prior knowledge including motor current and the angle of the end effector, the only sensor data in need to identify and compensate a certain CDM is the its motor current. Then, the pertinent compensation curve for the CDM device can be generated for the actuate motor to follow. The experimental results reveal that the competitive performance with other state-of-the-arts can be achieved. In the future research, we will integrate dynamic characteristics of CDM into the method to expand the scope of application.

Ultrasound image acquisition in conventional transesophageal echocardiography (TEE) requires complex manual operation of the probe in the esophagus based on the interpretation of ultrasound images and in-depth knowledge of the cardiac anatomy. In this work, we formulate the TEE probe guidance task as a reinforcement learning (RL) problem, and present the first learning-based solution to 3-DOF control of a TEE probe based on the ultrasound image feedback, named RL-TEE, in order to mimic the visual search and navigation strategies of expert echocardiographers. The probe-tissue interaction in TEE is carefully modeled in our framework by considering both the requirements for navigation towards the standard views and compliance in the esophageal environment. Furthermore, we propose a hybrid deep Q-network model that augments a convolutional neural network backbone with self-attention mechanisms to better capture spatial information in ultrasound images to guide navigation decisions. The presented methods are preliminarily validated in a TEE simulation environment built with data from 25 subjects to acquire four standard views of the heart. Our results show that the proposed method can effectively learn to accurately and compliantly guide the probe movement for TEE standard view acquisition tasks and has a good generalization ability to unseen patient data. Note to Practitioners—The motivation of this paper is to realize 3-DOF movement guidance of a TEE probe to acquire the standard views of the heart based on the real-time images, which can be applied to existing robotic control systems or used to assist novice echocardiographers in TEE examination, thereby relieving operator workload and improving ease of use. This paper suggests a novel approach that uses the deep RL technique to achieve automatic interpretation of TEE images and intelligent guidance of the probe movement. The RL framework is designed to take into account both the navigation efficiency and compliance with the esophageal environment for the targeted intracorporeal application. A hybrid deep Q-network model that augments a convolutional neural network with attention mechanisms is designed to better capture spatial information from ultrasound images to predict the probe movement. The effectiveness of the framework is preliminarily validated in extensive experiments in a simulation environment built with real patient data. The proposed method can be applied in clinical use to provide real-time TEE probe guidance for novice echocardiographers, and can be integrated with a robotic system to fully automate the TEE acquisition, thereby relieving the doctors from tedious manual operation to focus on the diagnosis and treatment.

Adaptively Informed Trees (AIT*) is an algorithm that uses the problem-specific heuristic to avoid unnecessary searches, which significantly improves its performance, especially when collision checking is expensive. However, the heuristic estimation in AIT* consumes lots of computational resources, and its asymmetric bidirectional searching strategy cannot fully exploit the potential of the bidirectional method. In this article, we propose an extension of AIT* called BiAIT*. Unlike AIT*, BiAIT* uses symmetrical bidirectional search for both the heuristic and space searching. The proposed method allows BiAIT* to find the initial solution faster than AIT*, and update the heuristic with less computation when a collision occurs. We evaluated the performance of BiAIT* through simulations and experiments, and the results show that BiAIT* can find the solution faster than state-of-the-art methods. We also analyze the reasons for the different performances between BiAIT* and AIT*. Furthermore, we discuss two simple but effective modifications to fully exploit the potential of the adaptively heuristic method.Note to Practitioners—This work is inspired by the adaptively heuristic method and the symmetrical bidirectional searching method. The article introduces a novel algorithm that uses the symmetrical bidirectional method to calculate the adaptive heuristic and efficiently search the state space. The problem-specific heuristic in BiAIT* is derived from a lazy-forward tree and a lazy-reverse tree, which are constructed without collision checking. The lazy-forward and lazy-reverse trees are enabled to meet in the middle, thus generating the effective and accurate heuristic. In BiAIT*, the lazy-forward and lazy-reverse trees share heuristic information and jointly guide the growth of the forward and reverse trees, which conduct collision checking and guarantee the feasibility of their edges. Compared with state-of-the-art methods, BiAIT* finds the initial heuristic and updates the heuristic more quickly. The proposed algorithm can be applied to industrial robots, medical robots, or service robots to achieve efficient path planning. The implementation of BiAIT* is available at https://github.com/Licmjy-CU/BiAITstar.

Balancing the trade-off between safety and efficiency is of significant importance for path planning under uncertainty. Many developed risk-aware path planners explicitly limit the probability of collision to an acceptable bound in uncertain environments. However, convex obstacles or Gaussian uncertainties are usually assumed to make the problem tractable in the existing method. These assumptions limit the generalization and application of path planners in real-world implementations. In this article, we propose to apply deep learning methods to the sampling-based planner, developing a novel risk bounded near-optimal path planning algorithm named neural risk-aware RRT (NR-RRT). Specifically, a deterministic risk contours map is maintained by perceiving the probabilistic nonconvex obstacles, and a neural network sampler is proposed to predict the next most-promising safe state. Furthermore, the recursive divide-and-conquer planning and bidirectional search strategies are used to accelerate the convergence to a near-optimal solution with guaranteed bounded risk. Worst-case theoretical guarantees can also be proven owing to a standby safety guaranteed planner utilizing a uniform sampling distribution. Simulation experiments demonstrate that the proposed algorithm outperforms the state-of-the-art for finding risk bounded low-cost paths in uncertain nonconvex environments with seen and unseen scene layouts. Note to Practitioners—This article is motivated by developing an efficient risk-aware path planner that can quickly find risk bounded solutions in uncertain nonconvex environments for practical applications, such as autonomous vehicles and search-and-rescue robots. Sampling-based planning approaches such as rapidly-exploring random tree (RRT) and its variants are popular for their good performance in exploring the state space. However, it is quite time-consuming to look for risk bounded paths in uncertain environments, especially under nonconvex and non-Gaussian constraints. The initial paths are often of poor quality. Therefore, we propose the NR-RRT algorithm to rapidly find near-optimal solutions with guaranteed bounded risk. It utilizes an informed bidirectional search strategy after having past experiences in the challenging environments. It can be applied in not only seen uncertain scenarios but also those have unseen scene layouts different from the training scenarios. However, the algorithm cannot handle the problem in environments that contain entirely unseen obstacles. In future research, we will address the problem of planning under robot model uncertainty.

Building a general and efficient path planning framework in uncertain nonconvex environments is challenging due to the safety constraints and complex configuration. Traditional avenues usually involve convexifying obstacles and presume Gaussian distribution, which are not universal. Meanwhile, the fast convergence of high-quality solutions is not guaranteed. Therefore, we develop a novel neural risk-bounded path planner to quickly find near-optimal solutions that have an acceptable collision probability in the complex environments. Firstly, we retrieve the nonconvex obstacles with arbitrary probabilistic uncertainties in the form of a deterministic point cloud map. A neural network sampler encodes it into a latent embedding and is trained with sufficient expert demonstrations, predicting states in the potential subspace. We construct a neural cost estimator to select the best informed state from those samples. Then, we recursively use the simple yet effective neural networks to march toward the start and goal bidirectionally. The collision risk of the intermediate connections is verified based on sum-of-squares optimization. Simulation results show that our approach significantly saves time and resources in finding comparable solutions over the state-of-the-art methods in the seen and unseen challenging environments.Note to Practitioners—More and more robots are deployed in unstructured environments, such as forests and subterranean caves. However, uncertainty in the environment situational awareness usually causes accidents. To quickly generate safe paths without over-conservation in uncertain complex environments, we propose a neural risk-bounded sampling-based path planner. Conventional methods consume lots of computation time and resources to generate satisfactory results. Our learning-based risk-bounded path planning framework can efficiently find paths with a guaranteed risk tolerance avoiding uncertain nonconvex static obstacles. It imitates the expert to generate informed states in a subspace that potentially contains the optimal solution. In practice, we need to formulate the observed uncertain obstacle at a grid map into the polynomial containing random variables and determine their probability distributions.

Reorienting objects by using supports is a practical yet challenging manipulation task. Owing to the intricate geometry of objects and the constrained feasible motions of the robot, multiple manipulation steps are required for object reorientation. In this work, we propose a pipeline for predicting various object placements from point clouds. This pipeline comprises three stages: a pose generation stage, followed by a pose refinement stage, and culminating in a placement classification stage. We also propose an algorithm to construct manipulation graphs based on point clouds. Feasible manipulation sequences are determined for the robot to transfer object placements. Both simulated and real-world experiments demonstrate that our approach is effective. The simulation results underscore our pipeline’s capacity to generalize to novel objects in random start poses. Our predicted placements exhibit a 20% enhancement in accuracy compared to the state-of-the-art baseline. Furthermore, the robot finds feasible sequential steps in the manipulation graphs constructed by our algorithm to accomplish object reorientation manipulation.Note to Practitioners—Object reorientation is a prevalent manipulation task in both domestic and industrial manufacturing scenarios. Extrinsic supporting items are often used to provide diverse object placements that allow for feasible grasp configurations for robotic manipulation. In previous methods, utilizing mesh models of objects was necessary to ascertain stable placements and construct manipulation graphs. In this work, we propose a data-driven approach to predict various object placements conditioned on point clouds. Moreover, we use predicted point cloud placements to construct manipulation graphs, which facilitate collision-free pick-and-place steps to reorient objects. Our approach demonstrates the capacity to generalize to novel objects. In future work, we will enhance the performance of our pipeline by optimizing the distance metric used for measuring pose discrepancies and improving the classifier model.

The fiber-guided laser scanner has been widely adopted in endoscopic microsurgeries due to its outstanding precision, swiftness, and dexterity for incision, ablation, and photocoagulation of some difficult-to-reach tissues, e.g., the laryngeal, respiratory, and alimentary sites. However, fabricating and actuating such precision surgical instruments are challenging, considering the strictly limited workspace and the high-performance requirements. This article proposes a compact fiber-guided laser scanner with two rotational degrees of freedom (DOFs), one of which is actuated with the stepper motor and the other controlled by electromagnetism. The novel design of the electromagnetically driven system allows the laser scanner for a more compact radial dimension while maintaining high precision in trajectory tracking tasks. In addition, we develop a robust control strategy based on visual feedback to achieve autonomous detecting and tracking of the trajectory targets. In vitro tests are subsequently performed to validate the efficacy and accuracy of our proposed system. The experimental results reveal that the average tracking error of our laser steering system is as low as$ {279.29~{\mu~{\mathrm {m}}}}$, which is deemed satisfactory in practical endoscopic microsurgeries. Note to Practitioners—This article is motivated by the unmet needs of in vivo laser endoscopic microsurgery in an automated fashion, and simultaneously the proposed methodology can be applied to other laser scanning-related scenarios. Existing solutions to in vivo laser scanning generally are associated with complex micro-electromechanical systems (MEMS), and hence high costs and relatively bulky footprint. Moreover, existing manual control schemes cannot enable high-precision laser scanning in an automated fashion. This article demonstrates a novel approach incorporating mechanism and electromagnetism to actuate the laser scanner, whereby achieving a highly compact design at low costs. Further, a closed-loop control strategy is developed for such an electromagnetically driven laser scanner, enhancing its scanning accuracy and reliability. This article theoretically analyzes the mechanics and electromagnetics of the laser scanner. Subsequently, it proposes a closed-loop control scheme through endoscopic visual feedback, allowing the laser scanner for autonomous target tracking. The pilot in vitro experiments demonstrate the feasibility and reliability of this holistic approach. Ultimately, the design and control of the electromagnetically driven laser scanner would be further optimized and validated for its laser ablation performance on ex vivo tissues or animal models in the future.

In medical robotics and image-guided surgery (IGS), registration is needed in order to align together the coordinate frames of robots, medical imaging modalities, surgical tools, and patients. Existing registration algorithms often assume one point set to be a noise-free model while the other to contain noise and outliers. However, in real scenarios, noise and outliers can exist in both point sets to be registered. To eliminate the above-mentioned challenge, in this paper, we formally formulate the Bi-directional Generalised Rigid Point Set Registration (Bi-GRPSR) problem where normal vectors are adopted, bi-directional probability density function (PDFs) and Hybrid Mixture Models (HMMs) are constructed to derive the objective function. Bi-GRPSR considering anisotropic positional noise is thus cast as a maximum likelihood estimation (MLE) problem, which is solved by the proposed Bi-directional Generalised Anisotropic Coherent Point Drift (Bi-AGCPD) where spatially nearby points are considered to move coherently and iterative expectation maximization (EM) steps are involved. Experimental results on two human bone point sets, under different settings of noise, outliers, and overlapping ratios, validate the effectiveness and improvements of Bi-AGCPD over existing probabilistic and learning-based methods.Note to Practitioners—This paper presents a novel rigid point set registration method that explicitly takes the anisotropic noise in both point sets into account. The proposed framework first formulates the probability density functions of generalised points in a bi-directional way, with which the bi-directional hybrid mixture model is built. The resulting objective function is minimised with the expectation maximisation technique. The algorithms are essential for real-world applications in that noise usually exists in both spaces to be registered and is generally anisotropic. The proposed method has demonstrated promising results on two human femur bone models, which indicates the great potential for it to be readily applied to related applications, especially medical scenarios, given that the two point sets are coarsely aligned. Future work will extend the presented method into scenarios of global registration.

The utilization of cloud infrastructure and its extensive range of Internet-accessible resources holds significant potential for advancing intelligent transportation and robotics. Over the past two decades, interest in cloud-edge collaborative simultaneous localization and mapping (SLAM) has grown markedly. Consequently, a comprehensive review of current trends in this field is crucial for both novice and experienced researchers. This paper examines robots and automation systems that rely on network-based data or code, particularly in the context of SLAM development. Applying SLAM to mobile robots with limited computing power is essential for achieving autonomous navigation, and cloud-edge collaborative SLAM has emerged as an efficient solution. The review is structured around four key benefits of cloud-edge collaborative SLAM: Assisted Cloud Computing, which provides access to cloud computation and reduces the burden on edge devices; Total Cloud Computing, where the majority of computation is offloaded to the cloud, while edge devices primarily handle sensing and low-cost pre-processing; Data Storage, enabling access to large datasets, such as high-resolution environment maps and extensive training datasets, enhancing overall performance; and Data Transmission, involving cloud-edge communication for efficient data transfer and data association. Additionally, we address the challenges in existing work and the development of asynchronous collaboration and implicit representation transmission, which could mitigate transmission latency in communication-constrained environments. We believe that this review will bridge the gap between SLAM systems and deployed robotic systems, promoting the advancement of cloud-edge collaborative SLAM.

This paper proposes a new approach to achieve direct visual servoing (DVS) based on discrete orthogonal moments (DOMs). DVS is performed in such a way that the extraction of geometric primitives, matching, and tracking steps in the conventional feature-based visual servoing pipeline can be bypassed. Although DVS enables highly precise positioning, it suffers from a limited convergence domain and poor robustness due to the extreme nonlinearity of the cost function to be minimized and the presence of redundant data between visual features. To tackle these issues, we propose a generic and augmented framework that considers DOMs as visual features. By using the Tchebichef, Krawtchouk, and Hahn moments as examples, we not only present the strategies for adaptively tuning the parameters and order of the visual features but also exhibit an analytical formulation of the associated interaction matrix. Simulations demonstrate the robustness and accuracy of our approach, as well as its advantages over the state-of-the-art. Real-world experiments have also been performed to validate the effectiveness of our approach.

Optical Coherence Tomography (OCT) is a non-invasive imaging technique that is instrumental in retinal disease diagnosis and treatment. Segmentation of retinal layers in OCT is an essential step, but remains challenging for common pixel-wise segmentation methods usually fail to obtain the correct layer topology. To tackle this challenge, we propose a novel Locate-to-Segment (L2S) framework to provide a layer region location guidance for pixel-wise labeling learning so as to obtain better segmentation with the correct topology and smooth boundaries. Specifically, a Structured Boundary Regression Network (SBRNet) is devised to first predict the surface positions. For effective learning on normal-size images, we design two regression branches to regress the top surface and eight layer widths separately in SBRNet to locate each layer region with absolutely correct orderings. Then, we take the prediction of SBRNet as an additional input for a common pixel-wise segmentation network to provide the guidance of correct topology. In this L2S manner, our framework takes merits of regression-based methods and pixel-wise labeling-based methods to obtain accurate segmentation with the correct topology and smooth continuous boundaries. Experimental results on a public retinal OCT dataset demonstrate the effectiveness of our method, outperforming state-of-the-art segmentation methods with the highest average Dice score of 90.29% and the lowest average MAD score of 0.782.

In medical robotics and image-guided surgery (IGS), registration is needed in order to align together the coordinate frames of robots, medical imaging modalities, surgical tools, and patients. Existing registration algorithms often assume one point set to be a noise-free model while the other to contain noise and outliers. However, in real scenarios, noise and outliers can exist in both point sets to be registered. To eliminate the above-mentioned challenge, in this paper, we formally formulate the Bi-directional Generalised Rigid Point Set Registration (Bi-GRPSR) problem where normal vectors are adopted, bi-directional probability density function (PDFs) and Hybrid Mixture Models (HMMs) are constructed to derive the objective function. Bi-GRPSR considering anisotropic positional noise is thus cast as a maximum likelihood estimation (MLE) problem, which is solved by the proposed Bi-directional Generalised Anisotropic Coherent Point Drift (Bi-AGCPD) where spatially nearby points are considered to move coherently and iterative expectation maximization (EM) steps are involved. Experimental results on two human bone point sets, under different settings of noise, outliers, and overlapping ratios, validate the effectiveness and improvements of Bi-AGCPD over existing probabilistic and learning-based methods.

Planning a collision-free path efficiently among obstacles is crucial in robotics. Conventional one-shot unidirectional path planning algorithms work well in the static environment, but cannot respond to the environment changes timely in the dynamic environment. To tackle this issue and improve the search efficiency, we propose a bidirectional incremental search method, Bidirectional Lifelong Planning A* (BLPA*), which searches in the forward and backward directions and performs incremental search bidirectionally when the environment changes. Furthermore, inspired by the robot perception range limitation and BLPA*, we propose the fractional bidirectional D* Lite (fBD* Lite(dp)), which constraints the forward search to the robot perception range and uses the backward search to expand the rest area. Our simulation results demonstrate BLPA* and mD* Lite(dp) can achieve superior performance in the dynamic environment. It reveals that the bidirectional incremental search strategy can be a general and efficient technique for graph-search-based robot path planning methods.

Cooperative object transportation using multiple robots has been intensively studied in the control and robotics literature, but most approaches are either only applicable to omnidirectional robots or lack a complete navigation and decision-making framework that operates in real time. This paper presents an autonomous nonholonomic multi-robot system and an end-to-end hierarchical autonomy framework for collaborative luggage trolley transportation. This framework finds kinematic-feasible paths, computes online motion plans, and provides feedback that enables the multi-robot system to handle long lines of luggage trolleys and navigate obstacles and pedestrians while dealing with multiple inherently complex and coupled constraints. We demonstrate the designed collaborative trolley transportation system through practical transportation tasks, and the experiment results reveal their effectiveness and reliability in complex and dynamic environments. (Video11Video demonstration: https://youtu.be/efnPERm0Rco.)

This paper presents a novel augmented reality (AR)-assisted orthopedic surgical robotic system based on Head-Mounted Display (HMD) devices. The proposed system can overlay the preoperative plans over the patient's anatomy and provide useful guidance for surgeons during interventions, with integrated calibration and registration components. A novel bi-directional generalised point set registration algorithm that utilises robust features is developed to accurately align the pre-operative CT and intra-operative patient spaces, which has been demonstrated to outperform existing registration methods. The efficacy of the system is both qualitatively and quantitatively assessed with an in vitro study simulating a total knee arthroplasty (TKA) procedure. The experimental results showed that 1) the system can successfully align the preoperative and intraoperative spaces, with the mean target registration error (TRE) being 2.7771 mm; 2) the models can be properly overlaid to the physical scenarios with the mean AR visualization accuracy being 6.9726 mm.

Human-aware path planner is essential for achieving harmonious coexistence between humans and robots in highly dynamic environments. In this paper, we propose an integrated framework to find the optimal path in the complex environment with considering collision risk, social norms, and crowded areas. In the proposed framework, a general dynamic group model (g-space) based on the Gaussian Mixed Model (GMM) is proposed as the social norms of dynamic groups, which not only considers the factors of humans (e.g., pose, quantity, distribution, psychology) but also establishes the proximity and human interacting constraints of dynamic groups. An integrated Collision Risk and Human Space (CR&HS) model is applied to achieve human-acceptable behaviors, in which both collision avoidance, human comfort, and interference-free constraints have been involved. Moreover, an Improved Virtual Doppler Method (IVDM) has been used to realize safety navigation to avoid the robot falling into the crowded area. Finally, the proposed framework has been utilized with the sampling-based rapidly-exploring random tree. Experimental results demonstrate that the proposed method can generate the optimal human-aware collision-free path in complex environments. Note to Practitioners—This paper aims to plan an optimal trajectory for the robot in highly dynamic environments. In this field, it is still a challenging task to plan a trajectory with collision-free, human-aware, and crowd-aware. To do that, we present an integrated framework to generate the optimal trajectory by involving the collision risk, social norms, and human density. First, the g-space model is adopted as interference-free constraints of dynamic groups. The integrated knowledge fusion model (CR&HS) then penalizes the manners which have higher collision risk and adverse effects on human interaction or human comfortable. Besides, human motion and density are provided to a robot by IVDM. The proposed framework is utilized in the sampling-based rapidly-exploring random tree as the evaluation module. Finally, the feasibility and reliability of the proposed method have been verified by experiments in different simulated environments. The proposed framework can be applied in most mobile service robots to achieve human-friendly manners.

An endoscope holder must understand the detailed surgical actions and the surgeons’ visual attention to keep important targets in the field of endoscopic view during operations. From an intensive analysis of the surgeons’ attention mechanism, we included that surgical actions, like cutting, suturing, etc., play an important role in determining the positions and weights of visual attention points during a dynamic surgical scene. To perform this process, this work proposes a Surgical Action-driven Visual Attention network (SAVAnet) and applies the network in autonomous endoscope control. Four scenarios are constructed in the da Vinci V-rep simulator: pick&place and needle exercise in a general laparoscopic training environment, needle driving with and without obstacle removal in an abdominal cavity, to create datasets for network training. The results show that the network has an outstanding performance in surgical action prediction with a high average accuracy of over 91%. Additionally, with surgical action guidance, the attention point prediction has higher accuracy and accords with surgeons’ visual attention. Finally, the acquired attention points are utilized to execute visual servoing in simulation. The results verify that the SAVAnet is feasible for autonomous endoscope control in real-time and lays a theoretical foundation for future sim-to-real execution. Note to Practitioners—This paper was motivated by the problem of endowing an endoscope with surgeons’ visual attention mechanism, which is affected by surgical actions, for autonomous endoscope control. An eye-tracking device has been utilized to detect surgeon’s visual attention in real-time and then control the endoscope to follow what the surgeon is looking at. However, this approach is susceptible to the surgical environment. Besides, many instrument detection and segmentation algorithms are developed for automatic surgical instrument tracking. However, surgeons’ visual attention does not always focus on the instruments during operations. In this work, we propose a novel SAVAnet to determine visual attention based on surgical actions. We prove from many qualitative and quantitative experiments that surgical actions play a significant role in determining visual attention. The designed SAVAnet can predict surgical actions correctly and then effectively guide the choice of visual attention. Finally, the simulation results show that the SAVAnet can endow endoscope with surgeons’ visual attention to perform self-control in real time. In future research, we will train the SAVAnet using real datasets and conduct more physical experiments on real surgical robots.

This paper proposes a network called points-guided sampling net (PGSN) to guide the sampling process in sampling-based motion planner by utilizing the geometric information of obstacles. The geometric information is extracted from the point cloud of obstacles. By analyzing the properties of the point cloud, we propose a VAE feature extraction net that incorporates the variational autoencoder (VAE) framework with unique architectures designed for point clouds. Furthermore, we design a multi-modal sampling net to model the probability distribution of the states based on training trajectories taken from different environments. Based on PGSN, we propose a sampling-based motion planning algorithm called the point-guided rapidly-exploring random tree (PG-RRT). Three experiments are conducted to verify the proposed PGSN: Exp I shows the proposed VAE feature extraction net can successfully extract geometric features from the inputted point cloud; Exp II verifies the multi-modal sampling net successfully chooses corresponding mode with respect to extracted features; Exp III demonstrates the efficacy of our PG-RRT algorithm by showing PG-RRT outperforms other algorithms. Moreover, we provide theoretical analysis and insights towards understanding our model. Note to Practitioners—Obstacles cause lots of the sampling space invalid, thus the traditional sampling-based motion planning (SBMP) algorithm is usually unable to generate a trajectory within a reasonable short period of time. To improve the success rate and efficiency of SBMP, this paper proposes a novel deep neural network called points-guided sampling net (PGSN). PGSN is designed to exploit: (1) environmental point clouds and (2) training trajectories from multiple environments with different obstacles. In the first step, the point clouds include important geometric information. To utilize this information, we adopt a variational autoencoder approach which combines an encoder and a decoder together to extract geometric features more accurately from point clouds. In the second step, trajectories from multiple environments have a multi-modal property which can be represented by a truncated multivariate Gaussian mixture model. We propose a multi-modal sampling net to learn optimal parameters of this model from the training trajectories, and to select corresponding mode based on the extracted features. Experiments demonstrate that the proposed algorithm is feasible and can achieve higher success rate than the state-of-the-art methods. Our method uses a single frame of point cloud to improve efficiency, therefore multiple point clouds from different perspective maybe needed when objects occlude with each other.

Sampling-based path planning algorithms usually implement uniform sampling methods to search the state space. However, uniform sampling may lead to unnecessary exploration in many scenarios, such as the environment with a few dead ends. Our previous work proposes to use the promising region to guide the sampling process to address the issue. However, the predicted promising regions are often disconnected, which means they cannot connect the start and goal states, resulting in a lack of probabilistic completeness. This work focuses on enhancing the connectivity of predicted promising regions. Our proposed method regresses the connectivity probability of the edges in the x and y directions. In addition, it calculates the weight of the promising edges in loss to guide the neural network to pay more attention to the connectivity of the promising regions. We conduct a series of simulation experiments, and the results show that the connectivity of promising regions improves significantly. Furthermore, we analyze the effect of connectivity on sampling-based path planning algorithms and conclude that connectivity plays an essential role in maintaining algorithm performance.Note to Practitioners—This work is derived from the promising region prediction for sampling-based path planning. The sampling-based path planning methods have been widely used in robotics due to their efficiency. To further improve the efficiency of these algorithms, sampling in the promising region predicted by a neural network is introduced into the sampling procedure. However, the connectivity of the promising region has yet to be considered, and it will affect the performance of the algorithms in several aspects. To demonstrate this problem, we compare the performance of the neural heuristic algorithms under different connectivity statuses in this paper. Furthermore, to enhance the connectivity of the predicted promising region, the novel prediction output and loss function are proposed. The simulation results show improvements in the algorithms after utilizing our method.

In this paper, we propose spatiotemporal co-attention hybrid neural network (SC-HNN), a novel hybrid neural network model with both spatial and temporal attention mechanisms for pose-invariant inertial odometry. The main idea is to extract both local and global features from a window of IMU measurements for velocity prediction. SC-HNN leverages the convolutional neural network (CNN) to capture the sectional features and long short-term memory (LSTM) recurrent neural network (RNN) to extract the long-range dependencies. Attention mechanisms are designed and embedded in both CNN and LSTM modules for better model representation. Specifically, in the CNN attention block, the convolved features are refined along both channel and element dimensions. For the LSTM module, softmax scoring is applied to update the weights of the hidden states along the temporal axis. We evaluate SC-HNN on the benchmark with the largest and most natural IMU data, RoNIN. Extensive ablation experiments demonstrate the effectiveness of our SC-HNN model. Compared with the state of the art, the 50th percentile accuracy of SC-HNN is 18.21% higher and the 90th percentile accuracy is 21.15% higher for all the phone holders not appeared in the training set. The real scenario inertial tracking trials in the CUHK campus further prove the superior generalization ability of the SC-HNN model. Note to Practitioners—This paper aims at improving the localization accuracy of deep inertial odometry. We focus on the problem of indoor localization only from the low-cost IMU embedded in the smartphone without any restriction on the phone’s daily use. IMU is a perfect solution for indoor localization because of its low power consumption, high privacy protection, and external infrastructure free. This paper suggests a novel hybrid convolutional and recurrent neural network with a set of carefully designed attention mechanisms to improve the representation ability of deep inertial odometry model. Specifically, the convolutional layer is applied to extract the local spatial features among the 6D IMU signals, following a cascaded channel attention module and element attention module to boost the representation ability of CNN. The complex long-term dependencies are then identified by the LSTM layers. To adaptively capture the temporal features of the multimodal inertial signals, an attention mechanism is applied to weigh the hidden states for the generation of the final features. The effectiveness of the SC-HNN design is validated by extensive ablation studies. To the best of our knowledge, our model is the first HNN fused attention mechanism for inertial tracking. Extensive experiments show that the proposed method outperforms the state of the art.

Current wireless capsule endoscopy (WCE) is limited in the long examination time and low flexibility since the capsule is passively moved by the natural peristalsis. Efforts have been made to facilitate active locomotion of WCE using magnetic actuation and localization technologies. In this work, we investigate the trajectory following problem of a robotic capsule under rotating magnetic actuation, in order to realize efficient and accurate navigation of the capsule in the narrow, complex intestinal environments. Specifically, four control strategies are developed based on the PD controller, adaptive controller (AC), model predictive controller (MPC) and robust multi-stage model predictive controller (RMMPC). In particular, the RMMPC method takes into account the uncertainty in the intestinal environment by modeling the intestinal peristalsis and friction in the controller design. We evaluate the proposed methods in simulation as well as in real-world experiments in several tubular environments, including plastic phantoms with complex shapes and an ex-vivo pig colon. The results have demonstrated the potential of the proposed control methods to realize accurate and efficient inspection of the intestine using active WCE. Our methods can be integrated with current WCE to improve the diagnostic accuracy and efficiency of the GI tract.Note to Practitioners—The motivation of this paper is to solve the trajectory following problem for active WCE in the human intestine to realize accurate, efficient and repeatable inspection of the gastrointestinal (GI) tract. We present four different control strategies for 5-DOF control of a robotic capsule endoscope actuated by a reciprocally rotating permanent magnet to make the capsule follow a predefined trajectory in a tubular environment. The accuracy and efficiency of the approach are validated in simulation and real-world experiments. The proposed trajectory following strategies can be integrated into existing WCE products to allow automatic and repeatable examination of the GI tract, and can also be extended to the locomotion of other tethered or untethered magnetic devices in the tubular environments for different medical and industrial applications. In the future, our proposed approach is expected to be combined with image-based automatic diagnosis of the GI tract to provide doctors with better tools for digestive examinations.

Registration is highly demanded in many real-world scenarios such as robotics and automation. Registration is challenging partly due to the fact that the acquired data is usually noisy and has many outliers. In addition, in many practical applications, one point set (PS) usually only covers a partial region of the other PS. Thus, most existing registration algorithms cannot guarantee theoretical convergence. This article presents a novel, robust, and accurate three-dimensional (3D) rigid point set registration (PSR) method, which is achieved by generalizing the state-of-the-art (SOTA) Bayesian coherent point drift (BCPD) theory to the scenario that high-dimensional point sets (PSs) are aligned and the anisotropic positional noise is considered. The high-dimensional point sets typically consist of the positional vectors and normal vectors. On one hand, with the normal vectors, the proposed method is more robust to noise and outliers, and the point correspondences can be found more accurately. On the other hand, incorporating the registration into the BCPD framework will guarantee the algorithm’s theoretical convergence. Our contributions in this article are three folds. First, the problem of rigidly aligning two general PSs with normal vectors is incorporated into a variational Bayesian inference framework, which is solved by generalizing the BCPD approach while the anisotropic positional noise is considered. Second, the updated parameters during the algorithm’s iterations are given in closed-form or with iterative solutions. Third, extensive experiments have been done to validate the proposed approach and its significant improvements over the BCPD. Note to Practitioners—This paper was motivated by the problem of 3D rigid PSR for computer-assisted surgery (CAS), especially in orthopedic applications. The proposed algorithm is also suitable for other scenarios where the initial coarse registration is conducted. The traditional registration methods are susceptible to noise (especially anisotropic noise), outliers, and incomplete partial data. This paper generalizes the recently proposed BCPD method to the six-dimensional scenario where anisotropic positional noise is considered and normal vectors are incorporated. The proposed noise model is decomposed into three parts to be solved alternately: the membership probability of mixture distributions, the soft correspondence estimation, and the model parameters (i.e., the rotation matrix, translation vector, the covariance matrix with the anisotropic positional error, and the concentration parameter with the estimation of the normal vectors). Especially, the convergence is guaranteed at the theoretical level using the variational inference theory. The experimental results demonstrate the superiority of our algorithm on registration accuracy, convergence speed, and robustness to noise, outliers, and partial data.

This article presents a closed-loop magnetic manipulation framework for robotic transesophageal echocardiography (TEE) acquisitions. Different from previous work on intracorporeal robotic ultrasound acquisitions that focus on continuum robot control, we first investigate the use of magnetic control methods for more direct, intuitive, and accurate manipulation of the distal tip of the probe. We modify a standard TEE probe by attaching a permanent magnet and an inertial measurement unit (IMU) sensor to the probe tip and replacing the flexible gastroscope with a soft tether containing only wires for transmitting ultrasound and IMU data and show that six-degree-of-freedom (DOF) localization and five-DOF closed-loop control of the probe can be achieved with an external permanent magnet based on the fusion of internal inertial measurement and external magnetic field sensing data. The proposed method does not require complex structures or motions of the actuator and the probe compared with existing magnetic manipulation methods. We have conducted extensive experiments to validate the effectiveness of the framework in terms of localization accuracy, update rate, workspace size, and tracking accuracy. In addition, our results obtained on a realistic cardiac tissue-mimicking phantom show that the proposed framework is applicable in real conditions and can generally meet the requirements for teleoperated TEE acquisitions.

Optical Coherence Tomography (OCT) is a rapidly growing and promising imaging technique, enabling non-invasive high-resolution visualization of biological tissues. Segmentation of tissue structures from OCT scans is essen-tial for disease diagnosis but remains challenging for the blurry boundaries and large volumes. Deep learning-based OCT segmentation algorithms always require large numbers of annotations for satisfying performance, which is hard to meet since manually labeling is time-consuming and labor-intensive. Therefore, we propose a novel semi-supervised OCT segmentation framework utilizing very few labeled scans, i.e., 5 samples, and abundant unlabeled data. Specifically, our framework con-sists of one shared encoder and two different decoder branches. For the two branches, we design a strong augmentation-consistent supervision module and a scaling transformation-consistent supervision module respectively to improve their generalization ability. Besides, cross consistency supervision with feature perturbations between two branches is proposed to incorporate their advantages for further regularization. With such multiple consistency supervision, we aim to enrich the diversity of unsupervised information so as to make full use of labeled and unlabeled data. Experimental results on a public retinal OCT dataset demonstrate the effectiveness of our method, achieving an average dice score of 87.25% in the case of only 5 labeled samples used. It outperforms the supervised baseline by 3.46% and the best semi-supervised model by 1.42% in our experiments.

In this work, we propose A2DIO, a novel hybrid neural network model with a set of carefully designed attention mechanisms for pose invariant inertial odometry. The key idea is to extract both local and global features from the window of IMU measurements for velocity prediction. A2DIO leverages the convolutional neural network (CNN) to capture the sectional features and long-short term memory (LSTM) recurrent neural network to extract long-range dependencies. In both CNN and LSTM modules, attention mechanisms are designed and embedded for better model representation. Specifically, in the CNN attention block, the convolved features are refined along both channel and spatial dimensions, respectively. For the LSTM module, softmax scoring is applied to update the weights of the hidden states along the temporal axis. We evaluate A2DIO on the benchmark with the largest and most natural IMU data, RoNIN. Extensive ablation experiments demonstrate the effectiveness of our A2DIO model. Compared with the state of the art, the 50th percentile accuracy of A2DIO is 18.21 % higher and the 90th percentile accuracy is 21.15 % higher for all the phone holders not appeared in the training set.

Autonomous mobile manipulation robots that can collect trolleys are widely used to liberate human resources and fight epidemics. Most prior robotic trolley collection solutions only detect trolleys with 2D poses or are merely based on spe-cific marks and lack the formal design of planning algorithms. In this paper, we present a novel mobile manipulation system with applications in luggage trolley collection. The proposed system integrates a compact hardware design and a progressive perception and planning framework, enabling the system to efficiently and robustly collect trolleys in dynamic and complex environments. For perception, we first develop a 3D trolley detection method that combines object detection and keypoint estimation. Then, a docking process in a short distance is achieved with an accurate point cloud plane detection method and a novel manipulator design. On the planning side, we formulate the robot's motion planning under a nonlinear model predictive control framework with control barrier functions to improve obstacle avoidance capabilities while maintaining the target in the sensors' field of view at close distances. We demonstrate our design and framework by deploying the system on actual trolley collection tasks, and their effectiveness and robustness are experimentally validated. (Video11Video demonstration: https://youtu.be/6SwjgGvRtno.)

This paper presents a novel, robust, and accurate three-dimensional (3D) rigid point set registration (PSR) method, which is achieved by generalizing the state-of-the-art (SOTA) Bayesian coherent point drift (BCPD) theory to the scenario that high-dimensional point sets(PSs) are aligned and that the anisotropic positional noise is considered. Our contributions in this paper are three folds. First, the problem of rigidly aligning two general point sets (PSs) with normal vectors is incorporated into a variational Bayesian inference framework, which is solved by generalizing the BCPD approach while the anisotropic positional noise is considered. Second, the updated parameters during the algorithm's iterations are given in closed-form or iterative solutions. Third, extensive experiments have been done to validate the proposed approach and its significant improvements over the BCPD.

Online state-time trajectory planning in highly dynamic environments remains an unsolved problem due to the curse of dimensionality of the state-time space. Existing state-time planners are typically implemented based on randomized sampling approaches or path searching on discrete graphs. The smoothness, path clearance, or planning efficiency is sometimes not satisfying. In this work, we propose a gradient-based planner on the state-time space for online trajectory generation in highly dynamic environments. To enable the gradient-based optimization, we propose a Timed-ESDT that supports distance and gradient queries with state-time keys. Based on the Timed-ESDT, we also define a smooth prior and an obstacle likelihood function that are compatible with the state-time space. The trajectory planning is then formulated to a MAP problem and solved by an efficient numerical optimizer. Moreover, to improve the optimality of the planner, we also define a state-time graph and conduct path searching on it to find a better initialization for the optimizer. By integrating the graph searching, the planning quality is significantly improved. Experiments on simulated and benchmark datasets demonstrate the superior performance of our proposes method over conventional ones.

This paper proposes a new network for reconstructing multi-object point cloud. Different from previous networks which reconstruct multi-object point cloud as a whole, our network iteratively reconstructs each individual object point cloud from a frame of multi-object point cloud. To achieve this goal, we have designed MO-Transformer, a transformer-based autoregressive network. During training, MO-Transformer takes a frame of multi-object point cloud and individual object point clouds as input. During testing, MO-Transformer iteratively reconstructs individual object point clouds only based on the input multi-object point cloud. To train the proposed MO-Transformer, we design a new loss function called separate Chamfer distance (SCD). In addition, we prove that SCD is an upper bound of the traditional Chamfer distance calculated based on the entire multi-object point cloud. The reconstruction experiment verifies the efficacy of our network in multi-object point cloud reconstruction. Furthermore, the reconstruction experiment also investigates the effect of different dimensions using a series of datasets. The ablation study experiment verifies the necessity of SCD in training MO-Transformer.

In environmental monitoring programs, mobile robots have been widely deployed for remote sensing, with the end objective of monitoring and mapping out environmental fields. Complex characteristics and correlations in natural phenomena make it challenging to establish a reliable framework for mobile sensing and field mapping. Furthermore, constraints of onboard resources will limit the ability of mobile robots to cover a large area. This article focuses on the active sensing problem in environmental field mapping and particularly exploits the use of intrinsic interactions among multivariate spatiotemporal data. A novel deep neural network of a hybrid CNN-RNN model is employed to learn the monitored multivariate spatiotemporal field. Specifically, a set of attention mechanisms is designed and embedded in the network, which is able to adaptively capture parameterwise dependencies among the monitored heterogeneous parameters and spatial correlations in geolocations of a surveyed field. The weights of inferred attention facilitate explicit interpretation of the driving parameters and geolocations. Some subregions of interest in the surveyed field are specified by their spatial attention distribution and are actively sensed by following the proposed coverage path planner. Experiments are carried out using a real-world dataset with multisource environmental imagery from a remote sensing program. Experimental results are obtained, which demonstrate the superior mapping performance of the proposed systematical methodology compared to baseline methods. Furthermore, the proposed model is able to quantitatively reveal the driving monitored parameters and geolocations in a regression process.Note to Practitioners—This article was motivated by the need for a practical and systematic approach for reconstruction and planning to execute robotic active sensing (AS) in environmental field mapping. Field robotic applications are not maneuverable in comparison with indoor scenarios due to severe conflict between the need for long execution endurance in the field and the very limited onboard resources. Traditional AS planners normally use statistical model-based informative metrics, which may lead to model misspecification in real-world phenomena. The developed framework in this article yields a novel attention-driven metric to guide AS and mapping. It relies on an attention-based hybrid neural network that reveals the driving variables in terms of the heterogeneities and complexities in a natural environment. The high-priority regions are maximized in a coverage path depending on the inferred spatial attention distribution while maintaining the travel cost of the sensing robots within an available energy budget. Experiments using a remote sensing dataset validate the reliable performance of the proposed framework, in environmental field mapping.

Collaborative robots are widely utilized in intelligent manufacturing to cooperate with the human to accomplish different assembly tasks. To improve the efficiency of human–robot cooperation, robots should be able to recognize human intentions and provide necessary assistance proactively. The major challenge for current human intention recognition methods is that they only deal with known human intentions of predefined tasks and lack of ability to learn unknown intentions corresponding to new tasks. This article introduces an evolving hidden Markov model (EHMM)-based approach to learn new human intentions incrementally by carrying out structure and parameter updating based on the observed sequence, in parallel with the recognition. The incremental learning ability makes it applicable in dynamic environments with changing tasks. A set of assistive execution policies has been developed for the robot to provide appropriate assistance to the human partner based on the intention recognition results in real time. Experiments have been carried out to verify the effectiveness of our approach in human–robot cooperative assembly tasks. The results show very high recognition accuracy (≥95.45%), and the human subjects show their high satisfaction with the intention learning ability of the proposed approach.Note to Practitioners—This article aims to effectively improve the productivity of human–robot cooperation by exploiting human adaptability and robot repeatability. Smooth cooperation requires the peer robot to provide proactive assistance to humans by inferring human intention after training. Moreover, the robot should also be able to learn untrained intentions online by human demonstrations. This is made possible by our proposed evolving hidden Markov model (EHMM) that unifies intention inference and incremental learning. Simplified cooperative assembly tasks have been designed to verify the proposed unified intention inference and learning model. A robotic assembly platform has been introduced to integrate the proposed EHMM with a perception module and a collaborative manipulation module. We have demonstrated, through experiments and surveys, that the proposed approach can promote efficacy and acceptance of human–robot cooperative assembly.

In medical image analysis (MIA) and computer-assisted surgery (CAS), aligning two multiple point sets (PSs) together is an essential but also a challenging problem. For example, rigidly aligning multiple point sets into one common coordinate frame is a prerequisite for statistical shape modelling (SSM). Accurately aligning the pre-operative space with the intra-operative space in CAS is very crucial to successful interventions. In this article, we formally formulate the multiple generalized point set registration problem (MGPSR) in a probabilistic manner, where both the positional and the normal vectors are used. The six-dimensional vectors consisting of both positional and normal vectors are called as generalized points. In the formulated model, all the generalized PSs to be registered are considered to be the realizations of underlying unknown hybrid mixture models (HMMs). By assuming the independence of the positional and orientational vectors (i.e., the normal vectors), the probability density function (PDF) of an observed generalized point is computed as the product of Gaussian and Fisher distributions. Furthermore, to consider the anisotropic noise in surgical navigation, the positional error is assumed to obey a multi-variate Gaussian distribution. Finally, registering PSs is formulated as a maximum likelihood (ML) problem, and solved under the expectation maximization (EM) technique. By using more enriched information (i.e., the normal vectors), our algorithm is more robust to outliers. By treating all PSs equally, our algorithm does not bias towards any PS. To validate the proposed approach, extensive experiments have been conducted on surface points extracted from CT images of (i) a human femur bone model; (ii) a human pelvis bone model. Results demonstrate our algorithm’s high accuracy, robustness to noise and outliers. Note to Practitioners—This paper was motivated by solving the problem of registering two or more PSs. Most existing registration approaches use only the positional information associated with each point, and thus lacks robustness to noise and outliers. Three significant improvements are brought by our proposed approach. First, the normal vectors that can be extracted from the point sets are utilized in the registration. Second, the positional error distribution is assumed to be anisotropic and inhomogeneous. Third, all the PSs to be registered are treated equally that means no PS is considered as the model one. The registration problem is cast into a maximum likelihood (ML) problem and solved under the expectation maximization (EM) framework. We have demonstrated through extensive experiments that the proposed registration approach achieves significantly improved accuracy, robustness to noise and outliers. The algorithm is particularly suitable for biomedical applications involving the registration procedures, such as image-guided surgery.

In this article, based on the rapidly-exploring random tree (RRT), we propose a novel and efficient motion planning algorithm using bidirectional RRT search. First, a RRT extend function is used to organize the sampled states under kinodynamic constraints. Meanwhile, the bidirectional search strategy is implemented to grow a forward tree and backward tree simultaneously in the tree extension process. When these two trees meet each other, the backward tree will act as a heuristic to guide the forward tree to continuously grow toward the goal state, where the algorithm switches to unidirectional search mode. Therefore, the two-point boundary value problem (BVP) in the connection process is avoided, and the extension process gets much accelerated. We also prove that probabilistic completeness is guaranteed. Numerical simulations are conducted to demonstrate that the proposed algorithm performs much better than the state-of-the-art algorithms in different environments.Note to Practitioners—The motivation of this work is to develop an efficient sampling-based motion planning algorithm for mobile robots. Conventional sampling-based algorithms are time-consuming to find a feasible solution under differential constraints. When applying bidirectional search strategy to improve them, the complex 2-point BVP is required to solve. In this article, the backward free is regarded as a heuristic to guide the tree growth. On the one hand, the advantage of bidirectional search is retained. On the other hand, the 2-point BVP is avoided. Therefore, the bidirectional-unidirectional technique can achieve efficient robot motion planning. The proposed algorithm can be extended to other specified sampling-based algorithms to further improve their performance. Besides, it can be also applied to autonomous driving, service robot and medical robots to achieve efficient motion planning.

Robot path planning in 3D space is a challenging problem for its complex configuration. Sampling-based algorithms have gained great success in solving path planning problems in 3D space, but the quality of the initial path is not guaranteed and the convergence to the optimal solution is slow. To address these problems, in this article, we present a novel sampling-based path planning framework enhanced by the deep neural network (DNN) with applications to 3D space. In the proposed framework, we first train the DNN with a number of successful path planning cases in 3D space. Then the DNN is utilized to predict the promising region where the feasible path probably exists for a given path planning problem. This predicted promising region serves as a nonuniform sampling heuristic to bias the sampling process of the path planner. In this way, the path planner can focus on the promising region in the exploration and exploitation process so that the path planning speed gets accelerated. We conduct numerical simulations to evaluate the performance of the proposed algorithm and the results show that it can perform much better than conventional path planning algorithms. Furthermore, we also investigate the performance of different DNN architectures for path planning in 3D space. Note to Practitioners—In this work, we aim to provide an efficient learning-based method to accelerate the robot path planning process in 3D space. Conventional path planning algorithms need to perceive the environment first, and then implement a series of calculations such as collision checking and data storing to generate a feasible path. When facing complex and high-dimensional environments, they do not perform well. But the proposed neural network method in this article can predict the promising region where the feasible path exists for any given environment. This prediction result is used to guide the path planning process so that the algorithm performance can get significantly improved. Apart from sampling-based algorithms, the proposed neural network model can also be extended to other types of path planning algorithms.

Point set registration (PSR) is an essential problem in surgical navigation and computer-assisted surgery (CAS). In CAS, PSR can be used to map the intraoperative surgical space with the preoperative volumetric image space. The performances of PSR in real-world surgical scenarios are sensitive to noise and outliers. This article proposes a novel point set registration approach where the additional features (i.e., the normal vectors) extracted from the point sets are utilized and the convergence of the algorithm is guaranteed from the theoretical perspective. More specifically, we formulate the PSR with normal vectors by generalizing the Bayesian coherent point drift (BCPD) into the 6-D scenario. The proposed algorithm is more accurate and robust to noise and outliers, and the theoretical convergence of the proposed approach is guaranteed. Our contributions of this article are summarized as follows. 1) The PSR problem with normal vectors is formally formulated through generalizing the BCPD approach. 2) The formulas for updating the parameters during the algorithm’s iterations are given in closed forms. 3) Extensive experiments have been done to verify the proposed approach and specifically its significant improvements over the BCPD has been validated.

Simultaneous magnetic actuation and localization is a promising technology to realize active wireless capsule endoscopy. In this article, an adaptive approach is proposed to efficiently propel and precisely locate a magnetically actuated capsule in unknown complex tubular environments. In order to track the capsule in real time in a large workspace, we improve upon our previous external sensor-based magnetic localization strategies by proposing an adaptive method to automatically activate an optimal subset of sensors during the capsule movement and developing a simplified multiple objects tracking algorithm to estimate the six-dimensional pose of the capsule in real time. Moreover, based on a study of capsule locomotion in tubular environments with different shapes, we propose to dynamically adjust the pose of the actuator to achieve efficient and robust propulsion of the capsule in an unknown tubular environment. The effectiveness of our proposed system is validated in extensive experiments on phantoms andex vivoanimal organs. Compared with the state-of-the-art, our method can realize the closed-loop control of a magnetic capsule without requiring sensing modules to be placed inside the capsule, which can reduce the space constraints and energy consumption of the capsule. Moreover, the system and algorithms for adaptive actuation and localization can provide an improved propulsion speed, workspace range, and localization update frequency as well as a comparable localization accuracy.

In this article, a two-level planner is proposed to provide a solution to the autonomous luggage trolley collection problem at the airport. In the higher level planner, a decision-making problem is tackled where a sequence of luggage trolleys is determined with which the robot can collect them one by one. Based on the traditional traveling salesman problem (TSP), this decision-making problem is formulated as an open dynamic traveling salesman problem with fixed start (ODTSP-FS). Incorporating the modified transition rule, elitist global update rule, and additional local update rule, an efficient algorithm is proposed to handle this decision-making problem. The experimental results demonstrate that the proposed algorithm achieves fast convergence and smaller cost compared with the state-of-the-art algorithms. In the lower level planner, based on the pipeline of rapid-exploring random tree (RRT) scheme, a novel real-time path planning algorithm is introduced, which can adjust itself to moving obstacles and moving targets by retaining the whole tree and using two rewiring strategies. Finally, the proposed two-level planner is evaluated in a simulation environment similar to the airport to validate the effectiveness and efficiency of the proposed algorithm.

As a common cause of anemia and gastrointestinal bleeding, angiodysplasia (AD) diagnosis in wireless capsule endoscopy (WCE) images is important in clinical. Current manual review requires undivided concentration of the gastroenterologists, which is laborious and time-consuming. The development of computational methods that can assist automated diagnosis of angiodysplasia is highly desirable. In this paper, we present a new approach, ADNet, for angiodysplasia segmentation using convolutional neural networks (CNNs). Compared with previous learning strategies, ADNet gains accuracy from attentionguided and domain-adversarial training via a multibranch CNN architecture. Specifically, the core branch is constructed for AD segmentation in a fully convolutional manner. Then we propose an attention module embedded in the attention branch to enhance network feature learning, which allows ADNet to focus on the most informative and AD relevant regions while processing. Furthermore, an adaptation branch is built to learn domain-invariant features by adversarial training, aiming to improve the performance when datasets are expanded while preventing the degradation induced by the variations in WCE image acquisition. ADNet is evaluated using two WCE datasets with angiodysplasia and the results show the accuracy gains we obtain, where the state-of-the-art segmentation performance on the public dataset of GIANA’17 is achieved.

Brain needle intervention is a specific diagnosis and therapy procedure in brain disorders, such as brain tumors and Parkinson’s disease. Preoperative needle path planning is a vital step to guarantee the patient’s safety and reduce lesions. For positioning accuracy in the CT/MRI environment, we have developed a novel needle intervention robot in our previous work. Because the robot is currently designed for the rigid needle, the task of preoperative path-planning is to search for an optimal Remote Center of Motion (RCM) for needle insertion. Therefore, this work proposes an RCM recommendation system using deep reinforcement learning. Considering the robot kinematics, this system takes the following criteria/constraints into consideration: clinical obstacle (blood vessels, tissues) avoidance (COA), mechanically inverse kinematics (MIK) and mechanically less motion (MLM) for the robot. We design a reward function to combine the above three criteria based on their corresponding importance level and utilize proximal policy optimization (PPO) as the main agent of reinforcement learning (RL). RL methods are proved to be competent in searching the RCM, which satisfies the above criteria simultaneously. On the one hand, the results present that RL agents obtain the success rate of finishing the designed task at 93%, which has reached the human level in the tests. On the other hand, the RL agents have the remarkable capability of combining more complex criteria/constraints in future work.

Robot path planning is difficult to solve due to the contradiction between the optimality of results and the complexity of algorithms, even in 2D environments. To find an optimal path, the algorithm needs to search all the state space, which costs many computation resources. To address this issue, we present a novel recurrent generative model (RGM), which generates efficient heuristic to reduce the search efforts of path planning algorithms. This RGM model adopts the framework of general generative adversarial networks (GAN), which consists of a novel generator that can generate heuristic by refining the outputs recurrently and two discriminators that check the connectivity and safety properties of heuristic. We test the proposed RGM module in various 2D environments to demonstrate its effectiveness and efficiency. The results show that, compared with a model without recurrence, the RGM successfully generates appropriate heuristic in both seen and new unseen maps with higher accuracy, demonstrating the good generalization ability of the RGM model. We also compare the rapidly-exploring random tree star (RRT*) with generated heuristic and the conventional RRT* in four different maps, showing that the generated heuristic can guide the algorithm to efficiently find both initial and optimal solutions in a faster and more efficient way.

Autonomous ultrasound (US) acquisition is an important yet challenging task, as it involves interpretation of the highly complex and variable images and their spatial relationships. In this work, we propose a deep reinforcement learning framework to autonomously control the 6-D pose of a virtual US probe based on real-time image feedback to navigate towards the standard scan planes under the restrictions in real-world US scans. Furthermore, we propose a confidence-based approach to encode the optimization of image quality in the learning process. We validate our method in a simulation environment built with real-world data collected in the US imaging of the spine. Experimental results demonstrate that our method can perform reproducible US probe navigation towards the standard scan plane with an accuracy of 4.91mm/4.65° in the intra-patient setting, and accomplish the task in the intra- and inter-patient settings with a success rate of 92% and 46%, respectively. The results also show that the introduction of image quality optimization in our method can effectively improve the navigation performance.

Interactions with either environments or expert policies during training are needed for most of the current imitation learning (IL) algorithms. For IL problems with no interactions, a typical approach is Behavior Cloning (BC). However, BC-like methods tend to be affected by distribution shift. To mitigate this problem, we come up with a Robust Model-Based Imitation Learning (RMBIL) framework that casts imitation learning as an end-to-end differentiable nonlinear closed-loop tracking problem. RMBIL applies Neural ODE to learn a precise multi-step dynamics and a robust tracking controller via Nonlinear Dynamics Inversion (NDI) algorithm. Then, the learned NDI controller will be combined with a trajectory generator, a conditional VAE, to imitate an expert’s behavior. Theoretical derivation shows that the controller network can approximate an NDI when minimizing the training loss of Neural ODE. Experiments on Mujoco tasks also demonstrate that RMBIL is competitive to the state-of-the-art generative adversarial method (GAIL) and achieves at least 30% performance gain over BC in uneven surfaces.

This paper presents a search-based partial motion planner for generating feasible trajectories of car-like robots in highly dynamic environments. The planner searches for smooth, safe, and near-time-optimal trajectories by exploring a state graph built on motion primitives. To enable fast online planning, we propose an efficient path searching algorithm based on the aggregation and pruning of motion primitives. We then propose a fast collision checking algorithm that takes into account the motions of moving obstacles. The algorithm linearizes relative motions between the robot and obstacles, and then checks collisions by calculating a point-line distance. Benefiting from the fast searching and collision checking algorithms, the planner can effectively explore the state-time space to generate near-time-optimal solutions. Experiments show that the proposed method can generate feasible trajectories within milliseconds while maintaining a higher success rate than up-to-date methods, which significantly demonstrates its advantages.

Human activities are hugely restricted by COVID-19, recently. Robots that can conduct inter-floor navigation attract much public attention since they can substitute human workers to conduct the service work. However, current robots either depend on human assistance or elevator retrofitting, and fully autonomous inter-floor navigation is still not available. As the very first step of inter-floor navigation, elevator button segmentation and recognition hold an important position. Therefore, we release the first large-scale publicly available elevator panel dataset in this work, containing 3,718 panel images with 35,100 button labels, to facilitate more powerful algorithms on autonomous elevator operation. Together with the dataset, a number of deep learning based implementations for button segmentation and recognition are also released to benchmark future methods in the community. The dataset is available at https://github.com/zhudelong/elevator_button_recognition

Point set registration (PSR) is an essential problem in communities of computer vision, medical robotics and biomedical engineering. This paper is motivated by considering the anisotropic characteristics of the error values in estimating both the positional and orientational vectors from the PSs to be registered. To do this, the multi-variate Gaussian and Kent distributions are utilized to model the positional and orientational uncertainties, respectively. Our contributions of this paper are three-folds: (i) the PSR problem using normal vectors is formulated as a maximum likelihood estimation (MLE) problem, where the anisotropic characteristics in both positional and normal vectors are considered; (ii) the matrix forms of the objective function and its associated gradients with respect to the desired parameters are provided, which can facilitate the computational process; (iii) two approaches of computing the normalizing constant in the Kent distribution are compared. We verify our proposed registration method on various PSs (representing pelvis and femur bones) in computer- assisted orthopedic surgery (CAOS). Extensive experimental results demonstrate that our method outperforms the state- of-the-art methods in terms of the registration accuracy and the robustness.

Accurate position feedback in a wide range is critical for medical microrobotics and robot-assisted examinations, such as colonoscopy, bronchoscopy and capsule endoscopy examination. Among the many modalities of positioning feedback, magnetic tracking is a preferable method due to the unique advantages of free line of sight, free energy storage and untethered connection. However, the field strength of the magnetic source decreases with the third power of the distance, limiting the effectiveness of position feedback at long distances. In order to maintain a consistently high tracking accuracy in a broad area, this paper presents a new dynamic tracking solution by applying a movable sensor array. In this new solution, the tracking accuracy of the magnet is first determined and optimized within a short range. When the target microrobot carrying the magnet exceeds this optimized range, the sensor array is relocated by an external robotic arm to keep the target in the effective tracking range. Moreover, we also propose a multi-point locating algorithm to minimize the varying background noise. Experimental results show that the proposed method increases the range of magnetic tracking and achieves a satisfactory level of tracking accuracy, which demonstrates significant potentials to improve the position feedback of microrobots in medical applications.

This paper presents a modular and reconfigurable mini-robotic system with 5 degrees of freedom (DoFs) towards minimally invasive surgery (MIS). The mini-robotic system consists of two modules, a 2-DoFs rotational end-effector, and a 3-DoFs positioning platform. The 2-DoFs rotational end-effector is based on a spring-spherical joint mechanism, whose rotation is controlled by Bowden-cable. The 3-DoFs positioning platform is based on the linear Delta parallel mechanism. Magnetic spherical joints are adopted to replace the traditional spherical joint. The magnetic joint connections enable fast assembling and disassemble of the end platform and kinematic chains. Different surgical instruments can be installed without changing the driver and control system. A flexible shaft actuates the 3-DoFs positioning platform to arrange the motors away from the manipulator side. Based on these structure characteristics, the 3-DoFs positioning platform’s size is dramatically reduced. The outer diameter of the current prototype is 32.5 mm. The single-axis positioning accuracy of the 3-DoFs positioning platform is within -1 mm to 0.85 mm. Three axes tracking experiments are also carried out, with the positioning errors of ± 1.2 mm for cylindrical curves and -1.5 mm to 2 mm for spherical helix curves. Static and dynamic load capabilities are also tested. Finally, the feasibility of the proposed system is demonstrated.

Active wireless capsule endoscopy (WCE) under magnetic actuation is a promising technology to reduce the inspection time and relieve the burden of physicians. In this paper, we propose a reciprocally rotating magnetic actuation method for trajectory following of a capsule and develop its dynamic model. For the trajectory following task, we investigate the closed-loop tracking control strategies based on different controllers to actuate the capsule in the complex environments. The effectiveness of our method is validated in extensive experiments in a simulation environment as well as in an ex-vivo pig colon. The results demonstrate that the proposed method can accurately and efficiently actuate the capsule to follow the desired trajectory in the complex environments, achieving tracking errors on the order of millimeter. Moreover, the experiments on the ex-vivo pig colon show that the proposed reciprocally rotating magnetic actuation method has the potential to reduce the clinical risks and improve the safety and clinical acceptability of this technology.

Grasp pose detection generates the position and orientation of the robot end-effector to grasp objects from the RGB or RGB-D image. In this paper, we propose a novel grasp pose detection network that generates 3-DOF grasp poses using the RGB image. The network follows the anchor-based object detection pipeline and incorporates the angle detection unit. Furthermore, we redesign the grasp angle predictor with a classification unit to increase the accuracy of grasp pose rotation estimation. Our method classifies the prediction angle densely in contrast with the previous regression method or sparse classification method. Moreover, an angle smooth label is designed to avoid the sudden change of the angle regression loss caused by the periodic property of the angle. We validate our algorithm on Cornell Grasp Dataset and obtain a higher detection accuracy than the state-of-the-art method. The real scenario experiment also proves the effectiveness of our method. The robot equipped with the parallel gripper achieves a 96.4% grasp success rate.

Point set registration (PSR) is an essential problem in surgical navigation and image-guided surgery (IGS). It can help align the pre-operative volumetric images with the intra-operative surgical space. The performances of PSR are susceptible to noise and outliers, which are the cases in real-world surgical scenarios. In this paper, we provide a novel point set registration method that utilizes the features extracted from the PSs and can guarantee the convergence of the algorithm simultaneously. More specifically, we formulate the PSR with normal vectors by generalizing the bayesian coherent point drift (BCPD) into the six-dimension scenario. Our contributions can be summarized as follows. (1) The PSR problem with normal vectors is formulated by generalizing the Bayesian coherent point drift (BCPD) approach; (2) The updated parameters during the algorithm's iterations are given in closed-forms; (3) Extensive experiments have been done to verify the proposed approach and its significant improvements over the BCPD has been validated. We have validated our proposed registration approach on both the human femur model. Results demonstrate that our proposed method outperforms the state-of-the-art registration methods and the convergence is guaranteed at the same time.

Nonrigid point set (PS) registration is an outstanding and fundamental problem in the fields of robotics, computer vision, medical image analysis, and image-guided surgery (IGS). The aim of a nonrigid registration problem is to align together two point sets where one has been deformed. The assumption of isotropic localization error is shared in the previous nonrigid registration algorithms. In this article, we have derived and presented a novel nonrigid registration algorithm, where the position localization error (PLE) is generalized to be anisotropic, which means that the error distribution is not the same in different spatial directions. The motivation of considering the anisotropic characteristic is that the PLE is actually different in three spatial directions in real applications of registrations, such as IGS. Mathematically, the difficulty in dealing with the anisotropic error case comes from the change from a standard deviation that is a scalar to a covariance matrix. The formulas for updating the parameters in both expectation and maximization steps are derived. More specifically, in the expectation step, we compute the posterior probabilities that represent the correspondences between points in two PSs. In the maximization step, given the current posteriors, the covariance matrix of the PLE and the nonrigid transformation are updated. To further speed up the proposed algorithm, the low-rank approximation variation of our method is also presented. We have demonstrated through experiments on both general and medical data sets (corrupted with noise) that the proposed algorithm outperforms the state-of-the-art ones in terms of registration accuracy and robustness to noise. More specifically, all the experimental results have passed the statistical tests at the 5% significance level.Note to Practitioners—This article was motivated by solving the problem of nonrigidly registering two point sets where one has been deformed and corrupted with anisotropic noise. Most existing registration methods generally assume the positional error to be the same in all directions, which in fact is not the case in real scenarios. This article presents a new robust method that assumes the positional error to be anisotropic, which is the case in point sets coming from the stereo reconstruction. The nonrigid registration problem is formulated as a maximum-likelihood (ML) problem and solved with the expectation–maximization (EM) technique. We have demonstrated through extensive experiments on both general and medical data sets that the proposed registration algorithm achieves significantly improved accuracy, robustness to noise, and outliers compared with state-of-the-art algorithms. The algorithm is particularly suitable for biomedical applications involving the registration, such as medical imaging and image-guided surgery (IGS).

Registering two point sets (PSs) is an essential problem in medical robotics and computer-assisted surgery (CAS). As one typical example, in computer-assisted orthopedic surgery (CAOS), the preoperative scan has to be aligned with the intraoperative scan accurately. In this article, we first formally formulate the generalized PS registration problem in a probabilistic manner. Especially, not only positional but also orientational information is incorporated into the registration. Notably, the positional error is assumed to obey a multivariate Gaussian distribution to accommodate the anisotropic noise. The expectation–maximization (EM) framework is utilized to solve the maximum likelihood (ML) problem. In the E-step, the correspondence probabilities between points in two generalized PSs are computed. In the M-step, the constrained optimization problem with respect to the rigid transformation matrix is reformulated as an unconstrained one. This is achieved by utilizing the Rodrigues parameterization to represent the rotation matrix. Both extensive simulated and real experiments are conducted to validate the proposed algorithm by comparing it with state-of-the-art registration methods.Note to Practitioners—This article was motivated by considering the anisotropic positional uncertainty into the rigid point set (PS) registration in the application of preoperative-to-intraoperative registration within image-guided surgery. We provide iterative solutions that compute the rotation and translation vector that aligns two 3-D PSs. The correspondences between points in two PSs are not known and regarded as hidden variables in the optimization process. Expectation–maximization technique is utilized to solve the maximum likelihood problem. We have demonstrated through experiments that our proposed approach can achieve lower registration error values than the compared state-of-the-art registration methods on various data sets. The readers should note that the proposed method is particularly suitable for cases that anisotropic noise is involved.

Registration is an essential problem in image-guided surgery (IGS) since it brings different involved coordinate frames together. Nonrigid or deformable registration still faces many challenges, such as two point sets (PSs) are partially overlapped. To tackle the challenges in the nonrigid registration, we introduce a new two-step point-based registration pipeline that includes two steps. In the first step, the rigid transformation between the two spaces is recovered where the orientation vectors are adopted. In the second step, built on the nonrigid coherent point drift (CPD) approach, the anisotropic positional noise is also assumed. Registration results on the human liver verify the proposed approach' great improvements over the other methods. First, the rotation and translation are recovered with smaller error values than the existing methods. Second, our registration method's performance is much more robust to the partial overlapping between two PSs. Third, the two-step registration framework achieves the best performances in most test cases when there is a localization error in acquiring the intraoperative data. Note to Practitioners-A novel registration approach is presented for image-guided liver surgery (LGLS). Compared with existing nonrigid registration methods, two significant changes (or improvements) exist in the proposed registration framework: 1) the normal vectors are extracted and utilized in the rigid registration step and 2) the anisotropic positional uncertainties are considered. In both steps, the registration problems are formulated as a maximum likelihood (ML) problems and dealt with the expectation-maximization (EM) technique. In both steps, the matrix form of the updated positional covariance is provided and can speed up the computational process. The readers are reminded that with extra information and a more general positional error assumption, our approach demonstrates improved performances in the case of partial-to-full alignment.

Simultaneous magnetic actuation and localization (SMAL) is a promising technology for accelerating and positioning the capsule in the human intestine for wireless capsule endoscopy (WCE). In this article, we propose a novel system that uses a rotating magnetic actuator and an external sensor array to achieve closed-loop SMAL for a capsule with two embedded magnetic rings. First, the state of the capsule is detected as “Stuck,” “Synchronous,” or “Missing” by studying the relationship between the theoretical actuating magnetic field and the measured total magnetic field. Then, the undesired interference with the localization system caused by the actuator is eliminated with an integral filter-based approach. Different models are proposed to solve the pose of the capsule according to the different states of the capsule, and the localization result is used to update the pose of the actuator to close the loop. Extensive experiments on phantoms and animal organs with different environmental conditions are carried out to validate the proposed framework. The state detection accuracy achieves 96.7%, and the capsule can be located with an accuracy of 5.5 mm and 5.2° in position and orientation, respectively. Experimental results show the feasibility of our proposed system and demonstrate the robustness, accuracy, actuation efficiency, and closed-loop performance of the system.Note to Practitioners—The motivation of this article is to solve the problem of controlling the movement of a wireless capsule endoscope in the human intestine to assist intestinal diagnosis and treatment. We present a feasible system design and corresponding algorithms to achieve closed-loop simultaneous actuation and localization of a robotic capsule. The design of a simple structure placed inside the capsule (i.e., two magnetic rings) and an external sensor array mounted on the examination bed can reduce the size and power consumption of the capsule, and the use of a rotating actuator helps improve the actuation efficiency. The proposed SMAL framework, which is composed of state detection, interference removal, multimodel localization, and actuator updating, can close the actuation-localization loop and improve the accuracy and robustness of the system. We demonstrate the superiority of the proposed framework compared with others through extensive experiments. In the future, our SMAL system can be combined with image- or ultrasound-based automatic diagnosis and is expected to provide doctors with better tools for digestive examinations.

Machines translation (MT) is an essential task in natural language processing or even in artificial intelligence. Statistical machine translation has been the dominant approach to MT for decades, but recently neural machine translation achieves increasing interest because of its appealing model architecture and impressive translation performance. In neural machine translation, an attention model is used to identify the aligned source words for the next target word, i.e., target foresight word, to select translation context. However, it does not make use of any information about this target foresight word at all. Previous work proposed an approach to improve the attention model by explicitly accessing this target foresight word and demonstrating substantial alignment tasks. However, this approach cannot be applied in machine translation tasks where the target foresight word is unavailable. This paper proposes several novel enhanced attention models by introducing hidden information (such as part-of-speech) of the target foresight word for the translation task. We incorporate the novel enhanced attention employing hidden information about the target foresight word into both recurrent and self-attention-based neural translation models and theoretically justify that such hidden information can make translation prediction easier. Empirical experiments on four datasets further verify that the proposed attention models deliver significant improvements in translation quality.

In this article, a novel informative path planning (IPP) framework is proposed for efficient robotic object search. We innovatively reformulate the object search into an IPP problem, which takes account of the knowledge of possible target object locations. To model the target object distribution knowledge, the semantic information of the focused environment is utilized to obtain the probabilities of finding the target object at possible locations. Then, the probability distribution is modeled by Gaussian mixture model (GMM) to generate an information map. Based on the map, a sampling-based IPP method is proposed to minimize the object search cost. It is worth noting that the object search path is planned with a tree structure and evaluated by a utility function that concerns both search information gain and path cost. Moreover, to improve the quality of the search path, a novel informative sampling strategy and a rewire mechanism are conceived. The performance of the proposed object search framework is fully evaluated through both simulation experiments and real-world tests with a mobile robot platform. Results demonstrated that our method can find the target object efficiently and robustly with shorter path length than three comparative methods in the literature and the mobile robot shows human-like behavior when searching for the target object.

Learning-based grasp pose detection algorithms have boosted the performance of robot grasping, but they usually need manually fine-tuning steps to find the balance between detection accuracy and efficient. In this paper, we discard these intermediate procedures, like sampling grasps and generating grasp proposals, and propose an end-to-end grasp pose detection model. Our model uses the RGB image as the input and predicts the single grasp pose in each small grid of the image. Furthermore, the best grasps are found by non-maximum suppression (NMS) strategy. The clustering and ranking procedures are left for NMS while the network only generates dense grasp predictions, which keeps the network simple and efficient. To achieve dense predictions, the predicted grasps of our detection model are represented by the 6 channels images with each pixel location representing a rated grasp. To the best of our knowledge, our model is the first neural network that attaches a grasp pose in pixel level. The model achieves 96.5% accuracy which costs 14ms for prediction of a 480×360 resolution RGB image in Cornell Grasp Dataset, and 90.4% robot grasping success rate for unknown objects with a parallel plate gripper in the real environment.

Robotic in-hand manipulation has been a longstanding challenge due to the complexity of modelling hand and object in contact and of coordinating finger motion for complex manipulation sequences. To address these challenges, the majority of prior work has either focused on model-based, low-level controllers or on model-free deep reinforcement learning that each have their own limitations. We propose a hierarchical method that relies on traditional, model-based controllers on the low-level and learned policies on the mid-level. The low-level controllers can robustly execute different manipulation primitives (reposing, sliding, flipping). The mid-level policy orchestrates these primitives. We extensively evaluate our approach in simulation with a 3-fingered hand that controls three degrees of freedom of elongated objects. We show that our approach can move objects between almost all the possible poses in the workspace while keeping them firmly grasped. We also show that our approach is robust to inaccuracies in the object models and to observation noise. Finally, we show how our approach generalizes to objects of other shapes.

This paper presents a robust and accurate approach for the rigid registration of pre-operative and intraoperative point sets in image-guided surgery (IGS). Three challenges are identified in the pre-to-intraoperative registration: the intra-operative 3D data (usually forms a 3D curve in space) (1) is often contaminated with noise and outliers; (2) usually only covers a partial region of the whole pre-operative model; (3) is usually sparse. To tackle those challenges, we utilize the tangent vectors extracted from the sparse intraoperative data points and the normal vectors extracted from the pre-operative model points. Our first contribution is to formulate a novel probabilistic distribution of the error between a pair of corresponding tangent and normal vectors. The second contribution is, based on the novel distribution, we formulate the registration of two multi-dimensional (6D) point sets as a maximum likelihood (ML) problem and solve it under the expectation maximization (EM) framework. Our last contribution is, in order to facilitate the computation process, the derivatives of the objective function with respect to desired parameters are presented. We conduct extensive experiments to demonstrate that our approach outperforms the state-of-the-art methods. Importantly, in the context of anteriro cruciate ligament (ACL) reconstruction, our method can achieve as low as 0.6795 mm mean target registration error (TRE) value with considerable noises and very limited overlapping ratios.

Wireless Capsule Endoscopy (WCE) has the advantage of reducing the invasiveness and pain of gastrointestinal examinations. In this work, we propose a system aimed at autonomously accelerating and locating the WCE inside the intestine for clinical applications. A rotating magnet controlled by a robotic arm is placed outside the patient's body to actuate the capsule with an internal magnetic ring, and the magnetic fields of the two sources are measured by an external sensor array. The original Multiple Objects Tracking method is improved by combining Normal Vector Fitting, Bézier Curve Gradient, and Spherical Linear Interpolation to estimate the 6-D pose of the WCE from a 5-D pose sequence. In order to close the actuation-localization loop, a strategy is presented to react to different states of the capsule. The proposed method is validated via experiments on phantoms as well as on animal intestines. The localization of the capsule shows an accuracy of 3.5mm in position and 9.4° in orientation, and the average update frequency of the estimated 6-D pose reaches 25Hz.

In this paper, we propose a novel algorithm that estimates the pose of the robot end effector using depth vision. The input to our system is the segmented robot hand point cloud from a depth sensor. Then a neural network takes a point cloud as input and outputs the position and orientation of the robot end effector in the camera frame. The estimated pose can serve as the input of the controller of the robot to reach a specific pose in the camera frame. The training process of the neural network takes the simulated rendered point cloud generated from different poses of the robot hand mesh. At test time, one estimation of a single robot hand pose is reduced to 10ms on gpu and 14ms on cpu, which makes it suitable for close loop robot control system that requires to estimate hand pose in an online fashion. We design a robot hand pose estimation experiment to validate the effectiveness of our algorithm working in the real situation. The platform we used includes a Kinova Jaco 2 robot arm and a Kinect v2 depth sensor. We describe all the processes that use vision to improve the accuracy of pose estimation of the robot end-effector. We demonstrate the possibility of using point cloud to directly estimate the robot's end-effector pose and incorporate the estimated pose into the controller design of the robot arm.

As one of the most promising areas, mobile robots draw much attention these years. Current work in this field is often evaluated in a few manually designed scenarios, due to the lack of a common experimental platform. Meanwhile, with the recent development of deep learning techniques, some researchers attempt to apply learning-based methods to mobile robot tasks, which requires a substantial amount of data. To satisfy the underlying demand, in this paper we build HouseExpo, a large-scale indoor layout dataset containing 35, 126 2D floor plans including 252, 550 rooms in total. Together we develop PseudoSLAM, a lightweight and efficient simulation platform to accelerate the data generation procedure, thereby speeding up the training process. In our experiments, we build models to tackle obstacle avoidance and autonomous exploration from a learning perspective in simulation as well as real-world experiments to verify the effectiveness of our simulator and dataset. All the data and codes are available online and we hope HouseExpo and PseudoSLAM can feed the need for data and benefit the whole community.

Inertial Measurement Unit (IMU) has long been a dream for stable and reliable motion estimation, especially in indoor environments where GPS strength limits. In this paper, we propose a novel method for position and orientation estimation of a moving object only from a sequence of IMU signals collected from the phone. Our main observation is that human motion is monotonous and periodic. We adopt the Extended Kalman Filter and use the learning-based method to dynamically update the measurement noise of the filter. Our pedestrian motion tracking system intends to accurately estimate planar position, velocity, heading direction without restricting the phone's daily use. The method is not only tested on the self-collected signals, but also provides accurate position and velocity estimations on the public RIDI dataset, i.e., the absolute transmit error is 1.28m for a 59-second sequence.

In this paper, we propose a novel path planning algorithm for the nonholonomic multiple mobile robot system with applications to a robotic autonomous luggage trolley collection system at airports. We consider this path planning algorithm as a Multiple Traveling Salesman Problem (MTSP). Our path planning algorithm consists of three parts. First, we use the Minimum Spanning Tree (MSP) algorithm to divide the MTSP into a number of independent TSPs, which achieves the task assignment for each mobile robot. Secondly, we implement a closed-loop forward control policy based on the kinematic model of the mobile robot to get a feasible and smooth path. The control cost of the path is used as the new metric in solving the TSPs. Finally, in order to adapt to our case, we modify the TSP as an Open Dynamic Traveling Salesman Problem with Fixed Start (ODTSP-FS) and implement an ant colony algorithm to achieve the path planning for each mobile robot. We evaluate our algorithm with simulation experiments and the experimental results demonstrate that our algorithm can quickly generate feasible and smooth paths for each robot while satisfying the nonholonomic constraints.

Wireless capsule endoscopy (WCE) is a novel imaging tool that allows the noninvasive visualization of the entire gastrointestinal (GI) tract without causing discomfort to the patients. Although convolutional neural networks (CNNs) have obtained promising performance for the automatic lesion recognition, the results of the current approaches are still limited due to the small lesions and the background interference in the WCE images. To overcome these limits, we propose a Third-order Attention guided Lesion Amplification Network (TALA-Net) for WCE image classification. The TALA-Net consists of two branches, including a global branch and an attention-aware branch. Specifically, taking the high-level features in the global branch as the input, we propose a Third-order Attention (ToA) module to generate attention maps that can indicate potential lesion regions. Then, an Attention Guided Lesion Amplification (AGLA) module is proposed to deform multiple level features in the global branch, so as to zoom in the potential lesion features. The deformed features are fused into the attention-aware branch to achieve finer-scale lesion recognition. Finally, predictions from the global and attention-aware branches are averaged to obtain the classification results. Extensive experiments show that the proposed TALA-Net outperforms state-of-the-art methods with an overall classification accuracy of 94.72% on the WCE dataset.

Accurate recognition of polyps is crucial for early colorectal cancer diagnosis and treatment. Wireless capsule endoscopy (WCE) is a noninvasive, wireless imaging tool that allows direct visualization of the entire colon without discomfort to patients and has the potential to revolutionize the screening workup for colorectal diseases. However, current manual review is laborious and time consuming, requiring the undivided concentration of the gastroenterologist. Computational methods that can assist automated polyp recognition will enhance the outcome both in terms of diagnostic accuracy and efficiency of WCE. This review introduces the computer-assisted algorithms as applied to colorectal polyp screening, focusing on the successes of deep-learning-based strategies in the WCE sequences. We survey key applications of WCE polyp recognition, covering deep-learning-based image-level classification, lesion region detection, and pixel-accurate segmentation. We conclude by discussing emerging research challenges, possible trends, and future directions.

Polyp recognition in colonoscopy images is crucial for early colorectal cancer detection and treatment. However, the current manual review requires undivided concentration of the gastroenterologist and is prone to diagnostic errors. In this article, we present an effective, two-stage approach called PLPNet, where the abbreviation “PLP” stands for the word “polyp,” for automated pixel-accurate polyp recognition in colonoscopy images using very deep convolutional neural networks (CNNs). Compared to hand-engineered approaches and previous neural network architectures, our PLPNet model improves recognition accuracy by adding a polyp proposal stage that predicts the location box with polyp presence. Several schemes are proposed to ensure the model's performance. First of all, we construct a polyp proposal stage as an extension of the faster R-CNN, which performs as a region-level polyp detector to recognize the lesion area as a whole and constitutes stage I of PLPNet. Second, stageII of PLPNet is built in a fully convolutional fashion for pixelwise segmentation. We define a feature sharing strategy to transfer the learned semantics of polyp proposals to the segmentation task of stage II, which is proven to be highly capable of guiding the learning process and improve recognition accuracy. Additionally, we design skip schemes to enrich the feature scales and thus allow the model to generate detailed segmentation predictions. For accurate recognition, the advanced residual nets and feature pyramids are adopted to seek deeper and richer semantics at all network levels. Finally, we construct a two-stage framework for training and run our model convolutionally via a single-stream network at inference time to efficiently output the polyp mask. Experimental results on public data sets of GIANA Challenge demonstrate the accuracy gains of our approach, which surpasses previous state-of-the-art methods on the polyp segmentation task (74.7 Jaccard Index) and establishes new top results in the polyp localization challenge (81.7 recall).

Visual localization is a fundamental capability in robotics and has been well studied for recent decades. Although many state-of-the-art algorithms have been proposed, great success usually builds on the assumption that the working environment is static. In most of the real scenes, the assumption cannot hold because there are inevitably moving objects, especially humans, which significantly degrade the localization accuracy. To address this problem, we propose a robust visual localization system building on top of a feature-based visual simultaneous localization and mapping algorithm. We design a dynamic region detection method and use it to preprocess the input frame. The detection process is achieved in a Bayesian framework which considers both the prior knowledge generated from an object detection process and observation information. After getting the detection result, feature points extracted from only the static regions will be used for further visual localization. We performed the experiments on the public TUM data set and our recorded data set, which shows the daily dynamic scenarios. Both qualitative and quantitative results are provided to show the feasibility and effectiveness of the proposed method.

Visual localization has been well studied in recent decades and applied in many fields as a fundamental capability in robotics. However, the success of the state of the arts usually builds on the assumption that the environment is static. In dynamic scenarios where moving objects are present, the performance of the existing visual localization systems degrades a lot due to the disturbance of the dynamic factors. To address this problem, we propose a novel sparse motion removal (SMR) model that detects the dynamic and static regions for an input frame based on a Bayesian framework. The similarity between the consecutive frames and the difference between the current frame and the reference frame are both considered to reduce the detection uncertainty. After the detection process is finished, the dynamic regions are eliminated while the static ones are fed into a feature-based visual simultaneous localization and mapping (SLAM) system for further visual localization. To verify the proposed method, both qualitative and quantitative experiments are performed and the experimental results have demonstrated that the proposed model can significantly improve the accuracy and robustness for visual localization in dynamic environments.Note to Practitioners-This article was motivated by the visual localization problem in dynamic environments. Visual localization is well applied in many robotic fields such as path planning and exploration as the basic capability for a mobile robot. In the GPS-denied environments, one robot needs to localize itself through perceiving the unknown environment based on a visual sensor. In real-world scenes, the existence of the moving objects will significantly degrade the localization accuracy, which makes the robot implementation unreliable. In this article, an SMR model is designed to handle this problem. Once receiving a frame, the proposed model divides it into dynamic and static regions through a Bayesian framework. The dynamic regions are eliminated, while the static ones are maintained and fed into a feature-based visual SLAM system for further visual localization. The proposed method greatly improves the localization accuracy in dynamic environments and guarantees the robustness for robotic implementation.

The optical tracking system (OTS) plays a vital role in the computer-assisted surgical navigation process, whereas the performance of the commonly used binocular stereo vision is affected by the line-of-sight problem and limited workspace. Thus, this article proposed a prior knowledge-based multicamera reconstruction model (PKRM) to both expand the tracking workspace and improve the tracking robust and computational efficiency of OTS when working in unstructured clinical conditions. This reconstruction model inherits the advantages of the geometrical method, data-driven method, and gating technique (GT). First, we added the geometric principle as the prior knowledge to optimize the training of the multicamera OTS reconstruction model through the Lagrange multiplier method; hence, the prior knowledge feedforward NN (PKFNN) was built. Second, besides the training features, the state of camera (SOC) was extracted in advance to determine the NN structure using GT. According to the SOC feature, the OTS can be self-adaptive to the changing field of view (FOV) caused by optical occlusion, which is frequently occurred in surgery. Furthermore, experiments were carried out to verify the performance of the proposed model, whose accuracy and runtime performed 0.4627 mm and 0.0016 ms, respectively. Results demonstrate that the proposed reconstruction model can achieve higher accuracy and computational efficiency than both the geometrical model and the data-driven model. Especially, by considering SOC as the state prior knowledge, the tracking robustness is enhanced when one or two of the four cameras are not working properly. Note to Practitioners-The original motivation for this article derives from both the line-of-sight limitation and robust demand for optical tracking of surgical instruments. The performance of the multicamera optical tracking system (OTS) depends on its reconstruction model. However, the geometric reconstruction model requires more calculation to obtain high accuracy, which will enlarge the latency and reduce the update rate. In our previous work, the reconstruction model based on the neural network (NN) has achieved accurate tracking in real-time, while the training of the model tends into local optimal values. Hence, we proposed the prior knowledge feedforward NN model to improve the accuracy and computational efficiency. Moreover, to guarantee the line-of-sight in the optical occlusion, the state of camera combining with the gating technique enables the OTS to be self-adaptive for changing the field of view, which greatly ensures the robust tracking process with larger workspace in case of line-of-sight obstructions.

Fruit detection is a fundamental task for automatic yield estimation. The goal is to detect all the fruits in images. The-state of the art of fruit detection algorithm, Faster R-CNN, shows a lack of detection advantage on small fruits. One of the reasons is only that single-level features and a classifier are used for localization of proposal candidates. In this article, we propose to incorporate a multiple classifier fusion strategy into a Faster R-CNN network for small fruit detection. We utilize features from three different levels to learn three classifiers for objectness classification in the stage of proposal localization. Probabilities from classifiers are combined by a simple convolutional layer to generate final objectness classification for proposal candidates. During training, in order to train a model with strong generalization capability, we propose to use correlation coefficients to measure the diversity of multiple classifiers. A novel loss function with classifier correlation is introduced to train the region proposal network. We evaluate the proposed model on two data sets of small fruits. Extensive experiments show that the proposed model outperforms the state-of-the-art detectors for fruit detection.

In a paired-point rigid registration, target registration error (TRE) is deemed to be the most important quality metric. TRE usually cannot be directly measured, and thus many TRE estimation algorithms have been proposed. However, target localization errors (TLEs) in two spaces are not considered in the definition of TRE. In this paper, we propose a new type of evaluation metric that is referred to as total TRE (TTRE) at a given target point with TLE incorporated. Statistics including mean, root mean square (rms), and covariance matrix of TTRE are derived without making any assumption of the TLE magnitude. TTRE and fiducial registration error (FRE) are proved to be uncorrelated when an ideal weighting scheme is adopted in solving the registration problem. The proposed error model is validated through extensive experiments. In the first experiment with random fiducials and targets, in 90% of the test cases, there shows no difference between the predicted and simulated TTRE statistics when six fiducials are used. In the second experiment of deep-brain stimulation surgery, the mean value of CC(TTRE,FRE) being 8.9246 × 10-4± 0.0389 was observed, which indicates that TTRE and FRE are uncorrelated. In the third experiment of surgical tool-tip tracking, the mean and standard deviation of percentage differences between predicted and simulated TTRE rms values are 2.22% ± 0.77% for the planar tool and 2.62% ± 0.59% for the textral tool. In summary, our proposed algorithm can well model the TTRE metric.

This paper introduces a robust generalized point cloud (PC) registration method that utilizes not only the positional but also the orientation information associated with each point. The proposed method solves the rigid PC registration problem in a probabilistic manner, which casts the problem into a maximum likelihood (ML) framework. A hybrid mixture model (HMM) is utilized to represent one generalized PC. In the HMM, a von-Mises-Fisher mixture model (FMM) is adopted to model the orientational uncertainty, while a Gaussian mixture model (GMM) is used to represent the positional uncertainty. An expectation-maximization (EM) algorithm is adopted to solve the optimization problem in an iterative manner to find the optimal rotation matrix and the translation vector between two generalized PCs. In both expectation step (E step) and maximization step (M step), orientational information is utilized, which can potentially improve the algorithm's robustness to noise and outliers. In the E step, the posterior probabilities that represent the degree of point correspondences in two PCs are computed. In the M step, an efficient closed-form solution to a rigid transformation matrix is developed. E and M steps will iterate until certain convergence criteria are satisfied. Extensive experiments under different noise levels and outlier ratios have been carried out on a data set of femur bone computed tomography images. Experimental results show that the proposed method outperforms the state-of-the-art ones in terms of accuracy, robustness, and convergence speed significantly. Note to Practitioners-This paper was motivated by solving the problem of registering two PCs. Most existing approaches generally use only the positional information associated with each point and thus lack robustness to noise and outliers. This paper suggests a new robust method that also adopts the normal vectors associated with each point. The registration problem is cast into a maximum likelihood (ML) problem and solved under the expectation-maximization (EM) framework. Closed-form solutions for estimating parameters in both expectation and maximization steps are provided in this paper. We have demonstrated through extensive experiments that the proposed registration algorithm achieves improved accuracy, robustness to noise and outliers, and faster convergence speed.

Aligning different views or representations of anatomy is an essential task in both medical imaging computing (MIC) and computer-assisted interventions' (CAIs') communities. Motivated by simultaneously registering multiple point sets (PSs) and further improving the algorithm's robustness to outliers and noise, in this paper, we propose a novel probabilistic approach to jointly register multiple generalized PSs. A generalized PS includes high-dimensional points consisting of both positional vectors and orientational information (or normal vectors). Hybrid mixture models (HMMs) combining Gaussian and von Mises-Fisher (VMF) distributions are used to model positional and orientational components of the generalized PSs. All generalized PSs are jointly registered using the expectation-maximization (EM) technique. In the E-step, the posterior probabilities representing point correspondence confidences are computed. In the M-step, the rigid transformation matrices, positional variances, and orientational concentration parameters are updated for each generalized PS. E and M steps will iterate until some termination condition is satisfied. We validate our algorithm using the surface points extracted from the human femur CT model. The experimental results demonstrate that the proposed algorithm outperforms the state-of-the-art ones in terms of the accuracy, robustness, as well as convergence speed. In addition, our algorithm is able to recover a better central PS than the state-of-the-art one does in the case of registering multiple PSs. Our algorithm is very suitable for registering complex structures arising in medical imaging. This paper was motivated by solving the problem of registering two or multiple point sets. Most existing approaches generally use only the positional information associated with each point and thus lack robustness to noise and outliers. This paper suggests a new robust method that also adopts the normal vectors associated with each point. The registration problem is cast into a maximum-likelihood (ML) problem and solved under the expectation-maximization (EM) framework. Closed-form solutions to estimating parameters in both expectation and maximization steps are provided in this paper. We have demonstrated through extensive experiments that the proposed registration algorithm achieves improved accuracy, robustness to noise and outliers, and faster convergence speed.

Rapidly random-exploring tree (RRT) and its variants are very popular due to their ability to quickly and efficiently explore the state space. However, they suffer sensitivity to the initial solution and slow convergence to the optimal solution, which means that they consume a lot of memory and time to find the optimal path. It is critical to quickly find a short path in many applications such as the autonomous vehicle with limited power/fuel. To overcome these limitations, we propose a novel optimal path planning algorithm based on the convolutional neural network (CNN), namely the neural RRT* (NRRT*). The NRRT* utilizes a nonuniform sampling distribution generated from a CNN model. The model is trained using quantities of successful path planning cases. In this article, we use the A* algorithm to generate the training data set consisting of the map information and the optimal path. For a given task, the proposed CNN model can predict the probability distribution of the optimal path on the map, which is used to guide the sampling process. The time cost and memory usage of the planned path are selected as the metric to demonstrate the effectiveness and efficiency of the NRRT*. The simulation results reveal that the NRRT* can achieve convincing performance compared with the state-of-the-art path planning algorithms. Note to Practitioners-The motivation of this article stems from the need to develop a fast and efficient path planning algorithm for practical applications such as autonomous driving, warehouse robot, and countless others. Sampling-based algorithms are widely used in these areas due to their good scalability and high efficiency. However, the quality of the initial path is not guaranteed and it takes much time to converge to the optimal path. To quickly obtain a high-quality initial path and accelerate the convergence speed, we propose the NRRT*. It utilizes a nonuniform sampling distribution and achieves better performance. The NRRT* can be also applied to other sampling-based algorithms for improved results in different applications.

In a human-robot coexisting environment, it is pivotal for a mobile service robot to arrive at the goal position safely and efficiently. In this article, an elastic band-based rapidly exploring random tree (EB-RRT) algorithm is proposed to achieve real-time optimal motion planning for the mobile robot in the dynamic environment, which can maintain a homotopy optimal trajectory based on current heuristic trajectory. Inspired by the EB method, we propose a hierarchical framework consisting of two planners. In the global planner, a time-based RRT algorithm is used to generate a feasible heuristic trajectory for a specific task in the dynamic environment. However, this heuristic trajectory is nonoptimal. In the dynamic replanner, the time-based nodes on the heuristic trajectory are updated due to the internal contraction force and the repulsive force from the obstacles. In this way, the heuristic trajectory is optimized continuously, and the final trajectory can be proved to be optimal in the homotopy class of the heuristic trajectory. Simulation experiments reveal that compared with two stateof-the-art algorithms, our proposed method can achieve better performance in dynamic environments.

Automatic polyp recognition in endoscopic images is challenging because of the low contrast between polyps and the surrounding area, the fuzzy and irregular polyp borders, and varying imaging light conditions. In this article, we propose a novel densely connected convolutional network with “unbalanced discriminant (UD)” loss and “category sensitive (CS)” loss (DenseNet-UDCS) for the task. We first utilize densely connected convolutional network (DenseNet) as the basic framework to conduct end-to-end polyp recognition task. Then, the proposed dual constraints, UD loss and CS loss, are simultaneously incorporated into the DenseNet model to calculate discriminative and suitable image features. The UD loss in our network effectively captures classification errors from both majority and minority categories to deal with the strong data imbalance of polyp images and normal ones. The CS loss imposes the ratio of intraclass and interclass variations in the deep feature learning process to enable features with large interclass variation and small intraclass compactness. With the joint supervision of UD loss and CS loss, a robust DenseNet-UDCS model is trained to recognize polyps from endoscopic images. The experimental results achieved polyp recognition accuracy of 93.19%, showing that the proposed DenseNet-UDCS can accurately characterize the endoscopic images and recognize polyps from the images. In addition, our DenseNet-UDCS model is superior in detection accuracy in comparison with state-of-the-art polyp recognition methods. Note to Practitioners-Wireless capsule endoscopy (WCE) is a crucial diagnostic tool for polyp detection and therapeutic monitoring, thanks to its noninvasive, user-friendly, and nonpainful properties. A challenge in harnessing the enormous potential of the WCE to benefit the gastrointestinal (GI) patients is that it requires clinicians to analyze a huge number of images (about 50 000 images for each patient). We propose a novel automatic polyp recognition scheme, namely, DenseNet-UDCS model, by addressing practical image unbalanced problem and small interclass variances and large intraclass differences in the data set. The comprehensive experimental results demonstrate superior reliability and robustness of the proposed model compared to the other polyp recognition approaches. Our DenseNet-UDCS model can be further applied in the clinical practice to provide valuable diagnosis information for GI disease recognition and precision medicine.

Wireless capsule endoscopy (WCE) is a novel imaging tool that allows noninvasive visualization of the entire gastrointestinal (GI) tract without causing discomfort to patients. Convolutional neural networks (CNNs), though perform favorably against traditional machine learning methods, show limited capacity in WCE image classification due to the small lesions and background interference. To overcome these limits, we propose a two-branch Attention Guided Deformation Network (AGDN) for WCE image classification. Specifically, the attention maps of branch1 are utilized to guide the amplification of lesion regions on the input images of branch2, thus leading to better representation and inspection of the small lesions. What's more, we devise and insert Third-order Long-range Feature Aggregation (TLFA) modules into the network. By capturing long-range dependencies and aggregating contextual features, TLFAs endow the network with a global contextual view and stronger feature representation and discrimination capability. Furthermore, we propose a novel Deformation based Attention Consistency (DAC) loss to refine the attention maps and achieve the mutual promotion of the two branches. Finally, the global feature embeddings from the two branches are fused to make image label predictions. Extensive experiments show that the proposed AGDN outperforms state-of-the-art methods with an overall classification accuracy of 91.29% on two public WCE datasets. The source code is available at https://github.com/hathawayxxh/WCE-AGDN.

Point set registration (PSR) is an important problem in computer vision, robotics and biomedical engineering communities. Usually, only positional information at each point is adopted in a registration. In this paper, the orientational vector (or normal vector) associated with each point is also utilized. Generalized point set registration is formulated and solved under the Expectation-Maximization (EM) framework. In the E-step, the posterior probabilities representing the correspondence probabilities are computed. In the Mstep, rigid transformation parameters including the rotation matrix, the translation vector are updated. The proposed algorithm stops when it converges to the optimal solution or a maximum number of iterations is achieved. The observed position set and normal vector set are assumed to follow Gaussian Mixture Models (GMMs) and Fisher distribution Mixture Models (FMMs), respectively. To further improve our algorithm's robustness, the hybrid mixture models (HMMs) are assumed to be inhomogeneous. Experimental results on the surface points extracted from a human femur' CT model show that our algorithm can achieve lower registration error, is more robust to noise and outliers than the state-of-the-art registration methods.

Unmanned Aerial Vehicles (UAVs) have been implemented for environmental monitoring by using their capabilities of mobile sensing, autonomous navigation, and remote operation. However, in real-world applications, the limitations of on-board resources (e.g., power supply) of UAVs will constrain the coverage of the monitored area and the number of the acquired samples, which will hinder the performance of field estimation and mapping. Therefore, the issue of constrained resources calls for an efficient sampling planner to schedule UAV-based sensing tasks in environmental monitoring. This paper presents a mission planner of coverage sampling and path planning for a UAV-enabled mobile sensor to effectively explore and map an unknown environment that is modeled as a random field. The proposed planner can generate a coverage path with an optimal coverage density for exploratory sampling, and the associated energy cost is subjected to a power supply constraint. The performance of the developed framework is evaluated and compared with the existing state-of-the-art algorithms, using a real-world dataset that is collected from an environmental monitoring program as well as physical field experiments. The experimental results illustrate the reliability and accuracy of the presented coverage sampling planner in a prior survey for environmental exploration and field mapping.

Modern agricultural robots like drones have been studied in automatic yield estimation in recent years. Fruit counting is a fundamental task in the automatic yield estimation, on which significant progress has been achieved by detection-based methods and segmentation-regression-based methods. However, for clustered-fruit counting, the existing methods lack advantages on the localization of small and occluded fruits or discrete number regression. In addition, it is observed that existing deep neural network based counting methods have high variances on fruit density map estimation. Aiming at solving these two problems and decreasing the regression variance, in this paper, we propose a density-map-estimation model with DropBlock regularization. For evaluating the proposed model, we propose a new Clustered-Fruit dataset. Extensive experiments show that the proposed model is effective and outperforms the state-of-the-art counting methods on the Clustered-Fruit dataset. Our dataset is available at Clustered-Fruit.

Non-rigid point set registration (PSR) is an outstanding and fundamental problem in fields of robotics, computer vision, medical image analysis and imageguided surgery. The aim of a non-rigid registration problem is to align together two point sets that have been deformed. We have derived and presented a novel registration algorithm that non-rigidly registers two point sets together. The assumption of isotropic localization error is shared in the previous non-rigid registration algorithms. In this paper, the position localization error is generalized to the anisotropic cases, which means that the error distribution is not the same in different spatial directions. The motivation of considering the anisotropic characteristic is that the point localization error is actually different in three spatial directions in real applications. Mathematically, the difficulty in dealing with the anisotropic error case comes from the change from a standard deviation that is a scalar to a covariance matrix. The formulas for updating the parameters in both expectation and maximization steps are derived. In the expectation step, we compute the posterior probabilities that represent the correspondences between points in two PSs. In the maximization step, given the current posteriors, the covariance matrix of the position localization error and the non-rigid transformation are updated. To facilitate the proposed algorithm, the low-rank approximation variation of our method is also presented. We have demonstrated through experiments that the proposed algorithm outperforms the state-of-the-art ones in terms of registration and accuracy and robustness to noise. More specifically, most of the experimental results have passed the statistical tests at the 5% significance level.

In a human-robot coexisting environment, reaching the target place efficiently and safely is pivotal for a mobile service robot. In this paper, a Risk-based Dual-Tree Rapidly exploring Random Tree (Risk-DTRRT) algorithm is proposed for the robot motion planning in a dynamic environment, which provides a homotopy optimal trajectory on the basis of a heuristic trajectory. A dual-tree framework consisting of an RRT tree and a rewired tree is proposed for the trajectory searching. The RRT tree is a time-based tree, considering the future trajectory predictions of the pedestrians, and this tree is utilized to generate a heuristic trajectory. However, the heuristic trajectory is usually nonoptimal. Then, a line-of-sight (LoS) control checking algorithm is proposed to detect whether two time-based nodes can be rewired with the least cost. On the basis of the LoS control checking algorithm, a tree rewiring algorithm is proposed to optimize the heuristic trajectory. The tree generated in the tree rewiring process is called the rewired tree. The trajectory generated by the Risk-DTRRT algorithm proves to be optimal in the homotopy class of the heuristic trajectory. The navigation run time and the lengths of the planned trajectories are selected to demonstrate the effectiveness of the proposed algorithm. The experimental results in both simulation studies and real-world implementations reveal that our proposed method achieves convincing performance in both static and dynamic environments. Note to Practitioners-This paper is motivated by planning optimized trajectories for the mobile service robots in dynamic environments with pedestrians. In this area, the sampling-based motion planning algorithms have been widely used for their high efficiency and robustness. However, the real-time optimality of the motion planning cannot be guaranteed due to the challenges caused by the moving pedestrians. In this paper, we propose a dual-tree framework to solve this problem. First, a classic Rapidly exploring Random Tree (RRT) is constructed to generate a heuristic trajectory. Then, instead of reconnecting the nodes on the heuristic trajectory directly, a rewired tree is built to optimize the heuristic trajectory. This proposed dual-tree framework can fully exploit the information of the RRT tree and ensure the completeness of the motion planning. The proposed motion planning algorithm also considers the constraints of the nonholonomic mobile robots, and it can be applied in most mobile service robots to improve their motion planning quality.

Foreground moving object segmentation is a fundamental problem in many computer vision applications. As a solution for foreground segmentation, background modeling has been intensively studied over past years and many effective algorithms have been developed. However, accurate foreground segmentation is still a difficult problem. Currently, most of the algorithms work solely within the color space, in which the segmentation performance is prone to be degraded by a multitude of challenges, such as illumination changes, shadows, automatic camera adjustments, and color camouflage. RGB-D cameras are active visual sensors that provide depth measurements along with color images. We present in this paper an innovative background modeling method by using both the color and depth information from an RGB-D camera. The proposed method is evaluated using a public RGB-D data set. Various experiments confirm that our method is able to achieve superior performance compared with existing well-known methods.

In this paper, we propose a novel path planning framework for autonomous exploration in unknown environments using a mobile robot. A graph structure is incrementally constructed along with the exploration process. The structure is the road map that represents the topology of the explored environment. To construct the road map, we design a sampling strategy to get random points in the explored environment uniformly. A global path from the current location of the robot to the target area can be found on this road map efficiently. We utilize a lazy collision checking method that only checks the feasibility of the generated global path to improve the planning efficiency. The feasible global path is further optimized with our proposed trajectory optimization method considering the motion constraints of the robot. This mechanism can facilitate the path cost evaluation for the next best view selection. In order to select the next best target region, we propose a utility function that takes into account both the path cost and the information gain of a candidate target region. Moreover, we present a target reselection mechanism to evaluate the target region and reduce the extra path cost. The efficiency and effectiveness of our approach are demonstrated using a mobile robot in both simulation and real experimental studies.

This paper presents a sensor fusion-based surgical instrument tracking system which uses multiple monocular modules. The system is an optical tracking system, which has been widely utilized in the image-guide surgery because of its high accuracy and precision. However, the line-of-sight occlusion problem which remains unresolved in current systems frustrates surgeons during the operation. To address this challenge, we propose a surgical instrument tracking system based on multiple monocular modules. The rationale is to enable the system to track the surgical instruments inside the surgical site from different views. Three sensor fusion algorithms are proposed to integrate all sensor data from the multimodule system. In order to show the feasibility of the tracking system, simulations and comparison experiments have been carried out. The intensive investigation results give a practical instruction to the real implementation of the proposed system in image-guided interventions. Moreover, an image-guided surgical trial by using a cadaver head has been carried out to validate the feasibility of the proposed system and the tracking algorithms. The results from both the simulation and the cadaver trial have shown the effectiveness of the proposed robust fusion algorithm.

Alignment of multiple point sets is an essential problem in medical imaging and computer-assisted surgery. For example, aligning multiple point sets into one common coordinate frame is a prerequisite for statistical shape modelling (SSM). In this paper, we first formally formulate the multiple generalized point cloud registration problem in a probabilistic manner. Not only positional but also the orientational information is utilized in the registration. All the observed generalized point sets to be registered are considered to be realizations of underlyinng unknown hybrid mixture models (HMMs). By (i) utilizing more enriched information, i.e. orientational information or normal vectors (ii) treating all point sets equally, our registration algorithm is more robust to outliers and does not bias towards any point set. Assuming that the positional and orientational data are co-independent, the probability density function (PDF) of an observed hybrid point is the multiplication of Gaussian and Fisher distributions. Notably, the positional error vector is assumed to obey a multivariate Gaussian distribution to accommodate anisotropic noise. Expectation maxmization (EM) framework is utilized to jointly estimate the parameters. In the E-step, the posteriors between points and underlying mixture model components are computed. In the M-step, the constrained optimization problem of the rigid transformation matrix is re-formulated as an unconstrained one using the Rodrigues Formula of a rotation matrix. Extensive experiments are conducted on CT data of a femur bone model to compare the proposed algorithm with the state-of-the-art registration methods. The experimental results demonstrate the algorithm's better accuracy, robustness to noise and outliers and faster convergence speed.

The-state-of-the-art of fruit detection with Faster R-CNN shows lack of detection advantage on small fruits. One of reasons is only single level features is used for localization of proposal candidates. In this paper, we propose to incorporate a multiple classifier fusion strategy into a Faster R-CNN network for small fruit detection. We utilize features from three different levels to learn three classifiers for objectness classification in the stage of proposal localization. Probabilities from classifiers are combined by a simple convolutional layer to generate final objectness classification for proposal candidates. In order to keep diversity of multiple classifiers, a novel loss term of classifier correlation is introduced into original loss function. Experimental results show that our model is feasible for detecting small fruits.

This paper introduces a robust point cloud registration method which utilizes not only positional but also the orientation information at each point. The proposed method takes a probabilistic approach which forms the problem as a hybrid mixture model, in which a Von-Mises-Fisher mixture model (FMM) is adopted to model the orientation part and a gaussian mixture model (GMM) is used to represent the position part. When two point clouds are optimally registered, the correspondence is the maximum of the posterior probability of the overall mixture model. Expectation-Maximization (EM) algorithm has been adopted to solve the optimization problem in an iterative manner to find the optimal rotation and translation between two point clouds. Extensive experiments under different noise levels and different outlier ratios have been carried out on a dataset of the femur CT images. Comparison results show that the proposed method outperforms the state-of-the-art methods under most of the experimental conditions, which indicates the validity of our method.

In this paper, we study the problem of autonomous exploration in unknown indoor environments using mobile robot. We use mutual information (MI) to evaluate the information the robot would get at a certain location. In order to get the most informative sensing location, we first propose a sampling method that can get random sensing patches in free space. Each sensing patch is extended to informative locations to collect information with true values. Then we use Gaussian Markov Random Fields (GMRF) to model the distribution of MI in environment. Compared with the traditional methods that employ Gaussian Process (GP) model, GMRF is more efficient. MI of every sensing location can be estimated using the training sample patches and the established GMRF model. We utilize an efficient computation algorithm to estimate the GMRF model hyperparameters so as to speed up the computation. Besides the information gain of the candidates regions, the path cost is also considered in this work. We propose a utility function that can balance the path cost and the information gain the robot would collect. We tested our algorithm in both simulated and real experiment. The experiment results demonstrate that our proposed method can explore the environment efficiently with relatively shorter path length.

Powered lower-limb orthoses and prostheses are attracting an increasing amount of attention in assisting daily living activities. To safely and naturally collaborate with human users, the key technology relies on an intelligent controller to accurately decode users' movement intention. In this work, we proposed an innovative locomotion recognition system based on depth images. Composed of a feature extraction subsystem and a finite-state-machine based recognition subsystem, the proposed approach is capable of capturing both the limb movements and the terrains right in front of the user. This makes it possible to anticipate the detection of locomotion modes, especially at transition states, thus enabling the associated wearable robot to deliver a smooth and seamless assistance. Validation experiments were implemented with nine subjects to trace a track that comprised of standing, walking, stair ascending, and stair descending, for three rounds each. The results showed that in steady state, the proposed system could recognize all four locomotion tasks with approximate 100% of accuracy. Out of 216 mode transitions, 82.4% of the intended locomotion tasks can be detected before the transition happened. Thanks to its high accuracy and promising prediction performance, the proposed locomotion recognition system is expected to significantly improve the safety as well as the effectiveness of a lower-limb assistive device.

Exploration region selection is an essential decision making process in autonomous robot exploration task. While a majority of greedy methods are proposed to deal with this problem, few efforts are made to investigate the importance of predicting long-term planning. In this paper, we present an algorithm that utilizes deep reinforcement learning (DRL) to learn exploration knowledge over office blueprints, which enables the agent to predict a long-term visiting order for unexplored subregions. On the basis of this algorithm, we propose an exploration architecture that integrates a DRL model, a next-best-view (NBV) selection approach and a structural integrity measurement to further improve the exploration performance. At the end of this paper, we evaluate the proposed architecture against other methods on several new office maps, showing that the agent can efficiently explore uncertain regions with a shorter path and smarter behaviors.

Alignment of two point clouds is an essential problem in medical robotics and computer-assisted surgery. In this paper, we first formally formulate the generalized point cloud registration problem in a probabilistic manner. Specifically, not only positional but also the orientational information are incorporated into registration. Notably, the positional error is assumed to obey a multivariate Gaussian distribution to accommodate anisotropic cases. Expectation conditional maximization framework is utilized to solve the problem. In E-step, the correspondence probabilities between points in two generalized point clouds are computed. In M -step, the constrained optimization problem with respect to the transformation matrix is re-formulated as an unconstrained one. Extensive experiments are conducted to compare the proposed algorithm with the state-of-the-art registration methods. The experimental results demonstrate the algorithm's robustness to noise and outliers, fast convergence speed.

Autonomous elevator operation is considered an intelligent solution in handling the inter-floor navigation problem of service robots. As one of the most fundamental steps, elevator button recognition starts to receive more and more attention. However, due to the challenging image conditions and severe class imbalance problem, the performance of existing results is unsatisfying. In this paper, we propose to combine an optical character recognition (OCR) network and the Faster RCNN architecture into a single neural network, called OCR-RCNN to facilitate an end-to-end training and elevator button recognition procedure. To verify our method, we collect a large dataset of elevator panels and carry out extensive comparative experiments. The experiment results show that our method can greatly outperform the traditional recognition pipelines, yielding an accurate and robust performance on recognizing untrained elevator buttons.

Wireless capsule endoscopy (WCE) enables clinicians to examine the digestive tract without any surgical operations, at the cost of a large amount of images to be analyzed. The main challenge for automatic computer-aided diagnosis arises from the difficulty of robust characterization of these images. To tackle this problem, a novel discriminative joint-feature topic model (DJTM) with dual constraints is proposed to classify multiple abnormalities in WCE images. We first propose a joint-feature probabilistic latent semantic analysis (PLSA) model, where color and texture descriptors extracted from same image patches are jointly modeled with their conditional distributions. Then the proposed dual constraints: visual words importance and local image manifold are embedded into the joint-feature PLSA model simultaneously to obtain discriminative latent semantic topics. The visual word importance is proposed in our DJTM to guarantee that visual words with similar importance come from close latent topics while the local image manifold constraint enforces that images within the same category share similar latent topics. Finally, each image is characterized by distribution of latent semantic topics instead of low level features. Our proposed DJTM showed an excellent overall recognition accuracy 90.78%. Comprehensive comparison results demonstrate that our method outperforms existing multiple abnormalities classification methods for WCE images.

In this paper, we propose a novel automatic computer-aided method to detect polyps for colonoscopy videos. To capture perceptually and semantically meaningful salient polyp regions, we first segment images into multilevel superpixels. Each level corresponds to different sizes of superpixels. Rather than adopting hand-designed features to describe these superpixels in images, we employ sparse autoencoder (SAE) to learn discriminative features in an unsupervised way. Then, a novel unified bottom-up and top-down saliency method is proposed to detect polyps. In the first stage, we propose a weak bottom-up (WBU) saliency map by fusing the contrast-based saliency and object-center-based saliency together. The contrast-based saliency map highlights image parts that show different appearances compared with surrounding areas, whereas the object-center-based saliency map emphasizes the center of the salient object. In the second stage, a strong classifier with multiple kernel boosting is learned to calculate the strong top-down (STD) saliency map based on samples directly from the obtained multilevel WBU saliency maps. We finally integrate these two-stage saliency maps from all levels together to highlight polyps. Experiment results achieve 0.818 recall for saliency calculation, validating the effectiveness of our method. Extensive experiments on public polyp datasets demonstrate that the proposed saliency algorithm performs better compared with state-of-the-art saliency methods to detect polyps.

In this paper, we propose a gait recognition method for service robots to conduct human following tasks. A walking sequence segmentation method is designed to extract the consecutive gait cycles from an arbitrary walking sequence. Based on the segmentation results, a novel hybrid gait feature is proposed to capture the static, dynamic, and trajectory features for each segmented key and supplementary gait cycles. A dataset of 25 human subjects is collected to evaluate the proposed method in three different walking paths with various walking directions. Experimental results show that the proposed method achieves satisfactory performance in terms of identification accuracy and Fcomb indexes on our dataset. Compared with five state-of-the-art gait recognition methods, the proposed method achieves the best performance on human gait recognition based on the walking sequences defined in our proposed dataset.

Target registration error (TRE) is widely adopted to evaluate the accuracy of a paired-point rigid registration (PPRR). However, TRE is defined in such a way that target localization error (TLE) is not considered. In this paper, we first propose a new type of error that is referred to as total target registration error (TTRE). The statistical model of TTRE is derived that we take the TLE in two spaces to be registered into consideration. Results in the first simulation show that the developed model can accurately estimate the simulated TTRE root-mean-square (RMS) (RMS percent differences|| <; 1.5% ± 2%) in all test cases. When all elements of diagonal FLE and TLE covariance matrices are independently generated from a uniform distribution that spans from 0 to 1mm and the number of fiducials N ≥ 6, the mean and covariance matrix of TTRE are well modelled. We have also theoretically proved and validated through the second simulation that TTRE and fiducial registration error (FRE) are uncorrelated (correlation coefficient (CC) <; 0.1). Finally, TTRE and TRE were found to exhibit a low correlation (0.37 <; CC <; 0.46).

Robots are increasingly operating in indoor environments designed for and shared with people. However, robots working safely and autonomously in uneven and unstructured environments still face great challenges. Many modern indoor environments are designed with wheelchair accessibility in mind. This presents an opportunity for wheeled robots to navigate through sloped areas while avoiding staircases. In this paper, we present an integrated software and hardware system for autonomous mobile robot navigation in uneven and unstructured indoor environments. This modular and reusable software framework incorporates capabilities of perception and navigation. Our robot first builds a 3D OctoMap representation for the uneven environment with the 3D mapping using wheel odometry, 2D laser and RGB-D data. Then we project multilayer 2D occupancy maps from OctoMap to generate the the traversable map based on layer differences. The safe traversable map serves as the input for efficient autonomous navigation. Furthermore, we employ a variable step size Rapidly Exploring Random Trees that could adjust the step size automatically, eliminating tuning step sizes according to environments. We conduct extensive experiments in simulation and real-world, demonstrating the efficacy and efficiency of our system. (Supplemented video link: https://youtu.be/6XJWcsH1fk0).

Flexible manipulator enables curvilinear accessibility through small incisions or natural orifices for minimally invasive surgery and diagnosis, which makes it a good choice for minimally invasive surgery. In order to control the robot precisely and safely, the real-time position and shape information of the robot need to be measured well. In this paper, we propose a magnetic tracking based tip pose and shape detection method for wire driven flexible robots. A permanent magnet is mounted at the distal end of the robot. Its magnetic field can be sensed with a sensor array. Therefore, position and orientation of the tip can be estimated utilizing the tracking method. A shape sensing algorithm is then carried out to estimate the real-time shape based on the tip pose. With the tip pose and shape display in the reconstructed visual environment, navigation can be achieved. This method provides the advantages that no sensors are needed to mount on the robot and has no line-of-sight problem. Experimental results verified the feasibility of the proposed method. A navigation error of 1.9mm is achieved.

This paper introduces a generic framework for field surveillance using consumer rotorcrafts and ground vehicles. Building such an autonomous system comes with two key challenges in persistent perception and obstacle avoidance. We begin with explaining two core algorithms to solve the challenges: an auto-landing algorithm that enables a quadrotor to land on a moving ground vehicle at a speed of 6.00 m/s, and an obstacle avoidance algorithm that ensures the safety of the quadrotor during searching process. On the basis of these algorithms, the architecture and infrastructure of Hawkeye framework are presented as well. Hawkeye is designed to be a generic platform with extensibility that allows integration of other domain applications. We demonstrate the potential of Hawkeye framework in a simulated agriculture monitoring mission and report its performance at the end of the paper.

Wireless capsule endoscopy (WCE) has become a widely used diagnostic technique for the digestive tract, at the price of a large volume of data that needs to be analyzed. To tackle this problem, a new computer-aided system using novel features is proposed in this paper to classify WCE images automatically. In the feature learning stage, to obtain the representative visual words, we first calculate the color scale invariant feature transform from the bleeding, polyp, ulcer, and normal WCE image samples separately and then apply K -means clustering on these features to obtain visual words. These four types of visual words are combined together to composite the representative visual words for classifying the WCE images. In the feature coding stage, we propose a novel saliency and adaptive locality-constrained linear coding (SALLC) algorithm to encode the images. The SALLC encodes patch features based on adaptive coding bases, which are calculated by the distance differences among the features and the visual words. Moreover, it imposes the patch saliency constraint on the feature coding process to emphasize the important information in the images. The experimental results exhibit a promising overall recognition accuracy of 88.61%, validating the effectiveness of the proposed method.

Cloud computing enables users to share computing resources on-demand. The cloud computing framework cannot be directly mapped to cloud robotic systems with ad hoc networks since cloud robotic systems have additional constraints such as limited bandwidth and dynamic structure. However, most multirobotic applications with cooperative control adopt this decentralized approach to avoid a single point of failure. Robots need to continuously update intensive data to execute tasks in a coordinated manner, which implies real-time requirements. Thus, a resource allocation strategy is required, especially in such resource-constrained environments. This paper proposes a hierarchical auction-based mechanism, namely link quality matrix (LQM) auction, which is suitable for ad hoc networks by introducing a link quality indicator. The proposed algorithm produces a fast and robust method that is accurate and scalable. It reduces both global communication and unnecessary repeated computation. The proposed method is designed for firm real-time resource retrieval for physical multirobot systems. A joint surveillance scenario empirically validates the proposed mechanism by assessing several practical metrics. The results show that the proposed LQM auction outperforms state-of-the-art algorithms for resource allocation.

Wireless Capsule Endoscopy (WCE) allows physicians to examine the entire digestive system without any surgical operation. Although it provides a noninvasive imaging approach to access the gastrointestinal (GI) tract, the biggest drawback of this technology is the large numbers of images need to be diagnosed. In this paper, a global and local saliency coding (GLSAC) method is proposed to detect polyps from WCE images. We first extract the Scale Invariant Feature Transform (SIFT) features from image patches and apply K-means method on these features to obtain visual words. Since saliency is a fundamental characteristic of feature coding, a novel coding strategy: the global and local saliency coding (GLSAC) is proposed. The calculation of the global saliency coding and local saliency coding is obtained by the exponential function of feature differences to the nearest visual code and all visual words. Specifically, the local saliency strategy considers coding bases based on the distribution of the distances between the features and visual words adaptively. Furthermore, we present a concentric circle-based spatial-rotation-invariant pooling strategy to obtain final image features from the patch features. Experiment results achieve promising 91.13% accuracy and 92.51% sensitivity, validating the effectiveness of the proposed method. Moreover, the comparison results show that our strategy outperforms the state-of-the-art methods on the polyp recognition task.

Wireless capsule endoscopy (WCE) needs computerized method to reduce the review time for its large image data. In this paper, we propose an improved bag of feature (BoF) method to assist classification of polyps in WCE images. Instead of utilizing a single scale-invariant feature transform (SIFT) feature in the traditional BoF method, we extract different textural features from the neighborhoods of the key points and integrate them together as synthetic descriptors to carry out classification tasks. Specifically, we study influence of the number of visual words, the patch size and different classification methods in terms of classification performance. Comprehensive experimental results reveal that the best classification performance is obtained with the integrated feature strategy using the SIFT and the complete local binary pattern (CLBP) feature, the visual words with a length of 120, the patch size of 8*8, and the support vector machine (SVM). The achieved classification accuracy reaches 93.2%, confirming that the proposed scheme is promising for classification of polyps in WCE images.

Wireless capsule endoscopy (WCE) enables noninvasive and painless direct visual inspection of a patient's whole digestive tract, but at the price of long time reviewing large amount of images by clinicians. Thus, an automatic computer-aided technique to reduce the burden of physicians is highly demanded. In this paper, we propose a novel color feature extraction method to discriminate the bleeding frames from the normal ones, with further localization of the bleeding regions. Our proposal is based on a twofold system. First, we make full use of the color information of WCE images and utilize K-means clustering method on the pixel represented images to obtain the cluster centers, with which we characterize WCE images as words-based color histograms. Then, we judge the status of a WCE frame by applying the support vector machine (SVM) and K-nearest neighbor methods. Comprehensive experimental results reveal that the best classification performance is obtained with YCbCr color space, cluster number 80 and the SVM. The achieved classification performance reaches 95.75% in accuracy, 0.9771 for AUC, validating that the proposed scheme provides an exciting performance for bleeding classification. Second, we propose a two-stage saliency map extraction method to highlight bleeding regions, where the first-stage saliency map is created by means of different color channels mixer and the second-stage saliency map is obtained from the visual contrast. Followed by an appropriate fusion strategy and threshold, we localize the bleeding areas. Quantitative as well as qualitative results show that our methods could differentiate the bleeding areas from neighborhoods correctly.

Multirobot comanipulation shows great potential in surpassing the limitations of single-robot manipulation in complicated tasks such as robotic surgeries. However, a dynamic multirobot setup in unstructured environments poses great uncertainties in robot configurations. Therefore, the coordination relationships between the end-effectors and other devices, such as cameras (hand–eye calibration) and tools (tool–flange calibration), as well as the relationships among the base frames (robot–robot calibration) have to be determined timely to enable accurate robotic cooperation for the constantly changing configuration of the systems. We formulated the problem of hand–eye, tool–flange, and robot–robot calibration to a matrix equation$\mathbf{AXB=YCZ}$. A series of generic geometric properties and lemmas were presented, leading to the derivation of the final simultaneous algorithm. In addition to the accurate iterative solution, a closed-form solution was also introduced based on quaternions to give an initial value. To show the feasibility and superiority of the simultaneous method, two nonsimultaneous methods were compared through thorough simulations under various robot movements and noise levels. Comprehensive experiments on real robots were also performed to further validate the proposed methods. The comparison results from both simulations and experiments demonstrated the superior accuracy and efficiency of the proposed simultaneous calibration method.

Wireless capsule endoscopy (WCE) is a revolutionary imaging technique that enables direct inspection of the gastrointestinal tract in a non-invasive way. However, viewing the large amounts of images is a very time-consuming and labor intensive task for clinicians. In this paper, we propose an automatic bleeding detection method in the WCE images. We propose a two-stage saliency map extraction method to highlight bleeding regions where the first-stage saliency map is created by means of different color channels mixer and the second-stage saliency map is obtained from the visual contrast in the RGB color space. Followed by an appropriate fusion strategy and threshold, we localize the bleeding areas in the WCE images. Then we extract statistic color features in the corresponding saliency region and non-saliency region respectively and fuse them together to represent the whole WCE images. Finally Support Vector Machine (SVM) is applied to carry out the experiment on 800 sample WCE images. Experiment result achieves an accuracy of 95.89%, sensitivity of 98.77% and specificity of 93.45%. This inspiring result demonstrates that the proposed method is very effective in detecting bleeding patterns in the WCE images. Our comparison studies with several state-of-the-art bleeding detection methods confirm that the proposed method achieves much better results than those of the alternative techniques.

In this paper, two gesture design methods for the humanoid robot NAO are proposed and compared. The first method is mapping human motions to the robot by an RGB-D sensor and kinematic modeling. The second method is based on hand-puppeteering. Thirteen subjects are recruited to design a forearm waving gesture for a NAO robot by the two methods. The obtained two groups of forearm waving gestures are then compared by another sixteen subjects. Our experimental results indicate that the forearm waving gestures obtained from the hand-puppeteering method are slower and have smaller range of motion than those obtained from the motion mapping method. Besides, people tend to perceive the forearm waving gestures obtained from the hand-puppeteering method as more likeable and as conveying the greeting message better than those obtained from the motion mapping method. This work contributes to a better understanding of the nature of the two gesture design methods and offers instructive reference for robot behavior designers on design method choosing.

Cloud technology elevates the potential of robotics with which robots possessing various capabilities and resources may share data and combine new skills through cooperation. With multiple robots, a cloud robotic system enables intensive and complicated tasks to be carried out in an optimal and cooperative manner. Multisensor data retrieval (MSDR) is one of the key fundamental tasks to share the resources. Having attracted wide attention, MSDR is facing severe technical challenges. For example, MSDR is particularly difficult when cloud cluster hosts accommodate unpredictable data requests triggered by multiple robots operating in parallel. In these cases, near real-time responses are essential while addressing the problem of the synchronization of multisensor data simultaneously. In this paper, we present a framework targeting near real-time MSDR, which grants asynchronous access to the cloud from the robots. We propose a market-based management strategy for efficient data retrieval. It is validated by assessing several quality-of-service (QoS) criteria, with emphasis on facilitating data retrieval in near real-time. Experimental results indicate that the MSDR framework is able to achieve excellent performance under the proposed management strategy in typical cloud robotic scenarios.

Optical means of instrument tracking has been widely used in image-guided interventions and considered the de facto standard for tracking rigid bodies with a direct line-of-sight. However, the occlusion problem which remains unresolved in current systems frustrates surgeons during the operation. To address this challenge, we propose a surgical instrument tracking system based on multiple reconfigurable monocular modules. The main approach is to enable the system to dynamically reconfigure the multiple monocular modules when occlusion occurs partially within the workspace. In this paper, we focus on the system architecture and an agile multicamera calibration method which only uses the customized tool for the surgical instrument tracking scenario. Additionally, two fast non-iterative algorithms are proposed and studied. In order to show the feasibility and superiority of the corresponding multicamera calibration algorithm, comparison experiments have carried out. The intensive investigation results give a practical instruction to the real implementation of the proposed system in image-guided interventions.

Ulcer is one of the most common symptoms of many serious diseases in the human digestive tract. Especially for the ulcers in the small bowel where other procedures cannot adequately visualize, wireless capsule endoscopy (WCE) is increasingly being used in the diagnosis and clinical management. Because WCE generates large amount of images from the whole process of inspection, computer-aided detection of ulcer is considered an indispensable relief to clinicians. In this paper, a two-staged fully automated computer-aided detection system is proposed to detect ulcer from WCE images. In the first stage, we propose an effective saliency detection method based on multi-level superpixel representation to outline the ulcer candidates. To find the perceptually and semantically meaningful salient regions, we first segment the image into multi-level superpixel segmentations. Each level corresponds to different initial region sizes of the superpixels. Then we evaluate the corresponding saliency according to the color and texture features in superpixel region of each level. In the end, we fuse the saliency maps from all levels together to obtain the final saliency map. In the second stage, we apply the obtained saliency map to better encode the image features for the ulcer image recognition tasks. Because the ulcer mainly corresponds to the saliency region, we propose a saliency max-pooling method integrated with the Locality-constrained Linear Coding (LLC) method to characterize the images. Experiment results achieve promising 92.65% accuracy and 94.12% sensitivity, validating the effectiveness of the proposed method. Moreover, the comparison results show that our detection system outperforms the state-of-the-art methods on the ulcer classification task.

The performance of robotic object manipulation relies heavily on the selection of object model. In this article, we develop a multilevel part-based object model by applying latent support vector machine to training a hierarchical object structure. We implement our method with a robot arm and a depth sensor in Robot Operating System, and then we compare the recognition performance of this model with established methods on a point cloud data set and show the manipulation performance of our model on three practical tasks. The result demonstrates that our robot recognizes and manipulates objects more accurately with this multilevel part-based object model.

In order to share information in the cloud for multi-robot systems, efficient data transmission is essential for real-time operations such as coordinated robotic missions. As a limited resource, bandwidth is ubiquitously required by applications among physical multi-robot systems. In this paper, we proposed a hierarchical auction-based mechanism, namely LQM (Link Quality Matrix)-auction. It consists of multiple procedures, such as hierarchical auction, proxy scheduling. Note that the proposed method is designed for real-time resource retrieval for physical multi-robot systems, instead of simulated virtual agents. We validate the proposed mechanism through real-time experiments. The results show that LQM-auction is suitable for scheduling a group of robots, leading to optimized performance for resource retrieval.

Wireless capsule endoscopy (WCE) enables non-invasive visual inspection of the patients' digestive tract. However, the huge number of images from the WCE has been a hurdle for doctors to handle and thus it is urgent to develop computer-aided diagnosis systems to identify problematic images. To tackle this problem, an innovative algorithm based on the integration of the Bag of Features (BoF) method and the saliency map is proposed to detect polyps from the WCE images in this study. The algorithm constitutes of four steps. In the first step, by applying the BoF method, the visual words of all images are calculated by inputting the extracted Scale Invariant Feature Transformation (SIFT) feature vectors to the K-means clustering procedure. Then we calculate the saliency and non-saliency maps of the WCE images. Following that, the histogram of the visual words of each image is calculated by integrating histograms in both saliency and non-saliency maps with various weights to represent the WCE image. Finally, polyp classification of the WCE images is conducted by Support Vector Machine (SVM) classifier. Experiments on 436 polyp images and 436 normal images are carried out to validate the proposed algorithm. The proposed method with the weight 0.9 on the saliency region achieves a best polyp detection accuracy of 92%, sensitivity of 87.9% and specificity of 93%, demonstrating that the proposed method provides a good characterization and description for polyp classification.

Tasks that are too hard for single robot can be easily carried out by multiple robots in a cooperative manner. If some/all robots have mobile bases, the cooperation is subjected to great uncertainties in both the robotic system and environment. Therefore, the relationships among all the base frames (robot-robot calibration) and the relationships between the end-effectors and the other devices such as cameras and tools (hand-eye and tool-flange calibrations) have to be calculated to enable the robots to cooperate. To address these challenges, in this paper, we propose a simultaneous hand-eye, tool-flange and robot-robot calibration method. Thorough simulations are conducted to show the superiority of the proposed simultaneous method under different noise levels and various numbers of robot movements. Furthermore, the comparison to two non-simultaneous calibration methods has also been carried out to show the efficiency and robustness of the proposed simultaneous method.

Wireless capsule endoscopy (WCE) has been widely used in hospitals in the last few years due to its advantage of non-invasive and painless nature. However, this new technology produces about 55,000 images for each patient and poses a great burden on the professional clinicians to review these images, thus an automatic computer-aided diagnosis technique is in high demand. In this paper, we propose a new feature integrating the Gabor filter and Monogenic-Local Binary Pattern (M-LBP) methods in color components for polyp detection. The new feature not only can represent shape and edge information under multi-resolution, but also preserve color information. The proposed method is composed of the following steps: the first step is to transform the original WCE images into different color space and extract the corresponding Gabor responses of the color components. Next the M_LBP descriptors applied on the resulting Gabor responses are concatenated together to characterize the images. Finally we apply Linear Discriminant Analysis (LDA) to reduce feature dimensions and conduct experiments with the Support Vector Machine (SVM) classifier on a set of images containing 436 polyp images and 436 normal images. The experimental results achieved an encouraging polyp detection accuracy of 91.43%, showing that the new feature provides a good characterization and description of the WCE images for polyp classification tasks. To compare the performance of the proposed method, several traditional features have been considered and the proposed method has surpassed the alternative techniques significantly.

Long term tracking is a challenging task for many applications. In this paper, we propose a novel tracking approach that can adapt various appearance changes such as illumination, motion, and occlusions, and owns the ability of robust reacquisition after drifting. We utilize a condensation-based method with an online support vector machine as a reliable observation model to realize adaptive tracking. To redetect the target when drifting, a cascade detector based on random ferns is proposed. It can detect the target robustly in real time. After redetection, we also come up with a new refinement strategy to improve the tracker's performance by removing the support vectors corresponding to possible wrong updates by a matching template. Extensive comparison experiments on typical and challenging benchmark dataset illustrate a robust and encouraging performance of the proposed approach.

Data association is a fundamental problem in multisensor fusion, tracking, and localization. The joint compatibility test is commonly regarded as the true solution to the problem. However, traditional joint compatibility tests are computationally expensive, are sensitive to linearization errors, and require the knowledge of the full covariance matrix of state variables. The paper proposes a posterior-based joint compatibility test scheme to conquer the three problems mentioned above. The posterior-based test naturally separates the test of state variables from the test of observations. Therefore, through the introduction of the robot movement and proper approximation, the joint test process is sequentialized to the sum of individual tests; therefore, the test has$O(n)$complexity (compared with$O(n^{2})$for traditional tests), where$n$denotes the total number of related observations. At the same time, the sequentialized test neither requires the knowledge to the full covariance matrix of state variables nor is sensitive to linearization errors caused by poor pose estimates. The paper also shows how to apply the proposed method to various simultaneous localization and mapping (SLAM) algorithms. Theoretical analysis and experiments on both simulated data and popular datasets show the proposed method outperforms some classical algorithms, including sequential compatibility nearest neighbor (SCNN), random sample consensus (RANSAC), and joint compatibility branch and bound (JCBB), on precision, efficiency, and robustness.

Wireless capsule endoscopy (WCE) can directly take digital images in the gastrointestinal tract of a patient. It has opened a new chapter in small intestine examination. However, a major problem associated with this technology is that too many images need to be manually examined by clinicians. Currently, there is no standard for capsule endoscopy image interpretation and classification. Most state-of-the-art CAD methods often suffer from poor performance, high computational cost, or multiple empirical thresholds. In this paper, a new method for rapid bleeding detection in the WCE video is proposed. We group pixels through superpixel segmentation to reduce the computational complexity while maintaining high diagnostic accuracy. Feature of each superpixel is extracted using the red ratio in RGB space and fed into support vector machine for classification. Also, the influence of edge pixels has been removed in this paper. Comparative experiments show that our algorithm is superior to the existing methods in terms of sensitivity, specificity, and accuracy.

This paper introduces a novel tracking framework for robots that can adapt various appearance changes of object and also owns the ability of reacquisition after drift. Two classifiers, LaRank and Online Random Ferns, are adopted to realize this tracking algorithm. The former one maintains the adaptive tracking using a Condensation-based method with an online support vector machine (SVM) as observation model, which also provides the reliable image patch samples to detector for updating. The other one is in charge of the task of detection in order to redetect the object when the target drifts. We also present a refinement strategy to improve the tracker's performance by discarding the support vector corresponding to possible wrong updates by a matching template after re-initialization. The experiments on benchmark dataset compare our tracking method with several other state-of-the-art algorithms, demonstrating a promising performance of the proposed framework.

Wireless capsule endoscopy (WCE) is a recently developed revolutionary medical technology which records the video of human's digestive tract noninvasively. However, reviewing a WCE video is a tired and time-consuming task for clinicians. Thus, WCE video automatic segmentation methods are emerging to reduce the review time for clinicians. In our previous work, a two-level WCE video segmentation approach has been proposed, which provides a novel approach to localize the boundaries more exactly and efficiently. However, it has an unsatisfactory performance in the small intestine/large intestine boundary detection. In this paper, we propose new features and an improved classifier to improve the previous two-level segmentation algorithm. In the rough level, color feature is utilized to draw a dissimilarity curve and an approximate boundary has been obtained. At the same time, training data for fine level can be directly labeled and collected between the two approximate boundaries of organs to overcome the difficulty of training data acquisition. In the fine level, a novel color uniform local binary pattern (CULBP) algorithm is proposed, which includes two kinds of patterns, color norm patterns and color angle patterns. The CULBP feature is more robust to variation of illumination and more discriminative for classification. Moreover, in order to elevate the performance of SVM classifier we proposed the Ada-SVM classifier which using RBFSVMs as component of Adaboost classifier. At last, an analysis of classification results of the Ada-SVM classifier is carried out to segment the WCE video into several meaningful parts, stomach, small intestine and large intestine. The experiments demonstrate a promising performance of the proposed method. The average precision and recall are as high as 91.37% and 88.50% in stomach/small intestine classification, 90.35% and 97.28% in small intestine/ large intestine classification.

Correspondence searching among different images is a fundamental problem in computer vision. It is important to find correspondences correctly and rapidly, especially for real-time tracking systems. Therefore, the definition of search areas in images is crucial. Traditional epipolar constraint is not noise-enduring; some reformative methods lack explicit geometric meanings. All of them cannot help defining rational search areas under noises. This paper proposes two new binocular imaging constraints with clear geometric meanings and strong restraining forces. Based on them, a novel searching strategy among multiimages is developed which can define optimal search areas with smallest sizes but best reliability. Practical algorithms for implementation are presented and experiments with real images are performed, validating the effectiveness of the proposed strategy.

Tumor in digestive tract is a common disease and wireless capsule endoscopy (WCE) is a relatively new technology to examine diseases for digestive tract especially for small intestine. This paper addresses the problem of automatic recognition of tumor for WCE images. Candidate color texture feature that integrates uniform local binary pattern and wavelet is proposed to characterize WCE images. The proposed features are invariant to illumination change and describe multiresolution characteristics of WCE images. Two feature selection approaches based on support vector machine, sequential forward floating selection and recursive feature elimination, are further employed to refine the proposed features for improving the detection accuracy. Extensive experiments validate that the proposed computer-aided diagnosis system achieves a promising tumor recognition accuracy of 92.4% in WCE images on our collected data.

Wireless Capsule Endoscopy (WCE) is a novel technique that allows visualization of the whole gastrointestinal (GI) tract especially the small intestine in a comfortable, non-invasive and efficacious way. The main disadvantage of WCE is that physicians need to examine a video of over 55,000 frames which is a time-consuming and labor intensive task. To address the problem, a strategy of WCE video clip abstraction based on linear discriminant analysis (LDA) is proposed in this paper. We extract multiple features based on which the frame differences are measured. Then the video clips are segmented using a non-parametric key-point detection algorithm and finally the most representative frames (MRFs) are extracted based on LDA algorithm. Experimental results demonstrate the proposed strategy achieves promising performances.

The direct view of the inner tract of the small intestine is not feasible until a recently revolutionary imaging technology, wireless capsule endoscopy (WCE), appeared in 2001. However, interpretation of the produced video data for the digestive tract on each patient is left to naked eyes of medical staffs. Such a process is very tedious and time-consuming with the average inspection time about two hours for a whole WCE video. To overcome this big problem, automatic WCE video analysis is required. In this paper, we propose a comparative study of several image based features that may be suitable for WCE video abstract, which may be a good candidate to reduce the burden of physicians. Color, texture and motion features built from images are investigated and compared to show their performance in representing video content for a WCE video abstract. Preliminary experimental results of these features for WCE video abstract are also demonstrated and discussed. It is found that textural and motion features may be suitable candidates for visual frame depiction for WCE video abstract in terms of visual content representation and compression ratio. Clinical validation of our work remains to be implemented in the near future.

In this paper, we propose a new scheme aimed for gastrointestinal (GI) tumor capsule endoscopy (CE) images classification, which utilizes sequential forward floating selection (SFFS) together with support vector machine (SVM). To achieve this goal, candidate features related to texture characteristics of CE images are extracted. With these candidate features, SFFS based on SVM is applied to select the most discriminative features that can separate normal CE images from tumor CE images. Comprehensive experiments on our present CE image data verify that it is promising to employ the proposed scheme to recognize tumor CE images.

In order to assess the compliance of ingestible wireless device (IWD) within safety guidelines, the SAR, and near fields of IWD in two realistic human body models, whose dielectric values are increased from the original by +/-10% and +/- 20% are studied using the finite-difference time-domain method. The radiation characteristics of the IWD in the human body models with changed and unchanged dielectric values are compared. Simulations are carried out at 13 scenarios where the IWD is placed at center positions of abdomens in the two models at the operation frequency of 430, 800, and 1200 MHz, respectively. Results show that variation of radiation intensity near the surface of abdomen is around 2.5, 2.6, and 3.5 dB within 20% variation of dielectric values corresponding to the frequency of 430, 800, and 1200 MHz, respectively. Electric fields in the anterior of the human body models are higher than those in the posterior for all scenarios. SAR values increase with the increase of conductivities of human body tissues, and usually decrease with the increase of relative permittivities of human body tissues. The effect of the dielectric values of human body on SAR is orientation-, human-body-, and frequency-dependent. A variation up to 20% in conductivities and relative permittivities alone or simultaneously always causes a SAR variation less than 10%, 20%, and 30% at the frequency of 430, 800, and 1200 MHz, respectively. As far as the compliance of safety was concerned, the IWD was safe to be used at the input power less than 12.6, 9.3, and 8.4 mW, according to the IEEE safety standards at the frequency of 430, 800, and 1200 MHz, respectively.

Node localization is a challenging problem in wireless sensor networks, especially in the scenarios of tuning multiple transmit-powers. In this article, we utilized particle filter to infer static node position from the correlations between radio frequency (RF) received signal strength indication (RSSI) and distance under multiple power settings. The RSSI based stochastic measurement model was analyzed and followed by the particle filter design. The simulation results verified the performance of proposed algorithm for localization. The proposed method is contributive in terms of making advantages of multiple transmit power for localization.

Wireless capsule endoscopy (WCE) has been gradually applied in hospitals due to its great advantage that it can directly view the entire small bowel in human body compared with traditional endoscopies and other imaging techniques for gastrointestinal diseases. However, a challenging problem with this new technology is that too many images produced by WCE causes a tough task to doctors, so it is very significant to help and relief the clinicians if we can develop computer based automatic detection system to prescreen the collected large amount of images and identify the images with potential problems. In this paper, we propose a new scheme aimed for small bowel tumor detection of WCE images. This new scheme utilizes texture feature, also a powerful clue used by physicians, to detect tumor images with support vector machine. We put forward a new idea of wavelet based local binary pattern as the textural features to discriminate tumor regions from normal regions, which take advantage of wavelet transform and uniform local binary pattern. With support vector machine as the classifier, three-fold cross validation experiments on our present image data verify that it is promising to employ the proposed texture features to recognize the small bowel tumor regions.

Capsule endoscopy has been proved efficient in examining the small intestine. A lot of work has been devoted to the study of capsule videos to assist the diagnosis. Different from previous approaches which worked off-line and couldn't be applied in active capsule controls, in this paper we proposed an in situ capsule video analysis method, which operates in real-time and provides the first event detection-based scheme for capsule techniques. Specifically, first we established a theoretical computing framework. The method addresses two key points: one is to merge the surgeon's expertise into the system, and the other is to identify unusual events. Then we evaluated the method by some empirical experiments. The preliminary results verified the usefulness of the method.

Wireless Heterogeneous Sensor Network (WHSN) facilitates ubiquitous information acquisition for Ambient Intelligence (AmI) systems. It is of great importance of power management and topology control for WHSN to achieve desirable network performances, such as clustering properties, connectivity and power efficiency. This paper proposes a game theoretic model of topology control to analyze the decentralized interactions among heterogeneous sensors. We study the utility function for nodes to achieve desirable frame success rate and node degree, while minimizing the power consumption. Specifically, we propose a static complete-information game formulation for power scheduling and then prove the existence of the Nash equilibrium with simultaneous move. Because the heterogeneous sensors typically react to neighboring environment based on local information and the states of sensors are evolving over time, the power-scheduling problem in WHSN is further formulated into a more realistic incomplete-information dynamic game model with sequential move. We then analyze the separating equilibrium, one of the perfect Bayesian equilibriums resulted from the dynamic game, with the sensors revealing their operational states from their actions. The sufficient and necessary conditions for the existence of separating equilibrium are derived for the dynamic Bayesian game, which provide theoretical basis to the proposed power scheduling algorithms, NEPow and BEPow. The primary contributions of this paper include applying game theory to analyze the distributed decision-making process of individual sensor nodes and to analyze the desirable utilities of heterogeneous sensor nodes. Simulations are presented to validate the proposed algorithms and the results show their ability of maintaining reliable connectivity, reducing power consumption, while achieving desirable network performances.

The traditional model reference adaptive system (MRAS) is of parallel configuration. By representing the reference model as an equivalent closed-loop subsystem, these two subsystems can share a controller virtually. An embedded structure of MRAS is thereby presented. In this paper, two approaches are developed for designing the adaptive laws. The adaptive law based on Lyapunov stability theory is designed for low-order systems. The adaptive law based on the recursive least-square (RLS) algorithm is suitable for both low- and high-order systems. The RLS-based adaptive law provides a quick convergence rate for the controller adaption. MRAS with RLS-based adaptation law performs better in trajectory tracking and model following. Simulation results show the effectiveness of the proposed method.

The actuation and guidance mechanism is necessary for next-generation capsule endoscope to carry out active examinations. We studied an actuation and guidance mechanism using magnetic method for a wireless active capsule endoscope. A small permanent magnet is enclosed inside the capsule as a seed, and an external magnetic field guides the seed through the small intestine. In this paper, optimization methods are discussed to calculate the economical control currents, so as to reduce the power consumption.We divided the working space into grids and used a filament-based algorithm to compute the magnetic field at grid centers, which proved to be fast and accurate. Using the pre-stored field data and an interpolation method, it is possible to achieve real-time control of the magnetic field at arbitrary points.

The wireless capsule endoscopy (WCE) has been widely used to detect the diseases in gastrointestinal tract. However, the contrast of many images it produced is rather dark due to some reasons, which causes some difficulties to diagnosis and computer aided diagnosis. To overcome this shortcoming, we propose a forward and backward anisotropic diffusion method based on the contrast space to enhance the capsule endoscopy images. Experimental results show that this new method can provide a better visualization of the wireless capsule endoscopy images than the forward and backward Beltrami flow and the contrast-limited adaptive histogram equalization so as to assist the inspection of WCE images.

This paper presents a new control approach with prediction to minimize the effects of time delays while ensuring stability and system performance. Two predictors at the slave and master sides are constructed assuming that the time delays in both transmission channels are measurable. Simulation and experimental results are compared with the scheme without prediction to show the effectiveness of this approach. The influence of data dropout to the proposed teleoperation system is studied in the experiment.

This paper studies robust performance analysis of H2/Hinfincontrol problem in time-varying systems. In the case where the state-space matrices of the system depend affinely on the uncertain parameters, it is known that recently developed extended or dilated linear matrix inequalities (LMIs) are effective to assess the robust performance in a less conservative fashion. This paper further probes into those preceding results and proposes a new form of extended LMIs for time-varying H2/Hinfincontrollers synthesis. The new method enables us to parameterize controllers without involving the Lyapunov variables in the parameterization. The numerical simulations prove the validity of this framework.

Salmonellais an important infectious disease of livestock and humans. Reported cases that humans caused bySalmonellahave remained high in the world over the past few years. The conventional susceptible-infected-recovery (SIR) models for outbreak analysis of infectious diseases divides the whole population into three groups, a susceptible group, an infected group and a recovery group. In this paper, a new model for the spread ofSalmonellais proposed by introducing the degree of risk for the closed population. Unlike the SIR models that only consider the disease transmission among humans, the proposed model with degree of risk also includes and emphasizes both transmission rates from the environment to humans and person to person. In addition, considering the seasonal effect, an improved model based on degree of risk is proposed, where the transmission rate among humans and the degree of risk can be quantified as a function of temperature. The model perdition of the number of infections from 2000 to 2002 is generally in good agreement with the actual observed datasets. Simulation studies show that the proposed models can deal with the pathogen infections likeSalmonellawith a low transmission rate and a seasonal feature.

Optical flow is the apparent motion of the brightness pattern in an image. It generally corresponds to the motion field of the captured scene in the image so that we can use it distinguish moving objects. The computation of optical flow is conventionally based on the assumption of uniform brightness, which however does not always hold. Numerous algorithms have been proposed to improve the computation precision of optical flow. In this paper, we propose to incorporate optical flow information into a novel geodesic active contour model for moving-object detection in monocular robots. Specifically, an active contour is formulated by using the level set method, which eliminates the need of a re-initialization procedure. The developed scheme alleviates the effect of optical flow noise, increasing the robustness of the perception of moving objects. The experimental results show that our algorithm can successfully track a moving target, e.g., a human being

A magnetic localization and orientation system is proposed for tracking wireless capsule endoscope. This system uses a small magnet enclosed in the capsule to serve as excitation source. When the capsule moves, the magnet establishes a static magnetic field around. With the magnetic sensor array composed of Honeywell 3-axis magnetic sensors, HMC1053, the magnetic intensities in some pre-determined spatial points can be detected, and the magnet's position and orientation parameters can be computed based on an algorithm. To initiate the system and obtain better tracking accuracy, we propose a calibration technique for the magnetic tracking system. The calibration includes sensitivity determination and nonlinearity adjustment, sensor center position and orientation adjustment. Based on the calibration procedures, the system can achieve satisfactory tracking accuracy with the average localization error 3.3 mm and the average orientation error 3.0deg

A mobile robot that is served for training the fencing athletes is introduced. The pilot and sensing units are designed and set on the robot at particular positions. To ensure a fast and smooth motion when starting, running and braking the robot, the driver unit is carefully designed on the base of analysing the torque-frequency curve of the step motor. This fencing training robot not only can used to train an athlete's each single special ability such as stepping, hitting accuracy, and response, but also can used to train the combined ability. At the same time, it can give a quantitative evaluation of training effect to guide the next training plan. The robot has been used in practice

A rectangular cell map representation and a neural dynamics based technique are proposed for real-time map building and complete coverage navigation (CCN) of a cleaning robot. The proposed model is compared with a triangular cell map based method proposed by Oh et al. (2004), which combined distance transform path planning, wall-following algorithm, and template based technique. Our method does not need any templates, even in unknown environments. A local map composed of rectangular cells is built through the proposed neural dynamics during CCN with restricted sensory information in unknown environments. The robot is able to dynamically build an accurate map of its immediate limited surroundings for its navigation. Comparison studies to triangular cell map based CCN approach show that the proposed model is capable of planning more reasonable and shorter coverage path in unknown environments

More and more wireless mobile sensor networks are employed by robotics to perform harsh tasks such as disaster rescue, emission sources localization or hazardous contaminants localization. There are a lot of network optimization problems to be solved in the protocol design of wireless mobile sensor networks (WMSN), such as rate control, flow control, congestion control, medium access control, queue management, power control and topology control etc. These issues involve several layers of the network protocol stack so that it's quite difficult to consider every single optimization problem of them in a holistic view. The majority of contemporary research works mainly deal with one or some of them in terms of certain applications or objectives. However, most of the proposed protocols are based on simulations or experiments which lack of sufficient mathematical or theoretical analysis to fully understand the convergence or stability. In order to study the theoretical basis of network algorithms, this paper briefly outlines the current methodologies exploited to design and optimize the performance of sensor networks. In addition, the paper investigates the theoretical aspects to make sense of the network optimization algorithms and give a survey mainly in terms of convex optimization, game theory and artificial intelligence

In this paper, a wireless sensor network based assistive system (WASN) is developed to help the deaf or the hearing-impaired people, particularly, help them to be aware of their environments. A brief review on the current assistive devices is also addressed first. The system architecture, components and specifications are presented for two typical application scenarios: smart home and smart school playground. A node platform is developed to implement the system. Finally, the system performance is evaluated by simulations, followed by the analysis of the feasibility and availability

Locomotion mechanism is a necessity for development of an active capsule endoscope. In this paper, we propose an inchworm-like locomotion mechanism, which is composed of a permanent magnet, a plunger, a solenoid and a cone-shaped polymer. When an alternating current is applied to the solenoid, the magnet will make a reciprocating movement under the electromagnetic force. With the friction difference produced by the polymer, the actuator could propel itself forward on a 2D surface. The working principle and kinetic model of the mechanism are analyzed. A prototype with a size of Phi12*37 mm were fabricated to study the feasibility of the mechanism. Experiments were carried out on the surfaces of various frictional coefficients. The results show that the actuator works more efficiently on the surface of the glass than on the paper, because the polymer has better clamping effect on the former. The velocity changes with the frequency and amplitude of the current. The mechanism consumes a low power and it can work under a relatively low voltage (less than 20 V). It has the potential for actuation of an active capsule endoscope

During the course of physiological information monitoring by wireless biomedical sensor networks, adverse biological effects will be caused by wireless radio frequency radiation, especially for long term, intensive and close inspection. This paper is concerned with bioeffect metric to evaluate the performance of wireless biosensor networks in terms of health effect consideration, and a price-based rate control algorithm to reduce the bioeffect. This paper first investigates the bioeffects caused by radiofrequency transmission of sensor node, including thermal effects and athermal effects. The bioeffects model is studied in both near-field and far-field, in relation to specific absorption rate (SAR). The main contribution of this work is that a normalized bioeffect metric, equivalent coefficient-of-absorption-and-bioeffects (CAB), is derived to evaluate and design the communication protocols for wireless biosensor networks. After identifying the factors that can reduce the adverse health effects in the communication system we present a bioeffect aware rate control algorithm for the system

Wireless sensor networks have been employed to enhance the healthcare system recently. During the course of physiological information monitoring by wireless biomedical sensor networks, adverse biological effects will be caused by wireless radio frequency radiation, especially for long term, intensive and close inspection. This paper is concerned with bioeffects metric to evaluate the performance of wireless biosensor networks in terms of health effect consideration, and a price-based rate control algorithm to reduce the bioeffects. This paper first investigates the bioeffects caused by radiofrequency transmission of sensor node, including thermal effects and athermal effects. The bioeffects model is studied in both near-field and far-field, in relation to specific absorption rate (SAR). The main contribution of this work is that a normalized bioeffects metric, equivalent coefficient-of-absorption-and-bioeffects (CAB), is derived to evaluate and design the communication protocols for wireless biosensor networks. After identifying the factors that can reduce the adverse health effects in the communication system, we use the proposed bioeffects metric to evaluate the performance of power control algorithm compared with the one without power control. Finally, we present a bioeffects aware rate control algorithm for the system

Area-covering operation is a special kind of path planning, which requires the robot path to cover every part of the workspace. In this paper, a neural dynamics based algorithm is proposed for real-time map building and area-covering operations. A local map composed of squared cells is built through the proposed neural dynamics during area-covering operations with limited sensory information in unknown environments. The robot is able to dynamically build an accurate map of its immediate limited surroundings for its navigation. The dynamics of each neuron in the topologically organized neural network is characterized by a shunting equation derived from Hodgkin and Huxley’s membrane equation. The robot can only sense a limited measurable range and the obtained sensory information are used for its navigation. The feasibility of the proposed algorithm is validated by simulation studies on cases under unknown environments.

This paper studies the problem of improving the fidelity of bilateral teleoperators under the stability constraint. First, a new fidelity measure for teleoperators is proposed. Then the teleoperator design problem is formulated as the optimization problem of maximizing the proposed fidelity measure under the stability constraint. The robust stability theory for unstructured uncertainty is applied to analyze the stability of bilateral teleoperators. Third, a new scheme to estimate adaptively the environmental impedance is developed to improve the fidelity of the systems. Finally, a case study is presented to validate the introduced approach.

To build a new wireless robotic capsule endoscope with external guidance for controllable and interactive GI tract examination, a sensing system is needed for tracking 3D location and 2D orientation of the capsule movement. An appropriate sensing approach is to enclose a small permanent magnet in the capsule. The magnet establishes a magnetic field around the patient's body. With the sensing data of magnetic sensor array outside the patient's body, the 3D location and 2D orientation of the capsule can be calculated. Higher localization and orientation accuracy can be obtained if more sensors and proper optimization algorithm are applied. In this paper, different nonlinear optimization algorithms are evaluated, and we have found that Levenberg-Marquardt method provides higher accuracy and faster speed. Simulations were done for investigating the de-noise ability of this algorithm based on different sensor arrays. Furthermore, the real experiment shows that the results are satisfactory with high accuracy.

When operating robots through the Internet, time delay and stability are two crucial issues. In this paper, we suggest that sound signal can be used to relay navigation information to the operator to substitute other sensory information such as two-dimensional vision and multidimensional force feedback. The rationale for using an acoustic interface is that it utilizes significantly less bandwidth than visual information, and it has higher stability compared with force feedback. A four-section psychological experiment was devised to investigate human teleoperation performance under different sensory feedback and evaluate the effectiveness of acoustic information. The preliminary tests results show that the human operator can sufficiently perceive the remote environment through audio feedback for remote teleoperation tasks.

To obtain shear and normal stress information on non-planar surfaces has long been a significant challenge. This paper described a new version of flexible tactile sensor skin for three-dimensional force measurement. The sensor array has been fabricated by MEMS technology. The sensor skin which can be bended 90/spl deg/ includes 4/spl times/4 force sensor cells. Each cell is hybrid integrated to flexible printing circuit board which consists of a E-shape diaphragm (50 /spl mu/m thick and 4000 /spl mu/m wide). Each cell exhibits a sensitivity of 228 mv/N to normal force and 34 mv/N to shear force in the designed force range of 2N. Design analysis, fabrication processes, and experimental results are presented in this paper.

The images captured by wireless capsule endoscope might have nonlinear spatial distortion, which makes accurate medical examination difficult. So it is a prerequisite to have this distortion corrected. Typically, the correction uses a calibration pattern, which might be a chessboard, dot, grid, or circle pattern. Based on this pattern, enough characteristic samples can be extracted accurately and conveniently, and mathematic model can be built for the distortion in the captured image with respect to the original calibration pattern. Then the correction parameters, including image centers and mapping polynomials, could be found to realize the correction. If the model is too complicated to be accurately built, correction using neural network is a good choice, since it does not rely on the mathematic model of the distortion.

Robotic target tracking has been used in a variety of applications. Due to limited sampling rate, sensory characteristics and processing delays, an important issue in such systems is thus to extrapolate ahead the trajectory (position, orientation, velocity and/or acceleration) of moving targets based on past observations. This paper introduces a novel on-line data-driven fuzzy clustering algorithm that is based on the maximum entropy principle for this particular task. In this algorithm, the fuzzy inference mechanism is extracted automatically from observed data without any human help, which thus eliminates the necessity of expert knowledge and a priori information on moving targets, as required by most traditional techniques. This algorithm does not require training, which enables it to work in a completely on-line fashion. Another important and distinct advantage of the algorithm exists in the fact that it is very fast and efficient in terms of computational cost and thus can be implemented in real time. In the mean time, the introduced algorithm has the ability to adapt quickly to the dynamics of moving targets. All these features make it especially suitable for the task to predict the trajectory of moving targets in robotic tracking. Simulation results show the effectiveness and efficiency of the presented algorithm.

A neural network used in a direct-adaptive control scheme can achieve trajectory tracking of a (highly) flexible joint robot holding an unknown payload without need for many learning repetitions. A modification of the Lyapunov stable nonlinear control method known as backstepping with tuning functions is derived to achieve this. Specifically, the introduction of appropriate weightings of the different tuning-function terms results in high performance. Also, a robust redesign of the tuning function method is presented to account for the uniform approximation (modeling) error of the neural network. This computationally burdensome method is made practical by taking advantage of the efficient structure of the CMAC neural network. Simulations with a (highly) flexible-joint robot show immediate compensation for a payload with performance nearly recovered after five seconds.

In this paper, a novel biologically inspired approach to real-time tracking control of a nonholonomic mobile robot is proposed. The proposed algorithm incorporates a neural dynamics model derived from a biological membrane equation with the conventional full-state tracking control technique. It is capable of generating real-time smooth and continuous velocity control signals that drive the mobile robot to follow desired trajectories. The proposed approach resolves the speed jump problem existing in some previous tracking controllers. In addition, it can track both continuous and discrete paths. The practicality and effectiveness of the proposed tracking controller were demonstrated by simulation and comparison results.

In this paper, a novel biologically inspired tracking controller is proposed for real-time navigation of nonholonomic mobile robots, which is inspired by the agonist/antagonist effects in muscular sensory motor reaction of a gated dipole neural model. Through the incorporation of the biological computation element, the proposed controller is capable of generating smooth, bounded acceleration command signals for the mobile robot to track reference paths. It resolves the velocity jump problem in the conventional backstepping controllers that result from the initial tracking errors or the discontinuities in discrete paths. In addition, the proposed controller removes the unpractical assumption of "perfect velocity tracking" in some existing approaches. The effectiveness and efficiency of the proposed approach are demonstrated by simulation and comparison studies.

This paper presents a voice-controlled mobile robotic system capable of recognizing voice commands and relaying them to a mobile robot. First, the specifications of the robot used will be presented. Second, a description of the HTK toolkit, the HMM, and the VCR software developed will be discussed. Finally, comparing our HTK-developed HMM against a commercially available Microsoft speech recognition engine (SDK 5.1) in terms of accuracy. The experimental evaluation and accuracy tests will show the ability of the VCR software to control a robot with simple human voice commands. Conclusions and future work are presented towards the end of the paper.

This paper describes an experimental study of the problem of collective sorting, in which multiple robots under simple reactive control rearrange initially randomly distributed objects of different classes into separate clusters so that each cluster contains objects of only one class. We specifically focus on the roles of three key system parameters - namely, population size, local communication among robots, and sensing range - on the performance of the collective sorting task of objects of two classes. Our experiments show that an increase in communication range has a similar positive effect on task performance as an increase in sensing range, especially when the sensing range is limited; however, this effect tends to be attenuated by the increase in the size of the robot collective.

Robotic target tracking has been used in a variety of applications. Due to limited sampling rate, sensory characteristics and processing delays, an important issue in such systems is to extrapolate ahead the trajectory (position, orientation, velocity, and/or acceleration) of moving targets from past observations. This paper introduces a novel online data-driven fuzzy clustering algorithm that is based on the Maximum Entropy Principle for this particular task. In this algorithm, the fuzzy inference mechanism is extracted automatically from observed data without human help, which thus eliminates the necessity of expert's knowledge and a priori information on moving targets, as required by most traditional techniques. This algorithm does not require training, which enables it to work in a completely online fashion. Another important and distinct advantage of the algorithm exists in the fact that it is very fast and efficient in terms of computational cost and thus can be implemented in real time. In the meantime, the introduced algorithm is able to adapt quickly to the dynamics of moving targets. All these desired features make it especially suitable for the task to predict the trajectory of moving targets in robotic tracking. Simulation results show the effectiveness and efficiency of the presented algorithm.

In this paper, an embedded fuzzy controller for a nonholonomic mobile robot is developed. The mobile robot was built based on the behavior-based artificial intelligence, where several levels of competences and behaviors are implemented. A class of fuzzy control laws is formulated using the Lyapunov's direct method, which can guarantee the convergence of the steering errors. Theoretical analysis of the fuzzy control algorithms for steering control of the mobile robot is performed. The requirements for a suitable rule base selection in the proposed fuzzy controller are provided, which can guarantee the asymptotical stability of the system. Simulation and experimental studies are conducted to investigate the performance of the proposed fuzzy controller. It can achieve the desired turn angle and make the mobile robot follow the target trajectory satisfactorily.

In this paper, a fuzzy controller is developed for of an autonomous nonholonomic mobile robot, which was successfully built with behavior-based artificial intelligence that is implemented by several levels of competences and behaviors. The Lyapunov's direct method is used to formulate a class of control laws that guarantee the convergence of the steering errors to zero. Certain constraints for the control laws are presented for the selection of a suitable rule base for the fuzzy controller, which makes the system asymptotically stable. The stability of the proposed fuzzy controller is proved theoretically and demonstrated by simulation studies. Experiments are also conducted to investigate the performance of the developed fuzzy controller.

For Internet-based tele-robotic systems (Internet robots), the most challenging and distinct difficulties are associated with Internet transmission delays, delay jitter and not-guaranteed bandwidth availability, which might lead to dramatic performance degradation or even instability. In this paper, a new approach to dealing with these problems is explored and implemented. Specifically, a rate-based end-to-end transport protocol is developed for real-time data transmission and an adaptive control scheme is developed to control the robot remotely. A mobile robot teleoperation system, ArtBot-I, is developed to verify and test the solutions. In the experiments, the users successfully guided a Pioneer-2 mobile robot through a laboratory environment remotely via the Internet using a web browser.

In this paper, a neural network based torque controller is proposed for real-time collision-free navigation of nonholonomic mobile robots. A torque resulted from the obstacles is incorporated in the control design based on the artificial potential technique, which locally pushes the robot away from the obstacles to avoid collisions. All the needed environment information can be obtained from on-board robot sensors that have limited visibility range only. A torque from a simply single-layer neural network is employed to learn the completely unknown robot dynamics. The system stability is guaranteed by a Lyapunov stability theory. The real-time fine control of mobile robots is achieved through the on-line learning of the neural network. The effectiveness of the proposed controller is demonstrated by simulation studies in both static and dynamic environments.

This paper presents a new visual gesture recognition method for the human-machine interface of mobile robot teleoperation. The interface uses seven static hand gestures, each of which represents an individual control command for the motion control of the remote robot. All the important aspects to develop such an interface are explored, including image acquisition, adaptive object segmentation with color image in RGB, HLS representation, morphological filtering, hand finding and labeling, and recognition with edge codes, template matching, and skeletonizing. By choosing processing methods and procedures properly, a higher ratio of correct recognition and a faster speed are achieved from the experiments.

In this paper we introduce a software and hardware structure for on-line mobile robotic systems. The system hardware configuration mainly consists of a commercially available Pioneer 2 PeopleBot mobile robot, a Sony PTZ video camera and a pair of BreezeNet indoor wireless Ethernet adaptors. The system employs a client-server software architecture in which the client server is insulated from the lower-level details of the mobile robot. This architecture is implemented on the real Internet and the preliminary result is promising. By adopting this modular structure, it will be very easy to construct an experimental platform for the research on diverse teleoperation topics such as remote control algorithms, interface designs, network protocols and applications etc.

A neural dynamics based approach is proposed for real-time motion planning with obstacle avoidance of a mobile robot in a nonstationary environment. The dynamics of each neuron in the topologically organized neural network is characterized by a shunting equation or an additive equation. The real-time collision-free robot motion is planned through the dynamic neural activity landscape of the neural network without any learning procedures and without any local collision-checking procedures at each step of the robot movement. Therefore the model algorithm is computationally simple. There are only local connections among neurons. The computational complexity linearly depends on the neural network size. The stability of the proposed neural network system is proved by qualitative analysis and a Lyapunov stability theory. The effectiveness and efficiency of the proposed approach are demonstrated through simulation studies.

Since data packets may get lost somewhere in the Internet connections, for real-time applications such as Internet-based teleoperation, delay boundary prediction plays an important role in determining properly whether a packet is lost or not. The predictors currently employed are lowpass filters based on the autoregressive and moving average (ARMA) models. However, recent studies and the results of the experiments in this paper show that the traditional ARMA model is not suitable because sometimes delays develop with quick and evident variation. In this paper, we present a novel adaptive algorithm for delay boundary prediction based on the maximum entropy principle (MEP). The results of our 3 successive working day experiments on 9 links which consists of academic, commercial and governmental ones among Northern America, Asia and Europe show that the MEP algorithm proposed has a better performance than the traditional ARMA method.

For Internet based teleoperation, the most difficult and distinct part is the unavoidable time-varying delays between human operators and remote robotic devices. Currently, the RTTs (roundtrip time or we can call it delay) are mostly treated as given conditions in application level. In this paper, after a statistical analysis of the huge RTT time series collected densely in a few continuous days using some linear and nonlinear methods, it is found that the RTT time series has a rather high degree of linear correlation between observations. It could be inferred that there is no high nonlinear dependence among observations. Thus, RTT is linearly predictable. We use the MEP (maximum entropy principle) method developed to predict next RTT value (one step ahead) and the results confirm our findings.

This paper describes an intelligent sensor and control system, robotic prosthetic eye system, with artificial eye model, biomedical electrodes and a micro controller. The system intends to provide a rehabilitation ocular implant device that can be used by people with ocular implant. The proposed system can acquire the dynamic natural eye orientation signal, which is sent to the micro controller to control the artificial eye to have the same orientation. This paper starts with a brief review of various eye movement detection methods and then a proposed system is set up to carry out experimental study. Pilot study has demonstrated its potential for clinical applications.

A biologically inspired neural network approach is proposed for real-time collision-free path planning and tracking control of a nonholonomic mobile robot in a nonstationary environment. The real-time robot trajectory with obstacle avoidance is rated by a topologically organized neural network, where the dynamics of each neuron is characterized by a shunting equation. The varying environment is represented by the dynamic activity landscape of the neural network. Where the neural activity propagation is subject to the kinematic constraint of the nonholonomic mobile robots. The real-time tracking velocities are generated by a novel neural dynamics based controller, which is based on two shunting models and the backstepping technique. Unlike the backstepping controllers that produce non-smooth velocity commands with sharp jumps, the proposed tracking controller is capable of generating smooth, continuous commands not suffering from velocity jumps. The effectiveness and efficiency of the proposed approach are demonstrated through simulation and comparison studies.

Describes two robotic prosthetic eye prototype models. The first model uses an external infrared sensor array mounted on a frame of a pair of eyeglasses to detect natural eye movement and to feed the control system to drive the artificial eye to move with the natural eye. The second model uses human brain EOG (electrooculography) signals picked up by electrodes placed on both sides of a person's head to carry out the same eye movement detection and control tasks as mentioned above. Theoretical issues on sensor failure detection and recovery, and signal processing techniques used in sensor data fusion are studied using statistical methods and artificial neural network based techniques. In addition, practical control system design and implementation using micro controllers are studied and implemented to carry out the natural eye movement detection and artificial robotic eye control tasks.

In this paper, shunting neural networks are proposed for dynamic planning and control of robots. The dynamic environment is represented by a neural activity landscape of a neural network, where each neuron in the topologically organized neural network is characterized by a shunting equation that is derived from Hodgkin and Huxley's (1952) biological membrane equation. The collision-free path is generated in real-time from the activity landscape without any explicit searching procedures and without any prior knowledge of the dynamic environment. The real-time tracking control of robots to follow the planned dynamic path is designed using shunting equation as well. The effectiveness and efficiency of the proposed approach are demonstrated through simulation and comparison studies. Simulation in several computer-synthesized virtual environments further demonstrates the advantages of the proposed approach with encouraging experimental results.

In this paper, dynamic collision-free trajectory generation in a nonstationary environment is studied using biologically inspired neural network approaches. The proposed neural network is topologically organized, where the dynamics of each neuron is characterized by a shunting equation or an additive equation. The state space of the neural network can be either the Cartesian workspace or the joint space of multi-joint robot manipulators. There are only local lateral connections among neurons. The real-time optimal trajectory is generated through the dynamic activity landscape of the neural network without explicitly searching over the free space nor the collision paths, without explicitly optimizing any global cost functions, without any prior knowledge of the dynamic environment, and without any learning procedures. Therefore the model algorithm is computationally efficient. The stability of the neural network system is guaranteed by the existence of a Lyapunov function candidate. In addition, this model is not very sensitive to the model parameters. Several model variations are presented and the differences are discussed. As examples, the proposed models are applied to generate collision-free trajectories for a mobile robot to solve a maze-type of problem, to avoid concave U-shaped obstacles, to track a moving target and at the same to avoid varying obstacles, and to generate a trajectory for a two-link planar robot with two targets. The effectiveness and efficiency of the proposed approaches are demonstrated through simulation and comparison studies.

This paper presents an approach of using FIR Median Hybrid Filters for analysis of eye tracking movements. The proposed filter can remove the eye blink artifact from the eye movement signal. The background of the project is described first. The whole idea is to put movements into eyes, which are used as static prosthesis, so that the ocular implant will have the same natural movement as the real eye. First step is to obtain the movement of the real eye. From the review of the eye movement methods, the electro-oculogram (EOG) is used to determine the eye position. Because the eye blink artifact is always corrupted in the EOG signal, it must be filtered out for the purpose of our project. The FIR Median Hybrid Filter is studied in the paper; its properties are explored with examples. Finally the filter is used to deal the real eye blink corrupted EOG signal. Examples are given of analysis procedure for eye tracking or a random moving target. The method is proved to be highly reliable.

As the population-aging problem is increasingly pressing on society, and the associated healthcare costs are taking up an incremental percentage of the GNP, various inexpensive support systems for elderly people staying alone at home are becoming very demanding. Fortunately, as the Internet continues to expand exponentially and accesses to the Internet become more prevalent in our daily life, home healthcare systems based on teleoperated mobile robot platform via the Internet become feasible. In this paper, we describe a feasibility study and a basic platform design of a teleoperated home healthcare system via the Internet, which include a literature review of the state of the art in research on this topic, discussions on some open problems and challenges facing researchers in this area and a basic platform design of a home healthcare system via the Internet current under implementation in our research lab.

In this paper, a novel biologically inspired neural network approach is proposed for real-time motion planning with obstacle avoidance of a nonholonomic car-like robot in a nonstationary environment. The dynamics of each neuron in the topologically organized neural network is characterized by a shunting equation derived from Hodgkin and Huxley's (1952) membrane equation. The robot configuration space constitutes the state space of the neural network. There are only local connections among neurons. Thus the computational complexity linearly depends on the neural network size. The neural activity propagation is subject to the kinematic constraints of the nonholonomic car-like robot. The real-time robot motion is planned through the dynamic neural activity landscape without any prior knowledge of the dynamic environment, without any learning procedures, and without any local collision checking procedures at each step of the robot movement. Therefore the model algorithm is computationally efficient. The stability of the neural network system is proved by qualitative analysis and a Lyapunov stability theory. Simulation in several computer-synthesized virtual environments further demonstrates the advantages of the proposed approach with encouraging experimental results.

A neural network approach to real-time collision-free navigation of holonomic 3-degree-of-freedom (DOF) robots in a nonstationary environment is proposed. This approach is based on a biologically inspired model for dynamic trajectory generation of a point robot or a multi-joint robot manipulator. The state space of the neural network is three-dimensional (3D), where two represent the spatial position in the 2D Cartesian workspace and one represents the orientation of the robot. This model is capable of generating a real-time optimal navigation path for 3-DOF robots through the dynamic neural activity landscape without explicitly optimizing any cost functions, without any learning process, and without any local collision checking procedures. Therefore it is computationally efficient. In addition, this model can deal with real-time navigation with sudden environmental changes, navigation of a robot with multiple targets, and navigation of multiple robots. The stability of the neural network is guaranteed by Lyapunov stability analysis. The effectiveness and efficiency are demonstrated through simulation studies.

A neural network approach is proposed for real-time collision-free motion planning of holonomic and nonholonomic car-like robots in a nonstationary environment. This model is capable of planning real-time robot motion with sudden environmental changes, motion of a car with multiple targets, and motion of multiple robots. The proposed neural network model is biologically inspired, where the dynamics of each neuron in the topologically organized neural network is characterized by a shunting equation or an additive equation. There are only local connections among neurons. The real-time optimal robot motion is planned through the dynamic neural activity landscape of the neural network without explicitly searching over the free workspace nor the collision paths, without explicitly optimizing any cost functions, without any prior knowledge of the dynamic environment, without any learning process, and without any local collision checking procedures. Therefore it is computationally efficient. The stability of the neural network is guaranteed by Lyapunov stability analysis. The effectiveness and efficiency of the proposed approach are demonstrated through simulation studies.

A neural network approach is proposed for real-time collision free trajectory generation in an environment with varying obstacles and moving target. This biologically inspired neural network is topologically organised. The dynamics of each neuron is characterised by a shunting equation or an additive equation. Each neuron has only local connections, and the optimal trajectories are generated without any explicitly optimising cost functions and without learning. Therefore the model is computationally efficient. The stability of the network is analytically proved using a Lyapunov function candidate. As examples, the proposed neural network is applied to trajectory formation for a mobile robot in solving maze-type problems, dynamically tracking moving target, and avoiding varying obstacles. The efficiency of the proposed approach is demonstrated through simulation and comparison studies.

To dynamically generating trajectory with collision free is very important but difficult for the robots in a nonstationary environment. In this paper, the real-time trajectory generation and obstacle avoidance are studied using biologically motivated neural network approaches. The optimal trajectory is generated through the neural dynamics of a topologically organised neural network. Each neuron of the neural net is characterised by a shunting equation or an additive equation. This model is computationally efficient and its stability is guaranteed. These neural network approaches were applied for solving maze-type problems, dynamically tracking moving target, and avoiding varying obstacles. The model parameter sensitivity and model variations are discussed. Simulations are included to demonstrate the proposed approaches.

The main objective of the research project reported in this paper is to design an assistive device that will help patients with eye-implant to have natural eye movement. The patients lose their eye for various reasons. The loss of an eye can be solved by the ocular implant. The artificial eye can be made like a real eye cosmetically. But the problem is that it is static and does not have the natural movement of an eye. We design an ocular assistive system to enable the artificial eye have the natural movement of a real eye. This paper starts with the literature review of the eye movement detection methods and followed by the description of the experimental system we designed and constructed. The paper is concluded with further considerations.

Two approaches to the evaluation of the manipulator regressor of a general n-degree-of-freedom (DOF) robot are presented. The first method is an energy-based approach using the Lagrangian formulation of robot dynamics as a starting point. A key fact used in deriving the solution is that the manipulator Lagrangian is linkwise additive. The second approach generates an iterative algorithm for efficient numerical evaluation of the regressor. It is obtained by reformulating the Newton-Euler recursion using vector analysis type techniques. In addition, a modified Slotine-Li algorithm for adaptive motion control is presented and is then applied in a simulation study to a 4-DOF PUMA-type robot, where the manipulator regressor is evaluated using the iterative algorithm proposed.>

The commenters point out an error in the adaptive impedance-control approach for robotic manipulators given in the above-titled paper by W.-S. Lu and Q.-H. Meng (ibid., vol.7, no.3, p.408-15, June 1991). Lu and Meng assert that their adaptive impedance-control scheme can compensate for uncertainty in the force measurements; however, it is shown that it is possible for the uncertainty in the force measurements to cause the parameter estimates to go unbounded. To remedy this possible instability problem, an additional auxiliary controller is suggested. In their reply, Lu and Meng fail to find any error in the Lyapunov analysis of their algorithm. They are not convinced that one should seek a theoretically asymptotically stable solution with a controller that has a highly undesirable feature.>

Two adaptive impedance control algorithms are presented. In this treatment, it is assumed that some parameters in the manipulator dynamics may be uncertain, and the measurements from the wrist force sensor utilized are imprecise. By introducing the concept of target-impedance reference trajectory (TIRT), which characterizes a desired dynamic relation of the end-point with the environment and a refined Lyapunov approach, it is shown that the adaptation mechanisms previously suggested can be injected into N. Hogan's (1987) conventional impedance control scheme. The two resulting algorithms are compared in terms of implementation feasibility as well as computation efficiency. Simulation results are presented to illustrate the proposed algorithms.>