Fast motion and low vibration are often conflicting requirements in industrial robots. Due to joint flexibility, operating at high accelerations can excite mechanical resonance, leading to vibrations that may accelerate gearbox wear and degrade control performance. Effectively suppressing these vibrations requires accurate modeling, identification, and compensation of joint flexibility across the entire workspace and under various payloads. The conventional two-mass model, which focuses on joint rotational stiffness in the gearbox, fails to capture the global flexibility characteristics, as flexibility also arises from bending in the bearings. This article presents a practical multi-local-mode-based model and the corresponding identification method to globally characterize a manipulator’s joint flexibility. Based on this model, a vibration suppression method is developed to mitigate mechanical resonance, achieving low vibration even at high acceleration. Comparative experiments demonstrate that the proposed approach effectively suppresses vibrations across the workspace and under various payload conditions.

SLAM (Simultaneous Localization and Mapping) has a wide application prospect in navigation of autonomous endoscopic minimally invasive surgery. However, traditional methods like ORB-SLAM2 often struggle in dynamic environments and exhibit significant modeling inaccuracies, particularly in bronchial scenes where respiratory motion induces continuous deformation of the airway structure. To address this challenge, we propose a novel monocular SLAM-based framework for dynamic bronchial environment reconstruction tailored to robotic lung invasive surgery. Firstly, a pseudo-static processing method that builds map sequences at identical respiratory phases across cycles was built. Furthermore, we present a voxel model optimization technique using curvature-consistent graph interpolation to refine the bronchial lumen surface, eliminating pores and redundancies in the voxel map. Experiments conducted on real patient demonstrate that our method has good accuracy and robustness. Compared to existing state-of-the-art methods, our framework achieves superior reconstruction completeness, showing strong potential for clinical use in robotic bronchoscopy.

Prostate cancer is the second most common cancer in men, and transrectal ultrasound (TRUS) guided biopsy is the standard method to diagnose prostate cancer. Accurate prostate segmentation in TRUS images is crucial for precise biopsy. Manual segmentation is laborious, while automated segmentation faces significant challenges due to the low signal-to-noise ratio, blurred boundaries, and presence of noise and artifacts. To address these issues, this paper proposes a 3D edge-attention denoising diffusion network, aiming to achieve high accuracy and generalizability for prostate segmentation in TRUS-guided biopsy. The proposed network incorporates an edge attention denoising U-Net (EAD U-Net) to extract and utilize desired edge information in TRUS images, improving the segmentation accuracy in challenging regions of the prostate. To reduce uncertainty and enhance network accuracy, we incorporate a Kalman fusion module, which utilizes the Kalman filter and all estimations from the EAD U-Net in reverse process to obtain the optimal segmentation estimation. The proposed network was evaluated using 1834 3D ultrasound images from two open-source datasets. Comparative experiments with existing methods demonstrate that our method surpasses state-of-the-art techniques, proving its effectiveness in prostate segmentation from TRUS images. The proposed method achieved an average Dice similarity coefficient of 92.92% and 94.0%, and the 95th percentile of Hausdorff distance of 1.07 mm and 0.77 mm on two datasets, demonstrating the potential to facilitate accurate MRI-TRUS fusion guided prostate biopsy.

The enhanced mobility brought by legged locomotion empowers quadrupedal robots to navigate through complex and unstructured environments. However, optimizing agile locomotion while accounting for the varying energy costs of traversing different terrains remains an open challenge. Most previous work focuses on planning trajectories with traversability cost estimation based on human-labeled environmental features. This human-centric approach is insufficient because it does not account for the varying capabilities of the robot locomotion controllers over challenging terrains. To address this, we introduce a novel real-world learning pipeline that unifies offline demonstrations, online reinforcement learning, and multi-modal perception to achieve robust legged navigation. The framework employs multiple training stages to develop a planner that guides the robot in avoiding obstacles and hardto- traverse terrains while reaching its goals. We first develop a novel traversability estimator in a robot-centric manner. The training of the navigation planner is directly performed in the real world using a sample efficient reinforcement learning method. With the proposed method, a quadrupedal robot learns to perform traversability-aware navigation through realworld interactions in diverse offroad and unstructured environments. Moreover, the robot demonstrates the ability to generalize the learned navigation skills to unseen scenarios.

Observing that the key for robotic action planning is to understand the target-object motion when its associated part is manipulated by the end effector, we propose to generate the 3D object-part scene flow and extract its transformations to solve the action trajectories for diverse embodiments. The advantage of our approach is that it derives the robot action explicitly from object motion prediction, yielding a more robust policy by understanding the object motions. Also, beyond policies trained on embodiment-centric data, our method is embodiment-agnostic, generalizable across diverse embodiments, and being able to learn from human demonstrations. Our method comprises three components: an object-part predictor to locate the part for the end effector to manipulate, an RGBD video generator to predict future RGBD videos, and a trajectory planner to extract embodiment-agnostic transformation sequences and solve the trajectory for diverse embodiments. Trained on videos even without trajectory data, our method still outperforms existing works significantly by 27.7% and 26.2% on the prevailing virtual environments MetaWorld and Franka-Kitchen, respectively. Furthermore, we conducted real-world experiments, showing that our policy, trained only with human demonstration, can be deployed to various embodiments.

Shape control of deformable objects under both rotational and translational deformations is important for versatile robotic applications. However, deformation control with full 6-degree-of-freedom (DoF) manipulation is an open problem, since modeling and describing rotational deformations lead to significant challenges. To tackle the problem, this paper proposes a novel method by introducing a co-rotated space for the modal graph representation of objects with unknown physical and geometric models. In this space, we design new deformation features that can encode local rotations while preserving a compact and low-frequency shape representation. Moreover, these features can be mapped analytically to the robot manipulation, enabling the design of adaptive control laws with guaranteed stability for unmodeled objects. Experiments on complex volumetric objects demonstrate the effectiveness and advantage of our method with raw, noisy, and unregistered point clouds. The results highlight the importance of integrating co-rotated features to address rotational deformations.

To achieve en bloc resection of bladder tumor and the anterior tumor resection in transurethral resection of bladder tumor (TURBT), a cystoscope transurethral continuum robotic system has been proposed. A continuum cystoscope in the system needs to bend more than 180° and its base has translation, axial rotation, and tilt degrees of freedom to achieve full bladder accessibility. Under the constant-curvature assumption, the analytical solution of inverse kinematics already exists for multi-segment continuum robots with variable segment lengths and continuum robots with two inextensible segments. However, there is a lack of analytical inverse kinematics solution for continuum robots with features of the continuum cystoscope. Therefore, this paper proposes a novel and efficient inverse kinematics solving algorithm for the continuum cystoscope used in TURBT. The proposed method simplifies the inverse kinematics problem by constructing a robot plane coordinate and uses geometric relationships to derive a non-linear constraint equation containing only one intermediate variable. By solving this non-linear equation, the solution to the entire inverse kinematics problem is obtained. Additionally, based on this inverse kinematics algorithm, the length of the continuum segment is designed to ensure full bladder accessibility. In the comparative experiments with the Jacobian-based method, which involves 12500 target poses, the proposed method solves 100% of the inverse kinematics problems with a much greater computational efficiency.

Witnessing the rapid progress and development in instance-level object pose estimation, increasing attention has shifted to the more challenging problem for unseen objects, which is in great demand for various robotic applications. In this paper, we propose the GPD, a novel framework for unseen object pose estimation, including both category-level and cross-category objects. The key innovation of the GPD model is the effective utilization of geometric primitives in target reconstruction and pose estimation, as it can generalize the learned primitive deformation across intra-class and inter-class instances. Additionally, we also design an advanced scheme for representative object feature extraction, including attention-aware excitation, multi-scale fusion, and semantic feature encoding. Extensive evaluations validate the effectiveness of individual innovation modules and the overall superior performance of the GPD. It not only achieves the SOTA results on category-level benchmarks CAMERA25 and REAL275, but also demonstrates impressive generalization ability across novel objects on the GraspNet-1Billion dataset. Furthermore, we deploy the trained GPD model for vision-guided robotic grasping experiments in simulation and real-world settings, again exhibiting its outstanding robustness and practicability in robotic manipulations. Note to Practitioners—This paper is motivated by the problem of unseen object pose estimation and robotic manipulation in unstructured environments. For intelligent robots expected to interact with their surroundings, rather than just passively perceiving them like surveillance cameras, 6Dof pose estimation is a critical capability. However, existing approaches generally face two key challenges. On the one hand, robots are likely to encounter unseen objects in real-world applications. Without the availability of prior models or specific training data for these unseen objects, instance-level and category-level methods may become ineffective or even fail to work. On the other hand, the error tolerance of precise tabletop robotic manipulation is very tight, and the varying lighting conditions and background noise impose higher robustness requirements on pose estimation algorithms. To address these difficulties, we propose a novel network that learns geometric primitive deformation for pose estimation. This model is less dependent on object prior information, thereby enhancing the generalization ability. Additionally, by incorporating cross-modal excitation and multi-scale fusion during feature extraction, our model can capture representative appearance and geometric information of objects for accurate pose estimation. Extensive experimental results on benchmark datasets quantitatively validate the superior performance of our approach. We also demonstrate its effectiveness in robotic applications through unseen object grasping experiments on Kinova and Franka Emika robot platforms. In the future, we plan to explore primitive combination schemes for compound object representation, enabling pose estimation for more complex-shaped objects.

This research investigates learning-based control of continuum robots in constrained environments without relying on analytical models. We propose a data-efficient stochastic control strategy incorporating online model updates to achieve precise manipulation even when arbitrary robot deformations occur due to environmental interactions. A localized Gaussian process regression approach accounting for state stochasticity is first presented to approximate the forward kinematics. The learned model enables uncertainty-aware stochastic predictions via the proposed scaled unscented transform (SUT)-based method for efficient exploration. Leveraging new data, online model updates are performed in a highly sample-efficient manner. Furthermore, a probabilistic model predictive control approach integrating the learned models and chance constraints based on Chebyshev’s inequality is developed for searching an optimal control sequence. Simulations and experiments are performed to demonstrate the effectiveness of the proposed approach for controlling continuum robots in constrained environments using limited observational data.Note to Practitioners—The motivation of this research is to solve the problem of controlling continuum robots in constraint environment. The flexibility of continuum robots significantly affects the manipulation accuracy, and the interaction between the continuum robot and environmental constraints can also lead to unpredictable behavior. Learning control methods that rely only on sensory data, provide a feasible solution to the aforementioned problem. However, current methods lack sample efficiency and the capability to handle unknown environmental constraints. This research proposes a learning control method which can control a flexible continuum robot in constrained environments with high data-efficiency and robustness even when the robot shape undergoes sudden deformations due to contact with obstacles.

A new control pipeline has been proposed for the Rope-Base Soft-tip Manipulator (RBSM) to execute the surface contact task to prevent the jamming and slipping problems. The control pipeline enables smooth surface-to-surface contact for the RBSM using only force sensors, eliminating the dependence on additional pose measurement of the window surface plane and soft-tip deformation information. The pipeline consists of three steps: free contact step implemented by an exponential force shape controller to avoid force overshoot to the window surface; orientation refinement step implemented by a force and torque combined controller to make the RBSM cleaning head surface stable adapt to the smooth window surface; and finally, a release normal force step to reduce head jamming and region covering with a pre-defined vibration-less cleaning trajectory for smooth cleaning on the slippery window surface. The proposed pipeline has been validated in a Rope base Cleaning Manipulator prototype to clean a common window surface. The force and velocity curves during the cleaning experiment show that the proposed method achieves smooth scraping and cleaning under unknown initial significant errors in surface orientation.

Accurate estimation of absolute depth from a monocular endoscope is a fundamental task for automatic navigation systems in robotic surgery. Previous works solely rely on uni-modal data (i.e., monocular images), which can only estimate depth values arbitrarily scaled with the real world. In this paper, we present a novel framework, SADER, which explores vision and robot kinematics to estimate the high-quality absolute depth for monocular surgical scenes. To jointly learn the multi-modal data, we introduce a self-distillation based two-stage training policy in the framework. In the first stage, a boosting depth module based on vision transformer is proposed to improve the relative depth estimation network that is trained in a self-supervised method. Then, we develop an algorithm to automatically compute the scale from robot kinematics. By coupling the scale and relative depth data, pseudo absolute depth labels for all images are yielded. In the second stage, we re-train the network with 3D loss supervised by pseudo labels. To make our method generalize to different endoscopes, the learning of endoscopic intrinsics is integrated into the network. In addition, we did cadaver experiments to collect new surgical depth estimation data about robotic laparoscopy for evaluation. Experimental results on public SCARED and cadaver data demonstrate that the SADER outperforms previous state-of-art even stereo-based methods with an accuracy error under 1.90 mm, proving the feasibility of our approach to recover the absolute depth with monocular inputs. Note to Practitioners—This paper aims to solve the problem of absolute monocular depth estimation in automatic surgical navigation by leveraging the multi-modal data from the robot-based endoscopic system. Accurate depth perception with real scales of the monocular scene is essential for the control of surgical robots in automatic navigation. However, current methods can only predict the relative depth of the surgical scene using monocular images. In this article, we propose a self-supervised learning-based method to achieve high-quality absolute depth estimation of monocular endoscopic images. It neither needs manual data annotation, nor other imaging modalities. The experiments extensively validate the feasibility and high performance of our framework for absolute depth estimation on monocular endoscopes. This absolute depth perception framework can be potentially encapsulated into the automatic navigation system in the near future.

This article introduces a multiview 3-D microscopic profilometry system equipped with tilted cameras adhering to the Scheimpflug condition and a vertically aligned projector. It utilizes a novel 3-D cross-ratio invariant (3D-CRI) model that offers inherent self-correction, enhanced global consistency, and computational efficiency. The inherent self-correction mitigates optical contaminations like multiple reflections by using deviations from the epipolar line for optimal candidate selection. The model is also designed as spatial lines to associate the approximate linear error distribution across depths, thereby facilitating its further correction (e.g., proposed embedded linear compensation method to address data inconsistency among various binocular projector-camera setups). Additionally, the model simplifies computational demands by directly correlating phase differences to spatial coordinates. Furthermore, a conversion method from the triangular stereo to the 3D-CRI model is developed, avoiding the complex per-pixel calibration. The experiments are conducted to verify robustness, globally consistent accuracy, and speed performance. The experimental results validate that the system is robust to multiple reflections by testing on the printed circuit boards with dense components. The global consistency is also verified by the experiments with better quantitative metrics within the whole measurement range over ten times the depth of field. In particular, the quantitative metrics of theZ-axis are halved. The speed performance shows that our method is the fastest among competing techniques.

Underwater object detection is critical for applications, such as environmental monitoring, resource exploration, and the navigation of autonomous underwater vehicles. However, accurately detecting small objects in underwater environments remains challenging due to noisy imaging conditions, variable illumination, and complex backgrounds. To address these challenges, we propose the adaptive residual attention network (ARAN), an optimized deep learning framework designed to enhance the detection and precise identification of diminutive targets in complex aquatic settings. ARAN incorporates the proposed Fusion path aggregation network (PANet), which refines spatial features by effectively distinguishing objects from their backgrounds. The framework integrates three novel modules: first, multiscale feature attention, which enhances low-level feature extraction; second, high–low feature residual learning, which rearranges channel and batch dimensions to capture pixel-level relationships through cross-dimensional interactions; and third, multilevel feature dynamic aggregation, which dynamically adjusts fusion weights to facilitate progressive multilevel feature fusion and mitigate conflicts in multiscale integration, ensuring that small objects are not overshadowed. Extensive experiments on four benchmark datasets demonstrate that ARAN significantly outperforms mainstream models, achieving state-of-the-art performance. Notably, on the CSIRO dataset, ARAN attains a mean average precision at 50% of 98%, precision of 94.7%, F2-score of 94.6%, and recall of 94.7%. These results confirm our model's superior accuracy, robustness, and efficiency in underwater object detection, highlighting its potential for practical deployment in challenging aquatic environments. We will release the code on GitHub upon acceptance of the article.

In 3D microscopic imaging, the extremely shallow depth of field presents a challenge for accurate 3D reconstruction in cases of significant defocus. Traditional calibration methods rely on the spatial extraction of feature points to establish spatial 3D information as the optimization objective. However, these methods suffer from reduced extraction accuracy under defocus conditions, which causes degradation of calibration performance. To extend calibration volume without compromising accuracy in defocused scenarios, we propose a per-pixel calibration based on multi-view 3D reconstruction errors. It utilizes 3D reconstruction errors among different binocular setups as an optimization objective. We first analyze multi-view 3D reconstruction error distributions under the poor-accuracy optical model by employing a multi-view microscopic 3D measurement system using telecentric lenses. Subsequently, the 3D proportion model is proposed for implementing our error-based per-pixel calibration, derived as a spatial linear expression directly correlated with the 3D reconstruction error distribution. The experimental results confirm the robust convergence of our method with multiple binocular setups. Near the focus volume, the multi-view 3D reconstruction error remains approximately m (less than 0.5 camera pixel pitch), with absolute accuracy maintained within 0.5% of the measurement range. Beyond tenfold depth of field, the multi-view 3D reconstruction error increases to around m (still less than 2 camera pixel pitches), while absolute accuracy remains within 1% of the measurement range. These high-precision measurement results validate the feasibility and accuracy of our proposed calibration.

Visualizing surgical scenes is crucial for revealing internal anatomical structures during minimally invasive procedures. Novel View Synthesis is a vital technique that offers geometry and appearance reconstruction, enhancing understanding, planning, and decision-making in surgical scenes. Despite the impressive achievements of Neural Radiance Field (NeRF), its direct application to surgical scenes produces unsatisfying results due to two challenges: endoscopic sparse views and significant photometric inconsistencies. In this paper, we propose uncertainty-aware conditional NeRF for novel view synthesis to tackle the severe shape-radiance ambiguity from sparse surgical views. The core of UC-NeRF is to incorporate the multi-view uncertainty estimation to condition the neural radiance field for modeling the severe photometric inconsistencies adaptively. Specifically, our UC-NeRF first builds a consistency learner in the form of multi-view stereo network, to establish the geometric correspondence from sparse views and generate uncertainty estimation and feature priors. In neural rendering, we design a base-adaptive NeRF network to exploit the uncertainty estimation for explicitly handling the photometric inconsistencies. Furthermore, an uncertainty-guided geometry distillation is employed to enhance geometry learning. Experiments on the SCARED and Hamlyn datasets demonstrate our superior performance in rendering appearance and geometry, consistently outperforming the current state-of-the-art approaches. Our code will be released at https://github.com/wrld/UC-NeRF.

Autonomous ground mobile robots rely on their configuration characteristics to prevent tip-overs and collisions, ensuring safe navigation in complex environments. However, complex configurations with specially designed links and joints produce a higher-dimensional workspace and bring significant challenges for path planning, especially in large-scale rough terrains. To address this, we propose a real-time multi-level terrain-aware path planning framework that integrates different levels of terrain awareness into the global and local layers. An implicit map representation is introduced at the global layer to enable efficient terrain analysis and path planning, while an iterative geometric evaluation is designed at the local layer to estimate configuration stability and improve path smoothness. By sharing the global layer information with the local layer, the framework enhances path planning efficiency and adaptability in complex environments. Its modular design supports diverse robot configurations and pathfinding algorithms, enabling effective autonomous navigation in large-scale 3D terrains with online or offline maps. Simulations and real-world experiments demonstrated that our approach outperforms state-of-the-arts across diverse environments, including uneven terrains, multi-layered structures, and complex debris fields. The results highlighted that our approach provides faster and safer path planning, more accurate and robust configuration-stability estimation, and higher success rates in traversing complex 3D environments.

Human beings possess a remarkable skill for fine in-hand manipulation, utilizing both intrafinger interactions (in-finger) and finger–environment interactions across a wide range of daily tasks. These tasks range from skilled activities like screwing light bulbs, picking and sorting pills, and in-hand rotation, to more complex tasks such as opening plastic bags, cluttered bin picking, and counting cards. Despite its prevalence in human activities, replicating these fine motor skills in robotics remains a substantial challenge. This study tackles the challenge of fine in-hand manipulation by introducing the dexterous and compliant (DexCo) hand system. The DexCo hand mimics human dexterity, replicating the intricate interaction between the thumb, index, and middle fingers, with a contractable palm. The key to maneuverable fine in-hand manipulation lies in its innovative soft hydraulic actuation, which strikes a balance between control complexity, dexterity, compliance, and motion accuracy within a compact structure, enhancing the overall performance of the system. The model of soft hydraulic actuation, based on hydrostatic force analysis, reveals the compliance of hand joints, which is also further extended to a dedicated robot operating system (ROS) package for DexCo hand simulation, considering both motion and stiffness aspects. Dedicated velocity and position teleoperation controllers are designed for implementing real physical manipulation tasks. The benchmark results show that the fingertip achieves a maximum repeatable finger strength of 34.4 N, a grasp cycle time of less than 2.04 s, and a maximum repeatability accuracy of 0.03 mm. Experimental results demonstrate the DexCo hand successfully performs complex fine in-hand manipulation tasks, providing a promising solution for advancing robotic manipulation capabilities toward the human level.

Stereo matching is a core task for many computer vision and robotics applications. Despite their dominance in traditional stereo methods, the hand-crafted Markov Random Field (MRF) models lack sufficient modeling accuracy compared to end-to-end deep models. While deep learning representations have greatly improved the unary terms of the MRF models, the overall accuracy is still severely limited by the hand-crafted pairwise terms and message passing. To address these issues, we propose a neural MRF model, where both potential functions and message passing are designed using data-driven neural networks. Our fully data-driven model is built on the foundation of variational inference theory, to prevent convergence issues and retain stereo MRF's graph inductive bias. To make the inference tractable and scale well to high-resolution images, we also propose a Disparity Proposal Network (DPN) to adaptively prune the search space of disparity. The proposed approach ranks 1ston both KITTI 2012 and 2015 leaderboards among all published methods while running faster than 100 ms. This approach significantly outperforms prior global methods, e.g., lowering D1 metric by more than 50% on KITTI 2015. In addition, our method exhibits strong cross-domain generalization and can recover sharp edges. The codes at https://github.com/aeolusguan/NMRF.

Given an input set of 3D point pairs, the goal of outlier-robust 3D registration is to compute some rotation and translation that align as many point pairs as possible. This is an important problem in computer vision, for which many highly accurate approaches have been recently proposed. Despite their impressive performance, these approaches lack scalability, often overflowing the 16GB of memory of a standard laptop to handle roughly 30,000 point pairs. In this paper, we propose a 3D registration approach that can process more than ten million (107) point pairs with over 99% random outliers. Moreover, our method is efficient, entails low memory costs, and maintains high accuracy at the same time. We call our method TEAR11https://github.com/tyhuang98/TEAR-release, as it involves minimizing an outlier-robust loss that computes Truncated Entry-wise Absolute Residuals. To minimize this loss, we decompose the original 6-dimensional problem into two subproblems of dimensions 3 and 2, respectively, solved in succession to global optimality via a customized branch-and-bound method. While branch-and-bound is often slow and unscalable, this does not apply to TEAR as we propose novel bounding functions that are tight and computationally efficient. Experiments on various datasets are conducted to validate the scalability and efficiency of our method.

Soft tissue tracking is crucial for computer-assisted interventions. Existing approaches mainly rely on extracting discriminative features from the template and videos to recover corresponding matches. However, it is difficult to adopt these techniques in surgical scenes, where tissues are changing in shape and appearance throughout the surgery. To address this problem, we exploit optical flow to naturally capture the pixel-wise tissue deformations and adaptively correct the tracked template. Specifically, we first implement an inter-frame matching mechanism to extract a coarse region of interest based on optical flow from consecutive frames. To accommodate appearance change and alleviate drift, we then propose an adaptive-template matching method, which updates the tracked template based on the reliability of the estimates. Our approach, Ada-Tracker, enjoys both short-term dynamics modeling by capturing local deformations and long-term dynamics modeling by introducing global temporal compensation. We evaluate our approach on the public SurgT benchmark, which is generated from Hamlyn, SCARED, and Kidney boundary datasets. The experimental results show that Ada-Tracker achieves superior accuracy and performs more robustly against prior works. Code is available at https://github.com/wrld/Ada-Tracker.

Recently, fiber optic sensors such as fiber Bragg gratings (FBGs) have been widely investigated for shape reconstruction and force estimation of flexible surgical robots. However, most existing approaches need precise model parameters of FBGs inside the fiber and their alignments with the flexible robots for accurate sensing results. Another challenge lies in online acquiring external forces at arbitrary locations along the flexible robots, which is highly required when with large deflections in robotic surgery. In this paper, we propose a novel data-driven paradigm for simultaneous estimation of shape and force along highly deformable flexible robots by using sparse strain measurement from a single-core FBG fiber. A thin-walled soft sensing tube helically embedded with FBG sensors is designed for a robotic-assisted flexible ureteroscope with large deflection up to 270° and a bend radius under 10 mm. We introduce and study three learning models by incorporating spatial strain encoders, and compare their performances in both free space without interactions as well as constrained environments with contact forces at different locations. The experimental results in terms of dynamic shape-force sensing accuracy demonstrate the effectiveness and superiority of the proposed methods.

Model Predictive Control (MPC) relies heavily on the robot model for its control law. However, a gap always exists between the reduced-order control model with uncertainties and the real robot, which degrades its performance. To address this issue, we propose the controller of integrating a data-driven error model into traditional MPC for quadruped robots. Our approach leverages real-world data from sensors to compensate for defects in the control model. Specifically, we employ the Autoregressive Moving Average Vector (ARMAV) model to construct the state error model of the quadruped robot using data. The predicted state errors are then used to adjust the predicted future robot states generated by MPC. By such an approach, our proposed controller can provide more accurate inputs to the system, enabling it to achieve desired states even in the presence of model parameter inaccuracies or disturbances. The proposed controller exhibits the capability to partially eliminate the disparity between the model and the real-world robot, thereby enhancing the locomotion performance of quadruped robots. We validate our proposed method through simulations and real-world experimental trials on a large-size quadruped robot that involves carrying a 20 kg un-modeled payload (84% of body weight).

Photometric bundle adjustment (PBA) is widely used in estimating the camera pose and 3D geometry by assuming a Lambertian world. However, the assumption of photometric consistency is often violated since the non-diffuse reflection is common in real-world environments. The photometric inconsistency significantly affects the reliability of existing PBA methods. To solve this problem, we propose a novel physically-based PBA method. Specifically, we introduce the physically-based weights regarding material, illumination, and light path. These weights distinguish the pixel pairs with different levels of photometric inconsistency. We also design corresponding models for material estimation based on sequential images and illumination estimation based on point clouds. In addition, we establish the first SLAM-related dataset of non-Lambertian scenes with complete ground truth of illumination and material. Extensive experiments demonstrated that our PBA method outperforms existing approaches in accuracy.

Object pose estimation plays a crucial role in robotic manipulation, however, its practical applicability still suffers from limited generalizability. This paper addresses the challenge of generalizable object pose estimation, particularly focusing on category-level object pose estimation for unseen object categories. Current methods either require impractical instance-level training or are confined to predefined categories, limiting their applicability. We propose VFM-6D, a novel framework that explores harnessing existing vision and language models, to elaborate object pose estimation into two stages: category-level object viewpoint estimation and object coordinate map estimation. Based on the two-stage framework, we introduce a 2D-to-3D feature lifting module and a shape-matching module, both of which leverage pre-trained vision foundation models to improve object representation and matching accuracy. VFM-6D is trained on cost-effective synthetic data and exhibits superior generalization capabilities. It can be applied to both instance-level unseen object pose estimation and category-level object pose estimation for novel categories. Evaluations on benchmark datasets demonstrate the effectiveness and versatility of VFM-6D in various real-world scenarios.

Estimating the rigid transformation with 6 degrees of freedom based on a putative 3D correspondence set is a crucial procedure in point cloud registration. Existing correspondence identification methods usually lead to large outlier ratios (>95% is common), underscoring the significance of robust registration methods. Many researchers turn to parameter search-based strategies (e.g., Branch-and-Bround) for robust registration. Although related methods show high robustness, their efficiency is limited to the high-dimensional search space. This paper proposes a heuristics-guided parameter search strategy to accelerate the search while maintaining high robustness. We first sample some correspondences (i.e., heuristics) and then just need to sequentially search the feasible regions that make each sample an inlier. Our strategy largely reduces the search space and can guarantee accuracy with only a few inlier samples, therefore enjoying an excellent trade-off between efficiency and robustness. Since directly parameterizing the 6-dimensional nonlinear feasible region for efficient search is intractable, we construct a three-stage decomposition pipeline to reparameterize the feasible region, resulting in three lower-dimensional sub-problems that are easily solvable via our strategy. Besides reducing the searching dimension, our decomposition enables the leverage of 1-dimensional interval stabbing at all three stages for searching acceleration. Moreover, we propose a valid sampling strategy to guarantee our sampling effectiveness, and a compatibility verification setup to further accelerate our search. Extensive experiments on both simulated and real-world datasets demonstrate that our approach exhibits comparable robustness with state-of-the-art methods while achieving a significant efficiency boost.

The manipulation of deformable objects, especially deformable linear objects (DLOs), represents an open challenge in robotics. The keys to solving the problem are accurately recognizing the topological model describing the entwined state of DLO(s) and a manipulation strategy based on it. The situation becomes more challenging in practical applications since the DLO(s) may be placed under complex and unencountered environments, making distinguishing between the targets and the background difficult. In addition, with the information obtained from a sensor system with a limited field of sense, a robot has to treat the encountered DLO(s) as multiple ones of an unknown quantity. This paper proposes a solution based on deep learning techniques for these complicated scenarios. The approach can derive a topological model describing the entangled structure of single or multiple DLOs. Based on the model, we proposed a strategy for untangling the DLO(s), considering both the possible self-tangling in one DLO and the tangling between multiple DLOs. The strategy ensures that the entangled DLO(s) can be arranged to be the neatest state given a limited field of view. The feasibility and effectiveness of the proposed solution were verified by untangling experiments utilizing a dual-arm robot system. Even if the exact quantity of the DLO(s) is unknown, the robots can still untangle the DLO(s). Moreover, the proposed approach performed robustness to unfamiliar background textures, which is preferable in practical applications. Dataset used in this paper can be found at https://github.com/lancexz/dlos-dataset.Note to Practitioners—This article proposes an automatic solution for untangling an unknown quantity of DLO(s) using robots. Issues concerned with complex backgrounds, an unknown quantity of DLOs, and limits of field of view, which are easily encountered in practical applications, are solved. Experiments demonstrated the feasibility and robustness of the approach in different scenarios. Potential applications for the proposed method include pipeline bin-picking tasks, rope-like object manipulation in household robots, and cable assembly in industrial areas. Moreover, the topological state recognition process can also be utilized in knot-tying studies and producing datasets. As the current system can not deal with scenes where DLO segments or crossings severely overlap, practitioners should integrate real-time tracking, deformation control method, and unqualified cases detection to avoid the appearance of such cases during manipulation. Future works should also consider the rebound caused by the elastic potential of DLOs.

Shape from Polarization (SfP) aims to recover surface normal using the polarization cues of light. The accuracy of existing SfP methods is affected by two main problems. First, the ambiguity of polarization cues partially results in false normal estimation. Second, the widely-used assumption about orthographic projection is too ideal. To solve these problems, we propose the first approach that com-bines deep learning and stereo polarization information to recover not only normal but also disparity. Specifically, for the ambiguity problem, we design a Shape Consistency-based Mask Prediction (SCMP) module. It exploits the inherent consistency between normal and disparity to identify the areas with false normal estimation. We replace the unreliable features enclosed by these areas with new features extracted by global attention mechanism. As to the orthographic projection problem, we propose a novel Viewing Direction-aided Positional Encoding (VDPE) strategy. This strategy is based on the unique pixel-viewing direction encoding, and thus enables our neural network to handle the non-orthographic projection. In addition, we establish a real-world stereo SfP dataset that contains various object categories and illumination conditions. Experiments showed that compared with existing SfP methods, our approach is more accurate. Moreover, our approach shows higher robustness to light variation.

Estimating depth from four large field of view (FoV) cameras has been a difficult and understudied problem. In this paper, we proposed a novel and simple system that can convert this difficult problem into easier binocular depth estimation. We name this system OmniVidar, as its results are similar to LiDAR, but rely only on vision. OmniVidar contains three components: (1) a new camera model to address the shortcomings of existing models, (2) a new multi-fisheye camera based epipolar rectification method for solving the image distortion and simplifying the depth estimation problem, (3) an improved binocular depth estimation network, which achieves a better balance between accuracy and efficiency. Unlike other omnidirectional stereo vision methods, OmniVidar does not contain any 3D convolution, so it can achieve higher resolution depth estimation at fast speed. Results demonstrate that OmniVidar outperforms all other methods in terms of accuracy and performance.

Localization in a pre-built map is a basic technique for robot autonomous navigation. Existing mapping and localization methods commonly work well in small-scale environments. As a map grows larger, however, more memory is required and localization becomes inefficient. To solve these problems, map sparsification becomes a practical necessity to acquire a subset of the original map for localization. Previous map sparsification methods add a quadratic term in mixed-integer programming to enforce a uniform distribution of selected landmarks, which requires high memory capacity and heavy computation. In this paper, we formulate map sparsification in an efficient linear form and select uniformly distributed landmarks based on 2D discretized grids. Furthermore, to reduce the influence of different spatial distributions between the mapping and query sequences, which is not considered in previous methods, we also introduce a space constraint term based on 3D discretized grids. The exhaustive experiments in different datasets demonstrate the superiority of the proposed methods in both efficiency and localization performance. The relevant codes will be released at https://github.com/fishmarch/SLAM_Map_Compression.

Scene reconstructions are often incomplete due to occlusions and limited viewpoints. There have been efforts to use semantic information for scene completion. However, the completed shapes may be rough and imprecise since respective methods rely on 3D convolution and/or lack effective shape constraints. To overcome these limitations, we propose a semantic scene completion method based on deformable deep implicit templates (DDIT). Specifically, we complete each segmented instance in a scene by deforming a template with a latent code. Such a template is expressed by a deep implicit function in the canonical frame. It abstracts the shape prior of a category, and thus can provide constraints on the overall shape of an instance. Latent code controls the deformation of template to guarantee fine details of an instance. For code prediction, we design a neural network that leverages both intra-and inter-instance information. We also introduce an algorithm to transform instances between the world and canonical frames based on geometric constraints and a hierarchical tree. To further improve accuracy, we jointly optimize the latent code and transformation by enforcing the zero-valued isosurface constraint. In addition, we establish a new dataset to solve different problems of existing datasets. Experiments showed that our DDIT outperforms state-of-the-art approaches.

This paper proposes a novel bin picking framework, two-stage grasping, aiming at precise grasping of cluttered small objects. Object density estimation and rough grasping are conducted in the first stage. Fine segmentation, detection, grasping, and pushing are performed in the second stage. A small object bin picking system has been realized to exhibit the concept of two-stage grasping. Experiments have shown the effectiveness of the proposed framework. Unlike traditional bin picking methods focusing on vision-based grasping planning using classic frameworks, the challenges of picking cluttered small objects can be solved by the proposed new framework with simple vision detection and planning.

Most existing methods for category-level pose estimation rely on object point clouds. However, when considering transparent objects, depth cameras are usually not able to capture high-quality data, resulting in point clouds with severe artifacts. Without a complete point cloud, existing methods are not applicable to challenging transparent objects. To tackle this problem, we present StereoPose, a novel stereo image based framework for category-level object pose estimation, ideally suited for transparent objects. For a robust estimation from pure stereo images, we develop a pipeline that decouples category-level pose estimation into object size estimation, initial pose estimation, and pose refinement. StereoPose then estimates object pose based on representation in the normalized object coordinate space (NOCS). To address the issue of image content aliasing, we further define a back-view NOCS map for the transparent object. The back-view NOCS aims to reduce the network learning ambiguity caused by content aliasing, and leverage informative cues on the back of the transparent object for more accurate pose estimation. To further improve the performance of the stereo framework, StereoPose is equipped with a parallax attention module for stereo feature fusion and an epipolar loss for improving the stereo-view consistency of network predictions. Extensive experiments on the public TOD dataset demonstrate the superiority of the proposed StereoPose framework for category-level 6D transparent object pose estimation. Code and demos will be available on the project homepage: www.cse.cuhk.edu.hk/~kaichen/stereopose.html.

Task automation of surgical robot has the potentials to improve surgical efficiency. Recent reinforcement learning (RL) based approaches provide scalable solutions to surgical automation, but typically require extensive data collection to solve a task if no prior knowledge is given. This issue is known as the exploration challenge, which can be alleviated by providing expert demonstrations to an RL agent. Yet, how to make effective use of demonstration data to improve exploration efficiency still remains an open challenge. In this work, we introduce Demonstration-guided EXploration (DEX), an efficient reinforcement learning algorithm that aims to overcome the exploration problem with expert demonstrations for surgical automation. To effectively exploit demonstrations, our method estimates expert-like behaviors with higher values to facilitate productive interactions, and adopts non-parametric regression to enable such guidance at states unobserved in demonstration data. Extensive experiments on 10 surgical manipulation tasks from SurRoL, a comprehensive surgical simulation platform, demonstrate significant improvements in the exploration efficiency and task success rates of our method. Moreover, we also deploy the learned policies to the da Vinci Research Kit (dVRK) platform to show the effectiveness on the real robot. Code is available at https://github.com/med-air/DEX.

Recent advancements toward perception and decision-making of flexible endoscopes have shown great potential in computer-aided surgical interventions. However, owing to modeling uncertainty and inter-patient anatomical variation in flexible endoscopy, the challenge remains for efficient and safe navigation in patient-specific scenarios. This paper presents a novel data-driven framework with self-contained visual-shape fusion for autonomous intelligent navigation of flexible endoscopes requiring no priori knowledge of system models and global environments. A learning-based adaptive visual servoing controller is proposed to online update the eye-in-hand vision-motor configuration and steer the endoscope, which is guided by monocular depth estimation via a vision transformer (ViT). To prevent unnecessary and excessive interactions with surrounding anatomy, an energy-motivated shape planning algorithm is introduced through entire endoscope 3-D proprioception from embedded fiber Bragg grating (FBG) sensors. Furthermore, a model predictive control (MPC) strategy is developed to minimize the elastic potential energy flow and simultaneously optimize the steering policy. Dedicated navigation experiments on a robotic-assisted flexible endoscope with an FBG fiber in several phantom environments demonstrate the effectiveness and adaptability of the proposed framework.

This paper presents a robotic in-hand manipulation technique that can be applied to pick an object too large to grasp in a prehensile manner, by taking advantage of its contact interactions with a curved, passive end-effector, and two flat support surfaces. First, the object is tilted up while being held between the end-effector and the supports. Then, the end-effector is tucked into the gap underneath the object, which is formed by tilting, in order to obtain a grasp against gravity. In this paper, we first examine the mechanics of tilting to understand the different ways in which the object can be initially tilted. We then present a strategy to tilt up the object in a secure manner. Finally, we demonstrate successful picking of objects of various size and geometry using our technique through a set of experiments performed with a custom-made robotic device and a conventional robot arm. Our experiment results show that object picking can be performed reliably with our method using simple hardware and control, and when possible, with appropriate fixture design.

Instance segmentation is a long-standing task for supporting robotic bin picking. However, objects of diverse classes can be closely packed with occlusions in cluttered and chaotic scenes, hence, even recent methods could have difficulty in locating clear and precise boundaries to distinguish nearby objects. In this work, we aim to improve the boundary quality of the instance masks for robust and precise instance segmentation in these challenging scenarios. Technical-wise, we first formulate an IoU-based Boundary-aware Mask head (IBM head) for predicting the instance-level mask, boundary, and their corresponding IoU scores. With this core module, we then follow the coarse-to-fine strategy and design our pipeline with two stages: an 1IoUNet to learn localization-based objectness cue and a hierarchical mask refiner to produce sharper and cleaner boundaries. We deploy the IBM head throughout the framework. Extensive experimental results on three grasping benchmarks manifest that our method attains the best instance segmentation performance, compared with the state-of-the-art approaches. Practically, we conduct real-world picking tests to show that with the objectness and boundary IoU scores as guidance, we are able to filter invalid (occluded) instances and select high-fidelity (exposed) instances for grasping.

Needle picking is a challenging manipulation task in robot-assisted surgery due to the characteristics of small slender shapes of needles, needles' variations in shapes and sizes, and demands for millimeter-level control. Prior works, heavily relying on the prior of needles (e.g., geometric models), are hard to scale to unseen needles' variations. In this paper, we present the first end- to-end learning method to train deep visuomotor policy for needle picking. Concretely, we propose DreamerfD to maximally leverage demonstrations to improve the learning efficiency of a state-of-the-art model-based reinforcement learning method, DreamerV2; Since Variational Auto-Encoder (VAE) in DreamerV2 is difficult to scale to high-resolution images, we propose Dynamic Spotlight Adaptation to represent control-related visual signals in a low-resolution image space; Virtual Clutch is also proposed to reduce per-formance degradation due to significant error between prior and posterior encoded states at the beginning of a rollout. We conducted extensive experiments in simulation to evaluate the performance, robustness, in-domain variation adaptation, and effectiveness of individual components of our method. Our method, trained by 8k demonstration timesteps and 140k online policy timesteps, can achieve a remarkable success rate of 80%. Furthermore, our method effectively demonstrated its superiority in generalization to unseen in-domain variations including needle variations and image disturbance, highlighting its robustness and versatility. Codes and videos are available at https://sites.google.com/view/DreamerfD.

Robotic bin packing is very challenging, especially when considering practical needs such as object variety and packing compactness. This paper presents SDF-Pack, a new approach based on signed distance field (SDF) to model the geometric condition of objects in a container and compute the object placement locations and packing orders for achieving a more compact bin packing. Our method adopts a truncated SDF representation to localize the computation, and based on it, we formulate the SDF -minimization heuristic to find optimized placements to compactly pack objects with the existing ones. To further improve space utilization, if the packing sequence is controllable, our method can suggest which object to be packed next. Experimental results on a large variety of everyday objects show that our method can consistently achieve higher packing compactness over 1,000 packing cases, enabling us to pack more objects into the container, compared with the existing heuristics under various packing settings. The code is publicly available at: https://github.com/kwpoon/SDF-Pack.

Offline evolutionary-based methodologies have supplied a successful motion planning framework for the quadrupedal jump. However, the time-consuming computation caused by massive population evolution in offline evolutionary-based jumping framework significantly limits the popularity in the quadrupedal field. This paper presents a time-friendly online motion planning framework based on meta-heuristic Differential evolution (DE), Latin hypercube sampling, and Configuration space (DLC). The DLC framework establishes a multidimensional optimization problem leveraging centroidal dynamics to determine the ideal trajectory of the center of mass (CoM) and ground reaction forces (GRFs). The configuration space is introduced to the evolutionary optimization in order to condense the searching region. Latin hypercube sampling offers more uniform initial populations of DE under limited sampling points, accelerating away from a local minimum. This research also constructs a collection of pre-motion trajectories as a warm start when the objective state is in the neighborhood of the pre-motion state to drastically reduce the solving time. The proposed methodology is successfully validated via real robot experiments for online jumping trajectory optimization with different jumping motions (e.g., ordinary jumping, flipping, and spinning).

Few-Shot 3D Point Cloud Object Detection (FS3D) is a challenging task, aiming to detect 3D objects of novel classes using only limited annotated samples for training. Considering that the detection performance highly relies on the quality of the latent features, we design a VAE-based prototype learning scheme, named prototypical VAE (P-VAE), to learn a probabilistic latent space for enhancing the diversity and distinctiveness of the sampled features. The network encodes a multi-center GMM-like posterior, in which each distribution centers at a prototype. For regularization, P-VAE incorporates a reconstruction task to preserve geometric information. To adopt P-VAE for the detection framework, we formulate Geometric-informative Prototypical VAE (GP-VAE) to handle varying geometric components and Class-specific Prototypical VAE (CP-VAE) to handle varying object categories. In the first stage, we harness GP-VAE to aid feature extraction from the input scene. In the second stage, we cluster the geometric-informative features into per-instance features and use CP-VAE to refine each instance feature with category-level guidance. Experimental results show the top performance of our approach over the state of the arts on two FS3D benchmarks. Quantitative ablations and qualitative prototype analysis further demonstrate that our probabilistic modeling can significantly boost prototype learning for FS3D.

Manhattan and Atlanta worlds hold for the structured scenes with only vertical and horizontal dominant directions (DDs). To describe the scenes with additional sloping DDs, a mixture of independent Manhattan worlds seems plausible, but may lead to unaligned and unrelated DDs. By contrast, we propose a novel structural model called Hong Kong world. It is more general than Manhattan and Atlanta worlds since it can represent the environments with slopes, e.g., a city with hilly terrain, a house with sloping roof, and a loft apartment with staircase. Moreover, it is more compact and accurate than a mixture of independent Manhattan worlds by enforcing the orthogonality constraints between not only vertical and horizontal DDs, but also horizontal and sloping DDs. We further leverage the structural regularity of Hong Kong world for the line-based SLAM. Our SLAM method is reliable thanks to three technical novelties. First, we estimate DDs/vanishing points in Hong Kong world in a semi-searching way. We use a new consensus voting strategy for search, instead of traditional branch and bound. This method is the first one that can simultaneously determine the number of DDs, and achieve quasi-global optimality in terms of the number of inliers. Second, we compute the camera pose by exploiting the spatial relations between DDs in Hong Kong world. This method generates concise polynomials, and thus is more accurate and efficient than existing approaches designed for unstructured scenes. Third, we refine the estimated DDs in Hong Kong world by a novel filter-based method. Then we use these refined DDs to optimize the camera poses and 3D lines, leading to higher accuracy and robustness than existing optimization algorithms. In addition, we establish the first dataset of sequential images in Hong Kong world. Experiments showed that our approach outperforms state-of-the-art methods in terms of accuracy and/or efficiency.

Image processing has significantly extended the practical value of the eye-in-hand camera, enabling and promoting its applications for quantitative measurement. However, fully vision-based pose estimation methods sometimes encounter difficulties in handling cases with deficient features. In this article, we fuse visual information with the sparse strain data collected from a single-core fiber inscribed with fiber Bragg gratings (FBGs) to facilitate continuum robot pose estimation. An improved extreme learning machine algorithm with selective training data updates is implemented to establish and refine the FBG-empowered (F-emp) pose estimatoronline. The integration of F-emp pose estimation can improve sensing robustness by reducing the number of times that visual tracking is lost given moving visual obstacles and varying lighting. In particular, this integration solves pose estimation failures under full occlusion of the tracked features or complete darkness. Utilizing the fused pose feedback, a hybrid controller incorporating kinematics and data-driven algorithms is proposed to accomplish fast convergence with high accuracy. The online-learning error compensator can improve the target tracking performance with a 52.3%–90.1% error reduction compared with constant-curvature model-based control, without requiring fine model-parameter tuning and prior data acquisition.

Soft robots have great potential in surgical applications due to their compliance and adaptability to their environment. However, their flexibility and nonlinearity bring challenges for precise modeling, sensing, and control, especially in constrained cavities. In this article, a simple, compact two-segment soft robot for flexible laser ablation is proposed. The proximal hydraulic-driven segment can offer omnidirectional bending so as to navigate toward lesions. The distal segment driven by tendons enables precise, fast steering of laser collimator for laser sweeping on lesion targets. The dynamics of such mechanical steering motion can be enhanced with a metal spring backbone integrated along the collimator, thus facilitating the control with certain linearity and responsiveness. A soft robot modeling and control scheme based on Koopman operators is proposed. We also design a disturbance observer so as to incorporate the controller feedback with real-time fiber optic shape sensing. Experimental validation is conducted on simulated orex-vivolaser ablation tasks, thus evaluating our control strategies in laser path following across various contours/patterns. As a result, such a simple compact laser manipulation can perform up to 6 Hz sweeping with precision of path following errors below 1 mm. Such modeling and control scheme could also be used on an endoscopic laser ablation robot with unsymmetric mechanism driven by two tendons.

Shape control of deformable objects is a challenging and important robotic problem. This article proposes a model-free controller using novel 3-D global deformation features based on modal analysis. Unlike most existing controllers using geometric features, our controller employs physically based deformation features designed by decoupling global deformation into low-frequency modes. Although modal analysis is widely adopted in computer vision and simulation, its usage in robotic deformation control is still an open topic. We develop a new model-free framework for the modal-based deformation control. Physical interpretation of the modes enables us to formulate an analytical deformation Jacobian matrix mapping the robot manipulation onto changes of the modal features. In the Jacobian matrix, unknown geometric and physical models of the object are treated as low-dimensional modal parameters, which can be used to linearly parameterize the closed-loop system. Thus, an adaptive controller with proven stability can be designed to deform the object while online estimating the modal parameters. Simulations and experiments are conducted using linear, planar, and volumetric objects under different settings. The results not only confirm the superior performance of our controller, but also demonstrate its advantages over the baseline method.

Achieving in-hand manipulation of a grasped object is challenging in robotic communities. This article proposes a planar intrinsic in-hand manipulation approach based on isometric transformation. We demonstrate that the isometric transformation (excluding reflections) can be reduced to rotations with two fixed centers, if more than two rotations around two different centers are performed. Furthermore, this article applies this theory to reposition and reorient a grasped object by performing three or more primitive rotations sequentially. We further analyze the feasible manipulation space, where the desired position and orientation of a grasped object can be attained. The theoretical results have been verified through simulations and experiments. In addition, we performed several real-world tasks using a novel robotic hand with two rotational degrees of freedom at each finger, demonstrating that the proposed in-hand manipulation method can reposition and reorient different grasped objects in$\text{SE}(2)$.

Manipulation in tight environment is challenging but increasingly common in vision-guided robotic applications. The significantly reduced amount of available feedback (limited visual cues, field of view, robot motion space, etc.) hinders solving the hand-eye relationship accurately. In this article, we propose a new generic approach for online robot–camera calibration that could deal with the least feedback input available in tight environment: an arbitrarily restricted motion space and a single feature point with unknown position for the robot end-effector. We introduce the interactive perception to generate prescribed but tunable robot motions to reveal high-dimensional sensory feedback, which is not obtainable from static images. We then define the interactive feature plane (IFP), whose spatial property corresponds to the robot-actuating trajectories. A depth-free adaptive controller is proposed based on image feedback, where the converged orientation of IFP directly harvests the data for solving the hand–eye relationship. Our algorithm requires neither external calibration sensors/objects nor large-scale data acquisition process. Simulations demonstrate the validity of our method to accurately calibrate different types of robot under various system set-ups. In experiments, we show good results of our algorithm in terms of accuracy and consistency under tight motion space compared to existing approaches using external objects and/or optimization.

Given a set of putative 3D-3D point correspondences, we aim to remove outliers and estimate rigid transformation with 6 degrees of freedom (DOF). Simultaneously estimating these 6 DOF is time-consuming due to high-dimensional parameter space. To solve this problem, it is common to decompose 6 DOF, i.e. independently compute 3-DOF rotation and 3-DOF translation. However, high non-linearity of 3-DOF rotation still limits the algorithm efficiency, especially when the number of correspondences is large. In contrast, we propose to decompose 6 DOF into$(2+1)$and$(1+2)\ DOF$. Specifically,$(2+1)DOF$represent 2-DOF rotation axis and 1-DOF displacement along this rotation axis.$(1+2)\ DOF$indicate 1-DOF rotation angle and 2-DOF displacement orthogonal to the above rotation axis. To compute these DOF, we design a novel two-stage strategy based on inlier set maximization. By leveraging branch and bound, we first search for$(2+1)\ DOF$, and then the remaining$(1+2)\ DOF$. Thanks to the proposed transformation decomposition and two-stage search strategy, our method is deterministic and leads to low computational complexity. We extensively compare our method with state-of-the-art approaches. Our method is more accurate and robust than the approaches that provide similar efficiency to ours. Our method is more efficient than the approaches whose accuracy and robustness are comparable to ours.

Industrial bin picking is a challenging task that requires accurate and robust segmentation of individual object instances. Particularly, industrial objects can have irregular shapes, that is, thin and concave, whereas in bin-picking scenarios, objects are often closely packed with strong occlusion. To address these challenges, we formulate a novel part-aware instance segmentation pipeline. The key idea is to decompose industrial objects into correlated approximate convex parts and enhance the object-level segmentation with part-level segmentation. We design a part-aware network to predict part masks and part-to-part offsets, followed by a part aggregation module to assemble the recognized parts into instances. To guide the network learning, we also propose an automatic label decoupling scheme to generate ground-truth part-level labels from instance-level labels. Finally, we contribute the first instance segmentation dataset, which contains a variety of industrial objects that are thin and have non-trivial shapes. Extensive experimental results on various industrial objects demonstrate that our method can achieve the best segmentation results compared with the state-of-the-art approaches.

Automatic laparoscope motion control is fundamentally important for surgeons to efficiently perform operations. However, its traditional control methods based on tool tracking without considering information hidden in surgical scenes are not intelligent enough, while the latest supervised imitation learning (IL)-based methods require expensive sensor data and suffer from distribution mismatch issues caused by limited demonstrations. In this paper, we propose a novel Imitation Learning framework for Laparoscope Control (ILLC) with reinforcement learning (RL), which can efficiently learn the control policy from limited surgical video clips. Specially, we first extract surgical laparoscope trajectories from unlabeled videos as the demonstrations and reconstruct the corresponding surgical scenes. To fully learn from limited motion trajectory demonstrations, we propose Shape Preserving Trajectory Augmentation (SPTA) to augment these data, and build a simulation environment that supports parallel RGB-D rendering to reinforce the RL policy for interacting with the environment efficiently. With adversarial training for IL, we obtain the laparoscope control policy based on the generated rollouts and surgical demonstrations. Extensive experiments are conducted in unseen reconstructed surgical scenes, and our method outperforms the previous IL methods, which proves the feasibility of our unified learning-based framework for laparoscope control.

Ultra-wideband (UWB) is a radio technology that is usually used for indoor localization. To complete the 3D localization task in a room, more than three UWB anchors should be installed without occlusion, then a UWB tag can be localized. An Inertial Measurement Unit (IMU) is usually fused to provide smoother results. However, the requirements for the installation of UWB anchors limit their application in a floor with multiple rooms and corridors. In this paper, we propose a localization method that needs fewer anchors in this environment. In addition to the range information from UWB anchors, non-holonomic constraints from the carrier and reference trajectory from the special environmental structure are added as observations of the error-state Extended Kalman Filter (EKF) system. The proposed method can complete the localization task in the corridor-like areas using a single UWB anchor, so fewer UWB anchors are needed in a floor with multiple rooms and corridors. The real-world experiment video is available at https://youtu.be/DT-02Id_94A.

Micro aerial vehicles usually have strap-down sensors on the vehicle body, leading to the severe coupling effect between perception and trajectory planning. As a result, visual-inertial simultaneous localization and mapping (VI-SLAM) technologies implemented on MAVs suffer from tracking failure problems, especially in featureless environments. To overcome these challenges, based on MAVs with movable camera mechanisms (e.g., gimbal stabilizer, pan-tilt, or bionic neck-eye system), we proposed two sampling-based algorithms for known and unknown environments respectively. The first active perception planning algorithm based on a scene richness model is developed with a built feature map for the environment. Differ from the first algorithm, the second one is modified for active localization in unknown 3D space. It is basically a time-based sampling-based approach that uses the same scene richness model. In addition, it also achieved a balance between exploitation and exploration. With the above solutions, the robustness of visual perception is improved while avoiding over-exploitation of known information. Simulation and real-world experiments are performed to verify the feasibility of our algorithms.

In this paper, we propose a novel variable-length estimation approach for shape sensing of extensible soft robots utilizing fiber Bragg gratings (FBGs). Shape reconstruction from FBG sensors has been increasingly developed for soft robots, while the narrow stretching range of FBG fiber makes it difficult to acquire accurate sensing results for extensible robots. Towards this limitation, we newly introduce an FBG-based length sensor by leveraging a rigid curved channel, through which FBGs are allowed to slide within the robot following its body extension/compression, hence we can search and match the FBGs with specific constant curvature in the fiber to determine the effective length. From the fusion with the above measurements, a model-free filtering technique is accordingly presented for simultaneous calibration of a variable-length model and temporally continuous length estimation of the robot, enabling its accurate shape sensing using solely FBGs. The performances of the proposed method have been experimentally evaluated on an extensible soft robot equipped with an FBG fiber in both free and unstructured environments. The results concerning dynamic accuracy and robustness of length estimation and shape sensing demonstrate the effectiveness of our approach.

This paper presents an optimal motion planning framework to generate versatile energy-optimal quadrupedal jumping motions automatically (e.g., flips, spin). The jumping motions via the centroidal dynamics are formulated as a 12-dimensional black-box optimization problem subject to the robot kino-dynamic constraints. Gradient-based approaches offer great success in addressing trajectory optimization (TO), yet, prior knowledge (e.g., reference motion, contact schedule) is required and results in sub-optimal solutions. The new proposed framework first employed a heuristics-based optimization method to avoid these problems. Moreover, a prioritization fitness function is created for heuristics-based algorithms in robot ground reaction force (GRF) planning, enhancing convergence and searching performance considerably. Since heuristics-based algorithms often require significant time, motions are planned offline and stored as a pre-motion library. A selector is designed to automatically choose motions with user-specified or perception information as input. The proposed framework has been successfully validated only with a simple continuously tracking PD controller in an open-source Mini-Cheetah by several challenging jumping motions, including jumping over a window-shaped obstacle with 30 cm height and left-flipping over a rectangle obstacle with 27 cm height. (Video*)

Estimating precise metric depth and scene reconstruction from monocular endoscopy is a fundamental task for surgical navigation in robotic surgery. However, traditional stereo matching adopts binocular images to perceive the depth information, which is difficult to transfer to the soft robotics-based surgical systems due to the use of monocular endoscopy. In this paper, we present a novel framework that combines robot kinematics and monocular endoscope images with deep unsupervised learning into a single network for metric depth estimation and then achieve 3D reconstruction of complex anatomy. Specifically, we first obtain the relative depth maps of surgical scenes by leveraging a brightness-aware monocular depth estimation method. Then, the corresponding endoscope poses are computed based on non-linear optimization of geo-metric and photometric reprojection residuals. Afterwards, we develop a Depth-driven Sliding Optimization (DDSO) algorithm to extract the scaling coefficient from kinematics and calculated poses offline. By coupling the metric scale and relative depth data, we form a robust ensemble that represents the metric and consistent depth. Next, we treat the ensemble as supervisory labels to train a metric depth estimation network for surgeries (i.e., MetricDepthS-Net) that distills the embeddings from the robot kinematics, endoscopic videos, and poses. With accurate metric depth estimation, we utilize a dense visual reconstruction method to recover the 3D structure of the whole surgical site. We have extensively evaluated the proposed framework on public SCARED and achieved comparable performance with stereo-based depth estimation methods. Our results demon-strate the feasibility of the proposed approach to recover the metric depth and 3D structure with monocular inputs.

Instance segmentation is an important task for supporting robotic grasping in auto-store scenarios. Accurate segmentation usually relies on the quantity and quality of available annotated training data. However, it requires tremendous cost to obtain these labels. In this work, without requiring any human annotations on real data, our proposed self-ensembling sim-to-real network, namely SESR, is able to generate precise instance masks for a wide variety of supermarket goods. We design our SESR with a teacher model and a student model trained with a self-ensembling strategy. We adopt different levels of consistency to bridge the sim-to-real gap and boost the model generalization ability. Also, we compile an auto-store bin-picking dataset covering various goods. Extensive experiments on both unseen scenarios and unseen objects validate the effectiveness and superiority of our method over others, and the robot arm demonstrations further show that our segmentation results can support real-time auto-store bin picking.

Image lines projected from parallel 3D lines intersect at a common point called the vanishing point (VP). Manhattan world holds for the scenes with three orthogonal VPs. In Manhattan world, given several lines in a calibrated image, we aim to cluster them by three unknown-but-sought VPs. The VP estimation can be reformulated as computing the rotation between the Manhattan frame and camera frame. To estimate three degrees of freedom (DOF) of this rotation, state-of-the-art methods are based on either data sampling or parameter search. However, they fail to guarantee high accuracy and efficiency simultaneously. In contrast, we propose a set of approaches that hybridize these two strategies. We first constrain two or one DOF of the rotation by two or one sampled image line. Then we search for the remaining one or two DOF based on branch and bound. Our sampling accelerates our search by reducing the search space and simplifying the bound computation. Our search achieves quasi-global optimality. Specifically, it guarantees to retrieve the maximum number of inliers on the condition that two or one DOF is constrained. Our hybridization of two-line sampling and one-DOF search can estimate VPs in real time. Our hybridization of one-line sampling and two-DOF search can estimate VPs in near real time. Experiments on both synthetic and real-world datasets demonstrated that our approaches outperform state-of-the-art methods in terms of accuracy and/or efficiency.

This paper proposes a novel pretext task to address the self-supervised video representation learning problem. Specifically, given an unlabeled video clip, we compute a series of spatio-temporal statistical summaries, such as the spatial location and dominant direction of the largest motion, the spatial location and dominant color of the largest color diversity along the temporal axis, etc. Then a neural network is built and trained to yield the statistical summaries given the video frames as inputs. In order to alleviate the learning difficulty, we employ several spatial partitioning patterns to encode rough spatial locations instead of exact spatial Cartesian coordinates. Our approach is inspired by the observation that human visual system is sensitive to rapidly changing contents in the visual field, and only needs impressions about rough spatial locations to understand the visual contents. To validate the effectiveness of the proposed approach, we conduct extensive experiments with four 3D backbone networks, i.e., C3D, 3D-ResNet, R(2+1)D and S3D-G. The results show that our approach outperforms the existing approaches across these backbone networks on four downstream video analysis tasks including action recognition, video retrieval, dynamic scene recognition, and action similarity labeling. The source code is publicly available at: https://github.com/laura-wang/video_repres_sts.

Surgical robots have been widely adopted with over 4000 robots being used in practice daily. However, these are telerobots that are fully controlled by skilled human surgeons. Introducing "surgeon-assist"-some forms of autonomy-has the potential to reduce tedium and increase consistency, analogous to driver-assist functions for lanekeeping, cruise control, and parking. This article examines the scientific and technical backgrounds of robotic autonomy in surgery and some ethical, social, and legal implications. We describe several autonomous surgical tasks that have been automated in laboratory settings, and research concepts and trends.

Sanding is a common yet important task in the manufacturing of many wooden objects (e.g. furniture, decoration box), where the coated layer attached on objects is removed after the interaction with sanding belts. Existing sanding operation is heavily dependent on manual works, which is highly labor-intensive and with the low consistency of quality, and the issues of safety and health also arise after continuous working in the noisy and dusty environment. To deal with the aforementioned, this paper presents the development of a new autonomous sanding robot. The autonomous capability of the developed robot is reflected in the whole procedure of sanding. In particular, the CAD model of the target object is automatically constructed with the structured-light technology, and the sanding behavior on the target surface is self-regulated under the desired impedance model. Such feature makes the robot capable of working towards uncertain objects with minimum human involvement. The proposed impedance controller has the model-free advantage, by using the adaptive neural networks (NNs) to compensate the uncertain dynamics and the unknown disturbances online. The stability of the closed-loop system is rigorously proved with Lyapunov methods, and experimental results on different objects are presented to validate the performance of the developed robot. The implementation of the developed robot can systematically address the problems associated with manual works. Note to Practitioners—The current working environment of sanding is not healthy or safe to humans, due to the nature of noise, dust, high-speed sanding belt. By controlling the robot to autonomously perform sanding tasks can keep humans away from such environment and hence systematically address the issues of health and safety. This paper presents a new impedance control method for sanding robot. In impedance control, the control goal is specified as a dynamic relationship between the contact position and the interaction force. In this paper, the contact position is determined by using the 3D vision sensor, while the interaction force is regulated by referring to the human experience. In addition, the influence caused by unmodeled factors (e.g. unknown dynamics of the sanded object) is also dealt with by using the techniques of NNs. Such setting can effectively guarantee the sanding quality and also avoid the physical damage to the sanded object. Therefore, it lays the foundation for the autonomous robotic sanding.

To realize a higher-level autonomy of surgical knot tying in minimally invasive surgery (MIS), automated suture grasping, which bridges the suture stitching and looping procedures, is an important yet challenging task needs to be achieved. This paper presents a holistic framework with image-guided and automation techniques to robotize this operation even under complex environments. The whole task is initialized by suture segmentation, in which we propose a novel semi-supervised learning architecture featured with a suture-aware loss to pertinently learn its slender information using both annotated and unannotated data. With successful segmentation in stereo-camera, we develop a Sampling-based Sliding Pairing (SSP) algorithm to online optimize the suture’s 3D shape. By jointly studying the robotic configuration and the suture’s spatial characteristics, a target function is introduced to find the optimal grasping pose of the surgical tool with Remote Center of Motion (RCM) constraints. To compensate for inherent errors and practical uncertainties, a unified grasping strategy with a novel vision-based mechanism is introduced to autonomously accomplish this grasping task. Our framework is extensively evaluated from learning-based segmentation, 3D reconstruction, and image-guided grasping on the da Vinci Research Kit (dVRK) platform, where we achieve high performances and successful rates in perceptions and robotic manipulations. These results prove the feasibility of our approach in automating the suture grasping task, and this work fills the gap between automated surgical stitching and looping, stepping towards a higher-level of task autonomy in surgical knot tying. Note to Practitioners—This paper aims to automate the suture grasping task in surgical knot tying by leveraging stereo visual guidance. To effectively robotize this procedure, it requires multidisciplinary knowledge to achieve suture segmentation, 3D shape reconstruction, and reliable automated grasping, while there are no existing works tackling this procedure especially using robots with RCM kinematics constraints and under complex environments. In this article, we propose a learning-driven method along with a 3D shape optimizer, which can conduct the suture segmentation and output its accurate spatial coordinates, serving as guidance for automated grasping operation. Apart from this, we introduce a unified function to optimize the grasping pose, and a vision-based grasping strategy is also proposed to intelligently complete this task. The experiments extensively validate the feasibility of our framework for automated suture grasp, and its successful completion can serve as a basis for the following looping manipulation, hence filling a step gap in robot-assisted knot tying. This framework can be also encapsulated into the medical robotic system, and by simply indicating (e.g. mouse click) the rough position of the suture’s tip in one camera frame, the overall framework can be initialized and further accomplish the suture grasping task, which further prompts a full autonomy of surgical knot tying in the near future.

We consider the uncalibrated vision-based control problem of robotic manipulators in this work. Though lots of approaches have been proposed to solve this problem, they usually require calibration (offline or online) of the camera parameters in the implementation, and the control performance may be largely affected by parameter estimation errors. In this work, we present new fully uncalibrated visual servoing approaches for position control of the 2DOFs planar manipulator with a fixed camera. In the proposed approaches, no camera calibration is required, and numerical optimization algorithms or adaptive laws for parameter estimation are not needed. One benefit of such features is that exponential convergence of the image position errors can be ensured regardless of the camera parameter uncertainties. Generally, existing uncalibrated approaches only can guarantee asymptotical convergence of the position errors. Moreover, different from most existing approaches which assume that the robot motion plane and the image plane are parallel, one of the proposed approaches allows the camera to be installed at a general pose. This also simplifies the controller implementation and improves the system design flexibility. Finally, simulation and experimental results are provided to illustrate the effectiveness of the presented fully uncalibrated visual servoing approaches.

Three-dimensional (3D) reconstruction of dynamic objects has broad applications, including object recognition and robotic manipulation. However, achieving high-accuracy reconstruction and robustness to motion simultaneously is a challenging task. In this paper, we present a novel method for 3D reconstruction of dynamic objectS, whose main features are as follows. Firstly, a structured-light multiplexing method is developed that only requires 3 patterns to achieve high-accuracy encoding. Fewer projected patterns require shorter image acquisition time, thus, the object motion is reduced in each reconstruction cycle. The three patterns, i.e. spatial-temporally encoded patterns, are generated by embedding a specifically designed spatial-coded texture map into the temporal-encoded three-step phase-shifting fringes. A temporal codeword and three spatial codewords are extracted from the composite patterns using a proposed extraction algorithm. The two types of codewords are utilized separately in stereo matching: the temporal codeword ensures the high accuracy, while the spatial codewords are responsible for removing phase ambiguity. Secondly, we aim to eliminate the reconstruction error induced by motion between frames abbreviated as motion induced error (MiE). Instead of assuming the object to be static when acquiring the 3 images, we derive the motion of projection pixels among frames. Using the extracted spatial codewords, correspondences between different frames are found, i.e. pixels with the same codewords are traceable in the image sequences. Therefore, we can obtain the phase map at each image-acquisition moment without being affected by the object motion. Then the object surfaces corresponding to all the images can be recovered. Experimental results validate the high reconstruction accuracy and precision of the proposed method for dynamic objects with different motion speeds. Comparative experiments show that the presented method demonstrates superior performance with various types of motion, including translation in different directions and deformation.

Given a set of putative 2D-2D line correspondences, we aim to identify correct matches. Existing methods exploit the geometric constraints. They are only applicable to structured scenes with orthogonality, parallelism and coplanarity. In contrast, we propose the first approach suitable for both structured and unstructured scenes. Instead of geometric constraint, we leverage the spatial regularity on sphere. Specifically, we propose to map line correspondences into vectors tangent to sphere. We use these vectors to encode both angular and positional variations of image lines, which is more reliable and concise than directly using inclinations, midpoints or endpoints of image lines. Neighboring vectors mapped from correct matches exhibit a spatial regularity called local trend consistency, regardless of the type of scenes. To encode this regularity, we design a neural network and also propose a novel loss function that enforces the smoothness constraint of vector field. In addition, we establish a large real-world dataset for image line matching. Experiments showed that our approach outperforms state-of-the-art ones in terms of accuracy, efficiency and robustness, and also leads to high generalization.

Existing vanishing point (VP) estimation methods rely on pre-extracted image lines and/or prior knowledge of the number of VPs. However, in practice, this information may be insufficient or unavailable. To solve this problem, we propose a network that treats a perspective image as input and predicts a spherical probability map of VP. Based on this map, we can detect all the VPs. Our method is reliable thanks to four technical novelties. First, we leverage the icosahedral spherical representation to express our probability map. This representation provides uniform pixel distribution, and thus facilitates estimating arbitrary positions of VPs. Second, we design a loss function that enforces the antipodal symmetry and sparsity of our spherical probability map to prevent over-fitting. Third, we generate the ground truth probability map that reasonably expresses the locations and uncertainties of VPs. This map unnecessarily peaks at noisy annotated VPs, and also exhibits various anisotropic dispersions. Fourth, given a predicted probability map, we detect VPs by fitting a Bingham mixture model. This strategy can robustly handle close VPs and provide the confidence level of VP useful for practical applications. Experiments showed that our method achieves the best compromise between generality, accuracy, and efficiency, compared with state-of-the-art approaches.

This paper presents a comprehensive inertial aided 3D LiDAR SLAM system with hybrid geometric primitives in large-scale environments, including a tightly-coupled LiDAR-Inertial-Odometry (LIO), a global mapping module supported by learning-based loop closure detection and a sub-maps matching algorithm. An efficient method is developed to simultaneously extract explicit plane features and point features from each raw point cloud. To make full use of the structural information of the surroundings, plane features and point features (ground and edge) are tracked across a fix-sized group of LiDAR keyframes in the local map. For effective loop closure detection in large-scale environments, we integrate the learning-based point cloud network and a keyframe sequence matching method to detect loops. Finally, a novel, deterministic and near real-time plane-driven sub-maps matching algorithm is proposed to close the loops. The proposed SLAM system is validated with experiments on different types of environments.

Laparoscopic Field of View (FOV) control is one of the most fundamental and important components in Minimally Invasive Surgery (MIS), nevertheless the traditional manual holding paradigm may easily bring fatigue to surgical assistants, and misunderstanding between surgeons also hinders assistants to provide a high-quality FOV. Targeting this problem, we here present a data-driven framework to realize an automated laparoscopic optimal FOV control. To achieve this goal, we offline learn a motion strategy of laparoscope relative to the surgeon’s hand-held surgical tool from our in-house surgical videos, developing our control domain knowledge and an optimal view generator. To adjust the laparoscope online, we first adopt a learning-based method to segment the two-dimensional (2D) position of the surgical tool, and further leverage this outcome to obtain its scale-aware depth from dense depth estimation results calculated by our novel unsupervised RoboDepth model only with the monocular camera feedback, hence in return fusing the above real-time 3D position into our control loop. To eliminate the misorientation of FOV caused by Remote Center of Motion (RCM) constraints when moving the laparoscope, we propose a novel rotation constraint using an affine map to minimize the visual warping problem, and a null-space controller is also embedded into the framework to optimize all types of errors in a unified and decoupled manner. Experiments are conducted using Universal Robot (UR) and Karl Storz Laparoscope/Instruments, which prove the feasibility of our domain knowledge and learning enabled framework for automated camera control.

Automatic surgical gesture recognition is fundamentally important to enable intelligent cognitive assistance in robotic surgery. With recent advancement in robot-assisted minimally invasive surgery, rich information including surgical videos and robotic kinematics can be recorded, which provide complementary knowledge for understanding surgical gestures. However, existing methods either solely adopt uni-modal data or directly concatenate multi-modal representations, which can not sufficiently exploit the informative correlations inherent in visual and kinematics data to boost gesture recognition accuracies. In this regard, we propose a novel online approach of multi-modal relational graph network (i.e., MRG-Net) to dynamically integrate visual and kinematics information through interactive message propagation in the latent feature space. In specific, we first extract embeddings from video and kinematics sequences with temporal convolutional networks and LSTM units. Next, we identify multi-relations in these multi-modal embeddings and leverage them through a hierarchical relational graph learning module. The effectiveness of our method is demonstrated with state-of-the-art results on the public JIGSAWS dataset, outperforming current uni-modal and multi-modal methods on both suturing and knot typing tasks. Furthermore, we validated our method on in-house visual-kinematics datasets collected with da Vinci Research Kit (dVRK) platforms in two centers, with consistent promising performance achieved. Our code and data are released at: https://www.cse.cuhk.edu.hk/~yhlong/mrgnet.html.

Soft robots have been applied widely to various constrained scenarios due to the advantages over traditional rigid manipulators such as softness, deformability and adaptability to constrained surroundings. To make full use of this merit, this paper proposes a method that integrates collision detection, localization and force estimation for a cable-driven soft manipulator without any prior geometrical knowledge of its surroundings. First of all, a collision detection algorithm is presented based upon Cosserat-rod statics by a threshold method through using the cable tension and the shape information, which are obtained by the load cells and the Vicon system, respectively. Secondly, a collision localization and force estimation method is proposed through optimizing the discrepancy between the actual and the theoretical shapes. Finally, experiments are carried out to validate these algorithms. The experimental results demonstrate that the site, the magnitude as well as the direction can be estimated.

Surgical instrument segmentation in robot-assisted surgery (RAS) - especially that using learning-based models - relies on the assumption that training and testing videos are sampled from the same domain. However, it is impractical and expensive to collect and annotate sufficient data from every new domain. To greatly increase the label efficiency, we explore a new problem, i.e., adaptive instrument segmentation, which is to effectively adapt one source model to new robotic surgical videos from multiple target domains, only given the annotated instruments in the first frame. We propose MDAL, a meta-learning based dynamic online adaptive learning scheme with a two-stage framework to fast adapt the model parameters on the first frame and partial subsequent frames while predicting the results. MDAL learns the general knowledge of instruments and the fast adaptation ability through the video-specific meta-learning paradigm. The added gradient gate excludes the noisy supervision from pseudo masks for dynamic online adaptation on target videos. We demonstrate empirically that MDAL outperforms other state-of-the-art methods on two datasets (including a real-world RAS dataset). The promising performance on ex-vivo scenes also benefits the downstream tasks such as robot-assisted suturing and camera control.

Autonomous grasping is an important factor for robots physically interacting with the environment and executing versatile tasks. However, a universally applicable, cost-effective, and rapidly deployable autonomous grasping approach is still limited by those target objects with fuzzy-depth information. Examples are transparent, specular, flat, and small objects whose depth is difficult to be accurately sensed. In this work, we present a solution to those fuzzy-depth objects. The framework of our approach includes two major components: one is a soft robotic hand and the other one is a Fuzzy-depth Soft Grasping (FSG) algorithm. The soft hand is replaceable for most existing soft hands/grippers with body compliance. FSG algorithm exploits both RGB and depth images to predict grasps while not trying to reconstruct the whole scene. Two grasping primitives are designed to further increase robustness. The proposed method outperforms reference baselines in unseen fuzzy-depth objects grasping experiments (84% success rate).

In this paper, we presented a new method for deformation control of deformable objects, which utilizes both visual and tactile feedback. At present, manipulation of deformable objects is basically formulated by assuming positional constraints. But in fact, in many situations manipulation has to be performed under actively applied force constraints. This scenario is considered in this research. In the proposed scheme a tactile feedback is integrated to ensure a stable contact between the robot end-effector and the soft object to be manipulated. The controlled contact force is also utilized to regulate the deformation of the soft object with its shape measured by a vision sensor. The effectiveness of the proposed method is demonstrated by a book page turning and shaping experiment.

Embryos/oocytes vitrification is an essential cryopreservation technique in IVF (in vitro fertilization) clinics. The reliable and effective transferring of embryos/oocytes is crucial to the subsequent steps in the whole procedure of vitrification. After each transferring, the straw needs to be replaced with a new one. Due to the uncertainties in the fabrication and installation, the exact knowledge of the kinematic model of the straw is usually unknown, and the relationship between the microscope and the straw is also unknown without calibration beforehand. In such situation, automatically transferring the oocytes from micropipette to the narrow tip of straw (0.7mm) is very challenging. In this paper, a new vision-guided robotic system is developed to automate the transferring of the oocyte without calibration. To this end, the unknown depth information is estimated then compensated by constructing a deep vision network through microscope image, and an approximate Jacobian control algorithm is also proposed to servo control the end tip of the uncalibrated straw to contact the micropipette with the vision feedback. After that, the oocyte is automatically transferred from the micropipette to the straw to finalize the task. The stability of the closed-loop control system is rigorously proved with Lyapunov methods, and the effectiveness of the developed robot is validated in experiments.

Autonomous delivery of suspended payloads with MAVs has many applications in rescue and logistics transportation. Robust and online estimation of the payload status is important but challenging especially in outdoor environments. The paper develops a novel real-time system for estimating the payload position; the system consists of a monocular fisheye camera and a novel encoder-based device. A Gaussian fusion-based estimation algorithm is developed to obtain the payload state estimation. Based on the robust payload position estimation, a payload controller is presented to ensure the re-liable tracking performance on aggressive trajectories. Several experiments are performed to validate the high performance of the proposed method.

Autonomous surgical execution relieves tedious routines and surgeon’s fatigue. Recent learning-based methods, especially reinforcement learning (RL) based methods, achieve promising performance for dexterous manipulation, which usually requires the simulation to collect data efficiently and reduce the hardware cost. The existing learning-based simulation platforms for medical robots suffer from limited scenarios and simplified physical interactions, which degrades the real-world performance of learned policies. In this work, we designed SurRoL, an RL-centered simulation platform for surgical robot learning compatible with the da Vinci Research Kit (dVRK). The designed SurRoL integrates a user-friendly RL library for algorithm development and a real-time physics engine, which is able to support more PSM/ECM scenarios and more realistic physical interactions. Ten learning-based surgical tasks are built in the platform, which are common in the real autonomous surgical execution. We evaluate SurRoL using RL algorithms in simulation, provide in-depth analysis, deploy the trained policies on the real dVRK, and show that our SurRoL achieves better transferability in the real world.

In this article, we focus on resolving the traffic flow prediction, robot path planning, and motion coordination problems in large-scale warehousing robotics systems with thousand-robot networks. The warehousing environment is partitioned into several sectors, and a hierarchical framework is developed, which includes a centralized prediction and planning level and a decentralized local coordination level. In the centralized level, a traffic flow prediction algorithm is first proposed to predict the evolution of the robot density distribution in a future horizon and estimate the future traffic heat value of each sector. Based on this, the sector-level robot path can be generated in the time-expended sector graph by comprehensively considering the traveling distance and the predicted traffic heat value and will be dynamically updated by considering the most recent traffic information. In the coordination level, local cooperative A* algorithm, incorporated with the conflict-based searching strategy, is implemented within each sector to generate conflict-free road-level paths for all the robots in the sector simultaneously, and the rolling planning scheme is utilized in order to immediately react to robot motion uncertainties and communication disconnections. The effectiveness and practical applicability of the proposed approach are validated by large-scale simulations with more than one 1000 robots and real laboratory experiments.Note to Practitioners—Considering practical situations and requirements in industrial warehouses and automated logistics systems, this article resolves the life-long planning and coordination problems of large-scale robot networks and ensures the practical execution performance in the presence of robot motion uncertainties and temporary communication disconnections. Our main idea is to reduce robot congestions and improve warehouse working efficiency by balancing the traffic flow in the whole environment. To achieve this, we present a traffic flow prediction algorithm to estimate the robot density distribution in a future horizon and take this information into consideration in sector-level path planning. The reliability, scalability, and the real-time performance of the proposed solution are achieved by the presented hierarchical system framework and the dynamic planning scheme. The proposed concept and approach can also be used to coordinate other large-scale systems with multirobot or multi-AGV networks. Simulation and experimental results suggest that the proposed solution is effective and practically applicable, but a saturation phenomenon of the system capacity can be observed under a very heavy workload. In the future, we will investigate the relation between the maximum system capacity and the environment structure and make further efforts to optimize the environment structure and road layout in order to improve the warehouse working efficiency.

This study investigates the use of a tendon-driven continuum robot to deform a soft object, whereas the robot body is deformed into an arbitrary shape to adapt to a constrained environment. A dynamic estimator (DE) is developed to approximate the Jacobian matrix that associates the actuator input with the deformed output of the soft object. This helps solve the singularity problem and reduce the effects of noise. Then a visual predictive controller (VPC) with a reference trajectory is developed to ensure a smooth operation. A linear extended-state observer (ESO) is further designed to measure the robot states, such that the controller can compensate for the estimation error. Simulations and experiments are performed to verify the proposed control approach.Note to Practitioners—The motivation of this article is to solve the problem of automatic deformation control of soft objects in restricted environments. The existing soft object deformation control is achieved using rigid robots in an open environment, but rigid robots are difficult to use in specific applications where the environment is restricted (e.g., natural orifice surgery). Flexible continuum robots with mechanical compliance can manipulate soft objects in narrow spaces. However, due to environmental constraints, the robot body may be deformed into any shape regardless of the input of the actuator. To solve the problem, this research provides a new visual servo control strategy that deforms soft objects using a continuum robot in a restricted environment. The proposed method can control a flexible robot to manipulate soft objects while taking into account the change in the robot configuration in a restricted environment.

The problem of consensus in networked agent systems is revisited and applied to vision-based localization. A class of new consensus dynamics is introduced first, and sufficient conditions including the persistence of excitation on the coupling matrix for reaching consensus are derived. As an application of the proposed consensus dynamics, an adaptive localization algorithm then is proposed for autonomous robots equipped with primarily visual sensors in GPS-denied environments. In the context of consensus over an undirected tree topology, the convergence of the proposed localization algorithm is proved. Finally, both numerical simulations and physical experiments are presented to show the effectiveness of the proposed localization algorithm. Our algorithm is simpler to implement and computationally cheaper compared to other localization methods. Moreover, it is immune to error accumulation and long-term stable, and the asymptotical convergence of the estimation errors can be theoretically guaranteed.

Given a set of 3D-to-2D point correspondences corrupted by outliers, we aim to robustly estimate the absolute camera pose. Existing methods robust to outliers either fail to guarantee high robustness and efficiency simultaneously, or require an appropriate initial pose and thus lack generality. In contrast, we propose a novel approach based on the robust "L2-minimizing estimate" (L2E) loss. We first define a novel cost function by integrating the projection constraint into the L2E loss. Then to efficiently obtain the global minimum of this function, we propose a hybrid strategy of a local optimizer and branch-and-bound. For branch-and-bound, we derive effective function bounds. Our approach can handle high outlier ratios, leading to high robustness. It can run reliably regardless of whether the initial pose is appropriate, providing high generality. Moreover, given a decent initial pose, it is suitable for real-time applications. Experiments on synthetic and real-world datasets showed that our approach outperforms state-of-the-art methods in terms of robustness and/or efficiency.

Cooperative adaptive cruise control (CACC) of intelligent vehicles contributes to improving cruise control performance, reducing traffic congestion, saving energy and increasing traffic flow capacity. In this paper, we resolve the CACC problem from the viewpoint of synchronization control, our main idea is to introduce the spatial-temporal synchronization mechanism into vehicle platoon control to achieve the robust CACC and to further realize the non-stop intersection control. Firstly, by introducing the cross-coupling based space synchronization mechanism, a distributed control algorithm is presented to achieve the single-lane CACC in the presence of vehicle-to-vehicle (V2V) communications, which enables autonomous vehicles to track the desired platoon trajectory while synchronizing their longitudinal velocities to keeping the expected inter-vehicle distance. Secondly, by designing the enter-time scheduling mechanism (temporal synchronization), a high-level intersection control strategy is proposed to command vehicles to form a virtual platoon to pass through the intersection without stopping. Thirdly, a Lyapunov-based time-domain stability analysis approach is presented. Compared with the traditional string stability based approach, the proposed approach guarantees the global asymptotical convergence of the proposed CACC system. Experiments in the small-scale simulated system demonstrate the effectiveness of the proposed approach.

Three-dimensional reconstruction of dynamic objects is important for robotic applications, for example, the robotic recognition and manipulation. In this paper, we present a novel 3D surface reconstruction method for moving objects. The proposed method combines the spatial-multiplexing and time-multiplexing structured-light techniques that have advantages of less image acquisition time and accurate 3D reconstruction, respectively. A set of spatial-temporal encoded patterns are designed, where a spatial-encoded texture map is embedded into the temporal-encoded three-step phase-shifting fringes. The specifically designed spatial-coded texture assigns high-uniqueness codeword to any window on the image which helps to eliminate the phase ambiguity. In addition, the texture is robust to noise and image blur. Combining this texture with high-frequency phase-shifting fringes, high reconstruction accuracy would be ensured. This method only requires 3 patterns to uniquely encode a surface, which facilitates the fast image acquisition for each reconstruction step. A filtering stereo matching algorithm is proposed for the spatial-temporal multiplexing method to improve the matching reliability. Moreover, the reconstruction precision is further enhanced by a correspondence refinement algorithm. Experiments validate the performance of the proposed method including the high accuracy, the robustness to noise and the ability to reconstruct moving objects.

This paper focuses on designing, modeling and validating a novel scalable multicopter whose deformation mechanism, called SNIAE-SSE, relies on a combination of simple non-intersecting angulated elements (SNIAEs) and straight scissor-like elements (SSEs). The proposed SNIAE-SSE mechanism has the advantages of single degree-of-freedom, fast actuation capability and large deformation ratio. In this work, enabled by the SNIAE-SSE mechanism, a quadcopter prototype with symmetrical and synchronous deformation is firstly developed, which facilitates a novel and controllably scalable multicopter system for us to analyze its modeling, as well as to validate its flight performance and dynamics during the deformation in several flight missions including hover, throwing, and morphing flying through a narrow window. Experimental results demonstrate that the developed scalable multicopter can maintain its stable flight behavior even both the folding and unfolding body deformations are fast performed, which indicates an excellent capability of the scalable multicopter to rapidly adapt to complex and dynamically changed environments.

This paper presents a novel solution for online trajectory planning of a full-size tractor-trailers vehicle composed of a car-like tractor and arbitrary number of passive full trailers. The motion planning problem for such systems was rarely addressed due to the complex nonlinear dynamics. A simulation-based prediction method is proposed to easily handle the complicated nonlinear dynamics and efficiently generate the obstacle-free and dynamically feasible trajectories. The vehicle dynamics model and a two-layer controller are used in the prediction. Implementation results on the real-world full-size industrial tractor-trailers vehicle are presented to validate the performance of the proposed methods.

In industry assembly lines, parts feeding machines are widely employed as the prologue of the whole procedure. They play the role of sorting the parts randomly placed in bins to the state with specified pose. With the help of the parts feeding machines, the subsequent assembly processes by robot arm can always start from the same condition. Thus it is expected that function of parting feeding machine and the robotic assembly can be integrated with one robot arm. This scheme can provide great flexibility and can also contribute to reduce the cost. The difficulties involved in this scheme lie in the fact that in the part feeding phase, the pose of the part after grasping may be not proper for the subsequent assembly. Sometimes it can not even guarantee a stable grasp. In this paper, we proposed a method to integrate parts feeding and assembly within one robot arm. This proposal utilizes a specially designed gripper tip mounted on the jaws of a two-fingered gripper. With the modified gripper, in-hand manipulation of the grasped object is realized, which can ensure the control of the orientation and offset position of the grasped object. The proposal in this paper is verified by a simulated assembly in which a robot arm completed the assembly process including parts picking from bin and a subsequent peg-in-hole assembly.

This paper presents a novel dataset targeting three types of challenging environments for autonomous driving, i.e., the industrial logistics environment, the undulating hill environment and the mixed complex urban environment. To the best of the author's knowledge, similar dataset has not been published in the existing public datasets, especially for the logistics environment collected in the functioning Hong Kong Air Cargo Terminal (HACT). Structural changes always suddenly appeared in the airport logistics environment due to the frequent movement of goods in and out. In the structureless and noisy hill environment, the non-flat plane movement is usual. In the mixed complex urban environment, the highly dynamic residence blocks, sloped roads and highways are included in a single collection. The presented dataset includes LiDAR, image, IMU and GPS data by repeatedly driving along several paths to capture the structural changes, the illumination changes and the different degrees of undulation of the roads. The baseline trajectories are provided which are estimated by Simultaneous Localization and Mapping (SLAM).

Surgical knot tying is one of the most fundamental and important procedures in surgery, and a high-quality knot can significantly benefit the postoperative recovery of the patient. However, a longtime operation may easily cause fatigue to surgeons, especially during the tedious wound closure task. In this paper, we present a vision-based method to automate the suture thread grasping, which is a sub-task in surgical knot tying and an intermediate step between the stitching and looping manipulations. To achieve this goal, the acquisition of a suture's three-dimensional (3D) information is critical. Towards this objective, we adopt a transfer-learning strategy first to fine-tune a pre-trained model by learning the information from large legacy surgical data and images obtained by the onsite equipment. Thus, a robust suture segmentation can be achieved regardless of inherent environment noises. We further leverage a searching strategy with termination policies for a suture's sequence inference based on the analysis of multiple topologies. Exact results of the pixel-level sequence along a suture can be obtained, and they can be further applied for a 3D shape reconstruction using our optimized shortest path approach. The grasping point considering the suturing criterion can be ultimately acquired. Experiments regarding the suture 2D segmentation and ordering sequence inference under environmental noises were extensively evaluated. Results related to the automated grasping operation were demonstrated by simulations in V-REP and by robot experiments using Universal Robot (UR) together with the da Vinci Research Kit (dVRK) adopting our learning-driven framework.

Robot-assisted vitreoretinal surgery can filter surgeons' hand tremors and provide safe, accurate tool manipulation. In this paper, we report the design, optimization, and evaluation of a novel tilt mechanism for a new Steady-Hand Eye Robot (SHER). The new tilt mechanism features a four-bar linkage design and has a compact structure. Its kinematic configuration is optimized to minimize the required linear range of motion (LRM) for implementing a virtual remote center-of-motion (V-RCM) while tilting a surgical tool. Due to the different optimization constraints for the robots at the left and right sides of the human head, two configurations of this tilt mechanism are proposed. Experimental results show that the optimized tilt mechanism requires a significantly smaller LRM (e.g. 5.08 mm along Z direction and 8.77 mm along Y direction for left side robot) as compared to the slider-crank tilt mechanism used in the previous SHER (32.39 mm along Z direction and 21.10 mm along Y direction). The feasibility of the proposed tilt mechanism is verified in a mock bilateral robot-assisted vitreoretinal surgery. The ergonomically acceptable robot postures needed to access the surgical field is also determined.

Robotic hands with anthropomorphism considerations are of prominent popularity in human-centered environment. Existing anthropomorphic robotic hands achieving part or most of human hand comparable dexterity have been applied as various robotic end-effectors and prosthetics. However, two deficiencies are evident that the design for a dexterous anthropomorphic hand is largely based on the intuition of designers and the dexterity of robotic hand is hard to evaluate. To tackle these two challenges, this paper summarizes 50 hand dexterity benchmarks (HD-marks) to evaluate hand dexterity comprehensively from three perspectives. Secondly, a novel 22-DOFs soft robotic hand (S-22) replicates human hand kinematics is used to demonstrate all the 50 HD-marks. Thirdly, 7 critical joint-based kinematic motions (K-motions) and their correlation with the 50 HD-marks are established. Therefore, a clear robotic hand design guideline is built by mapping the hand functional dexterity to the required joint kinematics.

In this paper, we propose a dynamic state estimation framework for lateral control of a heavy tractor-trailers system using only mass-produced low-cost sensors. This issue is challenging since the lateral velocity of the lead tractor is difficult to measure directly. The performance of existing dynamic model-based estimation methods will also be degraded, as different trailers and payloads cause the tractor model parameters to change. We address this issue by incorporating a kinematic estimator into a dynamic model-based estimation scheme. Accurate and reliable tire cornering stiffness and dynamics-informed lateral velocity of the lead tractor can be output in real-time by using our method. The stability and robustness of the proposed method are theoretically proved. The feasibility of our method is verified by full-scale experiments. It is also verified that the estimated model parameters and lateral states do improve the control performance by integrating the estimator into a lateral control system.

To make the controller implementation easier and to enhance the system robustness and control performance in the presence of the camera parameter uncertainties, it is very desired to develop vision-based control approaches without any offline or online camera calibration. In this article, we propose a new calibration-free image-based trajectory tracking control scheme for nonholonomic mobile robots with a truly uncalibrated fixed camera. By developing a novel camera-parameter-independent kinematic model, both offline and online camera calibration can be avoided in the proposed scheme, and any knowledge of the camera is not needed in the controller design. The proposed trajectory tracking control scheme can guarantee exponential convergence of the image position and velocity tracking errors. To illustrate the performance of the proposed scheme, experimental results are provided in this article. Note to Practitioners-This article was motivated by the vision-based motion control problem of mobile robots in uncalibrated environments. Existing vision-based motion control approaches for nonholonomic mobile robots generally depend on offline precise/coarse or online numerical/adaptive calibration of the camera intrinsic and extrinsic parameters and require precise or coarse knowledge of the camera in their implementation. This article presents a novel calibration-free image-based trajectory tracking control scheme, which can be implemented easily in real environments without any offline or online calibration of the camera parameters and can be used to efficiently control the motion of nonholonomic mobile robots with an arbitrarily placed and truly unknown overhead camera. Experimental results show that the proposed scheme can achieve satisfactory trajectory tracking control performance despite the lack of any knowledge about the camera intrinsic and extrinsic parameters and the presence of unknown camera lens distortions, and hence, can provide a simple but efficient solution to the vision-based motion control problem of nonholonomic mobile robots.

Precise 3-D surface reconstruction plays an important role in automated manipulation, industrial inspection, robotics, and so on. In this article, we present a novel 3-D surface reconstruction framework for stereo vision systems assisted with structured light projection. In the framework, a multistep matching scheme is proposed to establish a reliable correspondence between image pairs with high computation efficiency and accuracy. The successive matching steps can find the most precise correspondence through a step-by-step filtering procedure. To further enhance the precision, a correspondence refinement algorithm is presented. Phase maps with different frequencies are utilized as the code words for the multistep matching due to their high encoding accuracy and robustness to noise. This method does not require phase unwrapping or projector calibration, which improves the reconstruction precision and simplifies the operation. Selection strategies for the number of matching steps, the pattern frequencies, and the matching threshold are proposed. Furthermore, various 3-D reconstruction experiments are conducted using the proposed framework. Comparative experiments verify the advantages of the proposed framework compared with existing 3-D reconstruction methods regarding the accuracy and precision. The adaptability to scenarios with different motion speeds is demonstrated. Robustness and limitations of the framework are also revealed by conducting experiments in challenging scenarios. Note to Practitioners-This article is motivated by the precise 3-D surface reconstruction problem in automated robotic systems. In different scenarios, such as the reconstruction of the static objects or moving objects, the errors induced by sensor noise and motion should be taken into consideration. To enhance the measurement precision under these occasions, selection of pattern number and fringe frequencies has been a problem. To overcome these problems, this article proposes a novel framework for active stereo 3-D surface reconstruction. The framework utilizes multifrequency phase-shifting fringes to encode the reconstructed target. Then, a multistep matching method filters the candidates step by step to obtain the most precise corresponding pixel and avoid noise error accumulation. A refinement method is introduced to further improve the precision. Selection strategies of the number of matching steps, the fringe frequencies, and matching thresholds enable the 3-D reconstruction framework to be utilized on different occasions. In applications, limitations of the proposed method should be noted.

Industrial micro aerial vehicles (MAVs) with robotic manipulators have numerous applications in search and rescue tasks that reduce risks to human beings. However, such tasks distinctly require MAVs to have the capability of real-time autonomous navigation only with onboard sensors, especially in GPS-denied applications. This article introduces a new approach to onboard vision-based autonomous state estimation and mapping for MAVs' navigation in unknown environments. The algorithms run on board and do not need an external positioning system to assist autonomous navigation. The state estimator is developed to provide MAV's current pose on the basis of the extended Kalman filter by using image patch features. Inverse depth convergence monitoring and local bundle adjustment are utilized to improve the accuracy. The mapping algorithm for navigation is developed according to a real-time stereo matching method for three-dimensional perception. Finally, we have performed several experiments to demonstrate the effectiveness of the proposed approach.

Estimating the absolute camera pose requires 3D-to-2D correspondences of points and/or lines. However, in practice, these correspondences are inevitably corrupted by outliers, which affects the pose estimation. Existing outlier removal strategies for robust pose estimation have some limitations. They are only applicable to points, rely on prior pose information, or fail to handle high outlier ratios. By contrast, we propose a general and accurate outlier removal strategy. It can be integrated with various existing pose estimation methods originally vulnerable to outliers, and is applicable to points, lines, and the combination of both. Moreover, it does not rely on any prior pose information. Our strategy has a nested structure composed of the outer and inner modules. First, our outer module leverages our intersection constraint, i.e., the projection rays or planes defined by inliers intersect at the camera center. Our outer module alternately computes the inlier probabilities of correspondences and estimates the camera pose. It can run reliably and efficiently under high outlier ratios. Second, our inner module exploits our flow consensus. The 2D displacement vectors or 3D directed arcs generated by inliers exhibit a common directional regularity, i.e., follow a dominant trend of flow. Our inner module refines the inlier probabilities obtained at each iteration of our outer module. This refinement improves the accuracy and facilitates the convergence of our outer module. Experiments on both synthetic data and real-world images have shown that our method outperforms state-of-the-art approaches in terms of accuracy and robustness.

Although many vision-based control methods have been proposed for nonholonomic mobile robots, in their implementation, it is usually necessary to calibrate the camera intrinsic and/or extrinsic parameters using offline/online parameter estimation algorithms or online adaptation laws. To avoid the tediousness of camera calibration and to make the system performance highly robust to camera parameter uncertainties, in this article, we propose novel image-based pose stabilization control approaches for nonholonomic mobile robots with a truly uncalibrated overhead fixed camera. In the proposed approaches, only image position information of three feature points from an overhead camera is used for controller design, while information from other sensors (such as wheel encoders) is not required. Furthermore, either offline or online camera calibration is not necessary, and no knowledge about the camera intrinsic and extrinsic parameters is needed, which also can greatly simplify the controller implementation. Simulation and experimental results are given to demonstrate the feasibility and effectiveness of the proposed purely image-based pose stabilization approaches.

In this paper, we consider the self-repairing problem from the viewpoint of robotics and our objective is not only to restore the logical network topology but also to maintain the motion synchronization of the physical mobile robot formation. A gradient-based self-repairing algorithm which only relies on the local interactions among coupling robots is presented. More specifically, aiming to optimize the repair path in a distributed manner, a gradient generation and diffusion mechanism is presented first, which can generate a stable gradient distribution in the robot formation. Then, based on the recursive self-repairing technique and the proposed gradient distribution, several self-repairing rules as well as the corresponding individual control method are presented to solve the self-repairing problem. The improvement of the proposed algorithm on the motion synchronism of the robot formation and the optimality of the selected repair path are proved by theoretical analyses. Finally, the effectiveness and the practical applicability of the proposed algorithm are validated by simulations and real experiments.

We address the problem of video representation learning without human-annotated labels. While previous efforts address the problem by designing novel self-supervised tasks using video data, the learned features are merely on a frame-by-frame basis, which are not applicable to many video analytic tasks where spatio-temporal features are prevailing. In this paper we propose a novel self-supervised approach to learn spatio-temporal features for video representation. Inspired by the success of two-stream approaches in video classification, we propose to learn visual features by regressing both motion and appearance statistics along spatial and temporal dimensions, given only the input video data. Specifically, we extract statistical concepts (fast-motion region and the corresponding dominant direction, spatio-temporal color diversity, dominant color, etc.) from simple patterns in both spatial and temporal domains. Unlike prior puzzles that are even hard for humans to solve, the proposed approach is consistent with human inherent visual habits and therefore easy to answer. We conduct extensive experiments with C3D to validate the effectiveness of our proposed approach. The experiments show that our approach can significantly improve the performance of C3D when applied to video classification tasks. Code is available at https://github.com/laura-wang/video_repres_mas.

The image lines projected from parallel 3D lines intersect at a common point called the vanishing point (VP). Manhattan world holds for the scenes with three orthogonal VPs. In Manhattan world, given several lines in a calibrated image, we aim at clustering them by three unknown-but-sought VPs. The VP estimation can be reformulated as computing the rotation between the Manhattan frame and the camera frame. To compute this rotation, state-of-the-art methods are based on either data sampling or parameter search, and they fail to guarantee the accuracy and efficiency simultaneously. In contrast, we propose to hybridize these two strategies. We first compute two degrees of freedom (DOF) of the above rotation by two sampled image lines, and then search for the optimal third DOF based on the branch-and-bound. Our sampling accelerates our search by reducing the search space and simplifying the bound computation. Our search is not sensitive to noise and achieves quasi-global optimality in terms of maximizing the number of inliers. Experiments on synthetic and real-world images showed that our method outperforms state-of-the-art approaches in terms of accuracy and/or efficiency.

Point cloud based place recognition is still an open issue due to the difficulty in extracting local features from the raw 3D point cloud and generating the global descriptor, and it's even harder in the large-scale dynamic environments. In this paper, we develop a novel deep neural network, named LPD-Net (Large-scale Place Description Network), which can extract discriminative and generalizable global descriptors from the raw 3D point cloud. Two modules, the adaptive local feature extraction module and the graph-based neighborhood aggregation module, are proposed, which contribute to extract the local structures and reveal the spatial distribution of local features in the large-scale point cloud, with an end-to-end manner. We implement the proposed global descriptor in solving point cloud based retrieval tasks to achieve the large-scale place recognition. Comparison results show that our LPD-Net is much better than PointNetVLAD and reaches the state-of-the-art. We also compare our LPD-Net with the vision-based solutions to show the robustness of our approach to different weather and light conditions.

Inherent compliance plays an enabling role in soft robots, which rely on it to mechanically conform to the environment. However, it also limits the payload of the robots. Various variable stiffness approaches have been adopted to limit compliance and provide structural stability, but most of them can only achieve stiffening of discrete fixed regions which means compliance cannot be precisely adjusted for different needs. This paper offers an approach to enhance the payload with finely adjusted compliance for different needs. We have developed a manipulator that incorporates a novel variable stiffness mechanism and a sliding layer mechanism. The variable stiffness mechanism can achieve a 6.4 stiffness changing ratio with a miniaturized size (10 mm diameter for the testing prototype) through interlocking jamming layers with a honeycomb core. The sliding layer mechanism can actively shift the position of the stiffening regions through sliding of jamming layers. A model to predict the robot shape is derived with verifications via an experiment. The stiffening capacity of the variable stiffness mechanism is also empirically evaluated. A case study of a potential application in laparoscopic surgeries is showcased. The payload of the manipulator is investigated, and the prototype shows up to 57.8 percentage decrease of the vertical deflection due to an external load after reconfigurations.

A wide range of man-made environments can be abstracted as the Atlanta world. It consists of a set of Atlanta frames with a common vertical (gravitational) axis and multiple horizontal axes orthogonal to this vertical axis. This paper focuses on leveraging the regularity of Atlanta world for monocular SLAM. First, we robustly cluster image lines. Based on these clusters, we compute the local Atlanta frames in the camera frame by solving polynomial equations. Our method provides the global optimum and satisfies inherent geometric constraints. Second, we define the posterior probabilities to refine the initial clusters and Atlanta frames alternately by the maximum a posteriori estimation. Third, based on multiple local Atlanta frames, we compute the global Atlanta frames in the world frame using Kalman filtering. We optimize rotations by the global alignment and then refine translations and 3D line-based map under the directional constraints. Experiments on both synthesized and real data have demonstrated that our approach outperforms state-of-the-art methods.

In this paper, we study the cooperative path planning and motion coordination problems of the multi-robot system with large number of robots, aiming for practical applications in robotic warehouses and automated transportation systems. Particularly, we solve the life-long planning problem and guarantee the coordination performance in the presence of robot motion uncertainties. A hierarchical path planning and motion coordination structure is presented. The environment is divided into several sectors and a traffic heat-map is presented to describe the current sector-level traffic condition. In path planning level, the sector-level path is calculated by considering the path distance, the current traffic condition and the current robot uncertainty. In motion coordination level, local cooperative A* algorithm and conflict-based searching strategy are utilized within each sector to generate the collision-free local path of each robot in a rolling planning manner. The effectiveness and practical applicability of the proposed approach are validated by simulations with more than one thousand robots and real experiments.

In robotic-assisted laparoscopic surgery, the first assistant (FA) stands at the bedside assisting the intervention, while the surgeon sits at the console teleoperating the robot. Tasks for the FA include navigating new instruments into the surgeon's field-of-view and passing in or retracting materials from the body using hand-held tools. We previously developed ARssist, an augmented reality application based on an optical see-through head-mounted display, to aid the FA. In this paper, we refine the system and first perform a pilot study with three experienced surgeons for two specific tasks: instrument insertion and tool manipulation. The results suggest that ARssist would be especially useful for less experienced assistants and for difficult hand-eye configurations. We then perform a multi-user study with inexperienced subjects. The results show that ARssist can reduce navigation time by 34.57%, enhance insertion path consistency by 41.74%, reduce root-mean-square path deviation by 40.04%, and reduce tool manipulation time by 72.25%. Thus, ARssist has the potential to improve efficiency, safety and hand-eye coordination, especially for novice assistants.

Three-dimensional vision plays an important role in robotics. In this paper, we present a 3D surface reconstruction scheme based on combination of stereo matching and pattern projection. A two-step matching scheme is proposed to establish reliable correspondence between stereo images with high computation efficiency and accuracy. The first step (coarse matching) can quickly find the correlation candidates, and the second step (precise matching) is responsible for determining the most precise correspondence within the candidates. Two phase maps serve as codewords and are utilized in the two-step stereo matching, respectively. The phase maps are derived from phase-shifting patterns to provide robustness to the background noises. Only five patterns are required, which reduces the image acquisition time. Moreover, the precision is further enhanced by applying a correspondence refinement algorithm. The precision and accuracy are validated by experiments on standard objects. Furthermore, various experiments are conducted to verify the capability of the proposed method, which includes the complex object reconstruction, the high-resolution reconstruction, and the occlusion avoidance. The real-time experimental results are also provided.

This paper focuses on the localization and mapping problem on ground vehicles using odometric and monocular visual sensors. To improve the accuracy of vision based estimation on ground vehicles, researchers have exploited the constraint of approximately planar motion, and usually implemented it as a stochastic constraint on an SE(3) pose. In this paper, we propose a simpler algorithm that directly parameterizes the ground vehicle poses on SE(2). The out-of SE(2) motion perturbations are not neglected, but incorporated into an integrated noise term of a novel SE(2)-XYZ constraint, which associates an SE(2) pose and a 3D landmark via the image feature measurement. For odometric measurement processing, we also propose an efficient preintegration algorithm on SE(2). Utilizing these constraints, a complete visual-odometric localization and mapping system is developed, in a commonly used graph optimization structure. Its superior performance in accuracy and robustness is validated by real-world experiments in industrial indoor environments.

Most existing controllers for Car-Like Mobile Robots (CLMR) are designed to handle dynamic effects by decoupling speed and steering controls, also assume that full states are accessible, which are unrealistic for real-world applications. This paper presents a combined speed and steering control system for CLMR. To provide the essential state for the controller, a newly developed visual algorithm is adopted for estimating the high-update rate longitudinal and lateral velocities of the robot which cannot be accurately measured by wheel encoders due to the skidding and slipping effects. The stability of the proposed system can be guaranteed by Lyapunov method since the velocity estimation error, the speed tracking error and the lateral deviation converging to zero simultaneously. Real-world experiments are conducted on an electric autonomous tractor with online estimation to demonstrate the feasibility of the approach.

Large demand for robotics and automation has been reflected in the sanding works, as current manual operations are labor-intensive, without consistent quality, and also subject to safety and health issues. While several machines have been developed to automate one or two steps in the sanding works, the autonomous capability of existing solutions is relatively low, and the human assistance or supervision is still heavily required in the calibration of target objects or the planning of robot motion and tasks. This paper presents the development of an autonomous sanding robot, which is able to perform the sanding works on an unknown object automatically, without any prior calibration or human intervention. The developed robot works as follows. First, the target object is scanned then modeled with the structured-light camera. Second, the robot motion is planned to cover all the surfaces of the object with an optimized transition sequence. Third, the robot is controlled to perform the sanding on the object under the desired impedance model. A prototype of the sanding robot is fabricated and its performance is validated in the task of sanding a batch of wooden boxes. With sufficient degrees of freedom (DOFs) and the customization of the end effector, the developed robot is able to provide a general solution to the autonomous sanding on many other different objects.

3D lines in structured environments encode particular regularity like parallelism and orthogonality. We leverage this structural regularity to estimate the absolute and relative camera poses. We decouple the rotation and translation, and propose a novel rotation estimation method. We decompose the absolute and relative rotations and reformulate the problem as computing the rotation from the Manhattan frame to the camera frame. To compute this rotation, we propose an accurate and efficient two-step method. We first estimate its two degrees of freedom (DOF) by two image lines, and then estimate its third DOF by another image line. For these lines, we assume their associated 3D lines are mutually orthogonal, or two 3D lines are parallel to each other and orthogonal to the third. Thanks to our two-step DOF estimation, our absolute and relative pose estimation methods are accurate and efficient. Moreover, our relative pose estimation method relies on weaker assumptions or less correspondences than existing approaches. We also propose a novel strategy to reject outliers and identify dominant directions of the scene. We integrate it into our pose estimation methods, and show that it is more robust than RANSAC. Experiments on synthetic and real-world datasets demonstrated that our methods outperform state-of-the-art approaches.

Place recognition and loop-closure detection are main challenges in the localization, mapping and navigation tasks of self-driving vehicles. In this paper, we solve the loop-closure detection problem by incorporating the deep-learning based point cloud description method and the coarse-to-fine sequence matching strategy. More specifically, we propose a deep neural network to extract a global descriptor from the original large-scale 3D point cloud, then based on which, a typical place analysis approach is presented to investigate the feature space distribution of the global descriptors and select several super keyframes. Finally, a coarse-to-fine strategy, which includes a super keyframe based coarse matching stage and a local sequence matching stage, is presented to ensure the loop-closure detection accuracy and real-time performance simultaneously. Thanks to the sequence matching operation, the proposed approach obtains an improvement against the existing deep-learning based methods. Experiment results on a self-driving vehicle validate the effectiveness of the proposed loop-closure detection algorithm.

While the mechanism of Rope-Climbing provides much flexibility, it opens up challenges to the development of the controller for Robotic Manipulator installed on Rope-Climbing robot(RCR), which is called Rope-Climbing Robot Manipulator(RCRM) here. In particular, the deformable nature of the rope results in the vibration to the manipulator and hence affects the positioning of the end effector. In this paper, a new adaptive vision-based controller is proposed for RCRM, which enables the robot to carry out the high-accuracy task under the unknown vibration from the rope. The proposed controller guarantees the performance of the robot in twofold. First, the control problem is directly formulated in the image space such that the exact spatial relationship between the moving base of the manipulator (due to the vibrating rope) and the target (e.g. the wall) is not required. Second, novel adaptation laws are developed to estimate the vibration from the rope online and are cancelled out in the robot control input to stabilize the end effector. The stability of the closed-loop system is rigorously proved with Lyapunov methods, and experimental results are presented to illustrate the performance of the proposed controller.

Existing works on control of tractor-trailers systems only consider the kinematics model without taking dynamics into account. Also, most of them treat the issue as a pure control theory problem whose solutions are difficult to implement. This paper presents a trajectory tracking control approach for a full-scale industrial tractor-trailers vehicle composed of a carlike tractor and arbitrary number of passive full trailers. To deal with dynamic effects of trailing units, a force sensor is innovatively installed at the connection between the tractor and the first trailer to measure the forces acting on the tractor. The tractor's dynamic model that explicitly accounts for the measured forces is derived. A tracking controller that compensates the pulling/pushing forces in real time and simultaneously drives the system onto desired trajectories is proposed. The propulsion map between throttle opening and the propulsion force is proposed to be modeled with a fifth-order polynomial. The parameters are estimated by fitting experimental data, in order to provide accurate driving force. Stability of the control algorithm is rigorously proved by Lyapunov methods. Experiments of full-size vehicles are conducted to validate the performance of the control approach.

Wall polishing is a typical and essential procedure in the interior renovation. However, such works are mainly carried out by humans, which have the disadvantages of low efficiency, inconsistent quality, and issues of safety and health. A new vision-based impedance controller is proposed for polishing robots to automate the labor-intensive works. The desired impedance model is specified as the control objective to regulate the dynamic relationship between the interaction force and the motion of the robot end effector, where the motion is measured with the vision feedback. The use of the vision feedback guarantees the performance of the robot from two aspect. First, the vision feedback from the high-resolution camera ensures the accuracy of measurement of the robot end effector and hence guarantees the quality of polishing. Second, the concept of image moment is introduced such that the image Jacobian matrix is non-singular in a global sense, which guarantees the large working range of the robot. The dynamic stability of the closed-loop system is rigorously proved with Lyapunov methods, and experimental results are presented to illustrate the performance of the proposed controller.

This paper focuses on the motion estimation problem of ground vehicles using odometry and monocular visual sensors. While the keyframe-based batch optimization methods become the mainstream approach in mobile vehicle localization and mapping, the keyframe poses are usually represented by SE(3) in vision-based methods or SE(2) in methods based on range scanners. For a ground vehicle, this paper proposes a new SE(2)-constrained SE(3) parameterization of its poses, which can be easily achieved in the batch optimization framework using specially formulated edges. Utilizing such a parameterization of poses, a complete odometry-vision-based motion estimation system is developed. The system is designed in a commonly used structure of graph optimization, providing high modularity and flexibility for further implementation or adaptation. Its superior performance in terms of accuracy on a ground vehicle platform is validated by real-world experiments in industrial indoor environments.

Skeleton-based human action recognition has been a hot topic in recent years. Most existing studies are based on the skeleton data obtained from Kinect, which is noisy and unstable, in particular, in the case of occlusions. To cope with the noisy skeleton data and variation of viewpoints, this paper presents a view-invariant method for human action recognition by recovering the corrupted skeletons based on a 3D bio-constrained skeleton model and visualizing those body-level motion features obtained during the recovery process with images. The bio-constrained skeleton model is defined with two types of constraints: 1) constant bone lengths and 2) motion limits of joints. Based on the bio-constrained model, an effective method is proposed for skeleton recovery. Two types of new motion features, the Euclidean distance matrix between joints (JEDM), which contains the global structure information of the body, and the local dynamic variation of the joint Euler angles (JEAs) are used in describing human action. These two types of features are encoded into different motion images, which are fed into a two-stream convolutional neural network for learning different action patterns. The experiments on three benchmark datasets achieve better accuracy than the state-of-the-art approaches, which demonstrates the effectiveness of the proposed method.

The grasping control of an aerial manipulator in practical environments is challenging due to its complex kinematics/dynamics and motion constraints. This paper introduces a lightweight aerial manipulator, which is combined with an X8 coaxial octocopter and a 4-DoF manipulator. To address the grasping control problem, we develop an efficient scheme containing trajectory generation, visual trajectory tracking, and kinematic compensation. The NSGA-II method is utilized to implement the multiobjective optimization for trajectory planning. Motion constraints and collision avoidance are also considered in the optimization. A kinematic compensation-based visual trajectory tracking is introduced to address the coupled nature between manipulator and VAV body. No dynamic parameter calibration is needed. Finally, several experiments are performed to verify the stability and feasibility of the proposed approach.

The fast expanding 3C (Computer, Communication, and Consumer electronics) manufacturing leads to a high demand on the fabrication of USB cables. While several commercial machines have been developed to automate the process of stripping and soldering of USB cables, the operation of manipulating USB wires according to the color code is heavily dependent on manual works because of the deformation property of wires, probably resulting in the falling-off or the escape of wires during manipulation. In this paper, a new vision-based controller is proposed for robotic grasping and manipulation of USB wires. A novel two-level structure is developed and embedded into the controller, where Level-I is referred to as the grasping and manipulation of wires, and Level-II is referred to as the wire alignment by following the USB color code. The proposed formulation allows the robot to automatically grasp, manipulate, and align the wires in a sequential, simultaneous, and smooth manner, and hence to deal with the deformation of wires. The dynamic stability of the closed-loop system is rigorously proved with Lyapunov methods, and experiments are performed to validate the proposed controller.

Robot malfunction is inevitable in practical applications of the robot formation control due to uncontrolled crashing, system malfunction or communication loss. In this paper, we study the synchronous formation control problem in the presence of robot malfunctions. Our main idea is to improve the network connectivity and motion synchronism of the robot formation through a series of topology switchings and robot replacements. Firstly, the synchronous formation control method is introduced which enables the robots to tracking their desired trajectories while keeping predefined formation shapes. Secondly, a recursive switched topology control strategy is proposed to restore the formation shape as well as to improve the network connectivity and motion synchronism in the presence of robot malfunctions. Thirdly, the convergence analysis of the proposed control system is presented and a sufficient condition is obtained under an average dwell time scheme. What's more, the proposed approach is fully distributed and the communication delays between neighboring robots also have been taken into consideration. Simulation results demonstrate the effectiveness of the proposed approach.

The ability of rope-climbing robots in aloft operation is limited by its self-supporting and locomotion ability. In many applications, a given task is also too complex to be achieved by a single rope-climbing robot acting alone. The solution of multiple rope-climbing robots can overcome the limitations. However, existing control methods for rope-climbing robots are limited to single robot, and the open issue of coordination between multiple rope-climbing robots has not been systematically addressed. This paper presents a new synchronization scheme for position control of multiple rope-climbing robots, such that each robot moves to the corresponding desired position while synchronizing the heights between each other. Maintaining the same height is very important to guarantee the stability of the task-oriented manipulator installed among multiple robots, when it is performing the manipulation task. The development of the proposed controller is based on the singular perturbation approach, by treating the fast actuator dynamics as a perturbation of the slow robot dynamics, such that the lowest control complexity is achieved. The exponential stability of the overall system that consists of the fast and slow subsystems is proved by using Tikhonov’ s theorem. Experimental results are presented to illustrate the performance of the proposed controller.

Flexible continuum robots have exhibited unique advantages in working in an unstructured environment. Many applications require robots to actively control the deformation of soft objects, such as soft tissues in surgery. Thus, this study presents a robust model-predictive deformation control of a soft object using a flexible continuum robot. A linear approximation model for mapping from actuation space of a continuum robot to deformation space of a soft object is established. Jacobian matrix is estimated online by using a robust Geman-McClure estimator. Then, the deformation of the soft object is regulated by using a prediction horizon-based controller with exponential weighting for model uncertainty. The proposed control approach is effective in manipulating a soft object with a flexible continuum robot that is in contact with obstacles.

In this paper, we propose a novel 3D laparoscopic imaging system based on stereo-photogrammetry which is assisted by projecting patterns on the tissue surface. The proposed laparoscopic imaging system has three optic channels, two of which are responsible for stereo vision feedback and the other one is used for coded structured patterns projection. The projected patterns provide the robustness to homogeneous tissue surface since they add more features that can be relied on in the stereo matching. Image fiber bundles (100k pixels) and Gradient-index (GRIN) lenses are utilized to facilitate the remote image acquisition and miniaturization of the laparoscopic probe. Moreover, we adopt a digital micromirror device (DMD) and high-speed cameras to achieve fast pattern switching (up to 4 kHz) and high frame rate image acquisition. The system configuration allows for implementation of the time multiplexing pattern codification strategy in the 3D laparoscopic imaging system to enhance the reliability and resolution of the 3D surface reconstruction. A prototype is established, and various experiments are conducted. Comparative experimental results prove the advantages of our system design. The static and dynamic 3D reconstruction results validate the performance of the proposed 3D laparoscopic imaging system quantitatively and qualitatively.

Emerging applications of robotic manipulation of deformable objects have opened up new challenges in robot control. While several control techniques have been developed to manipulate deformable objects, the performance of existing methods is commonly limited by two issues: 1) implicit assumption that the physical contact between the end-effector and the object is always maintained, and 2) requirements of exact parameters of deformation model, which are difficult to obtain. This paper presents a new control scheme for robotic manipulation of deformable objects, which allows the robot to automatically contact then actively deform the deformable object by assessing the status of deformation in real time. Instead of designing multiple controllers and switching among them, the proposed method smoothly and stably integrates two control phases (i.e. region reaching and active deforming) into a single controller. The stability of the closed-loop system is rigorously proved with the consideration of the uncertain deformation model and uncalibrated cameras. Hence, the proposed control scheme enhances the autonomous capability of active deformable object manipulation. Experimental studies are conducted with different initial conditions to demonstrate the performance of the proposed controller.

Most existing trajectory tracking controllers are based on non-skidding and non-slipping assumptions, also assume that full states are accessible, which is unrealistic for real-world applications due to tire-road interaction. This paper presents a novel vision-based approach to achieve high performance tracking control of a Car-Like Mobile Robot (CLMR) with wheel skidding and slippage. A visual estimation algorithm is proposed to provide reliable position, velocity, skidding and slipping information to close the control loop. The stability of the proposed system can be guaranteed by Lyapunov method since the position tracking error and the estimation error converge to zero simultaneously. Simulation is made to validate the effectiveness of the developed controller in the presence of skidding and slipping with online visual estimator.

The problem of the leader-following formation control of nonholonomic mobile robots is addressed in this paper. A distributed formation control strategy using explicitly the coordination errors among robots is proposed without assuming that each follower robot knows the full state of the leader. First, a distributed estimation law is proposed for each follower robot to estimate the states, including the position, orientation and linear velocity of the leader. The distributed formation control law is then designed based on the estimated states of the leader and the neighborhood formation tracking error. Under some mild assumptions on the interaction graph among the leader and the follower robots and the velocity of the leader, asymptotic convergence of formation tracking errors to zero can be achieved. Finally, some numerical simulations and experiments on a group of nonholonomic mobile robots are presented to demonstrate the effectiveness of the proposed strategy.

Despite tremendous efforts made for years, trajectory tracking control (TC) of a nonholonomic mobile robot (NMR) without global positioning system remains an open problem. The major reason is the difficulty to localize the robot by using its onboard sensors only. In this paper, a newly designed adaptive trajectory TC method is proposed for the NMR without its position, orientation, and velocity measurements. The controller is designed on the basis of a novel algorithm to estimate position and velocity of the robot online from visual feedback of an omnidirectional camera. It is theoretically proved that the proposed algorithm yields the TC errors to asymptotically converge to zero. Real-world experiments are conducted on a wheeled NMR to validate the feasibility of the control system.

Most existing formation control approaches are based on the assumption that the global/relative position and/or velocity measurements of mobile robots are directly available. To extend the application domain and to improve the formation control performance, it is extremely necessary to avoid the use of position and velocity measurements in the design of formation controllers. In this paper, we propose new leader-following formation tracking control schemes for nonholonomic mobile robots with onboard perspective cameras, without using both position and velocity measurements. To address the unavailability issue of position measurements, the leader-follower kinematics model in the image space is developed, which can facilitate the complete elimination of measurement/estimation of the position information. Furthermore, feedback information from the perspective camera of the follower robot is used to design adaptive observers to estimate the leader linear velocity for feedforward compensation, which can handle the absence of velocity measurements such that the proposed schemes can be applied to control formations of mobile robots without mutual communication abilities. By using the Lyapunov stability theory, a rigorous stability analysis based on the nonlinear formation dynamics is provided to show that the global stability of the combined observer-controller closed-loop system can be guaranteed. Both simulation and experimental results are also given to demonstrate the performance of the proposed formation tracking control schemes.

This paper addresses the design of a vision-based method to automatically deform soft objects into desired two-dimensional shapes with robot manipulators. The method presents an innovative feedback representation of the object's shape (based on a truncated Fourier series) and effectively exploits it to guide the soft object manipulation task. A new model calibration scheme that iteratively approximates a local deformation model from vision and motion sensory feedback is derived; this estimation method allows us to manipulate objects with unknown deformation properties. Pseudocode algorithms are presented to facilitate the implementation of the controller. Numerical simulations and experiments are reported to validate this new approach.

Extrinsic calibration is one of the most important problems in robotics, whose objective is to estimate the relative poses among the sensors and the robot. Currently, most on-line solutions of extrinsic calibration are based on the Gaussian filters, which estimate all the system states iteratively. However, a large number of the system states are not related to calibration. Estimating all these redundant states may highly increase the dimensionality of the problem, and therefore, undermine the calibration performance in both efficiency and robustness. In this paper, we propose an innovative on-line calibration algorithm, called constraint Gaussian filter with virtual measurements (VMCGF). The nature of VMCGF is a filter with a compact state vector containing only the states of interest, also called essential states. Violating the modeling principles of the traditional Gaussian filters, the measurements cannot be expressed by the observation function with the essential states solely. Exploiting the constraints between the measurements and the essential states, virtual measurements are generated according to the properties of the generalized chi-square distribution. Although originally developed to solve the calibration problem, VMCGF is a general filtering algorithm, that can be applied to solve other problems that might suffer from redundant states. The implementation of VMCGF on a camera odometry calibration problem is introduced, and its observability properties are analyzed. Both simulations and experiments are conducted to validate our algorithm.

The expanding 3C (Computer, Communication, and Consumer electronics) manufacturing industry leads to a high demand on the soldering of flexible PCBs. Current manual soldering has the disadvantages of low output, low speed, and low efficiency, and upgrading soldering operations with robotic technologies is mainly limited by the property of deformation of flexible PCBs. In this paper, a novel robotic manipulation system is developed for automatic soldering of flexible PCBs, consisting of the hardware of a dual-arm configuration and the software of a cooperative control scheme. The proposed system works in a sequential manner, in the sense that a Cartesian-space region reaching controller drives an assistive arm to actively contact the PCB first, and a vision-based tracking controller activates a soldering arm after the deformation is stabilized. The proposed formulation eliminates uncertain deformation of flexible PCBs and thus guarantees the feasibility of robotic soldering.

Most studies on automated cell transportation are single-task oriented. Results from these investigations hardly meet the increasing demand for emerging cell surgery operations that usually require a series of manipulation tasks with multiple processing steps. In this paper, automated cell transportation to accomplish a multistep process in cell surgery was investigated. A novel control system that can manipulate grouped cells to move into different task regions sequentially and continuously without interruption was developed based on a robot-aided optical tweezers manipulation system. A potential field-based controller was designed to achieve multistep processing control, where the new concepts of contractive coalition and switching region were incorporated into tweezers-cell coalition. The success of this controller lies in simultaneously controlling the positions of the optical tweezers, trapping multiple cells effectively, and avoiding collisions in a unified manner. Simulations and experiments of transferring a group of cells to a number of task regions were performed to demonstrate the effectiveness of the proposed approach.

In this paper, we address the 3D position control of RCM-constrained instruments with monocular cameras. To compute the instrument's position from a single 2D image, we develop an innovative gradient descent algorithm which rotates and translates a line segment (over the plane spanned by the imaged instrument and the optical centre) until it best aligns with the manipulated tool. In contrast with other approaches in the literature, our algorithm only requires to simultaneously observe two feature points; the proposed iterative algorithm is not based on the exact solution, therefore it can still work with noisy image measurements. We derive a kinematic controller that uses the proposed position estimator to guide the 3D motion of a robotic instrument with a monocular camera. We evaluate the performance of our approach with numerical simulations and experiments.

In this paper, we propose a hierarchical pose graph optimization algorithm, which hierarchically divides a large pose graph into subgraphs and solves the optimization problem for each subgraph independently. Applying a modified graph partitioning algorithm, normalized cut, the original graph could be partitioned into subgraphs, which are conditionally independent on a set of key nodes. A modified normalized cut algorithm is applied to automatically partition a pose graph into several subgraphs, which are independent on each other when conditioned on a small number of keynodes. Preserving keynodes only, a simplified upper level graph is generated by a pose graph sparsification algorithm. Given the optimization results of all keynodes, which are obtained by solving the upper level graph, each subgraph in the lower level can be solved efficiently without concerning other subgraphs. The scale of each optimization is limited during graph partitioning. Therefore, the efficiency of the optimization is improved. Experiments with both public standard datasets and a dataset collected by an autonomous guided vehicle (AGV) system are conducted to test our algorithm.

In this paper, we address the problem of the image-based 3D pose computation of a semi-circle suturing needle using monocular image feedback for laparoscopy. We propose a constrained two-degree-of-freedom (2-DOF) geometry-based modelling method to parametrise the needle's 6-DOF pose, including depth information. The modelling solely relies on the simultaneous observation of the needle's apparent tip and junction. No external markers are needed for extra constraints. An adaptive controller combining gradient descent and vector-flow method is introduced to iteratively guide the needle's initial guessing pose to its real pose by minimizing image-based position errors. Experiments have been conducted using both numerical simulations and simulated laparoscopic scenarios to evaluate the performance of the algorithm.

In this paper, the task-space cooperative tracking control problem of networked robotic manipulators without task-space velocity measurements is addressed. To overcome the problem without task-space velocity measurements, a novel task-space position observer is designed to update the estimated task-space position and to simultaneously provide the estimated task-space velocity, based on which an adaptive cooperative tracking controller without task-space velocity measurements is presented by introducing new estimated task-space reference velocity and acceleration. Furthermore, adaptive laws are provided to cope with uncertain kinematics and dynamics and rigorous stability analysis is given to show asymptotical convergence of the task-space tracking and synchronization errors in the presence of communication delays under strongly connected directed graphs. Simulation results are given to demonstrate the performance of the proposed approach.

In this paper, we present a new feedback method to automatically servo-control the 3-D shape of soft objects with robotic manipulators. The soft object manipulation problem has recently received a great deal of attention from robotics researchers because of its potential applications in, e.g., food industry, home robots, medical robotics, and manufacturing. A major complication to automatically control the shape of an object is the estimation of its deformation properties, which determines how the manipulator's motion actively transforms into deformations. Note that these properties are rarely known beforehand, and its offline parametric identification is difficult and/or impractical to conduct in many applications. To cope with this issue, we developed a new algorithm that computes in real time the unknown deformation parameters of a soft object; this algorithm provides a valuable adaptive behavior to the deformation controller, something we cannot achieve with traditional fixed-model approaches. In contrast with most controllers in the literature, our new method can explicitly servo-control 3-D deformations (and not just 2-D image projections) in an entirely model-free way. To validate the proposed adaptive controller, we present a detailed experimental study with robotic manipulators.

In this paper, we propose a novel design of compliant safe joint, which has flexibility when the work load exceeds a predefined threshold. The compliance is generated by a spring. We design a special transmission mechanism to convert axial motion into circumferential motion such that the linear compliance can be converted into circular one. When the end-effector of a surgical robot actuated by the compliant safe joints collides with patient's body, the compliance of the joints will protect the patient by absorbing part of the collision energy. Because of the system's special mechanical structure, the control methods should be different when it works under different states. We propose a simple algorithm to choose control methods so that the system can work both under rigid and flexible states with different controllers. We have built a prototype to validate the design and the controller.

This paper presents the development of an endoscope manipulator with passive and active structures for functional endoscopic sinus surgery (FESS). The 5-DoF passive structure has three translations and two rotations (T3R2) that allows the surgeon to manually place the endoscope near to the entry point during. The 4-DoF motorized structure (T2R2) actively controls the endoscope's position based on the surgeon's input commands. We analyze the reciprocal screw of the passive and active structures. The motion control system is based on a real-time Linux kernel that processes the commands from the surgeon and controls the manipulator's active joints. A user control interface based on an IMU fastened on the surgeon's foot is developed; this interface measures the foot's posture and through a series of gestures, it provides the desired pan/tilt/zoom motions of the camera. The developed endoscope manipulator allows the surgeon to conduct ‘two-hand’ operations while retaining direct control of the camera. We present an experimental study to validate the performance of the robotic prototype.

In this paper, we address the positioning problem of remote centre of motion (RCM) mechanisms with uncalibrated image feedback from a monocular camera. Nowadays, RCM mechanisms are widely used in minimally invasive robotic surgery due to their ability to distally rotate a tool around a fixed entry port; note that in most surgical applications, the tools are typically controlled by manual/teleoperated motion commands given by a human user. In this paper, we depart from the traditional manual control scheme and derive sensor-based methods to automatically position the manipulated tool using real-time image feedback. To this end, we first characterise the mechanism's 3-DOF configuration with the angle of the image projected tool and scalar distances between feature points. To cope with uncertainty in the camera's calibration parameters, we propose two gradient descent estimators that adaptively compute the unknown Jacobian matrix; the stability of these algorithms is proved with Lyapunov theory. Finally, we derive a kinematic image-based controller and evaluate its performance with several positioning experiments.

In this paper, we present a new robotic uterine positioner for total laparoscopic hysterectomy. The robot is designed to actively position the patient's uterus during surgery, a lengthy and tedious task that is traditionally performed by a human assistant. Safety is simply the most important concern when developing robots for surgical purposes; we address this concern in the design of our robot from a mechanical perspective. To this end, we develop a 3-DOF robotic uterine positioner with an in-body remote center of motion (RCM); this key feature allows to prevent injuries to the patient when large motions occur at the cervix. A linearly-actuated arc-guided RCM mechanism is introduced to guarantee the rigidity and stability of the robot; The system's design allows to manipulate the uterus in a decoupled manner, thus control complexity can be reduced. Passive safety mechanisms are also implemented in all DOF of the robot in order to limit the interaction forces with the patient. Experiments, including an ex-vivo test conducted with cadaver, are conducted to verify the robot's performance.

This paper presents a novel and simple adaptive algorithm for estimating the position of a mobile robot with high accuracy in an unknown and unstructured environment by fusing images of an omnidirectional vision system with measurements of odometry and inertial sensors. Based on a new derivation where the omnidirectional projection can be linearly parameterized by the positions of the robot and natural feature points, we propose a novel adaptive algorithm, which is similar to the Slotine-Li algorithm in model-based adaptive control, to estimate the robot's position by using the tracked feature points in image sequence, the robot's velocity, and orientation angles measured by odometry and inertial sensors. It is proved that the adaptive algorithm leads to global exponential convergence of the position estimation errors to zero. Simulations and real-world experiments are performed to demonstrate the performance of the proposed algorithm.

Overlapping cells segmentation is one of the challenging topics in medical image processing. In this paper, we propose to approximately represent the cell contour as a set of sparse contour points, which can be further partitioned into two parts: the strong contour points and the weak contour points. We consider the cell contour extraction as a contour points locating problem and propose an effective and robust framework for segmentation of partially overlapping cells in cervical smear images. First, the cell nucleus and the background are extracted by a morphological filtering-based K-means clustering algorithm. Second, a gradient decomposition-based edge enhancement method is developed for enhancing the true edges belonging to the center cell. Then, a dynamic sparse contour searching algorithm is proposed to gradually locate the weak contour points in the cell overlapping regions based on the strong contour points. This algorithm involves the least squares estimation and a dynamic searching principle, and is thus effective to cope with the cell overlapping problem. Using the located contour points, the Gradient Vector Flow Snake model is finally employed to extract the accurate cell contour. Experiments have been performed on two cervical smear image datasets containing both single cells and partially overlapping cells. The high accuracy of the cell contour extraction result validates the effectiveness of the proposed method.

To enhance the safety of a humanoid robot when it is operating a complex environment, a number of methods that combine visual and force information have been presented. These methods are generally divided into two approaches. The first approach is to coordinate the visual controller and force controller in a parallel way, and the second approach is to coordinate them in series. However, these two approaches do not consider the interaction between the visual controller and force controller. Specifically, the first approach does not consider the interaction between the controllers. The second approach only considers the effect of the output of the visual controller on the force controller, while the effect of the force controller on the visual controller is not considered. This study presents a design for a new flexible controller for a humanoid robot that considers the interaction of visual and force information. The advantages of the proposed method are that it simultaneously incorporates the functions of a visual servo controller and a flexible controller as well as its ability to consider the interaction of visual and force information when a humanoid robot is operating.

In this paper, we address the set-point deformation control of elastic objects by fully-constrained grippers. We propose an uncalibrated Lyapunov-based algorithm that iteratively estimates the deformation Jacobian matrix, with no prior knowledge of the deformation and camera models. With this new method we show how, by combining pose information of the grippers with several visual measurements, we can independently control elastic deformations of unknown objects. We report experiments with a 6-DOF robot manipulator to validate this control approach.

In this paper, we address the visual servoing problem of robot manipulators with eye-in-hand cameras. To servo-control the image position of a feature point, traditional image-based controllers require the computation of the point's position vector with respect to the camera's frame. However, when the point's location is uncertain, the stability of traditional visual servoing controllers can not be rigorously guaranteed. To contribute to this problem, in this paper we present two new kinematic image-based controllers that do not require the exact location of static features. The first controller is a depth-free method that uses the camera's calibration matrix and visual feedback to compute a quasi-position vector of the feature point. The second controller uses adaptive control techniques to iteratively estimate the calibration matrix and the point's position vector. We prove the stability of both servo-controllers using Lyapunov theory, and present experimental results to evaluate its performance.

The trajectory tracking of underactuated water surface robots (or boats, surface vessels, etc.) has been an attractive topic over the past decade, and a lot of controllers are proposed for this challenging problem. However, most of the existing trajectory tracking controllers of the underactuated water surface robots assume the global positions of the robots can be accurately measured. In the working environments of the robots, the global position measurements are sometimes unstable or even unavailable. To avoid the direct position measurement, a new controller is proposed in this paper for the trajectory tracking of underactuated water surface robots by adopting the monocular visual feedback. This controller works on the basis of a novel adaptive algorithm for estimating global position of the robot online using visual feature tracking from a monocular camera, and its orientation and velocity measured by the AHRS (Attitude and Heading Reference System) sensor and visual odometry. It is proved by Lyapunov theory that the proposed adaptive visual servo controller gives rise to the asymptotic trajectory tracking and convergence of the position estimation to the actual position. An experiment is conducted to validate the effectiveness and robust performance of the proposed controller.

The view-based 3-D model retrieval system represents a 3-D model by its projected views. Most of the existing view-based 3-D model retrieval systems only analyze the features of the projected views, but not well consider the spatial arrangements of the viewpoints. Furthermore, most of these systems suffer from the high computational cost due to pairwise comparing the projected views of 3-D models. In this paper, we propose a new 3-D model descriptor called Sphere Image, which is defined as a collection of view features. A viewpoint of a 3-D model is regarded as a “pixel”: (1) The position of the viewpoint is denoted as the coordinate of the “pixel”. (2) The feature descriptor of the projected view is denoted as the value of the “pixel”. We also propose a probabilistic graphical model for 3-D model matching, and develop a 3-D model retrieval system to test our approach. We have conducted experiments based on the SHape REtrieval Contest (SHREC) 2012 generic 3-D model date set and the SHREC2009 partial 3-D model data set. Experimental results indicate that our system outperforms some state-of-the-art 3-D model retrieval systems.

Localization is one of the most difficult and costly problems in mobile robotics. To avoid this problem, this paper presents a new controller for the trajectory tracking of nonholonomic mobile robots using visual feedback without direct position measurement. This controller works on the basis of a novel adaptive algorithm for estimating the global position of the mobile robot online using natural visual features measured by a vision system and its orientation and velocity measured by odometry and Attitude and Heading Reference System (IMU&Compass) sensors. The nonholonomic motion constraint of mobile robots is fully taken into account, compared with most of the existing visual servo controllers for mobile robots. The Lyapunov theory is used to prove that the proposed adaptive visual servo controller gives rise to asymptotic tracking of a desired trajectory and convergence of the position estimation to the actual position. A graphical processing unit is adopted to implement the proposed adaptive controller in parallel to achieve real-time detection and tracking of visual features. Experiments on a mobile robot are conducted to validate the effectiveness and robust performance of the proposed controller.

Despite the recent progress in physically interactive and surgical robotics, the active deformation of compliant objects remains an open problem. The main obstacle comes from the difficulty to identify/estimate the object's deformation properties. This paper presents a new visually servoed deformation controller for unknown elastic objects. The control law is designed using the passivity-based framework. The proposed method exploits visual feedback to iteratively estimate the deformation Jacobian matrix, avoiding any identification steps. We prove that even in the presence of inexact estimations, the controller ensures input-to-state stability (i.e. dissipativity) with respect to time-varying disturbances. Finally, an experimental study with several deformation tasks is presented to validate the theory.

Localization is one of the most crucial and difficult problems for motion control of mobile robots despite of tremendous research efforts made for years. This paper presents a new vision-based controller for controlling a nonholonomic mobile robot to track a desired trajectory without directly measuring its position. A novel adaptive estimator is embedded into this new controller to estimate global position of the mobile robot online using natural visual features measured by a vision system, and its orientation and velocity measured by odometry/inertia/magnetic sensors. It is proved by Lyapunov theory that the proposed controller gives rise to asymptotic tracking of a desired trajectory and convergence of the position estimation to the actual position. The experiment is conducted to validate the proposed controller.

In nature, some animals exhibit impressive navigation capability using ambient magnetic field. Particularly, certain kinds of sea turtles can associate geomagnetism to spatial representation for positioning in transoceanic migration across the seemingly clueless sea. In robotics, the previous works on magnetic navigation can position a robot using ambient magnetic field, but they focused on an exploitation of extensively-explored magnetic map, i.e. the magnetic field of every inch of the region of interest is recorded for mapping. In this work, we propose an algorithm that is based on our analysis on how sea turtles navigate at sea under magnetic disruption as investigated and reported by biologists [1], [2]. We propose a direct likelihood method that generates pseudo training data to improve the estimation accuracy of the Gaussian mixture models. The experimental evaluation demonstrates that our localization algorithm exhibits stable and accurate positioning results. This work contrasts with the previous works which focused on magnetic localization using extensive data collection. On the contrary, we address whether magnetic localization is still feasible under scarce data samples and how to overcome this challenge.

In this paper, we propose a new vision-based controller to actively deform an unknown elastic object. Note that most deformation controllers in the literature require a-priori knowledge of the object's deformation properties. In contrast to this trend, we present a new Lyapunov-based method that online estimates the unknown deformation Jacobian matrix, avoiding any model identification or calibration steps. To achieve the desired object's deformation, we derive an innovative dynamic-state feedback velocity control law using the passivity-based framework. We present a detailed experimental study to validate the feasibility of our deformation controller.

Despite the recent progress in physically interactive and surgical robotics, the active deformation of compliant objects remains an open problem. The main obstacle to its implementation comes from the difficulty to identify or estimate the object's deformation model. In this paper, we propose a novel vision-based deformation controller for robot manipulators interacting with unknown elastic objects. We derive a new dynamic-state feedback velocity control law using the passivity-based framework. Our method exploits visual feedback to estimate the deformation Jacobian matrix in real time, avoiding any model identification steps. We prove that even in the presence of inexact estimations, the closed-loop dynamical system ensures input-to-state stability (i.e., full dissipativity) with respect to external disturbances. An experimental study with several deformation tasks is presented to validate the theory.

The nine degrees-of-freedom (DOF) inertial measurement units (IMU) are generally composed of three kinds of sensor: accelerometer, gyroscope and magnetometer. The calibration of these sensor suites not only requires turn-table or purpose-built fixture, but also entails a complex and laborious procedure in data sampling. In this paper, we propose a method to calibrate a 9-DOF IMU by using a set of casually sampled raw sensor measurement. Our sampling procedure allows the sensor suite to move by hand and only requires about six minutes of fast and slow arbitrary rotations with intermittent pauses. It requires neither the specially-designed fixture and equipment, nor the strict sequences of sampling steps. At the core of our method are the techniques of data filtering and a hierarchical scheme for calibration. All the raw sensor measurements are preprocessed by a series of band-pass filters before use. And our calibration scheme makes use of the gravity and the ambient magnetic field as references, and hierarchically calibrates the sensor model parameters towards the minimization of the mis-alignment, scaling and bias errors. Moreover, the calibration steps are formulated as a series of function optimization problems and are solved by an evolutionary algorithm. Finally, the performance of our method is experimentally evaluated. The results show that our method can effectively calibrate the sensor model parameters from one set of raw sensor measurement, and yield consistent calibration results.

Indoor aerial robots are useful in many applications due to their size, agility and ability to hover. However, tweaking a state-feedback controller to fly stably takes either intensive human supervision, or extensive modeling and identification, hence has never been trivial. In this paper, we give a successful flight controller design that can learn from a single demonstration performed by human and hover indoor aerial robots autonomously on maiden flight1.

To explore or exploit? In this paper, we discuss the long-standing exploration-exploration dilemma in context of designing a learning controller for stunt-style driving with scarce samples. By making an efficient use of a single demonstration by an expert, our algorithm leverages our intuitive understanding of driving to extract a coarse dynamics model from the collected driving data, then formulate the policy search in a setting of gradient update with a specially designed cost function. Both theoretical and empirical results are detailed and discussed.

For years, the puzzling cross-coupled responses between the control axes on the hingeless helicopters have long been haunting researchers. Different from previous works that underestimated the gyroscopic moments as a minor off-axis cross-coupling and misinterpreted the precession phenomena on the rotors of the hingeless helicopters as a phase-lag which is physically meaningless, this paper proposes a new method to relate both on-axis and off-axis responses by the influential gyroscopic moments through the actuation mechanism of the hingeless helicopters. Therefore, by this new method the debatable cross-coupling due to actuation dynamics can be directly and analytically quantified. This method is based on the fact that when the angular momentum of the spinning rotor is disturbed by the incremental lift along the main blades due to the varying cyclic pitch angle controlled by the servo mechanisms, the off-axis moments are induced to counteract the changes of the angular momentum according to the principle of gyroscope, and hence these gyroscopic moments directly exhibit the on-axis responses. This new method yields a parametric framework to examine the previously unexplained cross-coupled responses on the hingeless helicopters, and it shows that in hingeless helicopters except the aerodynamics the intricate nonlinearities are also attributed to their unintuitive actuation mechanisms. Finally, simulations as well as experiments have been carried out to validate the proposed modeling method.

Most unmanned aerial robots use a Global Navigation Satellite System (GNSS), such as GPS, GLONASS, and Galileo, for their navigation. However, from time to time the GNSS fails to function due to geographical restrictions and deliberated jamming. This paper proposes an Unscented Kalman Filter-based GPS/IMU integration method in order to accurately estimate the position and velocity of an unmanned miniature helicopter even when the GNSS malfunctions completely. Different from previous GPS/IMU integration methods that cannot propagate noisy inertial measurements to the position and velocity estimations on the rapid vibratory Vertical Take-Off and Landing (VTOL) platforms during the GNSS outage, this method novelly prioritises the propagations of the states in the Unscented Kalman Filter (UKF) algorithm and leverages the time-varying GNSS dilution of precision in line with the adjustments of the measurement noise covariances. Moreover, this method models the stochastic process in the inertial sensors by the acceleration white noise bias in addition to the commonly used random walking process. Without considering the specific actuation models that vary from vehicle to vehicle, this method can particularly be applied to the quivering unmanned helicopters which equipped with two-stroke engines. It yields a rapid and precise compensation for the sensor errors in order to effectively facilitate the propagations of inertial measurements to the position and velocity estimations. Finally, the superior performance of the proposed method in terms of accuracy and endurance is empirically demonstrated using our fully instrumented JR Voyager GSR helicopter.

This paper presents a self-rescue mechanism for a screw drive in-pipe robot, which only uses one DC motor. The robot has two working modes, Normal Working Mode and Self-rescue Mode. Under normal working mode, the robot propels itself in the pipe just as other classical screw drive robots. When the robot encounters the obstacle and gets jammed, the lock up mechanism and motion control mechanism of the robot are activated. Then, the robot changes from working mode to self-rescue mode and moves away in the reverse direction to avoid jamming in the pipe. The change of the working mode is determined by the characteristics of the mechanism. The proposed mechanism can be used as a safety protection method for the pipe robot. Experiments have been conducted to testify the proposed mechanism. Compared with those with screw drive mechanisms, robots with self-rescue mechanism are able to avoid jamming in the pipe.

This paper presents a new controller for locking a moving object in 3-D space at a particular position (for example the center) on the image plane of a camera mounted on a robot by actively moving the camera. The controller is designed to cope with both unknown robot dynamics parameters and unknown motion of the object. Based on the fact that the unknown position of the moving object appears linearly in the closed-loop dynamics of the system if the depth-independent image Jacobian is used, we developed a nonlinear observer to estimate the 3-D motion of the object on-line and an adaptive algorithm to estimate the robot dynamic parameters. With a full consideration of dynamic responses of the robot, we employed the Lyapunov method to prove asymptotic convergence of the image errors. Experimental results are presented to support the approach in this paper.

Diffractive gratings are often associated with the use of beam shaping device utilizing a monochromatic source. They could provide high flexibility in design and offer a precise control according to applications. The design of diffraction grating often makes several iterations through design and prototyping before completion. In this paper, we demonstrate a computer-aided design method and a mechanical method of prototyping diffractive grating optics for beam shaping, which aim at improve productivity through greater design flexibility, rapid fabrication and cost reduction. A description of the optical design is presented along with a discussion on the integrated patterning system.

This paper presents a novel adaptive controller for image-based visual servoing of a small autonomous helicopter to cope with uncalibrated camera parameters and unknown 3-D geometry of the feature points. The controller is based on the backstepping technique but differs from the existing backstepping-based methods because the controller maps the image errors onto the actuator space via a depth-independent interaction matrix to avoid estimation the depth of the feature points. The new design method makes it possible to linearly parameterize the closed-loop dynamics by the unknown camera parameters and coordinates of the feature points in the three dimensional space so that an adaptive algorithm can be developed to estimate the unknown parameters and coordinates on-line. Two potential functions are introduced in the controller to guarantee convergence of the image errors and to avoid trivial solutions of the estimated parameters. The Lyapunov method is used to prove the asymptotic stability of the proposed controller based on the nonlinear dynamics of the helicopter. Simulations have been also conducted to demonstrate the performance of the proposed method.

Different from previous works that require prior trim conditions on the helicopter, this paper proposes a hierarchical PD controller that is robust in controlling untrimmed and therefore critically unstable helicopters. This controller can yield asymptotic stability of the helicopter in horizontal motion control, which can be proven by the linear stability analysis. And this controller can flawlessly engage with traditional dual loop autopilot by using auto-varying references in an inner stabilizing loop. Moreover, to facilitate the controller design, this paper derives the dynamics of hingeless helicopters with an emphasis on gyroscopic effect. Finally, the stability and superior performance of the proposed controller are empirically demonstrated on an instrumented JR Voyager GSR helicopter.

This paper presents a distributed algorithm for a mobile sensor network to track targets with unknown motion. We formulates the target tracking as a multi-objective optimization problem which integrates the tracking quality, the energy saving and the network connectivity. To cope with sensing noises, we use the determinant of the covariance matrix of target estimation as the tracking quality measure and compute its partial derivatives for the optimization process. Virtual nodes are introduced to represent obstacles in the environment. Furthermore this algorithm can be extended to solve the problem of source tracking where sensors can only detect the density of the diffusing substances emitted by the source. Therefore a whole tracking framework has been set up which can be easily extended for applications under complicated situations. Simulations demonstrate the effectiveness of the proposed algorithm in energy conservation and tracking accuracy under different situations.

Autonomous deployment of mobile agents for coverage enhancement is an important issue in wireless sensor networks. The major challenge lies in the requirement of efficient distributed and localized computing. In addition, managing the coverage of heterogeneous sensing model is complicated due to the diversity of sensing ranges and the irregularity of coverage holes. This paper presents two distributed algorithms for maximizing the sensing coverage in heterogeneous sensor networks. The first algorithm is based on a circle packing technique. We prove the uniqueness of a circle packing up to a given triangulation and boundary conditions, thus the designated coverage layout can be achieved by controlling the boundary conditions. In the second algorithm, we give a formulation of virtual forces among sensor nodes to reduce redundant overlaps and avoid coverage holes. We prove that these virtual forces always give a quasioptimal local coverage. This method is applicable for deployment of sensor nodes in, not only an open field, but also any bounded field of interest and/or in the presence of obstacles. Numerical simulations are showed and these examples verify that the proposed algorithms always yield sensor deployments of wide coverage and collision free motions among sensor nodes. The proposed strategies utilize only the local information about a sensor node and its neighbors, thus providing distributed, efficient and scalable solutions to the deployment problem.

This paper presents a novel adaptive controller for image-based visual servoing using generalized image features. The key idea lies in the development of the depth-independent interaction matrix and the proposal of an adaptive algorithm for estimating the unknown geometric parameters of the features in the 3-D space. Furthermore, we derive the conditions for the asymptotic stability of the proposed controller and demonstrate that the conditions are satisfied for six types of common image features: points, lines, distances, angles, areas, and centroids. Experiments have been conducted to validate the proposed controller.

This paper presents a fully distributed algorithm for target tracking using a mobile sensor network. It tries to maintain the target being visible to the mobile network all the time while consuming as little motion energy as possible. Meanwhile the network connectivity is maintained. At every time, only the nodes around the target are activated while other nodes keep idle. Certain functions are defined to quantify the main aspects in the tracking such as the target escaping probability and the network connectivity status. They transform the tracking into a multi-objective optimization problem. To solve this global problem, a local motion strategy is proposed. Simulation results show that our algorithm yields good performance.

This paper presents a novel adaptive controller for image-based visual servoing of robots with an uncalibrated eye-in-hand camera using line features. The controller is developed based on three key ideas. First, we propose a new method that is similar to the Plucker coordinates, to represent projections of the lines features. The new representation leads to a depth-independent image Jacobian matrix and an error vector between real images and estimated projections of the lines, which are both linear to the unknown camera parameters. Second, an adaptive algorithm is developed to estimate the unknown camera parameters and the 3-D coordinates of the lines on-line. Third, a simple controller using the depth-independent image Jacobian is designed to control the projections of the lines to desired positions and orientations. The Lyapunov theory is used to prove the asymptotic convergence of the image error to zero based on the nonlinear robot dynamics. Finally, experiments have been conducted to demonstrate the performance of the proposed approach.

This paper presents a novel approach for image-based visual servoing of a robot manipulator with an eye-in-hand camera when the camera parameters are not calibrated and the 3-D coordinates of the features are not known. Both point and line features are considered. This paper extends the concept of depth-independent interaction (or image Jacobian) matrix, developed in earlier work for visual servoing using point features and fixed cameras, to the problem using eye-in-hand cameras and point and line features. By using the depth-independent interaction matrix, it is possible to linearly parameterize, by the unknown camera parameters and the unknown coordinates of the features, the closed-loop dynamics of the system. A new algorithm is developed to estimate unknown parameters online by combining the Slotine-Li method with the idea of structure from motion in computer vision. By minimizing the errors between the real and estimated projections of the feature on multiple images captured during motion of the robot, this new adaptive algorithm can guarantee the convergence of the estimated parameters to the real values up to a scale. On the basis of the nonlinear robot dynamics, we proved asymptotic convergence of the image errors by the Lyapunov theory. Experiments have been conducted to demonstrate the performance of the proposed controller.

This paper addresses the problem of deploying a set of mobile sensor nodes of heterogeneous sensing ranges to give a large and connected coverage. A novel deployment algorithm based on the circle packing technique is given. It iteratively enhances the coverage area of the sensor network from an initial random deployment, while guarantees the absence of coverage hole and obstacle avoidance. The sensing area of each node is modeled as a circular disc while its radius is bounded by the corresponding sensing range limit. The problem of placing these circular discs to cover a field is intuitively transformed to the circle packing problem: given the specified combinatorics of tangency patterns of n circles, find the label R denoting the radii of these circles. Since a unique packing exists for any given set of triangulations and boundary conditions, we can always find the minimum sensing range required for every interior node to satisfy such packing conditions. Though an extension from tangency packing to overlap packing, the interstices among triples (which represent coverage holes) can be eliminated. We have proven that the maximum global scaling factor to vanish all possible interstices is alpha = 3^((1/2)/2. Based on a number of numerical simulations, we have verified that the proposed algorithm always yields sensor deployments of wide coverage and minimize the sensing ranges required for every interior sensing node to satisfy the packing and boundary conditions.

This paper addresses the problem of tracking a mobile target using a mobile sensor network while minimizing the energy consumption and maintaining the network connectivity during the tracking process. While minimizing the tracking energy consumption is proved to be NP-complete, an approximately optimal solution named breadth-first leader-follower strategy is presented. Nodes close to the target predicted position find their following nodes based on breadth-first search and lead them to cover the probable region where the target may exist next time instant. Meantime the overall network connectivity can be maintained. We have proved that the energy consumption of the nodes moving under the control of the proposed algorithm is within a scalar factor of the optimal consumption. Simulation has been conducted to demonstrate the performance of the algorithm in different situations. The results show that our algorithm can yield good performance in target tracking while consuming little energy.

This paper presents a new controller for controlling a number of feature points on a robot manipulator to trace desired trajectories specified on the image plane of a fixed camera. The controller is designed to cope with the case when the intrinsic and extrinsic parameters of the camera as well as the robot parameters are not calibrated. The controller employs the depth-independent image Jacobian to map the errors on the image plane onto the joint space. By using the depth-independent image Jacobian, it is possible to linearly parameterize the unknown camera parameters in the closed loop dynamics of the system. A new algorithm is developed to estimate unknown parameters on-line. We have proved asymptotic convergence of the image errors by Lyapunov method with a full consideration of dynamic responses of the robot manipulator and demonstrated the performance by experiments.

This paper presents an effective algorithm for calibrating the extrinsic parameters between a camera and a laser range finder whose trace is invisible. On the basis of an analysis of three possible features, we propose to design a right-angled triangular checkerboard and to employ the invisible intersection points of the laser range finder’s slice plane with the edges of the checkerboard to set up the constraints equations. The extrinsic parameters are then calibrated by minimizing the algebraic errors between the measured intersections points and their corresponding projections on the image plane of the camera. We compared our algorithm with the existing methods by both simulations and the real data of a stereo measurement system. The simulation and experimental results confirmed that the proposed algorithm can yield more accurate results.

To facilitate training of endodontic operations, we have developed an interactive virtual environment to simulate endodontic shaping operations. This paper presents methodologies for dynamic modeling, visual/haptic display and model validation of endodontic shaping. We first investigate the forces generated in the course of shaping operations and discuss the challenging issues in their modeling. Based on the special properties and constraints associated with both pulpal tissue and endodontic files, we propose a dynamic model to simulate endodontic shaping, which is a smoothed particle based model derived for the pulpal tissue coupled with a finite element model for the endodontic files. The virtual environment has been implemented with both graphic and haptic interfaces. Furthermore, the effectiveness of the proposed model has been validated by experimental results through a novel robotic endodontic measurement system.

In the above titled paper (ibid., vol. 23, no. 3, pp. 443-458, Jun 07), Fig. 11 was printed with some mathematical symbols missing in lines 3, 6, 8, and 13. The corrected figure and its caption are presented here.

This paper presents a new controller for controlling a number of feature points on a robot manipulator to trace desired trajectories specified on the image plane of a fixed camera. It is assumed that the intrinsic and extrinsic parameters of the camera are not calibrated. A new adaptive algorithm is developed to estimate the unknown parameters online, based on three original ideas. First, we use the pseudoinverse of the depth-independent interaction matrix to map the image errors onto the joint space of the manipulator. By eliminating the depths in the interaction matrix, we can linearly parameterize the closed-loop dynamics of the manipulator. Second, to guarantee the existence of the pseudoinverse, the adaptive algorithm introduces a potential force to drive the estimated parameters away from the values that result in a singular Jacobian matrix. Third, to ensure that the estimated parameters are convergent to their true values up to a scale, we combine the Slotine-Li method with an online algorithm for minimizing the error between the estimated projections and real image coordinates of the feature points. We have proved asymptotic convergence of the image errors to zero by the Lyapunov theory based on the nonlinear robot dynamics. Experiments have been carried out to verify the performance of the proposed controller.

To facilitate training of endodontic operations, we have developed an interactive virtual environment to simulate endodontic shaping. This paper presents methodologies for haptic modeling, display and validation of endodontic shaping. We first investigate the forces generated in the course of shaping operations and discuss the challenging issues in their modeling. Based on the special properties and constraints associated with both pulpal tissue and endodontic files, we propose a haptic model to simulate endodontic shaping operations, which is a smoothed particle based dynamic model derived for the pulpal tissue coupled with a finite element model for the endodontic files. Furthermore, the effectiveness of the proposed model has been validated by experimental results through a novel robotic endodontic measurement system

Accurate estimation of interaction forces for endodontic shaping is fundamental to the interactive simulation of this operation. By applying new statistical learning techniques to this problem, this paper proposes a novel estimation method to acquire an optimized interaction force model to characterize input-output force mapping for endodontic shaping. We first present a novel robotic measurement system to acquire interaction forces in endodontic shaping and establish the needed training set. Then we propose a support vector regression model to learn the input-output force mapping for endodontic shaping. The regression model uses RBF kernel for training, and the optimized parameters of which are obtained by experiments. The learned model can convincingly estimate the interaction force resulting from endodontic shaping. And the effectiveness of the model has been evaluated by error measurement results

This paper presents a new controller for controlling a number of feature points on a robot manipulator to trace desired trajectories specified on the image plane of a fixed camera, provided that the intrinsic and extrinsic parameters of the camera are not calibrated. A new adaptive algorithm is developed to estimate the unknown parameters on-line based on three original ideas. First, we use the pseudo-inverse of the depth-independent interaction matrix, proposed in our earlier work, to map the image errors onto the joint space so that we can linearly parameterize the closed-loop dynamics of the system. Second, to guarantee existence of the pseudo-inverse, we introduce a potential force to drive the estimated parameters away from the values resulting in singular image interaction matrix. Third, to ensure that the estimated parameters are convergent to their true values up to a scale, we combine the Slotine-Li method with an on-line algorithm for minimizing the errors between the estimated projections and real image coordinates of the feature points. We have proved asymptotic convergence of the image errors by Lyapunov method and demonstrated the performance by experiments

This paper presents a new adaptive controller for dynamic tracking of a robot manipulator without visual velocity when the intrinsic and extrinsic parameters of the camera are not calibrated. Most controllers in the past require the measurement of the visual velocity or differentiation of the visual position. The measurement of the visual velocity is subject to big noises in general due to low sampling rates of the vision loop. To avoid performance decaying caused by measurement errors of the visual velocity, the controller we developed requires estimated visual velocity only. With a full consideration of dynamic responses of the robot manipulator, we employed the Lyapunov method to prove the convergence of the image errors of the trajectory to zero and the convergence of the estimated parameters to the real values up to a scale. Experiments have been conducted to demonstrate good convergence of the trajectory errors of the robot under the control of the proposed method

A system, employing the probe of an atomic force microscope to mechanically pattern various materials such as photoresist, semiconductors or polymers in the nanometer regime has been developed. The system was utilized for characterization of nanowires including carbon nanotubes (CNTs) and silicon nanowires (SiNWs)

This paper presents a novel algorithm, called ISOGRID (isometric grid-based algorithm), for autonomous deployment of mobile sensor networks. Upon an initial random placement of sensor nodes, the algorithm iteratively computes node movements to enhance sensing coverage and avoid obstacles while ensuring sensor connectivity. The principle is to redeploy the sensor nodes such that the communication graph approximates the layout of an isometric grid. Based on a number of simulation experiments, we have verified that the proposed algorithm always yields sensor deployments of wide coverage and desired topologies while ensuring collision-free motions of robots and promising wireless communication among sensor nodes. As the algorithm runs in a decentralized framework, it is computationally efficient and scalable. We also suggest another deployment algorithm, MEC (minimum enclosing circle-based algorithm), as an improvement to Minimax presented in (G. L. Wang, et al., March 2004) and extensively utilize it in the simulation examples to make comparison with ISOGRID

This paper presents an effective and efficient face normalization method based on eyes location. The face is rapidly detected based on boosted cascade of simple Haar-like features firstly. Then, the algorithm detects the position of pupils in the face image using the geometric relation between the face and the eyes. Finally, the algorithm normalizes the orientation, the scale and the grayscale of the face image. The experimental results demonstrated that this algorithm can detect and normalize the face image efficiently and accurately. The algorithm can be used in face recognition because the normalized faces can improve the recognition rate.

A robust model fitting technique is presented for recovery of structure and motion from a sequence. The error bound of the inliers is not needed and the outliers are assumed to be randomly (uniformly) distributed. Unlike other methods in the RANSAC family that need some prior information or a scale estimator to select the best consensus set, we estimate the parameters of the model directly based on Bayesian theory and kernel density estimation. Then the estimated parameters can be used directly or used to determine the best consensus set. The advantage of the proposed method is that it can robustly determine the best consensus set without any prior information of the data and need a lower computational cost than other auto-scale algorithms in the RANSAC family. The proposed method is applied to structure from planar motion estimation. The experiments indoor and outdoor have been done to verify the performance of the algorithm and the very promising results are obtained

This paper presents a new adaptive controller for a robot manipulator to control position of projections of unknown targets using the visual feedback from an eye-in-hand camera. The controller is designed to cope with the case when the intrinsic and extrinsic parameters of the camera are not calibrated. The controller employs the depth-independent image Jacobian to map the errors on the image plane onto the joint space. By using the depth-independent image Jacobian, it is possible to linearly parameterize the unknown camera parameters and the unknown coordinates of the target points in the closed loop dynamics of the system. A new algorithm is developed to estimate unknown parameters on-line. By minimizing the errors between the real and estimated projections of the target points on the image plane, this new adaptive algorithm can guarantee the convergence of the estimated parameters to the real values. With a full consideration of dynamic responses of the root manipulator, we employed the Lyapunov method to prove asymptotic convergence of the image errors. Experiments have been conducted to demonstrate the performance of the proposed controller

To ensure safety of railway operation, it is important to regularly check railway conditions such as deformation of the rails. To monitor rail deformation, this paper presents a method for detecting sources of vibrations a running train on-line by measuring accelerations, which include the train bogie's lateral acceleration, and the crossbeam's lateral and vertical accelerations. A series of detection algorithms including peak-peak value entropy comparison, weighted correlation coefficients comparison etc. are proposed in the method, according to different characters of vibrations from train itself and rail deformation. To eliminate the vibration due to the train itself, the algorithm employs the peak-peak value entropy comparison. To identify the order of the vibrations between crossbeam and bogie, a weighted correlation coefficient is applied. Weight center and maximum position are used to detect at last. The algorithms were implemented on a passenger train using ARM processor and real experiments were conducted on the train on the railway between Shenyang and Dalian in China. The experiments demonstrated that the proposed method can produce satisfactory results.

This paper presents a new adaptive controller for image-based dynamic control of a robot manipulator using a fixed camera whose intrinsic and extrinsic parameters are not known. To map the visual signals onto the joints of the robot manipulator, this paper proposes a depth-independent interaction matrix, which differs from the traditional interaction matrix in that it does not depend on the depths of the feature points. Using the depth-independent interaction matrix makes the unknown camera parameters appear linearly in the closed-loop dynamics so that a new algorithm is developed to estimate their values on-line. This adaptive algorithm combines the Slotine-Li method with on-line minimization of the errors between the real and estimated projections of the feature points on the image plane. Based on the nonlinear robot dynamics, we prove asymptotic convergence of the image errors to zero by the Lyapunov theory. Experiments have been conducted to verify the performance of the proposed controller. The results demonstrated good convergence of the image errors.

For Internet based real-time teleoperation systems, random time delay can cause instability in the closed loop control system and hence hinder task accomplishment. Event based control systems have been proposed to overcome the instability caused by the random time delay. High latency at the transport layer can still impede effective and reliable execution of tasks with high dexterity requirements. Network QoS based dynamic resource allocation has been proposed to increase the efficiency and reliability of task execution. However, these approaches only try to mitigate or overcome the effects of random time delay and do not address the cause of latency issues in the communication channel. This paper addresses the efficiency and reliability requirements for supermedia enhanced teleoperated systems by reducing the end-to-end transmission latency through the use of overlay networks. The proposed system reduces the transmission latency by using multiple, disjoint paths in overlay networks. The proposed system facilitates reliable and efficient task completion for tasks with high dexterity requirements. Experimental validation of the proposed teleoperated system using the PlanetLab Network is provided for the task of teleoperating a mobile manipulator system.

This paper describes a virtual reality interface between human and the Atomic Force Microscope (AFM), which allows the operator to perform nanomanipulation with an AFM tip in the virtual reality environment with haptic feedback. During operation, the tip-sample interaction forces and intermolecular forces between the tip and surface are modeled based on Lennard-Jones potential and JKR theory, respectively. Our objective is to provide a 3D virtual reality interface capable of displaying topography of surface for the users and allow them to predict the results for the manipulation.

Effective and efficient simulation of tissue-tool interactions is the key to a virtual endodontic training system. This paper presents a new force model for effectively simulating the interactions of the pulpal tissue with the endodontic tools. Based on the smoothed particle hydrodynamics (SPH) method, a particle-based pulpal tissue model is proposed, which takes into account characteristics of the tissue and constraints associated with both the tissue and current haptic devices. The stability, accuracy and speed of the proposed model are guaranteed by applying suitable smoothing kernels to calculate different forces applied to the pulpal tissue. Furthermore, a finite element model is used to model the deformation and dynamics of the endodontic tools. We also derive several boundary constraints associated with interactions of the pulpal tissue with the endodontic tools. With this force model, the virtual system can provide the convincing sense of touch of endodontic procedures in real time using current high fidelity haptic devices.

This paper presents a new adaptive controller for dynamic image-based visual servoing of a robot manipulator when the intrinsic and extrinsic parameters of the camera are not calibrated. To cope with nonlinear dependence of the image Jacobian on the unknown parameters, this controller employs a matrix called nonscaled image Jacobian which does not depend on the scale factors determined by the depths of feature points. By removing the scale factors, the camera parameters appear linearly in the close-loop dynamics so that a new algorithm, different from Slotine and Li’s, is developed to estimate their values on-line. In the parameter adaptation, in addition to the regressor term, our algorithm also uses the errors between the real and estimated projections of the feature points on the image plane so as to guarantee the convergence of the estimated parameters to the real values up to a scale. A new Lyapunov function is introduced to prove asymptotic convergence of the image errors based on the robot dynamics. Experiments have been conducted to demonstrate the performance of the proposed controller.

This paper presents a new and novel controller for dynamic image-based trajectory tracking of a robot manipulator in uncalibrated environments. The controller is designed to cope with the case when the homogenous transformation matrix between the root and the vision system is unknown. A new adaptive algorithm, different from the Slotine and Li’s method, has been developed to estimate a set of parameters corresponding to the unknown transformation matrix. With a full consideration of dynamic responses of the robot manipulator, we employed the Lyapunov method to prove the convergence of the image errors of the trajectory to zero and the convergence of the estimated parameters to the real values up to a scale. Simulations and experiments have been conducted to demonstrate good convergence of the trajectory errors of the robot and the estimated parameters under the control of the proposed method.

This paper reports a case study of using single 3D stereoscopic visual feedback for real-time teleoperation of dexterous tasks. In traditional teleoperation systems, real-time visual feedbacks of multiple monoscopic views of the robot workspace are provided for remote operator. However, it is difficult for the operator to control remote robot to perform dexterous tasks by looking at multiple video feedbacks at the same time. During teleoperation, remote operators usually find multiple 2D visual feedbacks confusing, especially when performing dexterous tasks that require accurate positioning and orientating of robot end-effectors. In this paper, we propose to provide single real-time 3D stereoscopic visual feedback for remote operators so that they perceive remote robot workspace with the sense of depth. This sense of 3D empowers remote operators to accurately position and orient robot end-effector with confidence. Experiments have been conducted to reveal the usefulness of real-time 3D stereoscopic video feedback over multiple monoscopic video feedback in real-time teleoperation.

This paper presents a new adaptive controller for dynamic image-based visual servoing of a robot manipulator when the intrinsic and extrinsic parameters of the camera are not calibrated. To cope with nonlinear dependence of the image Jacobian on the unknown parameters, this controller employs a matrix called nonscaled image Jacobian which does not depend on the scale factors determined by the depths of feature points. By removing the scale factors, the camera parameters appear linearly in the close-loop dynamics so that a new algorithm, different from Slotine and Li's, is developed to estimate their values on-line. In the parameter adaptation, in addition to the regressor term, our algorithm also uses the errors between the real and estimated projections of the feature points on the image plane so as to guarantee the convergence of the estimated parameters to the real values up to a scale. A new Lyapunov function is introduced to prove asymptotic convergence of the image errors based on the robot dynamics. Experiments have been conducted to demonstrate the performance of the proposed controller.

Endodontic treatment is one of the most common dental procedures employed in modern dentistry. Although there have been many works in medical simulation in the past two decades, little research has been done on modeling and simulation of endodontic procedures. This work presents a methodology to design and implement a virtual endodontics test bed for facilitating surgical training of root canal skills. In order to make the system more efficient and effective, the work focuses on the simulation of the most critical step in the entire endodontic procedure, i.e. shaping and cleaning. The paper first proposes a novel approach for estimating filing force and torque by developing a measurement system. Then, a hybrid modeling scheme for endodontic simulation is presented, which consists of a geometric-based endodontic instrument model, a physically-based root canal model, and a parameter identification method for the model.

Fixtures are tools to locate and securely hold workpieces for manufacturing operations such as inspection and machining. This paper presents a simple and complete algorithm for automatic and optimal fixture design on 3-D workpieces represented by discrete points. A new performance index is proposed to simultaneously address three fundamental requirements: form-closure constraint, minimum locating errors of the workpiece, and even load distribution among the fixturing points. To find the optimal future, the algorithm combines a local optimization procedure with a recursive problem decomposition strategv. The local optimization procedure searches for a solution in the direction of minimizing the performance index. When the performance index reaches a local minimum, the algorithm decomposes the discrete point set into subsets, and then carries out new rounds of local search in the subsets. The algorithm ends when a locally optimal solution is obtained. Compared to the exhaustive search, this algorithm is more efficient; and compared to other heuristic methods, the proposed algorithm is complete. The efficiency of this algorithm is demonstrated by numerical examples.

Matching corners between images is an important and difficult problem in stereo vision and many other vision applications, and no effective method has been developed to cope with general cases. In this paper, we present an improved SUSAN (smallest univalue segment assimilating nucleus) corner detection algorithm and an effective algorithm to establish corner correspondence between two images based on Delaunay triangulation. First we construct Delaunay triangulations among corners of each image and compute interior angles of the triangles. The corner correspondence is established based on an observation that these angles completely and uniquely characterize the corners and their values are not affected by scale change, less affected by rotation and translation to some extent. At the matching stage, we first obtain the most similar triangle pairs, and then extend their edges circularly until all matching corners are triangulated and mismatching corners are discarded. Experimental results are provided which illustrate the good performance of the algorithm.

This paper presents two novel methods to calibrate the 'attitude' of a camera from a long sequence of images captured by a camera mounted on a train. The 'attitude' refers to the camera's orientation with respect to the motion plane. For general plane motion with small rotation such as motion of a train, it is difficult to estimate the 'attitude' of a camera accurately. A long image sequence is used to overcome the noises in rotation estimation induced from the errors in correspondence estimation. We have developed an automated technique to process a large number of images and calibrate the 'attitude' of a camera mounted on the train automatically. Experiments with real indoor and outdoor images have been conducted and the results demonstrated that the methods can estimate the 'attitude' of a camera with good accuracy.

Significant research has been conducted in the field of Internet-based teleoperation. However, there is a lack of objective performance measures beyond completion time, which is dependent on several external factors to the system. This paper develops the concept of event-synchronization for supermedia enhanced Internet based teleoperation systems. Supermedia is the term used to refer to the different feedback streams; for example, video, haptic, temperature and others. This performance measure or system property is not affected by external factors; such as, the human operator and the communication characteristics. In addition, the design, which is based on Petri net theory, of systems satisfying this property is detailed. The experimental results obtained using a mobile manipulator bilateral teleoperation system, are given.

A complete and efficient algorithm is proposed for searching form-closure grasps of n hard fingers on the surface of a three-dimensional object represented by discrete points. Both frictional and frictionless cases are considered. This algorithm starts to search a form-closure grasp from a randomly selected grasp using an efficient local search procedure until encountering a local minimum. The local search procedure employs the powerful ray-shooting technique to search in the direction of reducing the distance between the convex hull corresponding to the grasp and the origin of the wrench space. When the distance reaches a local minimum in the local search procedure, the algorithm decomposes the problem into a few subproblems in subsets of the points according to the existence conditions of form-closure grasps. A search tree whose root represents the original problem is employed to perform the searching process. The subproblems are represented as children of the root node and the same procedure is recursively applied to the children. It is proved that the search tree generates O(KlnK/n) nodes in case a from-closure grasp exists, where K is the number of the local minimum points of the distance in the grasp space and n is the number of fingers. Compared to the exhaustive search, this algorithm is more efficient, and, compared to other heuristic algorithms, the proposed algorithm is complete in the discrete domain. The efficiency of this algorithm is demonstrated by numerical examples.

The coordination of multi-robots is required in many scenarios for efficiency and task completion. Combined with teleoperation capabilities, coordinating robots provide a powerful tool. Add to this the Internet and now it is possible for multi-experts at multi-remote sites to control multi-robots in a coordinated fashion. For this to be feasible there are several hurdles to be crossed including Internet type delays, uncertainties in the environment and uncertainties in the object manipulated. In addition, there is a need to measure and control the quality of tele-coordination. This paper proposes a measure for the quality of tele-coordination, referred to as the coordination index, and details the design procedure that ensures a system performs at a required index. The theory developed was tested by bilaterally tele-coordinating two mobile manipulators via the Internet. The experimental results confirmed the theory presented.

A novel computer input system-the Micro Input Devices System (MIDS)-is under development by merging MEMS sensors and existing wireless technologies. This system could potentially replace the functions of the mouse, pen, and keyboard as input devices to the computer. The system could also be used as a general wireless 3D motion sensing device. In this paper, we will present our work on using MIDS for motion sensing application of robot hands. MIDS is used to evaluate the performance of PD adaptive control and Impedance control schemes in manipulating a five-fingered robot hand and in manipulating this hand to grasp a ball. Experimental results indicate that MIDS is capable of obtaining real-time 3D acceleration/vibration data wirelessly for the robotic hand, hence eliminating the need to perform the time-consuming integration of the position sensor data to obtain acceleration. Moreover, our initial results also indicate that further exploration of this technology could eventually produce a new control-input device for robotic grasping manipulators. These results are presented in this paper.

With the rapid development of information technology, Internet has evolved from a simple data-sharing media to an amazing information world where people can enjoy different kinds of services. Recently, the use of the Internet has been expanded to the field of automation, i.e. using the Internet as a tool to control equipment located at remote sites. This paper presents a cooperative robot system consisting of a robot hand and a mobile robot carrying a stereo vision, which can be tele-operated by operators at different sites via the Internet. To overcome the instability and reliability problem caused by the random time delay of the Internet communication, we adopt an event as the reference for controller design of the system. A vision-based method is adopted to maintain interactions among the operations. Results obtained in teleoperation experiments among Hong Kong, the mainland China, and USA will be demonstrated to confirm the usefulness and effectiveness of the developed method and system.

While the classical notion of force closure is defined for actively controlled and coordinated robotic fingers, passive contacts play an equally important role in workpiece fixturing and often in robotic manipulation. This paper presents a description of passive forces arising at the normal and frictional contacts by passive physical means. The passive contacts generate reactive forces only as a response to an external force and/or any active force. Within the framework of rigid body contact, a contact system with passive forces is generally undeterminate. We present a general approach based on an application of the minimum norm principle. The model reveals some intricate properties of the passive contact forces, including internal forces at the passive and/or active contacts. Some practical implications of the passive nature in fixture design are discussed.

In real-time Internet based teleoperation, different robotic tasks have different dexterity requirements during task progress and thus different network resources are required for safe and reliable task accomplishment. In order to control remote manipulators efficiently and smoothly via the Internet, dynamic bandwidth allocation is crucial to successful accomplishment of robotic tasks controlled by remote operator. In this paper, a novel bandwidth allocation mechanism is developed based on the online measured task dexterity index of current dexterous tasks so that operators can control remote manipulators efficiently and smoothly even under poor network quality. Experiments have been conducted to demonstrate the effectiveness of the presented resource (bandwidth) allocation algorithm in Internet based teleoperation system.

A complete and efficient algorithm is proposed for searching form-closure grasps of n-hard fingers on 3-D objects represented by discrete points. Both frictional and frictionless cases are considered. This algorithm starts to search a form-closure grasp front a random grasp using an efficient local search procedure until encountering a local minimum. The local procedure is based on the powerful ray-shooting technique and searches in the direction of reducing the distance between the convex hull corresponding to the grasp and the origin of the wrench space. When the distance reaches a local minimum value, the algorithm decomposes the problem into sub-problems according to the existence conditions of form-closure grasps. A search tree whose root represents the original problem is employed to guide the searching process. The sub-problems are represented as children of the root node and the same procedure is recursively applied to the children. Theoretical analysis has been conducted on completeness and computational complexity of the algorithm. The efficiency of this algorithm is demonstrated by numerical examples.

This paper introduces new planning and control methods for supermedia-enhanced real-time telerobotic operations via the Internet. Supermedia is the collection of video, audio, haptic information, temperature, and other sensory feedback. However, when the communication medium used, such as the Internet, introduces random communication time delay, several challenges and difficulties arise. Most importantly, random communication delay causes instability, loss of transparency, and desynchronization in real-time closed-loop telerobotic systems. Due to the complexity and diversity of such systems, the first challenge is to develop a general and efficient modeling and analysis tool. This paper proposes the use of Petri net modeling to capture the concurrency and complexity of Internet-based teleoperation. Combined with the event-based planning and control method, it also provides an efficient analysis and design tool to study the stability, transparency, and synchronization of such systems. In addition, the concepts of event transparency and event synchronization are introduced and analyzed. This modeling and control method has been applied to the design of several supermedia-enhanced Internet-based telerobotic systems, including the bilateral control of mobile robots and mobile manipulators. These systems have been experimentally implemented in three sites test bed consisting of robotic laboratories in the USA, Hong Kong, and Japan. The experimental results have verified the theoretical development and further demonstrated the stability, event transparency, and event synchronization of the systems.

For a class of robot arms, a proportional-derivative (PD) controller plus gravity compensation yields the global asymptotic stability for regulation tasks, and some proportional-integral-derivative (PID) controllers guarantee local regulation without gravity cancellation. However, these controllers cannot render asymptotic stability for tracking tasks. In this paper, a simple decentralized continuous sliding PID controller for tracking tasks that yields semiglobal stability of all closed-loop signals with exponential convergence of tracking errors is proposed. A dynamic sliding mode without reaching phase is enforced, and terminal attractors, as well as saturated ones, are considered. A comparative experimental study versus PD control, PID control, and adaptive control for a rigid robot arm validates our design.

This paper addresses visual servoing of a robot manipulator with uncalibrated intrinsic and extrinsic parameters of the vision system and unknown physical parameters of the manipulator. A novel sliding mode visual feedback scheme is proposed to solve the problem of the robust trajectory tracking of a planar manipulator in the image frame without calibrating the camera parameters. The controller does not use visual velocity so as to achieve high and robust performance with low sampling rate of the vision system. It is proved by Lyapunov direct method that the tracking error of the robot converges to an arbitrarily small neighborhood of zero. The simulation is included to demonstrate the effectiveness of the controller proposed.

In this paper, we propose an approach to design a proper fixture layout for a 3D curved workpiece. The problem is tackled in a point set domain by discretizing the exterior surface of the workpiece into a dense collection of candidate fixturing points and then searching for a small proper set of fixturing points that satisfies the total restraint of the workpiece and reduces the workpiece positioning error. The algorithm first randomly selects an initial set of seven fixturing points and then iteratively improves it by exchanging with the other candidate points through a best-first strategy and a randomized motion. Finally, the algorithm has been implemented and its efficiency has been ascertained by two examples.

This paper concentrates on transparency and synchronization of supermedia in Internet based teleoperation. Supermedia is used to describe the collection of all the feedback streams in teleoperations, such as haptic, video, audio, temperature and others. Transparency and synchronization are introduced and analyzed from the event-based control perspective. The concepts of event-transparency and event-synchronization for event-based control telerobotic systems are developed and their implications are studied. To illustrate those concepts and their benefits, the teleoperation of a mobile manipulator via the Internet, where haptic, video and temperature information is fed back to the operator, is discussed. Experimental results will verify the event-transparency and event-synchronization of this event-based telerobotic system.

This paper presents a QoS based efficiency improving scheme for Internet-based teleoperation. One of the widely used QoS parameters for showing network status is network delay. With the help of event-based control technique, the stability of the teleoperation systems is guaranteed under random network delay. This paper investigates how to improve the efficiency of teleoperated tasks based on the network quality by introducing a Command Negotiator and a robot controller gain adjustment scheme using the measured QoS parameters into the proposed QoS based teleoperation systems. The presented methods improve the efficiency and system responses of the teleoperation systems when the network quality is poor. Moreover, a teleoperation experiment is presented to demonstrate the effectiveness of the proposed controller gains adjustment scheme.

The calibration accuracy of the intrinsic and extrinsic parameters of the vision system greatly affects the performance of visual servoing. We address the problem of controlling a planar manipulator using a fixed single camera without calibrating its intrinsic parameters and the transformation matrix between the robot base frame and the camera frame, and without measuring manipulator's depth. Based on an important observation that the unknown parameters can be separated from the unknown composite image Jacobian matrix, we propose an adaptive algorithm to estimate the unknown and mixed parameters on-line. It is proved with a full consideration of dynamics of the system by Lyapunov approach that the feature points of planar manipulator approach asymptotically to the desired ones on image plane and the estimated parameters are bounded under the control of the proposed visual servo controller. The performance has been confirmed by simulations and experiments.

The evolution of human-to-computer input devices lags far behind the evolution of processing power. In this paper, we present work on merging MEMS force sensors and existing wireless technologies to develop a novel multifunctional interface input system, the Micro Input Devices System (MIDS), which could potentially replace the mouse, the pen, and the keyboard as input devices to the computer. Moreover, initial experimental results indicate that further exploration of this technology could eventually produce a new control-input device for grasping robotic manipulators. We have thus far developed a prototype MIDS that consists of two MIDS rings, each packaged with commercial MEMS acceleration sensors to sense multi-axes motion, and a MIDS wrist watch that communicates with the rings and transmits data wirelessly to interface with a CPU. The system has been demonstrated to perform click and drawing motions successfully. A self-calibration method was also developed to resolve ambiguities in sensed motion for the MEMS sensors.

For the visual servoing tasks, it is required to calibrate accurately the homogeneous transformation matrix between the robot base frame and vision frame besides the intrinsic parameters of the vision system. In this paper, based on an important observation, that is, the unknown transformation matrix between the robot base frame and vision frame can be separated from the visual Jacobian matrix, and by virtue of decomposition of rotation matrix, we design a novel adaptive position-based visual servo controller for manipulators when the transformation matrix is not calibrated. It is proved with a full dynamics of the system by the Lyapunov approach that the motion of the manipulator approaches asymptotically to the desired trajectory. Simulations and experimental results both demonstrate the performance of this new controller.

The paper proposes a new haptic system for the pulse palpation via the Internet for remote diagnosis of patients without time and space restrictions. First, we design a pulse-detect sensor to obtains the pulse signals from the patient's wrist. Next, a server-client process is developed to transfer the pulse signals to the remote computer. Then, a haptic device is used to provide an interface for the doctor to feel the pulses of the patient at a remote distance. The haptic device regenerates the pulse signals by a motor-controlled cam system. To couple the doctor with the remote environment and give him a sense of telepresence, a trajectory follower is implemented: the cam follower moves up and down in accordance with the desired pulse trajectory. In addition, by considering the system as a one-dimensional manipulator, a PID force controller is realized to trace the desired force obtained from pressure sensors on the patient's side. As a result, the force felt by the doctor from the haptic device is closed to that he/she would feel from the patient's wrist directly.

This paper proposes a simple and efficient algorithm for computing a form-closure grasp on a 3D polyhedral object. This algorithm searches for a form-closure grasp from a "good" initial grasp in a promising search direction that pulls the convex hull of the primitive contact wrenches towards the origin of the wrench space. The "good" initial grasp is a set of contact points that minimizes the distance between the origin and the centroid of the primitive contact wrenches, and can be calculated by the quadratic programming. The local promising search direction at every step is readily determined by the ray-shooting based qualitative test algorithm developed in our early work. By using the "good" initial grasp, the iteration times of search can be significantly reduced so that a form-closure grasp can be found more efficiently. Since the algorithm adopts a local search strategy, its computational cost is less dependent on the complexity of the object surfacer. Finally, the algorithm was implemented and its efficiency ascertained by three examples.

Robotic operations carried out via the Internet face several challenges and difficulties. These range from human-computer interfacing and human-robot interaction to overcoming random time delay and task synchronization. These limitations are intensified when multi-operators at multisites are collaboratively teleoperating multirobots to achieve a certain task. In this paper, a new modeling and control method for Internet-based cooperative teleoperation is developed. Combining Petri net model and event-based planning and control theory, the new method provides an efficient way to model the concurrence and complexity of the Internet-based cooperative teleoperation. It also provides an efficient analysis tool to study the stability, transparency and synchronization of the system. Furthermore, the new modeling and control method enables us to design an Internet-based cooperative telerobotic system that is reliable, safe and intelligent. This new method has been experimentally implemented in a three site test bed consisting of robotic laboratories in the USA, Hong Kong and Japan. The experimental results have verified the theoretical development and further demonstrated the advantages of the new modeling and control method.

To implement a visual feedback controller, it is necessary to calibrate the homogeneous transformation matrix between the robot base frame and the vision frame besides the intrinsic parameters of the vision system. The calibration accuracy greatly affects the control performance. In this paper, we address the problem of controlling a robot manipulator using visual feedback without calibrating the transformation matrix. We propose an adaptive algorithm to estimate the unknown matrix online. It is proved by the Lyapunov method that the robot motion approaches asymptotically to the desired one and the estimated matrix is bounded under the control of the proposed visual feedback controller. The performance was confirmed by simulations and experiments.

Discusses the issue of hybrid position and force control of a two-manipulator system manipulating a flexible beam in trajectory tracking. Unlike our previous approach of set-point position control, (Sun and Liu, 1997, and Liu and Sun, 2000), in the trajectory tracking, the system coordinates are hard to regulate to the desired states with non-zero tracking velocities under continuous feedback control. In this study, we design a hybrid position and force tracking controller while using saturation control to compensate the effect of beam vibration dynamics to the tracking performance. All parameters and states used in the controller are readily available so that the proposed method is feasible in implementation. Under the proposed controller, the tracking error asymptotically converges to a predetermined boundary. Simulation results demonstrate the validity of the proposed approach.

Fixtures play an important role in many manufacturing operations such as inspection, machining and part fabrication. In the development of a fixture, it is desired that the feasible fixturing surfaces and optimal fixturing locations on the workpiece be selected automatically. An algorithm is presented to determine optimal fixturing locations which totally restrain the workpart in the fixture without being disturbed by any external force. First, based on observation that form-closure fixturing points exist on a set of surfaces if and only if the convex hull of the vertex contact wrenches resulting from the vertices of the surfaces contains the origin of R/sup 6/, an efficient approach is developed for finding an eligible set of fixturing surfaces. Second, we formulate the problem of determining optimal fixturing points on the eligible set of fixturing surfaces as a quadratic programming (QP) problem with the workpiece positioning accuracy as the performance index and the robust form-closure requirement as the linear constraints. Finally, the implementation for two numerical examples demonstrates the usefulness and efficiency of the proposed algorithm.

This paper presents an algorithm to calculate the fingertip positions, which can enhance the closure property for a n-finger grasp and ensure the kinematic feasibility. The algorithm starts by selecting a set of initial fingertip positions within the workspace of the robotic hand and on the surface of the object. If the currently selected grasp does not form closure, then the origin of the wrench space lies out side of the convex hull of the primitive contact wrenches. In this case the adjacent positions of each grip point can be generated to form a candidate set of grasps and the most promising one will be adopted based on the measure of a grasp from being a form closure. The measure is defined by considering the distance between the convex hull and the origin. The algorithm is executed iteratively until the origin is eventually contained by the convex hull. Finally, we illustrate the efficiency of the proposed algorithm with an implementation of three numerical examples.

Presents an integrated tactile feedback system for a multifingered robot hand to enable a human operator to feel contacts/interactions between the robot finger and the environment remotely. The system presented consists of a finger-shaped tactile sensor measuring contact areas on the fingertip and a tactile display rendering the contact information to the human operator. The tactile sensor, designed on the total internal reflection principle, can capture high resolution and high quality tactile images on the fingertip. The tactile display with 24 pins spaced at 2.5 mm uses DC solenoids structured in multi-layers to render the contacts between the fingertip and the environment. We have integrated the tactile sensor and the tactile display into a five-fingered robot hand system and verified the performance of the integrated system by experiments.

To implement a position-based visual feedback controller for a manipulator, it is necessary to calibrate the homogeneous transformation matrix between its base frame and the vision frame besides the intrinsic parameters of the vision system. In this paper, based on an important observation that the unknown transformation matrix can be separated from the visual Jacobian matrix, we design an adaptive controller for manipulators when the matrix is not calibrated. It is proved, with a full dynamics of the system, by the Lyapunov approach that the motion of the manipulator approaches asymptotically to the desired trajectory. Simulations and experimental results both demonstrate the performance of this new controller.

Fixtures play an important role in many manufacturing operations such as inspection, machining, and part fabrication. In the development of a fixture, it is desired that the feasible fixturing surfaces and optimal fixturing points on the workpiece be selected automatically. An algorithm is presented to automate the generation of optimal fixturing points, which totally restrains the workpiece in the fixture, i.e., satisfies the form-closure condition, as well as minimizes the workpiece positional errors. First, a heuristics is proposed for searching a set of fixturing surfaces capable of providing form-closure. Then, we formulate the problem of determining optimal fixturing points on the eligible set of fixturing surfaces as a quadratic programming problem with the workpiece positioning accuracy as the performance index and the robust form-closure requirement as the linear constraints. Finally, the implementation for three numerical examples demonstrates the usefulness and efficiency of the proposed algorithm.

We address the problem of computing optimal fingertip locations yielding n-finger form-closure grasps of polyhedral objects. Given grasping points of m fingers not in form-closure, we show that the calculation of optimal grip points for the n-m fingers can be viewed as a nonlinear programming problem. We propose a sufficient and necessary condition for the n-m fingers to achieve an n-finger form-closure grasp with the m fingers given. Based on this condition, we derive a set of nonlinear constraints for the nonlinear programming problem. Furthermore, we give a performance index serving as an objective function which measures the distance between the center of mass of the grasped object and the contact points to be determined. In addition, the algorithm proposed can be applied to the frictionless grasp. We have implemented the algorithm and verified its efficiency by two examples.

Presents a formulation of the synthesis of 3D frictional form-closure grasps of n robot fingers. First, using a recursive reduction technique, we transform the problem in 6D wrench space to one in three dimensions. Then we rewrite the sufficient and necessary condition for form-closure grasps in the equivalent form of the inconsistency of each of the two sets of linear inequalities. Then we propose three conditions to check the inconsistency of the inequality system, which geometrically indicates whether the convex region formed by the inequality system is empty. We have implemented the algorithms and confirmed their real-time efficiency for the synthesis of 3D form-closure grasps.

The use of the Internet is no longer limited to the transmission of data. In the past few years many successful attempts have been made to use the Internet as a command transmission media; through which control can be sent to remote systems and feedback can be obtained. But with this media come several limitations: delay, lost packets and disconnection. All of these limitations may cause instability in the system especially if the system loop is closed. All the previous work addressing these problems assumed several conditions; for example, time delay is constant or has an upper bound, control is not in real-time. A new real-time control approach is presented that deals with these limitations without any assumptions made regarding delay. The approach is based on event-based control, which was implemented on a mobile robot over the Internet. The commands sent to the robot are velocity and the feedback is force based on the environment. It will be shown that this approach results in a stable system. In addition, a new force feedback generation method is used.

A new formulation of an adaptive categorization game of ART-type neural networks is described in this paper. Learning automata dynamics is introduced into the adaptive categorization game in ART-type networks so that the vigilance parameter /spl rho/ is adjusted to give variable-sized clusters. Moreover, repeated game analysis is also conducted on the learning automata-based adaptive categorization game in order to study the asymptotic behaviors of the game when data patterns are passed to the ART networks one after the other.

This paper presents a simple and efficient algorithm to calculate the fingertip positions which ensure a form-closure grasp. The algorithm first arbitrarily chooses a grasp on the given faces of the object. If the selected grasp is not form-closure, the origin O of the wrench space lies outside of the convex hull of the primitive contact wrenches. In this case, the algorithm properly moves the fingertip positions at a fixed step on the faces so that the primitive contact wrenches move towards the origin O until the origin is eventually contained by the convex hull after iterations of the operation. The form-closure property is checked by our ray-shooting based qualitative test algorithm. The motion of the fingertip positions at every step is determined by a quadratic programming problem. Finally we have implemented the proposed algorithm and verified its efficiency with three numerical examples.

Factor analysis has been employed for data analysis in behavioral sciences for decades. In this paper, we propose to employ it in robot behavior studies so that important underlying factors that affect the decision-making in robot behavior actions can be extracted. Causal relationships among physical (observed) and logical (unobserved) perceptual dimensions are constructed. Factor analysis provides a simple mean for us to understand what the sensors data, that construct the robot behavioral perceptual space S, are measuring (logical perceptual space extraction). Learning can thus be conducted based on the logical dimensions of the perceptual space, which usually has much lower dimensionality than the original physical perceptual space, of robot behaviors. Analysis of simulated obstacle avoidance behavior is presented.

To implement a visual feedback controller, it a's necessary to calibrate the homogeneous transformation matrix between the robot base frame and the vision frame besides the intrinsic parameters of the vision system. The calibration accuracy greatly affects the control performance. We address the problem of controlling a robot manipulator using visual feedback without calibrating the transformation matrix. It is assumed that the vision system can measure the 3D position and orientation of the robot in real-time. Based on the fact that the visual Jacobian matrix can be represented in a linear form of elements of the transformation matrix, we propose a simple adaptive algorithm to estimate the unknown matrix on-line. This visual feedback controller greatly simplifies the implementation process of a robot-vision workcell and is especially useful when a pre-calibration is not possible, such as when a robot works with an active vision system carried by a mobile robot. It is proved by the Lyapunov approach that the robot position approaches asymptotically to the desired one and the estimated matrix is bounded under the control of this visual feedback controller. The performance has been confirmed by simulations and experiment.

A dynamic simulator can facilitate developments and applications of a multi-fingered robot hand. Existing dynamic simulators cannot effectively simulate dextrous manipulation of a multi-fingered robot hand due to the lack of capability to cope with frequent changes in contact constraints and grasping configurations as well as impulsive collision occurring during manipulation. We propose a unified framework to model free motions, collisions, and different contact motions including sticking, rolling, and sliding. Furthermore, a new transition model is also proposed to handle transitions between these contact motions. Based on our unified dynamic model, a 3D dynamic simulator has been developed to simulate dextrous manipulation tasks involving combination of different contacts. The simulation results presented confirm the validity of the dynamic model as well as efficiency of the developed simulator.

Assuming that grasp points of n-1 fingers on a polyhedral object are given and the n-1 fingers do not achieve a form-closure grasp, this paper proposes an efficient algorithm for computing all grasp points on the object for the n-th finger to achieve a form-closure grasp with other n-1 fingers. A sufficient and necessary condition is developed for forming the form-closure grasp. Based on the sufficient and necessary condition, we transform the grasp computation to a problem of: 1) computing the convex cone of the primitive contact wrenches of the n-1 fingers; and 2) checking the intersection between a circular region and a convex polygon in a 2D space. We have implemented the algorithm and confirmed its usefulness by an example.

The concept of adaptive categorization is introduced to ART-type networks in this paper. Adaptive categorization capability also improves learning performance in self-organizing systems and online learning systems. Classical ART-types networks, however, have only fixed single size cluster formation in categorization, which is controlled by the scalar vigilance parameter. This categorization methodology usually cannot give satisfactory results as the data pattern space is not covered thoroughly by fixed boundary clusters. A game-theoretic formulation and analysis on the competitive clustering nature of ART-type networks are presented. A game-theoretic vigilance parameter adaptation algorithm is then proposed to form variable sized clusters so that the data pattern space is covered much thoroughly. Simulations are presented to demonstrate reliable categorizations obtained from variable sized clusters using game-theoretic vigilance parameter adaptation.

This paper addresses the problem of computing n-finger form-closure grasps of polyhedral objects. Given grasping points of m fingers not in form-closure, we propose a sufficient and necessary condition for other n-m fingers to achieve an n-finger form-closure grasp with the m fingers. Based on this condition, it is demonstrated that the problem of computing grasping points of the n-m fingers can be formulated as an existence problem of a solution for a set of linear inequalities. Several sufficient conditions are also presented to guarantee the form-closure property of the n-finger grasp.

This paper formalizes qualitative test of 3D frictional form-closure grasps of n robotic fingers as a problem of linear programming (LP). It is well-known that a sufficient and necessary condition for form-closure grasps is that the origin of the wrench space lies inside the convex hull of primitive contact wrenches. We demonstrate that the problem of querying whether the origin lies inside the convex hull is equivalent to a ray-shooting problem, which is dual to a LP problem based on the duality between convex hulls and convex polytopes. Furthermore, this paper addresses a problem of minimizing the L/sub 1/ norm of the grasp forces balancing an external wrench, which can be also transformed to a ray-shooting problem. We have implemented the algorithms and confirmed their real-time efficiency for qualitative test and grasp force optimization.

This paper studies the problem of modeling and control of multiple cooperative underactuated manipulators handling a rigid object. We reveal holonomic property of such a system by presenting a smooth feedback controller subject to two conditions: 1) there are not fewer active joints than the degrees of freedom of the object; and 2) the Jacobian matrix with respect to passive joints is not singular. This controller is an extension of the PD plus gravity compensation scheme and its asymptotic stability is guaranteed by the LaSalle theorem. Furthermore, we develop a trajectory tracking controller that yields asymptotic convergence of position errors and bounded interaction forces simultaneously. The performance of the proposed controllers has been investigated by simulations on two 6-DOF underactuated manipulators and by experiments on the cooperative underactuated manipulator system developed at CMU.

This paper presents an efficient algorithm for computing all n-finger force-closure grasps on a polygonal object. This algorithm is based on a new qualitative test algorithm for force-closure grasps which recursively transforms the problem in the three dimensional wrench space to a problem in an one dimensional space. We demonstrate that nonforce-closure grasps are two convex polytopes in the space of n parameters that represent grasp points on sides of the polygon. Therefore, the force-closure grasp region is calculated by subtracting the convex polytopes from the parameter space. The qualitative test algorithm takes O(n/sup 3/) time and the grasp computation algorithm takes O(n/sup 3/ log n) time for n/spl les/3 and O(n/sup 3n/2/) time for n>3, where n is the number of the fingers. The efficiency of the algorithms is confirmed with simulations.

This paper formalizes qualitative test of 3D frictional form-closure grasps of n robotic fingers as a problem of linear programming (LP). It is well-known that a sufficient and necessary condition for form-closure grasps is that the origin of the wrench space lies inside the convex hull of primitive contact wrenches. We demonstrate that the problem of querying whether the origin is contained by the convex hull is equivalent to a ray-shooting problem, which is dual to a LP problem based on the duality between convex hulls and convex polytopes. Furthermore, this paper addresses a problem of minimizing the L/sub 1/ norm of the grasp force balancing an external wrench, which can be also transformed to a ray-shooting problem. We have implemented the algorithms and confirmed their real-time efficiency.

This paper verifies that the widely used PD plus a force control scheme is also suitable for controlling a system of two industrial robots manipulating a flexible sheet. It is proven by LaSalle's theorem that under the proposed control law, the desired rigid body motion can be achieved and the vibrations of the sheet at each contact are suppressed simultaneously. The offsets of all static deformations of the sheet with reference to the original positions decay to zero. The internal forces between the payload and the robots are well controlled to avoid any damage to the system. The investigation is based on the decomposition of the payload dynamics into two distinct dynamic subsystems, using a "clamped-free" model. The experiments on two CRS A460 robots manipulating a flexible aluminum sheet confirm these theoretical predictions.

We propose in this work the first model-based robust control method for a team of underactuated manipulators jointly manipulating a load. The method is based on feedback linearization of the nonlinear dynamic coupling between the torques applied at the actuated joints and the Cartesian acceleration of the load, combined with a variable structure controller. Singularities in the control method are addressed, and a sufficient condition for a singularity-free controller implementation is obtained. Simulation and experimental results are presented to validate the theory presented.

Robot behavior learning is an emerging research topic in robotics. By incorporating learning capability to robots, engineers are not required to hard-code appropriate actions under every possible situation. Actually, this is an impossible task. In this paper, an architecture of behavior learning/operating module (BLOM) for a robot system is proposed. In the BLOM architecture, several categories of situations and actions are formed and mappings among the situation and action categories are established. A Knight Tournament (KT) strategy is proposed for adaptive categorization of situation and action patterns in learning. A computer simulation on learning a robot behavior is also presented.

A single manipulator with passive joints is most likely nonholonomic systems, but multimanipulator systems may not. This paper investigates this issue by presenting a smooth feedback stabilization controller when the number of passive joints is not more than that of motion constraints associated to cooperations. This controller is a variation of the classical PD plus gravity compensation scheme and its asymptotic stability is guaranteed by LaSalle theorem. On the basis of this controller, we further discuss holonomy and nonholonomy conditions of multi-manipulator systems with passive joints. In addition, we propose a trajectory tracking controller which gives rise to asymptotic convergence of position errors and bounded interaction forces. Finally, we demonstrate asymptotic convergences of the proposed controllers with simulation study.

In this paper, a hybrid impedance control algorithm is proposed to stabilize a flexible beam handled by two manipulators to a desired position/orientation while suppressing its vibration, and simultaneously control the internal forces between the manipulators and beam. The algorithm combines impedance control and an I-type force feedback by designing a proper response of the interaction force including external and internal forces. No information about the vibration is used in the controller. The asymptotic stability is analyzed based on the vibration dynamics of the beam approximated by m assumed modes, where the number m can be as large as necessary. Three particular cases of using different mode functions are discussed under free-free, clamped-free and pinned-pinned boundary conditions. The validity of the proposed scheme is demonstrated by simulations.

Robotic manipulation of a general flexible object is an extremely difficult and challenging control problem. This paper shows that under a simple PD position feedback, the position/orientation of a general flexible object handled by two manipulators is able to approach the desired one and at the same time the vibration of each contact is suppressed. We use the "clamped-free" model to decompose the motion of the object into two components, a rigid and a flexible one, which allows us to treat them separately and achieve desired motions with a simple PD scheme. This is proved to work theoretically.

Two decentralized cooperation controllers are presented for trajectory tracking of two manipulators handling an object. The controllers control positions of the robots distributively by using trajectory errors of the object in the task space. In the first controller, the internal force between the object and the manipulator is controlled only by feedforward of the desired force. The second controller uses a force feedback. No communication is required between the manipulators in the both controllers. Their globally and exponentially asymptotic stability are guaranteed by Lyapunov functions.

This paper deals with the decentralized control of multiple manipulators in a class of cooperative tasks called holonomic cooperation and presents two joint-space controllers. The controllers decouple the position and force control by the joint-space orthogonalization method. The first controller implicitly realizes the force control through a feedforward of the desired force, but the second one includes a loop of the force feedback. Their global and exponential stability is guaranteed by Lyapunov functions. Unlike centralized controllers, our decentralized controllers are designed in the joint spaces of individual robots so that the implementation does not need to manipulate high dimensional matrices.

A new approach for modeling and cooperating two manipulators handling a deformable object is presented. Based on the decomposition of a deformable body into a reference component and a deformation component, a general deformable model is developed and the complex control task is divided into two subtask, i.e., the control of the reference motion and the control of the deformations. The position/force controllers and a null-space control law are proposed and demonstrated in the simulations.

This paper presents a general approach of model-based adaptive hybrid control for holonomically constrained manipulators. Both time-invariant and time-variant constraints are dealt with. This controller works on the basis of an orthogonalization principle of trajectory and force errors in the joint space and intrinsic characters of the manipulator dynamics. Its asymptotic stability is guaranteed by a passivity-based Lyapunov function. A distributed cooperation controller based on the same concept for multiple manipulators is also presented. Further, we show a parallel implementation of this controller using transputers on a six DOF direct drive manipulator, and some experimental results.

This paper presents a general approach for adaptively and distributively controlling multiple cooperative manipulators. The proposed approach does not adopt a centralized architecture but assigns a controller to each robot. Any communication requirement is determined by motion constraints existing in the cooperative system. All physical parameters of the manipulators or the load of the system are online estimated by a model-based adaptive algorithm. A Lyapunov function guarantees asymptotic convergence of trading errors of the trajectory and the interactive force among the robots. Performance of this controller is further shown by simulations on six DOF manipulators.

An algorithm for finding the shortest path of a disc among a set of polygonal obstacles is presented. Let N/sub t/ denote the total number of obstacle vertices and N/sub c/ the number of convex vertices. Our algorithm uses a radius-independent data structure called extended tangent graph (ETG) which registers collision-free tangents of the obstacles according to different discs and takes O(N/sub c//sup 2/) space. The ETG depends only on original obstacles, and it is constructed in advance without using any information about the disc in O((N/sub c/+k)N/sub t/) computation time, where k is the number of outer common tangents of the obstacles. It takes O(N/sub t/logN/sub t/) time to partially update the ETG to reflect the start and goal of a given disc. The shortest path is planned by a graph-search algorithm.>

This paper has a twofold objective: 1) to show that, when an adaptive law designed with a Lyapunov function and dependant also of the parametric error is introduced into a variable structure controller the existence of a sliding mode is no longer guaranteed for all t; and 2) a time varying boundary layer modulated by the tracking errors to propose a variable structure robot control with asymptotic stability. It is shown that the high activity on the actuators, typical of variable structure controllers, can be arbitrarily reduced in both adaptive and nonadaptive cases. These controllers give rise to the asymptotic stability and at the same time the sliding mode is guaranteed with very small chattering. A trade off arises in terms of chattering and robustness but the convergence of tracking errors is always assured. Computer simulations show the performance of the proposed controllers with comparison of two other algorithms.>

This paper deals with acceleration (or torque) constraints in motion planning of a manipulator among moving obstacles. It is known that motion planning with dynamic constraints in dynamic environments can be formulated in a state-time space. This paper shows that acceleration constraints of a manipulator can be represented as parabolas in a local small motion so that admissible velocities are efficiently computed. An algorithm is also presented for planning the optimal trajectory for a manipulator on a given path. To improve computation efficiency, this algorithm does not construct or search the total 3D state-time space but only a subspace corresponding to admissible regions in the 2D time-configuration and state spaces. Its usefulness is shown by a simulation for a 6 DOF manipulator.>

The tangent graph is a powerful graphic data structure for the path-planning of a mobile robot among obstacles because it has fewer edges than the widely used visibility graph. This paper proposes an efficient algorithm for computing the tangent graph of a set of polygonal obstacles, The algorithm at first detects common tangents of obstacle boundaries by moving-line processing that cooperatively moves a straight segment on the boundaries, and then checks for intersections among detected tangents and the obstacles both by a sequential approach and a sweep-line technique. It is proved that the moving-line processing takes O(MN) and O([M+R]N) computation time in the best and worst cases, respectively, where N expresses the number of obstacle vertices, and M denotes the number of convex segment chains of the obstacle boundaries, and R is a parameter representing the complexity of the boundaries.>

Introduces the concept of configuration patches (C-patches) for representing configurations of a polyhedral robot in a point-point contact with polyhedral obstacles, and proposes an approach called the inverse mapping method for computing an approximate representation with cubic cells of the configuration space (C-space) of the robot. The configuration patches represented in a local coordinate system, have simple geometric shapes like paralleltopes. The inverse mapping method constructs the approximation of a C-space by quantizing the C-patches in the local coordinate system. There is no need to check any intersection among the robot and the obstacles. The only process needed is a coordinate transformation from discrete points on the C-patches to configurations of the robot.

This paper describes an efficient algorithm of path planning for an internal gas pipe inspection robot. Since the purpose of this robot is to move in pipes to detect injures and scratches, the path should cover all pipes in a given map. The problem to find a path along which the robot can investigate all pipes at least so that the total distance is minimum is called 'Chinese Postman Problem'. Usually this problem is NP-complete. If all values of pipes are non-negative, the numerical programming is applied for finding the optimal path. However this is still too complicated to apply for searching the path. Motivated by this motivation, a method to find the optimal path based on the A* algorithm is proposed, which is treated more easily than the numerical programming. Finally some numerical simulation results are presented.

A tangent graph (T-graph) for path planning of mobile robots among polygonal and curved obstacles is proposed. In the T-graph, the nodes correspond to tangent points on obstacle boundaries and the edges represent collision-free common tangents of the obstacles or convex boundary segments between the tangent points. The T-graph requires O(K/sup 2/) memory, where K denotes number of convex segments of obstacle boundaries. Further, to avoid recomputation of configuration spaces of circular robots in a polygonal environment when their radii has changed, an extended tangent graph (ETG) is developed. The ETG does not depend on obstacles in configuration spaces of robots but on obstacles in real workspace. Consequently, it is possible to plan a path flexibly according to different robots' sizes and safety distances by the ETG.>

Presents an approach for optimizing the joint torques on the minimum-time trajectory in the criterion of minimum torque change. The torque change a summation of absolute values of differences between the torques at two discrete points, is a nonlinear function. To avoid the authors nonlinear optimization, the authors introduce a linear objective function which gives the same effect. On the basis of the linear objective function and dynamics of the system which constrains the joint torques and the internal forces between the manipulators and the object linearly, the linear programming approach is used to optimize the joint torques.>

When multiple mobile robots are working in the same environment, planning of collision-free coordinated motion is necessary; here, an algorithm for planning such a motion of two mobile robots, no matter how crude the constraints of obstacles are, is proposed. The situation is modeled as a Petri net, which is considered as a useful model for describing and analyzing a system in which it is possible for some events to occur concurrently but there are constraints on the concurrence. In the Petri net, all motion constraints of robots in their paths are arranged as its firing rules, and hence collision-free coordination between the robots can be easily planning by manipulation of the firing rules. The algorithm always finds a collision-free coordinated path of two robots if there actually exists such a path in the environment. Moreover, because the algorithm does not use any knowledge of movement of the robots, precise time-varying trajectory control is not required and realization of the coordination is easy. The algorithm works efficiently even in a complex environment, indebted to the generic properties of geographical quadtree modeling for the environment. The usefulness of the algorithm is shown by several simulations.>