Robot scooping is a challenging and important task in robotic tool manipulation research due to the complex relationship between the robot, the tool, and target objects/environment. Taking into account different tools, different target objects and varying environments, the required scooping manipulation strategy usually varies greatly. Even considering a specific type of spoon, the question of how to obtain a policy model that requires less demonstration data but shows better generalization capabilities deserves further exploration. In this paper, we propose a progressive learning framework for general robot scooping tasks, which requires a limited number of demonstrations but shows promising generalization capability. We first learn a scooping policy via human demonstrations with a specific setup. We then use this as a pre-train model for reinforcement learning in a curriculum manner to achieve a scooping strategy that is generalizable to different task setups. Finally, we evaluate the capabilities of the policy with a series of experiments both in simulation and on a real robot.

Robotic hand tool use has garnered significant attention from robotics researchers, because it enhances dexterity beyond the limitations imposed by manipulators with fixed tool configurations and human-involved manual tool changes. Despite extensive research, current methodologies predominantly focus on imitating human hand trajectories, often neglecting the pivotal role of tool-environment interaction. This study addresses this gap by exploring the task of cucumber peeling as a case study to implement contact-based demonstration strategies in robotic tool use. Our approach concentrates on the subtle tool contact behaviors that manifest through contact dynamics. Specifically, we select appropriate tool stiffness for the peeling tasks, which is captured via a handheld teaching device equipped with optical tactile sensors. Subsequently, object-level stiffness control strategies are employed to emulate these behaviors using a three-fingered robotic hand. Experimental results from real-world cucumber peeling trials substantiate our methodology, illustrating that the robotic hand can adjust contact through finger movements, thereby achieving humanlike peeling efficiency without necessitating alterations to the tool structure. This study not only demonstrates the feasibility of sophisticated tool use by robotic hands, but also highlights the critical importance of integrating tactile feedback to refine interaction with the environment.

Robotic grasping is a fundamental ability for a robot to interact with the environment. Current methods focus on how to obtain a stable and reliable grasping pose in object level, while little work has been studied on part (shape)-wise grasping which is related to fine-grained grasping and robotic affordance. Parts can be seen as atomic elements to compose an object, which contains rich semantic knowledge and a strong correlation with affordance. However, lacking a large part-wise 3D robotic dataset limits the development of part representation learning and downstream applications. In this paper, we propose a new large Language-guided SHape grAsPing datasEt (named LangSHAPE) to promote 3D part-level affordance and grasping ability learning. From the perspective of robotic cognition, we design a two-stage fine-grained robotic grasping framework (named LangPartGPD), including a novel 3D part language grounding model and a partaware grasp pose detection model, in which explicit language input from human or large language models (LLMs) could guide a robot to generate part-level 6-DoF grasping pose with textual explanation. Our method combines the advantages of humanrobot collaboration and LLMs’ planning ability using explicit language as a symbolic intermediate. To evaluate the effectiveness of our proposed method, we perform 3D part grounding and fine-grained grasp detection experiments on both simulation and physical robot settings, following language instructions across different degrees of textual complexity. Results show our method achieves competitive performance in 3D geometry fine-grained grounding, object affordance inference, and 3D part-aware grasping tasks. Our dataset and code are available on our project website https://sites.google.com/view/lang-shape.

This paper presents an efficient unified algorithm for the minimum Euclidean distance between two collections of compact convex sets, each of which can be a collection of convex primitives, such as ellipsoids, capsules, and cylinders, or a collection of triangles (i.e., triangle mesh) or a collection of points (i.e., point cloud) as special cases. The Euclidean distance between two compact convex sets is defined to be the smallest translation to bring them into intersection if they are separated or to separate them if they intersect, which can be computed by the well-known Gilbert-Johnson-Keerthi (GJK) and expanding polytope (EP) algorithms, respectively. While existing algorithms are aimed at computing the minimum Euclidean distance for a specific type of collections, algorithms for mixed situations always remain vacant. We discover that the smallest translation direction between any two compact convex sets determines the planes to bound and separate some other sets in two collections and can help quickly identify sets that do not have the minimum distance. In this way, the minimum distance between two collections can be efficiently computed, hundreds to thousands of times faster than the brute-force search. The computational efficiency of the proposed algorithm is verified with a number of numerical experiments in various scenarios.

This paper presents a new efficient way to quantitatively measure separation and penetration between a collection of convex primitives (incl. ellipsoids, capsules, cylinders, convex polyhedra, and triangles) and a point cloud or a triangle mesh. First, the minimum scaling factor of a convex primitive with respect to its centroid to contact a point or a triangle is proposed as a new distance metrics, which can be greater than, equal to, or less than one, implying that the point or the triangle is separated from, just contacts, or penetrates into the convex primitive. It can be computed mostly in closed form or occasionally with a 1-D gradient descent search, which is much faster than computing the Euclidean distance. Furthermore, an efficient algorithm is proposed to compute the smallest minimum scaling factor of convex primitives in a collection to a point cloud or a triangle mesh. It is based on the discovery that computing the minimum scaling factor of a convex primitive to a point or a triangle yields a plane separating more points or triangles from this or other convex primitives. Then, the overall smallest scaling factor can be found by checking only a few pairs of primitives and points or triangles, being significantly faster than the exhaustive search. In various numerical examples and comparison with the existing algorithms, the proposed metrics and algorithm show superior or comparable efficiency.

Tactile servoing is an effective approach to enabling robots to safely interact with unknown environments. One of the core problems in tactile servoing is to robustly converge the contact features to the desired ones via a dedicated controller. This paper proposes a Data-Driven Model Predictive Controller (DDMPC) to compute the motion command given the previous interaction experience and feature deviations in tactile space. Compared with the manually designed PID-based controller, the proposed controller depends on the sound control theory and its convergence is guaranteed from a computational perspective. It is applied to the balancing control of a rolling bottle on a robotic forearm covered by a custom tactile sensor array. The real experiment demonstrates the superior robustness of the proposed approach and shows its great potential for other tactile servoing scenarios with measurement noise, which is inevitable for current tactile sensors.

Anthropomorphic robotic arms, mimicking the structure and function of human arms, show great potential for helping people in various tedious and repetitive household tasks. However, such arms mostly consist of multiple serial links controlled independently by actuators at joints with high reduction ratios, posing challenges in household services in terms of load capacity, responsiveness, and safety. In this paper, we propose a high-performance anthropomorphic arm called TRX-Arm based on differential cable transmission, characterized by features of high dynamics, high load capacity, and inherent compliance. TRX-Arm is composed of three deferential cable-driven coupling joints and one independent roll joint. Thanks to the cable differential transmission, the joints are capable of achieving doubled torque and stiffness without replacing motors. To enhance safety in human-robot interaction, the actuators including motors, reducer, belt, and pulley are mounted at the shoulder near the base and drive the joints remotely using cables, thereby minimizing the inertia of the whole arm. The workspace of TRX-Arm has a volume of 1.56 m3, much larger than that of the human arm. Real experiments show its capabilities including high repeatability and load capacity as well as high dynamic behavior of a dual-arm robot platform built with TRX-Arms.

Objects of daily life are designed to suit the human hand. Without major modifications to these objects and our environments, robots will need end-effectors with human hand-like configuration and dexterity to efficiently operate on them. Tight integration of tactile and proprioceptive sensors are also critical to ensure robust execution of manipulation policies without sacrificing range-of-motion. Reliability is also key, and a mechanically robust, easy to repair end-effector is important to minimize downtime. To meet these challenges, we designed a 13 degree-of-freedom anthropomorphic hand with over 1000 tactile sensing elements, named TRX-Hand5. Also embedded within are positional encoders and cable tension sensors to provide proprioceptive perception. TRX-Hand5 has a novel biomimetic topology with six small posture motors in the palm to replicate the function of intrinsic hand muscles and five large power motors in the forearm to play the role of forearm flexor muscles. The whole hand weighs 2.6 kg with its dimensions comparable to those of an adult male’s hand and is capable of actuating its fingertips at over 200°/s while exerting up to 22 N of force. The system can be disassembled in modules for easy maintenance.

Quadruped locomotion is challenging for many learning-based algorithms. This is because it requires tedious manual tuning to cope with different types of terrains and is difficult to deploy in reality due to the sim-to-real gap between the training and the testing scenarios. This paper proposes a quadruped robot learning system for agile locomotion which does not require any pre-training and works well in various terrains. We introduce a hierarchical framework that uses reinforcement learning as the high-level policy to adjust the low-level trajectory generator for a better adaptability to various terrains. We compact the observation and the action spaces of reinforcement learning to deploy the proposed framework on a host computer interfaced with the robot. Besides, we design an omnidirectional trajectory generator guided by robot posture, which generates omnidirectional foot trajectories to interact with the environment. Experimental results and the supplementary video demonstrate that our hierarchical framework only trained in simulation can be easily deployed in the real world, and also has the advantages of fast convergence and good terrain adaptability.Note to Practitioners—This paper presents a hierarchical framework for quadruped robots. It combines a high-level reinforcement learning controller with a posture-guided trajectory generator to adaptively generate omnidirectional motions. Our method is easy to train as it converges fast and does not need to adjust a dozen or so of rewards. The quadruped robot can be deployed in a real environment directly after being trained in simulation. With the trained hierarchical framework deployed on a remote host computer, the robot works well in a variety of real-world environments unseen in the simulation.

To enrich legged robots with fast energy-efficient mobility on even terrain, wheeled-legged robots have emerged as a valued robot form in robotics research. This paper describes the complete development of a new wheeled-legged quadruped robot named Max, ranging from its mechanical design over system architecture to core algorithms implemented for it to realize various motion behaviors. Instead of attaching wheels to the distal ends of legs as in the existing wheeled-legged robot designs, this robot has wheels installed on the knees with a special switching mechanism to convert a leg between the legged and wheeled locomotion modes. This design keeps the wheeled leg lightweight, enabling the robot to preserve the motion agility as a quadruped robot while gaining the energy-efficiency as a four-wheel or even two-wheel mobile robot. An online locomotion generation method is proposed to compute the 6-D body trajectory of the robot in walking on the perceived terrain, while dynamic movements such as leaps and flips are generated by a unified trajectory optimizer, which is also used to generate the transition motions of the robot to transform into the wheeled mode. The diverse mobility of the proposed robot Max is verified with extensive experiments.Note to Practitioners—Empowering robots with all-terrain mobility is a fundamental open problem in developing a new generation of robots. To this end, combinations of wheels and legs have been explored for robots to possess both traversability on uneven terrains and efficiency on even terrains. This paper proposes a new wheeled-legged quadruped robot with focuses on the integrated design of wheeled legs, system architecture, and core algorithms implemented for various legged and wheeled locomotion behaviors. To embed wheels without adding additional motors and keep the light weight of original legs, a special switching mechanism is designed and integrated at the knee joints where wheels are installed. Algorithms for generating quadrupedal walk according to online perceived terrain information as well as other dynamic legged and wheeled motions are discussed and demonstrated. The system architecture for allocating all vision and motion algorithms is also presented. This work is intended to provide a whole picture of developing this new robot including both hardware and software aspects.

Accurate measuring and modeling of dynamic robot manipulation (e.g., tossing and catching) is particularly challenging, due to the inherent nonlinearity, complexity, and uncertainty in high-speed robot motions and highly dynamic robot–object interactions happening in very short distances and times. Most studies leverage extrinsic sensors such as visual and tactile feedback toward task or object-centric modeling of manipulation dynamics, which, however, may hit bottleneck due to the significant cost and complexity, e.g., the environmental restrictions. In this work, we investigate whether using solely the on-board proprioceptive sensory modalities can effectively capture and characterize dynamic manipulation processes. In particular, we present an object-agnostic strategy to learn the robot toss dynamics of arbitrary unknown objects from the spatio-temporal variations of robot toss movements and wrist-force/torque (F/T) observations. We then propose TossNet, an end-to-end formulation that jointly measures the robot toss dynamics and predicts the resulting flying trajectories of the tossed objects. Experimental results in both simulation and real-world scenarios demonstrate that our methods can accurately model the robot toss dynamics of both seen and unseen objects, and predict their flying trajectories with superior prediction accuracy in nearly real-time. Ablative results are also presented to demonstrate the effectiveness of each proprioceptive modality and their correlations in modeling the toss dynamics. Case studies show that TossNet can be applied on various real robot platforms for challenging tossing-centric robot applications, such as blind juggling and high-precise robot pitching.

Grasping and manipulating various kinds of objects cooperatively is the core skill of a dual-arm robot when deployed as an autonomous agent in a human-centered environment. This requires fully exploiting the robot's versatility and dexterity. In this work, we propose a general framework for dual-arm manipulators that contains two correlative modules. The learning-based dexterity-reachability-aware perception module deals with vision-based bimanual grasping. It employs an end-to-end evaluation network and probabilistic modeling of the robot's reachability to deliver feasible and dexterity-optimum grasp pairs for unseen objects. The optimization-based versatility-oriented control module addresses the online cooperative manipulation control by using a hierarchical quadratic programming formulation. Self-collision avoidance and dual-arm manipulability ellipsoid tracking with high reliability and fidelity are simultaneously achieved based on a learned lightweight distance proxy function and a speed-level tracking technique on Riemannian manifold. Intrinsic system safety is guaranteed, and a novel interface for skill transfer is enabled. A long-horizon rearrangement experiment, a bimanual turnover manipulation, and multiple comparative performance evaluation verify the effectiveness of the proposed framework.

Designing safety-critical control for robotic manipulators is challenging, especially in a cluttered environment. This article proposes an online control barrier function (CBF) construction method, which extracts CBF from distance samples and enforces the safety of the motion control of robotic manipulators. Specifically, the CBF guarantees the controlled invariant property for considering the system dynamics. The proposed method samples the distance function and determines the safe set. Then, the CBF is synthesized based on the safe set by a scenario-based sum-of-square program. Unlike most existing linearization-based approaches, our method preserves the volume of the feasible space for planning without approximating the signed distance function, which helps find a solution in a cluttered environment. The control law is obtained by solving a real-time CBF-based quadratic program. Moreover, our method guarantees safety with the probabilistic result validated on a 7-DOF manipulator in real and virtual environments. The experiments show that the manipulator is able to execute tasks where the potential clearance between obstacles is in millimeters.

For the image-based visual servoing (IBVS) of a manipulator with an unknown structure, the unavailability of the robot Jacobian matrix impedes the accurate control of the manipulator. To solve this issue, this article proposes a data-driven IBVS (DDIBVS) scheme combining model-free learning, matrix inversion estimation, feature tracking, and joint limits. On the one hand, a data-driven learning algorithm is designed, which enables an estimated end-effector velocity to approach the real one and outputs an estimated robot Jacobian matrix. On the other hand, we consider the desired velocity information of the visual feature to improve the tracking accuracy and design an auxiliary parameter to estimate the inversion operation and address the singularity problem. On this basis, a neural dynamic controller (NDC) is developed, which possesses learning, estimation, and control capabilities. Subsequently, the effectiveness, practicability, and superiority of the proposed method are evaluated through simulations and experiments conducted on a 7-degree-of-freedom (DOF) manipulator for visual servoing tasks.

To fully explore the potential of robots for dexterous manipulation, this paper presents a whole dynamic grasping process to achieve fluent grasping of a target object by the robot end-effector. The process starts from the phase of approaching the object over the phases of colliding with the object and letting it roll about the colliding point to the final phase of catching it by the palm or grasping it by the fingers of the end-effector. We derive a unified model for this hybrid dynamic manipulation process embodied as approaching-colliding-rolling-catching/grasping from the spatial vector based articulated body dynamics. Then, the whole process is formulated as a free-terminal constrained multi-phase optimal control problem (OCP). We extend the traditional differential dynamic programming (DDP) to solving this free-terminal OCP, where the backward pass of DDP involves constrained quadratic programming (QP) problems and we solve them by the primal-dual Augmented Lagrangian (PDAL) method. Simulations and real experiments are conducted to show the effectiveness of the proposed method for robotic dynamic grasping.

This paper proposes a novel efficient multi-phase trajectory generation algorithm for dynamic dexterous manipulation tasks, such as throwing, catching, dynamic regrasping, and dynamic handover, which can be decomposed into multiple manipulation primitives, including sticking, rolling, approaching, separating, colliding, and grasping. Each manipulation primitive is formulate as a free-terminal optimal control problem (OCP), aimed at computing the optimal pose (position and orientation) trajectories of the object and the robot subject to the pose and force linkage constraints between them and the expected force maintenance at contact. A single-arm regrasping task and a dual-arm dynamic handover task are conducted to demonstrate the effectiveness of the proposed algorithm.

Collaboration between humans and robots can be challenging because robots may have difficulty understanding a specific person’s intentions, particularly in complicated tasks such as co-manipulation and assembly in computer, communication, and consumer electronics (3C) manufacturing. These tasks require different weights on accuracy and speed for various fabrication steps, making traditional physical interaction inadequate. In this paper, we introduce a fuzzy reinforcement learning-based admittance controller that can infer humans’ intentions not only through physical interaction but also through natural language. During training, the natural language is encoded into a reward term to help the robot reach the human-intended convergence point, allowing us to develop a “personalized” policy. During testing, the language serves as a tool to help the robot understand and obey humans’ intentions when physical interaction alone is insufficient. For example, if the user finds it difficult to push the robot and needs it to move faster, they can say “it’s really slow,” while a request for high-accuracy operation can be conveyed through “the damping is too small.” With this algorithm, the robot can comprehend the intentions and act accordingly in such situations. Further results and videos can be found at: https://sites.google.com/view/hri-nlp.

Living objects are difficult to grasp since they can actively elude capture by adopting adversarial behaviors that are extremely hard to model or predict. In this case, an inappropriately strong contact force may hurt the struggling living objects and a grasping algorithm that can minimize the contact force whenever possible is required. To solve this challenging task, in this article, we present a reinforcement-learning (RL)-based algorithm with two stages: the pregrasp stage and the in-hand stage. In the pregrasp stage, the robot focuses on the living object's adversarial behavior and approaches it in a reliable manner. In particular, we use inverse RL to encode the living object's adversarial behavior into a reward function. The negative value of the learned reward function is then used to train a high-quality grasping policy that can compete with the living object's adversarial behavior with the RL framework. In the in-hand stage, we use RL to train a grasp policy such that the dexterous hand can grab the living object with the minimal force. A set of dense rewards are also specifically designed to encourage the robot to grasp and hold the living object persistently. To further improve the grasp performance, we explicitly take into account the structure of the dexterous robot hand by treating the hand as a graph and adopting graph convolutional network to formulate the grasping policy. We conduct a set of experiments to demonstrate the performance of our proposed method, in which the robot can grasp living objects with the success rate of 90% and 95% in the pregrasp and in-hand stages, respectively. The contact force applied by the robotic hand to the living object is dramatically reduced in comparison with the baseline grasping policy.

Centroidal dynamics, which describes the overall linear and angular motion of a robot, is often used in locomotion generation and control of legged robots. However, the equation of centroidal dynamics contains nonlinear terms mainly caused by the robot's angular motion and needs to be linearized for deriving a linear model-predictive motion controller. This paper proposes a new linearization of the robot's centroidal dynamics. By expressing the angular motion with exponential coordinates, more linear terms are identified and retained than in the existing methods to reduce the loss from the model linearization. As a consequence, a model-predictive control (MPC) algorithm is derived and shows a good performance in tracking angular motions on a quadruped robot.

Dynamic models play a key role in robot motion generation and control and the identification of inertial parameters is a critical component for obtaining an accurate dynamic model of a robot. This paper presents a novel iterative primitive shape division method for the inertia parameter identification of floating-base robots. Describing a robot by a set of primitive shapes with uniform mass distributions, the method iteratively divides the primitive shapes into smaller ones and refines their masses, which quickly converges to yielding the true inertia parameters of the robot. This method guarantees the physical consistency of the obtained parameters, possesses a high computational efficiency for online deployment, and works without contact force measurement. Furthermore, it can be used to estimate the position and magnitude of an external load applied to the robot. Simulations and experiments on a quadruped robot have been conducted to verify the effectiveness and efficiency of the proposed method.

This paper presents a unified controller for dual-arm robot dynamic multi-object nonprehensile transportation. The controller is composed of time-optimal path parameteri-zation (TOPP) and model predictive control (MPC) and aimed at efficiently and dynamically transporting objects using the dual-arm robot under physical constraints while avoiding the slippage of the objects. A force tracking controller without using the force sensor is also proposed to achieve accurate contact force control between the arms and objects. Experiments on the real robot show the effectiveness of the proposed TOPP-MPC-based controller.

Grasping of objects using multi-fingered robotic hands often fails due to small uncertainties in the hand motion control and the object's pose estimation. To tackle this problem, we propose a grasping adjustment strategy based on tactile seroving. Our technique employs feedback from a sensorized multi-fingered robotic hand to collaboratively servo the fingers and palm to achieve the desired grasp. We demonstrate the performance of our method through simulation and physical experiments by having a robot grasp different objects under conditions of variable uncertainty. The results show that our approach achieved a higher success rate and tolerated greater uncertainty than an open-looped grasp.

The performance of a camera network monitoring a set of targets depends crucially on the configuration of the cameras. In this paper, we investigate the reconfiguration strategy for the parameterized camera network model, with which the sensing qualities of the multiple targets can be optimized globally and simultaneously. We first propose to use the number of pixels occupied by a unit-length object in image as a metric of the sensing quality of the object, which is determined by the parameters of the camera, such as intrinsic, extrinsic, and distortional coefficients. Then, we form a single quantity that measures the sensing quality of the targets by the camera network. This quantity further serves as the objective function of our optimization problem to obtain the optimal camera configuration. We verify the effectiveness of our approach through extensive simulations and experiments, and the results reveal its improved performance on the AprilTag detection tasks. Codes and related utilities for this work are open-sourced and available at https://github.com/sszxc/MultiCam-Simulation.

The wheel-bipedal robot has the advantages of both wheeled robots and legged robots, but as a cost, it is more challenging to perform flexible movements in various surroundings while keeping it balanced. The inaccurate dynamics of the robot makes the balance problem even more intractable. To solve this problem, the robot Ollie is used as a testbed. The whole-body control (WBC) framework is adopted to enhance the dexterity of the robot with multiple degrees of freedom in the task space. Moreover, a learning-based adaptive technique is applied to assist the WBC such that the balance controller can be designed in the absence of the accurate dynamics. Physical experiments demonstrate that the robot can manage various actions, with the help of the combination of the WBC and the learning-based adaptive technique.

Intercepting an object in flight through nonpre-hensile manipulation is a challenging problem, which is aimed at catching and stopping a flying object using little contacts without completely restraining its relative motion to the robot. This paper presents a two-stage optimal trajectory generation method to tackle this problem. At the pre-catching stage, optimal position and attitude trajectories of the robot's end-effector to approach the object are generated by a variational method. At the post-catching stage, the end-effector's trajectories are generated to optimally eliminate the translational and rotational motion of the object and a convex-MPC algorithm combined with admittance control is used to realize the trajectory tracking. A series of simulations and experiments have been conducted to verify the effectiveness of the proposed method.

To accurately pour drinks into various containers is an essential skill for service robots. However, drink pouring is a dynamic process and difficult to model. Traditional deep imitation learning techniques for implementing autonomous robotic pouring have an inherent black-box effect and require a large amount of demonstration data for model training. To address these issues, an Explainable Hierarchical Imitation Learning (EHIL) method is proposed in this paper such that a robot can learn high-level general knowledge and execute low-level actions across multiple drink pouring scenarios. Moreover, with the EHIL method, a logical graph can be constructed for task execution, through which the decision-making process for action generation can be made explainable to users and the causes of failure can be traced out. Based on the logical graph, the framework is manipulable to achieve different targets while the adaptability to unseen scenarios can be achieved in an explainable manner. A series of experiments have been conducted to verify the effectiveness of the proposed method. Results indicate that EHIL outperforms the traditional behavior cloning method in terms of success rate, adaptability, manipulability, and explainability. Note to Practitioners—Pouring liquids is a common activity in people’s daily lives and all wet-lab industries. Drink pouring dynamic control is difficult to model, while the accurate perception of flow is challenging. To enable the robot to learn under unknown dynamics via observing the human demonstration, deep imitation learning can be used. To address the limitations of traditional deep neural networks, an Explainable Hierarchical Imitation Learning (EHIL) method is proposed in this paper. The proposed method enables the robot to learn a sequence of reasonable pouring phases for performing the task rather than simply execute the task via traditional behavior cloning. In this way, explainability and safety can be ensured. Manipulability can be achieved by reconstructing the logical graph. The target of this research is to obtain pouring dynamics via the learning method and realize the precise and quick pouring of drink from the source containers to various targeted containers with reliable performance, adaptability, manipulability, and explainability.

Redundant manipulators play a critical role in industry and academia, which can be controlled from the kinematic or dynamic perspective. The motion-force control of redundant manipulators is a core problem in robot control, especially for the task requiring keeping contact with objectives, such as cutting, polishing, deburring, etc. However, when a manipulator’s model structure is unknown, it is challenging to take motion-force control of redundant manipulators. This article proposes a data-driven-based motion-force control scheme, which solves the motion-force control problem from the kinematic perspective. The scheme can take effect and estimate the structure information, i.e., the model parameters involved in the forward kinematics when the structure of the manipulator is incomplete or unknown. A recurrent neural network is devised to find the solution to the scheme. Besides, the theoretical analysis is presented to prove the correctness of the scheme. Simulations and physical experiments running on seven degrees of freedom redundant manipulators illustrate the superb performance and practicability of the scheme intuitively. The key contribution of this article is that, for the first time, a motion-force control scheme aided with data-driven technology is proposed from a kinematic perspective for the redundant manipulators.

Pose estimation problems are fundamental in robotics. Most of these problems are challenging due to the nonconvex nature. This also sets up an obstacle for uncertainty description that is essential for pose integration and quality control. In this article, we show that a large class of related problems can be categorized as the quadratic pose estimation problems (QPEPs) and we propose a general quaternion-based mathematical model to unify these problems. To solve the nonconvex QPEPs, a Gröbner-basis method is investigated to derive their globally optimal and robust solutions. Furthermore, we develop the rules for characterizing the solvability and observability of these solutions. In addition, the uncertainty description, i.e., covariance matrix, as an important piece of information in robotic state estimation frameworks, is analyzed in detail. Theoretical results show that the covariance can be estimated via online optimization, in an efficient and unbiased manner. In this way, both the solution and covariance are guaranteed to be globally optimal. Through simulations and experiments, we show that the proposed QPEP-based solver is not only accurate, robust, and efficient but outperforms the representatives for covariance estimation. The designed algorithms are also assembled as a C++/MATLAB/Octave/ROS library, while these developed interfaces are built for main stream platforms and simultaneous localization and mapping schemes.

This paper presents a balance control technique for a novel wheel-legged robot. We first derive a dynamic model of the robot and then apply a linear feedback controller based on output regulation and linear quadratic regulator (LQR) methods to maintain the standing of the robot on the ground without moving backward and forward mightily. To take into account nonlinearities of the model and obtain a large domain of stability, a nonlinear controller based on the interconnection and damping assignment - passivity-based control (IDA-PBC) method is exploited to control the robot in more general scenarios. Physical experiments are performed with various control tasks. Experimental results demonstrate that the proposed linear output regulator can maintain the standing of the robot, while the proposed nonlinear controller can balance the robot under an initial starting angle far away from the equilibrium point, or under a changing robot height.

Quadruped locomotion is a challenging task for learning-based algorithms. It requires tedious manual tuning and is difficult to deploy in reality due to the reality gap. In this paper, we propose a quadruped robot learning system for agile locomotion which does not require any pre-training and works well in various real-world terrains. We introduce a hierarchical learning framework that uses reinforcement learning as the high-level policy to adjust the low-level trajectory generator for better adaptability to the terrain. We compact the observation and action space of the reinforcement learning to deploy it on a host computer in reality. Besides, we design a trajectory generator guided by robot posture, which can generate adaptive foot trajectory to interact with the environment. Experimental results show that our system can be easily deployed in reality while only trained in simulation, and also has the advantages of fast convergence and good terrain adaptability. The supplementary video demonstration is available at https://vsislab.github.io/hfql/.

In this paper, a learning-based whole-body loco-motion controller is proposed, which enables quadruped robots to perform running in the style of real animals. We use a low-level controller based on multi-rigid body dynamics to calculate desired torques for each joint, while the high-level neural network policy planning the expected gait and foothold. The policy is trained with reinforcement learning, so that the robot can track a variety of trajectories according to the gait patterns recorded from real-world dogs. We transfer the walking and running gait style to quadrupeds in simulation, involving pace, trot, high-speed gallop and natural transitions. The performance is evaluated by the synchronization rate of contact state between the policy result and the recorded sequence. In the experiments, the robot runs steadily at a speed of 2 m/s and showcases a notable synchronization rate of about 80%. Without prior knowledge, the policy demonstrates a realistic foothold distribution that covers the central area of the torso, which is prevalent in running animals.

Robot hand is essential for a fully functional robot and designing a good robot hand is a sophisticated job that challenges the designer’s knowledge and experience. This paper presents a computational framework for automatic optimal robot hand design based on reinforcement learning (RL), which considers desired grasping tasks, grasp control strategies, and performance quality measures altogether. The RL-based framework intends to grow finger joints with different types and link lengths at different positions from null. Then, the reward function for such a growing action is defined in terms of quality indexes of the generated robot hand to perform desired grasping tasks under expected control strategies. To demonstrate the effectiveness of this framework, in this paper we set the desired task to simply grasping objects of three primitive shapes (i.e., box, cylinder, and sphere) with predefined hand positions and strategies to close fingers to achieve grasps for each object. The force closure condition, quantitative stability indexes, and energy consumption of grasps as well as some penalty terms are used to assemble the reward function. Through simulation and practical prototype experiments, we show that capable robot hands can be automatically generated by the proposed framework. Potential factors that affect the output of the framework and deserve further exploration are also discussed.

The teleoperation system can transmit human intention to the remote robot, so that the system combines excellent robot operation performance and human intelligence. In this article, we have established a bilateral teleoperation system with force feedback from the arm and gripper. That is, the slave robot system can provide force feedback on both the wrist and the fingers, while the master robot system can render the slave feedback force and human interaction force, and control the slave robot accordingly. In addition, this paper also proposes the framework of the robot’s four-channel bilateral teleoperation control system, which is attributed to two situations: impedance control or admittance control. Finally, single-arm/single-arm, dual-arm/dual-arm bilateral teleoperation experiments prove the effectiveness of the bilateral teleoperation system and the four-channel controller architecture proposed in this paper.

In this paper, the gain scheduling technique is applied to design a balance controller for an autonomous bicycle with an inertia wheel. Previously, two different balance controllers are needed depending on whether the bicycle is stationary or dynamic. The switch between the two different controllers may cause the instability of the autonomous bicycle. Our proposed gain scheduled controller can balance the autonomous bicycle in both stationary and dynamic cases. A physical system is built and experiments are carried out to demonstrate the effectiveness of the gain scheduled controller.

Having the merits of chemical inertness and immunity to electromagnetic interference, light weight, small size, and softness, optical waveguides have attracted much attention in making tactile sensors recently. This paper presents a new design of waveguide using two layers of cores, one of which has an uniform width and the other has an incremental width. It is deduced and verified that the contact force can be derived from the light power loss in the uniform-width core, while the contact position can be derived from the light power loss in the other core together with the estimated force. By this dual-core design, a single waveguide can simultaneously and continuously measure the contact force and position along it, which makes it very suited for integration on some thin long robotic parts, such as robotic fingers. A hardware experiment has been conducted to demonstrate its effectiveness on a two-finger gripper in an assembly task. The dual-core waveguide achieves 2 mm spatial resolution and 0.1 N sensitivity.

The support function of a surface is a fundamental concept in mathematics and a crucial operation for algorithms in robotics, such as those for collision detection and grasp planning. It is possible to calculate the support function of a convex body in a closed form. For complex continuous, especially nonconvex, surfaces, however, this calculation can be far more difficult and no general solution is available so far, which limits the applicability of those related algorithms. This article first presents a branch-and-bound (B&B) algorithm to calculate the support function of complex continuous surfaces. An upper bound of the support function over a surface domain is derived. While a surface domain is divided into subdomains, the upper bound of the support function over any subdomain is proved to be not greater than the one over the original domain. Then, as the B&B algorithm sequentially divides the surface domain by dividing its subdomain having a greater upper bound than the others, the maximum upper bound over all subdomains is monotonically decreasing and converges to the exact value of the desired support function. Furthermore, with the aid of the B&B algorithm, this article derives new algorithms for the minimum distance between complex continuous surfaces and for globally optimal grasps on objects with continuous surfaces. A number of numerical examples are provided to demonstrate the effectiveness of the proposed algorithms.

This paper solves the problem of computing the best grasp in a discrete point set based on a popular grasp quality measure, namely the largest origin-centered ball contained in the grasp wrench set. So far, the solution to this problem is very limited. Noticing that the quality measure for a grasp is equal to the minimum value of the support function of its grasp wrench set over all directions and its computation together with force closure test can be fulfilled by evaluating the support function in a sequence of directions, we can quickly determine that a new grasp is worse whenever its support function in the sequence of directions or any other specific direction is less than the quality value of the current best grasp and avoid further computation. Furthermore, we enumerate candidate grasps in the point set in an adaptive way such that grasps that are more likely to outperform the current best grasp will be checked first, which helps find the best grasp earlier and significantly reduce the number of candidate grasps to be fully examined. With the aid of the adaptive enumeration and the quick comparison of grasps, the proposed algorithm takes tens of seconds to several hours on a normal PC to compute the best grasp in tens to hundreds of points on 3-D objects and it is two to several orders of magnitude faster than the brute-force search.

This paper presents an algorithm to compute the globally optimal fixture with frictionless contacts in a discrete point set on an object. The capability of a fixture to immobilize the object is evaluated by the minimum of the largest conflict of the object with the contacts over all motion directions, which can be reduced to the radius of the largest origin-centered ball contained in the convex hull of primitive contacts wrenches. All candidate fixtures (combinations of the discrete points of a certain number) are expressed in a tree structure, in which each node contains a discrete point representing a contact location and each path from the roots to a leaf represents a fixture. A preorder walk of the tree is performed to search for the optimal fixture. Necessary conditions are derived to predict whether the subtree of a node contains a better fixture. If any necessary condition is not met, then all the fixtures contained in the subtree can be simply rejected. By this means only a small portion of the tree will be traversed and the globally optimal fixture can be found more quickly. This algorithm has been tested on various three-dimensional objects and has been found to be over ten times faster than the brute-force search.

Having human-like motions will make humanoid robots more predictable and safer for the people around them. An effective way to realize this would be to use human motions as reference. Due to different kinematic and dynamic properties between humans and humanoid robots, however, a human motion could be physically infeasible for a robot and cause the robot to fall over. Therefore, it is necessary to modify and adapt an infeasible human motion to the robot. This paper presents a method for adapting human motions to humanoid robots based on a technique called time warping, which modifies the time line of a reference motion to speed up or slow down the motion. By doing this, the velocity and acceleration profiles of the motion are changed and it is possible to turn an infeasible motion into a feasible one. The optimal time warping is obtained through a generalized motion feasibility index that quantifies the feasibility of a motion considering the friction and center-of-pressure constraints. Thanks to the generality of the index, the proposed motion adaptation method can be applied to motions on arbitrary terrains or number of links in contact with the environment. Through dynamics simulation, we demonstrate that the method facilitates the reproduction of human motions on a humanoid robot.

This paper presents algorithms to compute the generalized distance between two separated or penetrating compact convex sets, which is defined as the minimum or maximum scale factor of a given gauge set such that the scaled gauge set intersects or is contained in the Minkowski difference of the two sets. The traditional Euclidean distance is a special case where the origin-centered unit ball is used as the gauge set. While the generalized distance was proposed almost a decade ago, the only practical method for its computation has been general-purpose numerical optimization, which is computationally expensive. In contrast, our geometry-based algorithms are efficient and guarantee globally optimal solutions. Important applications of the algorithms in robotics include collision detection and grasp planning. The algorithm for computing the penetration distance also provides an accurate and efficient approach to flatness error evaluation, which is a fundamental problem in manufacturing. We demonstrate that our algorithms possess superior efficiency and accuracy in these applications.

This paper proposes a new grasp force efficiency (GFE) measure that considers not only contact point locations but also the hand configuration and mechanism. GFE evaluates the largest wrench applied to the object that the grasp can resist with unit contact forces. Traditional GFE measures depend solely on the contact point locations without considering how the unit contact forces are generated. Intuitively, however, the actuators' effort required to generate unit contact forces depends on the hand configuration and mechanism and therefore should affect the grasp efficiency. For example, generating a unit contact force with an under-actuated finger would be more difficult than with a fully actuated finger. Our new GFE measure addresses this issue and is potentially useful for hand mechanism design as well as grasp planning. We also present a novel geometry-based iterative algorithm for computing the generalized penetration distance of a point in an arbitrary convex set. The algorithm allows unified and accurate computation of the new and traditional GFE measures with various criteria without linear approximation. Other applications of the algorithm include penetration depth computation of two convex objects for physics simulation.

This paper presents an efficient algorithm for computing a distance measure between two compact convex sets Q and A, defined as the minimum scale factor such that the scaled Q is not disjoint from A. An important application of this algorithm in robotics is the computation of the minimum distance between two objects, which can be performed by taking A as the Minkowski difference of the objects and Q as a set containing the origin in its interior. In this generalized definition, the traditional Euclidean distance is a special case where Q is the unit ball. While this distance measure was proposed almost a decade ago, there has been no efficient algorithm to compute it in general cases. Our algorithm fills this void and we demonstrate its superior efficiency compared to approaches based on general-purpose optimization.

Ray shooting is a well-studied problem in computer graphics and also has applications in robotics such as collision detection and contact force optimization. Unfortunately, most ray-shooting algorithms developed for graphics applications only allow 3-dimensional (3-D) objects represented as triangle meshes, and therefore are not suited for objects with parametric surfaces or general convex sets in high-dimensional space which robotics applications often require. In contact force optimization, for example, the problem is in the 6-dimensional (6-D) wrench space and it is desirable to consider the nonlinear friction cone without approximating it by a pyramid. This paper discusses existing and novel geometry-based ray-shooting algorithms applicable to general convex sets, and compares their performances in two robotics applications: computing the distance between two convex objects and optimizing contact forces in grasping.

This paper presents an efficient algorithm to compute the minimum of the largest wrenches that a grasp can resist over all wrench directions with limited contact forces, which equals the minimum distance from the origin of the wrench space to the boundary of a grasp wrench set. This value has been used as an important grasp quality measure in optimal grasp planning for over two decades, but there has been no efficient way to compute it until now. The proposed algorithm starts with a polytope containing the origin in the grasp wrench set and iteratively grows it such that the minimum distance from the origin to the boundary of the polytope quickly converges to the aforementioned value. The superior efficiency and accuracy of this algorithm over the previous methods have been verified through theoretical and numerical comparisons.

This paper presents algorithms for optimal grasping forces. The previous work reveals that contact forces with minimal sum or maximum of normal force components can be written as positive combinations of primitive contact forces, to which the corresponding primitive contact wrenches express the required resultant wrench as their positive combination with minimum coefficients in terms of theL1orL∞metric. On this basis, we first propose an algorithm to compute such a set of primitive contact forces and the minimum contact forces. Moreover, considering the uncertainty in the actual friction coefficient, we develop another algorithm to determine the minimum required friction coefficient and the corresponding minimum contact forces within a given limit on their magnitude so that such contact forces are more reliable in practice. As our algorithms do not rely on any general optimization technique, they are very efficient and easy to implement. These algorithms can also be used in grasp quality evaluation and optimal grasp planning.

This paper presents a control framework for a biped robot to maintain balance and walk on a rolling ball. The control framework consists of two primary components: a balance controller and a footstep planner. The balance controller is responsible for the balance of the whole system and combines a state-feedback controller designed by pole assignment with an observer to estimate the system's current state. A wheeled linear inverted pendulum is used as a simplified model of the robot in the controller design. Taking the output of the balance controller, namely the ideal center of pressure of the biped robot on the ball, as the input, the footstep planner computes the foot placements for the robot to track the ideal center of pressure and avoid a fall from the rolling ball. Simulation results show that the proposed controller can enable a biped robot to stably walk on balls of different sizes and rotate a ball to desired positions at desired speeds.

We present an efficient algorithm for solving the ray-shooting problem on high dimensional sets. Our algorithm computes the intersection of the boundary of a compact convex set with a ray emanating from an interior point of the set and represents the intersection point as a convex combination of a set of affinely independent points. We use our intersection algorithm to compute two types of optimal grasping forces, where either the sum or the maximum of normal force components is minimized. In our simulation, the algorithm converges well and performs the computations in tens of milliseconds on a laptop.

We present a new method to generate biped walking patterns for biped robots on uneven terrains. Our formulation uses a universal stability criterion that checks whether the resultant of the gravity wrench and the inertia wrench of a robot lies in the convex cone of the wrenches resulting from contacts between the robot and the environment. We present an algorithm to compute the feasible acceleration of the robot's CoM (center of mass) and use that algorithm to generate biped walking patterns. Our approach is more general and applicable to uneven terrains as compared with prior methods based on the ZMP (zero-moment point) criterion. We highlight its applications on some benchmarks.

Designing a fixture layout of an object can be reduced to computing the largest simplex and the resulting simplex is classified using the radius of the largest inscribed ball centered at the origin. We present three different algorithms to compute such a simplex: a simple randomized algorithm, an interchange algorithm, and a branch-and-bound algorithm. We evaluate their complexity and also present methods to combine different algorithms to improve the performance and highlight their performance on complex 3D models consisting of thousands of triangles. Our randomized algorithm computes a feasible fixture layout in linear time and is well-suited for realtime applications. The interchange algorithm computes an optimal simplex in linear time such that no single vertex can be changed to enlarge the simplex. The branch-and-bound algorithm computes the largest simplex by using lower and upper bounds on the radius of the inscribed ball.

Based on the distance algorithm by Gilbert et al., this paper presents a numerical algorithm for computing the intersection of the boundary of a compact convex set with a ray emanating from an interior point of the set, which is known as the ray-shooting problem. Affinely independent points on the boundary of the convex set are also determined such that the intersection point can be written as their convex combination. Because of its high efficiency and other good qualities, this algorithm provides superior solutions to three fundamental problems in robotic grasping, i.e., force-closure test, contact force optimization, and grasp quality evaluation, which can be formulated as the ray-shooting problem.

This paper presents an algorithm to compute the minimum distance from a point to a convex cone inn-dimensional space. The convex cone is represented as the set of all nonnegative combinations of a given set. The algorithm generates a sequence of simplicial cones in the convex cone, such that their distances to the single point converge to the desired distance. In many cases, the generated sequence is finite, and therefore, the algorithm has finite-convergence property. Recursive formulas are derived to speed up the computation of distances between the single point and the simplicial cones. The superior efficiency and effectiveness of this algorithm are demonstrated by applications to force-closure test, system equilibrium test, and contact force distribution, which are fundamental problems in the research of multicontact robotic systems. Theoretical and numerical comparisons with previous work are provided.

This correspondence deals with the dynamic force distribution (DFD) problem, i.e., computing the contact forces to equilibrate a dynamic external wrench on the grasped object. The sum of the normal force components is minimized for enhancing safety and saving energy. By this optimality criterion, the DFD problem can be transformed into a linear programming (LP) problem. Its objective function is the inner product of the dynamic external wrench and a vector, and the constraints on the vector, given by a set of linear inequalities, define a polytope. The solution to the LP problem can always be attained at the vertex of the polytope called the solution vertex. We notice that the polytope is determined by the grasp configuration. Along with the direction change of the dynamic external wrench, only the solution vertex moves to an adjacent vertex sequentially, whereas the polytope with all its vertices remains unchanged. Therefore, the polytope and the adjacencies of each vertex can be computed in the offline phase. Then, in the online phase, simply search the adjacencies of the old solution vertex for the new one. Without lost of optimality, such a DFD algorithm runs a thousandfold faster than solving the LP problem by the simplex method in real time.

This paper addresses a shortcut in the ray-shooting based algorithm proposed by Liu. His algorithm provides an efficient force-closure test of 3D frictional grasps, which is formulated as a linear programming (LP) problem. We prove that the optimal objective value of the LP formulation indicates the force-closure property of grasps directly. Thus computing Q, d/sub 1/, and d/sub 2/ can be omitted.

This paper presents a general algorithm for computing the optimal dynamic force distribution in multifingered grasping. It consists of two phases. In the offline phase, we select a spanning set for the required dynamic resultant wrench and find a corresponding spanning set for the total contact force. Then, in the online phase, the total contact force is obtained by decomposition of the resultant wrench into the former spanning set and a coefficient vector followed by positive combination of the latter spanning set with the vector. To make the online computation as simple as possible, iterative operation is executed offline and only arithmetic operation is employed online. To improve the grasping quality, the two spanning sets are selected elaborately.