This paper presented a regrasp planning method to eliminate grasp uncertainty while considering the geometric constraints of a fixture. The method automatically finds the Stable Placement Poses (SPPs) of an object on a Triangular Corner Fixture (TCF), elevates the object from its SPPs to dropping poses and finds the Deterministic Dropping Poses (DDPs), builds regrasp graphs by using the SPP-DDP pairs and their associated grasp configurations, and searches the graph to find regrasp motion sequences for precise assembly. Since the SPPs and their associated regrasps are constrained by the TCF’s geometry and have high precision, the final object poses regrasped via it has low uncertainty and can be directly used for assembly by position control. In the experimental section, we study the performance of analytical and learning-based methods for estimating the DDPs of different objects and quantitatively examine the proposed method’s ability to suppress uncertainty using assembly tasks like peg-in-hole insertion and sheathing tubes, aligning holes, mounting bearing housings, etc. The results demonstrate the method’s robustness and efficacy. Note to Practitioners—In production lines, robots interact with peripheral devices to improve efficiency and reduce uncertainty. In this work, we focus on a particular peripheral device – a Triangular Corner Fixture (TCF) made by three inclined and mutually perpendicular plates. We study using the TCF to improve manipulation precision. The inclined plates of the TCF form a gravity bucket that holds dropped objects in stable states under gravity. In a real scenario, a robot picks up an object and releases it above the TCF. The released object will reach a stable state on the TCF. Then, the robot regrasps and moves the stabilized object to the target pose with reduced uncertainty. Using the method proposed in this paper, a robot can automatically finish the above procedure by finding all the object’s stable states in the TCF, planning grasp configurations, invalidating infeasible states and grasps, building regrasp graphs and searching the graph to find a regrasp motion sequence that moves the object to a goal pose with high precision for assembly. In industrial applications, the proposed method has the potential to improve the flexibility of robotic systems for high-precision tasks. In the research fields, it may promote the research on sensorless manipulation and extrinsic manipulation, and push forward the studies in robotic regrasp.

A vacuum lifter is widely used to hold and pick up large, heavy, and flat objects. Conventionally, when using a vacuum lifter, a human worker watches the state of a running vacuum lifter and adjusts the object’s pose to maintain balance. In this work, we propose using a dual-arm robot to replace the human workers and develop planning and control methods for a dual-arm robot to raise a heavy plate with the help of a vacuum lifter. The methods help the robot determine its actions by considering the vacuum lifer’s suction position and suction force limits. The essence of the methods is two-fold. First, we build a Manipulation State Graph (MSG) to store the weighted logical relations of various plate contact states and robot/vacuum lifter configurations, and search the graph to plan efficient and low-cost robot manipulation sequences. Second, we develop a velocity-based impedance controller to coordinate the robot and the vacuum lifter when lifting an object. With its help, a robot can follow the vacuum lifter’s motion and realize compliant robot-vacuum lifter collaboration. Real-world experiments are carried out to investigate the proposed planning and control methods. The results show that a robot can effectively and flexibly work together with a vacuum lifter to manipulate large and heavy plate-like objects with the methods’ support. Note to Practitioners—This paper is motivated by the vacuum lifters used for transporting heavy plates in a factory that produces building materials. In the factory, a human worker attaches the suction cup of a vacuum lifter to a plate and controls the vacuum lifter to pull the plate up. Meanwhile, another human worker moves and lifts the plate to a goal pose, following the vacuum lifter while maintaining balance. The job is dangerous as the plate is heavy, and the vacuum lifter is not always strong enough to hold the plate firmly. Inspired by the usage and safety problem, we in this paper develop a planning and control method for a dual-arm robot to replace humans. The robot coordinates its motion to work with the vacuum lifter and performs lifting tasks. The vacuum lifter could remain operated by a human worker or be actuated by signals from the dual-arm robot or other third-party machines. The work is complementary to our previous study that developed planners for robots to use pulley blocks. They together provide extensive knowledge for using low-payload collaboratively robots to manipulate heavy plates.

When humans want to understand an object’s 3D shape, they watch the object from different viewpoints. Changing the viewpoint is either performed actively, i.e., moving eye sights or the human head, or passively, i.e., holding and reposing the object. Inspired by the humans’ passive policy, we propose a method to plan the motion for a dual-arm robot to hold and repose an object, capture multiple views using a stationary depth sensor mounted on the robot head, and obtain the object’s 3D shape from the multiple views. Primarily, we develop algorithms to determine the Next Best Configuration (NBC) for observation and Next Best Regrasp/Grasp (NBR/G) poses while considering elements like the confidence of captured partial point clouds, robotic manipulability, robotic motion distances, and sensing ranges. We study the necessity and influence of these elements on the time costs and surface coverage quality in the experimental section using several representative objects. The results show that the elements play essential roles in supporting specific actions or suppressing certain costs. They help to secure efficient robot motion and satisfactory 3D shape recovery quality.Note to Practitioners—This paper is motivated by the difficulties in using commercial 3D scanners. A commercial 3D scanner set usually comprises a scanning sensor, a rotating table, and editing software. To scan the 3D shape of an object, a human needs to place the object on the rotating table with different poses, let the scanner obtain several partial point clouds, and use the editing software to merge them into a final model. The human must carefully design the different poses by considering both the object’s self obstructions and stable placements, which is tiring and difficult to be applied to large-scale tasks like building 3D shape databases containing many objects. On the other hand, although several robotic solutions exist for automatic scanning, they either use an eye-in-hand scanner to scan a stationary object or an arm to hold and move an object for scanning. In the former case, the bottom or downward faces of the object cannot be covered. In the latter case, the surface blocked by the fingers of the holding hand will be lost. The method proposed by this paper plans dual-arm robot motion to grasp and move objects for scanning. It automatically determines pick-up, rotation, and handover to maximize scanning coverage. Compared with commercial scanners and existing robotic solutions, the method performs automatic scanning with high coverage and is more advantageous for scanning many objects without human intervention.

The work is inspired by the assembly of Soma block puzzles. Soma block puzzles usually include unstable intermediate states that require additional support to maintain stability temporarily. In the puzzles’ solution manual, we can observe that designers consider the characteristics that humans have two hands and can avoid an unstable intermediate state by using one hand to support the finished component and using the other hand to assemble an upcoming workpiece. Motivated by human behavior, this paper develops a planner that automatically finds an optimal assembly sequence for a dual-arm robot to build a woodblock structure while considering various constraints and supporting grasps from a second hand. It uses the mesh model of wood blocks and the final assembly state to generate possible assembly sequences and evaluate the optimal assembly sequence by considering the stability, graspability, assemblability, and the need for a second hand. Especially, the need for a second hand is resolved when supports from worktables and other workpieces are not enough to produce a stable assembly. A second hand can hold and support the unstable components so that the robot can further assemble new workpieces until the structure state becomes stable again. The output of the planner includes the optimal assembly orders, candidate grasps, assembly directions, and the supporting grasps (if needed). The output can help guide a dual-arm robot to perform motion planning and thus generate assembly motion. Experiments using various blocks and structures show the effectiveness of the proposed planner. Note to Practitioners—The presented planner can generate an optimal assembly order for a large variety of structures like decoration accessories, furniture, home interiors, frames, etc., in the practices. They can also be used for scenarios that need stacking or piling up multiple objects. The generated optimal assembly order is more friendly to dual-arm robot systems than previous assembly planners that ignored the merits of robotic collaboration. Also, the proposed assembly planner generates the necessary information for the motion planner, such as grasp poses and optimal assembly directions. A motion planner can directly use the generated results to plan robotic assembly motion.. The proposed assembly planner is expected to significantly reduce human effort and increase the efficiency of robotic assembly lines.

This paper proposes a combined planning and optimization method that enables a dual-arm robot to lift up and flip heavy plates using crane pulley blocks. The problem is motivated by the low payload of modern collaborative robots. Instead of directly manipulating heavy plates that collaborative robots cannot afford, the paper develops a planner for collaborative robots to operate crane pulley blocks. The planner assumes a target plate is pre-attached to the crane hook. It optimizes dual-arm action sequences and plans the robot’s dual-arm motion that pulls the rope of the crane pulley blocks to lift up the plate. The crane pulley blocks reduce the payload that each robotic arm needs to bear. When the plate is lifted up to a satisfying pose, the planner plans a sliding-pushing motion for one of the robot arms to tumble over the plate while considering force and moment constraints. The article presents the technical details of the planner and several experiments and analysis carried out using a dual-arm robot made by two Universal Robots UR3 arms. The influence of various parameters and optimization goals are investigated and compared in depth. The results show that the proposed planner is flexible and efficient. This paper is motivated by a cleaning process in a factory that produces sewage press machines. The pressboard of sewage press machines could be as heavy as 1000 kg. Human workers need to flip and clean both sides of the board before installing them to the main axis of a sewage machine. Their solution is using a gantry crane. They attach the board to the crane hook using bearing belts, activate the crane to lift up the board. When the board is raised to a satisfying pose, the workers turn the board over by pushing it. Motivated by human workers’ actions, we developed the planner presented in this paper. We assumed crane pulley blocks in the experiments and analysis, but in practice, they may be replaced with electronic ones to improve effort and efficiency. Using the electronic ones will be a sub-problem since there is no need for pulling ropes. The proposed method is expected to help a company’s technicians better judge if they need a heavy payload manipulator or keep their current crane equipment while employing several intelligent collaborative robots to operate them. As a result, it may help to accelerate the upgrade of manufacturing sites while reducing reforming budgets. Note to Practitioners—This paper is motivated by a cleaning process in a factory that produces sewage press machines. The pressboard of sewage press machines could be as heavy as 1000 kg. Human workers need to flip and clean both sides of the board before installing them to the main axis of a sewage machine. Their solution is using a gantry crane. They attach the board to the crane hook using bearing belts, activate the crane to lift up the board. When the board is raised to a satisfying pose, the workers turn the board over by pushing it. Motivated by human workers’ actions, we developed the planner presented in this paper. We assumed crane pulley blocks in the experiments and analysis, but in practice, they may be replaced with electronic ones to improve effort and efficiency. Using the electronic ones will be a sub-problem since there is no need for pulling ropes. The proposed method is expected to help a company’s technicians better judge if they need a heavy payload manipulator or keep their current crane equipment while employing several intelligent collaborative robots to operate them. As a result, it may help to accelerate the upgrade of manufacturing sites while reducing reforming budgets.

This article develops a mechanical screwing tool and its manipulation policies for two-finger parallel robotic grippers. The tool is based on a combined scissor-like element (SLE) and double-ratchet mechanism that converts the gripping motion of two-finger parallel grippers into a continuous rotation to realize tasks like fastening screws. The tool is entirely mechanical. There is no need for external cable connections. The manuscript includes two parts. For one thing, it shows the details of the tool design, optimizes the tool’s dimensions and effective stroke lengths, and studies the contacts and forces to achieve stable grasping and screwing. For another, it presents the related manipulation and control policies, including recognizing the tool, changing tool poses, and completing screw fastening tasks. The designed tool, together with the related manipulation and control policies, are analyzed and verified in several real-world applications. The results show that the tool has satisfying mechanical properties. Robots with parallel grippers can robustly and flexibly use the tool to fasten screws. The tool can also be used collaboratively with other tools to finish difficult tasks. In the future, similar tools are expected to replace special-purpose end-effectors or tool changers for more flexible robot integration.

This article develops a flexible and robust robotic system for autonomously drawing on 3-D surfaces. The system takes 2-D drawing strokes and a 3-D target surface (mesh or point clouds) as input. It maps the 2-D strokes onto the 3-D surface and generates a robot motion to draw the mapped strokes using visual recognition, grasp pose reasoning, and motion planning. The system is flexible compared to conventional robotic drawing systems as we do not fix drawing tools to the end of a robot arm. Instead, a robot recognizes and picks up pens online and holds the pens to draw 3-D strokes. Meanwhile, the system has high robustness thanks to the following crafts: First, a high-quality mapping method is developed to minimize deformation in the strokes. Second, visual detection is used to reestimate the drawing tool’s pose before executing each drawing motion. Third, force control is employed to compensate for noisy visual detection and calibration and ensure a firm touch between the pen tip and the surface. Fourth, error detection and recovery are implemented to deal with slippage and other anomalies. The planning and executions are performed in a closed-loop manner until the strokes are successfully drawn. We evaluate the system and analyze the necessity of the various crafts using different real-world tasks. The results show that the proposed system is flexible and robust to generate robotic motion that picks up the pens and successfully draws 3-D strokes on given surfaces.

Proximity perception is a technology that has the potential to play an essential role in the future of robotics. It can fulfill the promise of safe, robust, and autonomous systems in industry and everyday life, alongside humans, as well as in remote locations in space and underwater. In this survey article, we cover the developments of this field from the early days up to the present, with a focus on human-centered robotics. In this domain, proximity sensors are typically deployed in two scenarios: first, on the exterior of manipulator arms to support safety and interaction functionality, and second, on the inside of grippers or hands to support grasping and exploration. Therefore, based on this observation, in the beginning of this article, we propose a categorization to organize the use cases of proximity sensors in human-centered robotics. Then, we devote effort to present the sensing technologies and different measuring principles that have been developed over the years, also providing a summary in form of a table. Following, we review the literature regarding the applications that have been proposed. Finally, we give an overview of the most important trends that will shape the future of this domain.

In this study, a visual shock absorber capable of adapting to free-fall objects with various weights and speeds is designed and realized. The key element is a magnetic gear to passively absorb shock in the moment of contact, which is difficult for traditional feedback control to deal with. The magnetic gear allows the seamless transfer of control from the non-contact state to the contact state. 1000 Hz high-speed visual object tracking is used for preparation with position and velocity control in the object non-contact state. In the moment of object contact, the high backdrivability of the magnetic gear response by hardware provides high responsiveness to external force. After the impact, the plastic deformation control of a parallel-expressed Maxwell model handles the contact state.

Synergy provides a practical approach for expressing various postures of a multi-fingered hand. However, a conventional synergy defined for reproducing grasping postures cannot perform in-hand manipulation, e.g., tasks that involve simultaneously grasping and manipulating an object. Locking the position of particular fingers of a multi-fingered hand is essential for in-hand manipulation tasks either to hold an object or to fix unnecessary fingers. When using conventional synergy based control to manipulate an object, which requires locking some fingers, the coordination of joints is heavily restricted, decreasing the dexterity of the hand. We propose a functionally divided manipulation synergy (FDMS) method, which provides a synergy-based control to achieves both dimensionality reduction and in-hand manipulation. In FDMS, first, we define the function of each finger of the hand as either "manipulation" or "fixed." Then, we apply synergy control only to the fingers having the manipulation function, so that dexterous manipulations can be realized with a few control inputs. Furthermore, we propose the Synergy Switching Framework as a method for applying a finely defined FDMS to sequential task changes. The effectiveness of our method is experimentally verified.

In this study, Magripper, a highly backdrivable gripper, is developed to achieve high-speed hitting grasping executed seamlessly from reaching. The gripper is designed to achieve both high speed and environmental adaptability. The key element is backdrivability in terms of both hardware and control. In Magripper, a magnetic gear is introduced to passively absorb shock in the moment of contact as a means of hardware backdrivability, and backdrive control is implemented based on the Zener model. After developing a hitting grasping framework, high-speed hitting grasping tasks with a wood block, a wood cylinder, and a plastic coin are conducted using only servo control without sensors, such as cameras and tactile sensors. In particular, coin grasping with high-speed movement is very difficult because collisions with environmental objects such as the floor and desk, are likely, which may break a robot.

This study proposes a dynamic observable contact (DOC) hand as a new multifingered hand to ensure high- speed insertion in an assembly process with a small clearance between objects. To achieve insertion with a small clearance at high speed, a robot hand must realize both impact reduction and position-error compensation when the two objects contact each other. The DOC hand, with its features of 6-degrees-of- freedom dynamic passivity and object-pose observability, can realize both impact reduction and position-error compensation. To evaluate the effectiveness of the DOC hand, we construct a robot system using the DOC hand. We evaluate the performance of the system in the task of ring insertion with a small clearance (0-36um). The results indicate that the robot system performs with a higher speed than a human. In fact, the average cycle time is 2.42 s for the robot, whereas it is 2.58 s for a human. The DOC hand has opened up the possibility for achieving high-speed precision assembly using robots.

The robustness of the positioning and posturing of robot hands relative to target object and support surface is an important issue for autonomous grasping. For example, to perform a grasping action such as picking up thin objects from a table top, the position and posture of the hand must be controlled to keep adequate relative posture and distance to the support surface besides those between the hand and the target object. Because slight errors in the posture and position are enough to cause grasping failure, the positioning and posturing of the hand must be precise enough, specially when the hand is close to the target object and support surface. To improve the robustness of robotic grasping, in this paper we present a method by grasping control based on the relative posture and position between hand and support surface besides those between hand and target object, using proximity sensors. Proximity sensors are newly installed on fingernails besides on the fingertips. As the fingernail sensor, an integration of Time-of-Flight (TOF) sensor and photo-reflector is designed to realize long range detection, as well as with precise and high-speed detection regardless of the reflectance of support surfaces when approaching the support surface. By the sensors, the hand can approach the object and support surface coarsely first, and then can be controlled fast and precisely to realize adequate grasping motion along the support surface but without contact with the support face. The method has been implemented to a manipulator system, and successful grasping experiments have demonstrated the effectiveness of the proposed method.

In this study, we propose integrated control of a robotic hand and arm using only proximity sensing from the fingertips. An integrated control scheme for the fingers and for the arm enables quick control of the position and posture of the arm by placing the fingertips adjacent to the surface of an object to be grasped. The arm control scheme enables adjustments based on errors in hand position and posture that would be impossible to achieve by finger motions alone, thus allowing the fingers to grasp an object in a laterally symmetric grasp. This can prevent grasp failures such as a finger pushing the object out of the hand or knocking the object over. Proposed control of the arm and hand allowed correction of position errors on the order of several centimeters. For example, an object on a workbench that is in an uncertain positional relation with the robot, with an inexpensive optical sensor such as a Kinect, which only provides coarse image data, would be sufficient for grasping an object.

This study presents a robot-hand-arm system with high robustness and responsiveness by using a “Net-Structure Proximity Sensor.” The sensor, which we have developed and specially designed for a robot hand, directly detects an object being to be grasped and outputs analog voltage signals according to the position/posture error between the robot hand and the object. It has been confirmed that the robot hand is able to quickly adjust to and grasp an unknown object by applying a feed-back control method based on the sensor signals. This paper focuses on the integration of the proximity-based feedback control to a commonly-used vision-based control. These sensors work in complementary manner: a vision sensor is available for planning an approaching path of a robot hand by detecting large area, and a Net-Structure Proximity Sensor enables the robot hand to adjust the approaching error before grasping and to improve the certainty of the grasping. Two objective velocities are derived independently by the sensors. By adding the velocities with considering the reliability of the sensor information, the robot hand becomes to be able to perform approaching and adjustment to the target object simultaneously. Experimental results showed that the robot hand grasped a moving object with high success rate even in conditions where it was difficult to predict the trajectory of the object accurately.

Quick motion and soft touch control are important for autonomous grasping. To perform a grasping action, a hand must adjust its fingertips to match the object shape. It is also necessary to reduce the fingertip velocity on contact with the object. We propose a method of fingertip velocity control for fast approach and slow contact using proximity sensors installed on fingertips. The proposed control reduces the fingertip velocity at contact, decreasing the change of impulse force and thus making it appropriate for grasping soft or fragile objects. The proposed control uses time-to-contact (TTC), which is converted from sensor output. It is known that many animals, including humans, use TTC for collision avoidance. TTC represents the remaining time until collision considering the rate change of a control variable. We carried out grasping tests on various common objects using TTC. Experimental results show that the proposed control realizes fingertip alignment perpendicular to unknown shapes and surface objects and lowers the velocity at contact.

In this paper, we demonstrate a preliminary motion before grasping by a robot hand, for adjusting the object-fingertip distance and 2-axis postures simultaneously, using a Resistor Network Structure Proximity sensor (RNSP sensor). Through this motion (called “pre-shaping”) and the grasping of an object, the surface of each fingertip is brought into contact with the object surface so that in the next stage grasping can be undertaken. In the next stage, a force can be applied from the fingertips onto the object surface directly. The pre-shaping enhances the reliability of the feedback control for the after-contact tactile sensors. To realize the pre-shaping, we use fingertips equipped with RNSP sensors, which can detect the distance between the fingertip and the object, to determine the relative position between fingertips and an object. The RNSP sensor has a fast response (<;1 [ms]) and simple connectivity (only 6 wires), and can be mounted easily. Additionally, a characteristics of the RNSP sensor output can be designed by the arrangement of the sensor elements. To perform the pre-shaping by simple sensor feedback control based on the configuration between the fingertip and object, we designed the RNSP sensor so that it had the appropriate characteristics for the pre-shaping.