This paper presents an innovative continuous autofocus (C-AF) approach based on high-speed vision. It consistently provides focused images with stable and sufficiently high frame rates, aiming to improve the ability to track small, fast-moving objects in a highly magnified scene. To achieve this, we propose the concept of a dynamic-range focal sweep enabled by a high-speed camera and a focus-tunable liquid lens with high adjustment capability. The focal sweep consistently covers a small range around the object’s focus position, guided by previous depth results obtained through the depth-from-focus (DFF) technique. We conducted verification experiments to thoroughly analyze the capability of the proposed C-AF approach. By integrating a 2-axis Galvano mirror, we built a high-speed C-AF active vision system with rapid focus and pan-tilt adjustments. The comprehensive experiment results highlight the advanced capabilities of our seamless C-AF in tracking magnified objects.

This study proposes a dual-camera system for indoor high magnification surveillance which is capable of achieving always-in-focus and non-delay gaze control based on high-speed vision. The users are enabled to move the mouse freely on the wide-view screen while observing its in-focal zoom-in monitoring video in real-time. The proposed system consists of a wide-angle camera for wide-view and a Galvano mirror-enabled ultra-fast pan-tilt-zoom (PTZ) camera for zoom-in view. To achieve always-in-focus, a high-speed focus scanning system is proposed that is comprised of a high-speed camera, a parfocal zoom lens, and a gear mechanism. Through continuously reciprocating rotational motion of the focusing ring driven by the servo motor, the high-speed camera captures sets of images with varying focal distances. Moreover, we proposed a most-in-focus (MIF) frame extraction algorithm to select the sharpest images as output. The experimental results are obtained to confirm the effectiveness of our system.

In recent years, research and development have been carried out on manipulators equipped with multi-fingered robot hands as end-effectors to perform delicate and dexterous tasks. In high-speed movement of such multi-fingered hand-arms, the weight of the multi-fingered hands slows down the response of the arms. To solve this problem, we propose a control method in which a high-response hand is used to compensate for the delay in a low-response arm in the task of throwing a ball. In particular, the accuracy of the pitching motion is improved by predicting tracking errors based on the arm dynamics and using nonlinear model predictive control to compensate for arm tracking errors by using the hand motion. Experimental results of ball throwing are shown.

In recent years, there have been many studies on sports robots that can play against humans, including studies on the strategies that sports robots use by taking into account the physical conditions of their opponents. However, there have been few studies on strategies that take into account psychological conditions of the opponents, such as carelessness and habituation. This paper proposes a motion strategy for an air-hockey robot that intentionally coaxes the opponent to learn the robot’s attack sequence and then uses a different attack sequence to catch the opponent off guard. We explicitly model the change in reaction time during serial learning and represent the opponent’s reaction as an evaluation function. By applying this evaluation function to a game tree and selecting the optimal motion, the robot could catch the opponent by surprise. Actual experiments with several subjects confirmed the effectiveness of the proposed method.

In this paper, we propose the "multi-vision hand", in which a number of small high-speed cameras are mounted on the robot hand of a common 7 degrees-of-freedom robot. Also, we propose visual-servoing control by using a multi-vision system that combines the multi-vision hand and external fixed high-speed cameras. The target task was ball catching motion, which requires high-speed operation. In the proposed catching control, the catch position of the ball, which is estimated by the external fixed high-speed cameras, is corrected by the multivision hand in real-time. In experiments, the catch operation was successfully implemented by correcting the catch position, thus confirming the effectiveness of the multi-vision hand system.

In this paper, we propose a method for improving the maneuverability of master-slave systems. We aim at reproducing human skillfulness and dynamic performance in master-slave robots by using assist control for human operators. In this paper, we tackle a reaching task performed by a master-slave robot and propose an operation assist algorithm based on visual feedback control. The algorithm consists of visual object recognition for a slave robot, prediction of the operator's motion by a particle filter, estimation of the operator's intention, and operation assistance of the reaching motion. We verified the validity of the proposed system in experiments.

Paper folding is one of the most difficult tasks for multi-fingered robot hands because paper is deformable and its stiffness distribution is nonuniform. In this study, we aimed to achieve dexterous paper folding by extracting some dynamic motion primitives. Each primitive contains visual or force information, a physical model of a sheet of paper is used for analyzing its deformation, and a machine learning method is used for predicting its future state. We also propose a strategy for achieving valley folds in a sheet of paper twice in a row. One problem faced in folding paper is that, in the second fold, the crease line of the first fold disturbs the folding accuracy. We propose some new manipulation techniques to solve this problem. Finally, we show some demonstrations of paper folding achieved with a high success rate.

In this study, we design a novel air-hockey robot system that switches strategies according to the playing styles of its opponent. The system consists of a four-axis robot arm and two high-speed vision sensors. We control the robot using visual information received at a rate of 500Hz. The control system consists of three layers: motion control, short-term strategy, and long-term strategy. In the motion control layer, the robot is controlled by visual information of the puck. In the short-term strategy layer, motion of the robot is changed according to the motion characteristics of the puck. In the long-term strategy layer, the motion of the robot is changed according to the playing style of the opponent. By integrating the three control layers, the robot exhibits human-like reactions, which increase the appeal of the game. Experimental results verify the effectiveness of our proposed method.

In this paper, we propose an entirely new manipulation strategy for dynamic manipulation of a ribbon with a high-speed manipulator. The manipulation strategy involves manipulating the object at a constant, high speed. Then, we can assume that the dynamic behavior of the ribbon can be obtained by performing algebraic calculations of the robot motion using the proposed strategy. Based on this assumption, we derive a model of the ribbon and suggest a motion planning method using the proposed model. Finally, we show experimental results of shape control of a ribbon based on the proposed method.

Humans can perform fast and skillful manipulations using various parts of the body by effectively utilizing the dynamics of the targets. Visual sensation is the most important human sense used for such manipulations. Juggling is one such example involving skillful and dynamic manipulations, and visual information is essential for it to be successful. Previously, there have been several studies about robotic juggling. However, none of these studies have considered cases in which a human-like multifingered hand-arm is used for the robotic juggling. The purpose of this study is to achieve two-ball juggling using our robotic hand-arm, which has three general purpose fingers, and stereo vision. Image processing is executed at 500 fps using a high-speed vision system and graphics processing unit (GPU). The trajectory of the robotic hand-arm is generated based on the ball's estimated dropping position and moment, and the robot catches the ball. The juggling motion is achieved by repeating this cycle. Therefore, the results show that the robot successfully juggles two balls using our hand-arm system.

In this paper, we propose an entirely new manipulation strategy for dynamic manipulation of a flexible rope with a high-speed robot arm. The manipulation strategy involves manipulating the object at a constant, high speed. Then, we can assume that the dynamic behavior of the flexible rope can be obtained by performing algebraic calculations of the robot motion using the proposed strategy. Based on this assumption, we derive a model of the flexible rope and suggest a motion planning method using the proposed model. Finally, we show experimental results of rope deformation control based on the proposed method.

This video introduces an ultra high-speed robot as a milestone in the history of intelligent manipulation systems. To develop the ultra high-speed robot under the concept of dynamics matching, we began with the development of a 1 kHz vision system. Next, we developed a sensory-motor fusion system by introducing the 1 kHz vision system. In addition, we have developed a new high-torque mini actuator and a high-speed multi-fingered hand with these incorporated. Integration of these components brings real-time dexterous manipulations unlike commonly-used control based on prediction or learning.

In this research, we successfully demonstrated dexterous manipulation of sheet-like elastic objects, namely, playing cards, using a high-speed robot system. In particular, our goal was to achieve card flicking by vibrating a fingertip of a robot hand at high speed and card catching by using high-speed visual feedback based on a high-speed vision system. We discuss card grasping in the initial state based on the geometry conditions of the card and the kinematics of the robot hand, and we propose a strategy for card flicking. We also suggest a card catching method based on information obtained by the high-speed vision system. In addition, we obtain the card flicking conditions by analyzing the slip between the card and the fingertip of the robot hand. Finally, we show experimental results of card flicking and card catching.

The purpose of the work described in this paper is to achieve dynamic manipulation of a sheet-like flexible object. As one example, we examine dynamic folding of a cloth with two high-speed multifingered hands mounted on two sliders. First, dynamic folding by a human subject is analyzed in order to extract the necessary motions for realizing this task. Second, a model of a sheet-like flexible object is proposed by extending a linear flexible object model (algebraic equation) that takes advantage of high-speed robot motion. Third, motion planning of the robot system is performed by using the proposed model, and the simulation results are shown. Finally, an experiment was conducted with the robot motion obtained by the simulation.

In this paper, we propose an entirely new strategy for dexterous manipulation of a linear flexible object with a high-speed robot arm. The strategy involves manipulating the object at high speed. By moving the robot at high speed, we can assume that the dynamic behavior of the linear flexible object can be obtained by performing algebraic calculations of the robot motion. Based on this assumption, we derive a model of the linear flexible object and confirm the validity of the proposed model. Finally, we perform simulation of dynamic knotting based on the proposed model. Results of an experiment demonstrating dynamic knotting with a high-speed robot arm are shown.

This video introduces the demonstration of skillful manipulation using a high-speed robot system. The system consists of visual and tactile sensors at a rate of 1 kHz and a high-speed hand-arm manipulator. The high-speed sensory-motor fusion improves not just the speed of existing robot manipulations, but robotic skills by introducing the features peculiar to high-speed motion. Based on such a concept, new variations of skillful manipulation were achieved.

To achieve a human like grasping with a multi- fingered robot hand, the grasping force should be controlled without using information from the grasped object such as its weight and friction coefficient. In this study, we propose a method for detecting the slip of a grasped object using the force output of Center of Pressure (CoP) tactile sensors. CoP sensors can measure the center position of a distributed load and the total load applied on the surface of the sensor, within 1 ms. These sensors are arranged on the fingers of the robot hand, and their effectiveness as slip detecting sensors is confirmed in tests of slip detection during grasping. Finally, we propose a method for controlling grasping force to resist tangential force applied to the grasped object using a feedback control system with the CoP sensor force output.

In order to achieve as skillful handling as a human hand, it is necessary that a hand has the capability to manipulate tools. One important problem in tool handling with a multifingered hand is that the relative positions between the robot hand, the tool and the handled object change during the handling. For this reason, it is necessary to measure these relative positions in realtime, and to control the hand so as to cancel the changes. We have resolved this problem by using a high speed visual servoing. In this paper, as an example, a tweezers is handled by a multifingered hand. Further, a new method to control tools along with experiment result are shown.

In this paper the robotic throwing task is considered with the goal of achieving high-speed dynamic manipulation. We propose a kinetic chain approach for swing motion focused on torque transmission. In addition the release method using a robotic hand is analyzed for ball control. Experimental results are shown in which a high-speed manipulator throws a ball toward a target.

In this paper, we examine the relationship between a knotting process and the individual skills of which a robot hand is capable. To determine the necessary hand skills required for knotting, we first analyzed the knotting action performed by a human subject. We identified loop production, rope permutation, and rope pulling skills. To take account of handling of the two ends of the rope, we added a rope moving skill. We determined the characteristics of these skills using an intersection-based description. The knotting process was examined based on the analysis of knots and the characteristics of the robot hand skills. Finally, we show experimental results of an overhand knot and a half hitch performed using a high-speed multifingered hand system.

This paper describes a new realtime collision avoidance algorithm for a robot manipulator based on a new visual servo control. In the proposed algorithm, it is not necessary to compute the 3D position of the obstacle. Further, the manipulator can avoid the fast moving obstacle. The developed system uses several small light-weight high-speed vision chips placed on the surface of the robot manipulator. Experimental results of avoiding high-speed motion are shown.

This paper proposes a new strategy for making knots with a high-speed multifingered robot hand having tactile sensors. The strategy is divided into three skills: loop production, rope permutation, and rope pulling. Through these three skills, a knot can be made with a single multifingered robot hand. The dynamics of the rope permutation are analyzed in order to improve the success rate, and an effective tactile feedback control method is proposed based on the analysis. Finally, experimental results are shown.

In most previous studies, it has been difficult for a robot hand to regrasp a target quickly because its motion was static or quasi-static, in a constant contact state. In order to achieve high-speed regrasping, we propose a new strategy which we call dynamic regrasping. In this strategy, the regrasping task is achieved by throwing a target up and by catching it. In this paper, a regrasping strategy based on visual feedback is presented and experimental results using a high-speed multifingered robot hand and a high-speed vision system are shown. A cylinder is chosen as a specific example of target for dynamic regrasping, with which successful dynamic regrasping tasks are experimentally achieved

Speeding up robot motion provides not only improvement in operating efficiency but also improves dexterous manipulation by taking advantage of an unstable state or noncontact state. In this paper we describe a hybrid trajectory generator that produces high-speed manipulation. This algorithm produces both mechanical high-speed motion and sensor-based reactive motion. As an example of high-speed manipulation, a robotic ball control in a batting task has been achieved. Performance evaluation is also analyzed

This paper newly proposes the four-fingered robot hand with dual turning mechanism where two and two other fingers can independently rotate inner and outer circles with the common center, respectively. Due to this mechanical configuration, it has the particular rotating axis where the manipulation around the axis can be completely decomposed into the velocity control around the axis and the internal force control in the contact plane. We achieved a manipulation task around the axis with the time of 0.8[sec] for one rotation, while it is relatively slow for another axis.

A new multifingered hand system with quick finger motion of 180deg per 0.1s has been developed. With high-speed visual feedback at a rate of 1kHz, high-speed catching of a falling ball are achieved. In this catching task, a target was dropped from about 1m in height, and the speed of the falling ball became more than about 4m/s just before it hit the ground. The hand could precisely catch it using two fingertips, and the percentage of the success was almost 100%. This successful catching depends on not only high-speed precise motion of the hand, but also softness of the target surface. In this paper, the dynamical analysis of the high-speed catching is proposed, and the softness of the target surface makes very important roles in this tasks. As a result it is shown that the balanced impedance of the target surface and the feedback control of the hand is needed in order to achieve stable catching.

The coexistence of humans and industrial robots in a common workspace provides the advantage of increased flexibility in production or longer system up-time during maintenance. However, it is fundamentally necessary to guarantee the safety of the human. This paper presents an approach that uses a specialized tracking-vision-chip to realize a high-speed emergency-stop for safe human-robot-coexistence. The presented approach uses the ability of the vision-chip to perform pixel-parallel masking and fast summation-operations on binary images to detect whether the robot and human are too close. After initial evaluation in a (semi)-simulation, the approach was realized in an experimental system. Even with only a small 8-bit microcontroller controlling the vision-chip and the communication, a cycle time of more than 500Hz was achieved.

In order to achieve faster and more dexterous manipulations, we propose a strategy called "dynamic holding". In the dynamic holding condition, the object is held in a stable condition while moving at high-speed. In this paper, we first explain the concept of dynamic holding, then we describe an experiment of dribbling by a robot as an example of dynamic holding using a high-speed multifingered hand and a high-speed vision system.

A new robotic hand system with high-speed finger motion of 180 [deg] per 0.1 [s] has been developed. With high-speed visual feedback at a rate of 1 [kHz], various dynamic manipulations are achieved. As examples of dynamic manipulation we describe the dynamic active catching tasks for a falling ball and a falling cylinder using a high-speed multifingered hand. We achieved these difficult tasks by utilizing high-speed motion and impact forces actively. Experimental results using active catching strategy based on visual feedback are shown.

In this paper a robotic batting algorithm using a high-speed arm and high-speed stereo vision is proposed. With this strategy, the desired trajectory of the manipulator is generated so that both high-speed swing motion and tracking motion combine to meet the ball squarely with the bat. As a result the manipulator can follow the ball while swinging the bat at high speed even if it is difficult to predict the trajectory of a ball. Experimental results are shown in which a high-speed manipulator hits a ball thrown by a human.

In this paper a robotic catching algorithm based on a nonlinear mapping of visual information to the desired trajectory is proposed. The nonlinear mapping is optimized by learning based on constraints of dynamics and kinematics. As a result a reactive and flexible motion is obtained due to the real-time high-speed visual information. Experimental results on catching a moving object using a high-speed vision system and a manipulation are presented.

Research attempting to merge human and mechanical systems through a nervous system is one of the most dynamic fields in interface technology. Recently, artificial cochlear and retinal nerves were connected via a nervous system. However, there has been no report concerning the replacement of human tactile sensation by artificial sensors connected to the tactile sensory nerve system. The present study reports the development of a system which enables the touch strength applied to a robot hand to be sensed and applied to the hand of a human operator via the microstimulation method. The operator controls the grasping action of a robot hand equipped with tactile sensors at a remote location via a network. The contact information between an object and the robot hand is transmitted via the network, and the tactile sensory nerve of the operator is stimulated by a microelectrode. The force applied to the robot hand can be sensed as an applied force to the hand of the human operator. Applications of the proposed system include the presentation of tactile perception in remote manipulation. In addition, the proposed system is an effective method for transmitting sensation from artificial limbs. The present paper describes the experimental system and results.

In this paper we introduce a newly developed high-speed multi-fingered robotic hand. The hand has 8-joints and 3-fingers. A newly developed small harmonic drive gear and a high-power mini actuator are fitted in each finger link, and a strain gauge sensor is in each joint. The weight of the hand module is only 0.8 kg, but high-speed motion and high-power grasping are possible. The hand can close its joints at 180 deg per 0.1 s, and the fingertips have an output force of about 28 N. The hand system is controlled by a massively parallel vision system. Experimental results are shown in which a falling object was caught by the high-speed hand.

This paper discusses a design concept of a sensory-motor fusion system to achieve high performance in a dynamic changing environment. From the viewpoint of a dynamic system, the new concept called "dynamics matching" is proposed to match the dynamics constraints of a system. Based on this concept, we describe a high-speed vision chip that has a general purpose parallel processing array along with a photodetector all in a single silicon chip. Next we describe a new sensory-motor fusion system which consists of a hierarchical parallel processing system, a vision chip system, and a multifingered hand-arm. All sensory feedback, including visual feedback, can be achieved in 1 ms. In addition, as an application of the system, we demonstrate high-speed grasping using visual and force feedback.

In most conventional manipulation systems, changes in the environment cannot be observed in real time because the vision sensor is too slow. As a result the system is powerless under dynamic changes or sudden accidents. To solve this problem we have developed a grasping system using high-speed visual and force feedback. This is a multi-fingered hand-arm with a hierarchical parallel processing system and a high-speed vision system called SPE-256. The most important feature of the system is the ability to process sensory feedback at high speed, that is, in about 1 ms. By using an algorithm with parallel sensory feedback in this system, grasping with high responsiveness and adaptivity to dynamic changes in the environment is realized.