Table tennis robots have significantly advanced in performance owing to the rapid progress in deep learning and reinforcement learning technologies. However, these advancements often require a large number of training samples. Besides, research focused on the robot serve task remains relatively limited. In response to these problems, this paper proposes a sample-efficient adjustment-learning (SEAL) method for the serve task inspired by human experience in table tennis, which can inherently augment the available training samples without the need for additional sample collection. The adjustment learning does not require complex network structures but demonstrates superior performances. The models trained by adjustment learning have good generalization and robustness, that can adapt to different serve styles and reduce system transfer errors very efficiently. In addition, the random interpolation method during dataset generation stage is introduced, and the effectiveness of simultaneous learning in both joint space and Cartesian space is also demonstrated. For specific serve task, an accuracy of less than$\mathbf{3 0 ~ m m}$to any designated position at the first shot is achieved.

This work investigates the functionality of Semantic information in Contrastive Learning (SemCL). An advanced pretext task is designed: a contrast is performed between each object and its environment, taken from a scene. This allows the SemCL pretrained model to extract objects from their environment in an image, significantly improving the spatial understanding of the pretrained models. Downstream tasks of semantic/instance segmentation, object detection and depth estimation are implemented on PASCAl VOC, Cityscapes, COCO, KITTI, etc. SemCL pretrained models substantially outperform ImageNet pretrained counterparts and are competitive with well-known works on downstream tasks. The results suggest that a dedicated pretext task lever-aging semantic information can be powerful in benchmarks related to spatial understanding. The code is available at https://github.com/sjiang95/semcl.

We present One-Shot Affordance Learning (OSAL): a unified pipeline that learns manipulation for articulated objects by observing human demonstration only once. The key idea of our method is to embody affordance of articulated objects with an open-loop trajectory conditioned on a certain area of the object's surface. It serves as a simplified object-centric manipulation representation, which can be easily transferred into robot motion, while traditional methods fail to deal with the configuration difference between human hands and robot end effectors. Our system extracts the embodied affordance by focusing on hand action's effect on the object, and further grounds such affordance into object visual features through self-supervised learning for novel object configurations. We evaluated our method on a collection of real-life objects and furniture and demonstrated high success rates. With our system, humans only need to manipulate a novel object once with any gesture to transfer that manipulation skill to the robot, which we believe to be a highly efficient and user-friendly paradigm oriented for future real-life robots.

The tradeoff between the field-of-view and resolution of conventional onboard vision systems primarily results from their fixed optical components. We propose a novel active vision system, Falcon, as an optimal solution. This system comprises an electric zoom lens connected to a high-speed camera with a pair of galvanometer mirrors, enabling high-resolution imaging of a moving object across a wide range, from near to far. To ensure accurate calibration of the Falcon system, we introduce a novel mapping-based calibration method using external cameras. We also present a robust and lightweight visual feedback method that utilizes this mapping-based calibration for effective object tracking. The effectiveness of the Falcon system is verified by constructing a prototype and conducting tracking experiments in an indoor setting, which demonstrated the superior performance of our method. Additionally, we successfully achieved continuous and high-resolution imaging of a curved mirror on public roads while the vehicle was moving.

In daily life, we realize various complex tasks with our two upper limbs based on the so-called bimanual coordination phenomenon. A fundamental feature of bimanual coordination is the natural tendency to synchronize the motion of two upper limbs, resulting in some preferred patterns of interlimb coordination. In this study, based on the coarse-to-fine human-robot collaboration framework, we investigate the possibility of realizing human-robot interaction and collaboration for accurate manipulation utilizing voluntary bimanual coordination. The practical motivation of utilizing the voluntary bimanual coordination, which can be perceived as an indirect way of implementing interlimb transmission of force feedback information, is to avoid the bad effect on force feedback presentation due to the counterforce if the human-robot collaboration was realized in an unimanual manner. For experimental studies, we firstly evaluated the synchronous protocols in terms of in-phase and anti-phase between two arms. Based on the studied protocol, moving target tracking was then demonstrated as an application scenario of the proposed human-robot collaboration utilizing voluntary bimanual coordination.

Robot-catching of in-flight objects is a challenging task, requiring a high-frequency sequence of pose estimation, trajectory prediction, catching point determination, and motion planning. Considering the small working space and visual occlusion in the natural environment, we investigate robot-catching with a short-distance and partially observable trajectory in this paper. We introduce a marker-based high-speed visual tracking method to collect sufficient data from the limited trajectory. Besides, we design a new catching point selection strategy to achieve a timely and stable response of the robot arm. Based on the proposed method and the dynamics of a thrown ring, we get a success rate of 90% in experiments to catch the in-flight rings using a collaborative robot arm.

We propose an accurate and robust inter-vehicle distance estimation method using highspeed stereo vision. The framework involves two phases: a tracking phase, wherein a preceding vehicle is accurately and stably tracked by a tracking algorithm optimized for stereo high-speed vision, and a distance estimation phase, wherein the inter-vehicle distance is estimated via a highly accurate scale estimation and aggregation method for multiple scale-based distance estimations to ensure that it is more accurate and robust without introducing a delay. Further, we propose patch multiplexing to realize accurate and efficient aggregation even in situations where the scale changes rapidly (e.g., emergency braking). Through comparative analysis using three real-world scenarios, we verify that the accuracy of inter-vehicle distance estimation using our approach is comparable to that of laser rangefinders. We also demonstrate that differential quantities, such as velocity and acceleration, could be accurately estimated using an adaptive Kalman filter. Our results will help develop safe and accurate truck platooning and adaptive cruise control systems.

Augmented Human (AH) is a research field enhancing human physical abilities or supporting human activity using advanced technologies. As one of the AH approaches, previous studies have attached an actuator to a human body or tools used for an activity. The attached actuators are used to control their movements to support an activity. In this study, instead of attaching actuators, we propose to directly apply noncontact ultrasound force to a lightweight tool to manipulate it. The advantage of using noncontact force is that users do not need to wear a specific device and to process tools used for the activity. As a proof-of-concept system, we developed an ultrasound-based curveball system by which table tennis players can shoot a curveball regardless of their physical ability. In the system, a moving ping-pong ball (PPB) is a target tool for remote manipulation. The system curves the trajectory of a moving PPB by continuously focusing ultrasound on it. Users can control the curve timing and the curve direction (left or right) using a racket-shaped controller. In the user study, we conducted an actual table tennis match using the curveball system and qualitatively confirmed that the player using the system had the upper hand. Another user study using a ball dispenser quantitatively showed that the ultrasound-driven curveball increased the number of mistakes of the opponent player 2.95 times. These results indicate that the proposed concept is feasible.

In this study, human-robot interaction with multimodal perception interface combining human visual and haptic perception is introduced. In the proposed human-robot collaboration method, cognitive capabilities of human and accurate motion control capabilities of robot are integrated based on a coarse-to-fine strategy. Human operator is designated for coarse global motion under feedback interfaces utilizing human visual as well as haptic modalities. Simultaneously, fine local motion in an active manner without involving human intention-aware is realized by a robotic module with high-speed actuators and high-speed sensory feedback. Experiments demonstrated the effectiveness of the proposed method.

Physically moving real objects via a computational force connects computers and the real world and has been applied to tangible interfaces and mid-air display. Many researchers have controlled only a stationary real object by computational force. On the other hand, controlling a moving object can expand the real space that is controllable by the computer. In this paper, we explore the potential of computational force from the viewpoint of changing the trajectory of a moving object. Changing the trajectory is the primitive model to control a moving object, and it is the technological challenge requiring high-speed measurement and non-contact force with high-spatial resolution. As a proof-of-concept, we introduce Hopping-Pong changing the trajectory of a flying Ping-Pong Ball (PPB) using ultrasound force. The result shows that Hopping-Pong changes the trajectory of a PPB 344 mm. We conclude that a computational force is capable of controlling a moving object in the real world. This research contributes to expanding the computationally controlled space with applications for augmented sports, HCI and factory automation.

Human-robot collaboration taking both human and robot's advantages becomes very promising in recent years. In this study, we propose a new active assistant robotic system for human-robot collaboration with the aim of optimally combining the cognitive capabilities of human and accurate motion control capabilities of robot. Under our method, human operator is provided with pneumatic haptic feedback to guide the system for coarse global-motion. Fine local-motion in an active manner is realized by a dynamic compensation robotic module with high-speed visual feedback. Since the active assist behavior here is confined to local motion and is accordingly safe to human operator. Experiments of preliminary investigation for set-point positioning and contour tracing under human-robot collaboration were implemented. Experimental results verified the effectiveness of the proposed method.

In this paper, we propose a robotic assistance system for carrying out a highly accurate peg-in-hole task by using a high-speed vision system and high-accuracy actuators. In this system, the vision system measures the position and the posture of the workpieces, and the actuators are moved according to this information. By using the high-speed vision system and operating the system at 500 Hz, a hole formed in one workpiece can be made to follow the complicated motion of a peg operated by a human and can be accurately aligned with the peg. In addition, since the torque required to move the hole is calculated on the basis of the position and direction data of the workpieces in the coordinate plane of the image captured by the vision system, the system is not affected by camera installation errors. Moreover, by using two cameras and combining the information from them, we made the tracking field wider. In order to evaluate the performance of the system, we executed a peg-in-hole task using a 50μm peg and a 70μm hole, and showed that the system completed the task with a success rate of 90%.

The regrasping function of a robotic hand and arm has been investigated by many studies. Dynamic regrasping is performed by accelerating objects and it has the advantage of being able to perform the regrasp function at high speed. However, the difficulty of increasing the success rate is a persistent problem. In this study, we aimed to realize this continuous high-speed operation by increasing the success rate of the regrasping function. The handling of the Rubik's cube was used as the specific task to be performed. The action that was required to handle the Rubik's cube consisted of two types of regrasping motion and one type of one-face turn motion. In this study, a Rubik's cube was placed in a plane and manipulated by combining these three types of motion. Continuous operation was realized with a robotic hand and high-speed vision by utilizing environmental constraints in order to minimize the error. As a result, we succeeded 3 times in turning and regrasping in 1 s. Additionally, we were able to succeed 30 times in turning and regrasping in 10 s, with a success rate of 70%.

This video introduces our recent studies on human-robot collaboration based on dynamic compensation framework with the aim of optimally combining the cognitive capabilities of human and accurate motion capabilities of robot. Under the dynamic compensation approach, human operator is for cognitive global-motion without caring much about accuracy. Fine local-motion in an active manner is realized by a dynamic compensation robotic module based on high-speed visual feedback. Application scenarios from micro-manipulation to macro-manipulation are implemented.

This paper focuses on high-performance robotic contour tracking under the uncertainties that commonly exist in actual robotic applications. These uncertainties can be attributed to the robot itself (such as modeling errors or mechanical defects like backlash) or to environmental issues (such as calibration errors or misalignment of the workpiece). We propose a non-model-based dynamic compensation approach based on the coarse-to-fine philosophy, which enables contour tracking with both high speed and good accuracy. This is achieved by adopting a methodology in which a main robot performs fast but coarse motion, while an add-on module conducts accurate compensation for the overall uncertainties using a high-speed camera and high-speed compensation actuator. An algorithm called pre-compensated proportionalderivative sliding mode control (pre-compensated PD-SMC) is proposed to control the compensation actuator. The effectiveness of the proposed contour tracking approach and control algorithm are experimentally verified using two typical planar-contour shapes: a random smooth-curvature and rectangle.

The purpose of this work is to achieve high-speed dynamic manipulation. As one example, we deal with throwing a breaking ball. In this article we deal with two types of throwing with different grasping forms. We modeled a manipulator and a ball with a sphere rolling on the tip link of a three-degrees-of-freedom manipulator in the case where a ball is thrown with spin, and with a sphere held between the tip links in the case where the ball is thrown with less spin. We simulated the motion of the ball by using these models in each case. In addition, experimental results obtained with balls thrown by a high-speed manipulator showed that the spin rates with the two types of throwing differed by a factor of 4.3 at most.

Realizing accurate robotic manipulation of flexible objects is very challenging. It asks for dexterous manipulation skill, and demands good adaptation to dynamic uncertainties due to deformation. Unlike traditional methods that basically depend on complex modeling, we follow the opposite approach by exploring possible simplifications based on high-speed motion strategy as well as high-speed visual sensing. As an application task concerned with robotic needle threading, this work demonstrates our philosophy by proposing a novel and straightforward high-speed strategy to speed up the manipulation while solves the issue of deformation uncertainty at the same time. The methodology is based on a very simple physical fact, namely that by rotating a thread with sufficient velocity, the thread can be approximated as a rigid object caused by the constantly applied centrifugal force during the rotation. Therefore, the needle threading problem is converted into a simpler peg-insertion problem, and the complexity of interaction between robot and deformable thread is significantly simplified. Based on the proposed strategy, simple technical methods then become possible for adoption to accomplish the challenging task. Experiments show that the proposed approach allows the needle threading task to be realized very rapidly.

In this paper we focus on master-slave robot hand systems that can realize non-contact sensing and intuitive mapping between human hand motion and robot hand motion. Such a master-slave robot hand system can be effective from a viewpoint of usability. However, conventional systems are not able to adapt to dynamically changing environments because they have high latency from input to output. Therefore, we developed a fast-response master-slave robot hand system using a high-speed vision system and a high-speed robot hand. The latency of the proposed system is so small that humans cannot recognize it. The motion of a human hand is obtained with high-speed non-contact 3D sensing, and this motion is mapped to a high-speed robot hand, while taking account of structural differences between the human hand and the robot hand. We confirmed the effectiveness of our proposed system through experiments.

We propose a driving safety support system (DSSS) that employs a high-speed vision system installed in the environment surrounding, for instance, highways, urban roads, and intersections. The aim of the system is to recognize potentially dangerous traffic situations, including those that are undetectable from a moving vehicle, and to use this information for supporting safe driving. The system consists of a vision network of synchronized high-speed cameras that are capable of acquiring images at one-millisecond intervals, and vehicles that are capable of communicating with this network through communication hubs. We conducted collision avoidance experiments and demonstrated that, by introducing high-speed vision, the proposed system can resolve the issue of slow reaction time, which is common to environmental vision systems.

In this paper, we propose an entirely new manipulation strategy for dynamic manipulation of a ribbon with a high-speed manipulator. The manipulation strategy involves manipulating the object at a constant, high speed. Then, we can assume that the dynamic behavior of the ribbon can be obtained by performing algebraic calculations of the robot motion using the proposed strategy. Based on this assumption, we derive a model of the ribbon and suggest a motion planning method using the proposed model. Finally, we show experimental results of shape control of a ribbon based on the proposed method.

This paper presents a visual compliance strategy to deal with the problem of fast peg-and-hole alignment with large position and attitude uncertainty. With the use of visual compliance and adoption of a light-weight 3-DOF active peg, decoupled alignment for position and attitude is realized. The active peg is capable of high-speed motion and with less dynamic defects than a traditional robot arm. Two high-speed cameras, one configured as eye-in-hand and the other as eye-to-hand are adopted to provide with the task-space feedback. Visual constraints for effecting the visual compliant motion are analyzed. Alignment experiments show that peg-and-hole alignment with the proposed approach could be successfully realized with robust convergence, and on average, the alignment could be realized within 0.7 s in our experimental setting.

In this paper, we propose an entirely new manipulation strategy for dynamic manipulation of a flexible rope with a high-speed robot arm. The manipulation strategy involves manipulating the object at a constant, high speed. Then, we can assume that the dynamic behavior of the flexible rope can be obtained by performing algebraic calculations of the robot motion using the proposed strategy. Based on this assumption, we derive a model of the flexible rope and suggest a motion planning method using the proposed model. Finally, we show experimental results of rope deformation control based on the proposed method.

This video introduces an ultra high-speed robot as a milestone in the history of intelligent manipulation systems. To develop the ultra high-speed robot under the concept of dynamics matching, we began with the development of a 1 kHz vision system. Next, we developed a sensory-motor fusion system by introducing the 1 kHz vision system. In addition, we have developed a new high-torque mini actuator and a high-speed multi-fingered hand with these incorporated. Integration of these components brings real-time dexterous manipulations unlike commonly-used control based on prediction or learning.

In this research, we successfully demonstrated dexterous manipulation of sheet-like elastic objects, namely, playing cards, using a high-speed robot system. In particular, our goal was to achieve card flicking by vibrating a fingertip of a robot hand at high speed and card catching by using high-speed visual feedback based on a high-speed vision system. We discuss card grasping in the initial state based on the geometry conditions of the card and the kinematics of the robot hand, and we propose a strategy for card flicking. We also suggest a card catching method based on information obtained by the high-speed vision system. In addition, we obtain the card flicking conditions by analyzing the slip between the card and the fingertip of the robot hand. Finally, we show experimental results of card flicking and card catching.

The purpose of the work described in this paper is to achieve dynamic manipulation of a sheet-like flexible object. As one example, we examine dynamic folding of a cloth with two high-speed multifingered hands mounted on two sliders. First, dynamic folding by a human subject is analyzed in order to extract the necessary motions for realizing this task. Second, a model of a sheet-like flexible object is proposed by extending a linear flexible object model (algebraic equation) that takes advantage of high-speed robot motion. Third, motion planning of the robot system is performed by using the proposed model, and the simulation results are shown. Finally, an experiment was conducted with the robot motion obtained by the simulation.

In this paper, we propose an entirely new strategy for dexterous manipulation of a linear flexible object with a high-speed robot arm. The strategy involves manipulating the object at high speed. By moving the robot at high speed, we can assume that the dynamic behavior of the linear flexible object can be obtained by performing algebraic calculations of the robot motion. Based on this assumption, we derive a model of the linear flexible object and confirm the validity of the proposed model. Finally, we perform simulation of dynamic knotting based on the proposed model. Results of an experiment demonstrating dynamic knotting with a high-speed robot arm are shown.

This video introduces the demonstration of skillful manipulation using a high-speed robot system. The system consists of visual and tactile sensors at a rate of 1 kHz and a high-speed hand-arm manipulator. The high-speed sensory-motor fusion improves not just the speed of existing robot manipulations, but robotic skills by introducing the features peculiar to high-speed motion. Based on such a concept, new variations of skillful manipulation were achieved.

In this paper, we examine the relationship between a knotting process and the individual skills of which a robot hand is capable. To determine the necessary hand skills required for knotting, we first analyzed the knotting action performed by a human subject. We identified loop production, rope permutation, and rope pulling skills. To take account of handling of the two ends of the rope, we added a rope moving skill. We determined the characteristics of these skills using an intersection-based description. The knotting process was examined based on the analysis of knots and the characteristics of the robot hand skills. Finally, we show experimental results of an overhand knot and a half hitch performed using a high-speed multifingered hand system.

This paper proposes a new strategy for making knots with a high-speed multifingered robot hand having tactile sensors. The strategy is divided into three skills: loop production, rope permutation, and rope pulling. Through these three skills, a knot can be made with a single multifingered robot hand. The dynamics of the rope permutation are analyzed in order to improve the success rate, and an effective tactile feedback control method is proposed based on the analysis. Finally, experimental results are shown.