Anticipating a future scenario where the robot initiates its own actions and behaves voluntarily when collaborating with humans, our research focuses on human understanding and perception of unanticipated robot actions during physical human-robot interaction. While the current literature searches for key factors that make the human-robot collaboration successful, the question of how people experience the robot’s unanticipated action as cooperative or uncooperative seems to remain open. We designed a game-based experiment (N = 35) where the participant played a “catch-falling-coins” game by moving a robotic arm. Our experiment introduced unanticipated robot actions in an “active session” where the robot targeted higher-valued coins without first informing the participants. Through semi-structured interviews and statistical analysis of questionnaires (Big Five Personality Test, SAM, NARS and CH33), we examined the participants’ understanding of the robot’s “intention” and their positive or negative perception of the robot as cooperative or uncooperative. Among the participants who understood that the robot’s “intention” was to catch the higher-valued coins, the majority of them reported a positive perception of the robot (cooperative or helpful) while this was not the case among those who did not understand the robot’s intention. We also observed relevant relationships between some personality traits and a person’s understanding of the robot’s intention. Qualitative analysis of the interviews allowed us to structure the process of perception change during the game into three phases: confusion, investigation, and adaptation. We believe that our research contributes to the study of human perception, and particularly to the relationship between a human’s understanding of unanticipated robot actions and their positive or negative perception of the robot.

When generating robot motions based on instructions such as cooking recipes, ambiguity of the instructions and lack of necessary information are problematic for the robot. To solve this problem, we propose an efficient motion planning approach for a dual-arm robot by constructing a graph repre-senting a motion sequence based on a recipe consisting of verbal instructions and cooking images. A functional unit is generated based on the linguistic instructions in the recipe. Since most recipes lack the necessary information for executing the motion, we first consider extracting the information about the cooking motion like cutting from the food images of the recipe and supplementing it. In addition, to supplement the actions that humans perform unconsciously, we generate functional units for actions not explicitly mentioned in the recipe based on the current situation of the cooking process, and then connect them to the functional units generated from the recipe. Moreover, during the connection we consider the motion of the robot's arms in parallel for an efficient execution of the recipe, similar to those of a human. Through experiments, we demonstrate that for a given recipe, the proposed method can be used to generate a cooking sequence with the supplementary information needed, and executed by a dual-arm robot. The results show that the proposed method is effective and can simplify robot teaching in cooking tasks.

Unanticipated physical actions from the robot on humans [active physical human–robot interaction (pHRI)] may be inevitable with the deployment of robots in human-populated environments. However, it is still unclear how humans would perceive such actions and how the robot should execute them in a physically and psychologically safe manner. The objective of this article is to explore the possibility of quantifying the humans’ physical and mental state during an active physical interaction with a robot, by means of a laboratory experiment. We hypothesize that the active robot actions could cause measurable alterations in users’ data, which could be related to their perceptions and personalities. In the experiment, the user plays a visual game using the robot, which has a hidden task that results in active physical actions on the user. We collect data from physical and physiological sensors, and the perceptions and personalities via questionnaires and a semi-structured interview. Statistical analysis and clustering of the data collected from a total of 35 participants showed the relationships between participants’ physical and physiological data and their age, gender, perception, and personalities. Further developments based on these exploratory outcomes can be used to implement an active pHRI controller that can account for both the physical and the mental state of users.

This article presents a novel software architecture enabling the analysis of assembly actions from fine-grained hand motions. Unlike previous works that compel humans to wear ad-hoc devices or visual markers in the human body, our approach enables users to move without additional burdens. Modules developed are able to: (i) reconstruct the 3D motions of body and hands keypoints using multi-camera systems; (ii) recognize objects manipulated by humans, and (iii) analyze the relationship between the human motions and the manipulated objects. We implement different solutions based on OpenPose and Mediapipe for body and hand keypoint detection. Additionally, we discuss the suitability of these solutions for enabling real-time data processing. We also propose a novel method using Long Short-Term Memory (LSTM) deep neural networks to analyze the relationship between the detected human motions and manipulated objects. Experimental validations show the superiority of the proposed approach against previous works based on Hidden Markov Models (HMMs).

This paper proposes a robot assembly planning method by automatically reading the graphical instruction manuals designed for humans. Essentially, the method generates an Assembly Task Sequence Graph (ATSG) by recognizing a graphical instruction manual. An ATSG is a graph describing the assembly task procedure by detecting types of parts included in the instruction images, completing the missing information automatically, and correcting the detection errors automatically. To build an ATSG, the proposed method first extracts the information of the parts contained in each image of the graphical instruction manual. Then, by using the extracted part information, it estimates the proper work motions and tools for the assembly task. After that, the method builds an ATSG by considering the relationship between the previous and following images, which makes it possible to estimate the undetected parts caused by occlusion using the information of the entire image series. Finally, by collating the total number of each part with the generated ATSG, the excess or deficiency of parts are investigated, and task procedures are removed or added according to those parts. In the experiment section, we build an ATSG using the proposed method to a graphical instruction manual for a chair and demonstrate the action sequences found in the ATSG can be performed by a dual-arm robot execution. The results show the proposed method is effective and simplifies robot teaching in automatic assembly.

The purpose of this study is for a robot to learn picking motions in a logistics warehouse environment. The picking operation performed by a robot often fails owing to the inclination of items placed on a shelf, as well as the minimum clearance between the products and their vinyl packaging. Therefore, we considered acquiring a specific motion trajectory by reinforcement learning. However, because numerous types of items are handled in logistics warehouses, efficient learning is required. Therefore, in this research, we propose a method to efficiently exploration for learning picking an object by determining a focus exploration area for learning based on previous results of different objects.

We discuss grasp stability under gravity where each finger makes soft-finger contact with an object. By clustering polygon models of a finger and an object, the contact area between a finger and an object is obtained as the common area between an object cluster and a finger cluster. Then, by assuming the Winkler elastic foundation, the pressure distribution within the contact area is obtained. By using this pressure distribution, we show that we can judge grasp stability under soft-finger contact. We further consider defining a quality measure of a soft-finger grasp by assuming that although the gravitational force is applied to an object, the direction of gravity is unknown. To demonstrate the effectiveness of the proposed approach, we show several numerical examples.

Failure detection and correction is essential in robust systems. In robotics, failure detection has focused on traditional parts assembly, tool breakage, and threaded fastener assembly. However, not much work has focused on sub-mode failure classification. This is an important step in order to provide accurate failure recovery. Our work implemented a novel failure characterization scheme for cantilever snap assemblies. The approach identified exemplars that characterized salient features for specific deviations from a nominal trajectory. Then, a rule based approach with statistical measures was used to identify failure and classify failure sub-modes. Failure sub-mode classification was evaluated by using a reliability measure. Our work classified failure deviations with 88% accuracy. Varying success was experienced in correlating failure deviation modes. Cases with only 1-deviation had 86% accuracy, cases with 2-deviations had 67% accuracy, and cases with 3 deviations had 55% accuracy. Our work is an important step in failure characterization of complex geometrical parts and serves as a stepping stone to enact failure recovery.

This paper presents a knowledge model of everyday objects for semantic grasp. This model is intended for extracting the grasp areas of everyday objects and approach directions for grasping when the 3D point cloud data and the intended purpose are given. Parts that make up everyday objects have functions related to their manipulation. We therefore represent everyday objects in terms of connected parts of functional units. This knowledge model describes the structure of everyday objects and information on their manipulation. The structure of an everyday object describes component parts of the object in terms of simple shape primitives to provide geometrical information and describes connections between parts with kinematic attributes. The information on the structure is used to map the manipulation knowledge onto the 3D point cloud data. The manipulation knowledge of the object includes the grasp areas and approach directions for the intended purpose. Fine grasps suitable for the intended task can be generated by performing a grasp planning with consideration for stable grasp and the kinematics of the robot in the grasp areas and approach directions.

This paper proposes a method for bin-picking for objects without assuming the precise geometrical model of objects. We consider the case where the shape of objects are not uniform but are similarly approximated by cylinders. By using the point cloud of a single object, we extract the probabilistic properties with respect to the difference between an object and a cylinder and consider applying the probabilistic properties to the pick-and-place motion planner of an object stacked on a table. By using the probabilistic properties, we can also realize the contact state where a finger maintain contact with the target object while avoiding contact with other objects. We further consider approximating the region occupied by fingers by a rectangular parallelepiped. The pick-and-place motion is planned by using a set of regions in combination with the probabilistic properties. Finally, the effectiveness of the proposed method is confirmed by some numerical examples and experimental result.

Dexterous manipulation is an important function for working robots. Manipulator tasks such as grasping, assembly and disassembly can generally be divided into several motion primitives. We call such motion primitives “skills” and explain how most manipulator tasks can be composed of sequences of these skills. We will address the issues involved with various types of robots such as maintenance robots and service robots. We have considered hierarchizing the manipulation tasks of these robots since their tasks have become more complex than ever before. Additionally, as errors are seen likely to increase in complex tasks, it is important to implement effective error recovery technology. This paper presents our proposal for a new type of error recovery that uses the concepts of task stratification and error classification which can be expressed specifically using flow charts.

This paper proposes a method for planning the pick-and-place motion of an object by dual-arm manipulators. Our planner is composed of the offline and the online phases. The offline phase generates a set of regions on the object and the environment surfaces and calculates several parameters needed in the online phase. In the online phase, the planner selects a grasping pose of the robot and a putting posture of the object by searching for the regions calculated in the offline phase. By using the proposed method, we can also plan the trajectory of the robot, and the regrasping strategy of the dual-arm. Here, the putting posture of the object can be planned by considering stability of the object placed on the environment. The effectiveness of the proposed method is confirmed by simulation and experimental results by using the dual-arm robot NX-HIRO.

This paper proposes an object placement planner for a grasped object during pick-and-place tasks. The proposed planner automatically determines the pose of an object stably placed near a user assigned point on an environment surface. The proposed method first constructs a polygon model of the surrounding environment, and then clusters the polygon model of both the environment and the object where each cluster is approximated by a planar region. The placement of the object can be determined by selecting a pair of clusters between the object and the environment. We further impose several conditions to determine the pose of the object placed on the environment. We show that we can determine the position/orientation of the object placed on the environment for several cases such as hanging a mug cup on a bar. The effectiveness of the proposed research is confirmed through several numerical examples.

Snap assembly automation remains a challenging task. While progress is being made in localization of parts, force controllers, and control strategies, little work has been done to help the robot reason about its current state, such that if necessary, the robot can assume corrective actions to accomplish the task. Error prone situations caused by the unexpected motion of parts, localization errors, jamming or wedging, cannot be solved by force controllers alone. For this reason we propose a snap assemblies verification system for cantilever-snap fasteners. The verification works in concert with a control strategy that makes use of constraint designs embedded in the snap parts' physical design. The constrained assembly motion generates similar sensory-signal patterns across trials that facilitates force signal discrimination into higher level abstractions of intuitive behavior. This work's contribution is the design of a hierarchical taxonomy for cantilever-snap verification based on increasingly abstract layers that encode relative-change in the task's force signatures. A five-layered taxonomy is built on the concept that relative-change patterns can be classified through a small category set and aided by contextual information. The verification system yielded human apropos intuitive categorizations of task behavior for every state and effectively determined the assembly result. This simple yet effective approach will be expanded to perform probabilistic online system verification to aid in fault tolerance and the automation of cantilever-based snap assemblies.

This paper addresses the use of a service robot system working for humans in a daily life environment. The robot system described here always interacts with humans, and is therefore required to consider the satisfaction of users to provide suitable services. Our goal is to develop a robot system offering a hand-over service according to the preference of users. Here, we construct a task instruction system for the hand-over task utilizing the idea of Service Engineering as the first step in our research. The system can be used to instruct a robot to bring the indicated objects only by selecting task information, such as grasping position and posture, which can be utilized to estimate the user's preference.

This paper presents a grasping system for picking up an indicated object in a complex real-world environment using a parallel jaw gripper. The proposed grasping scheme comprises the following three main steps: (1) A user indicates a target object and provides the system with a task instruction on how to grasp it, (2) the system acquires geometric information about the target object and constructs a 3D environment model around the target by stereo vision using the information obtained from the task instruction, and (3) the system finds a grasp point based on grasp evaluation using the acquired information. As an example of the scheme, we examined the picking up of a cylindrical object by grasping at the brim. An important and advantageous feature of this scheme is that the user can easily instruct the robot on how to perform the object-picking task through simple clicking operations, and the robot can execute the task without exact models of the target object and the environment being available in advance.

This paper presents the development of Robot Technology Components (RTCs) using RT-Middleware (RTM) for designing assistive robotic arms and human-machine interfaces (HMIs) that can be used by the disabled. RTM is an open software platform for a component-oriented robot system. An RTC is the basic functional unit for constructing a robot system using RTM. The interface definition for the basic functionality of RTCs is standardized by the Object Management Group (OMG). Therefore, software modules written as RTCs can be shared and reused in other robot systems. One feature of an assistive robotic arm is that the arm is controlled by manual operation. Various HMIs for manually operating the robotic arms have been developed according to the abilities of each user and the requirements regarding the actions that the robot can perform. These HMIs can be interchangeably connected to the assistive robotic arms by implementing the software modules for the robotic arm and the HMI as RTCs with a common software interface.

In this paper, a training system designed to familiarize persons who have severe upper limb disabilities with robotic arm operation is described. The system combines a robotic arm simulator with various control interfaces and provides training tasks to learn how to operate robotic arms. In our training system, four test subjects used a virtual system to practice how to execute peg insertion tasks, and thus improved their ability to operate actual robotic arms. The training results, along with the results of post-training experiments conducted using an actual robotic arm, demonstrate the effectiveness of the training system.

A grasping and manipulation with a multi-fingered hand is comprised of one or more functional finger actions. We define a functional finger action as a ¿primitive action,¿ and fingers that are used in separate primitive actions in a grasping and manipulation as ¿functional finger isolation.¿ Various grasping and manipulations can be realized by assigning different primitive actions to the functional finger isolation form. This paper presents a towel picking task with a multi-fingered hand by assigning different primitive actions to fingers. In this paper, we describe the towel picking task by a series of primitive actions applied to the towel. Experimental results are given, and we also show that the specification of a hand for the towel picking can be derived by analyzing the primitive actions assigned to the fingers. By describing a grasping and manipulation task with primitive actions, it can be used with other type of robot hands.

To support disabled persons with less muscle strength like muscular dystrophy patients, a single switch and a scanning menu panel is introduced to input method for the manipulator control. The iArm manipulator and its simulator are connected to evaluate the input performance. The evaluation task is the handling of pegs on the pegboard with the manipulator. Muscle dystrophy patients tested our interface to control both of the simulator and the real manipulator with a pegboard. The view of the robot control by the subject is shown on the video.

We are developing a manipulator system in order to support disabled people with less muscle strength such as muscular dystrophy patients. Such a manipulator should have an easy interface for the users to control it. In this paper, we report on the construction of the user interface of the system using RT-middleware. RT-middleware is an open software platform for robot systems. Therefore other interface components or robot components which are adapted to other symptoms can be replaced with the interface without any change of the contents. A single switch is introduced as the input device for the manual control of the robot arm in our interface. The scanning menu panel is designed to perform various actions of the robot arm with the single switch. A manipulator simulation system was constructed to evaluate the input performance. Two muscular dystrophy patients tried of our interface to control the robot simulator and made comments.

We propose a method for generating robot motions by integrating multiple policies effective for task achievement. The method obtains a new policy efficiently based on the applied policies. However, there might be some states in which the applied policies fail to achieve the task. The failing states are found by means of a decrease in the state values. The policies for the states are then modified. In this paper, we applied this method to clutch assembly in order to demonstrate its validity for assembly tasks. We integrated insertion motion and search motion for the task and finally obtained the effective motion that accords to the task states.

This paper proposes a method for planning robot motions by integrating multiple knowledge that is effective in task achievement. The method efficiently obtains a new policy, which is a mapping from states to actions, on the basis of the knowledge presented in a state-action map. However, in some states, the applied knowledge fails to achieve a given task. In our method, the failing states are found by using the decrease in the state values, and the policy for these states is then modified. In order to demonstrate the validity of our method, we applied it to rearrangement tasks of multiple objects. The appropriate policies were obtained by integrating programs for similar tasks and a simple rule for the task process; moreover, a new knowledge that is effective in the rearrangement tasks was extracted from the obtained policies

Parameter tuning of force control is important for successful robotic assembly and ensuring high efficiency operations. In this paper, we present a method of designing damping control parameters for general assembly operations by considering the cycle time. In this method, optimal parameters are obtained through iterative simulations of assembly operations because it is difficult to estimate the cycle time analytically. We have applied the method to clutch assembly, which is a complicated insertion of a splined axis into movable toothed plates, and demonstrate how the operation can be sped up using the obtained parameters.

It is necessary to design appropriate parameters of force control in robotic assembly so as to achieve successful operations. Moreover, obtaining control parameters that can shorten the cycle time to perform operations is very important in industrial applications. In this paper, we propose a method to obtain control parameters through repetitive simulation of an assembly operation in order to estimate the cycle time because it is difficult to calculate the cycle time analytically. First, a simulator of a robotic peg-in-hole operation is developed based on preliminary experiments. We then formulate our proposed method as an optimization problem and solve the problem using the simulator. Finally, results of the optimization are presented.

Designing appropriate force control parameters is necessary in robotic assembly for successful operations. Moreover, obtaining control parameters that can shorten the cycle time to perform operations is very important in industrial applications. In this paper, we present a method to design damping control parameters that can shorten the cycle time and show the effectiveness of the parameters obtained through the method. In the method, sub-optimal parameters are obtained through repetitive simulations of assembly operations in order to estimate the cycle time because it is difficult to calculate the cycle time analytically. We apply the method to peg-in-hole operations and show experimental results using the control parameters obtained through the method.