With the development of real-time video streaming technology on the internet, live streaming services have been popular. In live streaming, it is common for users to have multiple live streams they want to watch simultaneously. Live streaming allows for simultaneous viewing across multiple windows. However, when engaging in simultaneous multi-window viewing, there is a risk of missing highlights from one stream while focusing on another. In this study, to avoid missing highlights during simultaneous viewing, we propose a method that predicts upcoming highlights by utilizing weighted moving averages that can capture trend changes in time series data in real time, based on chat text and audio from live streams. We evaluated the effectiveness of the proposed method using 23 YouTube Live streams, confirming that combining both textual and audio information enables effective real-time highlight prediction.

In this paper, we report on scene classification experiments in FPS (First-Person Shooter) game videos focusing solely on acoustic information. Generally, scene classification in videos considers both visual and acoustic information. However, the visual information in FPS games contains game-specific elements that make it challenging to construct scene classifiers with generalization capabilities. In contrast, acoustic information in FPS games tends to have more commonalities, such as weapon usage sounds, which can contribute to successful scene classification. Focusing on acoustic information that provides essential clues for scene identification in FPS games, in the experiments, we employed a simple CNN model to classify audio segments into combat and non-combat scenes. To evaluate the classification performance, we used an FPS game audio test set. The test set consists of audio segments extracted from the audio streams of FPS game videos. Each audio segment is assigned a scene label indicating whether it belongs to a combat or noncombat scene. The results show that game scenes can be roughly classified with a simple CNN model by focusing on just acoustic information of FPS games.

In this work, we propose an approach to manipulate objects by position-controlled robot hands: in-hand caging manipulation. In this method, an object is manipulated based on caging without force sensing or force control. An object is caged by a robot hand throughout manipulation, and we can locate the object around a goal by deformation of the cage without sensing the object configuration. In this paper, we considered 2-D polygonal objects as the targets of in- hand caging manipulation. We also considered some position- controlled hands as the devices to manipulate the objects. A motion planning algorithm for the hands was proposed and applied to planar in-hand caging manipulation. By moving the hands according to the result of the motion planning, it was possible to manipulate the objects without sensing. Our proposed method can be applied to a device such as a part feeder that aligns variously shaped parts in the same pose.

For virtual evaluation of universal design products, it is necessary to synthesize natural grasps for various hands including those with limited range of motion (ROM). In this paper, we study synthesizing grasps for digital hands with limited ROM in their thumbs' joints. We apply a contact-region-based method for grasp synthesis to this problem. In our previous study, two types of grasps were observed under the limitation of the thumb's ROM. One was a type that used the lateral region of the thumb, and the other was a type that used he same contact regions as healthy hands. In this paper, grasps are synthesized for two objects using three hand models whose lengths are different. With the same input information as the healthy hand, it is not possible to synthesize the grasps for the ROM-limited hand. On the other hand, when grasps are synthesized by changing the information about grasping target points of the thumb and the palm, feasible grasps for the ROM-limited hand can be found. For a camera object, the grasp using the thumb's radial region is automatically synthesized without changing the specified contact region. Also, for a spray object, a grasp that uses the same contact region as the healthy hand is synthesized. These results demonstrate our method can synthesize natural grasps even for ROM-limited hands, which will contribute to universal product design.

This paper presents a robotic 3D printer: a robot system that can assemble toy brick sculptures from their 3D CAD models. In this system, a 3D CAD model is automatically converted to a block model consisting of primitive toy blocks. Then an assembly plan of the block model is automatically generated, if feasible. According to the plan, an industrial robot assembles a brick sculpture layer by layer from bottom to top. We demonstrate successful assembly of several brick sculptures.

This paper describes a geometrical method for grasping objects wherein an object is caged by rigid parts of a robot hand and simultaneously grasped by soft parts attached to said rigid parts. We term this process as caging-based grasping. In this study, we derive concrete conditions for two- and three-dimensional caging-based grasping of objects of various shapes using circular robots and multi-fingered robot hands. Then, we validate the derived conditions by performing caging-based grasping experiments.

Picking of parts loaded in bulk is an industrial need. Thus bin-picking systems for various objects have ever been studied by various ways. However, it is difficult to recognize coil springs randomly placed in a pile by conventional machine vision techniques because of their shape characteristics. In this paper, we propose a method of recognition and pose estimation of coil springs. This method uses their highlights made by illumination for their recognition and pose estimation with stereo vision. We implemented this method as a bin-picking system with an industrial robot. Bin-picking of coil springs was successfully demonstrated on the system. Position errors were less than 2 mm. The average success rate for a coil spring in the part box was 94% when multiple retrials of picking were allowed. This rate could be improved by implementation of collision avoidance.

Industrial robots are used widely in factories and a method in which everyone can easily teach their “good” motions is demanded. For this purpose, our group proposed a robot programming method consisting of manual volume sweeping by an operator and automatic motion planning in order to generate motions with short cycle times. A swept volume is the space that robot bodies have passed without collisions, thus we can recognize that the swept volume is free space. In this paper, to reduce a burden of the robot programming we improve our method by calculation of swept volumes, display of them and motion planning totally online during the manual sweeping. Teaching experiments were performed to evaluate the performance of the proposed method. Paths generated by all operators had shorter cycle time than teaching/playback by direct teaching, which shows the effectiveness of the proposed method.

Graspless manipulation is easily interfered by external disturbances because the manipulated object is not completely held by a robot hand and supported by an environment such as a floor. Thus it is important to ensure the manipulation is executed robustly against some disturbances. In our works, a rigid-body-based analysis of indeterminate contact forces for quasi-static graspless manipulation has been proposed, and also joint torque optimization for robotic hands. The joint torques of the robot is determined in consideration of some robustness of manipulation against disturbances, which include changes or estimation errors of friction. In the analysis of contact forces in quasi-statics, a kinematic constraint on static friction is considered to exclude infeasible sets of frictional force, with considering treatment of kinetic friction. Additionally, new objective functions for computing optimal joint torques in both static and quasi-static graspless manipulation are proposed. Some numerical samples of both applications are shown to verify our proposed methods.

The authors study “view-based teaching/playback”: a method of robot programming with view-based image processing which can achieve more robustness against changes of task conditions than the conventional teaching/playback scheme. Reinforcement learning was integrated with this method to make necessary human teaching minimal and it was applied to robotic manipulation in a virtual environment in our previous study. However, the process of online reinforcement learning requires huge computation time. Moreover, the low success rate of reinforcement learning makes the expected time to learning success longer. Thus we accelerate the process of reinforcement learning in this paper to make our proposed method more practical. We also try to improve the success rate of reinforcement learning for further learning acceleration.

This paper proposes a novel approach to build a human grasp database that considers contact regions for grasping posture synthesis. To cover all types of hand surface regions contacting an object when grasping, the grasp taxonomy from other research was extended. For each pattern of region usage, we collected several posture variations that differ in the distance between regions. In total, 801 postures were collected for 107 patterns of region usage. A posture to grasp a new object was generated by interpolation with optimization-based adjustment using the proposed database, given a goal contact state of the hand with the object.

This paper proposed a method of modeling the range of motion (ROM) of the human hand, which has multiple joints that move in coordination. Traditionally, ROM of the hand was defined by independently bounding each joint angle from observation of extreme posutres. For example, the joint angle when fully extended (maximum) and that when fully flexed (minimum) were observed. However, it is difficult to express actual human's complex ROM with such simple boundaries. Therefore, we modeled ROM of the hand by defining outer boudary of collected various posture data. Each relationship between two of the joint angles was presented as a united area in which all the projected measured postures on the plane were minimally bounded using the a-convex hull algorithm. Such area was called "a coupled ROM" in this paper. Measurement and modeling experiments on four subjects were conducted to demonstrate basic characteristics of the proposed ROM model. The occupied volume of the proposed ROM was compared with that of the simple upper-lower ROM. The coordinatinated relationships were ranked and categorized for comparison among subjects.

Caging is a method to make an object inescapable from a closed region by rigid bodies. Position-controlled robot hands can capture an object and manipulate it via caging without force sensing or force control. However, the object in caging is freely movable in the closed region, which may not be allowed in some applications. In such cases, grasping is required. In this paper, we propose a new simple approach to grasping by position-controlled robot hands with the advantage of caging: caging-based grasping by a robot hand with rigid and soft parts. In caging-based grasping, we cage an object with the rigid parts of the hand, and construct a complete grasp with the soft parts. We formulate the caging-based grasping, and derive concrete conditions for caging-based grasping in planar and spatial cases, and show some experimental results.

In this paper, we present a new method for robot programming: view-based teaching/playback. It was developed to achieve more robustness to changes of task conditions than conventional teaching/playback without losing its general versatility. Our method is composed of two parts: teaching phase and playback phase. In the teaching phase, a human operator moves a manipulator to achieve a manipulation task. All the movements of the manipulator are recorded. All the images of the teaching scenes are also recorded by a camera. Then, a mapping from the recorded images to the corresponding movements is obtained as an artificial neural network. In the playback phase, the motion of the manipulator is determined by the output of the neural network using camera images. We applied this view-based teaching/playback to pushing tasks by an industrial manipulator. After multiple teaching demonstrations, pushing an object to the goal position was successfully achieved from some initial positions that were not identical to those in the demonstrations, without object models or camera calibrations.

This paper proposes a method to build an individual hand model that consists of a surface skin and an inside link model, which can be used to reconstruct hand behavior from motion capture (MoCap) data. Our system uses a static posture data, a palmar side photo and marker positions captured simultaneously by MoCap, to reduce extra time and effort demanded for each subject to build a model. From this modeling scan, several hand dimensions and marker positions are obtained. Joint centers are estimated based on regression analysis about joint centers, marker positions and some hand dimensions derived from magnetic resonance (MR) images of eight subjects. The skin surface is built by scaling a generic hand model so that it satisfies the measured dimensions. The proposed system will be validated through an experiment to build four subjects' hand models.

In this paper, we propose a robot programming method for industrial manipulators. It enables novice human operators to teach "good" robot motions in a short period of time. First in this method, a human operator makes a robot manipulator sweep a volume by its bodies. The manipulator is equipped with a force sensor on its wrist and damping-controlled; the operator can move it freely by grasping its end-effector. The swept volume stands for a part of the manipulator's free space, because the manipulator has passed through the volume without collisions. Next, the obtained swept volume is used by a motion planner to generate a well-optimized path of the manipulator automatically. Finally, the planned motion is executed by the manipulator. Even non-skilled operators can generate robot motions with short cycle time by doing the above procedure. The effectiveness of our method is successfully demonstrated in teaching experiments for a comparison with other conventional methods.

In this paper, three-dimensional caging by a multifingered hand (3D multifingered caging) is studied. Caging is a method of object constraining in which robot bodies surround an object and make it inescapable from the ldquocagerdquo composed of the robot bodies. In 3D multifingered caging, position-controlled robot hands can constrain objects and transport them, and force control is not necessary. Furthermore, even a robot hand with low degrees of freedom can constrain an object to manipulate. We show some examples of 3D multifingered caging. Then, we derive sufficient conditions for caging objects, and propose a method to plan configurations of a robot hand for caging with the sufficient conditions.

In this paper, we analyze indeterminate contact forces in robotic grasping and contact tasks. Previous studies by Omata and Nagata showed that there is a constraint on static friction forces, which is derived from contact kinematics, in rigid-body power grasps. The set of possible contact forces can be calculated using the constraint. This approach can be applied to not only mechanical analysis of power grasps but also that of other robotic contact tasks. However, there are some cases where Omata and Nagata’s formulation generates paradoxical results on contact forces. In this paper, we investigate this problem and propose a modified method to calculate the set of possible indeterminate contact forces. We also study how to reduce the computation.

This paper presents manipulation of small electronic parts whose size is submillimeter order or smaller. For assembly of small parts which are components of electronic devices, we need two key technologies: accurate position control and force control in handling them. In this paper, we utilize a four-degree-of-freedom assembly robot with a camera for position control. This vision-based assembly system gives us positioning accuracy of micrometer order. We also control the pressing load of the robot in pick-and-place of small objects indirectly with mechanical springs and a displacement sensor by changing resilience force of springs actively. We execute some experiments to handle small electronic parts using above-mentioned active force control and vision-based position control.

The present paper develops a model of "active" fingertip forces of a human index finger that allows the maximum fingertip force to be calculated for any specified position of the finger. The proposed model extends a passive fingertip force model by considering the mobility of the fingertip in terms of its deformation. The model selects the smaller force of the following two forces: the maximum force calculated using the musculoskeletal model and the maximum force calculated using the maximum deformation depth of the fingertip that lies within the ranges of motion of all the joints. A further extension of the previous passive fingertip force model is that a nonlinear relationship between muscle length and exerted force is considered. Maximum active fingertip forces were measured for three subjects and compared with the forces calculated using the proposed model and the previous model. The results of our model were in good agreement with the measured forces in contrast with the large values obtained using the passive fingertip force model.

The human eyes are a powerful sensor to detect valuable objects, and of the valuable structure among objects in the environment. Scientists executed experiments on these balls, for investigating the attention of human in searching things to buy and in exploring clues for problem solving. The shift of attention target toward insight has been reported in previous studies, but it has not been clarified how logical ideas of human is emerging for the short time attaining insight. This paper reports the discovery that the orbits of eyes' target movement just before insight can be regarded as auxiliary lines, i.e., lines which externalize the hidden relationship of objects which accelerate the human's deeper thought. The experiments were executed for subjects looking at artificial products and artworks and also for subjects designing a new city on a map. The snapshots of eye movements can be interpreted as reflecting the subjects' interest in combining basic parts to make a valuable structure. By showing the snapshot just before insight to subjects themselves, they tended to notice why they attained insight and could explain the value of the objects to others.

In this paper, the robustness of graspless manipulation (or nonprehensile manipulation) is investigated. We derive some new constraints for static frictional forces in graspless manipulation, which gives us more accurate evaluation of the robustness of graspless manipulation than previous studies. We also present a procedure to calculate a measure of the robustness based on linear programming. Numerical examples are shown to prove that our new method works well even when previous methods make an inappropriate evaluation

In this paper, we study excessive internal forces applied to manipulated objects in robotic contact tasks. Such internal forces may cause serious damage to the objects and/or robots; thus, prior assessment of the possibility of excessive internal forces is very important. We propose a method to judge the possibility of excessive internal forces on manipulated objects. The judgment is based on rigid-body mechanics with Coulomb friction and given by solving a series of linear programming problems. We show some numerical examples of the judgment by our proposed method.

Parameter tuning of force control is important for successful robotic assembly and ensuring high efficiency operations. In this paper, we present a method of designing damping control parameters for general assembly operations by considering the cycle time. In this method, optimal parameters are obtained through iterative simulations of assembly operations because it is difficult to estimate the cycle time analytically. We have applied the method to clutch assembly, which is a complicated insertion of a splined axis into movable toothed plates, and demonstrate how the operation can be sped up using the obtained parameters.

In this paper, we present a method of motion planning of multiple robot fingertips for graspless (or nonprehensile) manipulation. The method can automatically generate various graspless operations, including pushing and tumbling. By considering whether each robot finger should be position- or force-controlled, we can obtain robust manipulation plans against external disturbances. Some examples of planned graspless manipulation of a cuboid by two robot fingers are presented. We also show an experimental result of execution of planned graspless manipulation by a robot with a multi-fingered hand.

It is necessary to design appropriate parameters of force control in robotic assembly so as to achieve successful operations. Moreover, obtaining control parameters that can shorten the cycle time to perform operations is very important in industrial applications. In this paper, we propose a method to obtain control parameters through repetitive simulation of an assembly operation in order to estimate the cycle time because it is difficult to calculate the cycle time analytically. First, a simulator of a robotic peg-in-hole operation is developed based on preliminary experiments. We then formulate our proposed method as an optimization problem and solve the problem using the simulator. Finally, results of the optimization are presented.

Designing appropriate force control parameters is necessary in robotic assembly for successful operations. Moreover, obtaining control parameters that can shorten the cycle time to perform operations is very important in industrial applications. In this paper, we present a method to design damping control parameters that can shorten the cycle time and show the effectiveness of the parameters obtained through the method. In the method, sub-optimal parameters are obtained through repetitive simulations of assembly operations in order to estimate the cycle time because it is difficult to calculate the cycle time analytically. We apply the method to peg-in-hole operations and show experimental results using the control parameters obtained through the method.

In this paper, a flexible robotic assembly system with decentralized architecture is presented. The system is designed to achieve high reconfigurability so that it can adapt to changes in manufacturing environment; a new robot can be easily installed to the system and execute assembly tasks immediately, together with other devices. For easier reconfiguration, a semi-automated calibration method for positions of newly installed robots is integrated into the system. Our implementation of the assembly system consists of conventional manipulators and a belt conveyor. In the experiment, a new mobile manipulator was installed successfully and performed an assembly operation jointly with other existing manipulators.

To achieve various manipulation tasks by robot fingers, we have to use both position control and force control appropriately. In this paper, we propose a method to determine such control modes of robot fingers in graspless manipulation. Using the method, we can assign a position- or force-control mode to each robot finger so that the maximum manipulation stability will be achieved without excessive internal force. Desired finger forces are also determined for robot fingers to be force-controlled. We show some numerical examples of automatic determination of control modes in graspless manipulation of polyhedra on a plane by two robot fingers.

Proposes a flexible assembly system, where autonomous manufacturing devices and production management agents communicate with one another to accomplish production tasks. This architecture allows the system to arrange manufacturing devices independently of the type of products assembled and to assemble multiple products in parallel and asynchronous progress. This system also supports plug and produce, a system function that realizes easy reconfiguration. Thus, the system can return quick responses to breakdowns and changes in production quantities. This paper also presents an index for the general evaluation of reconfigurable manufacturing systems. With this index and the plug-and-produce function, the system can be reconfigured appropriately to adapt to changes in the manufacturing environments.

The stability of objects in graspless manipulation (or nonprehensile manipulation) is investigated. In contrast with the stability of grasps, it is crucial for the stability of graspless manipulation to evaluate frictional forces applied to objects in contact motion. We formulate the effect of Coulomb friction and propose a quantitative stability measure for graspless manipulation. The measure can be calculated approximately with linear programming. Finally, we show numerical examples. Our stability measure will be useful in planning of graspless manipulation.

In this paper, we propose a simple teaching method for industrial robots by human demonstration. The method can be divided into two parts: the teaching phase and the planning phase. In the teaching phase, a human operator demonstrates a manipulation of an object, and two cameras recognize the path of the object by observing markers attached on the object. In the planning phase, a motion planner generates a sequence of robot operations to play back the demonstrated path by pick-and-place and pushing. The proposed method incorporates automated camera calibration required for human demonstration, which enables labor-saving teaching and compensates the absolute positional error of industrial robots.

Graspless manipulation is to manipulate objects without grasping by just pushing, tumbling and so on. One of the main difficulties in graspless manipulation is planning. To reduce the load of computation, we adopt a two-step approach: 1) construction and simplification of contact state graph at geometry level; and 2) planning of manipulation at mechanics levels. In this paper, we focus the latter and propose an algorithm to plan mechanically feasible manipulation. It generates digraphs that represent C-subspaces for all the contact states, and unites them into one big graph, which we call a manipulation-feasibility graph. The manipulation plan can be obtained by searching the graph. This algorithm is implemented for planar graspless manipulation by multiple robot fingers, and the planned results obtained are shown.

Adaptation of biological systems can be viewed as decreasing local interaction among subsystems to realize effective cooperation. Conversely, it is important for effective cooperation to evaluate and control local interaction. Based on this idea we present a method of realizing cooperative rope turning by a man and a manipulator. The rotating motion of the manipulator arm for rope turning can be specified by frequency and phase difference between the arm and the rope. The frequency is determined in real-time by rhythm entrainment using a phase-locked loop. The phase difference is related to the quantity of local inter-action, that is, energy transfer from the arm to the rope. Therefore it is controlled to realize desired energy transfer, which is estimated from human demonstration. In the experiment the manipulator can successfully adapt its motion to that of the human partner.

This paper proposes a flexible assembly system that manages production by negotiating with autonomous manufacturing devices. With this mechanism, physical layout of the system is independent of the type of products to be assembled, and the system can asynchronously process multiple parallel production tasks. This system also supports plug and produce: a system function that enables easy reconfiguration. Thus, the system can be quickly accommodated to breakdowns and changes in production quantities. This paper also deals with an index for general evaluation of the manufacturing systems. With this index, we can plan appropriate improvement of the system layouts. Then we reconfigure the system using plug and produce function, in order to accommodate it to the changes in the manufacturing environments.

A control method of robots for human-robot cooperative manipulation is investigated. We propose estimating human motion using the minimum jerk model for smooth cooperation. Using a nonlinear least-squares method, we identify two parameters of the minimum jerk model in real-time. The estimated position of the human hand is used to determine the desired position of the end-effector of the manipulator in virtual compliance control. The effectiveness of the proposed method is verified by experimentation with an industrial 6-degree-of-freedom manipulator. Energy transfer in cooperative manipulation is studied for quantitative evaluation of achieved cooperation from the viewpoint of adaptation theory.

We aim at handling an unknown object by cooperating multiple mobile robots. The most important problem in this paper is obtaining a weight and mass center position of an object. Another important problem is to determine appropriate grasping points before handling. This method is proposed to avoid bad situations, such as making an object fall down, or some robots overloaded in handling. The algorithm we propose in this paper is a first step of grasping objects, which is "determination of initial grasping points". Initial grasping points are given by only a geometrical shape of an object. The robots are placed where the possibility that robots can lift the object with no trouble. This evaluation is based on the normal distribution model proposed in this paper. We are also going to discuss how to avoid and regrasp when initial grasping has failed. We show a simple numerical example of initial grasping and re-grasping, in planar object model.

The stability of an object manipulated by multiple robots (serial-link or wire-driven mechanisms) in contact tasks is investigated. We show that resultant force/moment applicable to the object through the contacts is represented as a polyhedral convex region in force/moment space. Then we propose a stability measure and a method of calculating it approximately by linear programming technique. We also present a linear programming-based algorithm to judge the possibility of excessive internal force in contact tasks by position-controlled robots.