In industrial assembly tasks, the in-hand pose of grasped objects needs to be known with high precision for subsequent manipulation tasks such as insertion. This problem (in-hand-pose estimation) has traditionally been addressed using visual recognition or tactile sensing. On the one hand, while visual recognition can provide efficient pose estimates, it tends to suffer from low precision due to noise, occlusions and calibration errors. On the other hand, tactile fingertip sensors can provide precise complementary information, but their low durability significantly limits their use in real-world applications. To get the best of both worlds, we propose an efficient method for in-hand pose estimation using off-the-shelf cameras and robot wrist force sensors, which requires no precise camera calibration. The key idea is to utilize visual and contact information adaptively to maximally reduce the uncertainty about the in-hand object pose in a Bayesian state estimation framework. As most of the uncertainty can be resolved from visual observations, our approach reduces the number of physical environment interactions while keeping a high pose estimation accuracy. Our experimental evaluation demonstrates that our approach can estimate object poses with sub-mm precision with an off-the-shelf camera and force-torque sensor.

In this paper, we present a diabolo model that can be used for training agents in simulation to play diabolo, as well as running it on a real dual robot arm system. We first derive an analytical model of the diabolo-string system and compare its accuracy using data recorded via motion capture, which we release as a public dataset of skilled play with diabolos of different dynamics. We show that our model outperforms a deep-learning-based predictor, both in terms of precision and physically consistent behavior. Next, we describe a method based on optimal control to generate robot trajectories that produce the desired diabolo trajectory, as well as a system to transform higher-level actions into robot motions. Finally, we test our method on a real robot system playing the diabolo, and throw it to and catch it from a human player.

Practical industrial assembly scenarios often require robotic agents to adapt their skills to unseen tasks quickly. While transfer reinforcement learning (RL) could enable such quick adaptation, much prior work has to collect many samples from source environments to learn target tasks in a model-free fashion, which still lacks sample efficiency on a practical level. In this work, we develop a novel transfer RL method named TRANSfer learning by Aggregating dynamics Models (TRANS-AM). TRANS-AM is based on model-based RL (MBRL) for its high-level sample efficiency, and only requires dynamics models to be collected from source environments. Specifically, it learns to aggregate source dynamics models adaptively in an MBRL loop to better fit the state-transition dynamics of target environments and execute optimal actions there. As a case study to show the effectiveness of this proposed approach, we address a challenging contact-rich peg-in-hole task with variable hole orientations using a soft robot. Our evaluations with both simulation and real-robot experiments demonstrate that TRANS-AM enables the soft robot to accomplish target tasks with fewer episodes compared when learning the tasks from scratch.

Robotic contact juggling is a challenging task in which robots must control the movement of a ball rapidly and indirectly without holding it while keeping the ball in and sometimes out of contact with the robot’s body. In this work, we address the problem of learning such robotic contact juggling from trial and error via model-based reinforcement learning (MBRL). The key insight is that complex robot-ball interactions of the contact juggling actually consist of a small set of simple dynamics that each corresponds to a distinct interaction "primitive" such as touching and releasing the ball. Accordingly, we develop a tailored MBRL method that incrementally fits a set of simple dynamics models to the movements of a robot and a ball while also learning a switching model that can select a proper dynamics model depending on the current state and action. The learned model can then be used in an MBRL framework to seek optimal juggling control. We demonstrated the effectiveness of our approach on a simulator of contact juggling performed by a robotic arm.

In industrial assembly tasks, the position of an object grasped by the robot has to be known with high precision in order to insert or place it. In real applications, this problem is commonly solved by jigs that are specially produced for each part. However, they significantly limit flexibility and are prohibitive when the target parts change often, so a flexible method to localize parts with high accuracy after grasping is desired. To solve this problem, we propose a method that can estimate the position of an object in the robot's hand to sub-millimeter precision, and can improve its estimate incrementally, using only minimal calibration and a force sensor. Our method is applicable to any robotic gripper and any rigid object that the gripper can hold, and requires only a force sensor. We demonstrate that the method can determine the position of an object to a precision of under 1 mm without using any part-specific jigs or equipment.

In this study, we present a novel control framework for assembly tasks with a soft robot. Typically, existing hard robots require high frequency controllers and precise force/torque sensors for assembly tasks. The resulting robot system is complex, entailing large amounts of engineering and maintenance. Physical softness allows the robot to interact with the environment easily. We expect soft robots to perform assembly tasks without the need for high frequency force/torque controllers and sensors. However, specific data-driven approaches are needed to deal with complex models involving nonlinearity and hysteresis. If we were to apply these approaches directly, we would be required to collect very large amounts of training data. To solve this problem, we argue that by leveraging softness and environmental constraints, a robot can complete tasks in lower dimensional state and action spaces, which could greatly facilitate the exploration of appropriate assembly skills. Then, we apply a highly efficient model-based reinforcement learning method to lower dimensional systems. To verify our method, we perform a simulation for peg-in-hole tasks. The results show that our method learns the appropriate skills faster than an approach that does not consider lower dimensional systems. Moreover, we demonstrate that our method works on a real robot equipped with a compliant module on the wrist.

Physical softness has been proposed to absorb impacts when establishing contact with a robot or its workpiece, to relax control requirements and improve performance in assembly and insertion tasks. Previous work has focused on special end effector solutions for isolated tasks, such as the peg-in-hole task. However, as many robot tasks require the precision of rigid robots, and their performance would degrade when simply adding compliance, it has been difficult to take advantage of physical softness in real applications. A wrist that could switch between soft and rigid modes could solve this problem, but actuators with sufficient strength for this state transition would increase the size and weight of the module and decrease the payload of the robot. To solve this problem, we propose a novel design of a soft module consisting of a cable-driven mechanism, which allows the robot end effector to change between soft and rigid mode while being very compact and light. The module effectively combines the advantages of soft and rigid robots, and can be retrofitted to existing robots and grippers while preserving the characteristics of the robotic system. We evaluate the effectiveness of our proposed design through experiments modeling assembly tasks, and investigate design parameters quantitatively.

Physically soft robots are promising for robotic assembly tasks as they allow stable contacts with the environment. In this study, we propose a novel learning system for soft robotic assembly strategies. We formulate this problem as a reinforcement learning task and design the reward function from human demonstrations. Our key insight is that the failed demonstrations can be used as constraints to avoid failed behaviors. To this end, we developed a teaching device with which humans can intuitively provide various demonstrations. Moreover, we leverage Physically-Consistent Gaussian Mixture Models to clearly assign Gaussian components to the successful and failed trials. We then create the reference trajectories via Gaussian Mixture Regressions, which fit the successful demonstrations while considering the failed ones. Finally, we apply a sample- efficient deep model-based reinforcement learning method to obtain robust strategies with a few interactions. To validate our method, we developed a real-robot experimental system composed of a rigid collaborative robot arm with a compliant wrist and the teaching device. Our results demonstrated that our method learned the assembly strategies with a higher success rate than when using only successful demonstrations.

Although picking up objects a few centimeters in size is a common task, achieving such ability in a robot manipulator remains challenging. We take a step toward solving this problem by focusing on the task of picking a 1.0-cm screw from a bulk bin using only tactile information to achieve the task. Inspired by how humans pick up small objects from a bin, we propose a "grasp-separate" strategy for robotic picking, which involves grasping many objects first and then separating a single object through manipulation in the fingers, for robotic picking. Based on this strategy, we developed a tactile-based screw bin-picking system. We trained a convolution neural network to estimate the number of screws in the fingers first and built a controller that generates manipulation behaviors to separate a screw using reinforcement learning. To compensate for the low resolution of off-the-shelf tactile sensor arrays, we adopted active sensing, which uses observations obtained during a predefined simple movement. We show that this approach enhances the estimation accuracy and manipulation performance. Furthermore, to enable flexible finger motion, such as between the thumb and the index finger in a human hand, we propose a soft robot finger structure that leverages compliant materials. A soft actor-critic algorithm successfully found dexterous screw separation behaviors in compliant soft robotic fingers. In the evaluation, the system obtained an average success rate of 80%, which was difficult to achieve without the grasp-separate manipulation technique.

We propose a novel image feature for textureless object detection. The feature is based on quantized gradient orientations those have been shown to be robust to cluttered backgrounds and illumination changes. We make this feature robust to the appearance changes of a targeted object itself induced by its transformations and small deformations. In our proposed method, we add small random values to the similarity transformation parameters and synthesize many model images. Then quantized orientations are extracted on these images and the orientations are cumulated at each pixel. The frequencies of selected features are utilized as weights when calculating scores. Our proposed feature is evaluated on publicly available dataset and achieve top-class performance both in speed and detection accuracy compared to state-of-the-art techniques.

Detecting and registering nonrigid surfaces are two important research problems for computer vision. Much work has been done with the assumption that there exists only one instance in the image. In this work, we propose an algorithm that detects and registers multiple nonrigid instances of given objects in a cluttered image. Specifically, after we use low level feature points to obtain the initial matches between templates and the input image, a novel high-order affinity graph is constructed to model the consistency of local topology. A hierarchical clustering approach is then used to locate the nonrigid surfaces. To remove the outliers in the cluster, we propose a deterministic annealing approach based on the Thin Plate Spline (TPS) model. The proposed method achieves high accuracy even when the number of outliers is nineteen times larger than the inliers. As the matches may appear sparsely in each instance, we propose a TPS based match growing approach to propagate the matches. Finally, an approach that fuses feature and appearance information is proposed to register each nonrigid surface. Extensive experiments and evaluations demonstrate that the proposed algorithm achieves promising results in detecting and registering multiple non-rigid surfaces in a cluttered scene.

There is a large number of possible facial attributes such as hairstyle, with/without glasses, with/without mustache, etc. Considering large number of facial attributes and their combinations, it is difficult to build attributes classifiers for all possible combinations needed in various applications, especially at the designing stage. To tackle this important and challenging problem, we propose a novel efficient facial attributes recognition algorithm using a learned spatial codebook. The Maximum Entropy and Maximum Orthogonality (MEMO) criterion is followed to learn the spatial codebook. With a spatial codebook constructed at the designing stage, attribute classifiers can be trained on demand with a small number of exemplars with high accuracy on the testing data. Meanwhile, up to 600 times speedup is achieved in the on-demand training process, compared to current state-of-the-art method. The effectiveness of the proposed method is supported by convincing experimental results.

Linear discrimination analysis (LDA) is one of the most popular feature extraction and classifier design techniques. It maximizes the Fisher-ratio between between-class scatter matrix and within-class scatter matrix under a linear transformation, and the transformation is composed of the generalized eigenvectors of them. However, Fisher criterion itself can not decide the optimum norm of transformation vectors for classification. In this paper, we show that actually the norm of the transformation vectors has strong influence on classification performance, and we propose a novel method to estimate the optimum norm of LDA under the ranking loss, re-weighting LDA. On artificial data and real databases, the experiments demonstrate the proposed method can effectively improve the performance of LDA classifiers. And the algorithm can also be applied to other LDA variants such as non parametric discriminant analysis (NDA) to improve theirs performance further.

In this paper we propose a domain partitioning RankBoost approach for face recognition. This method uses Local Gabor Binary Pattern Histogram (LGBPH) features for face representation, and adopts RankBoost to select the most discriminative features. Unlike the original RankBoost algorithm in Freund et al. (2003), weak hypotheses in our method make their predictions based on a partitioning of the similarity domain. Since the domain partitioning approach handles the loss function of a ranking problem directly, it can achieve a higher convergence speed than the original approach. Furthermore, in order to improve the algorithm's generalization ability, we introduce some constraints to the weak classifiers being searched. Experiment results on FERET database show the effectiveness of our approach.

Nowadays mobile devices maintain a lot of private information such as payment information, personal photographs and so on. In Japan, people are starting to use cellular phones as means of payment like pre-paid cards. If owners lost these devices and someone else picked it up, what would happen? To solve this problem, password is now widely used. In the case of password, it has to be complicated and long enough from the viewpoint of security, however it is troublesome for users to type such a long password. If someone set very easy password to avoid the troublesome process, it would be of no use. To solve both security and usability problem simultaneously, various types of biometrics have been proposed. Among many biometric solutions, since nowadays almost all of cellular phones have cameras, we propose face recognition that utilize these cameras. Through the verification process, users have only to take their facial photo using a camera equipped on mobile devices and wait for about one second. The system is effective enough to be used in the mobile devices in terms of performance, usability and hardware requirement.