Existing robotic musicians utilize detailed handcrafted instrument models to generate or learn policies for playing because model-free or inaccurate policy rollouts might easily damage or wear out fragile instruments. We introduce an approach to characterize geometric models of chordophones and their audio onset responses directly through audio-tactile exploration with a physical robot arm. Initially, the system refines prior estimates of string positions, provided by kinesthetic teaching or visual estimation, through repeated attempts to pluck individual strings. A subsequent stage implements a Safe Active Exploration paradigm based on Gaussian Processes to explore and characterize the audio onset response of feasible plucking motions while minimizing invalid attempts. The resulting models can be used to actuate an imprecise robotic arm to play sequences of notes with varying loudness on a Chinese Guzheng.

Markerless vision-based teleoperation that leverages innovations in computer vision offers the advantages of allowing natural and noninvasive finger motions for multifingered robot hands. However, current pose estimation methods still face inaccuracy issues due to the self-occlusion of the fingers. Herein, we develop a novel vision-based hand-arm teleoperation system that captures the human hands from the best viewpoint and at a suitable distance. This teleoperation system consists of an end-to-end hand pose regression network and a controlled active vision system. The end-to-end pose regression network (Transteleop), combined with an auxiliary reconstruction loss function, captures the human hand through a low-cost depth camera and predicts joint commands of the robot based on the image-to-image translation method. To obtain the optimal observation of the human hand, an active vision system is implemented by a robot arm at the local site that ensures the high accuracy of the proposed neural network. Human arm motions are simultaneously mapped to the slave robot arm under relative control. Quantitative network evaluation and a variety of complex manipulation tasks, for example, tower building, pouring, and multitable cup stacking, demonstrate the practicality and stability of the proposed teleoperation system.

Accurate estimation of the relative pose between an object and a robot hand is critical for many manipulation tasks. However, most of the existing object-in-hand pose datasets use two-finger grippers and also assume that the object remains fixed in the hand without any relative movements, which is not representative of real-world scenarios. To address this issue, a 6D object-in-hand pose dataset is proposed using a teleoperation method with an anthropomorphic Shadow Dexterous hand. Our dataset comprises RGB-D images, proprioception and tactile data, covering diverse grasping poses, finger contact states, and object occlusions. To overcome the significant hand occlusion and limited tactile sensor contact in real-world scenarios, we propose PoseFusion, a hybrid multi-modal fusion approach that integrates the information from visual and tactile perception channels. PoseFusion generates three candidate object poses from three estimators (tactile only, visual only, and visuo-tactile fusion), which are then filtered by a SelectLSTM network to select the optimal pose, avoiding inferior fusion poses resulting from modality collapse. Extensive experiments demonstrate the robustness and advantages of our framework. All data and codes are available on the project website: https://elevenjiang1.github.io/ObjectlnHand-Dataset/.

Vision-only motion reconstruction from monocular videos often produces artifacts such as foot sliding and jittering. Existing physics-based methods typically either simplify the problem to focus solely on foot-ground contacts, or they reconstruct full-body contacts within a physics simulator, necessitating the solution of a time-consuming bilevel optimization problem. To overcome these limitations, we present an efficient gradient-based method for reconstructing complex human motions (including highly dynamic and acrobatic movements) with physical constraints. Our approach reformulates human motion dynamics through a differentiable physical consistency loss within an augmented search space that accounts both for contacts and camera alignment. This enables us to transform the motion reconstruction task into a single-level trajectory optimization problem. Experimental results demonstrate that our method can reconstruct complex human motions from real-world videos in minutes, which is substantially faster than previous approaches. Additionally, the reconstructed results show enhanced physical realism compared to existing methods.

The key role of tactile sensing for human grasping and manipulation is widely acknowledged, but most industrial robot grippers and even multi-fingered hands are still designed and used without any tactile sensors. While the basic design principles for resistive or capacitive sensors are well known, several factors keep tactile sensing from large-scale deployment — high sensor costs, short lifespan, poor reliability, difficult production processes, a lack of suitable software and tools for system integration, and the unique requirement for tactile sensors to conform to application-specific shapes.In this work, we describe a very simple but efficient approach to design low-cost resistive matrix sensors, where sensor layout and geometry, taxel-size, and measurement sensitivity can be customized over a wide range. Sensor assembly needs nothing more than a hobby cutting plotter for precise cutting of aluminum tape and Velostat foils, as well as adhesive plastic tape. Our electronics combines transimpedance amplifiers with common Arduino microcontrollers, supporting standard communication protocols, and using either cabled or wireless data transfer to the host. We present three different application examples and sketch our ROS software for sensor calibration and visualization. All parts of our project, including detailed building instructions, bill-of-materials, electronics, and firmware are available open-source.

Planar pushing remains a challenging research topic, where building the dynamic model of the interaction is the core issue. Even an accurate analytical dynamic model is inherently unstable because physics parameters such as inertia and friction can only be approximated. Data-driven models usually rely on large amounts of training data, but data collection is time consuming when working with real robots.In this paper, we collect all training data in a physics simulator and build an LSTM-based model to fit the pushing dynamics. Domain Randomization is applied to capture the pushing trajectories of a generalized class of objects. When executed on the real robot, the trained recursive model adapts to the tracked object's real dynamics within a few steps. We propose the algorithm Recurrent Model Predictive Path Integral (RMPPI) as a variation of the original MPPI approach, employing state-dependent recurrent models. As a comparison, we also train a Deep Deterministic Policy Gradient (DDPG) network as a model-free baseline, which is also used as the action generator in the data collection phase. During policy training, Hindsight Experience Replay is used to improve exploration efficiency. Pushing experiments on our UR5 platform demonstrate the model's adaptability and the effectiveness of the proposed framework.

Established indoor robot navigation frameworks build on the separation between global and local planners. Whereas global planners rely on traditional graph search algorithms, local planners are expected to handle driving dynamics and resolve minor conflicts. We present a system to train neural-network policies for such a local planner component, explicitly accounting for humans navigating the space. DRL-agents are trained in randomized virtual 2D environments with simulated human interaction. The trained agents can be deployed as a drop-in replacement for other local planners and significantly improve on traditional implementations. Performance is demonstrated on a MiR-100 transport robot.

In this paper, we present a multimodal mobile teleoperation system that consists of a novel vision-based hand pose regression network (Transteleop) and an IMU (inertial measurement units)-based arm tracking method. Transteleop observes the human hand through a low-cost depth camera and generates not only joint angles but also depth images of paired robot hand poses through an image-to-image translation process. A keypoint-based reconstruction loss explores the resemblance in appearance and anatomy between human and robotic hands and enriches the local features of reconstructed images. A wearable camera holder enables simultaneous hand-arm control and facilitates the mobility of the whole teleoperation system. Network evaluation results on a test dataset and a variety of complex manipulation tasks that go beyond simple pick-and-place operations show the efficiency and stability of our multimodal teleoperation system.

Robust and accurate estimation of liquid height lies as an essential part of pouring tasks for service robots. However, vision-based methods often fail in occluded conditions while audio-based methods cannot work well in a noisy environment. We instead propose a multimodal pouring network (MP-Net) that is able to robustly predict liquid height by conditioning on both audition and haptics input. MP-Net is trained on a self-collected multimodal pouring dataset. This dataset contains 300 robot pouring recordings with audio and force/torque measurements for three types of target containers. We also augment the audio data by inserting robot noise. We evaluated MP-Net on our collected dataset and a wide variety of robot experiments. Both network training results and robot experiments demonstrate that MP-Net is robust against noise and changes to the task and environment. Moreover, we further combine the predicted height and force data to estimate the shape of the target container.

In this paper, we focus on the challenging perception problem in robotic pouring. Most of the existing approaches either leverage visual or haptic information. However, these techniques may suffer from poor generalization performances on opaque containers or concerning measuring precision. To tackle these drawbacks, we propose to make use of audio vibration sensing and design a deep neural network PouringNet to predict the liquid height from the audio fragment during the robotic pouring task. PouringNet is trained on our collected real-world pouring dataset with multimodal sensing data, which contains more than 3000 recordings of audio, force feedback, video and trajectory data of the human hand that performs the pouring task. Each record represents a complete pouring procedure. We conduct several evaluations on PouringNet with our dataset and robotic hardware. The results demonstrate that our PouringNet generalizes well across different liquid containers, positions of the audio receiver, initial liquid heights and types of liquid, and facilitates a more robust and accurate audio-based perception for robotic pouring.

In this paper, a novel and fast memetic evolutionary algorithm is presented which can solve fully constrained generic inverse kinematics with multiple end effectors and goal objectives, leaving high flexibility for the design of custom cost functions. The algorithm utilizes a hybridization of evolutionary and swarm optimization, combined with the limited-memory-Broyden-Fletcher-Goldfarb-Shanno with bound constraints algorithm for gradient-based optimization. Accurate solutions can be found in real-time and suboptimal extrema are robustly avoided, scaling well even for greatly higher degree of freedom. The algorithm provides a general framework for bounded continuous optimization which only requires two parameters for the number of individuals and elites to be set, and supports adding additional goals and constraints for inverse kinematics, such as minimal displacement between solutions, collision avoidance, or functional joint relations. Experimental results on several industrial and anthropomorphic robots as well as on virtual characters demonstrate the algorithm to be applicable for solving complex kinematic postures for different challenging tasks in robotics, human-robot interaction and character animation, including dexterous object manipulation, collision-free full-body motion, as well as animation post-processing for video games and films. Implementations are made available for Unity3D and robot operating system.

While the problem of inverse kinematics on serial kinematic chains is well researched, solving motion tasks quickly on more complex robots remains an open problem. Examples include dual-arm manipulation, grasping with multi-finger hands, and full-body motion generation for humanoids. In this paper, we introduce an open-source software package for ROS and MoveIt! that solves inverse kinematics and motion tasks on robots with arbitrary kinematic trees. The underlying memetic algorithm integrates evolutionary optimization, particle swarm optimization, and gradient methods. The optimization respects joint limits, effectively avoids local minima, and achieves fast convergence to accurate solutions. More importantly, the overall motion goal is specified using a set of weighted sub-goals, providing great flexibility and control of secondary objectives. Several application examples demonstrate how to combine the predefined sub-goals to achieve complex motion tasks.

Solving kinematic motion is a challenging field of research which is relevant for various applications in character animation and robotics. This paper presents a novel and fast hybrid evolutionary algorithm for inverse kinematics which can handle fully constrained and highly articulated geometries with multiple end effectors and individual objectives. Several experiments on the 42 DoF human body mannequin and other kinematic models demonstrate a robust multimodal and multi-objective optimisation, and the ability to evolve accurate solutions in real-time while offering maximum flexibility for the design of custom cost functions.

While solving inverse kinematics on serial kinematic chains is well researched, many methods still seem rather limited in jointly handling more complex geometries, including dexterous multi-finger hands or humanoid robots. In particular, object manipulation and motion tasks would benefit from the ability to define intermediate goals along the kinematic chains, such as an elbow position or wrist orientation. In this paper, we propose a fast hybrid evolutionary approach that is capable of solving inverse kinematics for multiple end effectors simultaneously, leaving high flexibility for specifying full-body postures with different objectives. Accurate solutions can be found in real-time and suboptimal extrema are robustly avoided. Our experimental results on the NASA Valkyrie and Shadow Dexterous Hand demonstrate that the algorithm is fast and can be efficiently applied for different robotic tasks which require flexible control of fully-constrained geometries.

Dextrous object manipulation with multi-fingered robot hands remains one of the key challenges of service robotics. So far, most theoretical approaches and simulators have concentrated on the search for and evaluation of static stable grasps, but with neither a model of the full hand-arm system nor the system dynamics. GraspIt! is probably the best-known simulator of this kind. In this work we present a simulator that uses the JBullet physics engine to realistically model grasps with multi-fingered hands. It supports manipulation tasks based on a complete arm and hand system, with full calculation of hand and object dynamics. A hybrid dynamics and kinematics approach avoids the oscillations introduced by the different size scale of the arm and hand, so that force-closure grasps are possible in addition to form-closure grasps. The software includes detailed models of our 24-DOF Shadow Dextrous hand and the 6-DOF Mitsubishi PA-10 robot arm. A real-time interface allows us to prepare or to replay and analyze grasp experiments performed on our real robots.

We consider in-hand manipulation tasks that consists of periodic movements. In order to improve the manipulation learning ability of a robot with a human-like hand, this paper introduces a segmentation method based on the techniques of action gist. Action gist is the key motion information in manipulation with the property of semantics. In the techniques of in-hand manipulation action gist, there is a Meta Motion Occurrence Histogram describing the motion information in the demonstration set. This paper proposes an algorithm related to the Meta Motion Occurrence Histogram to maximize the common motions in each segment, so as to figure out the best segmentation solution in the in-hand manipulation sequence. The experiments illustrate the performance of the proposed method, and discuss the possibility of segmentation fusing with the information from tactile sensor.

This paper presents a new adaptive iterative learning rule for binary couplings networks. Unlike previous approaches, the algorithm adapts to pattern correlations during learning and succeeds to store highly correlated patterns. Also, by supplying a set of default stabilities to the learning rule, the recall properties of the network can be adjusted for each pattern. Simulations results of pattern recall in a simple recursive network demonstrate the storage and associative memory properties of the trained network and show the advantage over older learning rules. Note that the adaption step of the learning rule can also be applied to other learning algorithms. Applications to multi-layer networks and hardware implementation are discussed.