Tactile sensing plays a pivotal role in human perception and manipulation tasks, allowing us to intuitively understand task dynamics and adapt our actions in real time. Transferring such tactile intelligence to robotic systems would help intelligent agents understand task constraints and accurately interpret the dynamics of both the objects they are interacting with and their own operations. While significant progress has been made in imbuing robots with this tactile intelligence, challenges persist in effectively utilizing tactile information due to the diversity of tactile sensor form factors, manipulation tasks, and learning objectives involved. To address this challenge, we present a unified tactile embedding space capable of predicting a variety of task-centric qualities over multiple manipulation tasks. We collect tactile data from human demonstrations across various tasks and leverage this data to construct a shared latent space for task stage classification, object dynamics estimation, and tactile dynamics prediction. Through experiments and ablation studies, we demonstrate the effectiveness of our shared tactile latent space for more accurate and adaptable tactile networks, showing an improvement of up to 84% over the single-task training.

Human-in-the-loop robot teleoperation interfaces enable operators to control robots to complete complex tasks, as seen by the success of teams in the DARPA Robotics Challenge (DRC). In this work, we compare two human-in-the-loop planning interfaces, a 2D keyboard and mouse (KBM) interface modeled after those used in the DRC and a 3D virtual reality (VR) interface, for teleoperating a robot to perform navigation and manipulation tasks. In our study, we investigated operator performance, and cognitive workload while using the interface, as well as the perceived usability of each. We found that participants had better performance in both task types when using the KBM interface, however they experienced fewer collisions between the robot and the world in the VR interface. Given these findings, we recommend utilizing a KBM interface in low-risk situations where task performance is the primary factor. In high-risk scenarios, where collisions can be detrimental, we recommend using VR. With this work we aim to contribute to building effective and intuitive interfaces for human-in-the-loop planning to allow robots to complete complex tasks in challenging environments.

Extreme environments, such as search and rescue missions, defusing bombs, or exploring extraterrestrial planets, are unsafe environments for humans to be in. Robots enable humans to explore and interact in these environments through remote presence and teleoperation and virtual reality provides a medium to create immersive and easy-to-use teleoperation interfaces. However, current virtual reality interfaces are still very limited in their capabilities. In this work, we aim to advance robot teleoperation virtual reality interfaces by developing an environment reconstruction methodology capable of recognizing objects in a robot’s environment and rendering high fidelity models inside a virtual reality headset. We compare our proposed environment reconstruction method against traditional point cloud streaming by having operators plan waypoint trajectories to accomplish a pick-and-place task. Overall, our results show that users find our environment reconstruction method more usable and less cognitive work compared to raw point cloud streaming.

This paper introduces an anytime synthesized motion planning algorithm for humanoid robots unifying locomotion and manipulation planning. It generates an entire set of motions to finish specific tasks in an environment containing obstacles by exploiting a powerful inverse kinematics (IK) engine. The IK engine can compute solutions allowing the robot to reposition its feet for meeting the task requirements. The presented planning algorithm has two primary beneficial capabilities. First, it is capable of generating a motion plan to complete a task handling multiple ordered or unordered actions. Second, it produces an initial solution very quickly, and then searches for the opportunity to improve the the solution during execution. The performance of the proposed algorithm is evaluated on the NASA-JSC Valkyrie humanoid robot by demonstrating an object pick up task in simulation and a box pick-and-place task in the real world.