Abstract Full‐body avatar presence is important for immersive social and environmental interactions in digital reality. However, current devices only provide three six degrees of freedom (DOF) poses from the headset and two controllers (i.e. three‐point trackers). Because it is a highly under‐constrained problem, inferring full‐body pose from these inputs is challenging, especially when supporting the full range of body proportions and use cases represented by the general population. In this paper, we propose a deep learning framework, DivaTrack, which outperforms existing methods when applied to diverse body sizes and activities. We augment the sparse three‐point inputs with linear accelerations from Inertial Measurement Units (IMU) to improve foot contact prediction. We then condition the otherwise ambiguous lower‐body pose with the predictions of foot contact and upper‐body pose in a two‐stage model. We further stabilize the inferred full‐body pose in a wide range of configurations by learning to blend predictions that are computed in two reference frames, each of which is designed for different types of motions. We demonstrate the effectiveness of our design on a large dataset that captures 22 subjects performing challenging locomotion for three‐point tracking, including lunges, hula‐hooping, and sitting. As shown in a live demo using the Meta VR headset and Xsens IMUs, our method runs in real‐time while accurately tracking a user's motion when they perform a diverse set of movements.

For many types of robots, avoiding obstacles is necessary to prevent damage to the robot and environment. As a result, obstacle avoidance has historically been an important problem in robot path planning and control. Soft robots represent a paradigm shift with respect to obstacle avoidance because their low mass and compliant bodies can make collisions with obstacles inherently safe. Here we consider the benefits of intentional obstacle collisions for soft robot navigation. We develop and experimentally verify a model of robot-obstacle interaction for a tip-extending soft robot. Building on the obstacle interaction model, we develop an algorithm to determine the path of a growing robot that takes into account obstacle collisions. We find that obstacle collisions can be beneficial for open-loop navigation of growing robots because the obstacles passively steer the robot, both reducing the uncertainty of the location of the robot and directing the robot to targets that do not lie on a straight path from the starting point. Our work shows that for a robot with predictable and safe interactions with obstacles, target locations in a cluttered, mapped environment can be reached reliably by simply setting the initial trajectory. This has implications for the control and design of robots with minimal active steering.

We describe a new series pneumatic artificial muscle (sPAM) and its application as an actuator for a soft continuum robot. The robot consists of three sPAMs arranged radially round a tubular pneumatic backbone. Analogous to tendons, the sPAMs exert a tension force on the robot's pneumatic backbone, causing bending that is approximately constant curvature. Unlike a traditional tendon driven continuum robot, the robot is entirely soft and contains no hard components, making it safer for human interaction. Models of both the sPAM and soft continuum robot kinematics are presented and experimentally verified. We found a mean position accuracy of 5.5 cm for predicting the end-effector position of a 42 cm long robot with the kinematic model. Finally, closed-loop control is demonstrated using an eye-in-hand visual servo control law which provides a simple interface for operation by a human. The soft continuum robot with closed-loop control was found to have a step-response rise time and settling time of less than two seconds.

Robotic needle steering systems for minimally invasive medical procedures require complementary medical imaging systems to track the needles in real time. Ultrasound is a promising imaging modality because it offers relatively low-cost, real-time imaging of the needle. Previous methods applied vibration to the base of the needle using a voice coil actuator, in order to make the needle visible in power Doppler ultrasound. We propose a new method for needle tip vibration, using electromagnetic actuation of small permanent magnets placed inside the needle to improve needle tip visibility in power Doppler imaging. Robotic needle insertion experiments using artificial tissue and ex vivo porcine liver showed that the electromagnetic tip vibration method can generate a stronger Doppler response compared to the previous base vibration method, resulting in better imaging at greater needle depth in tissue. It also eliminates previous issues with vibration damping along the shaft of the needle.