Transparent objects are ubiquitous in industry, pharmaceuticals, and households. Grasping and manipulating these objects is a significant challenge for robots. Existing methods have difficulty reconstructing complete depth maps for challenging transparent objects, leaving holes in the depth reconstruction. Recent work has shown neural radiance fields (NeRFs) work well for depth perception in scenes with transparent objects, and these depth maps can be used to grasp transparent objects with high accuracy. NeRF-based depth reconstruction can still struggle with especially challenging transparent objects and lighting conditions. In this work, we propose Residual-NeRF, a method to improve depth perception and training speed for transparent objects. Robots often operate in the same area, such as a kitchen. By first learning a background NeRF of the scene without transparent objects to be manipulated, we reduce the ambiguity faced by learning the changes with the new object. We propose training two additional networks: a residual NeRF learns to infer residual RGB values and densities, and a Mixnet learns how to combine background and residual NeRFs. We contribute synthetic and real experiments that suggest Residual-NeRF improves depth perception of transparent objects. The results on synthetic data suggest Residual-NeRF outperforms the baselines with a 46.1 % lower RMSE and a 29.5 % lower MAE. Real-world qualitative experiments suggest Residual-NeRF leads to more robust depth maps with less noise and fewer holes. Website: https://residual-nerf.github.io

This paper explores a novel approach to dexterous manipulation, aimed at levels of speed, precision, robustness, and simplicity suitable for practical deployment. The enabling technology is a Direct-drive Hand (DDHand) comprising two fingers, two DOFs each, that exhibit high speed and a light touch. The test application is the dexterous manipulation of three small and irregular parts, moving them to a grasp suitable for a subsequent assembly operation, regardless of initial presentation. We employed four primitive behaviors that use ground contact as a “third finger”, prior to or during the grasp process: pushing, pivoting, toppling, and squeeze-grasping. In our experiments, each part was presented from 30 to 90 times randomly positioned in each stable pose. Success rates varied from 83% to 100%. The time to manipulate and grasp was 6.32 seconds on average, varying from 2.07 to 16 seconds. In some cases, performance was robust, precise, and fast enough for practical applications, but in other cases, pose uncertainty required time-consuming vision and arm motions. The paper concludes with a discussion of further improvements required to make the primitives robust, eliminate uncertainty, and reduce this dependence on vision and arm motion.