Robot models, particularly those trained with large amounts of data, have recently shown a plethora of real-world manipulation and navigation capabilities. Several independent efforts have shown that given sufficient training data in an environment, robot policies can generalize to demonstrated variations in that environment. However, needing to finetune robot models to every new environment stands in stark contrast to models in language or vision that can be deployed zero-shot for open-world problems. In this work, we present Robot Utility Models (RUMs), a framework for training and deploying zero-shot robot policies that can directly generalize to new environments without any finetuning. To create RUMs efficiently, we develop new tools to quickly collect data for mobile manipulation tasks, integrate such data into a policy with multi-modal imitation learning, and deploy policies ondevice on the Hello Robot Stretch, a cheap commodity robot, with an external mLLM verifier for retrying. We train five such utility models for opening cabinet doors, opening drawers, picking up napkins, picking up paper bags, and reorienting fallen objects. Our system, on average, achieves 90% success rate in unseen, novel environments interacting with unseen objects. Primary among our lessons are the importance of training data over training algorithm and policy class, guidance about data scaling, necessity for diverse yet high-quality demonstrations, and a recipe for robot introspection and retrying to improve performance on individual environments.

Mobile manipulators for indoor human environments can serve as versatile devices that perform a variety of tasks, yet adoption of this technology has been limited. Reducing size, weight, and cost could facilitate adoption, but risks restricting capabilities. We present a novel design that reduces size, weight, and cost, while supporting a variety of tasks. The core design consists of a two-wheeled differential-drive mobile base, a lift, and a telescoping arm configured to achieve Cartesian motion at the end of the arm. Design extensions include a 1 degree-of-freedom (DOF) wrist to stow a tool, a 2-DOF dexterous wrist to pitch and roll a tool, and a compliant gripper. We justify our design with anthropometry and mathematical models of static stability. We also provide empirical support from teleoperating and autonomously controlling a commercial robot based on our design (the Stretch RE1 from Hello Robot Inc.) to perform tasks in real homes.

The demands of mobile manipulation are leading to a new class of multi-fingered hands with a premium on being lightweight and robust as well as being able to grasp and perform basic manipulations with a wide range of objects. A promising approach to addressing these goals is to use compliant, underactuated hands with selectively lockable degrees of freedom. This paper presents the design of one such hand that combines series-elastic actuation and electrostatic braking at the joints. A numerical analysis shows how the maximum pullout force varies as a function of kinematic parameters, spring forces at the joints and brake torques.

A persistent-if less glamorous—challenge in hospitals lies in the day-to-day work of moving and lifting patients with impaired mobility. This is a challenge intensified by our burgeoning aging population, the obesity epidemic, and our aging healthcare workforce. During manual patient handling, the predominant risk of staff injury is excessive back and shoulder loading. A mobile robotic nurse assistant (RoNA) is highly desired to enhance the efficacy and quality of care that nurses and their paraprofessional staff can provide. Such an assistant could improve a nurse's working conditions by off-loading some of his or her most physically demanding duties, thereby reducing the potential for self-injury or injury to the patient. Hstar Technologies is developing a revolutionary RoNA system that provides physical assistance to nurses in a hospital ward. The design of RoNA is a safe and robust system that works effectively in a hospital environment under direct and telepresence control by a nurse or physician. RoNA has a humanoid design featuring bimanual dexterous manipulators that employ a series-elastic-actuation (SEA) system. These electric actuators provide manipulator compliance, safety, flexibility and the strength to lift patients weighing up to 300lbs. RoNA also features an innovative humanoid upper torso, a unique mobile platform with holonomic drive and posture stability enhancement, intelligent navigation control with 3D sensing and perception capability, an intuitive and innovative human-robot interaction control interface, and a highly integrated plan for healthcare system assembly. We anticipate that robotic maneuvering assistants would increase job satisfaction, reduce lifting-related injuries, and extend the years of effective service nurses could render in hospitals. These effects would reduce hospital costs and ameliorate problems posed by the shortage of nursing staff.

For manipulation tasks, the transfer of objects between humans and robots is a fundamental way to coordinate activity and cooperatively perform useful work. Within this paper we demonstrate that robots and people can effectively and intuitively work together by directly handing objects to one another. First, we present experimental results that demonstrate that subjects without explicit instructions or robotics expertise can successfully hand objects to a robot and take objects from a robot in response to reaching gestures. Moreover, when handing an object to the robot, subjects control the object's position and orientation to match the configuration of the robot's hand, thereby simplifying robotic grasping and offering opportunities to simplify the manipulation task. Second, we present a robotic application that relies on this form of human-robot interaction. This application enables a humanoid robot to help a user place objects on a shelf, perform bimanual insertion tasks, and hold a box within which the user can place objects. By handing appropriate objects to the robot, the human directly and intuitively controls the robot. Through this interaction, the human and robot complement one another's abilities and work together to achieve results.

Ballard (1991) described the implications of having a visual system that could actively position the camera coordinates in response to physical stimuli. In humanoid robotic systems, or in any animate vision system that interacts with people, social dynamics provide additional levels of constraint and additional opportunities for processing economy. In this paper, we describe an integrated visual-motor system that was implemented on a humanoid robot to negotiate the robot's physical constraints, the perceptual needs of the robot's behavioral and motivational systems, and the social implications of the motor acts.