This study focuses on the robotic capability of ungrasping, or releasing, an object in a grasp from the gripper to the robot's environment. The presented technique enables the delicate release of a grasped object using non-static contacts, allowing for rolling and/or sliding. This dexterous manipulation capability is particularly relevant when ungrasping thin or slender objects, as will be demonstrated with real examples. We initially discuss the establishment of three-dimensional stability during ungrasping manipulation, ensuring robustness. Subsequently, we present a planning and control solution for three-dimensional ungrasping, building upon our previous planar version. A series of experiments across various test scenarios, ranging from precision placement to puzzle tiling, showcase the viability and effectiveness of our approach.

Large language models (LLMs) represent a significant advancement in integrating physical robots with AI-driven systems. In this research, we present a framework that leverages Robotics Decision-Making Models (RDMM) for decision-making in domain-specific contexts, enhancing robotic autonomy. This framework incorporates agent-specific knowledge representation, allowing robots to recall and utilize their capabilities and past experiences for improved decision-making. Unlike other approaches, our method prioritizes real-time, on-device solutions, successfully operating on hardware with as little as 8GB of memory. The framework integrates visual perception models, providing robots with a better understanding of their environment. Additionally, real-time speech recognition capabilities are included, improving the human-robot interaction experience. Experimental results show that the RDMM framework achieves planning accuracy of 93%. Furthermore, we introduce a novel dataset consisting of 27k planning instances and 1.3k annotated text-image samples, specifically curated from real-world robotic tasks in competition scenarios. The framework, benchmarks, datasets, and models developed in this work are publicly available on our project website at https://github.com/shadynasrat/RDMM.

Robotics collaborative drawing involves the inter-action between humans and robots to create of visual art using a variety of tools and materials, serving various functions such as communication, narration, and emotional representation. A creative technique within the human natural drawing process is known as inpainting, which involves reconstructing or editing elements in a drawing. This paper introduces PICaSo (Physical Inpainting on Canvas Solution), a robotic drawing system that enables multiple users to collaboratively create artwork on a canvas by integrating the inpainting process. PICaSo utilizes a fine-tuned text-to-image model to interpret natural language prompts into artistic renderings on canvas. Users guide the process by simple descriptive text and specifying desired drawing placement, empowering the robotic arm to autonomously translate these instructions into physical artworks. Our system’s innovation lies in its effective translation of digital inpainting processes into physical actions. By leveraging our erasing capability that enables selective removal of specific parts on the canvas without impacting neighboring areas, facilitating the creation of sequential drawings. This paper comprehensively outlines the capabilities of the proposed system, explores potential applications across various domains, and addresses technical challenges encountered during its development. Project website: shadynasrat.github.io/PICaSo

Although robotic portrait drawing has been a recurring topic in robotics, most robotic portrait drawing systems have focused on either speed or quality of the drawing due to various technical difficulties in pursuing both goals. In this work, we propose a novel robotic portrait drawing system that uses advanced machine-learning techniques and a variable line width Chinese calligraphy pen to draw a high-quality portrait in a short time. Our approach first detects the human keypoints from the incoming video stream and extracts the dominant human face from the video, and then uses a CycleGAN based algorithm to convert the image style into a black-and-white line drawing. After a number of optimization steps, we use a 6-DOF robotic arm and a calligraphy pen to quickly draw the portrait. The system has been openly demonstrated to the general public at the RoboWorld 2022 exhibition, where the system has drawn portraits of more than 40 visitors with a satisfaction rate of 95%.

Motion planning for high degree of freedom (DOF) robots is not an easy task, and often requires optimization in a high dimensional space. Still, a generic motion planner using a single cost function for optimization may not be optimal over a number of different tasks with various task specific constraints. In this paper, we present a motion planning system that utilizes both easy to communicate human preferences and dimensionality reduction to handle these issues. Joint trajectories with human preference costs are projected into the null space of the task space, which helps make the resulting optimization simpler and more reliable. In addition, we apply the dimensionality reduction for the optimization, which significantly lowers the computational load. The suggested controller has been successfully used in the DARPA Robotics Challenge (DRC) Finals to handle a number of manipulation tasks.

Common design principles for low cost humanoid robots include a low center of mass height and a large support area for increased static stability. However, such principles limit the bipedal mobility of the robot due to the kinematic constraints involved. In this paper, we present an efficient locomotion controller that utilizes automatically calculated heel and toe lift motions to overcome the kinematic constraints. This helps with uneven terrain traversal by providing additional support, and also enables a dynamic heel-strike toe-off gait with a large stride length. We demonstrate the controller in physically realistic simulations, and on the THOR-RD full-sized humanoid robot and DARwIn-OP miniature humanoid robot.

Major rule changes for the RoboCup Humanoid League in 2015 pose significant vision and locomotion challenges for disambiguating similarly colored objects and navigating soft terrain. These significant changes highlight the need for applying general purpose humanoid robotics approaches that can handle abrupt environment modifications, and we utilize the general purpose THOR (Tactical Hazardous Operations Robot) series of robot from the recent DARPA Robotics Challenge (DRC). Specific techniques for vision, kicking and autonomy complement software developed for robust deployments in the DRC. In this paper, we present these soccer playing techniques, which were validated in the Humanoid AdultSize league in Hefei. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Developing a reliable humanoid robot that operates in uncharted real-world environments is a huge challenge for both hardware and software. Commensurate with the technology hurdles, the amount of time and money required can also be prohibitive barriers. This paper describes Team THOR's approach to overcoming such barriers for the 2013 DARPA Robotics Challenge (DRC) Trials. We focused on forming modular components - in both hardware and software - to allow for efficient and cost effective parallel development. The robotic hardware consists of standardized and general purpose actuators and structural components. These allowed us to successfully build the robot from scratch in a very short development period, modify configurations easily and perform quick field repair. Our modular software framework consists of a hybrid locomotion controller, a hierarchical arm controller and a platform-independent operator interface. These modules helped us to keep up with hardware changes easily and to have multiple control options to suit various situations. We validated our approach at the DRC Trials where we fared very well against robots many times more expensive.

Bipedal humanoid robots will fall under unforeseen perturbations without active stabilization. Humans use dynamic full body behaviors in response to perturbations, and recent bipedal robot controllers for balancing are based upon human biomechanical responses. However these controllers rely on simplified physical models and accurate state information, making them less effective on physical robots in uncertain environments. In our previous work, we have proposed a hierarchical control architecture that learns from repeated trials to switch between low-level biomechanically-motivated strategies in response to perturbations. However in practice, it is hard to learn a complex strategy from limited number of trials available with physical robots. In this work, we focus on the very problem of efficiently learning the high-level push recovery strategy, using simulated models of the robot with different levels of abstraction, and finally the physical robot. From the state trajectory information generated using different models and a physical robot, we find a common low dimensional strategy for high level push recovery, which can be effectively learned in an online fashion from a small number of experimental trials on a physical robot. This learning approach is evaluated in physics-based simulations as well as on a small humanoid robot. Our results demonstrate how well this method stabilizes the robot during walking and whole body manipulation tasks.

During heavy work, humans utilize whole body motions in order to generate large forces. In extreme cases, exaggerated weight shifts are used to impart large impact forces. There have been approaches to design stable whole body impact motions based on precise dynamic models of the robot and the target object, but they have practical limitations as the uncertainty in the ensuing reaction forces can lead to instability. In the current work, we describe a motion controller for a humanoid robot that generates impacts at an end effector while keeping the robot body balanced before and after the impact. Instead of relying on the accuracy of the impact dynamics model, we use a simplified model of the robot and biomechanically motivated push recovery controllers to reactively stabilize the robot against unknown perturbations from the impact. We demonstrate our approach in physically realistic simulations, as well as experimentally on a small humanoid robot platform.

Dynamic bipedal walking is susceptible to external disturbances and surface irregularities, requiring robust feedback control to remain stable. In this work, we present a practical hierarchical push recovery strategy that can be readily implemented on a wide range of humanoid robots. Our method consists of low level controllers that perform simple, biomechanically motivated push recovery actions and a high level controller that combines the low level controllers according to proprioceptive and inertial sensory signals and the current robot state. Reinforcement learning is used to optimize the parameters of the controllers in order to maximize the stability of the robot over a broad range of external disturbances. The controllers are learned on a physical simulation and implemented on the Darwin-HP humanoid robot platform, and the resulting experiments demonstrate effective full body push recovery behaviors during dynamic walking.

Bipedal walking in human environments is made difficult by the unevenness of the terrain and by external disturbances. Most approaches to bipedal walking in such environments either rely upon a precise model of the surface or special hardware designed for uneven terrain. In this paper, we present an alternative approach to stabilize the walking of an inexpensive, commercially-available, position-controlled humanoid robot in difficult environments. We use electrically compliant swing foot dynamics and onboard sensors to estimate the inclination of the local surface, and use a online learning algorithm to learn an adaptive surface model. Perturbations due to external disturbances or model errors are rejected by a hierarchical push recovery controller, which modulates three biomechanically motivated push recovery controllers according to the current estimated state. We use a physically realistic simulation with an articulated robot model and reinforcement learning algorithm to train the push recovery controller, and implement the learned controller on a commercial DARwIn-OP small humanoid robot. Experimental results show that this combined approach enables the robot to walk over unknown, uneven surfaces without falling down.

We present a novel method to control a biped humanoid robot to walk on unknown inclined terrains, using an online learning algorithm to estimate in real-time the local terrain from proprioceptive and inertial sensors. Compliant controllers for the ankle joints are used to actively probe the surrounding surface, and the measured sensor data are combined to explicitly learn the global inclination and local disturbances of the terrain. These estimates are then used to adaptively modify the robot locomotion and control parameters. Results from both a physically-realistic computer simulation and experiments on a commercially available small humanoid robot show that our method can rapidly adapt to changing surface conditions to ensure stable walking on uneven surfaces.

We consider the problem of finding sufficiently simple models of high-dimensional physical systems that are consistent with observed trajectories, and using these models to synthesize new trajectories. Our approach models physical trajectories as least-time trajectories realized by free particles moving along the geodesics of a curved manifold, reminiscent of the way light rays obey Fermat's principle of least time. Finding these trajectories, unfortunately, requires finding a minimum-cost path in a high-dimensional space, which is generally a computationally intractable problem. In this work we show that this high-dimensional planning problem can often be solved nearly optimally in practice via deterministic search, as long as we can find a certain low-dimensional structure in the Lagrangian that describes our observed trajectories. This low-dimensional structure additionally makes it feasible to learn an estimate of a Lagrangian that is consistent with the observed trajectories, thus allowing us to present a complete approach for learning from and predicting high-dimensional physical motion sequences. We finally show experimental results applying our method to human motion and robotic walking gaits. In doing so, we furthermore demonstrate efficient path planning in a 990-dimensional space.