Legged robot navigation in unstructured and slippery terrains depends heavily on the ability to accurately identify the quality of contact between the robot's feet and the ground. Contact state estimation is regarded as a challenging problem and is typically addressed by exploiting force measurements, joint encoders and/or robot kinematics and dynamics. In contrast to most state of the art approaches, the current work introduces a novel probabilistic method for estimating the contact state based solely on proprioceptive sensing, as it is readily available by Inertial Measurement Units (IMUs) mounted on the robot's end effectors. Capitalizing on the uncertainty of IMU measurements, our method estimates the probability of stable contact. This is accomplished by approximating the multimodal probability density function over a batch of data points for each axis of the IMU with Kernel Density Estimation. The proposed method has been extensively assessed against both real and simulated scenarios on bipedal and quadrupedal robotic platforms such as ATLAS, TALOS and Unitree's GO1.

Visual Simultaneous Localization and Mapping (vSLAM)is the method of employing an optical sensor to map the robot’s observable surroundings while also identifying the robot’s pose in relation to that map. The accuracy and speed of vSLAM calculations can have a very significant impact on the performance and effectiveness of subsequent tasks that need to be executed by the robot, making it a key building component for current robotic designs. The application of vSLAM in the area of humanoid robotics is particularly difficult due to the robot’s unsteady locomotion. This paper introduces a pose graph optimization module based on RGB (ORB) features, as an extension of the KinectFusion pipeline (a well-known vSLAM algorithm), to assist in recovering the robot’s stance during unstable gait patterns when the KinectFusion tracking system fails. We develop and test a wide range of embedded MPSoC FPGA designs, and we investigate numerous architectural improvements, both precise and approximation, to study their impact on performance and accuracy. Extensive design space exploration reveals that properly designed approximations, which exploit domain knowledge and efficient management of CPU and FPGA fabric resources, enable real-time vSLAM at more than 30 fps in humanoid robots with high energy-efficiency and without compromising robot tracking and map construction. This is the first FPGA design to achieve robust, real-time dense SLAM operation targeting specifically humanoid robots. An open source release of our implementations and data can be found in [ 1 ].

Visual Simultaneous Localization and Mapping (vSLAM) is the process of mapping the robot's observed environment using an optical sensor, while concurrently determining the robot's pose with respect to that map. For humanoid robots, the implementation of vSLAM is particularly challenging, due to the intricate motions of the robot. In this work, we present a pose graph optimization module based on RGB features, as an extension on the KinectFusion pipeline (a well-known vSLAM algorithm), to help recover the robot's pose during unstable gait patterns where the KinectFusion tracking system fails. We implement and evaluate a plethora of embedded MPSoC FPGA designs and we explore several architectural optimizations, both precise and approximate, highlighting their effect on performance and accuracy. Properly designed approximations, which exploit domain knowledge and efficient management of CPU and FPGA fabric resources, enable real-time vSLAM (at more than 30 fps) in humanoid robots without compromising robot tracking and map construction. We show that a combination of precise and approximate optimizations and tuning of algorithmic parameters provide a speedup of up to 15.7X and 22.5X compared with the precise FPGA and ARM-only implementations, respectively, without violating the tight accuracy constraints.

In this article, we propose a deep learning frame-work that provides a unified approach to the problem of leg contact detection in humanoid robot walking gaits. Our formulation accomplishes to accurately and robustly estimate the contact state probability for each leg (i.e., stable or slip/no contact). The proposed framework employs solely propriocep-tive sensing and although it relies on simulated ground-truth contact data for the classification process, we demonstrate that it generalizes across varying friction surfaces and different legged robotic platforms and, at the same time, is readily transferred from simulation to practice. The framework is quantitatively and qualitatively assessed in simulation via the use of ground-truth contact data and is contrasted against state-of-the-art methods with an ATLAS, a NAO, and a TALOS humanoid robot. Furthermore, its efficacy is demonstrated in base estimation with a real TALOS humanoid. To reinforce further research endeavors, our implementation is offered as an open-source ROS/Python package, coined Legged Contact Detection (LCD).

In the next few years, new demanding applications will be supported on mobile platforms by reconciling two conflicting requirements: high performance (often with real-time limitations) and low power consumption. The objective of the vipGPU project is to develop hardware and software technology to provide efficient support for two such application scenarios, namely (a) simultaneous localization and mapping (SLAM) in mobile robotics systems, and (b) virtual reality (VR) in portable devices to simulate serious games with emphasis on simulating surgical interventions and medical training in general. In this project, we aim at developing a new heterogeneous platform consisting of hardware accelerators for low power embedded systems optimized (at the hardware and software level) for the implementation of the two applications mentioned above.

In the current paper we investigate the challenges of localizing walking humanoid robots using Visual SLAM (VSLAM). We propose a novel dense RGB-D SLAM framework that seamlessly integrates with the dynamic state of a humanoid, to provide real-time localization and dense mapping of its surroundings. Following the path of recent research in humanoid localization, in the current work we explore the integration between a VSLAM system and the humanoid state, by considering the gait cycle and the feet contacts. We analyze how these effects undermine the quality of data acquisition and association for VSLAM, by capturing the unilateral ground forces at the robot’s feet, and design a system that mitigates their impact.We evaluate our framework on both open and closed-loop bipedal gaits, using a low-cost humanoid platform, and demonstrate that it outperforms kinematic odometry and state-of-the-art dense RGB-D VSLAM methods, by continuously localizing the robot, even in the face of highly irregular and unstable motions.

Contact detection is an important topic in contemporary humanoid robotic research. Up to date control and state estimation schemes readily assume that feet contact status is known in advance. In this work, we elaborate on a broader question: in which gait phase is the robot currently in? We introduce an unsupervised learning framework for gait phase estimation based solely on proprioceptive sensing, namely joint encoder, inertial measurement unit and force/torque data. Initially, a meaningful physical explanation on data acquisition is presented. Subsequently, dimensionality reduction is performed to obtain a compact low-dimensional feature representation followed by clustering into three groups, one for each gait phase. The proposed framework is qualitatively and quantitatively assessed in simulation with ground-truth data of uneven/rough terrain walking gaits and insights about the latent gait phase dynamics are drawn. Additionally, its efficacy and robustness is demonstrated when incorporated in leg odometry computation. Since our implementation is based on sensing that is commonly available on humanoids today, we release an open-source ROS/Python package to reinforce further research endeavors.

Contemporary humanoids are equipped with visual and LiDAR sensors that are effectively utilized for Visual Odometry (VO) and LiDAR Odometry (LO). Unfortunately, such measurements commonly suffer from outliers in a dynamic environment, since frequently it is assumed that only the robot is in motion and the world is static. To this end, robust state estimation schemes are mandatory in order for humanoids to symbiotically co-exist with humans in their daily dynamic environments. In this article, the robust Gaussian Error-State Kalman Filter for humanoid robot locomotion is presented. The introduced method automatically detects and rejects outliers without relying on any prior knowledge on measurement distributions or finely tuned thresholds. Subsequently, the proposed method is quantitatively and qualitatively assessed in realistic conditions with the full-size humanoid robot WALK-MAN v2.0 and the mini-size humanoid robot NAO to demonstrate its accuracy and robustness when outlier VOLO measurements are present. Finally, in order to reinforce further research endeavours, our implementation is released as an open-source ROS/C++package.

Learning from Demonstration (LfD) is addressed in this work in order to establish a novel framework for Human-Robot Collaborative (HRC) task execution. In this context, a robotic system is trained to perform various actions by observing a human demonstrator. We formulate a latent representation of observed behaviors and associate this representation with the corresponding one for target robotic behaviors. Effectively, a mapping of observed to performed actions is defined, that abstracts action variations and differences between the human and robotic manipulators, and facilitates execution of newly-observed actions. The learned action-behaviors are then employed to accomplish task execution in an HRC scenario. Experimental results obtained regard the successful training of a robotic arm with various action behaviors and its subsequent deployment in HRC task accomplishment. The latter demonstrate the validity and efficacy of the proposed approach in human-robot collaborative setups.

This article presents a complete formulation of the challenging task of stable humanoid robot omnidirectional walk based on the Cart and Table model for approximating the robot dynamics. For the control task, we propose two novel approaches: preview control augmented with the inverse system for negotiating strong disturbances and uneven terrain and linear model-predictive control approximated by an orthonormal basis for computational efficiency coupled with constraints for improved stability. For the generation of smooth feet trajectories, we present a new approach based on rigid body interpolation, enhanced by adaptive step correction. Finally, we present a sensor fusion approach for sensor-based state estimation and an effective solution to sensors' noise, delay, and bias issues, as well as to errors induced by the simplified dynamics and actuation imperfections. Our formulation is applied on a real NAO humanoid robot, where it achieves real-time onboard execution and yields smooth and stable gaits.