The integration of satellite communication into mobile devices represents a paradigm shift in connectivity, yet the performance characteristics under motion and environmental occlusion remain poorly understood. We present the Starlink Robot, the first mobile robotic platform equipped with Starlink satellite internet, comprehensive sensor suite including upward-facing camera, LiDAR, and IMU, designed to systematically study satellite communication performance during movement. Our multi-modal dataset captures synchronized communication metrics, motion dynamics, sky visibility, and 3D environmental context across diverse scenarios including steady-state motion, variable speeds, and different occlusion conditions. This platform and dataset enable researchers to develop motion-aware communication protocols, predict connectivity disruptions, and optimize satellite communication for emerging mobile applications from smartphones to autonomous vehicles. In this work, we use LEOViz for real-time data collection and visualization. The project is available at https://starlinkrobot.github.io.

Achieving adaptive, multi-skilled, and energy-efficient locomotion is vital for advancing the operation of autonomous quadrupedal systems. This study presents eGAIT, a unified multi-skilled policy enabling energy-efficient and stable gait transitions across nine non-monotonic, velocity-optimized gaits, in response to dynamic velocity commands. The framework leverages a hybrid control architecture that integrates model-based and learning-based methods to address the entire locomotion pipeline. An MPC-based gait generator produces velocity-optimized trajectories, which are imitated through Proximal Policy Optimization (PPO), driven by a Adversarial Motion Prior (AMP) style reward to train distinct policies for specific velocity ranges. These policies are unified through a Hierarchical Reinforcement Learning (HRL) framework featuring a novel modified Deep Q-Network (eDQN) for real-time velocity-to-policy mapping. Training efficiency is enhanced by an auxiliary selector layer that guides velocity-policy mapping, while a sparsely activated stability reward mechanism ensures smooth gait transitions by incorporating geometric and rotational stability. Extensively validated in simulation and on a Unitree Go1 robot, eGAIT achieves a 100% success rate in velocity-to-policy mapping, a 35% improvement in energy efficiency, a 31% improvement in both velocity tracking and stability compared to the next best state-of-the-art method. This work advances autonomous quadrupedal locomotion, enabling longer, more efficient, and stable operations in dynamic environments. Supplementary materials and visualizations related to the paper can be found at: https://github.com/RPL-CS-UCL/egait/.

The increasing deployment of mobile robots in dynamic outdoor environments necessitates robotic systems capable of maintaining reliability amidst fluctuating wireless connectivity. While the Robot Operating System (ROS) has established itself as the de facto standard for such networked robotics, its abstraction of communication as an opaque, besteffort utility creates a critical bottleneck: it fails to leverage physical layer (PHY) information, resulting in degraded performance and unreliable execution in fluctuating networks. To address this, this paper presents MobileROS, a wireless-native robot operating system that transforms wireless communication from an external service into a core system resource. Grounded in the Symbiotic Paradigm, MobileROS establishes a bidirectional exchange where network conditions inform robotic decisions and mission requirements guide network resource allocation. Based on service mesh principles and domain-driven design, our architecture implements a Hub-Engines-Cells (HEC) model. It features a central Hub for global optimization, three specialized engines (the Radio Information Engine, the Cross Domain Engine, and the Physical Adaptive Engine) for crosslayer intelligence, and distributed Cells as functional units. A key mechanism, Application-Driven Bidirectional Dynamic Slicing, allows robots to actively reconfigure network resources based on semantic urgency, transforming the robot from a passive observer into an active network controller. We systematically evaluate MobileROS across three cities (London, Hong Kong, and Shenzhen) in five scenarios: distributed visual SLAM, cross-domain LiDAR perception, V2X autonomous driving, hybrid multi-robot collaboration against WebRTC baselines, and partition recovery validating CAP-theorem-aware failsafe mechanisms. Results demonstrate that MobileROS maintains significantly more stable performance than standard ROS in mobile wireless deployments.We provide implementation details athttps://github.com/MobileROS.

Decision Transformers (DTs) effectively capture long-range dependencies using self-attention but struggle with fine-grained local relationships, especially the Markovian properties in many offline-RL datasets. Conversely, Decision Convformer (DC) utilizes convolutional filters for capturing local patterns but shows limitations in tasks demanding long-term dependencies, such as Maze2d. To address these limitations and leverage both strengths, we propose the Long-Short Decision Transformer (LSDT), a general-purpose architecture to effectively capture global and local dependencies across two specialized parallel branches (self-attention and convolution). We explore how these branches complement each other by modeling various ranged dependencies across different environments, and compare it against other baselines. Experimental results demonstrate our LSDT achieves state-of-the-art performance and notable gains over the standard DT in D4RL offline RL benchmark. Leveraging the parallel architecture, LSDT performs consistently on diverse datasets, including Markovian and non-Markovian. We also demonstrate the flexibility of LSDT's architecture, where its specialized branches can be replaced or integrated into models like DC to improve their performance in capturing diverse dependencies. Finally, we also highlight the role of goal states in improving decision-making for goal-reaching tasks like Antmaze.

Understanding the traversability of terrain is essential for autonomous robot navigation, particularly in unstructured environments such as natural landscapes. Although traditional methods, such as occupancy mapping, provide a basic framework, they often fail to account for the complex mobility capabilities of some platforms such as legged robots. In this work, we propose a method for estimating terrain traversability by learning from demonstrations of human walking. Our approach leverages dense, pixel-wise feature embeddings generated using the DINOv2 vision Transformer model, which are processed through an encoder-decoder MLP architecture to analyze terrain segments. The averaged feature vectors, extracted from the masked regions of interest, are used to train the model in a reconstruction-based framework. By minimizing reconstruction loss, the network distinguishes between familiar terrain with a low reconstruction error and unfamiliar or hazardous terrain with a higher reconstruction error. This approach facilitates the detection of anomalies, allowing a legged robot to navigate more effectively through challenging terrain. We run real-world experiments on the ANYmal legged robot both indoor and outdoor to prove our proposed method. The code is open-source, while video demonstrations can be found on our website: https://rpl-cs-ucl.github.io/STEPP/

Visual localization involves estimating a query image's 6-DoF (degrees of freedom) camera pose, which is a fundamental component in various computer vision and robotic tasks. This paper presents LoGS, a vision-based localization pipeline utilizing the 3D Gaussian Splatting (GS) technique as scene representation. This novel representation allows high-quality novel view synthesis. During the mapping phase, structure-from-motion (SfM) is applied first, followed by the generation of a GS map. During localization, the initial position is obtained through image retrieval, local feature matching coupled with a PnP solver, and then a high-precision pose is achieved through the analysis-bysynthesis manner on the GS map. Experimental results on four large-scale datasets demonstrate the proposed approach's SoTA accuracy in estimating camera poses and robustness under challenging few-shot conditions. Codes can be found at: https://github.com/RPL-CS-UCL/gs_localization.

This article focuses on predicting how an object can be transformed to a semantically meaningful pose relative to another object, given only one or few examples. Current pose correspondence methods rely on vast 3D object datasets and do not actively consider semantic information, which limits the objects to which they can be applied. We present a novel method for learning cross-object pose correspondence. The proposed method detects interacting object parts, performs one-shot part correspondence, and uses geometric and visual-semantic features. Given one example of two objects posed relative to each other, the model can learn how to transfer the demonstrated relations to unseen object instances. Supplementary details can be found at https://sites.google.com/view/semantic-pose-correspondence

This paper presents Lite VLoc, a hierarchical vi-sual localization framework that uses a lightweight topo-metric map to represent the environment. The method consists of three sequential modules that estimate camera poses in a coarse-to-fine manner. Unlike dense 3D mapping methods, LiteVLoc reduces storage by avoiding geometric reconstruction. It uses a learning-based feature matcher to establish dense correspondences between sparse keyframes and observations, and then refines poses with a geometric solver, enabling robustness to viewpoint changes. The system assumes depth sensors or stereo camera for deployment. A novel dataset for the map-free relocalization task is also introduced. Extensive experiments including localization and navigation in both simulated and real-world scenarios have validate the system's performance and demonstrated its precision and efficiency for large-scale deployment. Code and data will be made publicly available at the webpage:https://rpl-cs-ucl.github.io/LiteVLoc.

State-of-the-art hierarchical localisation pipelines (HLoc) employ image retrieval (IR) to establish 2D-3D correspondences by selecting the top-k most similar images from a reference database. While increasing$k$improves localisation robustness, it also linearly increases computational cost and runtime, creating a significant bottleneck. This paper investigates the relationship between global and local descriptors, showing that greater similarity between the global descriptors of query and database images increases the proportion of feature matches. Low similarity queries significantly benefit from increasing k, while high similarity queries rapidly experience diminishing returns. Building on these observations, we propose an adaptive strategy that adjusts$k$based on the similarity between the query's global descriptor and those in the database, effectively mitigating the feature-matching bottleneck. Our approach reduces computational costs and processing time without sacrificing accuracy. Experiments on three indoor and outdoor datasets show that AIR-HLoc reduces feature matching time by up to 30% while preserving state-of-the-art accuracy. The results demonstrate that AIR-HLoc facilitates a latency-sensitive localisation system.

Path planning algorithms, such as the searchbased A*, are a critical component of autonomous mobile robotics, enabling robots to navigate from a starting point to a destination efficiently and safely. We investigated the resilience of the$A^{*}$algorithm in the face of potential adversarial interventions known as obstacle attacks. The adversary's goal is to delay the robot's timely arrival at its destination by introducing obstacles along its original path. We developed malicious software to execute the attacks and conducted experiments to assess their impact, both in simulation using TurtleBot in Gazebo and in real-world deployment with the Unitree Go1 robot. In simulation, the attacks resulted in an average delay of 36 %, with the most significant delays occurring in scenarios where the robot was forced to take substantially longer alternative paths. In real-world experiments, the delays were even more pronounced, with all attacks successfully rerouting the robot and causing measurable disruptions. These results highlight that the algorithm's robustness is not solely an attribute of its design but is significantly influenced by the operational environment. For example, in constrained environments like tunnels, the delays were maximized due to the limited availability of alternative routes.

Recent research on Simultaneous Localization and Mapping (SLAM) based on implicit representation has shown promising results in indoor environments. However, some challenges remain: the limited scene representation capability of implicit encoding, the uncertainty in the rendering process from implicit representations, and the disruption of consistency by dynamic objects. To address these challenges, we propose a dynamic visual SLAM system based on local-global fusion neural implicit representation, named DVN-SLAM. To improve the scene representation capability, we introduce a local-global fusion neural implicit representation that enables the construction of an implicit map while considering both global structure and local details. To tackle uncertainties arising from the rendering process, we design an information concentration loss for optimization, aiming to concentrate scene information on object surfaces. The proposed DVN-SLAM achieves competitive performance in localization and mapping across multiple datasets. More importantly, DVN-SLAM demonstrates robustness without semantic and optical flow prior in dynamic scenes, which sets it apart from other NeRF-based methods.

The creation of a metric-semantic map, which encodes human-prior knowledge, represents a high-level abstraction of environments. However, constructing such a map poses challenges related to the fusion of multi-modal sensor data, the attainment of real-time mapping performance, and the preservation of structural and semantic information consistency. In this paper, we introduce an online metric-semantic mapping system that utilizes LiDAR-Visual-Inertial sensing to generate a global metric-semantic mesh map of large-scale outdoor environments. Leveraging GPU acceleration, our mapping process achieves exceptional speed, with frame processing taking less than$7ms$, regardless of scenario scale. Furthermore, we seamlessly integrate the resultant map into a real-world navigation system, enabling metric-semantic-based terrain assessment and autonomous point-to-point navigation within a campus environment. Through extensive experiments conducted on both publicly available and self-collected datasets comprising 24 sequences, we demonstrate the effectiveness of our mapping and navigation methodologies. Note to Practitioners—This paper tackles the challenge of autonomous navigation for mobile robots in complex, unstructured environments with rich semantic elements. Traditional navigation relies on geometric analysis and manual annotations, struggling to differentiate similar structures like roads and sidewalks. We propose an online mapping system that creates a global metric-semantic mesh map for large-scale outdoor environments, utilizing GPU acceleration for speed and overcoming the limitations of existing real-time semantic mapping methods, which are generally confined to indoor settings. Our map integrates into a real-world navigation system, proven effective in localization and terrain assessment through experiments with both public and proprietary datasets. Future work will focus on integrating kernel-based methods to improve the map’s semantic accuracy.

In this paper, we propose a transformer-based architecture for predicting contact forces during a physical human-robot interaction. Our Neural Network is composed of two main parts: a Multi-Layer Perceptron called Transducer and a Transformer. The former estimates, based on the kinematic data from a motion capture suit, the current contact forces. The latter predicts – taking as input the same kinematic data and the output of the Transducer – the human motions and the contact forces over a time window in the future. We validated our approach by testing the network on directions of motions that were not provided in the training set. We also compared our approach to a purely Transformer-based network, showing a better prediction accuracy of the contact forces.

In this work, we present DiPPeR, a novel and fast 2D path planning framework for quadrupedal locomotion, leveraging diffusion-driven techniques. Our contributions include a scalable dataset generator for map images and corresponding trajectories, an image-conditioned diffusion planner for mobile robots, and a training/inference pipeline employing CNNs. We validate our approach in several mazes, as well as in real-world deployment scenarios on Boston Dynamic’s Spot and Unitree’s Go1 robots. DiPPeR performs on average 23 times faster for trajectory generation against both search based and data driven path planning algorithms with an average of 87% consistency in producing feasible paths of various length in maps of variable size, and obstacle structure. Website: https://rpl-cs-ucl.github.io/DiPPeR/

In this paper, we study the effectiveness of using a rigid movable palm for grasping varied objects, on a caging inspired gripper with three flexible fingers. This rigid palm extends to actively exert downwards force on objects, in contrast with existing methods, which combine movable palms with negative pressure to exert lifting forces on objects. We compare grasping with and without the palm, whilst also changing finger stiffness and fingertip angle, to analyse the effect on grasp success rate and stability over 24 design permutations. Reinforcement learning was used to train a unique grasping controller in every design case, aiming to achieve optimal grasping as the basis for comparison. Validation in both simulation and the real world was completed for every permutation. We demonstrated that the using palm improved success rates on average by 11% in simulation, 13% in the real world, and achieved a best real world success rate of 96% on 18 YCB benchmark food objects. Grasp stability against disturbances in three axes improved by 15% on average when using the palm. Our investigation determined fingertip angle had a large effect, whereas finger stiffness was less important.

We present DiPPeST, a novel image and goal conditioned diffusion-based trajectory generator for quadrupedal robot path planning. DiPPeST is a zero-shot adaptation of our previously introduced diffusion-based 2D global trajectory generator (DiPPeR). The introduced system incorporates a novel strategy for local real-time path refinements, that is reactive to camera input, without requiring any further training, image processing, or environment interpretation techniques. DiPPeST achieves 92% success rate in obstacle avoidance for nominal environments and an average of 88% success rate when tested in environments that are up to 3.5 times more complex in pixel variation than DiPPeR. A visual-servoing framework is developed to allow for real-world execution, tested on the quadruped robot, achieving 80% success rate in different environments and showcasing improved behavior than complex state-of-the-art local planners, in narrow environments. Website: https://rpl-cs-ucl.github.io/DiPPeSTweb/

Quadrupedal robots excel in mobility, navigating complex terrains with agility. However, their complex control systems present challenges that are still far from being fully addressed. In this paper, we introduce the use of Sample-Based Stochastic control strategies for quadrupedal robots, as an alternative to traditional optimal control laws. We show that Sample-Based Stochastic methods, supported by GPU acceleration, can be effectively applied to real quadruped robots. In particular, in this work, we focus on achieving gait frequency adaptation, a notable challenge in quadrupedal locomotion for gradient-based methods. To validate the effectiveness of Sample-Based Stochastic controllers we test two distinct approaches for quadrupedal robots and compare them against a conventional gradientbased Model Predictive Control system. Our findings, validated both in simulation and on a real 21Kg Aliengo quadruped, demonstrate that our method is on par with a traditional Model Predictive Control strategy when the robot is subject to zero or moderate disturbance, while it surpasses gradient-based methods in handling sustained external disturbances, thanks to the straightforward gait adaptation strategy that is possible to achieve within their formulation.

In this paper, we introduce a method to tackle the problem of robot local path planning among pushable objects –an open problem in robotics. In particular, we simultaneously train multiple agents in a physics-based simulation environment, utilizing an Advantage Actor-Critic algorithm coupled with a deep neural network. The developed online policy enables these agents to push obstacles in ways that are not limited to axial alignments, adapt to unforeseen changes in obstacle dynamics instantaneously, and effectively tackle local path planning in confined areas. We tested the method in various simulated environments to prove the adaptation effectiveness to various unseen scenarios in unfamiliar settings. Moreover, we have successfully applied this policy on an actual quadruped robot, confirming its capability to handle the unpredictability and noise associated with real-world sensors and the inherent uncertainties in unexplored object-pushing tasks.

Steering vectors (SVs) are a new approach to efficiently adjust language model behaviour at inference time by intervening on intermediate model activations. They have shown promise in terms of improving both capabilities and model alignment. However, the reliability and generalisation properties of this approach are unknown. In this work, we rigorously investigate these properties, and show that steering vectors have substantial limitations both in- and out-of-distribution. In-distribution, steerability is highly variable across different inputs. Depending on the concept, spurious biases can substantially contribute to how effective steering is for each input, presenting a challenge for the widespread use of steering vectors. Out-of-distribution, while steering vectors often generalise well, for several concepts they are brittle to reasonable changes in the prompt, resulting in them failing to generalise well. Overall, our findings show that while steering can work well in the right circumstances, there remain many technical difficulties of applying steering vectors to guide models' behaviour at scale.

This article presents a standardized human–robot teleoperation interface (HRTI) evaluation scheme for mobile manipulators. Teleoperation remains the predominant control type for mobile manipulators in open environments, particularly for quadruped manipulators. However, mobile manipulators, especially quadruped manipulators, are relatively novel systems to be implemented in the industry compared to traditional machinery. Consequently, no standardized interface evaluation method has been established for them. The proposed scheme is the first of its kind in evaluating mobile manipulator teleoperation. It comprises a set of robot motion tests, objective measures, subjective measures, and a prediction model to provide a comprehensive evaluation. The motion tests encompass locomotion, manipulation, and a combined test. The duration for each trial is collected as the response variable in the objective measure. Statistical tools, including mean value, standard deviation, and T-test, are utilized to cross-compare between different predictor variables. Based on an extended Fitts' law, the prediction model employs the time and mission difficulty index to forecast system performance in future missions. The subjective measures utilize the NASA-task load index and the system usability scale to assess workload and usability. Finally, the proposed scheme is implemented on a real-world quadruped manipulator with two widely-used HRTIs, the gamepad and the wearable motion capture system.

While mobile navigation has been focused on obstacle avoidance, Navigation Among Movable Obstacles (NAMO) via interaction with the environment, is a problem that is still open and challenging. This paper, presents a novel system integration to handle NAMO using visual feedback. In order to explore the capabilities of our introduced system, we explore the solution of the problem via graph-based path planning in a photorealistic simulator (NVIDIA Isaac Sim), in order to identify if the simulation-to-reality (sim2real) problem in robot navigation can be resolved. We consider the case where a wheeled robot navigates in a warehouse, in which movable boxes are common obstacles. We enable online real-time object localization and obstacle movability detection, to either avoid objects or, if it is not possible, to clear them out from the robot planned path by using pushing actions. We firstly test the integrated system in photorealistic environments, and we then validate the method on a real-world mobile wheeled robot (UCL MPPL) and its on-board sensory and computing system.

In this article, we propose a deep learning frame-work that provides a unified approach to the problem of leg contact detection in humanoid robot walking gaits. Our formulation accomplishes to accurately and robustly estimate the contact state probability for each leg (i.e., stable or slip/no contact). The proposed framework employs solely propriocep-tive sensing and although it relies on simulated ground-truth contact data for the classification process, we demonstrate that it generalizes across varying friction surfaces and different legged robotic platforms and, at the same time, is readily transferred from simulation to practice. The framework is quantitatively and qualitatively assessed in simulation via the use of ground-truth contact data and is contrasted against state-of-the-art methods with an ATLAS, a NAO, and a TALOS humanoid robot. Furthermore, its efficacy is demonstrated in base estimation with a real TALOS humanoid. To reinforce further research endeavors, our implementation is offered as an open-source ROS/Python package, coined Legged Contact Detection (LCD).

Mobile 3D Printing (M3DP), using printing-in-motion, is a powerful paradigm for automated construction. A mobile robot, equipped with its own power, materials and an arm-mounted extruder, simultaneously navigates and creates its environment. Such systems can be highly scalable, parallelizable and flexible. However, planning and controlling the motion of the arm and base at the same time is challenging and most deployments either avoid robot-base motion entirely or use human prescribed robot-base paths. In a previous paper, we developed a high-level planning algorithm to automate M3DP given a print task. The generated robot-base paths avoid collisions and maintain task reachability. In this paper, we extend this work to robot control. We develop and compare three different ways to integrate the long-duration planned path with a short horizon Model Predictive Controller. Experiments are carried out via a new M3DP system - Armstone. We evaluate and demonstrate our algorithm in a 250 m long multi-layer print which is about 5 times longer than any previous physical printing-in-motion system.

End-to-end scene text spotting has recently gained great attention in the research community. The majority of existing methods rely heavily on the location annotations of text instances (e.g., word-level boxes, word-level masks, and char-level boxes). We demonstrate that scene text spotting can be accomplished solely via text transcription, significantly reducing the need for costly location annotations. We propose a query-based paradigm to learn implicit location features via the interaction of text queries and image embeddings. These features are then made explicit during the text recognition stage via an attention activation map. Due to the difficulty of training the weakly-supervised model from scratch, we address the issue of model convergence via a circular curriculum learning strategy. Additionally, we propose a coarse-to-fine cross-attention localization mechanism for more precisely locating text instances. Notably, we provide a solution for text spotting via audio annotation, which further reduces the time required for annotation. Moreover, it establishes a link between audio, text, and image modalities in scene text spotting. Using only transcription annotations as supervision on both real and synthetic data, we achieve competitive results on several popular scene text benchmarks. The proposed method offers a reasonable trade-off between model accuracy and annotation time, allowing simplification of large-scale text spotting applications.

This paper proposes the design of a robotic gripper motivated by the bin-picking problem, where a variety of objects need to be picked from cluttered bins. The presented gripper design focuses on an enveloping cage-like approach, which surrounds the object with three hooked fingers, and then presses into the object with a movable palm. The fingers are flexible and imbue grasps with some elasticity, helping to conform to objects and, crucially, adding friction to cases where an object cannot be caged. This approach proved effective on a set of basic shapes, such as cuboids and cylinders, in which every object could be grasped. In particular, flat bottom parts could be grasped in a very stable manner, as demonstrated by testing grasps with multiple 5N and 10N disturbances. A set of supermarket items were also tested, highlighting promising features such as effective grasping of fruits and vegetables, as well as some limitations in the current embodiment, which is not always able to slip the fingers underneath objects.

In this paper, we explore the problem of task-consistent path planning for printing-in-motion via Mobile Manipulators (MM). MM offer a potentially unlimited planar workspace and flexibility for print operations. However, most existing methods have only mobility to relocate an arm which then prints while stationary. In this paper we present a new fully autonomous path planning approach for mobile material deposition. We use a modified version of Rapidly-exploring Random Tree Star (RRT*) algorithm, which is informed by a constrained Inverse Reachability Map (IRM) to ensure task consistency. Collision avoidance and end-effector reachability are respected in our approach. Our method also detects when a print path cannot be completed in a single execution. In this case it will decompose the path into several segments and reposition the base accordingly.

This paper introduces a novel deep learning approach to semantic segmentation of the shoreline environments with a high frames-per-second (fps) performance, making the approach readily applicable to autonomous navigation for Unmanned Surface Vehicles (USV). The proposed ShorelineNet is an efficient deep neural network of high performance relying only on visual input. ShorelineNet uses monocular visual input to produce accurate shoreline separation and obstacle detection compared to the state-of-the-art, and achieves this with real-time performance. Experimental validation on a challenging multi-modal maritime obstacle detection dataset, the MODD2 dataset, achieves a much faster inference (25fps on an NVIDIA Tesla K80 and 6fps on a CPU) with respect to the recent state-of-the-art methods, while keeping the performance equally high (73.1% F-score). This makes ShorelineNet a robust and effective model to be used for reliable USV navigation that require real-time and high-performance semantic segmentation of maritime environments.

Hybrid legged-wheeled robots such as the CEN-TAURO, are capable of varying their footprint polygon to carry out various agile motions. This property can be advantageous for wheeled-only planning in cluttered spaces, which is our focus. In this paper, we present an improved algorithm that builds upon our previously introduced preliminary footprint varying A* planner, which was based on the rectangular symmetry of the foot support polygon. In particular, we introduce a Theta* based planner with trapezium-like search, which aims to further reduce the limitations imposed upon the wheeled-only navigation of the CENTAURO robot by the low-dimensional search space, maintaining the real-time computational efficiency. The method is tested on the simulated and real full-size CENTAURO robot in cluttered environments.

Nowadays, robots are expected to enter in various application scenarios and interact with unknown and dynamically changing environments. This highlights the need for creating autonomous robot behaviours to explore such environments, identify their characteristics and adapt, and build knowledge for future interactions. To respond to this need, in this paper we present a novel framework that integrates multiple components to achieve a context-aware and adaptive interaction between the robot and uncertain environments. The core of this framework is a novel self-tuning impedance controller that regulates robot quasi-static parameters, i.e., stiffness and damping, based on the robot sensory data and vision. The tuning of the parameters is achieved only in the direction(s) of interaction or movement, by distinguishing expected interactions from external disturbances. A vision module is developed to recognize the environmental characteristics and to associate them to the previously/newly identified interaction parameters, with the robot always being able to adapt to the new changes or unexpected situations. This enables a faster robot adaptability, starting from better initial interaction parameters. The framework is evaluated experimentally in an agricultural task, where the robot effectively interacts with various deformable environments.

Contemporary humanoids are equipped with visual and LiDAR sensors that are effectively utilized for Visual Odometry (VO) and LiDAR Odometry (LO). Unfortunately, such measurements commonly suffer from outliers in a dynamic environment, since frequently it is assumed that only the robot is in motion and the world is static. To this end, robust state estimation schemes are mandatory in order for humanoids to symbiotically co-exist with humans in their daily dynamic environments. In this article, the robust Gaussian Error-State Kalman Filter for humanoid robot locomotion is presented. The introduced method automatically detects and rejects outliers without relying on any prior knowledge on measurement distributions or finely tuned thresholds. Subsequently, the proposed method is quantitatively and qualitatively assessed in realistic conditions with the full-size humanoid robot WALK-MAN v2.0 and the mini-size humanoid robot NAO to demonstrate its accuracy and robustness when outlier VOLO measurements are present. Finally, in order to reinforce further research endeavours, our implementation is released as an open-source ROS/C++package.

Hybrid legged-wheeled robots are able to adapt their leg configuration and height to vary their footprint polygons and go over obstacles or traverse narrow spaces. In this paper, we present a variable configuration wheeled motion planner based on the A* algorithm. It takes advantage of the agility of hybrid wheeled-legged robots and plans paths over low-lying obstacles and in narrow spaces. By imposing a symmetry on the robot polygon, the computed plans lie in a low-dimensional search space that provides the robot with configurations to safely negotiate obstacles by expanding or shrinking its footprint polygon. The introduced autonomous planner is demonstrated using simulations and real-world experiments with the CENTAURO robot.

In this paper, we present an implementation of a bi-manual teleoperation system, controlled by a human through three-dimensional (3D) skeleton extraction. The input data is given from a cheap RGB-D range sensor, such as the ASUS Xtion PRO. To achieve this, we have implemented a 3D version of the impressive OpenPose package, which was recently developed. The first stage of our method contains the execution of the OpenPose Convolutional Neural Network (CNN), using a sequence of RGB images as input. The extracted human skeleton pose localisation in two-dimensions (2D) is followed by the mapping of the extracted joint location estimations into their 3D pose in the camera frame. The output of this process is then used as input to drive the end-pose of the robotic hands relative to the human hand movements, through a whole-body inverse kinematics process in the Cartesian space. Finally, we implement the method as a ROS wrapper package and we test it on the centaur-like CENTAURO robot. Our demonstrated task is of a box and lever manipulation in real-time, as a result of a human task demonstration.

Bipedal robots have gained a lot of locomotion capabilities the past few years, especially in the control level. Navigation over complex and unstructured environments using exteroceptive perception, is still an active research topic. In this paper, we present a footstep planning system to produce foothold placements, using visual perception and proper environment modeling, given a black box walking controller. In particular, we extend a state-of-the-art search-based planning approach (ARA*) that produces 6DoF footstep sequences in 3D space for flat uneven terrain, to also handle rough curved surfaces, e.g. rocks. This is achieved by integrating both a curved patch modeling system for rough local terrain surfaces and a flat foothold contact analysis based on visual range input data, into the existing planning framework. The system is experimentally validated using real-world point clouds, while rough terrain stepping demonstrations are presented on the WALK-MAN humanoid robot, in simulation.

We present a new method to translate videos to commands for robotic manipulation using Deep Recurrent Neural Networks (RNN). Our framework first extracts deep features from the input video frames with a deep Convolutional Neural Networks (CNN). Two RNN layers with an encoder-decoder architecture are then used to encode the visual features and sequentially generate the output words as the command. We demonstrate that the translation accuracy can be improved by allowing a smooth transaction between two RNN layers and using the state-of-the-art feature extractor. The experimental results on our new challenging dataset show that our approach outperforms recent methods by a fair margin. Furthermore, we combine the proposed translation module with the vision and planning system to let a robot perform various manipulation tasks. Finally, we demonstrate the effectiveness of our framework on a full-size humanoid robot WALK-MAN.

Complex interactions with unstructured environments require the application of appropriate restoring forces in response to the imposed displacements. Impedance control techniques provide effective solutions to achieve this, however, their quasi-static performance is highly dependent on the choice of parameters, i.e. stiffness and damping. In most cases, such parameters are previously selected by robot programmers to achieve a desired response, which limits the adaptation capability of robots to varying task conditions. To improve the generality of interaction planning through task-dependent regulation of the parameters, this paper introduces a novel self-regulating impedance controller. The regulation of the parameters is achieved based on the robot's local sensory data, and on an interaction expectancy value. This value combines the interaction values from the robot state machine and visual feedback, to authorize the autonomous tuning of the impedance parameters in selective Cartesian axes. The effectiveness of the proposed method is validated experimentally in a debris removal task.

We present a new method to detect object affordances in real-world scenes using deep Convolutional Neural Networks (CNN), an object detector and dense Conditional Random Fields (CRF). Our system first trains an object detector to generate bounding box candidates from the images. A deep CNN is then used to learn the depth features from these bounding boxes. Finally, these feature maps are post-processed with dense CRF to improve the prediction along class boundaries. The experimental results on our new challenging dataset show that the proposed approach outperforms recent state-of-the-art methods by a substantial margin. Furthermore, from the detected affordances we introduce a grasping method that is robust to noisy data. We demonstrate the effectiveness of our framework on the full-size humanoid robot WALK-MAN using different objects in real-world scenarios.

This paper describes a new task-oriented grasping method to reorient a rigid object to its nominal pose, which is defined as the configuration that it needs to be grasped from, in order to successfully execute a particular manipulation task. Our method combines two key insights: (1) a visual 6 Degree-of-Freedom (DoF) pose estimation technique based on 2D-3D point correspondences is used to estimate the object pose in real-time and (2) the rigid transformation from the current to the nominal pose is computed online and the object is reoriented over a sequence of steps. The outcome of this work is a novel method that can be effectively used in the preparatory phase of a manipulation task, to permit a robot to start from arbitrary object placements and configure the manipulated objects to the nominal pose, as required for the execution of a subsequent task. We experimentally demonstrate the effectiveness of our approach on a full-size humanoid robot (WALK-MAN) using different objects with various pose settings under real-time constraints.

We present a novel and real-time method to detect object affordances from RGB-D images. Our method trains a deep Convolutional Neural Network (CNN) to learn deep features from the input data in an end-to-end manner. The CNN has an encoder-decoder architecture in order to obtain smooth label predictions. The input data are represented as multiple modalities to let the network learn the features more effectively. Our method sets a new benchmark on detecting object affordances, improving the accuracy by 20% in comparison with the state-of-the-art methods that use hand-designed geometric features. Furthermore, we apply our detection method on a full-size humanoid robot (WALK-MAN) to demonstrate that the robot is able to perform grasps after efficiently detecting the object affordances.

This paper describes a novel foot for biped robots designed to provide a reliable and low-cost solution for sensing the Center of Pressure (CoP) on flat and uneven surfaces. The foot uses a new method for detecting contact forces based on measuring the deflection of three flexural toes using Hall-effect magnetic field sensors. We experimentally compare five mathematical models for calculating the CoP coordinates from the sensor data. Results confirm that with the proposed method it is possible to obtain the same level of accuracy and reliability as with standard force sensing resistors, but without some the drawbacks of that approach.

We present a new bio-inspired system for automatically finding foot-scale curved surface patches in rough rocky terrain. These patches are intended to provide a reasonable set of choices for higher-level footfall selection algorithms, and are pre-filtered for several attributes - including location, curvature, and normal - that we observed humans to prefer. Input is from a 640 × 480 depth camera augmented with a 9-DoF inertial measurement unit to sense the direction of gravity. The system is capable of finding approximately 700 patches/second on commodity hardware, though the intention is not to find as many patches as possible but to reasonably sample upcoming terrain with quality patches. Sixty recordings of human subjects traversing rocky trails were analyzed to give a baseline for target patch properties. While the presented system is not designed to select a single foothold, it does find a set of patches for possible footholds which are statistically similar to the patches humans select.

Traditional segmentation algorithms for range images create partitions of connected and non-overlapping-but potentially irregularly shaped-regions corresponding to world surfaces. This paper presents an alternative paradigm based on regularly shaped curved patches (paraboloids) that model local contact regions potentially compatible with e.g. a robot's toe, heel, or fingertip. These patches randomly sample the environment surface but are not required to strictly partition it. They are fit to neighborhoods of the range data and then validated for fit quality and fidelity to the actual data-extrapolations (like hole-filling) which are not directly supported by data are avoided. Two different neighborhood formation methods based on k-d tree and triangle mesh data structures are compared, and results are presented for 10 datasets taken in natural rocky terrain.

We introduce a set of 10 bounded curved-surface patch types suitable for modeling local contact regions both in the environment and on a robot. We present minimal geometric parameterizations using the exponential map for spatial pose both in the usual 6DoF case and also for patches with revolute symmetry that have only 5DoF. We then give an algorithm to fit any patch type to point samples of a surface, with quantified uncertainty both in the input points (including nonuniform variance, common in data from range sensors) and in the output patch. Finally, we outline how such patches can be composed into a spatial patch map of the available contact surfaces both on and around a robot.