Fully autonomous vehicles promise enhanced safety and efficiency. However, ensuring reliable operation in challenging corner cases requires control algorithms capable of performing at the vehicle limits. We address this requirement by considering the task of autonomous racing and propose solving it by learning a racing policy using Reinforcement Learning (RL). Our approach leverages domain randomization, actuator dynamics modeling, and policy architecture design to enable reliable and safe zero-shot deployment on a real platform. Evaluated on the F1TENTH race car, our RL policy not only surpasses a state-of-the-art Model Predictive Control (MPC), but, to the best of our knowledge, also represents the first instance of an RL policy outperforming expert human drivers in RC racing. This work identifies the key factors driving this performance improvement, providing critical insights for the design of robust RL-based control strategies for autonomous vehicles.

Collaborative robots (cobots) are a promising technology for frontline workers in industry. They can support tasks that cannot be fully automated but are repetitive, fatiguing, boring, or dangerous for humans. Although cobots are explicitly designed to work with humans, they remain primarily non-intuitive and difficult to collaborate with. Thus, there is a need for new interaction approaches to facilitate efficient human-robot collaboration. Recently, we could see emerging examples of using augmented reality (AR) to assist a worker in collaborative task execution with a cobot. However, for such an approach to provide truly efficient support for the seamless bimanual task execution, we need to first investigate interaction methods offered by an AR interface. To that end, we performed a study with sixteen participants to compare eye-gaze and hand-ray pointing methods for part selection in collaborative, manual assembly tasks. The results of our study show that both techniques provide similar perceived usability, with the eye-gaze selection leading to significantly shorter completion times.

Velocity estimation is of great importance in autonomous racing. Still, existing solutions are characterized by limited accuracy, especially in the case of aggressive driving or poor generalization to unseen road conditions. To address these issues, we propose to utilize Unscented Kalman Filter (UKF) with a learned dynamics model that is optimized directly for the state estimation task. Moreover, we propose to aid this model with the online-estimated friction coefficient, which increases the estimation accuracy and enables zero-shot adaptation to the new road conditions. To evaluate the UKF-based velocity estimator with the proposed dynamics model, we introduced a publicly available dataset of aggressive maneuvers performed by an F1TENTH car, with side-slip angles reaching 40°. Using this dataset, we show that learning the dynamics model through UKF leads to improved estimation performance and that the proposed solution outperforms state-of-the-art learning-based state estimators by 17% in the nominal scenario. Moreover, we present unseen zero-shot adaptation abilities of the proposed method to the new road surface thanks to the proposed learning-based tire dynamics model with online friction estimation.

Motion planning is a mature area of research in robotics with many well-established methods based on optimization or sampling the state space, suitable for solving kinematic motion planning. However, when dynamic motions under constraints are needed and computation time is limited, fast kinodynamic planning on the constraint manifold is indispensable. In recent years, learning-based solutions have become alternatives to classical approaches, but they still lack comprehensive handling of complex constraints, such as planning on a lower-dimensional manifold of the task space while considering the robot's dynamics. This paper introduces a novel learning-to-plan framework that exploits the concept of constraint manifold, including dynamics, and neural planning methods. Our approach generates plans satisfying an arbitrary set of constraints and computes them in a short constant time, namely the inference time of a neural network. This allows the robot to plan and replan reactively, making our approach suitable for dynamic environments. We validate our approach on two simulated tasks and in a demanding real-world scenario, where we use a Kuka LBR Iiwa 14 robotic arm to perform the hitting movement in robotic Air Hockey.

While manipulating rigid objects is an extensively explored research topic, deformable linear object (DLO) manipulation seems significantly underdeveloped. A potential reason for this is the inherent difficulty in describing and observing the state of the DLO as its geometry changes during manipulation. This paper proposes an algorithm for fast-tracking the shape of a DLO based on the masked image. Having no prior knowledge about the tracked object, the proposed method finds a reliable representation of the shape of the tracked object within tens of milliseconds. This algorithm's main idea is to first skeletonize the DLO mask image, walk through the parts of the DLO skeleton, arrange the segments into an ordered path, and finally fit a B-spline into it. Experiments show that our solution outperforms the State-of-the-Art approaches in DLO's shape reconstruction accuracy and algorithm running time and can handle challenging scenarios such as severe occlusions, self-intersections, and multiple DLOs in a single image.

Space exploration has experienced a surge in interest and accessibility, with an increasing number of spacecraft launches. However, the scaling of space technology faces challenges as it heavily relies on human supervision and intervention. To overcome these limitations and enable greater autonomy, recent advancements in software and hardware, particularly in commercial off-the-shelf (COTS) components, have provided new opportunities. This paper introduces a prototype of a tightly-coupled hardware-software system that leverages a standard COTS computational platform and deep learning coprocessors to enable the efficient execution of deep learning workloads for space rovers. Integrated within the Robot Operating System 2 (ROS 2) framework, the system incorporates onboard sensors and offers rapid prototyping capabilities. By harnessing the benefits of COTS components and advanced software frameworks, this system represents a step towards achieving increased autonomy in space rovers, while also reducing development time. The presented system showcases the potential for future advancements in autonomous space exploration. The project documentation is publicly available: https://github.com/PUTvision/ros2_fpga_inference_node

Moon exploration has gained significant momentum in recent decades, with growing interest from space agencies and private investors. A diverse range of activities is associated with moon exploration, encompassing spacecraft design, payload transportation, launcher capabilities, resource identification and mining, and establishing a sustained presence on our only natural satellite. Such endeavors would require the shipment of both scientific and life-sustaining equipment. However, communication between Earth’s mission control and the Moon’s bases can still present challenges. In this paper, we introduce Cognition – a rover-lander distributed system that approaches this problem by distributing the data processing between the rover and the lander. The primary goal of the Cognition system is to optimize lunar surface exploration by minimizing data transmission to the Earth’s surface, prioritizing the transfer of valuable data, and augmenting the level of autonomy in the process.

Haptic sensing has recently been used effectively for legged robot localization in extreme scenarios where cam-eras and LiDAR might fail, such as dusty mines and foggy sewers. However, existing haptic sensing mainly relies on supervised classification, with training and evaluation executed over explicit terrain classes. Defining classes is a significant limitation to real-world applications, where prior labelling and handcrafted classes are often impractical. This paper proposes a novel haptic localization system based on a fully unsupervised terrain representation learned solely from the force/torque sensors located in the quadruped robot's feet. Instead of using the detected terrain class for localization, we propose an improved autoencoder architecture to generate a sparse map of encodings on the first run and to localize against this sparse map during subsequent runs. We compare our approach to a haptic localization system based on supervised terrain classification, showing that the unsupervised method has comparable or better performance than the supervised one for the same trajectories while clearly outperforming the proprioceptive odometry estimator available on the robot. Therefore, the proposed approach is well-suited for a routine maintenance application, increasing the platform's robustness.

Human-Robot Interaction (HRI) for collaborative robots has not changed since the introduction of the first cobot. The main interface to communicate with the robot remains a wired display - teach pendant (TP). While attempts are made to make the programming experience better - more intuitive touch-screen displays, it generally remains the same. With the recent rapid development of Augmented Reality (AR), the HRI of the cobot could drastically change. This paper explores AR-based implementations in robotics and categorizes them based on the type of the used device, with the main focus on the least explored category - mobile AR. Furthermore, two experiments are conducted to determine the user’s experience in robot programming using TP with a mobile-based AR interface. For this reason, an AR application prototype is developed as a co-interface to a TP. The results of the experiments are presented: the first examines the user’s needs that are missing in current solutions, while the second one analyses the user’s experience in using the robot with the AR interface. The obtained results suggest that users could benefit from mobile-based AR solutions in the commissioning and troubleshooting phase of the lifetime of the robot. However, at the same time, this solution is not advanced and accurate enough (yet) to encourage users to switch to the new platform and abandon the classical TP, while programming the robot.

Mobile robots are becoming very popular in real-world outdoors applications, where there are many challenges in robot control and perception. One of the most critical problems is to characterise the terrain traversed by the robot. This knowledge is indispensable for optimal terrain negotiation. Currently, most approaches are performing terrain classification from vision, but there is not enough research on terrain identification from a direct interaction of the robot with the environment. In our work, we proposed new methods for classification of force/torque data from an interaction of the legged robot foot with the ground, gathered during the walking process. We provided machine learning methods for terrain classification from raw force/torque signals for which we achieved 93% accuracy on a challenging dataset with 160 minutes of recorded fixed-length steps. We also worked on a dataset where the assumption of a fixed-length step is not valid. In this case, the final result is around 80% of accuracy. The most important fact is that the data in both cases was recorded while the robot was walking, no particular movements or controlled environment were needed. Additionally, we also proposed a clustering method which allows us to learn about the class membership based on the recorded data only, without any human supervision.

One of the major goals for mobile robots is to be able to traverse any kind of terrains. A possible way to achieve this goal is by the use of legged robots, as they have increased mobility. However, this would require them to be able to modify their gaits, based on the identification of the terrain that they are currently traversing. In this paper, we introduce a number of novel methods to address this issue of autonomous terrain classification and clustering, based on tactile data collected with a walking robot. The proposed learning methods are based on the Pitman-Yor process mixture of Gaussians, a Bayesian nonparametric prior, well-suited for density estimation. This model is initially used to learn the non-Gaussian distribution of the features produced from proprioceptive (force/torque) signals from the legs, registered during the interaction of one robot foot with a terrain. Then, we exploit its capacity on clustering and discovering structures in the data to identify terrains in the feature space. Experiments were conducted on a six-legged robot, thus demonstrating the applicability of the Pitman-Yor process mixture of Gaussians for terrain identification. In particular, we obtained a classification success rate of 82% and 51% accuracy, with our supervised learning and unsupervised learning approach respectively.

We propose a joint object pose estimation and categorization approach which extracts information about object poses and categories from the object parts and compositions constructed at different layers of a hierarchical object representation algorithm, namely Learned Hierarchy of Parts (LHOP) [7]. In the proposed approach, we first employ the LHOP to learn hierarchical part libraries which represent entity parts and compositions across different object categories and views. Then, we extract statistical and geometric features from the part realizations of the objects in the images in order to represent the information about object pose and category at each different layer of the hierarchy. Unlike the traditional approaches which consider specific layers of the hierarchies in order to extract information to perform specific tasks, we combine the information extracted at different layers to solve a joint object pose estimation and categorization problem using distributed optimization algorithms. We examine the proposed generative-discriminative learning approach and the algorithms on two benchmark 2-D multi-view image datasets. The proposed approach and the algorithms outperform state-of-the-art classification, regression and feature extraction algorithms. In addition, the experimental results shed light on the relationship between object categorization, pose estimation and the part realizations observed at different layers of the hierarchy.

This paper presents terrain classification method based on the intensity readings from Laser Range Finder. The classification is performed on the feature vectors obtained using statistical descriptors or Fourier Transform computed for the patches of the intensity map for each terrain sample. As a classifier Support Vector Machines were used. For the set of 12 terrains results of classification are reaching the level of 98% of the correctly recognized terrain samples. The proposed approach has a low computational cost, which is required for its real time applications. The article begins with the description of the experimental setup followed by the presentation of the proposed feature vectors for the registered intensity maps. Next, classification results, using introduced features, are given and compared to other approaches found in literature. At the end concluding remarks are given.

Multilegged robots working in urban environment have to cope with obstacles. Some robots avoid them, but there are obstacles which have to be traversed. In the paper a generalized scheme of urban obstacle negotiation process using the Finite State Machine model was presented. In each state the robot performs the movement or gathers the data from sensors. The transitions are made according to events which are external signals depending on the particular set of robot variables. The obtained automaton is generic with respect to the type of the obstacle. The same automaton describes negotiation of curbs, platforms and stairs (in ascent and descent phase). In this article at first the extended automaton is presented. Then the reduced form of the Finite State Machine is described. Next the experiments on simulator and on the real robot are demonstrated. The experimental set-up consists of a robot and an environment with a curb and stairs. The robot was able to negotiate these obstacles using the same automaton.