Monocular visual inertial odometry (VIO) has facilitated a wide range of real-time motion tracking applications, thanks to the small size of the sensor suite and low power consumption. To successfully bootstrap VIO algorithms, the initialization module is extremely important. Most initialization methods rely on the reconstruction of 3D visual point clouds. These methods suffer from high computational cost as state vector contains both motion states and 3D feature points. To address this issue, some researchers recently proposed a structureless initialization method, which can solve the initial state without recovering 3D structure. However, this method potentially compromises performance due to the decoupled estimation of rotation and translation, as well as linear constraints. To improve its accuracy, we propose novel structureless visual-inertial bundle adjustment to further refine previous structureless solution. Extensive experiments on real-world datasets show our method significantly improves the VIO initialization accuracy, while maintaining real-time performance.

The European Space Agency (ESA) and the European Space Resources Innovation Centre (ESRIC) created the Space Resources Challenge to invite researchers and companies to propose innovative solutions for Multi-Robot Systems (MRS) space prospection. This paper proposes the Resilient Exploration And Lunar Mapping System 2 (REALMS2), a MRS framework for planetary prospection and mapping. Based on Robot Operating System version 2 (ROS 2) and enhanced with Visual Simultaneous Localisation And Mapping (vSLAM) for map generation, REALMS2 uses a mesh network for a robust ad hoc network. A single graphical user interface (GUI) controls all the rovers, providing a simple overview of the robotic mission. This system is designed for heterogeneous multi-robot exploratory missions, tackling the challenges presented by extraterrestrial environments. REALMS2 was used during the second field test of the ESA-ESRIC Challenge and allowed to map around 60% of the area, using three homogeneous rovers while handling communication delays and blackouts.

This paper presents an integrated Reinforcement Learning (RL) and Model Predictive Control (MPC) framework for autonomous satellite docking with a partially filled fuel tank. Traditional docking control faces challenges due to fuel sloshing in microgravity, which induces unpredictable forces affecting stability. To address this, we integrate Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) RL algorithms with MPC, leveraging MPC’s predictive capabilities to accelerate RL training and improve control robustness. The proposed approach is validated through Zero-G Lab of SnT experiments for planar stabilization and high-fidelity numerical simulations for 6-DOF docking with fuel sloshing dynamics. Simulation results demonstrate that SAC-MPC achieves superior docking accuracy, higher success rates, and lower control effort, outperforming standalone RL and PPO-MPC methods. This study advances fuel-efficient and disturbance-resilient satellite docking, enhancing the feasibility of on-orbit refueling and servicing missions.

Online extrinsic calibration is crucial for building "power-on-and-go" moving platforms, like robots and AR devices. However, blindly performing online calibration for unobservable parameter may lead to unpredictable results. In the literature, extensive studies have been conducted on the extrinsic calibration between IMU and camera, from theory to practice. It is well-known that the observability of extrinsic parameter can be guaranteed under sufficient motion excitation. Furthermore, the impacts of degenerate motions are also investigated. Despite these successful analyses, we identify an issue with respect to the existing observability conclusion. This paper focuses on the observability investigation for straight line motion, which is a common-seen and fundamental degenerate motion in applications. We analytically prove that pure translational straight line motion can lead to the unobservability of the rotational extrinsic parameter between IMU and camera (at least one degree of freedom). By correcting the existing observability conclusion, our novel theoretical finding disseminates more precise principle to the research community and provides explainable calibration guideline for practitioners. Our analysis is validated by rigorous theory and experiments.

In robotics, motion capture systems have been widely used to measure the accuracy of localization algorithms. Moreover, this infrastructure can also be used for other computer vision tasks, such as the evaluation of Visual (-Inertial) SLAM dynamic initialization, multi-object tracking, or automatic annotation. Yet, to work optimally, these functionalities require having accurate and reliable spatial-temporal calibration parameters between the camera and the global pose sensor. In this study, we provide two novel solutions to estimate these calibration parameters. Firstly, we design an offline target-based method with high accuracy and consistency. Spatial-temporal parameters, camera intrinsic, and trajectory are optimized simultaneously. Then, we propose an online target-less method, eliminating the need for a calibration target and enabling the estimation of time-varying spatial-temporal parameters. Additionally, we perform detailed observability analysis for the target-less method. Our theoretical findings regarding observability are validated by simulation experiments and provide explainable guidelines for calibration. Finally, the accuracy and consistency of two proposed methods are evaluated with hand-held real-world datasets where traditional hand-eye calibration method do not work.

Accurate global localization is crucial for autonomous navigation and planning. To this end, various GPS-aided Visual-Inertial Odometry (GPS-VIO) fusion algorithms are proposed in the literature. This paper presents a novel GPS-VIO system that is able to significantly benefit from the online calibration of the rotational extrinsic parameter between the GPS reference frame and the VIO reference frame. The behind reason is this parameter is observable. This paper provides novel proof through nonlinear observability analysis. We also evaluate the proposed algorithm extensively on diverse platforms, including flying UAV and driving vehicle. The experimental results support the observability analysis and show increased localization accuracy in comparison to state-of-the-art (SOTA) tightly-coupled algorithms.

This investigation introduces a novel deep reinforcement learning-based suite to control floating platforms in both simulated and real-world environments. Floating platforms serve as versatile test-beds to emulate microgravity environments on Earth, useful to test autonomous navigation systems for space applications. Our approach addresses the system and environmental uncertainties in controlling such platforms by training policies capable of precise maneuvers amid dynamic and unpredictable conditions. Leveraging Deep Reinforcement Learning (DRL) techniques, our suite achieves robustness, adaptability, and good transferability from simulation to reality. Our deep reinforcement learning framework provides advantages such as fast training times, large-scale testing capabilities, rich visualization options, and ROS bindings for integration with real-world robotic systems. Being open access, our suite serves as a comprehensive platform for practitioners who want to replicate similar research in their own simulated environments and labs.

During natural disasters such as earthquakes, wildfires, hurricanes, landslides, tsunamis, CBRNE (chemical, medical, radiological, nuclear, or explosive) incidents, or terrorist threats, critical infrastructure can be severely damaged, with rescue workers facing immense obstacles. Providing help and reaching affected areas can be hazardous for human rescue personnel. To overcome this issue, 5G-enabled UAVs can play a significant role in deploying disaster-resilient networks. Such networks imply multiple challenges, including end-to-end communication reliability and low latency for UAV real-time control. In this paper, we focus on the end-to-end communication aspects of such networks. We present and implement our disaster-resilient network based on a combination of 5G-UAVs and satellite networks. We highlight the related challenges and define solutions based on 5G, Multi-access Edge Computing, UAVs, and dynamic geofencing. We integrate these solutions in an end-to-end network architecture and ultimately deploy it on a national joint 5G-satellite infrastructure to assess its performance. It performs well and demonstrates a recovery time of less than 30 seconds with $\mathbf{9 9 \%}$ network availability.

Exoskeletons are wearable robotic devices that surround the anatomy of the user to work in tandem with it. Depending on their structure, exoskeletons can be classified as rigid or flexible. The structure of flexible exoskeletons is made of soft materials, such as fabrics, which adapt to user motion. Therefore, these devices are prone to becoming misaligned with the user, due to improper fitting and slipping of the exoskeleton components on the user body. This article describes a cable-driven exosuit, calledLUXBIT, that favors its anatomical adaption to the user by arranging the fabric fibers and sewing patterns to transfer the mobilizing forces. This prototype integrates a novel deformable mechanism that promotes the natural lifting of the arm.LUXBITis intended for bimanual assistance in daily living, being equipped with a backpack to this end. The results analyzed in this article show thatLUXBITreduces muscle activity in the upper limbs’ flexion by ratios of up to 13.17% and allows the user to hold tiring postures for 62.91% longer.

The new era of space exploration demands a significant increase in the number of human and robotic missions, thus resulting in novel communication and service requirements. To satisfy such requirements, the fifth generation of mobile communication systems (5G), despite providing connectivity on Earth, has the potential to serve as a communication standard for space resource missions, particularly the ones targeting the Moon. In fact, 5G non-terrestrial networks (NTNs) are already in the standardization process and new techniques are being proposed in order to counteract the peculiarities of the non-terrestrial channel. However, going one step ahead and deploying constellations of satellites around the Earth or the Moon, requires first a detailed analysis and testing of the validity of the proposed techniques. Therefore, in this paper, we introduce the 5G Space Communications Lab, which has been developed with the purpose of simulating space-based 5G communications. The designed testbed proposed here increases the technology readiness level (TRL) of NTN-based 5G systems, demonstrating over a laboratory environment successful 5G communication via space links.

Space robotics applications, such as Active Space Debris Removal (ASDR), require representative testing before launch. A commonly used approach to emulate the microgravity environment in space is air-bearing based platforms on flat-floors, such as the European Space Agency's Orbital Robotics and GNC Lab (ORGL). This work proposes a control architecture for a floating platform at the ORGL, equipped with eight solenoid-valve-based thrusters and one reaction wheel. The control architecture consists of two main components: a trajectory planner that finds optimal trajectories connecting two states and a trajectory follower that follows any physically feasible trajectory. The controller is first evaluated within an introduced simulation, achieving a 100% success rate at finding and following trajectories to the origin within a Monte-Carlo test. Individual trajectories are also successfully followed by the physical system. In this work, we showcase the ability of the controller to reject disturbances and follow a straight-line trajectory within tens of centimeters.

Extraterrestrial rovers with a general-purpose robotic arm have many potential applications in lunar and planetary exploration. Introducing autonomy into such systems is desirable for increasing the time that rovers can spend gathering scientific data and collecting samples. This work investigates the applicability of deep reinforcement learning for vision-based robotic grasping of objects on the Moon. A novel simulation environment with procedurally-generated datasets is created to train agents under challenging conditions in unstructured scenes with uneven terrain and harsh illumination. A model-free off-policy actor-critic algorithm is then employed for end-to-end learning of a policy that directly maps compact octree observations to continuous actions in Cartesian space. Experimental evaluation indicates that 3D data representations enable more effective learning of manipulation skills when compared to traditionally used image-based observations. Domain randomization improves the generalization of learned policies to novel scenes with previously unseen objects and different illumination conditions. To this end, we demonstrate zero-shot sim-to-real transfer by evaluating trained agents on a real robot in a Moon-analogue facility. The source code and datasets are available at https://github.com/AndrejOrsula/drl_grasping.

The reliability of autonomous traverses of rovers is critical. It may be jeopardized by the accumulation of errors and the uncertainty propagation of their localization systems. Moreover, space environments are usually harsh, challenging and unpredictable. Teleoperation is complex due to the significant and unavoidable delay. For these reasons, a path planner that provides some level of autonomy with guarantees could increase the success rate of planetary exploration missions. This paper proposes a path planning solution that tackles increasing localization uncertainty and makes a trade-off between the collision risk and the path length. The planner uses the the Fast Marching Method (FMM) to produce a costmap aware of this uncertainty and calculate the optimal path for a level of confidence. This paper additionally presents several simulation and experimental using a wheeled robotic vehicle within a lunar analogue facility.

5G-enabled UAV-based services have become popular for civilian applications. At the same time, certain no-fly zones will be highly dynamic, e.g. accident areas, large outdoor public events, VIP convoys etc. An appropriate geofencing algorithm is required to avoid the no-fly zone in such scenarios. However, it is challenging to execute a high computing process such as a geofencing algorithm for a resource constraint UAV. This paper proposes an architecture and a geofencing algorithm for 5G-enabled UAV using Mobile Edge Computing (MEC). Also, the 5G-enabled UAV must fly within the coverage area during a mission. Hence, there must be an optimal trade-off between 5G coverage and distance to travel to design a new trajectory for a 5G-enabled UAV. To this end, we propose a cost minimization problem to generate a new trajectory while a no-fly zone exists. Specifically, we design a cost function considering 5G coverage and the velocity of the UAV. Then, we propose a geofencing algorithm running at the MEC by adopting the fast marching method (FMM) to generate a new trajectory for the UAV. Finally, a numerical example shows how the proposed geofencing algorithm generates an optimal trajectory for a UAV to avoid a dynamically created no-fly zone while on the mission.

Robot self-localization is essential for operating autonomously in open environments. When cameras are the main source of information for retrieving the pose, numerous challenges are posed by the presence of dynamic objects, due to occlusion and continuous changes in the appearance. Recent research on global localization methods focused on using a single (or multiple) Convolutional Neural Network (CNN) to estimate the 6 Degrees of Freedom (6-DoF) pose directly from a monocular camera image. In contrast with the classical approaches using engineered feature detector, CNNs are usually more robust to environmental changes in light and to occlusions in outdoor scenarios. This paper contains an attempt to empirically demonstrate the ability of CNNs to ignore dynamic elements, such as pedestrians or cars, through learning. For this purpose, we pre-process a dataset for pose localization with an object segmentation network, masking potentially moving objects. Hence, we compare the pose regression CNN trained and/or tested on the set of masked images and the original one. Experimental results show that the performances of the two training approaches are similar, with a slight reduction of the error when hiding occluding objects from the views.

In this paper we apply deep reinforcement learning techniques on a multicopter for learning a stable hovering task in a continuous state action environment. We present a framework based on OpenAI GYM, Gazebo, Robotic Operating System and RotorS MAV simulator, used for successfully training different agents to perform various tasks. The deep reinforcement learning method used for the training is a model-free, on-policy, actor-critic based algorithm called Trust Region Policy Optimization (TRPO). Two neural networks have been used as nonlinear function approximators. Our experiments show that such learning approach achieves successful results, and facilitates the process of controller design.

The reliability of aircraft inspection is of paramount importance to safety of flights. Continuing airworthiness of aircraft structures is largely based upon the visual detection of small defects made by trained inspection personnel with expensive, critical and time consuming tasks. At this aim, Unmanned Aerial Vehicles (UAVs) can be used for autonomous inspections, as long as it is possible to localize the target while flying around it and correct the position. This work proposes a solution to detect the airplane pose with regards to the UAVs position while flying autonomously around the airframe at close range for visual inspection tasks. The system works by processing images coming from an RGB camera mounted on board, comparing incoming frames with a database of natural landmarks whose position on the airframe surface is known. The solution has been tested in real UAV flight scenarios, showing its effectiveness in localizing the pose with high precision. The advantages of the proposed methods are of industrial interest since we remove many constraint that are present in the state of the art solutions.

Unmanned Aerial Vehicles are currently investigated as an important sub-domain of robotics, a fast growing and truly multidisciplinary research field. UAVs are increasingly deployed in real-world settings for missions in dangerous environments or in environments which are challenging to access. Combined with autonomous flying capabilities, many new possibilities, but also challenges, open up. To overcome the challenge of early identification of degradation, machine learning based on flight features is a promising direction. Existing approaches build classifiers that consider their features to be correlated. This prevents a fine-grained detection of degradation for the different hardware components. This work presents an approach where the data is considered uncorrelated and, using machine learning techniques, allows the precise identification of UAV's damages.

To perform a wide range of tasks service robots need to robustly extract knowledge about the world from the data perceived through the robot's sensors even in the presence of varying context-conditions. This makes the design and development of robot perception architectures a challenging exercise. In this paper we propose a robot perception architecture which enables to select and execute at runtime different perception graphs based on monitored context changes. To achieve this the architecture is structured as a feedback loop and contains a repository of different perception graph configurations suitable for various context conditions.

Aircraft tracking plays a key and important role in the Sense-and-Avoid system of Unmanned Aerial Vehicles (UAVs). This paper presents a novel robust visual tracking algorithm for UAVs in the midair to track an arbitrary aircraft at real-time frame rates, together with a unique evaluation system. This visual algorithm mainly consists of adaptive discriminative visual tracking method, Multiple-Instance (MI) learning approach, Multiple-Classifier (MC) voting mechanism and Multiple-Resolution (MR) representation strategy, that is called Adaptive M3tracker, i.e. AM3. In this tracker, the importance of test sample has been integrated to improve the tracking stability, accuracy and real-time performances. The experimental results show that this algorithm is more robust, efficient and accurate against the existing state-of-art trackers, overcoming the problems generated by the challenging situations such as obvious appearance change, variant surrounding illumination, partial aircraft occlusion, blur motion, rapid pose variation and onboard mechanical vibration, low computation capacity and delayed information communication between UAVs and Ground Station (GS). To our best knowledge, this is the first work to present this tracker for solving online learning and tracking freewill aircraft/intruder in the UAVs.

In this work we present an optimized fuzzy visual servoing system for obstacle avoidance using an unmanned aerial vehicle. The cross-entropy theory is used to optimise the gains of our controllers. The optimization process was made using the ROS-Gazebo 3D simulation with purposeful extensions developed for our experiments. Visual servoing is achieved through an image processing front-end that uses the Camshift algorithm to detect and track objects in the scene. Experimental flight trials using a small quadrotor were performed to validate the parameters estimated from simulation. The integration of cross-entropy methods is a straightforward way to estimate optimal gains achieving excellent results when tested in real flights.

This paper presents a Fuzzy Control application for a landing task of an Unmanned Aerial Vehicle, using the 3D-position estimation based on visual tracking of piecewise planar objects. This application allows the UAV to land on scenarios in which it is only possible to use visual information to obtain the position of the vehicle. The use of the homography permits a realtime estimation of the UAV's pose with respect to a helipad using a monocular camera. Fuzzy Logic allows the definition of a model-free control system of the UAV. The Fuzzy controller analyzes the visual information to generate altitude commands for the UAV to develop the landing task.

This article presents a real time Unmanned Aerial Vehicles UAVs 3D pose estimation method using planar object tracking, in order to be used on the control system of a UAV. The method explodes the rich information obtained by a projective transformation of planar objects on a calibrated camera. The algorithm obtains the metric and projective components of a reference object (landmark or helipad) with respect to the UAV camera coordinate system, using a robust real time object tracking based on homographies. The algorithm is validated on real flights that compare the estimated data against that obtained by the inertial measurement unit IMU, showing that the proposed method robustly estimates the helicopter's 3D position with respect to a reference landmark, with a high quality on the position and orientation estimation when the aircraft is flying at low altitudes, a situation in which the GPS information is often inaccurate. The obtained results indicate that the proposed algorithm is suitable for complex control tasks, such as autonomous landing, accurate low altitude positioning and dropping of payloads.

A neural behavior initiating agent (BIA) is proposed to integrate relevant compressed image information coming from others cooperating and specialized neural agents. Using this arrangement the problem of tracking and recognizing a moving icon has been solved by partitioning it into three simpler and separated tasks. Neural modules associated to those tasks proved to be easier to train and show a good general performance. The obtained neural controller can handle spurious images and solve an acute image related task in a dynamical environment. Under prolonged dead-lock conditions the controller shows traces of genuine spontaneity. The overall performance has been tested using a pan and tilt camera platform and real images taken from several objects, showing the good tracking results discussed in the paper.

In this paper we introduce a real-time trinocular system to control rotary wing Unmanned Aerial Vehicles based on the 3D information extracted by cameras located on the ground. The algorithm is based on key features onboard the UAV to estimate the vehicle's position and orientation. The algorithm is validated against onboard sensors and known 3D positions, showing that the proposed camera configuration robustly estimates the helicopter's position with an adequate resolution, improving the position estimation, especially the height estimation. The obtained results show that the proposed algorithm is suitable to complement or replace the GPS-based position estimation in situations where GPS information is unavailable or where its information is inaccurate, allowing the vehicle to develop tasks at low heights, such as autonomous landing, take-off, and positioning, using the extracted 3D information as a visual feedback to the flight controller.

This paper presents an implementation of two Fuzzy Logic controllers working in parallel for a pan-tilt camera platform on an UAV. This implementation uses a basic Lucas-Kanade tracker algorithm, which sends information about the error between the center of the object to track and the center of the image, to the Fuzzy controller. This information is enough for the controller to follow the object by moving a two axis servo-platform, regardless the UAV vibrations and movements. The two Fuzzy controllers for each axis, work with a rules-base of 49 rules, two inputs and one output with a more significant sector defined to improve the behavior of those controllers. The controllers have shown very good performances in real flights for statics objects, tested on the Colibri prototypes.