Autonomous spacecraft guidance for proximity operations can present significant challenges in uncertain environments, particularly in terms of safety and robustness. This paper proposes a novel deep reinforcement learning (DRL)-based guidance system that integrates Adaptive Domain Randomization (ADR) and relaxed Control Barrier Functions (CBFs) to enhance adaptability and enforce safety constraints. ADR systematically expands the range of environmental parameters during training, enabling the agent to generalize effectively to diverse conditions. CBFs provide a theoretical safety guarantee, ensuring stable manoeuvring even in failure scenarios. The proposed approach is evaluated in a simulated spacecraft proximity operation under nominal and degraded conditions, including actuator breakdown and sensor degradation. Results demonstrate that the proposed DRL agent configuration outperforms the baseline DRL approach, maintaining a lower relative position error and angular error.

Semantic segmentation is vital for autonomous car scene understanding. It provides more precise subject information than raw RGB images and this, in turn, boosts the performance of autonomous driving. Recently, self-attention methods show great improvement in image semantic segmentation. Attention maps help scene parsing with abundant relationships of every pixel in an image. However, it is computationally demanding. Besides, existing works focus either on channel attention, ignoring the pixel position factors, or on spatial attention, disregarding the impacts of the channels on each other. To address these problems, we present Fusion Attention Network based on self-attention mechanism to harvest rich contextual dependencies. This model consists of two chains: pyramid fusion spatial attention and fusion channel attention. We apply pyramid sampling in the spatial attention module to reduce the computation for spatial attention maps. Channel attention has a similar structure to the spatial attention. We also introduce a fusion technique to calculate contextual dependencies using features from both attention chains. We concatenate the results from spatial and channel attention modules as the enhanced attention map, leading to better semantic segmentation results. We conduct extensive experiments on popular datasets with different settings in addition to an ablation study to prove the efficiency of our approach. Our model achieves better results, on Cityscapes [7], compared to state-of-the-art methods, and also show good generalization capability on PASCAL VOC 2012 [9].

This article presents a novel flight assistance system, obstacle avoidance system for teleoperation (OAST), whose main role is to make teleoperation of small multirotor unmanned aerial vehicles (UAVs) safer and more efficient in closed spaces. The OAST allows the operator to avoid obstacles while keeping a liberty of movement. The UAV is controlled through a 3-D haptic controller and the OAST amends the user input to increase safety and efficiency of the teleoperation. The design of the OAST is verified in computerized experiments. Moreover, a simulation involving 20 participants is carried out to validate the proposed scheme. This experiment shows that the OAST improves the completion time of the scenarios by 41% on average while reducing the workload of the operator from 57 to 27 points on the NASA Task Load Index test. The number of collisions with the environment is all but eliminated in these scenarios.

Besides the large amount of information multispectral imaging offers, multispectral visual odometry remains overlooked due to the dissimilarity between modalities. In order to tackle the challenging feature matching between multispectral stereo images and to overcome the lack of robust multispectral visual localisation solutions, a novel approach is proposed in this paper. It consists in tracking features in each modality simultaneously, in a monocular manner and then, estimating motion in a windowed bundle adjustment framework and using the geometry of the stereo setup to recover the missing scale. The estimation is robustified by selecting adequate keyframes based on feature parallax and by maximising the mutual information between all the features reprojected in the stereo pair. Furthermore, the proposed multispectral visual odometry solution is integrated in an error-state Kalman filter framework to deal with challenging environments, where the quality of images is reduced. Two measurements models, using absolute and relative camera poses, are presented. The superiority of relative poses is then shown by providing a failure recovery algorithm which relies on inertial data when visual data are not accessible. The algorithm was tested on innovative series of visible-thermal multispectral datasets, acquired from a car with real driving conditions. An overall error of around 2% of the travelled distance was achieved on these datasets.

Planetary robotics navigation has attracted the great attention of many researchers in recent years. Localization is one of the most important problems for robots on another planet in the lack of GPS. The robots need to be able to know their location and the surrounding map in the environment concurrently, to work and communicate together on another planet. In the current work, a novel algorithm is designed to cooperatively localize a team of robots on another planet. Consequently, a robust algorithm is developed for cooperative Visual Odometry (VO) to localize each robot in a planetary environment while detecting both intra-loop closure and inter-loop closures using previously observed area by the robot and shared area from other robots, respectively. To validate the proposed algorithm, a comparison is provided between the proposed cooperative VO and the single version of VO. Accordingly, a planetary analogue real dataset is employed to investigate the accuracy of the proposed algorithm. The results promise the concept of cooperative VO to significantly increase the accuracy of localization.

Research on visual odometry and localisation is largely dominated by solutions developed in the visible spectrum, where illumination is a critical factor. Other parts of the electromagnetic spectrum are currently being investigated to generate solutions dealing with extreme illumination conditions. Multispectral setups are particularly interesting as they provide information from different parts of the spectrum at once. However, the main challenge of such camera setups is the lack of similarity between the images produced, which makes conventional stereo matching techniques obsolete.This work investigates a new way of concurrently processing images from different spectra for application to visual odometry. It particularly focuses on the visible and Long Wave InfraRed (LWIR) spectral bands where dissimilarity between pixel intensities is maximal. A new Multimodal Monocular Visual Odometry solution (MMS-VO) is presented. With this novel approach, features are tracked simultaneously, but only the camera providing the best tracking quality is used to estimate motion. Visual odometry is performed within a windowed bundle adjustment framework, by alternating between the cameras as the nature of the scene changes. Furthermore, the motion estimation process is robustified by selecting adequate keyframes based on parallax.The algorithm was tested on a series of visible-thermal datasets, acquired from a car with real driving conditions. It is shown that feature tracking could be performed in both modalities with the same set of parameters. Additionally, the MMS-VO provides a superior visual odometry trajectory as one camera can compensate when the other is not working.

In this paper, a bio-inspired monocular vision perception method combined with a learning-based reaction local planner for obstacle avoidance of micro UAVs is presented. The system is more computationally efficient than other vision-based perception and navigation methods such as SLAM and optical flow because it does not need to calculate accurate distances. To improve the robustness of perception against illuminance change, the input image is remapped using image moment which is independent of illuminance variation. After perception, a local planner is trained using deep reinforcement learning for mapless navigation. The proposed perception and navigation methods are evaluated in some realistic simulation environments. The result shows that this light-weight monocular perception and navigation system works well in different complex environments without accurate depth information.

Three‐dimensional (3D) local feature detection and description techniques are widely used for object registration and recognition applications. Although several evaluations of 3D local feature detection and description methods have already been published, these are constrained in a single dimensional scheme, i.e. either 3D or 2D methods that are applied onto multiple projections of the 3D data. However, cross‐dimensional (mixed 2D and 3D) feature detection and description are yet to be investigated. Here, the authors evaluated the performance of both single and cross‐dimensional feature detection and description methods on several 3D data sets and demonstrated the superiority of cross‐dimensional over single‐dimensional schemes.

The 3-D automatic target recognition (ATR) has many advantages over its 2-D counterpart, but there are several constraints in the context of small low-cost unmanned aerial vehicles (UAVs). These limitations include the requirement for active rather than passive monitoring, high equipment costs, sensor packaging size, and processing burden. We, therefore, propose a new structure from motion (SfM) 3-D ATR architecture that exploits the UAV's onboard sensors, i.e., the visual band camera, gyroscope, and accelerometer, and meets the requirements of a small UAV system. We tested the proposed 3-D SfM ATR using simulated UAV reconnaissance scenarios and found that the performance was better than classic 3-D light detection and ranging (LIDAR) ATR, combining the advantages of 3-D LIDAR ATR and passive 2-D ATR. The main advantages of the proposed architecture include the rapid processing, target pose invariance, small template size, passive scene sensing, and inexpensive equipment. We implemented the SfM module under two keypoint detection, description and matching schemes, with the 3-D ATR module exploiting several current techniques. By comparing SfM 3-D ATR, 3-D LIDAR ATR, and 2-D ATR, we confirmed the superior performance of our new architecture.

In this paper, a real-time setup and an implementation of a Proportional Derivative (PD) controller for orientation and comparison between PD and BackStepping (BS) controllers for linear positioning are presented using a Pelican quadrotor from Ascending Technologies (AscTec). An onboard Inertial Measurement Unit (IMU) was used for orientation control and Optitrack Vision Tracking System for linear positioning control. A linear Kalman filter was implemented for linear velocity estimation. The software and hardware integration was achieved with the help of the Robot Operating System (ROS). Simulations and experiments with this drone platform are achieved in order to implement different controller algorithms and analyse them in order to achieve better aircraft performance.

Unmanned aerial vehicles (UAV) equipped with a navigation system and an embedded camera can be used to estimate the position of a desired target. The relative position of the UAV along with knowledge of camera orientation and imagery data can be used to produce bearing measurements that allow estimation of target position. The filter methods applied are prone to biases due to noisy measurements. Further noise may be encountered depending on the UAV trajectory for target localisation. This work presents the implementation of an Unscented Kalman Filter (UKF) to estimate the position of a target on the 3D cartesian plane within a small indoor scenario. A small UAV with a single board computer, equipped with a frontal camera and moving in an oval trajectory at a fixed height was employed. Such a trajectory enabled an experimental comparison of UAV simulation data with UAV real-time flight data for indoor conditions. Optitrack Motion system and the Robot Operative System (ROS) were used to retrieve the drone position and exchange information at high rates.

Situational awareness is increasingly important across many applications. Having a more adept sense of situational awareness leads to better understanding and prediction in various scenarios. This is apparent with the increasing use of infrared cameras. The benefits of infrared imaging make it an attractive option for use in ground vehicles. However, they are limited in their field-of-views. We propose an automated mosaicing method, to improve situational awareness, using infrared images from a vehicle mounted camera. Within our method we also propose a novel key frame selection algorithm, for efficient real time mosaicing. We validate our algorithm using different driving speeds, showing that it is robust across different driving scenarios.

3D object recognition is proven superior compared to its 2D counterpart with numerous implementations, making it a current research topic. Local based proposals specifically, although being quite accurate, they limit their performance on the stability of their local reference frame or axis (LRF/A) on which the descriptors are defined. Additionally, extra processing time is demanded to estimate the LRF for each local patch. We propose a 3D descriptor which overrides the necessity of a LRF/A reducing dramatically processing time needed. In addition robustness to high levels of noise and non-uniform subsampling is achieved. Our approach, namely Histogram of Distances is based on multiple L2-norm metrics of local patches providing a simple and fast to compute descriptor suitable for time-critical applications. Evaluation on both high and low quality popular point clouds showed its promising performance.

With the recent increase of interest concerning multi-spectral systems, limitations of classic stereo setups have been overcome to tackle complex navigation problems such as night-time navigation or collision avoidance. However, multi-spectral stereo matching still remains a challenging issue as similarity between stereo pairs is reduced. In this work, we address the problem of visual navigation for multi-modal stereo setups. More precisely, we focus on finding correspondences between visible and long-wave infrared (thermal) images for pose estimation purposes. We present a new visual odometry method for air vehicles using mutual information and phase congruency. Mutual information performs an efficient cross-modality statistical analysis and phase congruency provides a robust spatial information. Hence, both of them are used as criteria for multi-spectral stereo matching. Temporal matching is then performed using feature tracking in each modality separately. This way, cross-modality processing is kept to a minimum, which reduces inaccuracy. This method provides an important number of quad-matches (stereo and temporal), which helps to select the best keypoints for computing a precise motion estimation. In our case, the selection process is performed using a RANSAC scheme. Extensive experimental results show the attractiveness of the proposed technique as a navigation solution. Moreover, results show that tracking features in both modalities makes the technique robust to abrupt navigation changes.

In this paper, we propose a novel image representation approach to tackle illumination variations in stereo matching problems. Images are mapped using their Fourier transforms which are convolved with a set of monogenic filters. Frequency analysis is carried out at different scales to account for most image content. The phase congruency and the local weighted mean phase angle are then computed over all the scales. The original image is transformed into a new representation using these two mappings. This representation is invariant to illumination and contrast variations. More importantly, it is generic and can be used with most sparse as well as dense stereo matching algorithms. In addition, sequential feature matching or tracking can also benefit from our approach in varying radiometric conditions. We demonstrate the improvements introduced with our image mappings on well-established data sets in the literature as well as on our own experimental scenarios that include high dynamic range imagery. The experiments include both dense and sparse stereo and sequential matching algorithms where the latter is considered in the very challenging visual odometry framework.

In this paper, we investigate the problem of visual odometry for ground vehicles based on the simultaneous utilization of multispectral cameras. It encompasses a stereo rig composed of an optical (visible) and thermal sensors. The novelty resides in the localization of the cameras as a stereo setup rather than two monocular cameras of different spectrums. To the best of our knowledge, this is the first time such task is attempted. Log-Gabor wavelets at different orientations and scales are used to extract interest points from both images. These are then described using a combination of frequency and spatial information within the local neighborhood. Matches between the pairs of multimodal images are computed using the cosine similarity function based on the descriptors. Pyramidal Lucas-Kanade tracker is also introduced to tackle temporal feature matching within challenging sequences of the data sets. The vehicle egomotion is computed from the triangulated 3-D points corresponding to the matched features. A windowed version of bundle adjustment incorporating Gauss-Newton optimization is utilized for motion estimation. An outlier removal scheme is also included within the framework to deal with outliers. Multispectral data sets were generated and used as test bed. They correspond to real outdoor scenarios captured using our multimodal setup. Finally, detailed results validating the proposed strategy are illustrated.

In this work, fast and efficient appearance-based methods for visual loop-closure detection are proposed. The widely used technique based on the Bag-of-Words image representation has shown some limitations especially with aliasing problem. In this work, however, an appearance-based approach for loop closure detection using local invariant and colour features is proposed. The first technique uses Bayes Decision Theory for loop closure detection based on Gaussian Mixture Model (GMM). A new technique based on the combination of GMM with the KD-Tree data structure is presented as well. The techniques have been validated using monocular image sequences from several environments.

This paper presents an Image Based Visual Servo control design for Fixed Wing Unmanned Aerial Vehicles tracking locally linear infrastructure in the presence of wind using a body fixed imaging sensor. Visual servoing offers improved data collection by posing the tracking task as one of controlling a feature as viewed by the inspection sensor, although is complicated by the introduction of wind as aircraft heading and course angle no longer align. In this work it is shown that the effects of wind alter the desired line angle required for continuous tracking to equal the wind correction angle as would be calculated to set a desired course. A control solution is then sort by linearizing the interaction matrix about the new feature pose such that kinematics of the feature can be augmented with the lateral dynamics of the aircraft, from which a state feedback control design is developed. Simulation results are presented comparing no compensation, integral control and the proposed controller using the wind correction angle, followed by an assessment of response to atmospheric disturbances in the form of turbulence and wind gusts.

This paper proposes a complete strategy to optimally filter, enhance and register 3D point clouds captured by commodity RGBD cameras. Starting from the raw data grabbed from multiple viewpoints, we build the scene that gathers all the clouds in one consistent view. The process begins with the innovative adaptation of Kalman filter to Kinect's output. The resulting point cloud is subject to an outlier removal technique and a pre-alignment based on 3D features is performed. Finally, the alignment is refined using Iterative Closest Point (ICP) algorithm. The output of this research work is a consistent 3D model which can be directly used in virtual reality applications, or any 3D rendering process. Test results on real data are presented to validate our approach, and to justify the choice of its different modules.

Commodity RGBD cameras such as Kinect sensor have recently proven a large success in many indoor robotics and computer vision applications. Nevertheless, tracking and motion estimation algorithms cannot rely on Kinect raw outputs because of their low accuracy. These consumer cameras can only produce precise depth measures within a close range. However, they do suffer from potential noises when the target is further away from permitted. This paper proposes an innovative adaptation of Kalman filtering scheme to improve the accuracy of Kinect as a real-time tracking device. We present a detailed proof of Kalman filter adaptation on Kinect data, and we demonstrate the robustness of our approach on a real dataset.

In this paper, we propose and compare two control schemes to achieve the visual tracking of structured linear infrastructures using a quad rotor UAV. The two schemes are both based on Hough parameters. The first one has the same advantages of Position Based Visual Servoing (PBVS) scheme, without the need of full 3D position estimation, while the second is an Image Based Visual Servoing (IBVS) scheme based on an optimal LQR servo controller. Simulation results based on a full nonlinear model of the quad rotor show the validity of the two proposed approaches and their main advantages.

Acoustic data have already been used for security (and military) applications such as sniper detection and gun firing localization, pedestrians and cars detection and tracking. However, in some situations, the measurement results of the acoustic data (range and bearing) alone appear to be insufficient to locate accurately targets especially because of the presence of the background noise or due to the effect of the reverberation on estimating the true time delay of arrival of the acoustic wave. Furthermore, the influence of the errors related to the sensors synchronisation is also negatively impacting. In this work, a recursive technique based on Unscented Kalman Filter (UKF) will be used to estimate the position of the acoustic source, in addition to using video data to improve the accuracy of the acoustic measurement by fusing the two types of data (acoustic and video). This combination showed through experimental results improvement in terms of target localisation accuracy.

This paper proposes a novel feature descriptor combining signatures of intensity and colour for increased descriptive power. The proposed approach, instead of just the grey channel, uses all three channels of an RGB image, colour codes pixels around the feature corresponding to Gaussian clusters on a CIExy colour chart and then distributes them into bins combined with an intensity gradient histogram. It is shown for a number of examples - 12 image sets - that this descriptor, dubbed HISTInC, outperforms a standalone gradient histogram for a number of conditions, giving on average a 27% increase in precision. Furthermore, in the interest of preserving memory, HIST-InC with a reduced (by 22%) intensity signature is also tested and yields mixed results.

Recent developments in the infrared industry and the availability of relatively cheaper infrared cameras attracted more attention towards the simultaneous utilization of visible and infrared images and solving the correspondence problem between them. The Microsoft Kinect sensor represents one of the most widespread examples of low cost multimodal setups. In this context, we investigate the feasibility of matching features extracted from cross-spectral stereo cameras using a sparse approach. First, a set of stable features are extracted from both images using the Scale Invariant Feature Transform (SIFT). Then, descriptors are computed from local self-similarities around the selected key points. The combination of image moments is also investigated and expected to improve the matching process. Experimental results show the performance of this approach to the multimodal correspondence challenge.

The problem treated in this paper is about optimal visual sensors placement and deployment with estimation of the appropriate location of an interceptor to be placed. The main objective of this research is to ensure accurate coverage of the monitoring space with a minimum number of directional “field-of-view” (FOV) cameras and in the same time to decrease the interception time of an intruder at any entry of a target traveling between two areas. Firstly, we define the coverage problem by realistic and consistent assumptions taking into account the capabilities and limits of the cameras. Then, a novel method based on Binary Particle Swarm Optimization (BPSO) and inspired probability is proposed for solving the cameras placement for coverage problem. The performances of the proposed approach are discussed and compared with several stochastic algorithms such as genetic algorithms, immune system and some adapted versions of evolutionary algorithms based on the BPSO. In the second part of the paper, we address an improved optimal polynomial time algorithm for computing the worst-case breach coverage in directional FOV sensor networks, which represent the best position of an interceptor to be placed.

It has been known that time delays may not only degrade the performance of the closed-loop control systems, but also induce instability due to the introduction of extra phase lag. The focus of this paper is to find out a way to design a robust controller for networked control systems (NCSs) with time-varying communication delays. To this aim, a mixed fuzzy-PID/neural network compensating scheme is proposed so as to alleviate the influence resulting from uncertain communication delays while maintaining system performance. A condition ensuring stability of the NCS is derived from the system's input–output viewpoint. The effectiveness and superiority of our proposed approach are fully verified in an experimentally networked pneumatic control system.

Most optical flow algorithms are based on the assumption of brightness constancy of individual pixels while moving on the image plane. Although being attractive analytically, this assumption is often violated under non-ideal visual conditions resulting in poor flow estimates. This paper presents an approach to support the validity of the assumption under such conditions. The method describes the grey-level of each pixel by the content of its neighbourhood using a geometric moment rather than its individual intensity function value. Then, the description of each pixel is normalised and made insensitive to fluctuations of intensity. As a result, the optical flow algorithm becomes much more reliable and robust against visual phenomena like varying illumination, specular reflections and shadows. The proposed approach is applied to a regression method and comprehensive results on synthetic and real data are reported. 1

Capture mechanisms are used to transfer objects between two vehicles in the space with no physical contact. A sphere (canister) detection and tracking method using an enhanced Hough transform technique and Hinfinfilter is proposed. The presented system aims to assist in the capture operation, currently investigated the European Space Agency and other partners, and to be used in space missions as an alternative to docking or berthing operations. Test results show the robustness and reliability of the proposed method. They also demonstrate the low computational and memory complexities needed.