We present a pursuit law for capturing a moving target with multiple aerial drones in a bounded environment. The objective is to develop strategies to allow a set of drones to perform a mission cooperatively to complete the task. The presented method combines two established pursuit laws: Group Deviated Pure Pursuit (GDP) and Proportional Navigation Guidance (PNG), and the resulting control architecture is in a cascade form. The results were validated in simulation and experimentation on quadrotor aerial drones. The obtained results confirmed the efficacy of the group mixed pursuit strategy, especially in the case of an agile and faster target. A video of the experimentation result can be seen on: https://youtu.be/cRKRUOV-lV4.

Fusing geometric cues with visual appearance is an imperative theme for RGB-D indoor semantic segmentation . Existing methods commonly adopt convolutional modules to aggregate multi-modal features, paying little attention to explicitly leveraging the long-range dependencies in feature fusion . Therefore, it is challenging for existing methods to accurately segment objects with large-scale variations. In this paper, we propose a novel transformer-based fusion scheme, named TransD-Fusion, to better model contextualized awareness. Specifically, TransD-Fusion consists of a self-refinement module, a calibration scheme with cross-interaction, and a depth-guided fusion. The objective is to first improve modality-specific features with self- and cross-attention, and then explore the geometric cues to better segment objects sharing a similar visual appearance. Additionally, our transformer fusion benefits from a semantic-aware position encoding which spatially constrains the attention to neighboring pixels . Extensive experiments on RGB-D benchmarks demonstrate that the proposed method performs well over the state-of-the-art methods by large margins.

RGB-D saliency detection aims to fuse multi-modal cues to accurately localize salient regions. Existing works often adopt attention modules for feature modeling, with few methods explicitly leveraging fine-grained details to merge with semantic cues. Thus, despite the auxiliary depth information, it is still challenging for existing models to distinguish objects with similar appearances but at distinct camera distances. In this paper, from a new perspective, we propose a novel Hierarchical Depth Awareness network (HiDAnet) for RGB-D saliency detection. Our motivation comes from the observation that the multi-granularity properties of geometric priors correlate well with the neural network hierarchies. To realize multi-modal and multi-level fusion, we first use a granularity-based attention scheme to strengthen the discriminatory power of RGB and depth features separately. Then we introduce a unified cross dual-attention module for multi-modal and multi-level fusion in a coarse-to-fine manner. The encoded multi-modal features are gradually aggregated into a shared decoder. Further, we exploit a multi-scale loss to take full advantage of the hierarchical information. Extensive experiments on challenging benchmark datasets demonstrate that our HiDAnet performs favorably over the state-of-the-art methods by large margins. The source code can be found in https://github.com/Zongwei97/HIDANet/.

Efficiently exploiting multi-modal inputs for accurate RGB-D saliency detection is a topic of high interest. Most existing works leverage cross-modal interactions to fuse the two streams of RGB-D for intermediate features' enhancement. In this process, a practical aspect of the low quality of the available depths has not been fully considered yet. In this work, we aim for RGB-D saliency detection that is robust to the low-quality depths which primarily appear in two forms: inaccuracy due to noise and the misalignment to RGB. To this end, we propose a robust RGB-D fusion method that benefits from (1) layer-wise, and (2) trident spatial, attention mechanisms. On the one hand, layer-wise attention (LWA) learns the trade-off between early and late fusion of RGB and depth features, depending upon the depth accuracy. On the other hand, trident spatial attention (TSA) aggregates the features from a wider spatial context to address the depth misalignment problem. The proposed LWA and TSA mechanisms allow us to efficiently exploit the multi-modal inputs for saliency detection while being robust against low-quality depths. Our experiments on five bench-mark datasets demonstrate that the proposed fusion method performs consistently better than the state-of-the-art fusion alternatives. The source code is publicly available at: https://github.com/Zongwei97/RFnet.

Deep Reinforcement Learning (DRL) is highly efficient for solving complex tasks such as drone obstacle avoidance using cameras. However, these methods are often limited by the camera perception capabilities. In this paper, we demonstrate that point-goal navigation performances can be improved by using cameras with a wider Field-Of-View (FOV). To this end, we present a DRL solution based on equirectangular images and demonstrates its relevance, especially compared to its perspective version. Several visual modalities are compared: ground truth depth, RGB, and depth directly estimated from these$360^{\circ}$RGB images using Deep Learning methods. Next, we propose a spherical adaptation to take into account the spherical distortions of omnidirectional images in the convolutional neural networks (CNNs) used in the actor-critic network and show a significant improvement in navigation performance. Finally, we modify the perspective depth estimation network using this spherical adaptation and demonstrate a further performance improvement.

Solving Direct Shooting Model Predictive Control (MPC) optimization problems online can be computationally expensive if a large horizon is used while also maintaining a dense time sampling. In these cases, it is accepted that tradeoffs between computational load and performances should be sought in order to meet real-time feasibility requirements. However, making the problem more tractable for the hardware should not necessarily imply a decrease in performances. One technique that has been proposed in the literature makes use of control input parameterizations to decrease the numerical complexity of nonlinear MPC problems without necessarily affecting the performances significantly. In this paper, we review the use of parameterizations and propose a simple Sequential Quadratic Programming algorithm for nonlinear MPC. We then benchmark the performances of the solver in simulation, showing that parameterizations allow to attain good performances with (significantly) lower computation times than state-of-the-art solvers.

Recent RGBD-based models for saliency detection have attracted research attention. The depth clues such as boundary clues, surface normal, shape attribute, etc., contribute to the identification of salient objects with complicated scenarios. However, most RGBD networks require multi-modalities from the input side and feed them separately through a two-stream design, which inevitably results in extra costs on depth sensors and computation. To tackle these inconveniences, we present in this paper a novel fusion design named modality-guided subnetwork (MGSnet). It has the following superior designs: 1) Our model works for both RGB and RGBD data, and dynamically estimates depth if not available. Taking the inner workings of depth-prediction networks into account, we propose to estimate the pseudo-geometry maps from RGB input — essentially mimicking the multi-modality input. 2) Our MGSnet for RGB SOD results in real-time inference but achieves state-of-the-art performance compared to other RGB models. 3) The flexible and lightweight design of MGS facilitates the integration into RGBD two-streaming models. The introduced fusion design enables a cross-modality interaction to enable further progress but with a minimal cost.

Visual servoing control schemes, such as Image-Based (IBVS), Pose Based (PBVS) or Hybrid-Based (HBVS) have been extensively developed over the last decades making possible their uses in a large number of applications. It is well-known that the main problems to be handled concern the presence of local minima or singularities, the visibility constraint, the joint limits, etc. Recently, Model Predictive Path Integral (MPPI) control algorithm has been developed for autonomous robot navigation tasks. In this paper, we propose a MPPI-VS framework applied for the control of a 6-DoF robot with 2D point, 3D point, and Pose Based Visual Servoing techniques. We performed intensive simulations under various operating conditions to show the potential advantages of the proposed control framework compared to the classical schemes. The effectiveness, the robustness and the capability in coping easily with the system constraints of the control framework are shown.

Recently, Model Predictive Path Integral (MPPI) control algorithm has been extensively applied to autonomous navigation tasks, where the cost map is mostly assumed to be known and the 2D navigation tasks are only performed. In this paper, we propose a generic MPPI control framework that can be used for 2D or 3D autonomous navigation tasks in either fully or partially observable environments, which are the most prevalent in robotics applications. This framework exploits directly the 3D-voxel grid acquired from an on-board sensing system for performing collision-free navigation. We test the framework, in realistic RotorS-based simulation, on goal-oriented quadrotor navigation tasks in a cluttered environment, for both fully and partially observable scenarios. Preliminary results demonstrate that the proposed framework works perfectly, under partial observability, in 2D and 3D cluttered environments.

Spherical cameras and the latest image processing techniques open up new horizons. In particular, methods based on Convolutional Neural Networks (CNNs) now give excellent results for optical flow estimation on perspective images. However, these approaches are highly dependent on their architectures and training datasets. This paper proposes to benefit from years of improvement in perspective images optical flow estimation and to apply it to omnidirectional ones without training on new datasets. Our network, OmniFlowNet, is built on a CNN specialized in perspective images. Its convolution operation is adapted to be consistent with the equirectangular projection. Tested on spherical datasets created with Blender1and several equirectangular videos realized from real indoor and outdoor scenes, OmniFlowNet shows better performance than its original network without extra training.

We consider the problem in which a supervisor or remote pilot provides a real-time linear velocity reference to a multirotor aerial robot, either through a traditional remote control handset, a modern haptic interface, or semi-autonomous guidance control system. In all such cases, the goal is to servo-control the vehicle's velocity to the set point as quickly and as efficiently as possible. The challenge is to achieve this robustly in the presence of unknown wind disturbances and in situations in which the vehicle moves into global position system (GPS) denied environments (indoors, urban canyons, forests) where estimation of the vehicle's velocity is challenging. These situations include unclutterred environments, poor visibility environments caused by poor lighting, and poorly textured visual environments where laser- and vision-based sensors become unreliable. The approach taken is to develop a coupled nonlinear complementary velocity aided attitude filter that provides estimates of both the inertial and body-fixed frame linear velocities, as well as the attitude of a multirotor aerial vehicle, that functions effectively even when only the inertial measurement unit and barometric sensor measurements are available. When full inertial velocity measurements are available (from GPS, Vicon, or a vision system), the filter additionally estimates the external wind speed. In this paper, we formally present the proposed filter along with experimental results and a comparison of the filter to recent results in the literature and in situations in which inertial reference frame velocities are available intermittently. The proposed filter is computationally simple to implement, easy to calibrate and tune, and provides an excellent base level functionality for modern multirotor aerial robotic systems that will be required to function robustly in a variety of environments.

A nonlinear inertial-aided image-based visual servo control approach for the stabilization of (almost) fully actuated autonomous underwater vehicles (AUVs) is proposed. It makes use of the homography matrix between two images of a planar scene as feedback information while the system dynamics are exploited in a cascade manner in a control design: An outer-loop control defines a reference setpoint based on the homography matrix and an inner-loop control ensures the stabilization of the setpoint by assigning the thrust and torque controls. Unlike conventional solutions that only consider the system kinematics, the proposed control scheme is novel in considering the full system dynamics (incorporating all degrees of freedom, nonlinearities, and couplings, as well as interactions with the surrounding fluid) and in not requiring information of the relative depth and normal vector of the observed scene. Augmented with integral corrections, the proposed controller is robust with respect to model uncertainties and disturbances. The almost global asymptotic stability of the closed-loop system is demonstrated, which is the largest domain of attraction one can achieve by means of continuous feedback control. Simulation results illustrating these properties on a realistic AUV model subjected to a sea current are presented and finally experimental results on a real AUV are reported.

Flight performance of aerial robotic vehicles is critically dependent on the quality of the state estimates provided by onboard sensor systems. The attitude estimation problem has been extensively studied over the last ten years and the development of low complexity, high performance, robust non-linear observers for attitude has been one of the enabling technologies fueling the growth of small scale aerial robotic systems. The velocity aided attitude estimation problem, that is simultaneous estimation of attitude and linear velocity of an aerial platform, has only been tackled using the non-linear observer approach in the last few years. Prior contributions have lead to non-linear observers for which either there is no stability analysis or for which the analysis is extremely complex. In this paper, we propose a simple relaxation of the state space, allowing scaled rotation matrices R ∈ ℝ3×3such that RXT= uI where X = uR̂ and u > 0 is a positive scalar, along with additional observer dynamics to force u → 1 asymptotically. With this simple augmentation of the observer state space, we propose a non-linear observer with a straightforward Lyapunov stability analysis that demonstrates almost global asymptotic convergence along with local exponential convergence. Simulations as well as experimental results are provided to demonstrate the performance of the proposed observer.

This paper deals with state feedback control of chained systems based on a Nonlinear Model Predictive Control (NMPC) strategy. Chained systems can model many common nonholonomic vehicles. We establish a relation between the degree of nonholonomy and the minimum length of the control horizon so as to make the NMPC feasible. A necessary condition on the control horizon of NMPC is given and theoretically proved whatever the dimension of the chained system considered. This relation is used to design a NMPC-based control strategy for chained systems. One of the advantages of NMPC is the capability of taking into account the constraints on state and on control variables. The theoretical results are illustrated through simulations on a (2,5) chained system, describing a car-like vehicle with one trailer. Difficult motion objectives that require a lateral displacement are considered.

Image-based visual servoing (IBVS) is now recognized to be an efficient and robust control strategy to guide robots using only visual data. Classical IBVS is commonly based on the Cartesian coordinates of points in the image. Although the convergence of the visual features to the desired ones is generally achieved, classical IBVS can lead to unnecessary displacements of the camera, such as the camera retreat problem in the case of a pure rotation around the optical axis. In contrast, IBVS based on the polar coordinates of points, is well adapted to carry out rotations around the optical axis but less adapted to manage translations. To take advantage of the benefits of each approach, we propose a new strategy for visual servoing based on a switching controller. The controller switches between Cartesian-based and polar-based approaches thanks to a switching signal provided by a decision maker. With the proposed controller, unnecessary 3D displacements are minimized without any 3D reconstruction and visibility constraints can be taken into account. The local stability of the closed-loop switching system is proved. A comparison with classical controllers and advanced controllers is performed by simulations. This comparative study illustrates the effectiveness of the proposed controller in terms of displacement in the image space and in the 3D space.

This paper deals with the image-based visual servoing (IBVS), subject to constraints. Robot workspace limitations, visibility constraints, and actuators limitations are addressed. These constraints are formulated into state, output, and input constraints, respectively. Based on the predictive-control strategy, the IBVS task is written into a nonlinear optimization problem in the image plane, where the constraints can be easily and explicitly taken into account. Second, the contribution of the image prediction and influence of the prediction horizon are pointed out. The image prediction is obtained due to a model. The latter can be a local model based on the interaction matrix or a nonlinear global model based on 3-D data. Its choice is discussed with respect to the constraints to be handled. Finally, simulations that were obtained with a 6-degree-of-freedom (DOF) free-flying camera highlight the potential advantages of the proposed approach with respect to the image prediction and the constraint handling.

The purpose of this paper is to show what image prediction can bring to image-based visual servoing. The visual feature prediction is obtained thanks to the interaction matrix. Based on a model predictive control strategy, the visual servoing task is formulated into an optimization problem. The error between the reference features and the predicted features is to be minimized over a receding prediction horizon. Numerous simulations highlight the interest of prediction, especially for difficult configurations such as large motion and rotation.

This paper deals with image based visual servoing (IBSV) by a visual predictive control (VPC) approach. Based on nonlinear model predictive control (NMPC), the visual servoing problem is formulated into a nonlinear constrained minimization problem in the image plane. A global model describing the behavior of the robotic system equipped with the camera is used to predict the evolution of the visual feature on a future horizon. The main interest of this method is the capability to easily take into account different constraints like mechanical limitations and/or visibility constraints. Simulation experiments are performed on a planar manipulator with an omnidirectional camera. Comparisons with the classical control law based on the interaction matrix highlight the efficiency and the robustness of the proposed approach, especially in difficult initial configurations and large displacements.