In this paper, we present a new compact vision sensor consisting of two fisheye event cameras mounted back-to-back, which offers a full 360-degree view of the surrounding environment. We describe the optical design, projection model and practical calibration using the incoming stream of events, of the novel stereo camera, called SFERA. The potential of SFERA for real-time target tracking is evaluated using a Bayesian estimator adapted to the geometry of the sphere. Real-world experiments with a prototype of SFERA, including two synchronized Prophesee EVK4 cameras and a DJI Mavic Air 2 quadrotor, show the effectiveness of the proposed system for aerial surveillance.

International audience

Autonomous driving systems require robust object detection in complex environments. Event cameras outperform RGB cameras under challenging lighting conditions, but face limitations due to the scarcity of available datasets and lack of specialized training. To narrow the gap between RGB- and event-based detection accuracy and avoid the high complexity of real-time RGB-event fusion, in this paper, we propose a knowledge distillation framework. Our approach uses both modalities during training but relies solely on sparse event data at inference and transfers knowledge from a robust RGB-based teacher model. We build on the success of DETR (DEtection TRansformer) and we leverage an event-aware masked knowledge distillation mechanism, to boost event-based detection accuracy. Experiments on the DSEC-DET dataset demonstrate that our method not only excels in challenging driving scenarios where RGB images are unreliable, but also surpasses the state-of-the-art in event-based object detection.

In this article, we present a new spherical image representation, called uniform spherical mapping of omnidirectional images (UniphorM), and show its strong potential in robotic vision. UniphorM provides an accurate and distortion-free representation of a 360-degree image, by relying on multiple subdivisions of an icosahedron and its associated Voronoi diagrams. The geometric mapping procedure is described in detail, and the tradeoff between pixel accuracy and computational complexity is investigated. To demonstrate the benefits of UniphorM in real-world problems, we applied it to direct visual attitude estimation and visual place recognition (VPR), by considering dual-fisheye images captured by a camera mounted on multiple robotic platforms. In the experiments, we measured the impact of the number of subdivision levels of the icosahedron on the attitude estimation error, time efficiency, and size of convergence domain of an existing visual gyroscope, using UniphorM and three competing mapping algorithms. A similar evaluation procedure was carried out for VPR. Finally, two new omnidirectional image datasets, one recorded with a hexacopter, calledSVMIS+, the other based on theMapillaryplatform, have been created and released for the entire research community.

In this article, we propose an original solution to the problem of surface reconstruction of large-scale unknown environments, with multiple cooperative robots. As they progress through the 3-D environment, the robots rely on volumetric maps obtained via a TSDF representation to extract discrete incomplete surface elements (ISEs), and a list of candidate viewpoints is generated to cover them. A next-best-view planning approach, which approximately solves a traveling salesman problem (TSP) via greedy allocation, is then used to iteratively assign these viewpoints to the robots. Two multiagent architectures, a centralized one (TSP-Greedy Allocation or TSGA) and a distributed one (dist-TSGA), in which the robots locally compute their maps and share them, are developed and compared. Extensive numerical and real-world experiments with multiple aerial and ground robots in challenging 3-D environments show the flexibility and effectiveness of our surface representation of a volumetric map. The experiments also shed light on the nexus between reconstruction accuracy and surface completeness, and between total distance traveled and execution time.

Motivated by the limited flight time of batterypowered multi-rotor UAVs, in this paper we address the problem of generating energy-optimal trajectories for a planar quadrotor. More specifically, by considering an accurate electrical model for the brushless DC motors and rest-to-rest maneuvers between two predefined boundary states, we explicitly compute the minimum-energy curves by adopting a free and a fixed end-time optimal control formulation. The numerical solution of these optimal control problems hinges upon a simple yet effective indirect projected gradient method. Simulation experiments illustrate the theory in a variety of realistic flight scenarios.

In this paper, we introduce and solve for the first time, the photometric stereo problem for low-cost 360-degree cameras. In particular, we present a spherical image irradiance equation which is adapted to twin-fisheye cameras, and an original algorithm for the estimation of light directions based on the specular highlights observed on mirror balls. Extensive experiments with synthetic and real-world images captured by a Ricoh Theta V camera, demonstrate the effectiveness and robustness of the proposed 3D reconstruction pipeline. To foster reproducible research, the image dataset and code developed for this paper are made publicly available at the address: home.mis.u-picardie.fr/~fabio/Photoophere.html.

In this paper, we study the state-constrained consensus problem and introduce a new family of distributed algorithms based on subspace projection methods which are simple to implement and which preserve, under some suitable conditions, the consensus value of the original discrete-time agreement protocol. The proposed theory is supported by extensive numerical experiments for the constrained 2D rendezvous of single-integrator robots.

In this paper, we propose a novel cluster-based Next-Best-View path planning algorithm to simultaneously explore and inspect large-scale unknown environments with multiple Unmanned Aerial Vehicles (UAVs). In the majority of existing informative path-planning methods, a volumetric criterion is used for the exploration of unknown areas, and the presence of surfaces is only taken into account indirectly. Unfortunately, this approach may lead to inaccurate 3D models, with no guarantee of global surface coverage. To perform accurate 3D reconstructions and minimize runtime, we extend our previous online planner based on TSDF (Truncated Signed Distance Function) mapping, to a fleet of UAVs. Sensor configurations to be visited are directly extracted from the map and assigned greedily to the aerial vehicles, in order to maximize the global utility at the fleet level. The performances of the proposed TSGA (TSP-Greedy Allocation) planner and of a nearest neighbor planner have been compared via realistic numerical experiments in two challenging environments (a power plant and the Statue of Liberty) with up to five quadrotor UAVs equipped with stereo cameras.

We present a new quasi-affine reconstruction of a scene and its application to camera self-calibration. We refer to this reconstruction as QUARCH (QUasi-Affine Reconstruction with respect to Camera centers and the Hodographs of horopters). A QUARCH can be obtained by solving a semidefinite programming problem when, (i) the images have been captured by a moving camera with constant intrinsic parameters, and (ii) a vague knowledge of the relative orientation (under or over 120 degrees) between camera pairs is available. The resulting reconstruction comes close enough to an affine one allowing thus an easy upgrade of the QUARCH to its affine and metric counterparts. We also present a constrained Levenberg-Marquardt method for nonlinear optimization subject to Linear Matrix Inequality (LMI) constraints so as to ensure that the QUARCH LMIs are satisfied during optimization. Experiments with synthetic and real data show the benefits of QUARCH in reliably obtaining a metric reconstruction.

In this paper, we present a new direct omnidirectional visual gyroscope for mobile robotic platforms. The gyroscope estimates the 3D orientation of a camera-robot by comparing the current spherical image with that acquired at a reference pose. By transforming pixel intensities into a Mixture of Photometric Potentials, we introduce a novel image-similarity measure which can be seamlessly integrated into a classical nonlinear least-squares optimization scheme, offering an extended convergence domain. Our method provides accurate and robust attitude estimates, and it is easy-to-use since it involves a single tuning parameter, the width of the photometric potentials (Gaussian functions, in this work) controlling the power of attraction of each pixel. The visual gyroscope has been successfully tested on spherical image sequences generated by a twin-fisheye camera mounted on the end-effector of a robot arm and on a fixed-wing UAV.

In this paper, we present a new characterization of mobility for formations of unicycle robots defined by distance-bearing constraints. In fact, by introducing a simple reduction procedure which associates a prescribed formation with a “macro-robot”, we extend the classification by type proposed by Campion et al., to multi-agent systems. To simplify the classification task, which only leverages the nonslip condition for a conventional centered wheel, we assume that the robots are disposed at the vertices of a regular convex polygon. We demonstrate the practical utility of the notion of macro-robot in a trajectory-tracking control problem for a formation of unicycles.

In this paper, we consider a non-conventional hexarotor whose propellers can be simultaneously tilted about two orthogonal axes: in this way, its underactuation degree can be easily adapted to the task at hand. For a given tilt profile, the minimum-energy trajectory between two prescribed boundary states is explicitly determined by solving an optimal control problem with respect to the angular accelerations of the six brushless motors. We also perform, for the first time, a systematic study of the singularities of the control allocation matrix of the hexarotor, showing the presence of subtle singular configurations that should be carefully avoided in the design phase. Numerical experiments conducted with the FAST-Hex platform illustrate the theory and delineate the pros and cons of dual-tilting paradigm in terms of maneuverability and energy efficiency.

A major limitation of existing battery-powered quadrotor UAVs is their reduced flight endurance. To address this issue, by leveraging the electrical model of a brushless DC motor, we explicitly determine minimum-energy paths between a predefined initial and final configuration of a quadrotor by solving an optimal control problem with respect to the angular accelerations of the four propellers. As a variation on this problem, if the total energy consumption between two boundary states is fixed, minimum-time and/or minimum-control-effort trajectories are computed for the aerial vehicle. The theory is illustrated for the DJI Phantom 2 quadrotor in three realistic scenarios.

This paper proposes a novel use of haptic feedback for human navigation with a mobile robot. Assuming that a path-planner has provided a mobile robot with an obstacle-free trajectory, the vehicle must steer the human from an initial to a desired target position by only interacting with him/her via a custom-designed vibro-tactile bracelet. The subject is free to decide his/her own pace and a warning vibrational signal is generated by the bracelet only when a large deviation with respect to the planned trajectory is detected by the vision sensor on-board the robot. This leads to a cooperative navigation system that is less intrusive, more flexible and easy-to-use than the ones existing in literature. The effectiveness of the proposed system is demonstrated via extensive real-world experiments.

This paper proposes a novel active target tracking strategy for a team of cooperating quadrotors equipped with 3-D range-finding sensors. The work builds upon previous research of the authors, and adopts a realistic nonlinear dynamic model for the quadrotors. A hierarchical controller is designed for the generation and tracking of the desired optimal trajectories of the aerial vehicles, and a discrete-time Kalman filter is used for fusing their local estimates of the target position. Under suitable conditions, it is shown that the cost function for the D-optimality criterion that the quadrotors aim at collaboratively reduce, possesses a single global minimum and no local minima. Numerical simulations and real-world experiments show the effectiveness of the proposed control strategy.

This paper presents a new algorithm based on previous results of the authors, for the estimation of the yaw angle of an omnidirectional camera/robot undergoing a 6-DoF rigid motion. Our real-time algorithm is uncalibrated, robust to noisy data, and it only relies on the projection of 3-D parallel lines as image features. Numerical and real-world experiments conducted with an eye-in-hand robot manipulator, which we used to simulate the 3-D motion of a Micro unmanned Aerial Vehicle (MAV), show the accuracy and reliability of our estimation algorithm.

This paper presents a new cooperative active target-tracking strategy for a team of double-integrator aerial vehicles equipped with 3-D range-finding sensors. Our strategy is active because it moves the vehicles along paths that minimize the combined uncertainty about the target's position. We propose a gradient-based control approach that encompasses the three major optimum experimental-design criteria and relies on the Kalman filter for estimation fusion. We derive analytical lower and upper bounds on the target's position uncertainty by exploiting the monotonicity property of the Riccati differential equation arising from the Kalman-Bucy filter. These bounds allow us to study the impact of sensors' accuracy and target's dynamics on the steady-state performance of our coordination algorithm. Finally, in the case that the position of the vehicles is not perfectly known, we introduce a more challenging problem, termed Active Cooperative Localization and Multi-target Tracking (ACLMT). In this problem, the vehicles move in the 3-D space in order to maximize the accuracy of their own position estimate and that of multiple moving targets.

This paper proposes a new cooperative active target-tracking strategy for a team of heterogeneous robots equipped with 3-D range-finding sensors. Our strategy is active, in the sense that the robots will track one or multiple moving targets while minimizing the combined uncertainty about the targets' position. We introduce a gradient-based control approach that encompasses the three major optimum experimental design criteria and relies only on robots' relative position measurements. The Kalman-Bucy filter is used for estimation fusion. Applications of the proposed strategy are shown to an experimental scenario featuring a team of double-integrator aerial vehicles and nonholonomic ground robots cooperatively tracking the motion of a human subject for a gait-monitoring task.

The Kuka Control Toolbox (KCT) is a collection of MATLAB functions for motion control of KUKA robot manipulators, developed to offer an intuitive and high-level programming interface to the user. The toolbox, which is compatible with all 6 DOF small and low payload KUKA robots that use the Eth.RSIXML, runs on a remote computer connected with the KUKA controller via TCP/IP. KCT includes more than 30 functions, spanning operations such as forward and inverse kinematics computation, point-to-point joint and Cartesian control, trajectory generation, graphical display and diagnostics. The flexibility, ease of use and reliability of the toolbox is demonstrated through two applicative examples.

The paper deals with a general class of leader-follower formations of unicycle robots induced by a constraint function that depends on the position and the orientation of the vehicles. We study the flexibility of such formations by introducing the notion of formation internal dynamics, characterize its equilibria and give sufficient geometric conditions for their existence. In particular, we show that the displacement and the relative orientation of each follower with respect to the leader's reference frame are fixed if and only if the robots either move along circular paths or parallel straight lines. These equilibrium configurations always exist if the trajectory of the leader is a circle of sufficiently small curvature or a straight line.

Planar catadioptric stereo vision sensors (PCS) combine a pinhole camera with two or more planar mirrors. PCS have recently received an increasing attention since a stereo view can be easily obtained without the need of exact multi-camera synchronization and calibration. In this paper we present a rigorous analytical treatment of the imaging geometry of PCS, propose new mirror calibration algorithms and introduce new multi-view properties that can be used for eye-in-hand camera localization. The effectiveness of the algorithms is shown via extensive simulation and real-data experiments on a robotic manipulator.

The paper proposes an original solution to the range identification problem for perspective dynamical systems. The depth of a static point observed by a pinhole camera undergoing a predefined 3-D motion, is estimated from its 2-D projection on the image plane. The proposed nonlinear observer relies on the immersion and invariance (I&I) methodology and offers several advantages over the existing range estimators. The paper also provides an analytical study of nonlinear observability performed with the extended output Jacobian. Extensive simulation experiments illustrate the theory and show the effectiveness of the proposed design.

The paper studies the cyclic pursuit problem in presence of connectivity constraints among single-integrator agents. The robots, each one pursuing its leading neighbor along the line of sight rotated by a common offset angle, are supposed to have a communication set described by a disk of constant radius. Given the initial position of the agents, we determine the communication radii that preserve the connectivity of the robots while they rendezvous at a point or converge to an evenly spaced circle formation. The special case that the initial condition is a linear combination of the eigenvectors of the dynamic matrix of the system, is studied in detail. On the other hand, given the communication radii, we find the set of initial conditions that guarantee the robots remain always connected. As a final contribution, once assigned a ldquonon-optimalrdquo radius, we study the stability of the hybrid system describing the dynamics of the robotic network under variable connectivity levels.

The paper proposes a new geometric approach to the stabilization of a hierarchical formation of unicycle robots. Hierarchical formations consist of elementary leader-follower units disposed on a rooted tree: each follower sees its relative leader as a fixed point in its own reference frame. Robots' linear velocity and trajectory curvature are forced to satisfy some given bounds. The major contribution of the paper is to study the effect of these bounds on the admissible trajectories of the main leader. In particular, we provide recursive formulas for the maximum velocity and curvature allowed for the main leader, so that the robots can achieve the desired formation while respecting their input constraints. An original formation control law is proposed and the asymptotic stabilization is proved. Simulation experiments illustrate the theory and show the effectiveness of the proposed designs.

This paper deals with vision-based localization for leader–follower formation control. Each unicycle robot is equipped with a panoramic camera that only provides the view angle to the other robots. The localization problem is studied using a new observability condition valid for general nonlinear systems and based on the extended output Jacobian. This allows us to identify those robot motions that preserve the system observability and those that render it nonobservable. The state of the leader–follower system is estimated via the extended Kalman filter, and an input-state feedback control law is designed to stabilize the formation. Simulations and real-data experiments confirm the theoretical results and show the effectiveness of the proposed formation control.

The paper introduces a new vision-based range estimator based upon the Immersion and Invariance (I&I) methodology, for leader-follower formation control. The proposed reduced-order nonlinear observer achieves global exponential convergence of the observation error to zero and it is extremely simple to implement and to tune. A Lyapunov analysis is provided to show the stability of the closed-loop system arising from the combination of the range estimator and an input-state feedback controller. Simulation experiments illustrate the theory and show the effectiveness of the proposed design.

The paper focuses on leader-follower formations of nonholonomic mobile robots. A formation control alternative to those existing in the literature is introduced. We show that the geometry of the formation imposes a bound on the maximum admissible curvature of leader trajectory. A peculiar feature of the proposed strategy is that the followers position is not rigidly fixed with respect to the leader reference frame but varies in suitable cones centered in the leader reference frame. Our approach also applies to hierarchical multirobot formations described by rooted tree graphs. Simulation experiments confirm the effectiveness of the proposed control schemes.

This paper focuses on leader-follower formations of mobile robots equipped with panoramic cameras and extend earlier works in the literature addressing both the vision-based localization and control problems. First, a new sufficient analytical condition for localizability is proved and used to shed light on the geometrical meaning of formation localization using uncalibrated vision sensors, here performed with the unscented Kalman filter. Second, we design a feedback control law based on dynamic extension in order to extend the applicability of our control scheme also to the case of distant robots.