We present in this paper a new open-source simulator based on CoppeliaSim and ROS for the popular Franka Emika Robot (FER) fully integrated in the ViSP ecosystem, a powerful library for Visual-Servoing. The simulator features a dynamic model that has been accurately identified from a real robot, leading to more realistic simulations. The C++ Api closely follows the ViSP class of the real robot allowing to narrow the gap between simulation code and real control software deployment. Conceived as a multipurpose research simulation platform, it is well suited for visual servoing applications as well as, in general, for any pedagogical purpose in robotics. All the software, models and CoppeliaSim scenes presented in this work are publicly available under free GPL-2.0 license.

Brain-Computer Interfaces (BCIs) enable users to interact with computers without any dedicated movement, bringing new hands-free interaction paradigms. In this paper we study the combination of BCI and Augmented Reality (AR). We first tested the feasibility of using BCI in AR settings based on Optical See-Through Head-Mounted Displays (OST-HMDs). Experimental results showed that a BCI and an OST-HMD equipment (EEG headset and Hololens in our case) are well compatible and that small movements of the head can be tolerated when using the BCI. Second, we introduced a design space for command display strategies based on BCI in AR, when exploiting a famous brain pattern called Steady-State Visually Evoked Potential (SSVEP). Our design space relies on five dimensions concerning the visual layout of the BCI menu; namely: orientation, frame-of-reference, anchorage, size and explicitness. We implemented various BCI-based display strategies and tested them within the context of mobile robot control in AR. Our findings were finally integrated within an operational prototype based on a real mobile robot that is controlled in AR using a BCI and a HoloLens headset. Taken together our results (4 user studies) and our methodology could pave the way to future interaction schemes in Augmented Reality exploiting 3D User Interfaces based on brain activity and BCIs.

We present in this paper a modular real-time model-based visual tracker. It is able to fuse different types of measurement, that is, edge points, textured points, and depth map, provided by one or multiple vision sensors. A confidence index is also proposed for determining if the outputs of the tracker are reliable or not. As expected, experimental results show that the more various measurements are combined, the more accurate and robust is the tracker. The corresponding C++ source code is available for the community in the ViSP library.

Augmented reality (AR) allows to seamlessly insert virtual objects in an image sequence. In order to accomplish this goal, it is important that synthetic elements are rendered and aligned in the scene in an accurate and visually acceptable way. The solution of this problem can be related to a pose estimation or, equivalently, a camera localization process. This paper aims at presenting a brief but almost self-contented introduction to the most important approaches dedicated to vision-based camera localization along with a survey of several extension proposed in the recent years. For most of the presented approaches, we also provide links to code of short examples. This should allow readers to easily bridge the gap between theoretical aspects and practical implementations.

Moving obstacle avoidance is a fundamental requirement for any robot operating in real environments, where pedestrians, bicycles, and cars are present. In this paper, we propose and validate a framework for avoiding moving obstacles during visual navigation with a wheeled mobile robot. Visual navigation consists of following a path, represented as an ordered set of key images, which have been acquired by an on-board camera in a teaching phase. While following such a path, our robot is able to avoid static and moving obstacles, which were not present during teaching, and which are sensed by an on-board lidar. The proposed approach takes explicitly into account obstacle velocities, estimated using an appropriate Kalman-based observer. The velocities are then used to predict the obstacle positions within a tentacle-based approach. Finally, our approach is validated in a series of real outdoor experiments, showing that when the obstacle velocities are considered, the robot behavior is safer, smoother, and faster than when it is not.

Moving obstacle avoidance is a fundamental requirement for any robot operating in real environments, where pedestrians, bicycles and cars are present. In this work, we design and validate a new approach that takes explicitly into account obstacle velocities, to achieve safe visual navigation in outdoor scenarios. A wheeled vehicle, equipped with an actuated pinhole camera and with a lidar, must follow a path represented by key images, without colliding with the obstacles. To estimate the obstacle velocities, we design a Kalman-based observer. Then, we adapt the tentacles designed in [1], to take into account the predicted obstacle positions. Finally, we validate our approach in a series of simulated and real experiments, showing that when the obstacle velocities are considered, the robot behaviour is safer, smoother, and faster than when it is not.

In this paper, we design and validate a new tentacle-based approach, for avoiding obstacles during appearance-based navigation with a wheeled mobile robot. In the past, we have developed a framework for safe visual navigation. The robot follows a path represented as a set of key images, and during obstacle circumnavigation, the on-board camera is actuated to maintain scene visibility. In those works, the model used for obstacle avoidance was obtained using a potential vector field. Here, a more sophisticated and efficient method, that exploits the robot kinematic model, and predicts collision at look-ahead distances, is designed and integrated in that framework. Outdoor experiments comparing the two models show that the new approach presents many advantages. Higher speeds and precision can be attained, very cluttered scenarios involving large obstacles can be successfully dealt with, and the control inputs are smoother.

This paper proposes an adaptation of classical image-based visual servoing to a generalised imaging model where cameras are modelled as sets of 3D viewing rays. This new model leads to a generalised visual servoing control formalism that can be applied to any type of imaging system whether it be multi-camera, catadioptric, non-central, etc. In this paper the generalised 3D viewing cones are parameterised geometrically via Plucker line coordinates. The new visual servoing model is tested on an a-typical stereo-camera system with non-overlapping cameras. In this case no 3D information is available from triangulation and the system is comparable to a 2D visual servoing system with non-central ray-based control law. I.

Image-based visual servo (IBVS) is a simple, efficient and robust technique for vision-based control. Although technically a local method in practice it demonstrates almost global convergence. However IBVS performs very poorly for cases that involve large rotations about the optical axis. It is well known that re-parameterizing the problem by using polar, instead of Cartesian coordinates, of feature points overcomes this limitation. First, simulation and experimental results are presented to show the complementarity of these two parameterizations. We then describe a new hybrid visual servo strategy based on combining polar and Cartesian image Jacobians.

This document provides a summary to a short video with the same title. The video shows the French intelligent transportation vehicle CyCab performing visual path following using only monocular vision. All phases of the process are shown with a spoken commentary. In the teaching phase, the user drives the robot manually while images from the camera are stored. Key images with corresponding images features are stored as a map together with 2D and 3D local information. In the navigation phase, CyCab follows the learned path by tracking the images features projected from the map and with a simple visual servoing control law.

We present a vision-based method to control the displacement of robot arm mounted on an underwater remotely operated vehicle (ROV). A closed-loop system based on an eye-to-hand visual servoing approach has been designed to achieve this task. We show that, using such an approach, the measuring of the manipulator motion with proprioceptive sensors is not required to precisely control the end-effector motion. To maintain the end effector in the field of view, the camera orientation is also controlled. The results presented show the validity of the approach.

We propose a 2D visual motion estimation method which can be exploited to achieve a dynamic positioning (e.g. by gaze control) with respect to a sea-bottom area of interest of a video camera mounted on a subsea vehicle. It mainly involves a dominant 2D motion robust estimation step from underwater video sequences. Optimizations carried out on the motion estimation code have made possible the use of our algorithm in "application-related real-time" for scientific exploration or inspection tasks. We have developed a friendly and efficient interface to perform this algorithm in an operational context. Experiments dealing with complex real underwater scenes are reported and validate the approach.

This paper presents the first version of our platform for automatically building the structure of a video sequence. The first application uses semi-automatic tools based only on image analysis for building interactive videos: decomposing the video into shots, extracting and tracking objects within each shot and linking occurrences of similar objects among the shots. The second application provides the end user with a powerful browser to navigate through any preprocessed hypervideo.

In this paper, we propose a visual servoing of the orientation of a camera mounted on a robot, using human eye movements. The study aims at the remote survey of sensitive sites or the intervention in contaminated nuclear sites. Visual data stemmed from an eye-tracker, providing the direction of the look of an operator, is generally noisy and sometimes inconsistent. Filtering, smoothing and stabilizing these eye tracker data are thus necessary in order to move the camera in the direction specified by the operator's eye. The visual servoing control scheme, as well as experimental results, are also presented.