This paper introduces PlugAndFilter, a framework designed to enhance the performance of multi-modal image registration, particularly for real-time video registration tasks. The improvements provided by PlugAndFilter include not only better registration quality for individual image pairs but also the transformation of the image registration methods into a more robust video registration system. These enhancements are made possible by three proposed contributions: Spatial and Temporal outlier detection, along with Confidence-based keypoint accumulation. PlugAndFilter is compatible with a wide range of thermal-visible registration models, and any registration method capable of producing keypoint matches can be integrated. The proposed implementation is optimized for real-time video registration on edge devices, with key design decisions highlighted to support this goal.

Bicubic interpolation is widely used in real-time image processing systems because of its quality. The real-time implementation of bicubic interpolation requires a lot of hardware resources, especially the number of multipliers because it represents high computational complexity. In this article, a set of algorithms that approximate the bicubic interpolation and reduce the hardware resource consumption are proposed. The proposed algorithms are based on combining linear and cubic interpolations. These algorithms are surveyed and compared in terms of interpolation quality, number of adders, number of multipliers, adaptive logic modules, lookup tables (LUTs), registers, and maximum operating frequency. These algorithms are implemented and tested on an Intel Cyclone V target. This article provides various choices of interpolation algorithms to cater to different application requirements, including accuracy, hardware resource consumption, and throughput performance. The implementation codes are available at github.com/DreamIP/Interpolation.

Hardware in the loop simulation has become a fundamental tool for the safe and rapid development of embedded systems. Dynamically and partially reconfigurable FPGA provide an energy efficient solution for high performance computing in embedded systems, such as computer vision, with limited resources. Finally 3D simulation with realistic physics simulation is required by designers of Unmanned Aerial Vehicle (UAV) and related missions. The combination of the three techniques are required to design UAV with reconfigurable HW/SW embedded systems that can self-adapt to different mission phases according to environment changes. But they require different complex and specific skills from separated communities and so are not considered simultaneously. In this paper we demonstrate a complete framework that we apply to a UAV case simulated with the well adopted Gazebo 3D simulation tool including the Ardupilot model. According to usual practices in Robotics, we use Robot Operating System (ROS) middleware over Linux that we implement on a separated Intel Cyclone V FPGA board including HW/SW interfaces. As a convincing case study we implement, besides software classical navigation tasks, a vision-based emergency-landing security task and a detect and tracking classic mission application (TLD) that can run in different HW and SW versions dynamically configured on the FPGA according to mission steps simulated with Gazebo.

In this paper we introduce a new geometric calibration algorithm, and a geometric method of 3D reconstruction using a panoramic microwave radar and a camera. These two sensors are complementary, considering the robustness to environmental conditions and depth detection ability of the radar on one hand, and the high spatial resolution of a vision sensor on the other hand. This makes the approach well adapted for large scale outdoor cartography. Firstly, we address the global calibration problem which consists in finding the exact transformation between radar and camera coordinate systems. The method is based on the optimization of a non-linear criterion obtained from a set of radar-to-image target correspondences. Unlike existing methods, no special configuration of the 3D points is required, only the knowledge of inter-targets distance is needed. This makes the method flexible and easy to use by a non expert operator. Secondly, we present a 3D reconstruction method based on sensors geometry. Both methods have been validated with synthetic and real data.

In this letter, we propose HOG-Dot, a method for the direct computation of the polar image gradients coordinates from the pixels values. The proposed algorithm, to be used as the first step of the Histogram of Oriented Gradient (HOG) pipeline, approximates the exact gradient with its projection onto a versor chosen among the projection plane set. Instead of non linear computations, the HOG-Dot method exploits linear operations while introducing a bounded approximation error with respect to other HOG approaches, thus resulting a more suitable solution for embedded devices. Concerning the state of the art, it also achieves improved accuracy with the mathematical spatial gradient formulation .

Loop-closure detection, which is the ability to recognize a previously visited place, is of primary importance for robotic localization and navigation problems. We here introduce SAIL-MAP, a method for loop-closure detection based on vision only, applied to topological simultaneous localization and mapping (SLAM). Our method allows the matching of camera images using a novel saliency-based feature detector and descriptor. These features have been designed to benefit from the robustness to viewpoint change and image perturbations of bio-inspired saliency algorithms. Additionally, the same algorithm is used for the detector and descriptor. The results obtained on different large-scale data sets demonstrate the efficiency of the proposed solution for localization problems.

Over the past few years, a new category of sensor is emerging. It is named intelligent sensor or smart sensor. One type of this sensor is smart camera. This paper focuses on smart camera used in the field of robotics and more precisely in the extraction of features. Object tracking, autonomous navigation or 3D mapping are some application examples of feature extraction. We have selected interest points feature that can be detected by several algorithms. One of these algorithms is Harris & Stephen, that has a simple principle based on the calculations of multiples derivatives and gives acceptable results. The smart camera also contains an FPGA as a processing device which will be used for the implementation of Harris & Stephen algorithm. Other modules were also added to the system to deliver more appropriate results.

We introduce CAPH, a new domain-specific language (DSL) suited to the implementation of stream-processing applications on field programmable gate arrays (FPGA). CAPH relies upon the actor/dataflow model of computation. Applications are described as networks of purely dataflow actors exchanging tokens through unidirectional channels. The behavior of each actor is defined as a set of transition rules using pattern matching. The CAPH suite of tools currently comprises a reference interpreter and a compiler producing both SystemC and synthetizable VHDL code. We describe the implementation, with a preliminary version of the compiler, of a simple real-time motion detection application on a FPGA-based smart camera platform. The language reference manual and a prototype compiler are available from http://wwwlasmea.univ-bpclermont.fr/Personnel/Jocelyn.Serot/caph.html.

Due to the various devices composing a smart camera system, various languages have to be known by the designer (like HDL and C/C++). Most of vision applications designers are software programmers and do not have a good knowledge of HDLs (VHDL). This paper presents a new high-level methodology for implementing vision applications on smart camera platforms. This methodology is based on a soft-core approach to manage the whole system and a dataflow (actor-oriented) language to design the processing elements. We discuss in particular interfacing constraints.

We describe a spatio-temporal triangulation method to be used with rolling shutter cameras. We show how a single pair of rolling shutter images enables the computation of both structure and motion of rigid moving objects. Starting from a set of point correspondences in the left and right images, we introduce the velocity and shutter characteristics in the triangulation equations. This results in a non-linear error criterion whose minimization in the least square sense provides the shape and velocity parameters. Unlike previous work on rolling shutter cameras, the constraining assumption of a-priori knowledge about the object geometry is removed and a full 3D motion model is considered. The aim of this work is thus to make the use of rolling shutter cameras of a broader interest. Experimental evaluation results confirm the feasibility of the approach.

A novel architecture dedicated to image processing is presented. The most original aspect of the approach is the use of a high density FPGA to build a versatile embedded smart camera. We show the main advantages of such a system based on a CMOS imaging device and a programmable chip. With its structure, the system proposes a high degree of versatility and allows the implementation of parallel image processing algorithms. As a result, an architecture of template tracking is proposed.

The main goal of this paper, is the study of image-based stereo visual servoing. A pair of cameras is mounted on the end-effector of the manipulator arm. The visual features are the pair of images of an unknown object. The developed control laws use either the raw image points, or the estimated 3D coordinates. The experimental setup is challenging: large rotations are involved, images are noisy, and cameras are coarsely calibrated. In this setup, the trajectory of the end-effector differs notably, sometimes leading the arm near its joint range limits. Experimental results demonstrate that using pixel coordinates is disadvantageous, compared with 3D coordinates estimated from the same pixel data.

In this paper, the problem of controlling a motion by visual servoing around an unknown object is addressed. These works can be interpreted as an initial step towards a perception goal of an unmodeled object. The main purpose is to perform motion with regard to the object in order to discover several viewpoint of the object. The originality of our work is based on the choice and extraction of visual features in accordance with motions to be performed. The notion of invariant feature is introduced to control the navigational task around the unknown object. A real-time experimentation with a complex object is realized and shows the generality of the proposed ideas.

We propose several results about trajectory generation by visual servoing. The approach consists of defining a specific task function which allows one to take into account the time varying aspect of the reference feature and to synthesize a control law in the sensor space. This control law ensures the trajectory control in the image space and reduces the tracking error. Under specific conditions, the trajectory of the camera can be ensured in the robot workspace. The main goal of this work is to demonstrate the effectiveness of this approach through experimental results. In the experiments, we used a Cartesian robot and a real time vision system. A CCD camera was mounted on the end effector of the robot. We present two types of trajectory. The first one is a helical trajectory parallel to a cube side. The second one involves passing around a cube. This latter is built by linking several elementary trajectories (rotation and translation).

Describes an approach to the problem of trajectory generation in a workspace by visual servoing. Visual servoing is based on an array of measurements taken from a set of images and used each time as an error function to compute a control vector. This is applied to the system (robot and camera) and enables it to move in order to reach a desired situation, at the end of the task, directly depicted in the image. The originality of this work is based on the concept of a time varying reference feature. Classically, in visual servoing, the reference features are static and the task to be achieved is similar to a positioning task. We define a specific task function which allows us to take into account the time varying aspect and we synthesize a new control law in the sensor space. This control law ensure the trajectory control in the workspace. Considering that any trajectories in workspace can be depicted as a combination of rotation and translation, we have tested our approach using these two elementary trajectories.

Visual servoing is based on an array of measurements taken from a set of images and used each time as an error function to compute a control vector. This is applied to the system (robot and camera) and makes it move in order to reach a desired situation, at the end of the task, directly depicted in the image. The originality of this recent work consists in improving the visual servoing approach. To do this, we consider a signal sensor vector constructed of a geometrical feature ((x,y) point coordinates, line parameters, etc.) and its first derivative. In this paper, we show how to work out the interaction matrix. We have tested this new approach on a workstation in the case of the point feature. We have also implemented it on our robotic platform. The overall results show a great improvement due to the action of this new signal sensor. We are extending this approach to more complex features.