This study introduces a novel method to mitigate temporal illumination variations in whisk-broom hyperspectral imaging under varying environmental illumination conditions. Whisk-broom hyperspectral imaging captures high resolution spectra pixel-by-pixel sequentially, a process susceptible to sunlight fluctuations over time, particularly when imaging cultural artifacts outdoors. Despite sunlight's broad spectrum, its variability over time can compromise the quality of hyperspectral images, affecting data analysis. Prior strategies suggested a supplementary single compensating vertical scan alongside the standard row-wise raster scan. However, this strategy fails when the additional single vertical scan is performed near or on a black frame. Building on this, our study proposes using two scans, one performed orthogonally to the traditional row-wise scan, to counteract illumination variations. Furthermore, we formulate the compensation problem in logarithmic space, exploiting the low-dimensional structure of the reflectance and illumination spectra. Using total variation penalization in the cost function enhances smoothness. Our method achieves robust compensation for changes in environmental illumination. We also demonstrate that we can use multiple columns from the column-wise scan without significantly decreasing the compensation quality while reducing acquisition time. We illustrate the application of our methods to hyperspectral images of stained-glass windows of the historic Cathedrale Notre-Dame d'Amiens in France.

Direct Visual Servoing (DVS) is a technique that controls the robot motion by using the pixel intensities captured by a camera. DVS demonstrates high accuracy at convergence, prompting the development of various methods aimed at expanding its convergence domain.In this paper, we propose a mathematical characterization of the DVS convergence domain with closed-form expressions for the controlled degrees of freedom. From these expressions, we concluded that the extent of the convergence domain is related to the presence of isotropic or defocus blur, a phenomenon that had only been observed previously as a trend in empirical experiments.

In this article, we propose an original solution to the problem of surface reconstruction of large-scale unknown environments, with multiple cooperative robots. As they progress through the 3-D environment, the robots rely on volumetric maps obtained via a TSDF representation to extract discrete incomplete surface elements (ISEs), and a list of candidate viewpoints is generated to cover them. A next-best-view planning approach, which approximately solves a traveling salesman problem (TSP) via greedy allocation, is then used to iteratively assign these viewpoints to the robots. Two multiagent architectures, a centralized one (TSP-Greedy Allocation or TSGA) and a distributed one (dist-TSGA), in which the robots locally compute their maps and share them, are developed and compared. Extensive numerical and real-world experiments with multiple aerial and ground robots in challenging 3-D environments show the flexibility and effectiveness of our surface representation of a volumetric map. The experiments also shed light on the nexus between reconstruction accuracy and surface completeness, and between total distance traveled and execution time.

Abstract We propose a method for eliminating the temporal illumination variations in whisk-broom (point-scan) hyperspectral imaging. Whisk-broom scanning is useful for acquiring a spatial measurement using a pixel-based hyperspectral sensor. However, when it is applied to outdoor cultural heritages, temporal illumination variations become an issue due to the lengthy measurement time. As a result, the incoming illumination spectra vary across the measured image locations because different locations are measured at different times. To overcome this problem, in addition to the standard raster scan, we propose an additional perpendicular scan that traverses the raster scan. We show that this additional scan allows us to infer the illumination variations over the raster scan. Furthermore, the sparse structure in the illumination spectrum is exploited to robustly eliminate these variations. We quantitatively show that a hyperspectral image captured under sunlight is indeed affected by temporal illumination variations, that a Naïve mitigation method suffers from severe artifacts, and that the proposed method can robustly eliminate the illumination variations. Finally, we demonstrate the usefulness of the proposed method by capturing historic stained-glass windows of a French cathedral.

In this paper, we introduce and solve for the first time, the photometric stereo problem for low-cost 360-degree cameras. In particular, we present a spherical image irradiance equation which is adapted to twin-fisheye cameras, and an original algorithm for the estimation of light directions based on the specular highlights observed on mirror balls. Extensive experiments with synthetic and real-world images captured by a Ricoh Theta V camera, demonstrate the effectiveness and robustness of the proposed 3D reconstruction pipeline. To foster reproducible research, the image dataset and code developed for this paper are made publicly available at the address: home.mis.u-picardie.fr/~fabio/Photoophere.html.

In this paper, we propose a novel cluster-based Next-Best-View path planning algorithm to simultaneously explore and inspect large-scale unknown environments with multiple Unmanned Aerial Vehicles (UAVs). In the majority of existing informative path-planning methods, a volumetric criterion is used for the exploration of unknown areas, and the presence of surfaces is only taken into account indirectly. Unfortunately, this approach may lead to inaccurate 3D models, with no guarantee of global surface coverage. To perform accurate 3D reconstructions and minimize runtime, we extend our previous online planner based on TSDF (Truncated Signed Distance Function) mapping, to a fleet of UAVs. Sensor configurations to be visited are directly extracted from the map and assigned greedily to the aerial vehicles, in order to maximize the global utility at the fleet level. The performances of the proposed TSGA (TSP-Greedy Allocation) planner and of a nearest neighbor planner have been compared via realistic numerical experiments in two challenging environments (a power plant and the Statue of Liberty) with up to five quadrotor UAVs equipped with stereo cameras.

The direct use of the entire photometric image information as dense features for visual servoing brings several advantages. First, it does not require any feature detection, matching, or tracking process. Thanks to the redundancy of visual information, the precision at convergence is highly accurate. However, the corresponding highly nonlinear cost function reduces the convergence domain. In this paper, we propose visual servoing based on the analytical formulation of Gaussian mixtures to enlarge the convergence domain. Pixels are represented by two-dimensional Gaussian functions that denote a “power of attraction.” In addition to the control of the camera velocities during the servoing, we also optimize the Gaussian spreads allowing the camera to precisely converge to a desired pose even from a far initial one. Simulations show that our approach outperforms the state of the art and real experiments show the effectiveness, robustness, and accuracy of our approach.

In this paper, we propose a descriptor for image matching under multiple mirror reflections. Indeed, existing adaptations of SIFT for the mirror transformation are not successful when object and mirrors orientations are not constrained. Hence, we propose to combine MIFT and Affine-SIFT descriptors as the Affine Mirror Invariant Feature Transform (AMIFT). The experimental results and given evaluation show that our proposed descriptor outperforms MIFT and ASIFT on both synthetic and real images datasets.

The advantages of using the entire photometric image information as visual feature are: it does not require any feature detections, matching or tracking process. To enlarge the convergence domain, we propose to accomplish visual servoing based on the analytical formulation of Gaussian mixtures to model the images. During the servoing, we consider the optimization of the Gaussian spreads allowing the camera to converge to a desired pose even from a far initial one. Simulation that overcomes the state-of-the-art and real experiments highlight the success of our approach.

In this paper, we present a new automatic camera control to achieve a relevant information framing. This camera control will be performed using a visual servoing framework in order to reach a salient area in a plane and in space. The relevant framing will be modelled by maximizing the saliency-based Gaussian Mixture Model (GMM) feature in the image. Furthermore, in order to achieve a realistic automatic camera control, we add an obstacles avoidance constraint and we ensure a relevant orientation during the motion. We validate our contribution in different synthetic 2D and 3D environments. Finally, we test our approach on a dense 3D points cloud model and in a real environment with a robot.

This work is part of a global framework that aims to build a complete set of algorithms for autonomous robot exploration, navigation and map building using a sole omnidirectional camera. It focuses on topology extraction of navigable free spaces to yield autonomous robot exploration. The paper proposes a new algorithm, the Omnidirectional Delta Medial Axis (ODMA), with linear-time implementation to extract robust topology of the robot's local free space. It first introduces an adapted metric to cope with the deformations involved by the catadioptric sensor, and that is conformal with the ground 2D-metric. Then, it produces a pruned skeleton of the free space, inspired by the delta medial axis. Experimental results validate the approach for both skeleton precision and stability with respect to the robot position and free space topology.

The problem of time to contact or time to collision (TTC) estimation is largely discussed in perspective images. However, a few works have dealt with images of catadioptric sensors despite of their utility in robotics applications. The objective of this paper is to develop a novel model for estimating TTC with catadioptric images relative to a planar surface, and to demonstrate that TTC can be estimated only with derivative brightness and image coordinates. This model, called “gradient based time to contact”, does not need high processing such as explicit estimation of optical flow and feature detection and/or tracking. The proposed method allows to estimate TTC and gives additional information about the orientation of planar surface. It was tested on simulated and real datasets.

Medial axes are known to be very sensitive to shape irregularities. In this paper, we develop a solution to compute a stable medial axis of noisy discrete shapes. It introduces a parameter up to which a deformation (noise) of the shape is considered irrelevant, and thus ignored in the discrete Euclidian Medial Axis computation. We show the linearity property of the proposed algorithm and compare it with two recent state of the art methods: The Gamma Integer Medial Axis and the Discrete Linear Lambda Medial Axis using a single pruning parameter. Based on Kimia's database (216 binary images), we present comparative experimental results with respect to skeletonization quality, noise sensitivity and computation time.

This paper proposes a method for human action classification by utilizing correlations between SURF based descriptors. This approach provides us a novel type of descriptor that can be used for action classification. The method proposed is tested using an SVM classification technique. For evaluation purposes, the KTH action recognition dataset, which is a standard benchmark for this area is used as it is one of the most well known and challenging dataset. The method proposed was able to successfully classify different action classes.

Detection of moving objects is known to be a critical first step in many vision-based applications such as video surveillance. Background Subtraction Algorithms (BSA) offer a solution to detect all foreground pixels in a frame by comparing them with a background model. However, this is still a very challenging task especially when dynamic scenes are involved (camera jitter, noise, etc). In this paper, we present a new method for dynamic background subtraction operation based on invariant moments (Hu Set). Each pixel is modeled as a set of moments calculated from its neighborhood and stored using codebook construction. Experimental results on a set of outdoor scenes show that our method outperforms traditional BSA.

In this paper, we propose a robust and fast active contour based method to free space detection in omnidirectional images where the problem of falsely detected obstacles is solved. We define a new functional energy formulation including altitude estimation of keypoints extracted nearby the active contour modeling the free space. The free space, so extracted, could help the robot in real time navigation and environment exploration tasks. To validate the efficiency of the proposed approach, the paper shows comparative results achieved with a classical formulation and our formulation of active contour energies, using images acquired by a robot exploring unknown indoor and outdoor environments, with no prior knowledge of the shape or the extend of the free space.

Omnidirectional cameras allow direct tracking and motion estimation of planar regions in images during a long period of time. However, using only one camera leads to plane and trajectory reconstruction up to a scale factor. We propose to develop dense plane tracking based on omnidirectional stereovision to answer this issue. The presented method estimates simultaneously the parameters of several 3D planes along with the camera motion in a spherical model formulation. Results show the efficiency of the approach.

This paper contributes to adapt parametric and geometric active contour methods in a new framework to handle real time free space extraction while taking advantage of the properties of omnivision. Both methods were formally and algorithmically adapted and improved. Some comparative results, achieved on unknown indoor and outdoor images, are presented to validate the efficiency of our two snake based approaches. We also show that active contours can be applied to make a robot navigate autonomously, only using real omni images, thanks to the extracted free space skeleton.

Omnidirectional catadioptric cameras offer a large field of view and a complete information about the world surrounding the robot. In this paper, we describe a method to perform a fast and robust extraction of the omnidirectional free space using active contour models. The extracted free space could help the robot in real time navigation and environment exploration tasks. The presented approach will be compared to classical active contour methods usually applied to object segmentation. Some comparative results achieved in indoor and outdoor environments are shown to validate the approach.

Visual servoing has been based on geometric features for a long time. Recent works have highlighted the interest of taking into account the photometric information of the entire image. This approach was tackled with images of perspective cameras. We propose, in this paper, to adapt this technique to central cameras. This generalization allows to apply this kind of method to wide field of view cameras. We also propose to adapt gradient computation to take into account distorsions of such cameras. Several experiments have been successfully done with a fisheye camera.

Robot map navigation and localization are challenging tasks that require the solving of the data association problem for local and global features. Data fusion allows the advantages of two or more sensors to be combined, and complementary cooperation can be obtained. This paper presents two methods to embed depth information in omnidirectional images using the extrinsic calibration of a 2D laser range finder and a central catadioptric camera. The methods presented do not require a visible laser beam, but they assume the planar checkerboard patterns are visible for both the catadioptric camera and the 2D laser range finder. Unlike other approaches, the methods proposed used an invisible laser trace, and they are evaluated at pixel error level using ground truth data from the calibration patterns projected in the omnidirectional image. Results include a mean square error analysis of all calibration poses, and laser point projection on indoor omnidirectional images. We think that embedding range information in omnidirectional images is an interesting tool for data fusion approaches, which can be used in robot map building and localization.

In this paper we present an optimal omnidirectional visual sensor which can replace perspective camera network for traffic diagnosis. The proposed system has the advantage, by the number and the designed mirror, to generate a single view of the crown and junction ways of the crossroads by maximizing the number of useless pixels. So, the percentage of pixels utilized directly for subsequent phases of image processing is optimal. We describe the methodology used to design such a sensor. In addition, to assess our sensor, we also developed image processing methods that provide useful indicators for estimating the state of the traffic as the crossroads occupancy rate, the vehicle speed and the flow of vehicles. Finally, we compare this optimal sensor to others that consist of parabolic, hyperbolic or spherical mirror to observe the scene. We prove that optimal sensor has better results than others.

We are investigating the mobile robot indoor localization and environment mapping using an omnidirectional stereovision sensor. It uses four parabolic mirrors and an orthographic camera, giving four images of the same scene. At least, only two mirrors are needed. Using four mirrors gives redundancy. We propose to exploit the images of vertical lines. This paper presents a new method in order to match these lines in the four images. Contrary to existing approaches, we took into account the four sub-images existence in the design of this method, in order to exploit redundancy. This brought an original algorithm combining matching and pose estimation of vertical lines from the 3D environment. Experimental results will be presented to validate this approach.

Robot vision has a lot to win as well with wide field of view induced by catadioptric cameras as with redundancy brought by stereovision. Merging these two characteristics in a single sensor is obtained by combining a single camera and multiple mirrors. This paper proposes a 3D model tracking algorithm that allows a robust tracking of 3D objects using stereo catadioptric images given by this sensor. The presented work relies on an adapted virtual visual servoing approach, a non-linear pose computation technique. The model take into account central projection and multiple mirrors. Results show robustness in illumination changes, mistracking and even higher robustness with four mirrors than with two.

In this paper, we propose a highly efficient algorithm to model the human skin color. The underlying algorithm involves generating a discrete Cosine transform (DCT) at each pixel location, using the surrounding points. These DCT coefficients are assumed to follow a generalized Gaussian distribution (GGD). Next, the model parameters are estimated using the maximum-likelihood (ML) criterion applied to a set of training skin samples. Finally, each pixel is classified as skin or the opposite if its likelihood ratio is above some threshold. The experimental results show that our model avoids excessive false detection while still retaining a high degree of correct detection.

This paper describes a new compact omnidirectional stereovision sensor that combines a single orthographic camera and four paraboloidal mirrors. Its geometry has been designed with the help of a stochastic optimization approach in order to minimize the 3D reconstruction error In comparison with state-of-the-art sensors described in the literature, better results are obtained for this sensor during simulations. We will especially compare it with a classical configuration with two mirrors. Two criteria will be used. the 3D reconstruction accuracy and the field of view. We illustrate the advantages of our sensor within a framework of a simulation using a realistic environment and a ray-tracing software.

International audience

You can create catadioptric omnidirectional stereovision using several mirrors with a single camera. These systems have interesting advantages, for instance in the case of mobile robot navigation and environment reconstruction. Our paper aims at estimating the” quality” of such stereovision system. What happens when the number of mirrors increases? Is it better to increase the base-line or to increase the number of mirrors? We propose some criteria and a methodology to compare different significant categories (seven): three already existing systems and four new designs that we propose. We also study and propose a global comparison between the best configurations.

Omnidirectional cameras offer a much wider field of view than the perspective ones and alleviate the problems due to occlusions. However, both types of cameras suffer from the lack of depth perception. A practical method for obtaining depth in computer vision is to project a known structured light pattern on the scene avoiding the problems and costs involved by stereo vision. This paper is focused on the idea of combining omnidirectional vision and structured light with the aim to provide 3D information about the scene. The resulting sensor is formed by a single catadioptric camera and an omnidirectional light projector. It is also discussed how this sensor can be used in robot navigation applications.

The present study proposes an innovative approach to qualitative mobile robot's localization using the concept of integral invariant on omnidirectional images. They are invariant depending on the image transformations caused by the movements of the robot. Several methods have been suggested to construct such invariants but they often rely on hypotheses about the transformation group which do not hold any more when dealing with omnidirectional sensors. These sensors benefit from an increasing interest in mobile robotics because of their field of view but they require adaptations of classical methods. This paper presents a method based on group averaging to construct invariant features which could be used to recognize a place with this type of sensors.

Central catadioptric sensors enable to acquire panoramic images on a 360 degree field of view while preserving a single viewpoint. These advantages account for the growing use of these sensors in applications such as surveillance, navigation or modelling. However, the deformations of the image do not allow to apply classical perspective image algorithms or operators. Typically, straight line detection in perspective image becomes a delicate and complex conic detection problem in central catadioptric image. Previous methods proposed in the literature were essentially motivated by particular cases such as horizontal line detection or paracatadioptric line detection. In this paper, we propose an algorithm which consists in performing the detection in the space of the equivalent sphere which is the unified domain of central catadioptric sensors. On this sphere, real lines are projected into great circles that we detect thanks to the Hough transform. We also propose to apply this unifying model in order to perform the calibration of the intrinsic parameters required for the projection on the sphere. We show results on synthetic and real catadioptric images (parabolic, hyperbolic) to demonstrate the relevance of the detection on the sphere.

The recovery of the three-dimensional structure of the environment is a pre-requisite for many tasks in mobile robotics. Unfortunately, calibration acts as a brake upon visual adaptation and robotic autonomy. We provide tools and constraints based on structured light for a complete and efficient uncalibrated Euclidean reconstruction of the environment. Experimental results achieved both on simulated and real data validate the method.

Deals with an absolute mobile robot self-localization algorithm given a two-dimensional cartography of the robot's environment. The proposed method only requires the goniometrical observations. We present an original solution to compute the robot's pose estimation by an interpretation tree (IT) search. Our approach introduces the subdivision of the robot's evolution field into rectangles. The matching between the observations and the visual landmarks is effected by judicious angular tests (heuristics for the IT). Moreover, we show that our matching method implicitly takes uncertainties and inaccuracies into account. Finally, some experimental results with real noisy omnidirectional images, provided by our omnidirectional vision sensor SYCLOP, are shown.

We present a complete localisation method for mobile robots. We deal with low level image processing, model recognition through an invariant, matching with real noisy goniometrical observations and location estimation through a triangulation technique. We show how useful invariants may be in order to solve the matching problem. Moreover, we point out how invariants are powerful to discriminate features (in images) in order to provide a quick and reliable robot location. Our localisation technique is absolute, i.e. it needs neither a (exterior) position prediction nor a resetting technique. Finally, we present experimental results with a substantial set of real noisy images that demonstrate the reliability of our localisation method.

In this article we present a navigation system allowing a mobile robot to be localized in an indoor environment which is only partially known. This system integrates an environment map updating module allowing the mobile robot to estimate the position of new vertical landmarks along its path. An extended Kalman filter is used on the one hand to estimate the mobile robot position and on the other hand to extract observations which will be used to determine the positions of unlisted landmarks. The integration of new landmarks into the environment global map is managed from the covariance matrix associated with each unlisted landmark. We present the experimental results we have got with SARAH, our mobile robot.

We first present a short description of our conic sensor. Furthermore, we pay special attention to the projective invariant, represented by the cross-ratio. We end the paper by presenting a new 1D cross-ratio application that enables us to localise our mobile robot SARAH efficiently. With the method that we propose using, the camera does not need to be calibrated and this vision system, based on an omnidirectional sensor, is a major advantage for the navigation of an autonomous mobile robot.

We present a summary of the most significant techniques, used in the last few years, concerning the coded structured light methods employed to get 3D information. In fact, depth perception is one of the most important subjects in computer vision. Stereovision is an attractive and widely used method, but, rather limited to make 3D surface maps, due to the correspondence problem. The correspondence problem can be improved using a method based on a structured light concept, projecting a given pattern on the measuring surfaces, although some relations between the projected pattern and the reflected one must be solved. This relationship can be directly found codifying the projected light, so that, each imaged region of the projected pattern carries the necessary information to solve the correspondence problem. We do not need to mention the numerous advantages in accurate obtention of 3D information for many research subjects, such as: robotics, autonomous navigation, shape analysis, and so on.

The absolute necessity of obtaining 3D information of structured and unknown environments in autonomous navigation reduce considerably the set of sensors that can be used. The necessity to know, at each time, the position of the mobile robot with respect to the scene is indispensable. Furthermore, this information must be obtained in the least computing time. Stereo vision is an attractive and widely used method, but, it is rather limited to make fast 3D surface maps, due to the correspondence problem. The spatial and temporal correspondence among images can be alleviated using a method based on structured light. This relationship can be directly found codifying the projected light; then each imaged region of the projected pattern carries the needed information to solve the correspondence problem. We present the most significant techniques, used in recent years, concerning the coded structured light method.

Mobile robots use actually a combination of internal and external sensors to determine their position and orientation in a path following period. Incremental encoders, gyrometers are generally used to give an approximative estimation of localization. Nevertheless, the cumulative drifts of these internal sensors must be periodically corrected with an exteroceptive sensor. So, the authors present, in this paper, an optical omnidirectional sensor which can give, with an adapted software, an absolute localization. This sensor is made of a CCD video camera associated with a conic shaped reflector; so, a view of a 2 /spl pi/ radian field is available to compute the position of the robot. The authors report the matching algorithm allowing a local observed scene to be replaced in the global navigation area. The authors conduct the accuracy analysis of this global positioning system, and present the experimental results.