We introduce an active learning framework for content-based image retrieval for video surveillance that can be trained ad-hoc for a single camera in a matter of minutes. This technique allows searching for both, known and unknown objects, given a region of interest. The process does not require prior labelled data and treats image retrieval as a binary classification task, in which frames can be similar or different from a query image. The technique is compatible with any pre-trained deep feature extractor. In addition, we propose a novel label propagation algorithm that benefits from (1) visual similarity of image pairs and (2) the semantic representation of the feature vectors from a pre-trained deep feature extractor. This approach allows to reduce the amount of labels needed, while avoiding the propagation of errors. Our experiments with three use-cases from a nuclear facility show the validity of the proposed method, which achieves high precision and recall while requiring minimal amounts of labelled data. • Present a relevant video frame retrieval pipeline that can be trained interactively. • Propose an active learning algorithm that leverages visual and semantic similarity. • Visual similarity makes label propagation less prone to introducing errors.

In this paper we introduce a novel public dataset for developing and benchmarking indoor localization systems. We have selected and 3D mapped a set of representative indoor environments including a large office building, a conference room, a workshop, an exhibition area and a restaurant. Our acquisition pipeline is based on a portable LiDAR SLAM backpack to map the buildings and to accurately track the pose of the user as it moves freely inside them. We introduce the calibration procedures that enable us to acquire and geo-reference live data coming from different independent sensors rigidly attached to the backpack. This has allowed us to collect long sequences of spherical and stereo images, together with all the sensor readings coming from a consumer smartphone and locate them inside the map with centimetre accuracy. The dataset addresses many of the limitations of existing indoor localization datasets regarding the scale and diversity of the mapped buildings; the number of acquired sequences under varying conditions; the accuracy of the ground-truth trajectory; the availability of a detailed 3D model and the availability of different sensor types. It enables the benchmarking of existing and the development of new indoor localization approaches, in particular for deep learning based systems that require large amounts of labeled training data.

This paper presents a new technique to solve the Indoor Visual Place Recognition problem from the Deep Learning perspective. It consists on an image retrieval approach supported by a novel image similarity metric. Our work uses a 3D laser sensor mounted on a backpack with a calibrated spherical camera i) to generate the data for training the deep neural network and ii) to build a database of geo-referenced images for an environment. The data collection stage is fully automatic and requires no user intervention for labelling. Thanks to the 3D laser measurements and the spherical panoramas, we can efficiently survey large indoor areas in a very short time. The underlying 3D data associated to the map allows us to define the similarity between two training images as the geometric overlap between the observed pixels. We exploit this similarity metric to effectively train a CNN that maps images into compact embeddings. The goal of the training is to ensure that the L2 distance between the embeddings associated to two images is small when they are observing the same place and large when they are observing different places. After the training, similarities between a query image and the geo-referenced images in the database are efficiently retrieved by performing a nearest neighbour search in the embeddings space.

Ego-motion estimation and localization in large environments are key components in any assistive technology for real-time user orientation and navigation. We consider the case where a large known environment is explored without a priori assumptions on the initial location. In particular we propose a framework that uses a single portable 3D sensor to solve the place recognition problem and continuously tracks its position even when leaving the known area or when significant changes occur in the observed environment. We cast the place recognition step as a classification problem and propose an efficient search space reduction considering only navigable areas where the user can be localized. Classification hypotheses are then discarded exploiting temporal consistency w.r.t. a relative tracker that exploits only the sensor input data. The solution uses a compact classifier whose representation scales well with the map size. After being localized, the user is continuously tracked exploiting the known environment using an efficient data structure that provides constant access time for nearest neighbor searches and that can be streamed to keep only the local region close to the last known position in memory. Robust results are achieved by performing a geometrically stable selection of points, efficiently filtering outliers and integrating the relative tracker based on previous observations. We experimentally show that such a framework provides good localization results and that it scales well with the environment map size yielding real-time performance for both place recognition and tracking.

Ferns ensembles offer an accurate and efficient multiclass non-linear classification, commonly at the expense of consuming a large amount of memory. We introduce a two-fold contribution that produces large reductions in their memory consumption. First, an efficient L0 regularised cost optimisation finds a sparse representation of the posterior probabilities in the ensemble by discarding elements with zero contribution to valid responses in the training samples. As a by-product this can produce a prediction accuracy gain that, if required, can be traded for further reductions in memory size and prediction time. Secondly, posterior probabilities are quantised and stored in a memory-friendly sparse data structure. We reported a minimum of 75% memory reduction for different types of classification problems using generative and discriminative ferns ensembles, without increasing prediction time or classification error. For image patch recognition our proposal produced a 90% memory reduction, and improved in several percentage points the prediction accuracy.

Precise 3D mapping and 6DOF trajectory estimation using exteroceptive sensors are key problems in many fields. Real-time moving laser sensors gained popularity due to their precise depth measurements, high frame rate and large field of view. We propose an optimization framework for Simultaneous Localization And Mapping that properly models the acquisition process in a scanning-while-moving scenario. Each measurement is correctly reprojected in the map reference frame by considering a continuous time trajectory which is defined as the linear interpolation of a discrete set of control poses in SE3. The trajectory estimation is performed using the sensor readings only, i.e., no external motion measurement units are used. An efficient data structure that makes use of a hybrid sparse voxelized representation for large map management allows to perform global optimization over trajectories, resetting the accumulated drift when loops are detected. We experimentally show that such framework improves localization and mapping w.r.t. solutions that compensate the distortion effects without including them in the optimization step. Moreover, we show that the proposed map structure provides linear or constant time operations w.r.t. the map size in order to perform real time SLAM and it can handle very large maps.

We describe a method for non-invasive, accurate and efficient 3D reconstruction of occluded scenes, from a minimal number of X-ray and range scan image acquisitions. The residuals of generalised epipolar constraints (GEC) are incorporated in a highly efficient bundle adjustment minimization, to obtain maximum likelihood estimations of the X-ray image calibration parameters from correspondences between scene points, image points and apparent contours of scene objects. Furthermore, we propose a multimodal template adequate for accurate joint calibration of X-ray and range scan images. It offers crucial advantages for security applications, such as minimal scene occlusion and an agile data acquisition. Finally we describe a shape-from-silhouettes method based on the state of the art, able to reconstruct scene objects with general 3D shapes. We combine these proposals in a full system for 3D reconstruction of occluded scenes, and use it to demonstrate the practical and computational advantages of the methods herein described, with respect to previous proposals, using both synthetic and real data experiments.

Closed receptacle contents can be reconstructed without invasive approaches by coupling X-ray and 3D data acquisitions. We present a solution to the reconstruction problem that exploits 3D data acquisitions of the environment to initialize the X-ray projection calibration and to align the final reconstruction to a global frame. In particular we show that the problem can be solved, without employing calibration patterns, by precisely locating the X-ray emitters and receiver plates w.r.t. a global reference frame. We then exploit matching contours extracted from the X-ray images to calibrate the projection system and to perform the reconstruction of parametric objects. We provide experimental results to measure the reconstruction precision of the overall system using both synthetic data and real experiments.

This paper addresses the problem of training of operators for telemanipulation tasks. In particular, it describes the development of a physics based virtual environment that allows a user to train in the control of an innovative robotic tool designed for the retrieval of spent nuclear fuels.

This paper presents a new robotic device whose main objective is to increase the flexibility of current systems used to carry out inspection and manipulation of spent nuclear fuel inside dry storage unit cells. Instead of using a rigid kinematic chain, the proposed device is based on the idea of a capsule that can be deployed inside a cell with the appropriate tool by means of a conventional crane and cable. The device is equipped with extendable actuators that immobilize the whole system inside the cell by pushing against the walls. This provides a stable platform for those tools that have to execute precision tasks. The system also comprises an industrial tool changer that allows the deployment of the required tool for each specific task. A first prototype of the device and a tool to retrieve spent nuclear fuel were built, and the results of the initial experiments are reported at the end of this paper.

We present an efficient method for accurate automatic registration of two geometrically complex 3D range scans by using 3D lines. We first detect edges from the associated 2D reflectance images and collect 3D edge contours by only taking into account valid foreground points. Then we use an efficient split-and-merge line fitting algorithm to detect 3D lines. We build a fast search codebook to efficiently match the two sets of 3D lines. This is done by computing the orientation angle and distance of pairs of 3D lines in each set, both of which are invariant under rigid transformations. Finally we recover the rigid transformation between two scans using an efficient RANSAC algorithm with robust transformation estimation that exploits two sets of corresponding 3D lines. We conclude presenting experimental results that demonstrate efficiency and accuracy of our proposed method.

This paper presents a process combining range and intensity based techniques, in order to get better 3D models than those obtained using these techniques separately. The procedure needs an initial estimation for internal and external camera parameters for two or more intensity images. The technique uses passive triangulation to refine initial camera calibrations and ensure a good registration of range and video data sets. Afterwards, corresponding points from the intensity images are triangulated and introduced in the original range cloud of points. The objective is to complete the models in areas where data is missing or to increase the resolution in areas of high interest and 3D contents.

In this paper, we present techniques combining intensity and range data to achieve more realistic three-dimensional models from real world scenes. The principal novelty is the use of the intensity information not only for texturing the final 3D model but also to improve the segmentation of the range data. The technique registers video and range, i.e., the camera parameters are computed in order to get both data in the same reference frame. Edge extraction techniques are then applied to select corresponding edges and a line-straightening algorithm is used to correct the 3D model. Results are presented for both indoor and outdoor environments.

In recent years, several methods of reconstructing 3D scenes from range images have been proposed. Currently, the factor limiting a fully automated reconstruction process is the lack of a sufficiently scalable planning algorithm for the acquisition step. We present a technique of iteratively planning the next view for a range scanner in an initially unknown, large indoor environment. When designing an algorithm suitable for architectural environments, two main problems have to be addressed. First, an objective function has to be defined to optimise the parameters of the next view. We propose an objective function based on the analysis of occlusions which takes into account both a quality criterion and the cost of the next acquisition. Second, the parameters of the next view have to be computed efficiently for a large search space with eight degrees of freedom. Our technique exploits hardware-accelerated rendering (OpenGL) in order to perform the expensive visibility computation, which reduces the computation time of one planning step to a few minutes. Results are shown for two large indoor scenes-an artificial scene and a real world room-with numerous self occlusions.

When aiming at the automated reconstruction of real world scenes from range images, one has to address the problem of planning the image acquisition. Although solutions for small objects in well defined environments are already available, the insufficient scalability of these approaches to large scenes and to a high number of degrees of freedom limits their applicability. In this paper we present a new planning algorithm with emphasis on practical usability in initially unknown, large indoor environments. Using a surface representation of seen and unseen parts of the environment, we propose an objective function based on the analysis of occlusions. In addition to previous approaches, we take into account both a quality criterion and the cost of the next acquisition. By optimising this objective function, the parameters of the next view are computed efficiently for a large search space with eight degrees of freedom (3D position, viewing direction, field of view, and resolution). Our technique exploits hardware-accelerated rendering (OpenGL) in order to perform the expensive visibility computation, which reduces the computation time of one planning step to a couple of minutes. Results are shown for two large indoor scenes-an artificial scene and a real world room-with numerous self occlusions.

This paper presents a new 3D scene analysis system that automatically reconstructs the 3D model of real-world scenes from multiple range images acquired by a laser range finder on board of a mobile robot. The reconstruction is achieved through an integrated procedure including range data acquisition, geometrical feature extraction, planning the next view, registration and integration of multiple views. The system relies only on the acquired data. Experimental results on real range images are presented in the paper. Direct applications of this technique include 3D reconstruction and/or update of architectural or industrial plans into a CAD model, design verification of buildings, navigation of autonomous robots, and input to virtual reality systems.

This paper presents an active sensing process for the efficient reconstruction of a real world environment using multiple views of a scene captured by a laser range finder onboard a mobile robot. To resolve the ambiguities caused by occlusions there is the need to acquire multiple range views at different viewpoints. A strategy for optimising the selection of the next sensing position and orientation based on previously detected occlusions is proposed. No a priori knowledge about the scene is used for planning the next views. The system evaluates the set of capture points from which all the necessary data can be acquired aiming at minimising its number. Among these points it then selects the one(s) maximising the occluded areas to be resolved.