Humans learn to recognize categories of objects, even when exposed to little language supervision. Behavioral studies and the successes of self-supervised learning (SSL) models suggest that this learning may hinge on modeling spatial regularities of visual features. However, SSL models rely on geometric image augmentations such as masking portions of an image or aggressively cropping it, which are not known to be performed by the brain. Here, we propose CO-SSL, an alternative to geometric image augmentations to model spatial co-occurrences. CO-SSL aligns local representations (before pooling) with a global image representation. Combined with a neural network endowed with small receptive fields, we show that it outperforms previous methods by up to on ImageNet-1k when not using cropping augmentations. In addition, CO-SSL can be combined with cropping image augmentations to accelerate category learning and increases the robustness to internal corruptions and small adversarial attacks. Overall, our work paves the way towards a new approach for modeling biological learning and developing self-supervised representations in artificial systems.

While a traditional camera only captures one point of view of a scene, a plenoptic or light-field camera, is able to capture spatial and angular information in a single snapshot, enabling depth estimation from a single acquisition. In this paper, we present a new metric depth estimation algorithm using only raw images from a multi-focus plenoptic camera. The proposed approach is especially suited for the multi-focus configuration where several micro-lenses with different focal lengths are used. The main goal of our blur aware depth estimation (BLADE) approach is to improve disparity estimation for defocus stereo images by integrating both correspondence and defocus cues. We thus leverage blur information where it was previously considered a drawback. We explicitly derive an inverse projection model including the defocus blur providing depth estimates up to a scale factor. A method to calibrate the inverse model is then proposed. We thus take into account depth scaling to achieve precise and accurate metric depth estimates. Our results show that introducing defocus cues improves the depth estimation. We demonstrate the effectiveness of our framework and depth scaling calibration on relative depth estimation setups and on real-world 3D complex scenes with ground truth acquired with a 3D lidar scanner.

Abstract This paper presents a novel calibration algorithm for plenoptic cameras, especially the multi-focus configuration, where several types of micro-lenses are used, using raw images only. Current calibration methods rely on simplified projection models, use features from reconstructed images, or require separated calibrations for each type of micro-lens. In the multi-focus configuration, the same part of a scene will demonstrate different amounts of blur according to the micro-lens focal length. Usually, only micro-images with the smallest amount of blur are used. In order to exploit all available data, we propose to explicitly model the defocus blur in a new camera model with the help of our newly introduced Blur Aware Plenoptic (BAP) feature. First, it is used in a pre-calibration step that retrieves initial camera parameters, and second, to express a new cost function to be minimized in our single optimization process. Third, it is exploited to calibrate the relative blur between micro-images. It links the geometric blur, i.e., the blur circle, to the physical blur, i.e., the point spread function. Finally, we use the resulting blur profile to characterize the camera’s depth of field. Quantitative evaluations in controlled environment on real-world data demonstrate the effectiveness of our calibrations.

This paper presents a novel calibration algorithm for Multi-Focus Plenoptic Cameras (MFPCs) using raw images only. The design of such cameras is usually complex and relies on precise placement of optic elements. Several calibration procedures have been proposed to retrieve the camera parameters but relying on simplified models, reconstructed images to extract features, or multiple calibrations when several types of micro-lens are used. Considering blur information, we propose a new Blur Aware Plenoptic (BAP) feature. It is first exploited in a pre-calibration step that retrieves initial camera parameters, and secondly to express a new cost function for our single optimization process. The effectiveness of our calibration method is validated by quantitative and qualitative experiments.

We present a temporal Bayesian filter for semantic segmentation of a video sequence. Each pixel is a random variable following a discrete probabilistic distribution function representing possible semantic classes. Bayesian filtering consists in two main steps: 1) a prediction model and 2) an observation model (likelihood). We propose to use a datadriven prediction function derived from a dense optical flow between images t and t + 1 achieved by a deep neural network [1]. Moreover, the observation function uses a semantic segmentation network. The resulting approach is evaluated on the public dataset Cityscapes. We show that using the temporal filtering increases the accuracy of the semantic segmentation.

Learning complex behaviors through reinforcement learning is particularly challenging when reward is only available upon successful completion of the full behavior. In manipulation robotics, so-called shaping rewards are often used to overcome this problem. However, these usually require human engineering or (partial)world models describing, e.g., the kinematics of the robot or high-level modules for perception. Here we propose an alternative method to learn an object palm-touching task through a weakly-supervised and stagewise learning of simpler tasks. First, the robot learns to fixate the object with its cameras. Second, the robot learns eye-hand coordination by learning to fixate its end effector. Third, using the previously acquired skills an informative shaping reward can be computed which facilitates efficient learning of the object palm-touching task. We demonstrate in simulation that learning the full task with this shaping reward is comparable to learning with an informative supervised reward.

In this paper, we present a novel approach, called Deep MANTA (Deep Many-Tasks), for many-task vehicle analysis from a given image. A robust convolutional network is introduced for simultaneous vehicle detection, part localization, visibility characterization and 3D dimension estimation. Its architecture is based on a new coarse-to-fine object proposal that boosts the vehicle detection. Moreover, the Deep MANTA network is able to localize vehicle parts even if these parts are not visible. In the inference, the networks outputs are used by a real time robust pose estimation algorithm for fine orientation estimation and 3D vehicle localization. We show in experiments that our method outperforms monocular state-of-the-art approaches on vehicle detection, orientation and 3D location tasks on the very challenging KITTI benchmark.

Due to its ability to learn complex behaviors in high-dimensional state-action spaces, deep reinforcement learning algorithms have attracted much interest in the robotics community. For a practical reinforcement learning implementation on a robot, it has to be provided with an informative reward signal that makes it easy to discriminate the values of nearby states. To address this issue, prior information, e.g. in the form of a geometric model, or human supervision are often assumed. This paper proposes a method to learn binocular fixations without such prior information. Instead, it uses an informative reward requiring little supervised information. The reward computation is based on an anomaly detection mechanism which uses convolutional autoencoders. These detectors estimate in a weakly supervised way an object's pixellic position. This position estimate is affected by noise, which makes the reward signal noisy. We first show that this affects both the learning speed and the resulting policy. Then, we propose a method to partially remove the noise using regression on the detection change given sensor data. The binocular fixation task is learned in a simulated environment on an object training set with various shapes and colors. The learned policy is compared with another one learned with a highly informative and noiseless reward signal. The tests are carried out on the training set and on a test set of new objects. We observe similar performances, showing that the environment-encoding step can replace the prior information.

Given an untextured 3D car models dataset, are we able to learn a robust make and model classifier which will be applied on real color images? One solution consists in finding a common representation between synthetic edge images and real color images. To address this issue, we introduce novel edge-color invariant features for 2D/3D car fine-grained classification. These features are learned simultaneously on real color and edge images of cars using a deep architecture. Using these accurate features, we propose to learn on edges synthetic images a fine-grained classifier which will be afterwards applied to real color images. The 3D car models dataset allows to automatically generate multi-views synthetic images using non-photorealistic edge rendering. Experimentally, we show efficiency of our learned edge-color invariant features for make and model recognition.

We propose a new approach for accurate car pose estimation in images using only a dataset of 3D untextured models. Our algorithm detects both a car and its 3D pose. It is based on the matching of 3D models with the car in the image. With a part detector based on Convolutional Neural Networks, interest points corresponding to predefined 3D parts are extracted from the image. Then, we use the car geometry to find which parts are relevant across viewpoints. Finally, a 2D/3D pose estimator is used to recover the 3D pose of the car. The main contribution is to learn appearance and geometry models from 3D models dataset only. Experiments show that the method is very competitive for car detection and coarse viewpoint classification and improves the 3D pose estimation over the state-of-the-art methods.

This paper proposes a novel model-based tracking approach for 3-D localization. One main difficulty of standard model-based approach lies in the presence of low-level ambiguities between different edges. In this paper, given a 3-D model of the edges of the environment, we derive a multiple hypotheses tracker which retrieves the potential poses of the camera from the observations in the image. We also show how these candidate poses can be integrated into a particle filtering framework to guide the particle set toward the peaks of the distribution. Motivated by the UAV indoor localization problem where GPS signal is not available, we validate the algorithm on real image sequences from UAV flights.

This paper presents a novel 3-D servoing approach using dense depth maps to perform robotic tasks. With respect to position-based approaches, our method does not require the estimation of the 3-D pose (direct), nor the extraction and matching of 3-D features (dense), and only requires dense depth maps provided by 3-D sensors. Our approach has been validated in various servoing experiments using the depth information from a low-cost Red Green Blue-Depth (RGB-D) sensor. Positioning tasks are properly achieved despite noisy measurements, even when partial occlusions or scene modifications occur. We also show that, in cases where a reference depth map cannot be easily available, synthetic ones generated with a rendering engine still lead to satisfactory positioning performances. Application of the approach to the navigation of a mobile robot is also demonstrated.

This paper proposes a novel 3D servoing approach using dense depth maps to achieve robotic tasks. With respect to position-based approaches, our method does not require the estimation of the 3D pose (direct), nor the extraction and matching of 3D features (dense) and only requires dense depth maps provided by 3D sensors. Our approach has been validated in servoing experiments using the depth information from a low cost RGB-D sensor. Positioning tasks are properly achieved despite the noisy measurements, even when partial occlusions or scene modifications occur.

This paper proposes a vision-based algorithm to autonomously track and chase a moving target with a small-size flying UAV. The challenging constraints associated with the UAV flight led us to consider a density-based representation of the object to track. The proposed approach to estimate the target's position, orientation and scale, is built on a robust color-based tracker using a multi-part representation. This object tracker can handle large displacements, occlusions and account for some image noise due to partial loss of wireless video link, thanks to the use of a particle filter. The information obtained from the visual tracker is then used to control the position and yaw angle of the UAV in order to chase the target. A hierarchical control scheme is designed to achieve the tracking task. Experiments on a quad-rotor UAV following a small moving car are provided to validate the proposed approach.

Classic registration methods for model-based tracking try to align the projected edges of a 3D model with the edges of the image. However, wrong matches at low level can make these methods fail. This paper presents a new approach allowing to retrieve multiple hypothesis on the camera pose from multiple low-level hypothesis. These hypothesis are integrated into a particle filtering framework to guide the particle set toward the peaks of the distribution. Experiments on simulated and real video sequences show the improvement in robustness of the resulting tracker.

This paper presents a 3D model-based tracking suitable for indoor position control of an unmanned aerial vehicle (UAV). Given a 3D model of the edges of its environment, the UAV locates itself thanks to a robust multiple hypothesis tracker. The pose estimation is then fused to inertial data to provide the translational velocity required for the control. A hierarchical control is used to achieve positioning tasks. Experiments on a quad-rotor aerial vehicle validate the proposed approach.

Color-based tracking methods have proved to be efficient for their robustness qualities. The drawback of such global representation of an object is the lack of information on its spatial configuration, making difficult the tracking of more complex motions. This issue can be overcome by using several kernels weighting pixels locations.