Accurate intrinsic calibration is essential for camera-based 3D perception, yet, it typically requires targets of well-known geometry. Here, we propose a camera self-calibration approach that infers camera intrinsics during application, from monocular videos in the wild. We propose to explicitly model projection functions and multi-view geometry, while leveraging the capabilities of deep neural networks for feature extraction and matching. To achieve this, we build upon recent research on integrating bundle adjustment into deep learning models, and introduce a self-calibrating bundle adjustment layer. The self-calibrating bundle adjustment layer optimizes camera intrinsics through classical Gauß-Newton steps and can be adapted to different camera models without re-training. As a specific realization, we implemented this layer within the deep visual SLAM system DROID-SLAM, and show that the resulting model, DroidCalib, yields state-of-the-art calibration accuracy across multiple public datasets. Our results suggest that the model generalizes to unseen environments and different camera models, including significant lens distortion. Thereby, the approach enables performing 3D perception tasks without prior knowledge about the camera. Code is available at https://github.com/boschresearch/droidcalib.

Most approaches to camera calibration rely on calibration targets of well-known geometry. During data acquisition, calibration target and camera system are typically moved w.r.t. each other, to allow image coverage and perspective versatility. We show that moving the target can lead to small temporary deformations of the target, which can introduce significant errors into the calibration result. While static inaccuracies of calibration targets have been addressed in previous works, to our knowledge, none of the existing approaches can capture time-varying, dynamic deformations. To achieve high-accuracy calibrations despite moving the target, we propose a way to explicitly model dynamic target deformations in camera calibration. This is achieved by using a low-dimensional deformation model with only few parameters per image, which can be optimized jointly with target poses and intrinsics. We demonstrate the effectiveness of modeling dynamic deformations using different calibration targets and show its significance in a structure-from-motion application.

It is well known that the robustness of many computer vision algorithms can be improved by employing large field of view cameras, such as omnidirectional cameras. To avoid obstructions in the field of view, such cameras need to be mounted in an exposed position. Alternatively, a multicamera setup can be used. However, this requires the extrinsic calibration to be known. In the present work, we propose a method to calibrate a fisheye multi-camera rig, mounted on a mobile platform. The method only relies on feature correspondences from pairwise overlapping fields of view of adjacent cameras. In contrast to existing approaches, motion estimation or specific motion patterns are not required. To compensate for the large extent of multi-camera setups and corresponding viewpoint variations, as well as geometrical distortions caused by fisheye lenses, captured images are mapped into virtual camera views such that corresponding image regions coincide. To this end, the scene geometry is approximated by the ground plane in close proximity and by infinitely far away objects elsewhere. As a result, low complexity feature detectors and matchers can be employed. The approach is evaluated using a setup of four rigidly coupled and synchronized wide angle fisheye cameras that were attached to four sides of a mobile platform. The cameras have pairwise overlapping fields of view and baselines between 2.25 and 3 meters.

Knowledge of motion with respect to the ground plane is required in many computer vision applications such as obstacle avoidance, egomotion estimation, and online calibration. The homography matrix comprises motion as well as ground plane information. Estimation of the homography matrix is challenging, as measurements are often not only corrupted by sparse gross outliers, but might also contain other structures, which are inconsistent with the ground plane such as curbstones and sidewalks. Several well studied algorithms regarding the identification of sparse gross outliers already exist. However, identifying structural outliers remains a challenging problem due the outliers' inner coherence. In homography and plane estimation structural outliers often cause plane fits that do not correspond to any physical plane in the scene. We make use of the large field of view of fisheye cameras by exploiting that outlier identification can be performed more robustly in the near field where motion parallax vectors are large. More sensitive data can then be tested subsequently based on the preceding results. The main contribution of this paper is twofold. First, we present a statistical analysis of parallax amplitudes that are to be expected due to the distance of a point from the ground plane and measurement noise. This leads to a statistical test for outliers with local adaptive thresholds. Second, we embed this concept into an extended Kalman filter for efficient processing. Furthermore, we emphasize the importance of warping captured images into a common frame previous to feature detection and matching to avoid distortion effects and to equalize search regions. We demonstrate the robustness of our approach and the effects of prewarping on the estimation using real data.

This paper presents an approach for online estimation of the extrinsic calibration parameters of a multi-camera rig. Given a coarse initial estimate of the parameters, the relative poses between cameras are refined through recursive filtering. The approach is purely vision based and relies on plane induced homographies between successive frames. Overlapping fields of view are not required. Instead, the ground plane serves as a natural reference object. In contrast to other approaches, motion, relative camera poses, and the ground plane are estimated simultaneously using a single iterated extended Kalman filter. This reduces not only the number of parameters but also the computational complexity. Furthermore, an arbitrary number of cameras can be incorporated. Several experiments on synthetic as well as real data were conducted using a setup of four synchronized wide angle fisheye cameras, mounted on a moving platform. Results were obtained, using both, a planar and a general motion model with full six degrees of freedom. Additionally, the effects of uncertain intrinsic parameters and nonplanar ground were evaluated experimentally.

In this paper, we introduce a new algorithm for continuous estimation of camera motion and zoom. We show that this method can be used to warp previously encoded frames into the perspective of the frame that is currently encoded. In particular, for so-called video or dynamic texture, that is, sequences exhibiting some kind of temporal periodicity or stationarity, a frame with a given temporal distance can be warped and added into the reference picture buffer to complement conventional reference frames. The rationale is that with these sequences, frames, or parts of frames, tend to reappear in a similar form after some time. We show that by combining a specifically tailored method for frame warping with exploitation of dynamic texture properties, bitrate savings surpassing the state-of-the-art can be achieved with a recent version of the HM software.

Although facial features are considered to be essential for humans to understand sign language, no prior research work has yet examined their significance for automatic sign language recognition or presented some evaluation results. This paper describes a vision-based recognition system that employs both manual and facial features, extracted from the same input image. For facial feature extraction an active appearance model is applied to identify areas of interest such as the eyes and mouth region. Afterwards a numerical description of facial expression and lip outline is computed. An extensive evaluation was performed on a new sign language corpus, which contains continuous articulations of 25 native signers. The obtained results proved the importance of integrating facial expressions into the classification process. The recognition rates for isolated and continuous signing increased in signer-dependent as well as in signer-independent operation mode. Interestingly, roughly two of ten signs were recognized just from the facial features.