Lenslet-based light field cameras offer rich geometric and radiometric data enabling advanced imaging applications such as 3D geometry estimation and novel view synthesis supporting complex illumination phenomena. However, manufacturing variations necessitate robust calibration techniques where metrically grounded measurements are desired. This work introduces a principal plane and pupil-centric camera model that addresses limitations in existing calibration methods that prevent their use in commonly occurring optical configurations including telephoto and wide-field modes. By incorporating pupil and principal plane locations and pupil magnification, we derive a numerically stable, generally applicable, and physically grounded intrinsic model, a 4D distortion model describing complex aberrations, and a distance-normalized image-space reprojection error for balanced performance across spatial and angular domains. We evaluate on two commercial camera types across five focal configurations, comparing to two state of the art methods, and show superior reprojection and pose estimation accuracy and stable operation across all modes. We also show improved rectification performance in both view synthesis and epipolar plane consistency. Our approach generalizes to a variety of light field cameras including commercially available and bespoke systems, providing a robust foundation for calibrated light field processing. Code and data are publicly available.

The majority of image processing approaches assume images are in or can be rectified to a perspective projection. However, in many applications it is beneficial to use non conventional cameras, such as fisheye cameras, that have a larger field of view (FOV). The issue arises that these large-FOV images can not be rectified to a perspective projection without significant cropping of the original image. To address this, we propose Rectified Convolutions (RectConv); a new approach for adapting pre-trained convolutional networks to operate with new non-perspective images, without any retraining. Replacing the convolutional layers of the network with RectConv layers allows the network to see both rectified patches and the entire FOV. We demonstrate RectConv adapting multiple pre-trained networks to perform segmentation and detection on fisheye imagery from two publicly available datasets. Our method requires no additional data or training, and operates directly on the native image as captured from the camera. We believe this work represents a step toward extending the vast resources available for perspective images to operate across a broad range of camera geometries.

There are a multitude of emerging imaging technologies that could benefit robotics. However the need for bespoke models, calibration and low-level processing represents a key barrier to their adoption. In this work we present NOCaL, Neural Odometry and Calibration using Light fields, a semi-supervised learning architecture capable of interpreting previously unseen cameras without calibration. NOCaL learns to estimate camera parameters, relative pose, and scene appearance. It employs a scene-rendering hypernetwork pre-trained on a large number of existing cameras and scenes, and adapts to previously unseen cameras using a small supervised training set to enforce metric scale. We demonstrate NOCaL on rendered and captured imagery using conventional cameras, demonstrating calibration-free odometry and novel view synthesis. This work represents a key step toward automating the interpretation of general camera geometries and emerging imaging technologies. Code and datasets are available at https://roboticimaging.org/Projects/NOCaL/.

While an exciting diversity of new imaging devices is emerging that could dramatically improve robotic perception, the challenges of calibrating and interpreting these cameras have limited their uptake in the robotics community. In this work we generalise techniques from unsupervised learning to allow a robot to autonomously interpret new kinds of cameras. We consider emerging sparse light field (LF) cameras, which capture a subset of the 4D LF function describing the set of light rays passing through a plane. We introduce a generalised encoding of sparse LFs that allows unsupervised learning of odometry and depth. We demonstrate the proposed approach outperforming monocular, stereo and conventional techniques for dealing with 4D imagery, yielding more accurate odometry and depth maps and delivering these with metric scale. We anticipate our technique to generalise to a broad class of LF and sparse LF cameras, and to enable unsupervised recalibration for coping with shifts in camera behaviour over the lifetime of a robot. This work represents a first step toward streamlining the integration of new kinds of imaging devices in robotics applications.