Inverse rendering is an ill-posed problem. Previous work has sought to resolve this by focussing on priors for object or scene shape or appearance. In this work, we instead focus on a prior for natural illuminations. Current methods rely on spherical harmonic lighting or other generic representations and, at best, a simplistic prior on the parameters. This results in limitations for the inverse setting in terms of the expressivity of the illumination conditions, especially when taking specular reflections into account. We propose a conditional neural field representation based on a variational auto-decoder and a transformer decoder. We extend Vector Neurons to build equivariance directly into our architecture, and leveraging insights from depth estimation through a scale-invariant loss function, we enable the accurate representation of High Dynamic Range (HDR) images. The result is a compact, rotation-equivariant HDR neural illumination model capable of capturing complex, high-frequency features in natural environment maps. Training our model on a curated dataset of 1.6 K HDR environment maps of natural scenes, we compare it against traditional representations, demonstrate its applicability for an inverse rendering task and show environment map completion from partial observations.

Humans excel at constructing panoramic mental models of their surroundings, maintaining object permanence and inferring scene structure beyond visible regions. In contrast, current artificial vision systems struggle with persistent, panoramic understanding, often processing scenes egocentrically on a frame-by-frame basis. This limitation is pronounced in the Track Any Point (TAP) task, where existing methods fail to track 2D points outside the field of view. To address this, we introduce TAP-Vid 360, a novel task that requires predicting the 3D direction to queried scene points across a video sequence, even when far outside the narrow field of view of the observed video. This task fosters learning allocentric scene representations without needing dynamic 4D ground truth scene models for training. Instead, we exploit 360 videos as a source of supervision, resampling them into narrow field-of-view perspectives while computing ground truth directions by tracking points across the full panorama using a 2D pipeline. We introduce a new dataset and benchmark, TAP360-10k comprising 10k perspective videos with ground truth directional point tracking. Our baseline adapts CoTracker v3 to predict per-point rotations for direction updates, outperforming existing TAP and TAP-Vid 3D methods.

Gaze estimation methods typically regress gaze directions directly from images using a deep network. We show that equipping a deep network with an explicit 3D shape model can: i) improve gaze estimation accuracy, ii) perform well with lower resolution inputs at high frame rates and, importantly, iii) provide a much richer understanding of the eye-region and its constituent gaze system, thus lending itself to a wider range of applications. We use an ‘eyes and nose’ 3D Morphable Model (3DMM) to capture relevant local 3D facial geometry and appearance, and we equip this with a geometric vergence model of gaze to give an ‘active-gaze 3DMM’. Latent codes are used to express eye-region shape, appearance, pose, scale and gaze directions, with these being regressed using a tiny Swin transformer. We achieve fast real time at 89 fps without fitted model rendering and 34 fps with rendering. Our system shows state-of-the-art results on the Eyediap dataset, which provides 3D training supervision and highly competitive results on ETH-XGaze, despite a lack of 3D supervision and without modelling the kappa angle. Indeed, our method can learn with only the ground truth gaze target point and the camera parameters, without access to the ground truth gaze origin points, thus significantly widening applicability.

We present a progressive 3D registration framework that is a highly-efficient variant of classical non-rigid Iterative Closest Points (N-ICP). Since it uses the Laplace-Beltrami operator for deformation regularisation, we view the overall process as Laplacian ICP (L-ICP). This exploits a ‘small deformation per iteration’ assumption and is progressively coarse-to-fine, employing an increasingly flexible deformation model, an increasing number of correspondence sets, and increasingly sophisticated correspondence estimation. Correspondence matching is only permitted within predefined vertex subsets derived from domain-specific feature extractors. Additionally, we present a new benchmark and a pair of evaluation metrics for 3D non-rigid registration, based on annotation transfer. We use this to evaluate our framework on a publicly-available dataset of 3D human head scans (Headspace). The method is robust and only requires a small fraction of the computation time compared to the most popular classical approach, yet has comparable registration performance.

We propose a self-supervised method for relative pose estimation for road scenes. By exploiting the approximate planarity of the local ground plane, we can extract a self-supervision signal via cross-projection between images using a homography derived from estimated ground-relative pose. We augment cross-projected perceptual loss by including classical image alignment in the network training loop. We use pretrained semantic segmentation and optical flow to extract ground plane correspondences between approximately aligned images and RANSAC to find the best fitting homography. By decomposing to ground-relative pose, we obtain pseudo labels that can be used for direct supervision. We show that this extremely simple geometric model is competitive for visual odometry with much more complex self-supervised methods that must learn depth estimation in conjunction with relative pose. Code and result videos: github.com/brucemuller/homographyVO.

While omni-directional sensors provide holistic representations typical deep learning frameworks reduce the benefits by introducing distortions and discontinuities as spherical data is supplied as planar input. On the other hand, recent spherical convolutional neural networks (CNNs) often require significant memory and parameters, thus enabling execution only at very low resolutions and shallow architectures. We propose HexNet, an orientation-aware deep learning framework for spherical signals, that allows for fast computation as we exploit standard planar network operations on an efficiently arranged projection of the sphere. Furthermore, we introduce a graph-based version for partial spheres, allowing us to compete at high-resolution with planar CNNs using residual network architectures. Our kernels operate on the tangent of the sphere and thus standard feature weights, pretrained on perspective data, can be transferred, enabling spherical pretraining on ImageNet. As our design is free of distortions and discontinuity, our orientation-aware CNN becomes a new state of the art for semantic segmentation on the recent 2D3DS dataset, and the omni-directional version of SYNTHIA introduced in this work. Moreover, we experimentally show the benefit of our spherical representation over standard images on the Cityscapes dataset by reducing distortion effects of planar CNNs. We implement object detection for the spherical domain. Rotation invariant classification and segmentation tasks are additionally presented for comparison to prior art.

We propose a self-supervised method for relative pose estimation. Unlike existing self-supervised methods, we do not train a dense depth estimation network in conjunction with our pose network and hence avoid the complexity and ambiguity of this much harder problem. Instead, we use a very simple geometric model in which we assume the local road scene is planar. By estimating a 9D ground-relative pose, we are able to perform cross-projection between images via the ground plane using only a homography to compute a self-supervised appearance loss between overlapping images. We use a geometric matching architecture that can handle arbitrary pose pairs and use a pretrained feature extractor to compute a perceptual appearance loss. Our approach is competitive with more complex visual odometry methods that estimate dense depth maps. Code: github.com/brucemuller/homographyVO

Inverse rendering is an ill-posed problem. Previous work has sought to resolve this by focussing on priors for object or scene shape or appearance. In this work, we instead focus on a prior for natural illuminations. Current methods rely on spherical harmonic lighting or other generic representations and, at best, a simplistic prior on the parameters. We propose a conditional neural field representation based on a variational auto-decoder with a SIREN network and, extending Vector Neurons, build equivariance directly into the network. Using this, we develop a rotation-equivariant, high dynamic range (HDR) neural illumination model that is compact and able to express complex, high-frequency features of natural environment maps. Training our model on a curated dataset of 1.6K HDR environment maps of natural scenes, we compare it against traditional representations, demonstrate its applicability for an inverse rendering task and show environment map completion from partial observations.

We consider the problem of structure-from-motion (SfM) for images with fixed calibration but varying principal point. This scenario occurs for archival imagery taken using historic glass plate and film cameras without fiducial markers, when images have been inconsistently cropped or when image plates are broken into multiple fragments.We derive initialisation and pose estimation methods and regularisation penalties tuned specifically for this scenario leading to a complete archival imagery SfM pipeline. We illustrate the performance of our methods on challenging real world examples from image archives. Specifically, we use archival images of the East coast of Greenland from the British Arctic Air Route Expedition (BAARE). This is of particular glaciological interest for measuring historic ice loss. We use a modern digital elevation model (ArcticDEM), masked to stable regions, as ground truth to evaluate our method.

In this paper we present methods for estimating shape from polarisation and shading information, i.e. photo-polarimetric shape estimation, under varying, but unknown, illumination, i.e. in an uncalibrated scenario. We propose several alternative photo-polarimetric constraints that depend upon the partial derivatives of the surface and show how to express them in a unified system of partial differential equations of which previous work is a special case. By careful combination and manipulation of the constraints, we show how to eliminate non-linearities such that a discrete version of the problem can be solved using linear least squares. We derive a minimal, combinatorial approach for two source illumination estimation which we use with RANSAC for robust light direction and intensity estimation. We also introduce a new method for estimating a polarisation image from multichannel data and provide methods for estimating albedo and refractive index. We evaluate lighting, shape, albedo and refractive index estimation methods on both synthetic and real-world data showing improvements over existing state-of-the-art.

In this paper we show how to perform scene-level inverse rendering to recover shape, reflectance and lighting from a single, uncontrolled image using a fully convolutional neural network. The network takes an RGB image as input, regresses albedo, shadow and normal maps from which we infer least squares optimal spherical harmonic lighting coefficients. Our network is trained using large uncontrolled multiview and timelapse image collections without ground truth. By incorporating a differentiable renderer, our network can learn from self-supervision. Since the problem is ill-posed we introduce additional supervision. Our key insight is to perform offline multiview stereo (MVS) on images containing rich illumination variation. From the MVS pose and depth maps, we can cross project between overlapping views such that Siamese training can be used to ensure consistent estimation of photometric invariants. MVS depth also provides direct coarse supervision for normal map estimation. We believe this is the first attempt to use MVS supervision for learning inverse rendering. In addition, we learn a statistical natural illumination prior. We evaluate performance on inverse rendering, normal map estimation and intrinsic image decomposition benchmarks.

We tackle the problem of building city- or country-scale seamless mosaics of the road network from millions of street-level images. These “orthomosaics” provide a virtual top-down, orthographic view, as might be captured by a satellite though at vastly reduced cost and avoiding limitations caused by atmospheric interference or occlusion by tree cover. We propose a novel, highly efficient planar visual odometry method that scales to millions of images. This includes a fast search for potentially overlapping images, relative pose estimation from approximate ground plane projected images and a largescale optimisation, which we call motion-from-homographies, that exploits multiple motion, GPS and control point priors. Since even city-scale orthomosaics have petapixel resolution, we work with a tile-based mosaic representation which is more efficient to compute and makes web-based, real-time interaction with the images feasible. Our orthomosaics are seamless both within tiles and across tile boundaries due to our proposed novel variant of gradient-domain stitching. We show that our orthomosaics are qualitatively superior to those produced using state-of-the-art structure-from-motion output yet our pose optimisation is several orders of magnitude faster. We evaluate our methods on a dataset of 1.4M images that we collected.

In this paper, we show how to estimate shape (restricted to a single object class via a 3D morphable model) using solely a semantic segmentation of a single 2D image. We propose a novel loss function based on a probabilistic, vertex-wise projection of the 3D model to the image plane. We represent both these projections and pixel labels as mixtures of Gaussians and compute the discrepancy between the two based on the geometric Rényi divergence. The resulting loss is differentiable and has a wide basin of convergence. We propose both classical, direct optimisation of this loss ("analysis-by-synthesis") and its use for training a parameter regression CNN. We show significant ad-vantages over existing segmentation losses used in state-of-the-art differentiable renderers Soft Rasterizer and Neural Mesh Renderer.

Three-dimensional morphable models (3DMMs) are powerful statistical tools for representing the 3D shapes and textures of an object class. Here we present the most complete 3DMM of the human head to date that includes face, cranium, ears, eyes, teeth and tongue. To achieve this, we propose two methods for combining existing 3DMMs of different overlapping head parts: (i). use a regressor to complete missing parts of one model using the other, and (ii). use the Gaussian Process framework to blend covariance matrices from multiple models. Thus, we build a new combined face-and-head shape model that blends the variability and facial detail of an existing face model (the LSFM) with the full head modelling capability of an existing head model (the LYHM). Then we construct and fuse a highly-detailed ear model to extend the variation of the ear shape. Eye and eye region models are incorporated into the head model, along with basic models of the teeth, tongue and inner mouth cavity. The new model achieves state-of-the-art performance. We use our model to reconstruct full head representations from single, unconstrained images allowing us to parameterize craniofacial shape and texture, along with the ear shape, eye gaze and eye color.

In this paper, we bring together two divergent strands of research: photometric face capture and statistical 3D face appearance modelling. We propose a novel lightstage capture and processing pipeline for acquiring ear-to-ear, truly intrinsic diffuse and specular albedo maps that fully factor out the effects of illumination, camera and geometry. Using this pipeline, we capture a dataset of 50 scans and combine them with the only existing publicly available albedo dataset (3DRFE) of 23 scans. This allows us to build the first morphable face albedo model. We believe this is the first statistical analysis of the variability of facial specular albedo maps. This model can be used as a plug in replacement for the texture model of the Basel Face Model and we make our new albedo model publicly available. We ensure careful spectral calibration such that our model is built in a linear sRGB space, suitable for inverse rendering of images taken by typical cameras. We demonstrate our model in a state of the art analysis-by-synthesis 3DMM fitting pipeline, are the first to integrate specular map estimation and outperform the Basel Face Model in albedo reconstruction.

Abstract We present a fully-automatic statistical 3D shape modeling approach and apply it to a large dataset of 3D images, the Headspace dataset, thus generating the first public shape-and-texture 3D morphable model (3DMM) of the full human head. Our approach is the first to employ a template that adapts to the dataset subject before dense morphing. This is fully automatic and achieved using 2D facial landmarking, projection to 3D shape, and mesh editing. In dense template morphing, we improve on the well-known Coherent Point Drift algorithm, by incorporating iterative data-sampling and alignment. Our evaluations demonstrate that our method has better performance in correspondence accuracy and modeling ability when compared with other competing algorithms. We propose a texture map refinement scheme to build high quality texture maps and texture model. We present several applications that include the first clinical use of craniofacial 3DMMs in the assessment of different types of surgical intervention applied to a craniosynostosis patient group.

We explore the use of convolutional neural networks (CNNs) for filling voids in digital elevation models (DEM). We propose a baseline approach using a fully convolutional network to predict complete from incomplete DEMs, which is trained in a supervised fashion. We then extend this to a shadow-constrained CNN (SCCNN) by introducing additional loss functions that encourage the restored DEM to adhere to geometric constraints implied by cast shadows. At the training time, we use automatically extracted cast shadow maps and known sun directions to compute the shadow-based supervisory signal in addition to the direct DEM supervision. At the test time, our network directly predicts restored DEMs from an incomplete DEM. One key advantage of our SCCNN model is that it is characterized by both CNN data inference and geometric shadow cues. It thus avoids data restoration that may violate shadowing conditions. Both our baseline CNN and SCCNN outperform the inverse distance weighting (IDW)-based interpolation method, with the shadow supervision enabling SCCNN to obtain the best performance.

In this article, we provide a detailed survey of 3D Morphable Face Models over the 20 years since they were first proposed. The challenges in building and applying these models, namely, capture, modeling, image formation, and image analysis, are still active research topics, and we review the state-of-the-art in each of these areas. We also look ahead, identifying unsolved challenges, proposing directions for future research, and highlighting the broad range of current and future applications.

Three-dimensional Morphable Models (3DMMs) are powerful statistical tools for representing the 3D surfaces of an object class. In this context, we identify an interesting question that has previously not received research attention: is it possible to combine two or more 3DMMs that (a) are built using different templates that perhaps only partly overlap, (b) have different representation capabilities and (c) are built from different datasets that may not be publicly-available? In answering this question, we make two contributions. First, we propose two methods for solving this problem: i. use a regressor to complete missing parts of one model using the other, ii. use the Gaussian Process framework to blend covariance matrices from multiple models. Second, as an example application of our approach, we build a new head and face model that combines the variability and facial detail of the LSFM with the full head modelling of the LYHM. The resulting combined model achieves state-of-the-art performance and outperforms existing head models by a large margin. Finally, as an application experiment, we reconstruct full head representations from single, unconstrained images by utilizing our proposed large-scale model in conjunction with the Face-Warehouse blendshapes for handling expressions.

We show how to train a fully convolutional neural network to perform inverse rendering from a single, uncontrolled image. The network takes an RGB image as input, regresses albedo and normal maps from which we compute lighting coefficients. Our network is trained using large uncontrolled image collections without ground truth. By incorporating a differentiable renderer, our network can learn from self-supervision. Since the problem is ill-posed we introduce additional supervision: 1. We learn a statistical natural illumination prior, 2. Our key insight is to perform offline multiview stereo (MVS) on images containing rich illumination variation. From the MVS pose and depth maps, we can cross project between overlapping views such that Siamese training can be used to ensure consistent estimation of photometric invariants. MVS depth also provides direct coarse supervision for normal map estimation. We believe this is the first attempt to use MVS supervision for learning inverse rendering.

In this paper, we propose a hybrid depth imaging system in which a polarisation camera is augmented by a second image from a standard digital camera. For this modest increase in equipment complexity over conventional shape-from-polarisation, we obtain a number of benefits that enable us to overcome longstanding problems with the polarisation shape cue. The stereo cue provides a depth map which, although coarse, is metrically accurate. This is used as a guide surface for disambiguation of the polarisation surface normal estimates using a higher order graphical model. In turn, these are used to estimate diffuse albedo. By extending a previous shape-from-polarisation method to the perspective case, we show how to compute dense, detailed maps of absolute depth, while retaining a linear formulation. We show that our hybrid method is able to recover dense 3D geometry that is superior to state-of-the-art shape-from-polarisation or two view stereo alone.

We address semantic segmentation on omnidirectional images, to leverage a holistic understanding of the surrounding scene for applications like autonomous driving systems. For the spherical domain, several methods recently adopt an icosahedron mesh, but systems are typically rotation invariant or require significant memory and parameters, thus enabling execution only at very low resolutions. In our work, we propose an orientation-aware CNN framework for the icosahedron mesh. Our representation allows for fast network operations, as our design simplifies to standard network operations of classical CNNs, but under consideration of north-aligned kernel convolutions for features on the sphere. We implement our representation and demonstrate its memory efficiency up-to a level-8 resolution mesh (equivalent to 640 x 1024 equirectangular images). Finally, since our kernels operate on the tangent of the sphere, standard feature weights, pretrained on perspective data, can be directly transferred with only small need for weight refinement. In our evaluation our orientation-aware CNN becomes a new state of the art for the recent 2D3DS dataset, and our Omni-SYNTHIA version of SYNTHIA. Rotation invariant classification and segmentation tasks are additionally presented for comparison to prior art.

We propose a novel biophysical and dichromatic reflectance model that efficiently characterises spectral skin reflectance. We show how to fit the model to multispectral face images enabling high quality estimation of diffuse and specular shading as well as biophysical parameter maps (melanin and haemoglobin). Our method works from a single image without requiring complex controlled lighting setups yet provides quantitatively accurate reconstructions and qualitatively convincing decomposition and editing.

A face image contains geometric cues in the form of configurational information and contours that can be used to estimate 3D face shape. While it is clear that 3D reconstruction from 2D points is highly ambiguous if no further constraints are enforced, one might expect that the face-space constraint solves this problem. We show that this is not the case and that geometric information is an ambiguous cue. There are two sources for this ambiguity. The first is that, within the space of 3D face shapes, there are flexibility modes that remain when some parts of the face are fixed. The second occurs only under perspective projection and is a result of perspective transformation as camera distance varies. Two different faces, when viewed at different distances, can give rise to the same 2D geometry. To demonstrate these ambiguities, we develop new algorithms for fitting a 3D morphable model to 2D landmarks or contours under either orthographic or perspective projection and show how to compute flexibility modes for both cases. We show that both fitting problems can be posed as a separable nonlinear least squares problem and solved efficiently. We demonstrate both quantitatively and qualitatively that the ambiguity is present in reconstructions from geometric information alone but also in reconstructions from a state-of-the-art CNN-based method.

We present a method for estimating surface height directly from a single polarisation image simply by solving a large, sparse system of linear equations. To do so, we show how to express polarisation constraints as equations that are linear in the unknown height. The local ambiguity in the surface normal azimuth angle is resolved globally when the optimal surface height is reconstructed. Our method is applicable to dielectric objects exhibiting diffuse and specular reflectance, though lighting and albedo must be known. We relax this requirement by showing that either spatially varying albedo or illumination can be estimated from the polarisation image alone using nonlinear methods. In the case of illumination, the estimate can only be made up to a binary ambiguity which we show is a generalised Bas-relief transformation corresponding to the convex/concave ambiguity. We believe that our method is the first passive, monocular shape-from-x technique that enables well-posed height estimation with only a single, uncalibrated illumination condition. We present results on real world data, including in uncontrolled, outdoor illumination.

Morphable models are useful shape priors for biometric recognition tasks. Here we present an iterative process of refinement for a 3D Morphable Model (3DMM) of the human ear that employs data augmentation. The process employs the following stages 1) landmark-based 3DMM fitting; 2) 3D template deformation to overcome noisy over-fitting; 3) 3D mesh editing, to improve the fit to manual 2D landmarks. These processes are wrapped in an iterative procedure that is able to bootstrap a weak, approximate model into a significantly better model. Evaluations using several performance metrics verify the improvement of our model using the proposed algorithm. We use this new 3DMM model-booting algorithm to generate a refined 3D morphable model of the human ear, and we make this new model and our augmented training dataset public.

We propose a shape template morphing approach suitable for any class of shapes that exhibits approximate reflective symmetry over some plane. The human face and full head are examples. A shape morphing algorithm that constrains all morphs to be symmetric is a form of deformation regulation. This mitigates undesirable effects seen in standard morphing algorithms that are not symmetry-aware, such as tangential sliding. Our method builds on the Coherent Point Drift (CPD) algorithm and is called Symmetry-aware CPD (SACPD). Global symmetric deformations are obtained by removal of asymmetric shear from CPD's global affine transformations. Symmetrised local deformations are then used to improve the symmetric template fit. These symmetric deformations are followed by Laplace-Beltrami regularized projection which allows the shape template to fit to any asymmetries in the raw shape data. The pipeline facilitates construction of statistical models that are readily factored into symmetrical and asymmetrical components. Evaluations demonstrate that SA-CPD mitigates tangential sliding problem in CPD and outperforms other competing shape morphing methods, in some cases substantially. 3D morphable models are constructed from over 1200 full head scans, and we evaluate the constructed models in terms of age and gender classification. The best performance, in the context of SVM classification, is achieved using the proposed SA-CPD deformation algorithm.

Abstract Important sources of shape variability, such as articulated motion of body models or soft tissue dynamics, are highly nonlinear and are usually superposed on top of rigid body motion which must be factored out. We propose a novel, nonlinear, rigid body motion invariant Principal Geodesic Analysis (PGA) that allows us to analyse this variability, compress large variations based on statistical shape analysis and fit a model to measurements. For given input shape data sets we show how to compute a low dimensional approximating submanifold on the space of discrete shells, making our approach a hybrid between a physical and statistical model. General discrete shells can be projected onto the submanifold and sparsely represented by a small set of coefficients. We demonstrate two specific applications: model‐constrained mesh editing and reconstruction of a dense animated mesh from sparse motion capture markers using the statistical knowledge as a prior.

In this paper, we develop an interest point detector and binary feature descriptor for spherical images. We take as inspiration a recent framework developed for planar images, BRISK (Binary Robust Invariant Scalable Keypoints), and adapt the method to operate on spherical images. All of our processing is intrinsic to the sphere and avoids the distortion inherent in storing and indexing spherical images in a 2D representation. We discretise images on a spherical geodesic grid formed by recursive subdivision of a triangular mesh. This leads to a multiscale pixel grid comprising mainly hexagonal pixels that lends itself naturally to a spherical image pyramid representation. For interest point detection, we use a variant of the Accelerated Segment Test (AST) corner detector which operates on our geodesic grid. We estimate a continuous scale and location for features and descriptors are built by sampling onto a regular pattern in the tangent space. We evaluate repeatability, precision and recall on both synthetic spherical images with known ground truth correspondences and real images.

We present a fully automatic pipeline to train 3D Morphable Models (3DMMs), with contributions in pose normalisation, dense correspondence using both shape and texture information, and high quality, high resolution texture mapping. We propose a dense correspondence system, combining a hierarchical parts-based template morphing framework in the shape channel and a refining optical flow in the texture channel. The texture map is generated using raw texture images from five views. We employ a pixel-embedding method to maintain the texture map at the same high resolution as the raw texture images, rather than using per-vertex color maps. The high quality texture map is then used for statistical texture modelling. The Headspace dataset used for training includes demographic information about each subject, allowing for the construction of both global 3DMMs and models tailored for specific gender and age groups. We build both global craniofacial 3DMMs and demographic sub-population 3DMMs from more than 1200 distinct identities. To our knowledge, we present the first public 3DMM of the full human head in both shape and texture: the Liverpool-York Head Model. Furthermore, we analyse the 3DMMs in terms of a range of performance metrics. Our evaluations reveal that the training pipeline constructs state-of-the-art models.

In this paper we present a differential approach to photo-polarimetric shape estimation. We propose several alternative differential constraints based on polarisation and photometric shading information and show how to express them in a unified partial differential system. Our method uses the image ratios technique to combine shading and polarisation information in order to directly reconstruct surface height, without first computing surface normal vectors. Moreover, we are able to remove the non-linearities so that the problem reduces to solving a linear differential problem. We also introduce a new method for estimating a polarisation image from multichannel data and, finally, we show it is possible to estimate the illumination directions in a two source setup, extending the method into an uncalibrated scenario. From a numerical point of view, we use a least-squares formulation of the discrete version of the problem. To the best of our knowledge, this is the first work to consider a unified differential approach to solve photo-polarimetric shape estimation directly for height. Numerical results on synthetic and real-world data confirm the effectiveness of our proposed method.

Abstract We present intrinsic methods to address the fundamental problem of segmenting a mesh into a specified number of patches with a uniform size and a controllable overlap. Although never addressed in the literature, such a segmentation is useful for a wide range of processing operations where patches represent local regions and overlaps regularize solutions in neighbour patches. Further, we propose a symmetry‐aware distance measure and symmetric modification to furthest‐point sampling, so that our methods can operate on semantically symmetric meshes. We introduce quantitative measures of patch size uniformity and symmetry, and show that our segmentation outperforms state‐of‐the‐art alternatives in experiments on a well‐known dataset. We also use our segmentation in illustrative applications to texture stitching and synthesis where we improve results over state‐of‐the‐art approaches.

In this paper, we present a complete pipeline for computing structure-from-motion from the sequences of spherical images. We revisit problems from multiview geometry in the context of spherical images. In particular, we propose methods suited to spherical camera geometry for the spherical-n-point problem (estimating camera pose for a spherical image) and calibrated spherical reconstruction (estimating the position of a 3-D point from multiple spherical images). We introduce a new probabilistic interpretation of spherical structure-from-motion which uses the von Mises-Fisher distribution to model noise in spherical feature point positions. This model provides an alternate objective function that we use in bundle adjustment. We evaluate our methods quantitatively and qualitatively on both synthetic and real world data and show that our methods developed for spherical images outperform straightforward adaptations of methods developed for perspective images. As an application of our method, we use the structure-from-motion output to stabilise the viewing direction in fully spherical video.

In this paper we present a method for computing dense correspondence between a set of 3D face meshes using functional maps. The functional maps paradigm brings with it a number of advantages for face correspondence. First, it allows us to combine various notions of correspondence. We do so by proposing a number of face-specific functions, suited to either within-or between-subject correspondence. Second, we propose a groupwise variant of the method allowing us to compute cycle-consistent functional maps between all faces in a training set. Since functional maps are of much lower dimension than point-to-point correspondences, this is feasible even when the input meshes are very high resolution. Finally, we show how a functional map provides a geometric constraint that can be used to filter feature matches between non-rigidly deforming surfaces.

In this paper, we present a reflectance-aware optical flow technique for nonrigidly aligning images captured in different lighting environments. We consider alignment between two specific lighting conditions of particular relevance to object capture using a light stage. The technique uses optical flow combined with three image transformation functions, namely a) the Illumination-Independent image transformation, b) the image colour transformation and c) the specular-invariant projection. We explore two types of photometric image alignment problem, i) aligning images captured with different spherical gradient patterns but the same types of light source and ii) aligning the spherical gradient sequence to an image captured with a different light source. Our results show that a previously proposed method can accurately solve the first problem and we propose a model-based approach to solving the second.

We present an approach to modeling ear-to-ear, high-quality texture from one or more partial views of a face with possibly poor resolution and noise. Our approach is example-based in that we reconstruct texture with patches from a database composed of previously seen faces. A 3D morphable model is used to establish shape correspondence between the observed data across views and training faces. The database is built on the mesh surface by segmenting it into uniform overlapping patches. Texture patches are selected by belief propagation so as to be consistent with neighbors and with observations in an appropriate image formation model. We also develop a variant that is insensitive to light and camera parameters, and incorporate soft symmetry constraints. We obtain textures of higher quality for degraded views as small as 10 pixels wide, than a standard model fitted to non-degraded data. We further show applications to super-resolution where we substantially improve quality compared to a state-of-the-art algorithm, and to texture completion where we fill in missing regions and remove facial clutter in a photorealistic manner.

In this paper we describe how to perform Principal Components Analysis in "shell space". Thin shells are a physical model for surfaces with non-zero thickness whose deformation dissipates elastic energy. Thin shells, or their discrete counterparts, can be considered to reside in a shell space in which the notion of distance is given by the elastic energy required to deform one shape into another. It is in this setting that we show how to perform statistical analysis of a set of shapes (meshes in dense correspondence), providing a hybrid between physical and statistical shape modelling. The resulting models are better able to capture non-linear deformations, for example resulting from articulated motion, even when training data is very sparse compared to the dimensionality of the observation space.

In this paper, we propose a novel approach to seamless texture stitching on a 3D mesh. The main idea is to blend the sampled images by least-angle selection of gradients in overlapping patches. This is in contrast to previous works which focus on vertex- or face-based strategies with additional heuristics for robustness. Patches are obtained by growing a uniform mesh segmentation via geodesic projections. Blending is achieved by formulating a screened Poisson equation using discrete differential operators. The processing pipeline further includes an optional step of color transformation for calibration and correction. This is applied to a zippered mesh based on range scans and a morphable model fitted to face photographs.

In this paper, we present a complete framework to inverse render faces with a 3D Morphable Model (3DMM). By decomposing the image formation process into geometric and photometric parts, we are able to state the problem as a multilinear system which can be solved accurately and efficiently. As we treat each contribution as independent, the objective function is convex in the parameters and a global solution is guaranteed. We start by recovering 3D shape using a novel algorithm which incorporates generalization error of the model obtained from empirical measurements. We then describe two methods to recover facial texture, diffuse lighting, specular reflectance, and camera properties from a single image. The methods make increasingly weak assumptions and can be solved in a linear fashion. We evaluate our findings on a publicly available database, where we are able to outperform an existing state-of-the-art algorithm. We demonstrate the usability of the recovered parameters in a recognition experiment conducted on the CMU-PIE database.

This paper presents a new database suitable for both 2-D and 3-D face recognition based on photometric stereo (PS): the Photoface database. The database was collected using a custom-made four-source PS device designed to enable data capture with minimal interaction necessary from the subjects. The device, which automatically detects the presence of a subject using ultrasound, was placed at the entrance to a busy workplace and captured 1839 sessions of face images with natural pose and expression. This meant that the acquired data is more realistic for everyday use than existing databases and is, therefore, an invaluable test bed for state-of-the-art recognition algorithms. The paper also presents experiments of various face recognition and verification algorithms using the albedo, surface normals, and recovered depth maps. Finally, we have conducted experiments in order to demonstrate how different methods in the pipeline of PS (i.e., normal field computation and depth map reconstruction) affect recognition and verification performance. These experiments help to 1) demonstrate the usefulness of PS, and our device in particular, for minimal-interaction face recognition, and 2) highlight the optimal reconstruction and recognition algorithms for use with natural-expression PS data. The database can be downloaded from http://www.uwe.ac.uk/research/Photoface.

In this paper, we present a framework for the groupwise processing of a set of meshes in dense correspondence. Such sets arise when modeling 3D shape variation or tracking surface motion over time. We extend a number of mesh processing tools to operate in a groupwise manner. Specifically, we present a geodesic-based surface flattening and spectral clustering algorithm which estimates a single class-optimal flattening. We also show how to modify an iterative edge collapse algorithm to perform groupwise simplification while retaining the correspondence of the data. Finally, we show how to compute class-optimal texture coordinates for the simplified meshes. We present alternative algorithms for topologically symmetric data which yield a symmetric flattening and low-resolution mesh topology. We present flattening, simplification, and texture mapping results on three different data sets and show that our approach allows the construction of low-resolution 3D morphable models.

In this paper, we present a complete framework to inverse render faces from single images using a 3D Morphable Model (3DMM). A 3DMM is a linear statistical model of 3D shape and texture [2]. In general, inverse rendering of faces from single photographs is ill-posed, as the same appearance can be obtained by different underlaying factors. For instance, a red pixel can be caused by skin colour, red illumination, or an increased camera sensitivity in the red channel. A combination of these factors is also possible. For an object of known shape under complex natural illumination, the well known work of Ramamoorthi [4] shows how the spherical harmonic domain can be used to estimate one or more of: illumination, surface texture and reflectance properties. We revisit this classical formulation in the context of 3DMMs. Previous methods for fitting a 3DMM based on analysis-by-synthesis recover all parameters in a single, nonconvex objective function [2, 3]. To reduce the threat of getting stuck in local minima, Romdhani introduced a fitting algorithm which incorporates features like edges and specular highlights into the cost function [5]. These fitting algorithms make limited assumptions about the illumination environment and only model ambient light and one directional light source. Zhang and Samaras [6] used spherical harmonics to model unconstrained illumination, although at the cost of assuming a simple Lambertian reflectance model.

In this paper we show how to simplify a 3D morphable model. Our method only requires knowledge of the original highest resolution statistical model and leads to low resolution models in which the model statistics are a subset of the original high resolution model. We employ an iterative edge collapse strategy, where the deleted edge is chosen as a function of the model statistics. We show that the expected value of the Quadric Error Metric can be computed in closed form for a PCA deformable model. Model parameters obtained using the model at any resolution (lower) can be used to reconstruct a high resolution surface, providing a route to super-resolution. We provide experimental results for a statistical face model, showing how the simplified models improve the efficiency of model fitting. We are able to decrease the model resolution and fitting time by factors of approximately 10 and 4 respectively whilst inducing an error which is only slightly larger than the fitting error of the original model.

We present a shape-from-shading algorithm for Lambertian surfaces of uniform but unknown albedo, illuminated by unknown, arbitrarily complex environment lighting. Our approach is based on a first order spherical harmonic approximation to the reflectance map. This is estimated from the image using surface normals interpolated from boundary points. The shape-from-shading step minimises local brightness error and an edge sensitive smoothness constraint. This involves the solution of a linear least squares problem with a quadratic equality constraint, the global optimum of which can be found using the method of Lagrange multipliers. We demonstrate the performance of the algorithm on complex objects rendered under realworld illumination.

In this paper, we present a robust and efficient method to statistically recover the full 3D shape and texture of faces from single 2D images. We separate shape and texture recovery into two linear problems. For shape recovery, we learn empirically the generalization error of a 3D morphable model using out-of-sample data. We use this to predict the 2D variance associated with a sparse set of 2D feature points. This knowledge is incorporated into a parameter-free probabilistic framework which allows 3D shape recovery of a face in an arbitrary pose in a single step. Under the assumption of diffuseonly reflectance, we also show how photometric invariants can be used to recover texture parameters in an illumination insensitive manner. We present empirical results with comparison to the state-of-the-art analysis-by-synthesis methods and show an application of our approach to adjusting the pose of subjects in oil paintings.

In this paper we derive formal constraints relating terrain elevation and observed cast shadows. We show how an optimisation framework can be used to refine surface estimates using shadowing constraints from one or more images. The method is particularly applicable to the digital elevation models produced by the Shuttle Radar Topography Mission (SRTM), which have an abundance of voids in mountainous areas where elevation data is missing. Cast shadow maps are detected automatically from multi-spectral satellite imagery using a simple heuristic which is reliable over varying types of surface cover. We show that the combination of our shadow segmentation and terrain correction methods can restore the structure of mountain ridges in interpolated SRTM voids using five satellite images, decreasing the RMS error by over 25%.

In this paper, we present a new method to statistically recover the full 3D shape of a face from a set of sparse feature points. We attribute noise in the feature point positions to generalisation error of the model. We learn the variance of these feature points empirically using out-of-sample data. This allows the shape reconstruction to probabilistically model the way in which feature points deviate from their true position. We are able to reduce the reconstruction error by as much as 12%.

State-of-the-art photometric shape and reflectance analysis uses polarized spherical gradient illumination images captured in a light stage. The quality of the estimated surface geometry degrades with interreflection, rotationally asymmetric reflectance lobes and light discretisation. We show that introducing additional parameters to model distortions in the diffuse reflectance lobe results in an underdetermined linear system. Results from existing approaches can be refined to satisfy this system using quadratic programming. We also describe how robust shape recovery can be achieved using a minimal four image set.

A key problem in shape-from-shading is how to simultaneously satisfy data-closeness and regularisation (such as surface smoothness) constraints. This paper makes two contributions towards solving this problem. The first is to desc ribe a smoothness constraint which preserves surface structure by adaptively smoothing according to the intensity gradient magnitude. The second is to derive a framework which seeks to strictly satisfy this constraint while maintaining zero brightness error. Experimental results on both synthetic and real world imagery demonstrate that our method is both robust and accurate and outperforms a number of existing techniques.

In this paper we present a method for face shape and albedo estimation which uses a morphable model in conjunction with non-Lambertian shape-from-shading. We use surface normal and albedo estimates to construct a spherical harmonic basis which can be used generatively to model face appearance variation under arbitrarily complex illumination. This allows us to perform illumination insensitive face recognition given only a single gallery image. In contrast to other similar methods, our surface normal and albedo estimates are not constrained by a statistical model and are instead inferred from shading cues. We present recognition results on the Yale Face Database B.

In this paper we revisit the process of constructing a high resolution 3D morphable model of face shape variation. We demonstrate how the statistical tools of thin-plate splines and Procrustes analysis can be used to construct a morphable model that is both more efficient and generalises to novel face surfaces more accurately than previous models. We also reformulate the probabilistic prior that the model provides on the distribution of parameter vector lengths. This distribution is determined solely by the number of model dimensions and can be used as a regularisation constraint in fitting the model to data without the need to empirically choose a parameter controlling the trade off between plausibility and quality of fit. As an example application of this improved model, we show how it may be fitted to a sparse set of 2D feature points (approximately 100). This provides a rapid means to estimate high resolution 3D face shape for a face in any pose given only a single face image. We present experimental results using ground truth data and hence provide absolute reconstruction errors. On average, the per vertex error of the reconstructed faces is less than 3.6 mm.

In this paper we consider diffuse and specular reflectance from surfaces modeled as distributions of glossy microfacets. In contrast to previous work, we describe the relative contribution of both of these components in the same terms, namely with resource to Fresnel theory. This results in a more highly constrained model with a reduced number of parameters. Also, the need for ad hoc and physically meaningless specular and diffuse reflectance coefficients is removed. This ensures that the conservation of energy is obeyed and only physically plausible mixtures of the two components are allowed. In our model, both specular and diffuse reflectance are related to the roughness and refractive index of the surface. We show how physically meaningful parameters of a surface can be measured from uncalibrated imagery and that our model fits observed BRDF data more accurately than comparable existing models.

In this paper we consider diffuse reflectance from surfaces modelled as distributions of specular microfacets. We show how the relative contributions of specular and diffuse reflectance can be found analytically for such microfacet models using Fresnel theory. This removes the need for ad hoc specular and diffuse reflectance coefficients, ensuring the conservation of energy and only allowing physically plausible mixtures of the two components. We show how the amount of light that can be diffusely reflected is dependent on the angle of incidence and physical properties of the surface.

In this paper, we show how to extract gender discriminating features from 2.5D facial needle-maps. The standard eigenspace analysis method for non-Euclidean data is principal geodesic analysis (PGA). Based on PGA, we propose a novel supervised weighted PGA method which incorporates local weights into standard PGA to improve gender discriminating capability of the extracted features. The weight map is iteratively optimized from the labeled data, which is different from other gender relevant weights used in the literature. Experimental results illustrate the effectiveness of this method and its successful application to gender classification.

In this paper we show how to estimate facial surface reflectance properties (a slice of the BRDF and the albedo) in conjunction with facial shape from a single image. We show how an estimate of the surface reflectance function can be made by fitting a curve to the scattered and noisy reflectance samples provided by the estimated shape. We present a novel statistical face shape constraint which we term dasiamodel-based integrabilitypsila which we enforce on the field of surface normals. We iteratively interleave the two processes of estimating reflectance properties based on the current shape estimate and updating the shape estimate based on the current estimate of the reflectance function. We show that the method is capable of recovering accurate shape and reflectance information from single images using both synthetic and real world imagery.

This paper addresses the problem of determining skin reflectance parameters, and studies their stability and discriminating power for different individuals. Our study uses radiance data captured by a Cyberware 3030 range scanner. We analyse the data using a layered reflectance model based on the Beckmann-Kirchhoff wave scattering model. The parameters of this model are the thickness of the skin layers, and their roughness. We investigate how the parameters of this model vary between different subjects, and how they vary for the same subject under different conditions.

In this paper, we perform gender classification based on 2.5D facial surface normals (facial needle-maps), and present two novel principal geodesic analysis (PGA) methods, weighted PGA and supervised PGA, to parameterize the facial needle-maps, and compare their performances with PGA for gender classification. Experimental results demonstrate the feasibility of gender classification based on facial needle-maps, and show that incorporating weights or pairwise relationships of labeled data into PGA improves the gender discriminating powers in the leading eigenvectors and the gender classification accuracy.

The aim in this paper is to show how to use the 2.5D facial surface normals \n(needle-maps) recovered using shape from shading (SFS) to improve \nthe performance of gender classification. We incorporate principal geodesic \nanalysis (PGA) into SFS to guarantee the recovered needle-maps is a possible \nexample defined by a statistical model. Because the recovered facial needlemaps \nsatisfy data-closeness constraint, they not only give the facial shape \ninformation, but also combine the image intensity implicitly. Experiments \nshow that this combination gives better gender classification performance \nthan using facial shape or texture information alone.

We focus on the problem of developing a coupled statistical model that can be used to recover facial shape from brightness images of faces. We study three alternative representations for facial shape. These are the surface height function, the surface gradient, and a Fourier basis representation. We jointly capture variations in intensity and the surface shape representations using a coupled statistical model. The model is constructed by performing principal components analysis on sets of parameters describing the contents of the intensity images and the facial shape representations. By fitting the coupled model to intensity data, facial shape is implicitly recovered from the shape parameters. Experiments show that the coupled model is able to generate accurate shape from out-of-training-sample intensity images.

This paper describes a method for cast shadow removal from obliquely illuminated images of faces. The method draws on a statistical model of surface normal directions. The model is fitted to shadowed facial images using robust statistics and constraints provided by shape-from-shading. Regions associated with poor fit residuals are associated with shadow regions. We illustrate the method on the Yale B database where it gives both good shadow map estimates and fills-in the facial surface in the shadow regions. 1

In this paper we investigate whether the 2.5D shape information delivered by a novel shape-from-shading algorithm can be used for illumination insensitive face recognition. We present a robust and efficient facial shape-fromshading algorithm which uses principal geodesic analysis to model the variation in surface orientation across a face. We show how this algorithm can be used to recover accurate facial shape and albedo from real world images. Our second contribution is to use the recovered 2.5D shape information in a variety of recognition methods. We present a novel recognition strategy in which similarity is measured in the space of the principal geodesic parameters. We also use the recovered shape information to generate illumination normalised prototype images on which recognition can be performed. Finally we show that, from a single input image, we are able to generate the basis images employed by a number of well known illumination-insensitive recognition algorithms. We also demonstrate that the principal geodesics provide an efficient parameterisation of the space of harmonic basis images.

This paper describes how a facial albedo map can be recovered from a single image using a statistical model that captures variations in surface normal direction. We fit the model to intensity data using constraints on the surface normal direction provided by Lambert's law and then use the differences between observed and reconstructed image brightness to estimate the albedo. We show that this process is stable under varying illumination. We then show how eigenfaces trained on albedo maps may provide a better representation for illumination insensitive recognition than those trained on raw image intensity.

In this paper, we show how a statistical model of facial shape can be embedded within a shape-from-shading algorithm. We describe how facial shape can be captured using a statistical model of variations in surface normal direction. To construct this model, we make use of the azimuthal equidistant projection to map the distribution of surface normals from the polar representation on a unit sphere to Cartesian points on a local tangent plane. The distribution of surface normal directions is captured using the covariance matrix for the projected point positions. The eigenvectors of the covariance matrix define the modes of shape-variation in the fields of transformed surface normals. We show how this model can be trained using surface normal data acquired from range images and how to fit the model to intensity images of faces using constraints on the surface normal direction provided by Lambert's law. We demonstrate that the combination of a global statistical constraint and local irradiance constraint yields an efficient and accurate approach to facial shape recovery and is capable of recovering fine local surface details. We assess the accuracy of the technique on a variety of images with ground truth and real-world images.

This paper describes how facial shape can be modelled using a statistical model that captures variations in surface normal direction. To construct this model, we make use of the azimuthal equidistant projection to map surface normals from the unit sphere to points on a local tangent plane. The variations in surface normal direction are captured using the covariance matrix for the projected point positions. This allows us to model variations in face shape using a standard point distribution model. We train the model on fields of surface normals extracted from range data and show how to fit the model to intensity data using constraints on the surface normal direction provided by Lambert's law. We demonstrate that this process yields accurate facial shape recovery and allows an estimate of the albedo map to be made from single, real world face images.

This paper describes how facial shape can be modelled using a statistical model that captures variations in surface normal direction. We fit the model to intensity data using constraints on the surface normal direction provided by Lambert's law. We demonstrate that this process yields improved facial shape recovery and allows realistic synthesis of views under novel illumination and pose.

This paper describes how a facial albedo map can be recovered from a single image using a statistical model that captures variations in surface normal direction. We fit the model to intensity data using constraints on the surface normal direction provided by Lambert's law and then use the differences between observed and reconstructed image brightness to estimate the albedo map. We show that this process is stable under varying illumination and use the process to render images under novel illumination.

Single image facial view synthesis using SFS

We present a parameter-free method for estimating the BRDF of a subject's skin from a single image. We show how the technique can be used to remove specularities caused by perspiration or oil on the skin's surface and demonstrate that this yields improved analysis using shape from shading.