Correspondence-based statistical shape modeling (SSM) stands as a powerful technology for morphometric analysis in clinical research. SSM facilitates population-level characterization and quantification of anatomical shapes such as bones and organs, aiding in pathology and disease diagnostics and treatment planning. Despite its potential, SSM remains under-utilized in medical research due to the significant overhead associated with automatic construction methods, which demand complete, aligned shape surface representations. Additionally, optimization-based techniques rely on bias-inducing assumptions or templates and have prolonged inference times as the entire cohort is simultaneously optimized. To overcome these challenges, we introduce Point2SSM++, a principled, self-supervised deep learning approach that directly learns correspondence points from point cloud representations of anatomical shapes. Point2SSM++ is robust to misaligned and inconsistent input, providing SSM that accurately samples individual shape surfaces while effectively capturing population-level statistics. Furthermore, we present extensions of Point2SSM++ tailored for dynamic spatiotemporal and multi-anatomy scenarios, showcasing the broad versatility of the framework. Through extensive validation across diverse anatomies, evaluation metrics, and clinically relevant downstream tasks, we demonstrate Point2SSM++’s superiority over existing state-of-the-art deep learning models and the traditional approach. Point2SSM++ substantially enhances the feasibility of SSM generation and significantly broadens its array of potential clinical applications. • Self-supervised approach for predicting anatomical correspondences from point clouds. • Outperforms statistical shape modeling methods on bones and organ populations. • Validation on downstream clinical tasks: craniosynostosis and femur pathology. • Extends to multi-anatomy learning such as full spine vertebrae. • Extends to 4D spatiotemporal data for longitudinal studies and dynamic shape analysis.

Transformers have emerged as the state-of-the-art architecture in medical image registration, outperforming convolutional neural networks (CNNs) by addressing their limited receptive fields and overcoming gradient instability in deeper models. Despite their success, transformer-based models require substantial resources for training, including data, memory, and computational power, which may restrict their applicability for end users with limited resources. In particular, existing transformer-based 3D image registration architectures face two critical gaps that challenge their efficiency and effectiveness. Firstly, although window-based attention mechanisms reduce the quadratic complexity of full attention by focusing on local regions, they often struggle to effectively integrate both local and global information. Secondly, the granularity of tokenization, a crucial factor in registration accuracy, presents a performance tradeoff: smaller voxel-size tokens enhance detail capture but come with increased computational complexity, higher memory usage, and a greater risk of over-fitting. We present EFFICIENTMORPH, a transformer-based architecture for unsupervised 3D image registration that balances local and global attention in 3D volumes through a plane-based attention mechanism and employs a Hi-Res tokenization strategy with merging operations, thus capturing finer details without compromising computational efficiency. Notably, EFFICIENTMORPH sets a new benchmark for performance on the OASIS dataset with ~16-27x fewer parameters. https://github.com/MedVIC-Lab/Efficient.Morph.Registration

We present Point2SSM, a novel unsupervised learning approach for constructing correspondence-based statistical shape models (SSMs) directly from raw point clouds. SSM is crucial in clinical research, enabling population-level analysis of morphological variation in bones and organs. Traditional methods of SSM construction have limitations, including the requirement of noise-free surface meshes or binary volumes, reliance on assumptions or templates, and prolonged inference times due to simultaneous optimization of the entire cohort. Point2SSM overcomes these barriers by providing a data-driven solution that infers SSMs directly from raw point clouds, reducing inference burdens and increasing applicability as point clouds are more easily acquired. While deep learning on 3D point clouds has seen success in unsupervised representation learning and shape correspondence, its application to anatomical SSM construction is largely unexplored. We conduct a benchmark of state-of-the-art point cloud deep networks on the SSM task, revealing their limited robustness to clinical challenges such as noisy, sparse, or incomplete input and limited training data. Point2SSM addresses these issues through an attention-based module, providing effective correspondence mappings from learned point features. Our results demonstrate that the proposed method significantly outperforms existing networks in terms of accurate surface sampling and correspondence, better capturing population-level statistics. The source code is provided at https://github.com/jadie1/Point2SSM.

Learning compact representations for shapes (binary images) is important for many applications. Although neural network models are very powerful, they usually involve many parameters, require substantial tuning efforts and easily overfit small datasets, which are common in shape-related applications. The state-of-the-art approach, ShapeOdds, as a latent Gaussian model, can effectively prevent overfitting and is more robust. Nonetheless, it relies on a linear projection assumption and is incapable of capturing intrinsic nonlinear shape variations, hence may leading to inferior representations and structure discovery. To address these issues, we propose Infinite ShapeOdds (InfShapeOdds), a Bayesian nonparametric shape model, which is flexible enough to capture complex shape variations and discover hidden cluster structures, while still avoiding overfitting. Specifically, we use matrix Gaussian priors, nonlinear feature mappings and the kernel trick to generalize ShapeOdds to a shape-variate Gaussian process model, which can grasp various nonlinear correlations among the pixels within and across (different) shapes. To further discover the hidden structures in data, we place a Dirichlet process mixture (DPM) prior over the representations to jointly infer the cluster number and memberships. Finally, we exploit the Kronecker-product structure in our model to develop an efficient, truncated variational expectation-maximization algorithm for model estimation. On synthetic and real-world data, we show the advantage of our method in both representation learning and latent structure discovery.

Unsupervised representation learning via generative modeling is a staple to many computer vision applications in the absence of labeled data. Variational Autoencoders (VAEs) are powerful generative models that learn representations useful for data generation. However, due to inherent challenges in the training objective, VAEs fail to learn useful representations amenable for downstream tasks. Regularization-based methods that attempt to improve the representation learning aspect of VAEs come at a price: poor sample generation. In this paper, we explore this representation-generation trade-off for regularized VAEs and introduce a new family of priors, namely decoupled priors, or dpVAEs, that decouple the representation space from the generation space. This decoupling enables the use of VAE regularizers on the representation space without impacting the distribution used for sample generation, and thereby reaping the representation learning benefits of the regularizations without sacrificing the sample generation. dpVAE leverages invertible networks to learn a bijective mapping from an arbitrarily complex representation distribution to a simple, tractable, generative distribution. Decoupled priors can be adapted to the state-of-the-art VAE regularizers without additional hyperparameter tuning. We showcase the use of dpVAEs with different regularizers. Experiments on MNIST, SVHN, and CelebA demonstrate, quantitatively and qualitatively, that dpVAE fixes sample generation for regularized VAEs.

Multi-label probabilistic maps, a.k.a. probabilistic segmentations, parameterize a population of intimately co-existing anatomical shapes and are useful for various medical imaging applications, such as segmentation, anatomical atlases, shape analysis, and consensus generation. Existing methods to estimate probabilistic segmentations rely on ad hoc intermediate representations (e.g., average of Gaussian-smoothed label maps and smoothed signed distance maps) that do not necessarily conform to the underlying generative process. Generative modeling of such maps could help discover as well as aide in the statistical analysis of sub-groups in a population via clustering and mixture modeling techniques. In this paper, we propose an estimation of multi-label probabilistic maps and showcase their favorable performance for modeling anatomical shapes such as the left atrium of the human heart and brain structures. The proposed formulation relies on a constrained optimization in the natural parameter space of the exponential family form of categorical distributions. A smoothness prior provides generalizability in the model and helps achieve greater performance in modeling tasks for unseen samples. We demonstrate and compare the effectiveness of the proposed method for Bayesian image segmentation, multi-atlas segmentation, and shape-based clustering.

We present a novel approach for improving the shape statistics of medical image objects by generating correspondence of skeletal points. Each object's interior is modeled by an s-rep, i.e., by a sampled, folded, two-sided skeletal sheet with spoke vectors proceeding from the skeletal sheet to the boundary. The skeleton is divided into three parts: the up side, the down side, and the fold curve. The spokes on each part are treated separately and, using spoke interpolation, are shifted along that skeleton in each training sample so as to tighten the probability distribution on those spokes' geometric properties while sampling the object interior regularly. As with the surface/boundary-based correspondence method of Cates et al., entropy is used to measure both the probability distribution tightness and the sampling regularity, here of the spokes' geometric properties. Evaluation on synthetic and real world lateral ventricle and hippocampus data sets demonstrate improvement in the performance of statistics using the resulting probability distributions. This improvement is greater than that achieved by an entropy-based correspondence method on the boundary points.

Shape models provide a compact parameterization of a class of shapes, and have been shown to be important to a variety of vision problems, including object detection, tracking, and image segmentation. Learning generative shape models from grid-structured representations, aka silhouettes, is usually hindered by (1) data likelihoods with intractable marginals and posteriors, (2) high-dimensional shape spaces with limited training samples (and the associated risk of overfitting), and (3) estimation of hyperparameters relating to model complexity that often entails computationally expensive grid searches. In this paper, we propose a Bayesian treatment that relies on direct probabilistic formulation for learning generative shape models in the silhouettes space. We propose a variational approach for learning a latent variable model in which we make use of, and extend, recent works on variational bounds of logistic-Gaussian integrals to circumvent intractable marginals and posteriors. Spatial coherency and sparsity priors are also incorporated to lend stability to the optimization problem by regularizing the solution space while avoiding overfitting in this high-dimensional, low-sample-size scenario. We deploy a type-II maximum likelihood estimate of the model hyperparameters to avoid grid searches. We demonstrate that the proposed model generates realistic samples, generalizes to unseen examples, and is able to handle missing regions and/or background clutter, while comparing favorably with recent, neural-network-based approaches.

While traditional dental fillings are molded during a dental visit, dental restoration (e.g. inlays and onlays) are fabricated in a dental lab to offer a long lasting reparative solution to tooth decay or similar structural damage. Such process requires dental technicians who are highly trained experts in tooth anatomy to pick an appropriate standard tooth model from a tooth database. The success of a restoration process primarily relies on the acquisition and modeling of an accurate 3D shape of the occlusal surface of interest for manufacturing purposes. Based on a single optical image, this paper provides an economical and automated solution for tooth restoration where user intervention is kept at the minimal. The inherit relation between the photometric information and the underlying 3D shape is formulated as a coupled statistical model where the effect of illumination is modeled using Spherical Harmonics. Moreover, shape and texture alignment is accomplished using a proposed definition of anatomical jaw landmarks which are automatically detected. The system is evaluated on database of 32 jaws for crown, inlay, and onlay restoration. Results shows a promising performance for using the proposed approach in clinical application.

Image formation involves understanding the sensors characteristics and object reflectance. In dentistry, for example an accurate three‐dimensional (3D) representation of the human jaw may be used for diagnostic and treatment purposes. Photogrammetry can offer a flexible, cost‐effective solution in that regard. Nonetheless there are several challenges, such as non‐friendly image acquisition environment inside the human mouth, problems with lighting (specularity effects because of saliva, gum discolourisation, and occlusion because of the tongue in the lower jaw), and errors because of the data acquisition sensors (e.g. camera calibration errors, lens distortion and so on). In this study, the authors focus on the 3D surface reconstruction aspect for human jaw modelling based on physical surface characteristics and sensor properties. Owing to apparent lens distortion imposed by near‐field imaging, the authors propose a new flexible calibration for lens radial distortion based on a single image of a sphere. The authors propose a non‐Lambertian shape‐from‐shading (SFS) algorithm under perspective projection which benefits from camera calibration parameters. Our experiments provide quantitative metric results for the proposed approach. The reflectance of the tooth surface is modelled by the Oren–Nayar reflectance model for rough surfaces whose roughness parameter is physically computed from an optical surface profiler measurements. As compared to state‐of‐the‐art SFS approaches, our approach is able to recover geometric details of tooth occlusal surface. This work is fundamental for establishing an optical‐based approach for reconstructing the human jaw, that is inexpensive and does not use ionising radiation.

Phenomenological appearance models capture surface appearance through mathematical modelling of the reflection process. Theoretically, the space of all possible images of a fixed‐pose object under all possible illumination conditions is infinite dimensional. Nonetheless, because of their low‐frequency nature, irradiance signals can be represented using low‐order basis functions. Discounting subsurface scattering and surface emittance, this work seeks to address the question; how to compactly and accurately represent image irradiance under unknown general illumination, given that a surface point sees its surrounding world through the local upper hemisphere oriented by the surface normal at this point. In this study, we formulate the image formation process of isotropic surface reflectance under arbitrary distant illumination in the frequency space while addressing the physical compliance of hemispherical basis for representing surface reflectance, for example, Helmholtz reciprocity and isotropy. The term ‘irradiance harmonics’ is also defined which enables decoupling illumination and reflectance from the underlying geometry and pose. We provide a closed form of the energy content being maintained by different reflectance modes of the proposed irradiance harmonics. Since specular materials tend to require more basis functions when compared with diffuse ones, the presented harmonics captures same cumulative energy content, by providing larger number of orthogonal irradiance basis, at lower illumination orders when compared to similar basis in literature.

Precise knowledge of the 3D shape of clinical crowns is crucial for the treatment of malocclusion problems as well as several endodontic procedures. While computed tomography would present such information, it is believed there is no threshold radiation dose below which it is considered safe. In this study, the authors propose an appearance based approach which allows for the reconstruction of plausible human jaw 3D models given a single optical image with unknown illumination. Appearance bases are analytically constructed using the frequency‐based representation of the irradiance equation while incorporating prior information about natural illumination and teeth reflectance. The inherent relation between the photometric information and the underlying 3D shape is formulated as a statistical model where the coupled effect of illumination and reflectance is modelled using the Helmholtz hemispherical harmonics‐based irradiance harmonics whereas the principle component regression is deployed to carry out the estimation of 3D shapes. The authors' approach relaxes limiting assumptions of conventional shape‐from‐shading approaches while being able to reconstruct tooth occlusal surface with challenging conditions, such as scattered specular spots and significant changes in colour and albedo characteristics resulting from tooth filling. Vis‐à‐vis dental applications, the results demonstrate a significant increase in accuracy in favour of the proposed approach.

Conventional subspace construction approaches suffer from the need of "large-enough" image ensemble rendering numerical methods intractable. In this paper, we propose an analytic formulation for low-dimensional subspace construction in which shading cues lie while preserving the natural structure of an image sample. Using the frequency-space representation of the image irradiance equation, the process of finding such subspace is cast as establishing a relation between its principal components and that of a deterministic set of basis functions, termed as irradiance harmonics. Representing images as matrices further lessen the number of parameters to be estimated to define a bilinear projection which maps the image sample to a lower-dimensional bilinear subspace. Results show significant impact on dimensionality reduction with minimal loss of information as well as robustness against noise.

Accurate 3D modeling of the human teeth/jaw helps patients avoid the discomfort of the mold process, and improves the data accuracy for oral orthodontists and dental care personnel. Since the surface of the human tooth is almost textureless, Shape from Shading (SFS) has been successfully adopted in solving this problem. In this paper, we attempt to improve the limitations of previous 3D tooth reconstruction algorithms by developing a new approach for shape reconstruction from single image shading with Two-Dimensional Principle Component Analysis (2D-PCA) shape priors. The surface reflectance is modeled using Oren-Nayar-Wolff model which accounts for the retro-reflection case and we experimentally prove that the teeth surface follows the microfacet theory. Our formulation exploits the shape priors as extracted from a set of training CT scans of real human jaws. Our experiments provide promising quantitative metric results for the proposed approach. This work is fundamental for establishing an optical-based approach for reconstructing the human jaw that is inexpensive and does not use ionizing radiation.

Under local illumination assumption, phenomenological appearance models capture surface appearance through the mathematical modelling of the reflection process. Theoretically, due to the arbitrariness of the lighting function, the space of all possible images of a fixed‐pose object under all possible illumination conditions is infinite dimensional. Nonetheless, due to their low‐ frequency nature, irradiance signals can be represented using low‐order basis functions, where spherical harmonics (SH) has been extensively adopted. When capturing image irradiance from a single viewpoint, the visible part of the object's surface constructs the upper hemisphere of the surface normals where the SH is no longer orthonormal. In this paper, we propose the use of hemispherical harmonics (HSH) to model image irradiance of convex Lambertian objects perceived from single viewpoint under unknown distant as well as near illumination. We prove analytically, and validate experimentally, that the Lambertian reflectance kernel has a more compact harmonic expansion in the hemispherical domain when compared to its spherical counterpart. Our experiments illustrate that, despite of having poor approximation accuracy under very close lights, such behavior improves exponentially with little increase in the distance to the light source relative to the object size.

Image formation involves understanding sensor characteristics and object reflectance. In dentistry, an accurate 3-D representation of the human jaw may be used for diagnostic and treatment purposes. Photogrammetry can offer a flexible, cost effective solution for accurate 3-D representation of the human teeth, which can be used for diagnostic and treatment purposes. Nonetheless there are several challenges, such as the non-friendly image acquisition environment inside the human mouth, problems with lighting and errors due to the data acquisition sensors. In this paper, we focus on the 3D surface reconstruction aspect for human teeth based on a single image. We introduce a more realistic formulation of the shape-from-shading (SFS) problem by considering the image formation components; the camera, the light source, and the surface reflectance. We propose a non-Lambertian SFS algorithm under perspective projection which benefits from camera calibration parameters. We take into account the attenuation of illumination due to near-field imaging. The surface reflectance is modeled using Oren-Nayar-Wolff model which accounts for the retro-reflection case. Our experiments provide promising quantitative metric results for the proposed approach.

Discounting subsurface scattering and surface emittance, this work seeks to address the question; how to accurately represent image irradiance under unknown general illumination and reflectance, given that a surface point sees its surrounding world through the local upper hemisphere oriented by the surface normal at this point. In this paper, we formulate the image formation process of isotropic surfaces under arbitrary distant illumination in the frequency space while addressing the physical compliance of hemispherical basis for representing surface reflectance, e.g. Helmholtz reciprocity and isotropy. This representation is further reduced in dimensionality through analytically deriving the principal components of the image irradiance. A database of natural illumination and real world surface materials are used to compute a general-purpose basis which captures the full behavior of illumination and reflectance in a lower-dimensional subspace for image irradiance representation. Compared to similar basis in literature, our proposed basis achieves higher accuracy levels at lower illumination orders.

The visual appearance of real-world surfaces is the net result of surface reflectance characteristics when exposed to illumination. Appearance models can be constructed using phenomenological models which capture surface appearance through mathematical modeling of the reflection process. This yields an integral equation, known as reflectance equation, describing the surface radiance, which depends on the interaction between the incident light field and the surface bidirectional reflectance distribution function (BRDF). The BRDF is a function defined on the cartesian product of two hemispheres corresponding to the incident and outgoing directions; the natural way to represent such a hemispherical function is to use hemispherical basis. However, due to their compactness in the frequency space, spherical harmonics (SH) have been extensively used for this purpose. In this paper, we address the geometrical compliance of hemispherical basis for representing surface BRDF. We propose a tensor product of the hemispherical harmonics (HSH) to provide a compact and efficient representation for arbitrary BRDFs, while satisfying the Helmholtz reciprocity property. We provide an analytical analysis and experimental justification that for a given approximation order, our proposed hemispherical basis provide better approximation accuracy when compared to Zernike-based basis, while avoiding the high computational complexity inherited from such polynomials. We validate our proposed Helmholtz HSH-based basis functions on Oren-Nayar and Cook-Torrance BRDF physical models.

Surface irradiance signals are turned into outgoing radiance through the surface reflectance function, which can be significantly perturbed by the illumination conditions. Due to their low-frequency nature, irradiance signals can be represented using low-order basis functions, where spherical harmonics (SH) have been extensively used to provide such basis. When capturing image irradiance from a single viewpoint, the visible part of the object's surface constructs the upper hemisphere of the surface normals where the SH are no longer orthonormal. This reduced domain paves the way for even lower-dimensional approximation since full spherical representation is not needed. While harmonic basis are known to be optimal under distant light, light coming from near-by objects and indoor environments are common near light scenarios; it is essential to relax distant light assumption. Considering light source(s) distributed uniformly over the upper hemisphere, we propose the use of hemispherical harmonics (HSH) to model image irradiance of convex Lambertian objects perceived from single viewpoint under unknown near illumination. We prove analytically, and experimentally validated, that the Lambertian kernel has a more compact harmonic expansion in the hemispherical domain when compared to its spherical counterpart. We illustrate that HSH provide an efficient and accurate low-dimensional representation of image irradiance of Lambertian objects under near lighting conditions in contrast to SH.

A surface reflectance function represents the process of turning irradiance signals into outgoing radiance. Irradiance signals can be represented using low-order basis functions due to their low-frequency nature. Spherical harmonics (SH) have been used to provide such basis. However the incident light at any surface point is defined on the upper hemisphere; full spherical representation is not needed. We propose the use of hemispherical harmonics (HSH) to model images of convex Lambertian objects under distant illumination. We formulate and prove the addition theorem for HSH in order to provide an analytical expression of the reflectance function in the HSH domain. We prove that the Lambertian kernel has a more compact harmonic expansion in the HSH domain when compared to its SH counterpart. Our experiments illustrate that the 1st order HSH outperforms 1st and 2nd order SH in the process of image reconstruction as the number of light sources grows.

We discuss a statistical shape-from-shading framework for images of general and unknown illumination. To overcome arbitrary illumination, the framework makes use of the fact that general lighting can be expressed using low-order spherical harmonics for convex Lambertian objects. We cast the classical shape-from-shading equation as a Partial Least Squares (PLS) regression problem, which allows for the rapid computation of the solution. Results show accurate shape recovery with respect to ground truth data.

This paper introduces a framework for long-distance face recognition using dense and sparse stereo reconstruction, with texture of the facial region. Two methods to determine correspondences of the stereo pair are used in this paper: (a) dense global stereo-matching using maximum-a-posteriori Markov Random Fields (MAP-MRF) algorithms and (b) Active Appearance Model (AAM) fitting of both images of the stereo pair and using the fitted AAM mesh as the sparse correspondences. Experiments are performed using combinations of different features extracted from the dense and sparse reconstructions, as well as facial texture. The cumulative rank curves (CMC), which are generated using the proposed framework, confirms the feasibility of the proposed work for long distance recognition of human faces.

We introduce a framework for face recognition at a distance based on sparse-stereo reconstruction. We develop a 3D acquisition system that consists of two CCD stereo cameras mounted on pan-tilt units with adjustable baseline. We first detect the facial region and extract its landmark points, which are used to initialize an AAM mesh fitting algorithm. The fitted mesh vertices provide point correspondences between the left and right images of a stereo pair; stereo-based reconstruction is then used to infer the 3D information of the mesh vertices. We perform experiments regarding the use of different features extracted from these vertices for face recognition. The cumulative rank curves (CMC), which are generated using the proposed framework, confirms the feasibility of the proposed work for long distance recognition of human faces with respect to the state-of-the-art [3].

We present a new statistical shape-from-shading framework for images of unknown illumination. The object (e.g., face) to be reconstructed is described by a parametric model. To deal with arbitrary illumination, the framework makes use of recent results that general lighting can be expressed using low-order spherical harmonics for convex Lambertian objects. The classical shape-from-shading equation is modified according to this framework. Results show accurate shape recovery with respect to ground truth data.