Abstract We propose a novel framework for the statistical modeling and analysis of the spatio‐temporal shape variability in articulated 4D (i.e., 3D + time) shapes such as human bodies and animals. We treat articulated 3D shapes, represented using parametric models such as SMPL or its variants, as elements of the product space of shape and pose parameters. 4D shapes can then be seen as trajectories in this space, which has a nonlinear Riemannian structure. Our key contribution is to treat these trajectories as elements of a Riemannian shape space and propose computational tools that ( 1 ) perform temporal alignment of such trajectories to account for variations in their execution rates, ( 2 ) compute geodesics between trajectories, and thus 4D shapes, even when they exhibit different execution rates, and ( 3 ) statistically model the spatio‐temporal variability of collections of 4D shapes, enabling us to compute statistical summaries such as means and principal modes of variation. We derive a simple, yet efficient, framework for characterizing populations of 4D shapes using statistical models, which in turn can be used as a generative model for synthesizing novel 4D shapes by sampling from these distributions. We demonstrate the effectiveness of the proposed framework using publicly available 4D human and animal datasets, and show that it outperforms the state‐of‐the‐art both in terms of accuracy and computational efficiency. Our code, dataset, and videos that illustrate the results are available at https://arti4d.github.io/Arti4D/ .

Explainable Artificial Intelligence methods have gained significant traction for their ability to elucidate the decision-making processes of black-box models, particularly in high-stakes fields such as healthcare and finance. Among these, Local Interpretable Model-agnostic Explanations (LIME) stands out as a widely adopted post-hoc, model-agnostic approach that interprets black-box predictions by constructing an interpretable surrogate model on perturbed instances to approximate the local behavior of the original model around a given instance. However, the effectiveness of LIME can depend on the quality of the training data used by the black-box model. When trained on limited or low-quality data, the black-box model may yield inaccurate predictions for perturbed samples, resulting in poorly defined local decision boundaries and consequently unreliable explanations. This limitation is especially problematic in data-scarce settings. To overcome this challenge, we propose TransLIME, a novel end-to-end explainable transfer learning framework that improves the local fidelity and stability of LIME on limited tabular datasets by transferring relevant explainability knowledge from a related auxiliary source domain with a shifted distribution. Also, in TransLIME, only representative source prototype explanations obtained through clustering are transferred to the target domain, thereby reducing cross-domain exposure of both data and explanatory information during transfer. Experimental evaluations on real-world datasets demonstrate the effectiveness of the proposed framework in improving explanation quality in target domains with limited data.

Explainable Artificial Intelligence (XAI) methods, such as Local Interpretable Model-Agnostic Explanations (LIME), have advanced the interpretability of black-box machine learning models by approximating their behavior locally using interpretable surrogate models. However, LIME's inherent randomness in perturbation and sampling can lead to locality and instability issues, especially in scenarios with limited training data. In such cases, data scarcity can result in the generation of unrealistic variations and samples that deviate from the true data manifold. Consequently, the surrogate model may fail to accurately approximate the complex decision boundary of the original model. To address these challenges, we propose a novel Instance-based Transfer Learning LIME framework (ITL-LIME) that enhances explanation fidelity and stability in data-constrained environments. ITL-LIME introduces instance transfer learning into the LIME framework by leveraging relevant real instances from a related source domain to aid the explanation process in the target domain. Specifically, we employ clustering to partition the source domain into clusters with representative prototypes. Instead of generating random perturbations, our method retrieves pertinent real source instances from the source cluster whose prototype is most similar to the target instance. These are then combined with the target instance's neighboring real instances. To define a compact locality, we further construct a contrastive learning-based encoder as a weighting mechanism to assign weights to the instances from the combined set based on their proximity to the target instance. Finally, these weighted source and target instances are used to train the surrogate model for explanation purposes. Experimental evaluation with real-world datasets demonstrates that ITL-LIME greatly improves the stability and fidelity of LIME explanations in scenarios with limited data. Our code is available at https://github.com/rehanrazaa/ITL-LIME.

We propose a novel framework for the statistical analysis of genus-zero 4D surfaces, i.e., 3D surfaces that deform and evolve over time. This problem is particularly challenging due to the arbitrary parameterizations of these surfaces and their varying deformation speeds, necessitating effective spatiotemporal registration. Traditionally, 4D surfaces are discretized, in space and time, before computing their spatiotemporal registrations, geodesics, and statistics. However, this approach may result in suboptimal solutions and, as we demonstrate in this paper, is not necessary. In contrast, we treat 4D surfaces as continuous functions in both space and time. We introduce Dynamic Spherical Neural Surfaces (D-SNS), an efficient smooth and continuous spatiotemporal representation for genus-0 4D surfaces. We then demonstrate how to perform core 4D shape analysis tasks such as spatiotemporal registration, geodesics computation, and mean 4D shape estimation, directly on these continuous representations without upfront discretization and meshing. By integrating neural representations with classical Riemannian geometry and statistical shape analysis techniques, we provide the building blocks for enabling full functional shape analysis. We demonstrate the efficiency of the framework on 4D human and face datasets. The source code and additional results are available at https://4d-dsns.github.io/DSNS/.

Abstract Neural fields have emerged as a powerful framework for representing continuous multidimensional signals such as images and videos, 3D and 4D objects and scenes, and radiance fields. While efficient, achieving high‐quality representation requires the use of wide and deep neural networks. These, however, are slow to train and evaluate. Although several acceleration techniques have been proposed, they either trade memory for faster training and/or inference, rely on thousands of fitted primitives with considerable optimization time, or compromise the smooth, continuous nature of neural fields. In this paper, we introduce Gaussian Neural Fields (GNF), a novel compact neural decoder that maps learned feature grids into continuous non‐linear signals, such as RGB images, Signed Distance Functions (SDFs), and radiance fields, using a single compact layer of Gaussian kernels defined in a high‐dimensional feature space. Our key observation is that neurons in traditional MLPs perform simple computations, usually a dot product followed by an activation function, necessitating wide and deep MLPs or high‐resolution feature grids to model complex functions. In this paper, we show that replacing MLP‐based decoders with Gaussian kernels whose centers are learned features yields highly accurate representations of 2D (RGB), 3D (geometry), and 5D (radiance fields) signals with just a single layer of such kernels. This representation is highly parallelizable, operates on low‐resolution grids, and trains in under 15 seconds for 3D geometry and under 11 minutes for view synthesis. GNF matches the accuracy of deep MLP‐based decoders with far fewer parameters and significantly higher inference throughput. The source code is publicly available at https://grbfnet.github.io/ .

Abstract Patch-based matching is a technique meant to measure the disparity between pixels in a source and target image and is at the core of various methods in computer vision. When the subpixel disparity between the source and target images is required, the cost function or the target image has to be interpolated. While cost-based interpolation is easier to implement, multiple works have shown that image-based interpolation can increase the accuracy of the disparity estimate. In this paper we review closed-form formulae for subpixel disparity computation for one dimensional matching, e.g., rectified stereo matching, for the standard cost functions used in patch-based matching. We then propose new formulae to generalize to high-dimensional search spaces, which is necessary for unrectified stereo matching and optical flow. We also compare the image-based interpolation formulae with traditional cost-based formulae, and show that image-based interpolation brings a significant improvement over the cost-based interpolation methods for two dimensional search spaces, and small improvement in the case of one dimensional search spaces. The zero-mean normalized cross correlation cost function is found to be preferable for subpixel alignment. A new error model, based on very broad assumptions is outlined in the Supplementary Material to demonstrate why these image-based interpolation formulae outperform their cost-based counterparts and why the zero-mean normalized cross correlation function is preferable for subpixel alignement.

The method overview. The explicit representation on the right side represents the shared canonical space and the view space using 12 feature planes. The gray arrows indicate the feature projection from the canonical representation. The left side shows the deformation between the canonical space and the view space. Pairwise features (e.g., X Y − Z T ) from the canonical and view representations are aggregated to obtain the final feature vector. ⨂ indicates feature aggregation. Density and appearance decoders, which estimate the geometry and color, are conditioned on the scene’s latent s i . Existing explicit and implicit-explicit hybrid neural representations for novel view synthesis are scene-specific. In other words, they represent only a single scene and require retraining for every novel scene. Implicit scene-agnostic methods rely on large multilayer perception (MLP) networks conditioned on learned features. They are computationally expensive during training and rendering times. In contrast, we propose a novel plane-based representation that learns to represent multiple static and dynamic scenes during training and renders per-scene novel views during inference. The method consists of a deformation network, explicit feature planes, and a conditional decoder. Explicit feature planes are used to represent a time-stamped view space volume and a shared canonical volume across multiple scenes. The deformation network learns the deformations across shared canonical object space and time-stamped view space. The conditional decoder estimates the color and density of each scene constrained by a scene-specific latent code. We evaluated and compared the performance of the proposed representation on static (NeRF) and dynamic (Plenoptic videos) datasets. The results show that explicit planes combined with tiny MLPs can efficiently train multiple scenes simultaneously. The project page: https://anonpubcv.github.io/cplanes/ . • We present a novel multi-scene representation that uses twelve explicit feature planes. • The method uses encoder-less generalization to learn discriminative features for each scene. • We represent multiple dynamic scenes without relying on optical flow estimation. • The auto-decoded latent (the scene dimension) can interpolate between scenes. • It achieves state-of-the-art rendering results for both static and dynamic scenes.

Semantic instance completion aims to recover the complete 3D shapes of foreground objects together with their labels from a partial 2.5D scan of a scene. Previous works have relied on full supervision, which requires ground-truth annotations, in the form of bounding boxes and complete 3D objects. This has greatly limited their real-world application because the acquisition of ground-truth data is very costly and time-consuming. To address this bottleneck, we propose a Weakly-Supervised Semantic Instance Completion Network (WSSIC-Net), which learns real-world partial point cloud object completion without requiring the ground truth of complete 3D objects. Instead, WSSIC-Net leverages 3D ground-truth bounding boxes, partial objects of a raw scene, and unpaired synthetic 3D point clouds. More specifically, a 3D detector is used to encode partial point clouds into proposal features, which are then fed into two branches. The first branch uses fully supervised box prediction based on proposal features. The second branch, hereinafter called instance completion, leverages the proposal features as partial object features to achieve weakly-supervised instance completion. A Generative Adversarial Network (GAN) completes the partial features of the 2.5D foreground objects of real-world scenes using only unpaired but semantically-consistent complete synthetic point clouds. In our experiments, we demonstrate that the fully-supervised 3D detection and the weakly-supervised instance completion complement one another. The qualitative and quantitative evaluations on the ScanNet v2 dataset demonstrate that the proposed "weakly-supervised" approach consistently achieves comparable performance to the state-of-the-art "fully supervised" methods.

Point cloud completion aims to reconstruct complete 3D shapes from partial scans. The long-range dependencies between points and shape perception are crucial for this task. While Transformers are effective due to their global processing ability, the quadratic complexity of their attention mechanism makes them unsuitable for long sequences when computational resources are constrained. As an alternative, State Space Models (SSMs) provide a memory-efficient solution for handling long-range dependencies, yet applying them directly to unordered point clouds presents challenges because of their intrinsic causality requirements. Existing methods attempt to address this by sorting points along a single axis. This, however, often overlooks complex causal relationships in 3D space since adjacency relationships based on Euclidean distance between points in the 3D space may not be preserved by this linear arrangement. To overcome this issue, we introduce CompletionMamba, a novel SSM-based network designed to harness SSMs for capturing both global and local dependencies within a point cloud. Initially, the input point cloud is causally structured by rearranging its coordinates. Then, a local SSM framework is proposed that defines neighborhood spaces around each point based on Euclidean distance, enhancing the causal structure. Although local SSM enhances relationships in short and long distance sequences, it still lacks full shape modeling of point cloud. To address this, we propose a novel shape-aware Mamba by integrating the shape code of each 3D shape into the model, enabling shape information propagation to all points. Our experiments show that CompletionMamba achieves state-of-the-art performance on both the MVP and PCN datasets.

While latent diffusion models (LDMs) excel at creating imaginative images, they often lack precision in semantic fidelity and spatial control over where objects are generated. To address these deficiencies, we introduce the Box-it-to-Bind-it (B2B) module—a novel, training-free approach for improving spatial control and semantic accuracy in text-to-image (T2I) diffusion models. B2B targets three key challenges in T2I: catastrophic neglect, attribute binding, and layout guidance. The process encompasses two main steps: (i)Object generation, which adjusts the latent encoding to guarantee object generation and directs it within specified bounding boxes, and (ii)Attribute binding, ensuring that generated objects adhere to their specified attributes in the prompt. B2B is designed as a compatible plug-and-play module for existing T2I models like Stable Diffusion and Gligen, markedly enhancing models’ performance in addressing these key challenges. We assess our technique on the well-established CompBench and TIFA score benchmarks, and HRS dataset where B2B not only surpasses methods specialized in either attribute binding or layout guidance but also uniquely excels by integrating these capabilities to deliver enhanced overall performance.

How can one analyze detailed 3D biological objects, such as neuronal and botanical trees, that exhibit complex geometrical and topological variation? In this paper, we develop a novel mathematical framework for representing, comparing, and computing geodesic deformations between the shapes of such tree-like 3D objects. A hierarchical organization of subtrees characterizes these objects - each subtree has a main branch with some side branches attached - and one needs to match these structures across objects for meaningful comparisons. We propose a novel representation that extends the Square-Root Velocity Function (SRVF), initially developed for Euclidean curves, to tree-shaped 3D objects. We then define a new metric that quantifies the bending, stretching, and branch sliding needed to deform one tree-shaped object into the other. Compared to the current metrics such as the Quotient Euclidean Distance (QED) and the Tree Edit Distance (TED), the proposed representation and metric capture the full elasticity of the branches (i.e., bending and stretching) as well as the topological variations (i.e., branch death/birth and sliding). It completely avoids the shrinkage that results from the edge collapse and node split operations of the QED and TED metrics. We demonstrate the utility of this framework in comparing, matching, and computing geodesics between biological objects such as neuronal and botanical trees. We also demonstrate its application to various shape analysis tasks such as (i) symmetry analysis and symmetrization of tree-shaped 3D objects, (ii) computing summary statistics (means and modes of variations) of populations of tree-shaped 3D objects, (iii) fitting parametric probability distributions to such populations, and (iv) finally synthesizing novel tree-shaped 3D objects through random sampling from estimated probability distributions.

This paper proposes a novel transformer-based framework to generate accurate class-specific object localization maps for weakly supervised semantic segmentation (WSSS). Leveraging the insight that the attended regions of the one-class token in the standard vision transformer can generate class-agnostic localization maps, we investigate the transformer's capacity to capture class-specific attention for class-discriminative object localization by learning multiple class tokens. We present the Multi-Class Token transformer, which incorporates multiple class tokens to enable class-aware interactions with patch tokens. This is facilitated by a class-aware training strategy that establishes a one-to-one correspondence between output class tokens and ground-truth class labels. We also introduce a Contrastive-Class-Token (CCT) module to enhance the learning of discriminative class tokens, enabling the model to better capture the unique characteristics of each class. Consequently, the proposed framework effectively generates class-discriminative object localization maps from the class-to-patch attentions associated with different class tokens. To refine these localization maps, we propose the utilization of patch-level pairwise affinity derived from the patch-to-patch transformer attention. Furthermore, the proposed framework seamlessly complements the Class Activation Mapping (CAM) method, yielding significant improvements in WSSS performance on PASCAL VOC 2012 and MS COCO 2014. These results underline the importance of the class token for WSSS.

Most existing learning-based 3D point cloud completion methods ignore the fact that the completion process is highly coupled with the viewpoint of a partial scan. However, the various viewpoints of incompletely scanned objects in real-world applications are normally unknown and directly estimating the viewpoint of each incomplete object is usually time-consuming and leads to huge annotation cost. In this paper, we thus propose an unsupervised viewpoint representation learning scheme for 3D point cloud completion without explicit viewpoint estimation. To be specific, we learn abstract representations of partial scans to distinguish various viewpoints in the representation space rather than the explicit estimation in the 3D space. We also introduce a Viewpoint-Aware Point cloud Completion Network (VAPCNet) with flexible adaption to various viewpoints based on the learned representations. The proposed viewpoint representation learning scheme can extract discriminative representations to obtain accurate viewpoint information. Reported experiments on two popular public datasets show that our VAPCNet achieves state-of-the-art performance for the point cloud completion task. Source code is available at https://github.com/FZH92128/VAPCNet.

3D face reconstruction plays a major role in many human-robot interaction systems, from automatic face authentication to human-computer interface-based entertainment. To improve robustness against occlusions and noise, 3D face reconstruction networks are often trained on a set of in-the-wild face images preferably captured along different viewpoints of the subject. However, collecting the required large amounts of 3D annotated face data is expensive and time-consuming. To address the high annotation cost and due to the importance of training on a useful set, we propose an Active Learning (AL) framework that actively selects the most informative and representative samples to be labeled. To the best of our knowledge, this paper is the first work on tackling active learning for 3D face reconstruction to enable a label-efficient training strategy. In particular, we propose a Reinforcement Active Learning approach in conjunction with a clustering-based pooling strategy to select informative view-points of the subjects. Experimental results on 300W-LP and AFLW2000 datasets demonstrate that our proposed method is able to 1) efficiently select the most influencing view-points for labeling and outperforms several baseline AL techniques and 2) further improve the performance of a 3D Face Reconstruction network trained on the full dataset.

Abstract 3D point clouds can represent complex 3D objects of arbitrary topologies and with fine‐grained details. They are, however, hard to regress from images using convolutional neural networks, making tasks such as 3D reconstruction from monocular RGB images challenging. In fact, unlike images and volumetric grids, point clouds are unstructured and thus lack proper parameterization, which makes them difficult to process using convolutional operations. Existing point‐based 3D reconstruction methods that tried to address this problem rely on complex end‐to‐end architectures with high computational costs. Instead, we propose in this paper a novel mechanism that decouples the 3D reconstruction problem from the structure (or parameterization) learning task, making the 3D reconstruction of objects of arbitrary topologies tractable and thus easier to learn. We achieve this using a novel Teacher‐Student network where the Teacher learns to structure the point clouds. The Student then harnesses the knowledge learned by the Teacher to efficiently regress accurate 3D point clouds. We train the Teacher network using 3D ground‐truth supervision and the Student network using the Teacher's annotations. Finally, we employ a novel refinement network to overcome the upper‐bound performance that is set by the Teacher network. Our extensive experiments on ShapeNet and Pix3D benchmarks, and on in‐the‐wild images demonstrate that the proposed approach outperforms previous methods in terms of reconstruction accuracy and visual quality.

Abstract Recent advances in digital technologies have lowered the costs and improved the quality of digital pathology Whole Slide Images (WSI), opening the door to apply Machine Learning (ML) techniques to assist in cancer diagnosis. ML, including Deep Learning (DL), has produced impressive results in diverse image classification tasks in pathology, such as predicting clinical outcomes in lung cancer and inferring regional gene expression signatures. Despite these promising results, the uptake of ML as a common diagnostic tool in pathology remains limited. A major obstacle is the insufficient labelled data for training neural networks and other classifiers, especially for new sites where models have not been established yet. Recently, image synthesis from small, labelled datasets using Generative Adversarial Networks (GAN) has been used successfully to create high-performing classification models. Considering the domain shift and complexity in annotating data, we investigated an approach based on GAN that minimized the differences in WSI between large public data archive sites and a much smaller data archives at the new sites. The proposed approach allows the tuning of a deep learning classification model for the class of interest to be improved using a small training set available at the new sites. This paper utilizes GAN with the one-class classification concept to model the class of interest data. This approach minimizes the need for large amounts of labelled data from the new site to train the network. The GAN generates synthesized one-class WSI images to jointly train the classifier with WSIs available from the new sites. We tested the proposed approach for follicular lymphoma data of a new site by utilizing the data archives from different sites. The synthetic images for the one-class data generated from the data obtained from different sites with minimum amount of data from the new site have resulted in a significant improvement of 15% for the Area Under the curve (AUC) for the new site that we want to establish a new follicular lymphoma classifier. The test results have shown that the classifier can perform well without the need to obtain more training data from the test site, by utilizing GAN to generate the synthetic data from all existing data in the archives from all the sites.

We propose a novel framework to learn the spatiotemporal variability in longitudinal 3D shape data sets, which contain observations of objects that evolve and deform over time. This problem is challenging since surfaces come with arbitrary parameterizations and thus, they need to be spatially registered. Also, different deforming objects, hereinafter referred to as 4D surfaces, evolve at different speeds and thus they need to be temporally aligned. We solve this spatiotemporal registration problem using a Riemannian approach. We treat a 3D surface as a point in a shape space equipped with an elastic Riemannian metric that measures the amount of bending and stretching that the surfaces undergo. A 4D surface can then be seen as a trajectory in this space. With this formulation, the statistical analysis of 4D surfaces can be cast as the problem of analyzing trajectories embedded in a nonlinear Riemannian manifold. However, performing the spatiotemporal registration, and subsequently computing statistics, on such nonlinear spaces is not straightforward as they rely on complex nonlinear optimizations. Our core contribution is the mapping of the surfaces to the space of Square-Root Normal Fields (SRNF) where the [Formula: see text] metric is equivalent to the partial elastic metric in the space of surfaces. Thus, by solving the spatial registration in the SRNF space, the problem of analyzing 4D surfaces becomes the problem of analyzing trajectories embedded in the SRNF space, which has a euclidean structure. In this paper, we develop the building blocks that enable such analysis. These include: (1) the spatiotemporal registration of arbitrarily parameterized 4D surfaces even in the presence of large elastic deformations and large variations in their execution rates; (2) the computation of geodesics between 4D surfaces; (3) the computation of statistical summaries, such as means and modes of variation, of collections of 4D surfaces; and (4) the synthesis of random 4D surfaces. We demonstrate the performance of the proposed framework using 4D facial surfaces and 4D human body shapes.

In stereo vision, self-similar or bland regions can make it difficult to match patches between two images. Active stereo-based methods mitigate this problem by projecting a pseudo-random pattern on the scene so that each patch of an image pair can be identified without ambiguity. However, the projected pattern significantly alters the appearance of the image. If this pattern acts as a form of adversarial noise, it could negatively impact the performance of deep learning-based methods, which are now the de-facto standard for dense stereo vision. In this paper, we propose the Active-Passive SimStereo dataset and a corresponding benchmark to evaluate the performance gap between passive and active stereo images for stereo matching algorithms. Using the proposed benchmark and an additional ablation study, we show that the feature extraction and matching modules of a selection of twenty selected deep learning-based stereo matching methods generalize to active stereo without a problem. However, the disparity refinement modules of three of the twenty architectures (ACVNet, CascadeStereo, and StereoNet) are negatively affected by the active stereo patterns due to their reliance on the appearance of the input images.

Abstract We propose a deep reinforcement learning‐based solution for the 3D reconstruction of objects of complex topologies from a single RGB image. We use a template‐based approach. However, unlike previous template‐based methods, which are limited to the reconstruction of 3D objects of fixed topology, our approach learns simultaneously the geometry and topology of the target 3D shape in the input image. To this end, we propose a neural network that learns to deform a template to fit the geometry of the target object. Our key contribution is a novel reinforcement learning framework that enables the network to also learn how to adjust, using pruning operations, the topology of the template to best fit the topology of the target object. We train the network in a supervised manner using a loss function that enforces smoothness and penalizes long edges in order to ensure high visual plausibility of the reconstructed 3D meshes. We evaluate the proposed approach on standard benchmarks such as ShapeNet, and in‐the‐wild using unseen real‐world images. We show that the proposed approach outperforms the state‐of‐the‐art in terms of the visual quality of the reconstructed 3D meshes, and also generalizes well to out‐of‐category images.

Estimating depth from RGB images is a long-standing ill-posed problem, which has been explored for decades by the computer vision, graphics, and machine learning communities. Among the existing techniques, stereo matching remains one of the most widely used in the literature due to its strong connection to the human binocular system. Traditionally, stereo-based depth estimation has been addressed through matching hand-crafted features across multiple images. Despite the extensive amount of research, these traditional techniques still suffer in the presence of highly textured areas, large uniform regions, and occlusions. Motivated by their growing success in solving various 2D and 3D vision problems, deep learning for stereo-based depth estimation has attracted a growing interest from the community, with more than 150 papers published in this area between 2014 and 2019. This new generation of methods has demonstrated a significant leap in performance, enabling applications such as autonomous driving and augmented reality. In this paper, we provide a comprehensive survey of this new and continuously growing field of research, summarize the most commonly used pipelines, and discuss their benefits and limitations. In retrospect of what has been achieved so far, we also conjecture what the future may hold for deep learning-based stereo for depth estimation research.

3D reconstruction is a longstanding ill-posed problem, which has been explored for decades by the computer vision, computer graphics, and machine learning communities. Since 2015, image-based 3D reconstruction using convolutional neural networks (CNN) has attracted increasing interest and demonstrated an impressive performance. Given this new era of rapid evolution, this article provides a comprehensive survey of the recent developments in this field. We focus on the works which use deep learning techniques to estimate the 3D shape of generic objects either from a single or multiple RGB images. We organize the literature based on the shape representations, the network architectures, and the training mechanisms they use. While this survey is intended for methods which reconstruct generic objects, we also review some of the recent works which focus on specific object classes such as human body shapes and faces. We provide an analysis and comparison of the performance of some key papers, summarize some of the open problems in this field, and discuss promising directions for future research.

Abstract We propose a framework for statistical modeling of the 3D geometry and topology of botanical trees. We treat botanical trees as points in a tree‐shape space equipped with a proper metric that captures the geometric and the topological differences between trees. Geodesics in the tree‐shape space correspond to the optimal sequence of deformations, i.e. bending, stretching, and topological changes, which align one tree onto another. In this way, the 3D tree modeling and synthesis problem becomes a problem of exploring the tree‐shape space either in a controlled fashion, using statistical regression, or randomly by sampling from probability distributions fitted to populations in the tree‐shape space. We show how to use this framework for (1) computing statistical summaries, e.g. the mean and modes of variations, of a population of botanical trees, (2) synthesizing random instances of botanical trees from probability distributions fitted to a population of botanical trees, and (3) modeling, interactively, 3D botanical trees using a simple sketching interface. The approach is fast and only requires as input 3D botanical tree models with a known upright orientation.

We propose an algorithm for generating novel 3D tree model variations from existing ones via geometric and structural blending. Our approach is to treat botanical trees as elements of a tree-shape space equipped with a proper metric that quantifies geometric and structural deformations. Geodesics, or shortest paths under the metric, between two points in the tree-shape space correspond to optimal deformations that align one tree onto another, including the possibility of expanding, adding, or removing branches and parts. Central to our approach is a mechanism for computing correspondences between trees that have different structures and a different number of branches. The ability to compute geodesics and their lengths enables us to compute continuous blending between botanical trees, which, in turn, facilitates statistical analysis, such as the computation of averages of tree structures. We show a variety of 3D tree models generated with our approach from 3D trees exhibiting complex geometric and structural differences. We also demonstrate the application of the framework in reflection symmetry analysis and symmetrization of botanical trees.

Abstract We introduce co‐variation analysis as a tool for modeling the way part geometries and configurations co‐vary across a family of man‐made 3D shapes. While man‐made 3D objects exhibit large geometric and structural variations, the geometry, structure, and configuration of their individual components usually do not vary independently from each other but in a correlated fashion. The size of the body of an airplane, for example, constrains the range of deformations its wings can undergo to ensure that the entire object remains a functionally‐valid airplane. These co‐variation constraints, which are often non‐linear, can be either physical, and thus they can be explicitly enumerated, or implicit to the design and style of the shape family. In this article, we propose a data‐driven approach, which takes pre‐segmented 3D shapes with known component‐wise correspondences and learns how various geometric and structural properties of their components co‐vary across the set. We demonstrate, using a variety of 3D shape families, the utility of the proposed co‐variation analysis in various applications including 3D shape repositories exploration and shape editing where the propagation of deformations is guided by the co‐variation analysis. We also show that the framework can be used for context‐guided orientation of objects in 3D scenes.

Recent developments in elastic shape analysis (ESA) are motivated by the fact that it provides a comprehensive framework for simultaneous registration, deformation, and comparison of shapes. These methods achieve computational efficiency using certain square-root representations that transform invariant elastic metrics into euclidean metrics, allowing for the application of standard algorithms and statistical tools. For analyzing shapes of embeddings of in , Jermyn et al. [1] introduced square-root normal fields (SRNFs), which transform an elastic metric, with desirable invariant properties, into the metric. These SRNFs are essentially surface normals scaled by square-roots of infinitesimal area elements. A critical need in shape analysis is a method for inverting solutions (deformations, averages, modes of variations, etc.) computed in SRNF space, back to the original surface space for visualizations and inferences. Due to the lack of theory for understanding SRNF maps and their inverses, we take a numerical approach, and derive an efficient multiresolution algorithm, based on solving an optimization problem in the surface space, that estimates surfaces corresponding to given SRNFs. This solution is found to be effective even for complex shapes that undergo significant deformations including bending and stretching, e.g., human bodies and animals. We use this inversion for computing elastic shape deformations, transferring deformations, summarizing shapes, and for finding modes of variability in a given collection, while simultaneously registering the surfaces. We demonstrate the proposed algorithms using a statistical analysis of human body shapes, classification of generic surfaces, and analysis of brain structures.

Co-located haptic feedback in mixed and augmented reality environments can improve realism and user performance, but it also requires careful system design and calibration. In this poster, we determine the thresholds for perceiving co-location errors through two psychophysics experiments in a typical fine-motor manipulation task. In these experiments we simulate the two fundamental ways of implementing VHAR systems: first, attaching a real tool; second, augmenting a virtual tool. We determined the just-noticeable co-location errors for position and orientation in both experiments and found that users are significantly more sensitive to co-location errors with virtual tools. Our overall findings are useful for designing visuo-haptic augmented reality workspaces and calibration procedures.

In this article, we investigate the relationship between the range of optimal parameters of active contour models and the shape of the target object. We focus on the weights of the internal and external energy terms of the snakes functional. Our contributions are 3-fold. First, we propose a normalization step that brings the search space for optimal parameters into a bounded range. Secondly, we perform a systematic study of the behaviour of active contour models for all possible settings of their parameters and on a large set of synthetic geometric shapes. We introduce the concept of stability diagrams as a novel approach for assessing the stability of active contour models given a range of parameter values. Finally, we show that over a series of evolving shapes the region of the parameter domain that corresponds to suitable coefficients for segmentation, hereinafter referred to as feasible solution region, follows a predictable trend. Using shape diagrams as a metric for characterizing shapes quantitatively, we are able to correlate the shape of the objects to segment with the location and extent of the feasible solution region in the parameter domain.

State-of-the-art 3D shape classification and retrieval algorithms, hereinafter referred to as shape analysis, are often based on comparing signatures or descriptors that capture the main geometric and topological properties of 3D objects. None of the existing descriptors, however, achieve best performance on all shape classes. In this article, we explore, for the first time, the usage of covariance matrices of descriptors, instead of the descriptors themselves, in 3D shape analysis. Unlike histogram -based techniques, covariance-based 3D shape analysis enables the fusion and encoding of different types of features and modalities into a compact representation. Covariance matrices, however, are elements of the non-linear manifold of symmetric positive definite (SPD) matrices and thus \BBL2 metrics are not suitable for their comparison and clustering. In this article, we study geodesic distances on the Riemannian manifold of SPD matrices and use them as metrics for 3D shape matching and recognition. We then: (1) introduce the concepts of bag of covariance (BoC) matrices and spatially-sensitive BoC as a generalization to the Riemannian manifold of SPD matrices of the traditional bag of features framework, and (2) generalize the standard kernel methods for supervised classification of 3D shapes to the space of covariance matrices. We evaluate the performance of the proposed BoC matrices framework and covariance -based kernel methods and demonstrate their superiority compared to their descriptor-based counterparts in various 3D shape matching, retrieval, and classification setups.

Visuo-haptic augmented reality systems enable users to see and touch digital information that is embedded in the real world. PHANToM haptic devices are often employed to provide haptic feedback. Precise co-location of computer-generated graphics and the haptic stylus is necessary to provide a realistic user experience. Previous work has focused on calibration procedures that compensate the non-linear position error caused by inaccuracies in the joint angle sensors. In this article we present a more complete procedure that additionally compensates for errors in the gimbal sensors and improves position calibration. The proposed procedure further includes software-based temporal alignment of sensor data and a method for the estimation of a reference for position calibration, resulting in increased robustness against haptic device initialization and external tracker noise. We designed our procedure to require minimal user input to maximize usability. We conducted an extensive evaluation with two different PHANToMs, two different optical trackers, and a mechanical tracker. Compared to state-of-the-art calibration procedures, our approach significantly improves the co-location of the haptic stylus. This results in higher fidelity visual and haptic augmentations, which are crucial for fine-motor tasks in areas such as medical training simulators, assembly planning tools, or rapid prototyping applications.

Several descriptors have been proposed in the past for 3D shape analysis, yet none of them achieves best performance on all shape classes. In this paper we propose a novel method for 3D shape analysis using the covariance matrices of the descriptors rather than the descriptors themselves. Covariance matrices enable efficient fusion of different types of features and modalities. They capture, using the same representation, not only the geometric and the spatial properties of a shape region but also the correlation of these properties within the region. Covariance matrices, however, lie on the manifold of Symmetric Positive Definite (SPD) tensors, a special type of Riemannian manifolds, which makes comparison and clustering of such matrices challenging. In this paper we study covariance matrices in their native space and make use of geodesic distances on the manifold as a dissimilarity measure. We demonstrate the performance of this metric on 3D face matching and recognition tasks. We then generalize the Bag of Features paradigm, originally designed in Euclidean spaces, to the Riemannian manifold of SPD matrices. We propose a new clustering procedure that takes into account the geometry of the Riemannian manifold. We evaluate the performance of the proposed Bag of Covariance Matrices framework on 3D shape matching and retrieval applications and demonstrate its superiority compared to descriptor-based techniques.

We propose in this paper a fully automatic approach for image-based plant stornata phenotyping. Given a microscopic image of a plant leaf surface, our goal is to automatically detect stornata cells and measure their morphological and structural features, such as stornata opening length and width, and size of the guard cells. The main challenge in developing such tool is the lack of contrast between the stornata cell region and its surrounding background. Our approach uses template matching to detect individual stornata cells and local analysis to measure stornata features within the detected stornata regions. It is fully automatic and computationally efficient. Thus, it will enable plant biologists to perform large scale analysis of stornata morphology, which in turn will help in developing understanding and controlling plant's response to various environmental stresses (e.g. drought and soil salinity).

Visuo-haptic augmented reality systems enable users to see and touch digital information that is embedded in the real world. Precise co-location of computer graphics and the haptic stylus is necessary to provide a realistic user experience. PHANToM haptic devices are often used in such systems to provide haptic feedback. They consist of two interlinked joints, whose angles define the position of the haptic stylus and three sensors at the gimbal to sense its orientation. Previous work has focused on calibration procedures that align the haptic workspace within a global reference coordinate system and developing algorithms that compensate the non-linear position error, caused by inaccuracies in the joint angle sensors. In this paper, we present an improved workspace calibration that additionally compensates for errors in the gimbal sensors. This enables us to also align the orientation of the haptic stylus with high precision. To reduce the required time for calibration and to increase the sampling coverage, we utilize time-delay estimation to temporally align external sensor readings. This enables users to continuously move the haptic stylus during the calibration process, as opposed to commonly used point and hold processes. We conducted an evaluation of the calibration procedure for visuo-haptic augmented reality setups with two different PHANToMs and two different optical trackers. Our results show a significant improvement of orientation alignment for both setups over the previous state of the art calibration procedure. Improved position and orientation accuracy results in higher fidelity visual and haptic augmentations, which is crucial for fine-motor tasks in areas including medical training simulators, assembly planning tools, or rapid prototyping applications. A user friendly calibration procedure is essential for real-world applications of VHAR.

Visuo-haptic augmented reality systems enable users to see and touch digital information that is embedded in the real world. Precise colocation of computer graphics and the haptic stylus is necessary to provide a realistic user experience. PHANToM haptic devices are often used in such systems to provide haptic feedback. They consist of two interlinked joints, whose angles define the position of the haptic stylus and three sensors at the gimbal to sense its orientation. Previous work has focused on a calibration procedures that align the haptic workspace within a global reference coordinate system and an algorithms that compensate the non-linear position error, which is caused by inaccuracies in the joint angle sensors. In our science and technology paper “Comprehensive Workspace Calibration for Visuo-Haptic Augmented Reality” [1], we present an improved workspace calibration that additionally compensates for errors in the gimbal sensors. This enables us to also align the orientation of the haptic stylus with high precision. To reduce the required time for calibration and to increase the sampling coverage, we utilize time-delay estimation to temporally align external sensor readings. This enables users to continuously move the haptic stylus during the calibration process, as opposed to commonly used point and hold processes. This demonstration showcases the complete workspace calibration procedure as described in our paper including a mixed reality demo scenario, that allows users to experience the calibrated workspace. Additionally, we demonstrate an early stage of our proposed future work in improved user guidance during the calibration procedure using visual guides.

Reflection symmetry is an important feature of an object. Main goals in symmetry analysis include quantifying the amount of asymmetry in an object and finding the nearest symmetric object to a given asymmetric one. Samir et al. [19] achieved these goals using a shape distance between representations of curves termed square-root velocity functions. We extend their work by defining shape descriptors based on this representation. The descriptors are based on asymmetry measures computed for a set of reflections of a curve and are invariant to all shape preserving transformations (translation, scale, rotation and re-parameterization). We utilize these descriptors for retrieval of shapes in the Flavia leaf database and a subset of a handwritten digit dataset. We show that we outperform the commonly used angle function and other state of the art descriptors.

Elastic shape analysis on non-linear Riemannian manifolds provides an efficient and elegant way for simultaneous comparison and registration of non-rigid shapes. In such formulation, shapes become points on some high dimensional shape space. A geodesic between two points corresponds to the optimal deformation needed to register one shape onto another. The length of the geodesic provides a proper metric for shape comparison. However, the computation of geodesics, and therefore the metric, is computationally very expensive as it involves a search over the space of all possible rotations and re- parameterization. This problem is even more important in shape retrieval scenarios where the query shape is compared to every element in the collection to search. In this paper, we propose a new procedure for metric approximation using the framework of kernel functions. We will demonstrate that this provides a fast approximation of the metric while preserving its invariance properties.

The shapes of plant leaves are of great importance to plant biologists and botanists, as they can help in distinguishing plant species, measuring their health, analyzing their growth patterns, and understanding relations between various species. We propose a statistical model that uses the Squared Root Velocity Function representation and a Riemannian elastic metric to model the observed variability in the shape of plant leaves. We show that under this representation, one can compute sample means and principal modes of variations and can characterize the observed shapes using probability models, such as Gaussians, on the tangent spaces at the sample means. The approach is fully automatic and does not require precomputing correspondences between the shapes. We validate these statistical models by analyzing their classification performance on standard benchmarks and show their utility as generative models for random sampling.

In this paper, we present an efficient 3D object retrieval method invariant to scale, orientation and pose. Our approach is based on the dense extraction of discriminative local descriptors extracted from 2D views. We aggregate the descriptors into a single vector signature using tensor products. The similarity between 3D models can then be efficiently computed with a simple dot product. Experiments on the SHREC12 commonly-used benchmark demonstrate that our approach obtains superior performance in searching for generic shapes.

During the last decade, Visuo-Haptic Augmented Reality (VHAR) systems have emerged that enable users to see and touch digital information that is embedded in the real world. They pose unique problems to developers, including the need for precise augmentations, accurate colocation of haptic devices, and efficient concurrent processing of multiple, realtime sensor inputs to achieve low latency. We think that this complexity is one of the main reasons, why VHAR technology has only been used in few user interface research projects. The proposed project's main objective is to pioneer the development of a widely applicable VHAR runtime environment, which meets the requirements of realtime, low latency operation with precise co-location, haptic interaction with deformable bodies, and realistic rendering, while reducing the overall cost and complexity for developers. A further objective is to evaluate the benefits of VHAR user interfaces with a focus on medical training applications, so that creators of future medical simulators or other haptic applications recognize the potential of VHAR.

In the last decade, Augmented Reality has become more mature and is widely adopted on mobile devices. Exploring the available information of a user's environment is one of the key applications. However, current mobile Augmented Reality interfaces are very limited compared to the recently emerging big data exploration tools for desktop computers. Our vision is to bring powerful Visual Analytic tools to mobile Augmented Reality.

Abstract We argue that full surface correspondence (registration) and optimal deformations (geodesics) are two related problems and propose a framework that solves them simultaneously. We build on the Riemannian shape analysis of anatomical and star‐shaped surfaces of Kurtek et al. and focus on articulated complex shapes that undergo elastic deformations and that may contain missing parts. Our core contribution is the re‐formulation of Kurtek et al.'s approach as a constrained optimization over all possible re‐parameterizations of the surfaces, using a sparse set of corresponding landmarks. We introduce a landmark‐constrained basis, which we use to numerically solve this optimization and therefore establish full surface registration and geodesic deformation between two surfaces. The length of the geodesic provides a measure of dissimilarity between surfaces. The advantages of this approach are: (1) simultaneous computation of full correspondence and geodesic between two surfaces, given a sparse set of matching landmarks (2) ability to handle more comprehensive deformations than nearly isometric, and (3) the geodesics and the geodesic lengths can be further used for symmetrizing 3D shapes and for computing their statistical averages. We validate the framework on challenging cases of large isometric and elastic deformations, and on surfaces with missing parts. We also provide multiple examples of averaging and symmetrizing 3D models.

We address the problem of automatic recognition of functional parts of man-made 3D shapes in the presence of significant geometric and topological variations. We observe that under such challenging circumstances, the context of a part within a 3D shape provides important cues for learning the semantics of shapes. We propose to model the context as structural relationships between shape parts and use them, in addition to part geometry, as cues for functionality recognition. We represent a 3D shape as a graph interconnecting parts that share some spatial relationships. We model the context of a shape part as walks in the graph. Similarity between shape parts can then be defined as the similarity between their contexts, which in turn can be efficiently computed using graph kernels. This formulation enables us to: (1) find part-wise semantic correspondences between 3D shapes in a nonsupervised manner and without relying on user-specified textual tags, and (2) design classifiers that learn in a supervised manner the functionality of the shape components. We specifically show that the performance of the proposed context-aware similarity measure in finding part-wise correspondences outperforms geometry-only-based techniques and that contextual analysis is effective in dealing with shapes exhibiting large geometric and topological variations.

We propose in this paper an interactive sketch-based system for simulating oriental brush strokes on complex shapes. We introduce a contour-driven approach where the user inputs contours to represent complex shapes, the system estimates automatically the optimal trajectory of the brush, and then renders them into oriental ink painting. Unlike previous work where the brush trajectory is explicitly specified as input, we automatically estimate this trajectory given the outline of the shape to paint. Existing methods can be classified into: (1) methods that explicitly model a virtual 3D brush and mimic its effect on a paper [Wang and Wang 2007], and (2) methods that simulate the rendering effect on a 2D canvas without an explicit 3D brush model [Okabe et al. 2007]. Our approach falls into the second category. Figure 1 shows four results generated by our algorithm.

The description of 3D shapes with features that possess descriptive power and invariant under similarity transformations is one of the most challenging issues in content based 3D model retrieval. Spherical harmonics-based descriptors have been proposed for obtaining rotation invariant representations. However, spherical harmonic analysis is based on latitude-longitude parameterization of a sphere which has singularities at each pole. Consequently, features near the two poles are over represented while features at the equator are under-sampled, and variations of the north pole affects significantly the shape function. In this paper we discuss these issues and propose the usage of spherical wavelet transform as a tool for the analysis of 3D shapes represented by functions on the unit sphere. We introduce three new descriptors extracted from the wavelet coefficients, namely: (1) a subset of the spherical wavelet coefficients, (2) the L1and, (3) the L2energies of the spherical wavelet sub-bands. The advantage of this tool is three fold; first, it takes into account feature localization and local orientations. Second, the energies of the wavelet transform are rotation invariant. Third, shape features are uniformly represented which makes the descriptors more efficient. Spherical wavelet descriptors are natural extension of 3D Zernike moments and spherical harmonics. We evaluate, on the Princeton shape benchmark, the proposed descriptors regarding computational aspects and shape retrieval performance

We describe our preliminary findings in applying the spherical parametrization and geometry images to the task of 3D shape matching and similarity based comparison of polygon soup models. Unlike traditional approach where multiple 2D views of the same object are required to capture the relevant geometry features, our proposed technique uses spherical parametrization and geometry images for 3D shape matching. This technique reduces the problem to the 2D space without computing 2D projections and guarantees the preservation of small details. Moreover, we take advantage from the hierarchical nature of the parametrization process, through the progressive mesh simplification, to derive a multiresolution analysis technique. Our proposed algorithm is invariant to similarity transformations such as rotation and scaling. The efficiency of this approach is discussed through a set of experiments

In this paper, we present a new framework for analyzing and segmenting point-sampled 3D objects. Our method first computes for each surface point the surface curvature distribution by applying the principal component analysis on local neighborhoods with different sizes. Then we model in the four dimensional space the joint distribution of surface curvature and position features as a mixture of Gaussians using the expectation maximization algorithm. Central to our method is the extension of the scale-space theory from the 2D domain into the three-dimensional space to allow feature analysis and classification at different scales. Our algorithm operates directly on points requiring no vertex connectivity information. We demonstrate and discuss the performance of our framework on a collection of point sampled 3D objects.