A photorealistic and controllable 3D caricaturization framework for faces is introduced. We start with an intrinsic Gaussian curvature-based surface exaggeration technique, which, when coupled with texture, tends to produce oversmoothed renders. To address this, we resort to 3D Gaussian Splatting (3DGS), which has recently been shown to produce realistic free-viewpoint avatars. Given a multiview sequence, we extract a FLAME mesh, solve a curvatureweighted Poisson equation, and obtain its exaggerated form. However, directly deforming the Gaussians yields poor results, necessitating the synthesis of pseudo–groundtruth caricature images by warping each frame to its exaggerated 2D representation using local affine transformations. We then devise a training scheme that alternates real and synthesized supervision, enabling a single Gaussian collection to represent both natural and exaggerated avatars. This scheme improves fidelity, supports local edits, and allows continuous control over the intensity of the caricature. In order to achieve real-time deformations, an efficient interpolation between the original and exaggerated surfaces is introduced. We further analyze and show that it has a bounded deviation from closed-form solutions. In both quantitative and qualitative evaluations, our results outperform prior work, delivering photorealistic, geometry-controlled caricature avatars. Project page: https://c4ricaturegs.github.io

Dimensionality reduction is a fundamental task that aims to simplify complex data by reducing its feature dimensionality while preserving essential patterns, with core applications in data analysis and visualisation. To preserve the underlying data structure, multi-dimensional scaling (MDS) methods focus on preserving pairwise dissimilarities, such as distances. They optimise the embedding to have pairwise distances as close as possible to the data dissimilarities. However, the current standard is limited to embedding data in Riemannian manifolds. Motivated by the lack of asymmetry in the Riemannian metric of the embedding space, this paper extends the MDS problem to a natural asymmetric generalisation of Riemannian manifolds called Finsler manifolds. Inspired by Euclidean space, we define a canonical Finsler space for embedding asymmetric data. Due to its simplicity with respect to geodesics, data representation in this space is both intuitive and simple to analyse. We demonstrate that our generalisation benefits from the same theoretical convergence guarantees. We reveal the effectiveness of our Finsler embedding across various types of non-symmetric data, highlighting its value in applications such as data visualisation, dimensionality reduction, directed graph embedding, and link prediction.

Recent advances in image editing, driven by image diffusion models, have shown remarkable progress. However, significant challenges remain, as these models often struggle to follow complex edit instructions accurately and frequently compromise fidelity by altering key elements of the original image. Simultaneously, video generation has made remarkable strides, with models that effectively function as consistent and continuous world simulators. In this paper, we propose merging these two fields by utilizing image-to-video models for image editing. We reformulate image editing as a temporal process, using pretrained video models to create smooth transitions from the original image to the desired edit. This approach traverses the image manifold continuously, ensuring consistent edits while preserving the original image’s key aspects. Our approach achieves stateof-the-art results on text-based image editing, demonstrating significant improvements in both edit accuracy and image preservation. Visit our project page.

Image editing has advanced significantly with the introduction of text-conditioned diffusion models. Despite this progress, seamlessly adding objects to images based on textual instructions without requiring user-provided input masks remains a challenge. We address this by leveraging the insight that removing objects (Inpaint) is significantly simpler than its inverse process of adding them (Paint), attributed to inpainting models that benefit from segmentation mask guidance. Capitalizing on this realization, by implementing an automated and extensive pipeline, we curate a filtered large-scale image dataset containing pairs of images and their corresponding object-removed versions. Using these pairs, we train a diffusion model to inverse the inpainting process, effectively adding objects into images. Unlike other editing datasets, ours features natural target images instead of synthetic ones while ensuring source-target consistency by construction. Additionally, we utilize a large Vision-Language Model to provide detailed descriptions of the removed objects and a Large Language Model to convert these descriptions into diverse, natural-language instructions. Our quantitative and qualitative results show that the trained model surpasses existing models in both object addition and general editing tasks. Visit our project page for the released dataset and trained models.

CoordFlow is a pixel-wise Implicit Neural Representation (INR) framework for video compression, achieving state-of-the-art results among pixel-wise methods and rivaling both frame-wise and classic techniques. By segmenting videos into layers, each represented by specialized neural networks with built-in motion compensation, CoordFlow boosts performance and enables additional practical video tasks.

We develop a framework to quantify predictive uncertainty in survival analysis, providing a reliable lower predictive bound (LPB) for the true, unknown patient survival time. Recently, conformal prediction has been used to construct such valid LPBs for *type-I right-censored data*, with the guarantee that the bound holds with high probability. Crucially, under the type-I setting, the censoring time is observed for all data points. As such, informative LPBs can be constructed by framing the calibration as an estimation task with covariate shift, relying on the conditionally independent censoring assumption. This paper expands the conformal toolbox for survival analysis, with the goal of handling the ubiquitous *general right-censored setting*, in which either the censoring or survival time is observed, but not both. The key challenge here is that the calibration cannot be directly formulated as a covariate shift problem anymore. Yet, we show how to construct LPBs with distribution-free finite-sample guarantees, under the same assumptions as conformal approaches for type-I censored data. Experiments demonstrate the informativeness and validity of our methods in simulated settings and showcase their practical utility using several real-world datasets.

When matching parts of a surface to its whole, a fundamental question arises: Which points should be included in the matching process? The issue is intensified when using isometry to measure similarity, as it requires the validation of whether distances measured between pairs of surface points should influence the matching process. The approach we propose treats surfaces as manifolds equipped with geodesic distances, and addresses the partial shape matching challenge by introducing a novel criterion to meticulously search for consistent distances between pairs of points. The new criterion explores the relation between intrinsic geodesic distances between the points, geodesic distances between the points and surface boundaries, and extrinsic distances between boundary points measured in the embedding space. It is shown to be less restrictive compared to previous measures and achieves state-of-the-art results when used as a loss function in training networks for partial shape matching.

The advent of vision-language pre-training techniques enhanced substantial progress in the development of models for image captioning. However, these models frequently produce generic captions and may omit semantically important image details. This limitation can be traced back to the image-text datasets; while their captions typically offer a general description of image content, they frequently omit salient details. Considering the magnitude of these datasets, manual reannotation is impractical, emphasizing the need for an automated approach. To address this challenge, we leverage existing captions and explore augmenting them with visual details using "frozen" vision experts including an object detector, an attribute recognizer, and an Optical Character Recognizer (OCR). Our proposed method, FuseCap, fuses the outputs of such vision experts with the original captions using a large language model (LLM), yielding comprehensive image descriptions. We automatically curate a training set of 12M image-enriched caption pairs. These pairs undergo extensive evaluation through both quantitative and qualitative analyses. Subsequently, this data is utilized to train a captioning generation BLIP-based model. This model outperforms current state-of-the-art approaches, producing more precise and detailed descriptions, demonstrating the effectiveness of the proposed data-centric approach. We release this large-scale dataset of enriched image-caption pairs for the community.

Abstract Learning functions defined on non‐flat domains, such as outer surfaces of non‐rigid shapes, is a central task in computer vision and geometry processing. Recent studies have explored the use of neural fields to represent functions like light reflections in volumetric domains and textures on curved surfaces by operating in the embedding space. Here, we choose a different line of thought and introduce a novel formulation of partial shape matching by learning a piecewise smooth function on a surface. Our method begins with pairing sparse landmarks defined on a full shape and its part, using feature similarity. Next, a neural representation is optimized to fit these landmarks, efficiently interpolating between the matched features that act as anchors. This process results in a function that accurately captures the partiality. Unlike previous methods, the proposed neural model of functions is intrinsically defined on the given curved surface, rather than the classical embedding Euclidean space. This representation is shown to be particularly well‐suited for representing piecewise smooth functions. We further extend the proposed framework to the more challenging part‐to‐part setting, where both shapes exhibit missing parts. Comprehensive experiments highlight that the proposed method effectively addresses partiality in shape matching and significantly outperforms leading state‐of‐the‐art methods in challenging benchmarks. Code is available at https://github.com/davidgip74/Learning-Partiality-with-Implicit-Intrinsic-Functions

Abstract. The choice of a proper regularization measure plays an important role in the field of image processing. One classical approach treats color images as two- dimensional surfaces embedded in a five-dimensional spatial-chromatic space. In this case, a natural regularization term arises as the image surface area. Choosing the chromatic coordinates as dominating over the spatial ones, we can think of the image spatial coordinates could as a parameterization of the image surface manifold in a three-dimensional color space. Minimizing the area of the image manifold leads to the Beltrami flow or mean curvature flow of the image surface in the three-dimensional color space, while minimizing the elastica of the image surface yields an additional interesting regularization. Recently, we proposed a color elastica model, which minimizes both the surface area and the elastica of the image manifold. In this paper, we propose to modify the color elastica and introduce two new models for color image regularization. The revised measures are motivated by the relations between the color elastica model, Euler’s elastica model, and the total variation model for gray level images. Compared to our previous color elastica model, the new models are direct extensions of Euler’s elastica model to color images. The proposed models are nonlinear and challenging to minimize. To overcome this difficulty, two operator-splitting methods are suggested. Specifically, nonlinearities are decoupled by the introduction of new vector- and matrix-valued variables. Then, the minimization problems are converted to initial value problems which are time-discretized by operator splitting. Each subproblem, after splitting, either has a closed-form solution or can be solved efficiently. The effectiveness and advantages of the proposed models are demonstrated by comprehensive experiments. The benefits of incorporating the elastica of the image surface as regularization terms compared to common alternatives are empirically validated.

Reconstruction of geometric structures from images using supervised learning suffers from limited available amount of accurate data. One type of such data is accurate real-world RGB-D images. A major challenge in acquiring such ground truth data is the accurate alignment between RGB images and the point cloud measured by a depth scanner. To overcome this difficulty, we consider a differential optimization method that aligns a colored point cloud with a given color image through iterative geometric and color matching. In the proposed framework, the optimization minimizes the photometric difference between the colors of the point cloud and the corresponding colors of the image pixels. Unlike other methods that try to reduce this photometric error, we analyze the computation of the gradient on the image plane and propose a different direct scheme. We assume that the colors produced by the geometric scanner camera and the color camera sensor are different and therefore characterized by different chromatic acquisition properties. Under these multimodal conditions, we find the transformation between the camera image and the point cloud colors. We alternately optimize for aligning the position of the point cloud and matching the different color spaces. The alignments produced by the proposed method are demonstrated on both synthetic data with quantitative evaluation and real scenes with qualitative results.

Isometric feature mapping is an established time-honored algorithm in manifold learning and non-linear dimensionality reduction. Its prominence can be attributed to the output of a coherent global low-dimensional representation of data by preserving intrinsic distances. In order to enable an efficient and more applicable isometric feature mapping, a diverse set of sophisticated advancements have been proposed to the original algorithm to incorporate important factors like sparsity of computation, conformality, topological constraints and spectral geometry. However, a significant shortcoming of most approaches is the dependence on large-scale dense-spectral decompositions and the inability to generalize to points far away from the sampling of the manifold. In this paper, we explore an unsupervised deep learning approach for computing distance-preserving maps for non-linear dimensionality reduction. We demonstrate that our framework is general enough to incorporate all previous advancements and show a significantly improved local and non-local generalization of the isometric mapping. Our approach involves training with only a few landmark points and avoids the need for population of dense matrices as well as computing their spectral decomposition.

We propose a novel multi-view camera pipeline for the reconstruction and registration of dynamic clothing. Our proposed method relies on a specifically designed pattern that allows for precise video tracking in each camera view. We triangulate the tracked points and register the cloth surface in a fine-grained geometric resolution and low localization error. Compared to state-of-the-art methods, our registration exhibits stable correspondence, tracking the same points on the deforming cloth surface along the temporal sequence. As an application, we demonstrate how the use of our registration pipeline greatly improves state-of-the-art pose-based drivable cloth models. Furthermore, we propose a novel model, Garment Avatar , for driving cloth from a dense tracking signal which is obtained from two opposing camera views. The method produces realistic reconstructions which are faithful to the actual geometry of the deforming cloth. In this setting, the user wears a garment with our custom pattern which enables our driving model to reconstruct the geometry. Our code and data are available at https://github.com/HalimiOshri/Pattern-Based-Cloth-Registration-and-Sparse-View-Animation. The released data includes our pattern and registered mesh sequences containing four different subjects and 15k frames in total.

The goal of the alignment problem is to align a (given) point cloud P = {p1, ⋯,pn} to another (observed) point cloud Q = {q1, ⋯,qn}. That is, to compute a rotation matrix R ∈ ℝ3×3and a translation vector t ∈ ℝ3that minimize the sum of paired distances between every transformed point Rpi− t, to its corresponding point qi, over every i ∈ {1, ⋯,n}. A harder version is the registration problem, where the correspondence is unknown, and the minimum is also over all possible correspondence functions from P to Q. Algorithms such as the Iterative Closest Point (ICP) and its variants were suggested for these problems, but none yield a provable non-trivial approximation for the global optimum.We prove that there always exists a "witness" set of 3 pairs in P × Q that, via novel alignment algorithm, defines a constant factor approximation (in the worst case) to this global optimum. We then provide algorithms that recover this witness set and yield the first provable constant factor approximation for the: (i) alignment problem in O(n) expected time, and (ii) registration problem in polynomial time. Such small witness sets exist for many variants including points in d-dimensional space, outlier-resistant cost functions, and different correspondence types.Extensive experimental results on real and synthetic datasets show that, in practice, our approximation constants are close to 1 and our error is up to x10 times smaller than state-of-the-art algorithms.

Models related to the Euler's elastica energy have proven to be useful for many applications including image processing. Extending elastica models to color images and multichannel data is a challenging task, as stable and consistent numerical solvers for these geometric models often involve high order derivatives. Like the single channel Euler's elastica model and the total variation models, geometric measures that involve high order derivatives could help when considering image formation models that minimize elastic properties. In the past, the Polyakov action from high energy physics has been successfully applied to color image processing. Here, we introduce an addition to the Polyakov action for color images that minimizes the color manifold curvature. The color image curvature is computed by applying the Laplace--Beltrami operator to the color image channels. When reduced to gray-scale images, while selecting appropriate scaling between space and color, the proposed model minimizes Euler's elastica operating on the image level sets. Finding a minimizer for the proposed nonlinear geometric model is a challenge we address in this paper. Specifically, we present an operator-splitting method to minimize the proposed functional. The nonlinearity is decoupled by introducing three vector-valued and matrix-valued variables. The problem is then converted into solving for the steady state of an associated initial-value problem. The initial-value problem is time split into three fractional steps, such that each subproblem has a closed form solution, or can be solved by fast algorithms. The efficiency and robustness of the proposed method are demonstrated by systematic numerical experiments.

Some face recognition methods are designed to utilize geometric information extracted from depth sensors to overcome the weaknesses of single-image based recognition technologies. However, the accurate acquisition of the depth profile is an expensive and challenging process. Here, we introduce a novel method that learns to recognize faces from stereo camera systems without the need to explicitly compute the facial surface or depth map. The raw face stereo images along with the location in the image from which the face is extracted allow the proposed CNN to improve the recognition task while avoiding the need to explicitly handle the geometric structure of the face. This way, we keep the simplicity and cost efficiency of identity authentication from a single image, while enjoying the benefits of geometric data without explicitly reconstructing it. We demonstrate that the suggested method outperforms both existing single-image and explicit depth based methods on largescale benchmarks, and even capable of recognize spoofing attacks. We also provide an ablation study that shows that the suggested method uses the face locations in the left and right images to encode informative features that improve the overall performance.

Multidimensional scaling (MDS) is a dimensionality reduction tool used for information analysis, data visualization and manifold learning. Most MDS procedures embed data points in low-dimensional euclidean (flat) domains, such that distances between the points are as close as possible to given inter-point dissimilarities. We present an efficient solver for classical scaling, a specific MDS model, by extrapolating the information provided by distances measured from a subset of the points to the remainder. The computational and space complexities of the new MDS methods are thereby reduced from quadratic to quasi-linear in the number of data points. Incorporating both local and global information about the data allows us to construct a low-rank approximation of the inter-geodesic distances between the data points. As a by-product, the proposed method allows for efficient computation of geodesic distances.

Intel® RealSense™ SR300 is a depth camera capable of providing a VGA-size depth map at 60 fps and 0.125mm depth resolution. In addition, it outputs an infrared VGA-resolution image and a 1080p color texture image at 30 fps. SR300 form-factor enables it to be integrated into small consumer products and as a front facing camera in laptops and Ultrabooks™. The SR300 depth camera is based on a coded-light technology where triangulation between projected patterns and images captured by a dedicated sensor is used to produce the depth map. Each projected line is coded by a special temporal optical code, that enables a dense depth map reconstruction from its reflection. The solid mechanical assembly of the camera allows it to stay calibrated throughout temperature and pressure changes, drops, and hits. In addition, active dynamic control maintains a calibrated depth output. An extended API LibRS released with the camera allows developers to integrate the camera in various applications. Algorithms for 3D scanning, facial analysis, hand gesture recognition, and tracking are within reach for applications using the SR300. In this paper, we describe the underlying technology, hardware, and algorithms of the SR300, as well as its calibration procedure, and outline some use cases. We believe that this paper will provide a full case study of a mass-produced depth sensing product and technology.

Many shape analysis methods treat the geometry of an object as a metric space that can be captured by the Laplace-Beltrami operator. In this paper, we propose to adapt the classical Hamiltonian operator from quantum mechanics to the field of shape analysis. To this end, we study the addition of a potential function to the Laplacian as a generator for dual spaces in which shape processing is performed. We present general optimization approaches for solving variational problems involving the basis defined by the Hamiltonian using perturbation theory for its eigenvectors. The suggested operator is shown to produce better functional spaces to operate with, as demonstrated on different shape analysis tasks.

We introduce the first completely unsupervised correspondence learning approach for deformable 3D shapes. Key to our model is the understanding that natural deformations (such as changes in pose) approximately preserve the metric structure of the surface, yielding a natural criterion to drive the learning process toward distortion-minimizing predictions. On this basis, we overcome the need for annotated data and replace it by a purely geometric criterion. The resulting learning model is class-agnostic, and is able to leverage any type of deformable geometric data for the training phase. In contrast to existing supervised approaches which specialize on the class seen at training time, we demonstrate stronger generalization as well as applicability to a variety of challenging settings. We showcase our method on a wide selection of correspondence benchmarks, where we outperform other methods in terms of accuracy, generalization, and efficiency.

Constructing fast numerical solvers for partial differential equations (PDEs) is crucial for many scientific disciplines. A leading technique for solving large-scale PDEs is using multigrid methods. At the core of a multigrid solver is the prolongation matrix, which relates between different scales of the problem. This matrix is strongly problem-dependent, and its optimal construction is critical to the efficiency of the solver. In practice, however, devising multigrid algorithms for new problems often poses formidable challenges. In this paper we propose a framework for learning multigrid solvers. Our method learns a (single) mapping from discretized PDEs to prolongation operators for a broad class of 2D diffusion problems. We train a neural network once for the entire class of PDEs, using an efficient and unsupervised loss function. Our tests demonstrate improved convergence rates compared to the widely used Black-Box multigrid scheme, suggesting that our method successfully learned rules for constructing prolongation matrices.

This paper explores a fully unsupervised deep learning approach for computing distance-preserving maps that generate low-dimensional embeddings for a certain class of manifolds. We use the Siamese configuration to train a neural network to solve the problem of least squares multidimensional scaling for generating maps that approximately preserve geodesic distances. By training with only a few landmarks, we show a significantly improved local and nonlocal generalization of the isometric mapping as compared to analogous non-parametric counterparts. Importantly, the combination of a deep-learning framework with a multidimensional scaling objective enables a numerical analysis of network architectures to aid in understanding their representation power. This provides a geometric perspective to the generalizability of deep learning.

A classical approach for surface classification is to find a compact algebraic representation for each surface that would be similar for objects within the same class and preserve dissimilarities between classes. We introduce Self Functional Maps as a novel surface representation that satisfies these properties, translating the geometric problem of surface classification into an algebraic form of classifying matrices. The proposed map transforms a given surface into a universal isometry invariant form defined by a unique matrix. The suggested representation is realized by applying the functional maps framework to map the surface into itself. The key idea is to use two different metric spaces of the same surface for which the functional map serves as a signature. Specifically, in this paper, we use the regular and the scale invariant surface laplacian operators to construct two families of eigenfunctions. The result is a matrix that encodes the interaction between the eigenfunctions resulted from two different Riemannian manifolds of the same surface. Using this representation, geometric shape similarity is converted into algebraic distances between matrices.

We present a method to match three dimensional shapes under non-isometric deformations, topology changes and partiality. We formulate the problem as matching between a set of pair-wise and point-wise descriptors, imposing a continuity prior on the mapping, and propose a projected descent optimization procedure inspired by difference of convex functions (DC) programming.

Reconstructing the detailed geometric structure of a face from a given image is a key to many computer vision and graphics applications, such as motion capture and reenactment. The reconstruction task is challenging as human faces vary extensively when considering expressions, poses, textures, and intrinsic geometries. While many approaches tackle this complexity by using additional data to reconstruct the face of a single subject, extracting facial surface from a single image remains a difficult problem. As a result, single-image based methods can usually provide only a rough estimate of the facial geometry. In contrast, we propose to leverage the power of convolutional neural networks to produce a highly detailed face reconstruction from a single image. For this purpose, we introduce an end-to-end CNN framework which derives the shape in a coarse-to-fine fashion. The proposed architecture is composed of two main blocks, a network that recovers the coarse facial geometry (CoarseNet), followed by a CNN that refines the facial features of that geometry (FineNet). The proposed networks are connected by a novel layer which renders a depth image given a mesh in 3D. Unlike object recognition and detection problems, there are no suitable datasets for training CNNs to perform face geometry reconstruction. Therefore, our training regime begins with a supervised phase, based on synthetic images, followed by an unsupervised phase that uses only unconstrained facial images. The accuracy and robustness of the proposed model is demonstrated by both qualitative and quantitative evaluation tests.

The Gromov-Hausdorff (GH) distance is traditionally used for measuring distances between metric spaces. It was adapted for non-rigid shape comparison and matching of isometric surfaces, and is defined as the minimal distortion of embedding one surface into the other, while the optimal correspondence can be described as the map that minimizes this distortion. Solving such a minimization is a hard combinatorial problem that requires precomputation and storing of all pairwise geodesic distances for the matched surfaces. A popular way for compact representation of functions on surfaces is by projecting them into the leading eigenfunctions of the Laplace-Beltrami Operator (LBO). When truncated, the basis of the LBO is known to be the optimal for representing functions with bounded gradient in a min-max sense. Methods such as Spectral-GMDS exploit this idea to simplify and efficiently approximate a minimization related to the GH distance by operating in the truncated spectral domain, and obtain state of the art results for matching of nearly isometric shapes. However, when considering only a specific set of functions on the surface, such as geodesic distances, an optimized basis could be considered as an even better alternative. Moreover, current simplifications of approximating the GH distance introduce errors due to low rank approximations and relaxations of the permutation matrices. Here, we define the geodesic distance basis, which is optimal for compact approximation of geodesic distances, in terms of Frobenius norm. We use the suggested basis to extract the Geodesic Distance Descriptor (GDD), which encodes the geodesic distances information as a linear combination of the basis functions. We then show how these ideas can be used to efficiently and accurately approximate the metric spaces matching problem with almost no loss of information. We incorporate recent methods for efficient approximation of the proposed basis and descriptor without actually computing and storing all geodesic distances. These observations are used to construct a very simple and efficient procedure for shape correspondence. Experimental results show that the GDD improves both accuracy and efficiency of state of the art shape matching procedures.

It has been recently shown that neural networks can recover the geometric structure of a face from a single given image. A common denominator of most existing face geometry reconstruction methods is the restriction of the solution space to some low-dimensional subspace. While such a model significantly simplifies the reconstruction problem, it is inherently limited in its expressiveness. As an alternative, we propose an Image-to-Image translation network that jointly maps the input image to a depth image and a facial correspondence map. This explicit pixel-based mapping can then be utilized to provide high quality reconstructions of diverse faces under extreme expressions, using a purely geometric refinement process. In the spirit of recent approaches, the network is trained only with synthetic data, and is then evaluated on “in-the-wild” facial images. Both qualitative and quantitative analyses demonstrate the accuracy and the robustness of our approach.

The L1norm has been tremendously popular in signal and image processing in the past two decades due to its sparsity-promoting properties. More recently, its generalization to non-Euclidean domains has been found useful in shape analysis applications. For example, in conjunction with the minimization of the Dirichlet energy, it was shown to produce a compactly supported quasi-harmonic orthonormal basis, dubbed as compressed manifold modes [14]. The continuous L1norm on the manifold is often replaced by the vector ℓ1norm applied to sampled functions. We show that such an approach is incorrect in the sense that it does not consistently discretize the continuous norm and warn against its sensitivity to the specific sampling. We propose two alternative discretizations resulting in an iteratively-reweighed ℓ2norm. We demonstrate the proposed strategy on the compressed modes problem, which reduces to a sequence of simple eigendecomposition problems not requiring non-convex optimization on Stiefel manifolds and producing more stable and accurate results.

Fast and robust three-dimensional reconstruction of facial geometric structure from a single image is a challenging task with numerous applications. Here, we introduce a learning-based approach for reconstructing a three-dimensional face from a single image. Recent face recovery methods rely on accurate localization of key characteristic points. In contrast, the proposed approach is based on a Convolutional-Neural-Network (CNN) which extracts the face geometry directly from its image. Although such deep architectures outperform other models in complex computer vision problems, training them properly requires a large dataset of annotated examples. In the case of three-dimensional faces, currently, there are no large volume data sets, while acquiring such big-data is a tedious task. As an alternative, we propose to generate random, yet nearly photo-realistic, facial images for which the geometric form is known. The suggested model successfully recovers facial shapes from real images, even for faces with extreme expressions and under various lighting conditions.

The introduction of consumer RGB-D scanners set off a major boost in 3D computer vision research. Yet, the precision of existing depth scanners is not accurate enough to recover fine details of a scanned object. While modern shading based depth refinement methods have been proven to work well with Lambertian objects, they break down in the presence of specularities. We present a novel shape from shading framework that addresses this issue and enhances both diffuse and specular objects' depth profiles. We take advantage of the built-in monochromatic IR projector and IR images of the RGB-D scanners and present a lighting model that accounts for the specular regions in the input image. Using this model, we reconstruct the depth map in real-time. Both quantitative tests and visual evaluations prove that the proposed method produces state of the art depth reconstruction results.

3D scanners are growing in their popularity as many new applications and products are becoming a commodity. These applications are often tethered to a computer and/or require expensive and specialized hardware. In this note we demonstrate that it is possible to achieve good 3D reconstruction on a mobile device. We describe a novel approach for mobile phone scanning which utilizes a smart-phone and a cheap laser pointer with a cylindrical lens which produces a line pattern attached to the phone using a 3D printed adapter as demonstrated in Figure 1. Our 3D reconstruction

We investigate a novel global orientation regression approach for articulated objects using a deep convolutional neural network. This is integrated with an in-plane image derotation scheme, DeROT, to tackle the problem of per-frame fingertip detection in depth images. The method reduces the complexity of learning in the space of articulated poses which is demonstrated by using two distinct state-of-the-art learning based hand pose estimation methods applied to fingertip detection. Significant classification improvements are shown over the baseline implementation. Our framework involves no tracking, kinematic constraints or explicit prior model of the articulated object in hand. To support our approach we also describe a new pipeline for high accuracy magnetic annotation and labeling of objects imaged by a depth camera.

The popularity of low-cost RGB-D scanners is increasing on a daily basis. Nevertheless, existing scanners often cannot capture subtle details in the environment. We present a novel method to enhance the depth map by fusing the intensity and depth information to create more detailed range profiles. The lighting model we use can handle natural scene illumination. It is integrated in a shape from shading like technique to improve the visual fidelity of the reconstructed object. Unlike previous efforts in this domain, the detailed geometry is calculated directly, without the need to explicitly find and integrate surface normals. In addition, the proposed method operates four orders of magnitude faster than the state of the art. Qualitative and quantitative visual and statistical evidence support the improvement in the depth obtained by the suggested method.

Multidimensional-scaling (MDS) is an information analysis tool. It involves the evaluation of distances between data points, which is a quadratic space-time problem. Then, MDS procedures find an embedding of the points in a low dimensional Euclidean (flat) domain, optimizing for the similarity of inter-points distances. We present an efficient solver for Classical Scaling (a specific MDS model) by extending the distances measured from a subset of the points to the rest, while exploiting the smoothness property of the distance functions. The smoothness is measured by the L2 norm of the Laplace-Beltrami operator applied to the unknown distance function. The Laplace Beltrami reflects the local differential relations between points, and can be computed in linear time. Classical-scaling is thereby reformulated into a quasi-linear space-time complexities procedure.

Segmenting an image into an arbitrary number of coherent regions is at the core of image understanding. Many formulations of the segmentation problem have been suggested over the past years. These formulations include, among others, axiomatic functionals, which are hard to implement and analyze, and graph-based alternatives, which impose a non-geometric metric on the problem. We propose a novel method for segmenting an image into an arbitrary number of regions using an axiomatic variational approach. The proposed method allows to incorporate various generic region appearance models, while avoiding metrication errors. In the suggested framework, the segmentation is performed by level set evolution. Yet, contrarily to most existing methods, here, multiple regions are represented by a single non-negative level set function. The level set function evolution is efficiently executed through the Voronoi Implicit Interface Method for multi-phase interface evolution. The proposed approach is shown to obtain accurate segmentation results for various natural 2D and 3D images, comparable to state-of-the-art image segmentation algorithms.

A proof of the optimality of the eigenfunctions of the Laplace--Beltrami operator (LBO) in representing smooth functions on surfaces is provided and adapted to the field of applied shape and data analysis. It is based on the Courant--Fischer min-max principle adapted to our case. The theorem we present supports the new trend in geometry processing of treating geometric structures by using their projection onto the leading eigenfunctions of the decomposition of the LBO. Utilization of this result can be used for constructing numerically efficient algorithms to process shapes in their spectrum. We review a couple of applications as possible practical usage cases of the proposed optimality criteria. We refer to a scale invariant metric, which is also invariant to bending of the manifold. This novel pseudometric allows constructing an LBO by which a scale invariant eigenspace on the surface is defined. We demonstrate the efficiency of an intermediate metric, defined as an interpolation between the scale invariant and the regular one, in representing geometric structures while capturing both coarse and fine details. Next, we review a numerical acceleration technique for classical scaling, a member of a family of flattening methods known as multidimensional scaling (MDS). There, the optimality is exploited to efficiently approximate all geodesic distances between pairs of points on a given surface and thereby match and compare between almost isometric surfaces. Finally, we revisit the classical principal component analysis (PCA) definition by coupling its variational form with a Dirichlet energy on the data manifold. By pairing the PCA with the LBO we can efficiently handle cases that go beyond the scope defined by the observation set that is handled by regular PCA.

An important operation in geometry processing is finding the correspondences between pairs of shapes. Measures of dissimilarity between surfaces, has been found to be highly useful for nonrigid shape comparison. Here, we analyze the applicability of the spectral kernel distance, for solving the shape matching problem. To align the spectral kernels, we introduce the iterative closest spectral kernel maps (ICSKM) algorithm. The ICSKM algorithm farther extends the iterative closest point algorithm to the class of deformable shapes. The proposed method achieves state-of-the-art results on the Princeton isometric shape matching protocol applied, as usual, to the TOSCA and SCAPE benchmarks.

Shape recovery based on shading variations of a lighted object was recently revisited with improvements that allow for the photometric stereo approach to serve as a competitive alternative for other shape reconstruction methods. However, most efforts of using photometric stereo tend to ignore some factors that are relevant in practical applications. The approach we consider tackles the photometric stereo reconstruction in the case of near-field imaging which means that both camera and light sources are close to the imaged object. The known challenges that characterize the problem involve perspective viewing geometry, attenuation of light and possibly missing regions. Here, we pay special attention to the question of how to faithfully model these aspects and by the same token design an efficient and robust numerical solver. We present a well-posed mathematical representation that integrates the above assumptions into a single coherent model. The surface reconstruction in our near-field scenario can then be executed efficiently in linear time. The merging strategy of the irradiance equations provided for each light source allows us to consider a characteristic expansion model which enables the direct computation of the surface. We evaluate several types of light attenuation models with nonuniform albedo and noise on synthetic data using four virtual sources. We also demonstrate the proposed method on surface reconstruction of real data using three images, each one taken with a different light source.

An efficient method for precise computation of image-aware geodesic distances for image editing algorithms is proposed. It exploits the connection between image representation as a mapping from a Cartesian grid and as a collection of its level sets, organized into a tree structure. The distance computation is reformulated in the domain of the image level sets, where it can be calculated without introducing approximation errors, which are unavoidable when working the image domain. Advantages of the proposed approach are demonstrated for image segmentation application.

Shape recovery of an object based on shading variations resulting from different light sources has recently been reconsidered. Improvements have been made that allow for the photometric stereo approach to serve as a competitive alternative to other shape reconstruction methods. However, most photometric stereo methods tend to ignore factors that are relevant in practical applications. The setup considered in this paper tackles photometric stereo reconstruction in the case of a specific near-field imaging. This means that both the camera and the light sources are close to the imaged object, where close can be loosely considered as a setup having similar distances between lights, camera, and object. The known challenges that characterize the problem involve perspective viewing geometry, point light sources, and images that may include shadowed regions. Here, we pay special attention to the question of how to faithfully model these aspects and at the same time design an efficient and robust numerical solver. We present a mathematical formulation that integrates the above assumptions into a single coherent model based on quasi-linear PDEs. The well-posedness is proved showing uniqueness of a weak (i.e., Lipschitz continuous) solution. The surface reconstruction in our near-field scenario can then be executed efficiently in linear time. The merging strategy of the irradiance equations provided for each light source allows us to consider a characteristic expansion model which enables the direct computation of the surface. We evaluate several types of light attenuation models with a nonuniform albedo and noise on synthetic data. We also demonstrate the proposed method on surface reconstruction of real data using three images, each one taken with a different light source by a working prototype. We demonstrate the accuracy of the proposed method compared to other methods that ignore the near-field setup and assume distant, parallel beam light sources.

Reconstruction of 3D objects Based on images is useful in many applications. One of the methods Based on multi-image data is the Photometric Stereo technique relying on several photographs of the observed object from the same point of view, each one taken under a different illumination condition. The common approach is to estimate the gradient field of the surface by minimizing a functional, integrating the distance from the camera and thereby obtaining the geometry of the observed object. We propose an alternative method that consists of a novel differential approach for multi-image Photometric Stereo and permits a direct solution of a novel PDE Based model without going through the gradient field while naturally dealing with shadowed regions. The mathematical well-posed ness of the problem in terms of numerical stability yields a fast algorithm that efficiently converges, even for pictures of sizes in the order of several mega pixels affected by noise.

Abstract The use of Laplacian eigenbases has been shown to be fruitful in many computer graphics applications. Today, state‐of‐the‐art approaches to shape analysis, synthesis, and correspondence rely on these natural harmonic bases that allow using classical tools from harmonic analysis on manifolds. However, many applications involving multiple shapes are obstacled by the fact that Laplacian eigenbases computed independently on different shapes are often incompatible with each other. In this paper, we propose the construction of common approximate eigenbases for multiple shapes using approximate joint diagonalization algorithms, taking as input a set of corresponding functions (e.g. indicator functions of stable regions) on the two shapes. We illustrate the benefits of the proposed approach on tasks from shape editing, pose transfer, correspondence, and similarity.

Abstract We present a new framework for point cloud denoising by patch‐collaborative spectral analysis. A collaborative generalization of each surface patch is defined, combining similar patches from the denoised surface. The Laplace–Beltrami operator of the collaborative patch is then used to selectively smooth the surface in a robust manner that can gracefully handle high levels of noise, yet preserves sharp surface features. The resulting denoising algorithm competes favourably with state‐of‐the‐art approaches, and extends patch‐based algorithms from the image processing domain to point clouds of arbitrary sampling. We demonstrate the accuracy and noise‐robustness of the proposed algorithm on standard benchmark models as well as range scans, and compare it to existing methods for point cloud denoising.

An isomorphism between two graphs is a connectivity preserving bijective mapping between their sets of vertices. Finding isomorphisms between graphs, or between a graph and itself (automorphisms), is of great importance in applied sciences. The inherent computational complexity of this problem is as yet unknown. Here, we introduce an efficient method to compute such mappings using heat kernels associated with the graph Laplacian. While the problem is combinatorial in nature, in practice we experience polynomial runtime in the number of vertices. As we demonstrate, the proposed method can handle a variety of graphs and is competitive with state-of-the-art packages on various important examples.

In nature, different animals of the same species frequently exhibit local variations in scale. New developments in shape matching research thus increasingly provide us with the tools to answer such fascinating questions as the following: How should we measure the discrepancy between a small dog with large ears and a large one with small ears? Are there geometric structures common to both an elephant and a giraffe? What is the morphometric similarity between a blue whale and a dolphin? Currently, there are only two methods that allow us to quantify similarities between surfaces which are insensitive to deformations in size: scale invariant local descriptors and global normalization methods. Here, we propose a new tool for shape exploration. We introduce a scale invariant metric for surfaces that allows us to analyze nonrigid shapes, generate locally invariant features, produce scale invariant geodesics, embed one surface into another despite changes in local and global size, and assist in the computational study of intrinsic symmetries where size is insignificant.

According to the uniformization theorem, any surface can be conformally mapped into a domain of a constant Gaussian curvature. The conformal factor indicates the local scaling introduced by such a mapping. This process could be used to compute geometric quantities in a simplified flat domain with zero Gaussian curvature. For example, the computation of geodesic distances on a curved surface can be mapped into solving an eikonal equation in a plane weighted by the conformal factor. Solving an eikonal equation on the weighted plane can then be done by regular sampling of the domain using, for example, the celebrated fast marching method (FMM). The connection between the conformal factor on the plane and the surface geometry can be justified analytically. Still, in order to construct consistent numerical solvers that exploit this relation, one needs to prove that the conformal factor is bounded. We provide theoretical bounds of the conformal factor and introduce optimization formulations that control its behavior. It is demonstrated that without such restrictions the numerical results are unboundedly inaccurate. Putting all ingredients in the right order, we introduce a method for computing geodesic distances on a two-dimensional manifold by using the FMM on a weighted flat domain. It is also shown how a metric on a curved domain can be reconstructed by reformulating the nonflat metric restoration problem into a weighted flat domain, again, with bounded weights for consistent results.

We introduce an (equi-)affine invariant diffusion geometry by which surfaces that go through squeeze and shear transformations can still be properly analyzed. The definition of an affine invariant metric enables us to construct an invariant Laplacian from which local and global geometric structures are extracted. Applications of the proposed framework demonstrate its power in generalizing and enriching the existing set of tools for shape analysis.

Detection and description of affine-invariant features is a cornerstone component in numerous computer vision applications. In this note, we analyze the notion of maximally stable extremal regions (MSERs) through the prism of the curvature scale space, and conclude that in its original definition, MSER prefers regular (round) regions. Arguing that interesting features in natural images usually have irregular shapes, we propose alternative definitions of MSER which are free of this bias, yet maintain their invariance properties.

We consider curve evolution based on comparing distributions of features, and its applications for scene segmentation. In the first part, we promote using cross-bin metrics such as the Earth Mover's Distance (EMD), instead of standard bin-wise metrics as the Bhattacharyya or Kullback-Leibler metrics. To derive flow equations for minimizing functionals involving the EMD, we employ a tractable expression for calculating EMD between one-dimensional distributions. We then apply the derived flows to various examples of single image segmentation, and to scene analysis using video data. In the latter, we consider the problem of segmenting a scene to spatial regions in which different activities occur. We use a nonparametric local representation of the regions by considering multiple one-dimensional histograms of normalized spatiotemporal derivatives. We then obtain semisupervised segmentation of regions using the flows derived in the first part of the paper. Our results are demonstrated on challenging surveillance scenes, and compare favorably with state-of-the-art results using parametric representations by dynamic systems or mixtures of them.

The Beltrami image flow is an effective nonlinear filter, often used in color image processing. It was shown to be closely related to the median, total variation, and bilateral filters. It treats the image as a two-dimensional manifold embedded in a hybrid spatial-feature space. Minimization of the image surface area yields the Beltrami flow. The corresponding diffusion operator is anisotropic and strongly couples the spectral components. Thus, there is so far no implicit or operator–splitting-based numerical scheme for the partial differential equation that describes the Beltrami flow in color. Usually, this flow is implemented by explicit schemes, which are stable only for very small time steps and therefore require many iterations. At the other end, vector extrapolation techniques accelerate the convergence of vector sequences, without explicit knowledge of the sequence generator. In this paper, we propose using vector extrapolation techniques for accelerating the convergence of the explicit schemes for the Beltrami flow. Experiments demonstrate fast convergence and efficiency compared to explicit schemes.

We present an efficient O( n ) numerical algorithm for first-order approximation of geodesic distances on geometry images, where n is the number of points on the surface. The structure of our algorithm allows efficient implementation on parallel architectures. Two implementations on a SIMD processor and on a GPU are discussed. Numerical results demonstrate up to four orders of magnitude improvement in execution time compared to the state-of-the-art algorithms.

This paper explores similarity criteria between non-rigid shapes. Broadly speaking, such criteria are divided into intrinsic and extrinsic, the first referring to the metric structure of the objects and the latter to the geometry of the shapes in the Euclidean space. Both criteria have their advantages and disadvantages; extrinsic similarity is sensitive to non-rigid deformations of the shapes, while intrinsic similarity is sensitive to topological noise. Here, we present an approach unifying both criteria in a single distance. Numerical results demonstrate the robustness of our approach in cases where using only extrinsic or intrinsic criteria fail.

Symmetry and self-similarity is the cornerstone of Nature, exhibiting itself through the shapes of natural creations and ubiquitous laws of physics. Since many natural objects are symmetric, the absence of symmetry can often be an indication of some anomaly or abnormal behavior. Therefore, detection of asymmetries is important in numerous practical applications, including crystallography, medical imaging, and face recognition, to mention a few. Conversely, the assumption of underlying shape symmetry can facilitate solutions to many problems in shape reconstruction and analysis. Traditionally, symmetries are described as extrinsic geometric properties of the shape. While being adequate for rigid shapes, such a description is inappropriate for non-rigid ones. Extrinsic symmetry can be broken as a result of shape deformations, while its intrinsic symmetry is preserved. In this paper, we pose the problem of finding intrinsic symmetries of non-rigid shapes and propose an efficient method for their computation.

Addressed here is the problem of constructing and analyzing expression-invariant representations of human faces. We demonstrate and justify experimentally a simple geometric model that allows to describe facial expressions as isometric deformations of the facial surface. The main step in the construction of expression-invariant representation of a face involves embedding of the facial intrinsic geometric structure into some low-dimensional space. We study the influence of the embedding space geometry and dimensionality choice on the representation accuracy and argue that compared to its Euclidean counterpart, spherical embedding leads to notably smaller metric distortions. We experimentally support our claim showing that a smaller embedding error leads to better recognition.

An efficient approach for face compression is introduced. Restricting a family of images to frontal facial mug shots enables us to first geometrically deform a given face into a canonical form in which the same facial features are mapped to the same spatial locations. Next, we break the image into tiles and model each image tile in a compact manner. Modeling the tile content relies on clustering the same tile location at many training images. A tree of vector-quantization dictionaries is constructed per location, and lossy compression is achieved using bit-allocation according to the significance of a tile. Repeating this modeling/coding scheme over several scales, the resulting multiscale algorithm is demonstrated to compress facial images at very low bit rates while keeping high visual qualities, outperforming JPEG-2000 performance significantly.

We introduce a short-time kernel for the Beltrami image enhancing flow. The flow is implemented by "convolving" the image with a space dependent kernel in a similar fashion to the solution of the heat equation by a convolution with a Gaussian kernel. The kernel is appropriate for smoothing regular (flat) 2-D images, for smoothing images painted on manifolds, and for simultaneously smoothing images and the manifolds they are painted on. The kernel combines the geometry of the image and that of the manifold into one metric tensor, thus enabling a natural unified approach for the manipulation of both. Additionally, the derivation of the kernel gives a better geometrical understanding of the Beltrami flow and shows that the bilateral filter is a Euclidean approximation of it. On a practical level, the use of the kernel allows arbitrarily large time steps as opposed to the existing explicit numerical schemes for the Beltrami flow. In addition, the kernel works with equal ease on regular 2-D images and on images painted on parametric or triangulated manifolds. We demonstrate the denoising properties of the kernel by applying it to various types of images and manifolds.

The experience of motion sickness in a virtual environment may be measured through pre and postexperiment self-reported questionnaires such as the Simulator Sickness Questionnaire (SSQ). Although research provides converging evidence that users of virtual environments can experience motion sickness, there have been no controlled studies to determine to what extent the user's subjective response is a demand characteristic resulting from pre and posttest measures. In this study, subjects were given either SSQ's both pre and postvirtual environment immersion, or only postimmersion. This technique tested for contrast effects due to demand characteristics in which administration of the questionnaire itself suggested to the participant that the virtual environment may produce motion sickness. Results indicate that reports of motion sickness after immersion in a virtual environment are much greater when both pre and postquestionnaires are given than when only a posttest questionnaire is used. The implications for assessments of motion sickness in virtual environments are discussed.

We present a new segmentation method for extracting thin structures embedded in three-dimensional medical images based on modern variational principles. We demonstrate the importance of the edge alignment and homogeneity terms in the segmentation of blood vessels and vascular trees. For that goal, the Chan-Vese minimal variance method is combined with the boundary alignment, and the geodesic active surface models. An efficient numerical scheme is proposed. In order to simultaneously detect a number of different objects in the image, a hierarchal approach is applied.

Recently, it was proven empirically that facial expressions can be modelled as isometries, that is, geodesic distances on the facial surface were shown to be significantly less sensitive to facial expressions compared to Euclidean ones. Based on this assumption, the 3DFACE face recognition system was built. The system efficiently computes expression invariant signatures based on isometry-invariant representation of the facial surface. One of the crucial steps in the recognition system was embedding of the face geometric structure into a Euclidean (flat) space. Here, we propose to replace the flat embedding by a spherical one to construct isometric invariant representations of the facial image. We refer to these new invariants as spherical canonical images. Compared to its Euclidean counterpart, spherical embedding leads to notably smaller metric distortion. We demonstrate experimentally that representations with lower embedding error lead to better recognition. In order to efficiently compute the invariants we introduce a dissimilarity measure between the spherical canonical images based on the spherical harmonic transform.

Gamut mapping deals with the need to adjust a color image to fit into the constrained color gamut of a given rendering medium. A typical use for this tool is the reproduction of a color image prior to its printing, such that it exploits best the given printer/medium color gamut, namely the colors the printer can produce on the given medium. Most of the classical gamut mapping methods involve a pixel-by-pixel mapping and ignore the spatial color configuration. Recently proposed spatial-dependent approaches for gamut mapping are either based on heuristic assumptions or involve a high computational cost. In this paper, we present a new variational approach for space-dependent gamut mapping. Our treatment starts with the presentation of a new measure for the problem, closely related to a recent measure proposed for Retinex. We also link our method to recent measures that attempt to couple spectral and spatial perceptual measures. It is shown that the gamut mapping problem leads to a quadratic programming formulation, guaranteed to have a unique solution if the gamut of the target device is convex. An efficient numerical solution is proposed with promising results.

We discuss the synthesis between the 3D and the 2D data in three-dimensional face recognition. We show how to compensate for the illumination and racial expressions using the 3D facial geometry and present the approach of canonical images, which allows us to incorporate geometric information into standard face recognition approaches.

Segmentation in volumetric images deals with separating 'objects' from their 'background' in a given 3D data. Usually, one starts with 'edge detectors' that give binary clues on the locations of the objects boundaries. Classical edge detectors that can be adopted from 2D are the Marr-Hildreth, and Haralick or Canny edge detectors. Next, usually one integrates these clues into meaningful contours or surfaces that indicate the boundaries of the objects. We use our recent variational explanation for the Marr-Hildreth and the Haralick-Canny like edge detectors to extend these classical operators. We combine these operators with a minimal deviation measure that can be tuned to the problem at hand. Finally, an improved 'geometric active surface model' is defined.

Isometric surfaces share the same geometric structure, also known as the "first fundamental form." For example, all possible bendings of a given surface that includes all length preserving deformations without tearing or stretching the surface are considered to be isometric. We present a method to construct a bending invariant signature for such surfaces. This invariant representation is an embedding of the geometric structure of the surface in a small dimensional Euclidean space in which geodesic distances are approximated by Euclidean ones. The bending invariant representation is constructed by first measuring the intergeodesic distances between uniformly distributed points on the surface. Next, a multidimensional scaling technique is applied to extract coordinates in a finite dimensional Euclidean space in which geodesic distances are replaced by Euclidean ones. Applying this transform to various surfaces with similar geodesic structures (first fundamental form) maps them into similar signature surfaces. We thereby translate the problem of matching nonrigid objects in various postures into a simpler problem of matching rigid objects. As an example, we show a simple surface classification method that uses our bending invariant signatures.

The most popular lossy image compression method used on the Internet is the JPEG standard. JPEG's good compression performance and low computational and memory complexity make it an attractive method for natural image compression. Nevertheless, as we go to low bit rates that imply lower quality, JPEG introduces disturbing artifacts. It is known that, at low bit rates, a down-sampled image, when JPEG compressed, visually beats the high resolution image compressed via JPEG to be represented by the same number of bits. Motivated by this idea, we show how down-sampling an image to a low resolution, then using JPEG at the lower resolution, and subsequently interpolating the result to the original resolution can improve the overall PSNR performance of the compression process. We give an analytical model and a numerical analysis of the down-sampling, compression and up-sampling process, that makes explicit the possible quality/compression trade-offs. We show that the image auto-correlation can provide a good estimate for establishing the down-sampling factor that achieves optimal performance. Given a specific budget of bits, we determine the down-sampling factor necessary to get the best possible recovered image in terms of PSNR.

We propose an efficient and deterministic algorithm for computing the one-dimensional dilation and erosion (max and min) sliding window filters. For a p-element sliding window, our algorithm computes the 1D filter using 1.5 + o(1) comparisons per sample point. Our algorithm constitutes a deterministic improvement over the best previously known such algorithm, independently developed by van Herk (1992) and by Gil and Werman (1993) (the HGW algorithm). Also, the results presented in this paper constitute an improvement over the Gevorkian et al. (1997) (GAA) variant of the HGW algorithm. The improvement over the GAA variant is also in the computation model. The GAA algorithm makes the assumption that the input is independently and identically distributed (the i.i.d. assumption), whereas our main result is deterministic. We also deal with the problem of computing the dilation and erosion filters simultaneously, as required, e.g., for computing the unbiased morphological edge. In the case of i.i.d. inputs, we show that this simultaneous computation can be done more efficiently then separately computing each. We then turn to the opening filter, defined as the application of the min filter to the max filter and give an efficient algorithm for its computation. Specifically, this algorithm is only slightly slower than the computation of just the max filter. The improved algorithms are readily generalized to two dimensions (for a rectangular window), as well as to any higher finite dimension (for a hyperbox window), with the number of comparisons per window remaining constant. For the sake of concreteness, we also make a few comments on implementation considerations in a contemporary programming language.

A voxel-based method for flattening a surface in 3D space into 2D while best preserving distances is presented. Triangulation or polyhedral approximation of the voxel data are not required. The problem is divided into two main parts: Voxel-based calculation of the minimal geodesic distances between points on the surface and finding a configuration of points in 2D that has Euclidean distances as close as possible to these distances. The method suggested combines an efficient voxel-based hybrid distance estimation method, that takes the continuity of the underlying surface into account, with classical multidimensional scaling (MDS) for finding the 2D point configuration. The proposed algorithm is efficient, simple, and can be applied to surfaces that are not functions. Experimental results are shown.

An automatic cortical gray matter segmentation from a three-dimensional (3-D) brain images [magnetic resonance (MR) or computed tomography] is a well known problem in medical image processing. In this paper, we first formulate it as a geometric variational problem for propagation of two coupled bounding surfaces. An efficient numerical scheme is then used to implement the geodesic active surface model. Experimental results of cortex segmentation on real 3-D MR data are provided.

Presents a novel technique for texture mapping on arbitrary surfaces with minimal distortion by preserving the local and global structure of the texture. The recent introduction of the fast marching method on triangulated surfaces has made it possible to compute a geodesic distance map from a given surface point in O(n lg n) operations, where n is the number of triangles that represent the surface. We use this method to design a surface flattening approach based on multi-dimensional scaling (MDS). MDS is a family of methods that map a set of points into a finite-dimensional flat (Euclidean) domain, where the only data given is the corresponding distance between every pair of points. The MDS mapping yields minimal changes of the distances between the corresponding points. We then solve an "inverse" problem and map a flat texture patch onto a curved surface while preserving the structure of the texture.

The problem of binarization of gray level images acquired under nonuniform illumination is reconsidered. Yanowitz and Bruckstein (1989) proposed to use an adaptive threshold surface, determined by interpolation of the image gray levels at points where the image gradient is high. The rationale is that a high image gradient indicates probable object edges, and there the image values are between the object and background gray levels. The threshold surface was determined by successive overrelaxation as the solution of the Laplace equation. This work proposes a different method to determine an adaptive threshold surface. In this new method, inspired by multiresolution approximation, the threshold surface is constructed with considerably lower computational complexity and is smooth, yielding faster image binarizations and better visual performance.

Isometric surfaces share the same geometric structure also known as the first fundamental form. For example, bending of a given surface, that includes length preserving deformations without tearing or stretching the surface, are considered to be isometric. We present a method to construct a bending invariant canonical form for such surfaces. This invariant representation is an embedding of the intrinsic geodesic structure of the surface in a finite dimensional Euclidean space, in which geodesic distances are approximated by Euclidean ones. The canonical representation is constructed by first measuring the intergeodesic distances between points on the surfaces. Next, multi-dimensional scaling (MDS) techniques are applied to extract a finite dimensional flat space in which geodesic distances are represented as Euclidean ones. The geodesic distances are measured by the efficient fast marching on triangulated domains numerical algorithm. Applying this transform to various objects with similar geodesic structures (similar first fundamental form) maps isometric objects into similar canonical forms. We show a simple surface classification method based on the bending invariant canonical form.

We use an unconditionally stable numerical scheme to implement a fast version of the geodesic active contour model. The proposed scheme is useful for object segmentation in images, like tracking moving objects in a sequence of images. The method is based on the Weickert-Romeney-Viergever (additive operator splitting) AOS scheme. It is applied at small regions, motivated by the Adalsteinsson-Sethian level set narrow band approach, and uses Sethian's (1996) fast marching method for re-initialization. Experimental results demonstrate the power of the new method for tracking in color movies.

A simplified color image formation model is used to construct an algorithm for image reconstruction from CCD sensors samples. The proposed method involves two successive steps. The first is motivated by Cok's template matching technique, while the second step uses steerable inverse diffusion in color. Classical linear signal processing techniques tend to oversmooth the image and result in noticeable color artifacts along edges and sharp features. The question is how should the different color channels support each other to form the best possible reconstruction. Our answer is to let the edges support the color information, and the color channels support the edges, and thereby achieve better perceptual results than those that are bounded by the sampling theoretical limit.

We introduce a new geometrical framework based on which natural flows for image scale space and enhancement are presented. We consider intensity images as surfaces in the (x, I) space. The image is, thereby, a two-dimensional (2-D) surface in three-dimensional (3-D) space for gray-level images, and 2-D surfaces in five dimensions for color images. The new formulation unifies many classical schemes and algorithms via a simple scaling of the intensity contrast, and results in new and efficient schemes. Extensions to multidimensional signals become natural and lead to powerful denoising and scale space algorithms.

This paper presents a new algorithm for planning the time-optimal motion of a robot travelling with limited velocity from a given location to a given destination on a surface in the presence of moving obstacles. Additional constraints such as space variant terrain traversability and fuel economy can be accommodated. A multivalued distance map is defined and applied in computing optimal trajectories. The multivalued distance map incorporates constraints imposed by the moving obstacles, surface topography, and terrain traversability. It is generated by an efficient numerical curve propagation technique.

A general geometrical framework for image processing is presented. We consider intensity images as surfaces in the (x, I) space. The image is thereby a two dimensional surface in three dimensional space for gray level images. The new formulation unifies many classical schemes, algorithms, and measures via choices of parameters in "master" geometrical measure. More important, it is a simple and efficient tool for the design of natural schemes for image enhancement, segmentation, and scale space. Here we give the basic motivation and apply the scheme to enhance images. We present the concept of an image as a surface in dimensions higher than the three dimensional intuitive space. This will help us handle movies, color, and volumetric medical images.

We introduce a new geometrical framework for image processing. This framework finds a seamless link between the TV-L/sub 1/ and the L/sub 2/ norms that are often used in image processing, based on the geometry of the image and its interpretation as a surface. It unifies most of the current "scale space" models for images by a simple selection of one parameter, yet more important, it enables one to introduce new methods to deal with images in a simple and natural way. A functional called "Polyakov (1981) action", borrowed from high energy physics, is shown to be useful for image enhancement in color, texture, volumetric medical data, movies, and more.

A geometric approach for 3D object segmentation and representation is presented. The segmentation is obtained by deformable surfaces moving towards the objects to be detected in the 3D image. The model is based on curvature motion and the computation of surfaces with minimal areas, better known as minimal surfaces. The space where the surfaces are computed is induced from the 3D image (volumetric data) in which the objects are to be detected. The model links between classical deformable surfaces obtained via energy minimization, and intrinsic ones derived from curvature based flows. The new approach is stable, robust, and automatically handles changes in the surface topology during the deformation.

A new boundary detection approach for shape modeling is presented. It detects the global minimum of an active contour model's energy between two points. Initialization is made easier and the curve cannot be trapped at a local minimum by spurious edges. We modify the "snake" energy by including the internal regularization term in the external potential term. Our method is based on the interpretation of the snake as a path of minimal length in a Riemannian metric, or as a path of minimal cost. We then make use of a new efficient numerical method to find the shortest path which is the global minimum of the energy among all paths joining the two end points. The method is extended to closed contours, given only one point on the objects boundary by using a topology-based saddle search routine. We show examples of our method applied to real aerial and medical images.

A new approach of edge integration for shape modeling is presented. It is used to find the global minimum of an active contour model's energy between two points. Initialization is made easier and the curve is not trapped at a local minimum by spurious edges. We modify the "snake" energy by including the internal regularization term in the external potential term. Our method is based on the interpretation of the snake as a path of minimal length in a Riemannian metric, or as a path of minimal weighted distance. We then make use of a new numerical method to find the shortest path which is the global minimum of the energy among all paths joining the two endpoints. We show examples of our method applied to real aerial and medical images.

A framework for generating differential affine invariant signatures based on the gray level images of planar shapes is introduced. Invariant signatures and their corresponding arclengths are computed for planar shapes. These signatures are useful for pattern recognition and classification under partial occlusion. We deal with implementable signatures, which practically means using up to second order derivatives. An approximation of the affine curvature signature is introduced. In this case the Euclidean curvature is used for generating the affine arclength. Both curvatures are computed from the gray level image, using the implicit representation of the object's boundary as it appears in the image. We also present robust signatures when 'projection invariance' of the gray levels is assumed. An invariant gradient magnitude along the geometric scale space is defined and used as an invariant edge enhancer. The geometric heat equation for weighted (by the enhancer) affine arclength definition is shown to yield an invariant denoising algorithm. It is used to clean noisy images before computing invariant features. The denoising operation deforms the geometry of the object in a predictable invariant way, unlike traditional image denoising algorithms, so that the mapping between planar shapes after the denoising is preserved.

A local smoothing operator applied in the reverse direction is used to obtain planar shape enhancement and exaggeration. Inversion of a smoothing operator is an inherently unstable operation. Therefore, a stable numerical scheme simulating the inverse smoothing effect is introduced. Enhancement is obtained for short time spans of evolution. Carrying the evolution further yields shape exaggeration or caricaturization effect. Introducing attraction forces between the evolving shape and the initial one, yields an enhancement process that converges to a steady state. These forces depend on the distance of the evolving curve from the original one and on local properties. Results of applying the unrestrained and restrained evolution on planar shapes, based on a stabilized inverse geometric heat equation, are presented showing enhancement and caricaturization effects.

Finding the shortest path between points on a surface is a challenging global optimization problem. It is difficult to devise an algorithm that is computationally efficient, locally accurate and guarantees to converge to the globally shortest path. In this paper a two stage coarse-to-fine approach for finding the shortest paths is suggested. In the first stage the algorithm of Ref. 10 that combines a 3D length estimator with graph search is used to rapidly obtain an approximation to the globally shortest path. In the second stage the approximation is refined to become a shorter geodesic curve, i.e., a locally optimal path. This is achieved by using an algorithm that deforms an arbitrary initial curve ending at two given surface points via geodesic curvature shortening flow. The 3D curve shortening flow is transformed into an equivalent 2D one that is implemented using an efficient numerical algorithm for curve evolution with fixed end points, introduced in Ref. 9.

A novel scheme for the detection of object boundaries is presented. The technique is based on active contours deforming according to intrinsic geometric measures of the image. The evolving contours naturally split and merge, allowing the simultaneous detection of several objects and both interior and exterior boundaries. The proposed approach is based on the relation between active contours and the computation of geodesics or minimal distance curves. The minimal distance curve lays in a Riemannian space whose metric as defined by the image content. This geodesic approach for object segmentation allows to connect classical "snakes" based on energy minimization and geometric active contours based on the theory of curve evolution. Previous models of geometric active contours are improved as showed by a number of examples. Formal results concerning existence, uniqueness, stability, and correctness of the evolution are presented as well.>

We present a new algorithm for determining minimal length paths between two regions on a three dimensional surface. The numerical implementation is based on finding equal geodesic distance contours from a given area. These contours are calculated as zero sets of a bivariate function designed to evolve so as to track the equal distance curves on the given surface. The algorithm produces all paths of minimal length between the source and destination areas on the surface given as height values on a rectangular grid.>

A new approach for the reconstruction of a smooth three dimensional object from its two dimensional gray level image is presented. An algorithm based on topological properties of simple smooth surfaces is provided to solve the problem of global reconstruction. Classifying singular points in the shading image as maxima minima and two kinds of saddle points, serves as the key to the solution of the problem. This classification is performed globally with no assumptions on the local behavior of characteristics near singular points. The global reconstruction procedure, being deterministic and using topological properties of the surface performs better than other approaches proposed so far, based on classification of singular points according to the local behavior of characteristics in their neighborhood. The proposed algorithm is simple, easy to implement and works remarkably fast on a parallel machine.

This paper presents a new algorithm for planning the time-optimal motion of a robot traveling with limited velocity from a given location to a given destination on a surface in the presence of moving obstacles. Additional constraints such as space variant terrain traversability and fuel economy can be accommodated. A multilayer distance map is defined and applied in computing optimal trajectories. The multilayer distance map incorporates constraints imposed by the moving obstacles, surface topography and terrain traversability. It is generated by an efficient numerical curve propagation technique.