The introduction of consumer RGB-D scanners set off a major boost in 3D computer vision research. Yet, the precision of existing depth scanners is not accurate enough to recover fine details of a scanned object. While modern shading based depth refinement methods have been proven to work well with Lambertian objects, they break down in the presence of specularities. We present a novel shape from shading framework that addresses this issue and enhances both diffuse and specular objects' depth profiles. We take advantage of the built-in monochromatic IR projector and IR images of the RGB-D scanners and present a lighting model that accounts for the specular regions in the input image. Using this model, we reconstruct the depth map in real-time. Both quantitative tests and visual evaluations prove that the proposed method produces state of the art depth reconstruction results.

3D scanners are growing in their popularity as many new applications and products are becoming a commodity. These applications are often tethered to a computer and/or require expensive and specialized hardware. In this note we demonstrate that it is possible to achieve good 3D reconstruction on a mobile device. We describe a novel approach for mobile phone scanning which utilizes a smart-phone and a cheap laser pointer with a cylindrical lens which produces a line pattern attached to the phone using a 3D printed adapter as demonstrated in Figure 1. Our 3D reconstruction

We investigate a novel global orientation regression approach for articulated objects using a deep convolutional neural network. This is integrated with an in-plane image derotation scheme, DeROT, to tackle the problem of per-frame fingertip detection in depth images. The method reduces the complexity of learning in the space of articulated poses which is demonstrated by using two distinct state-of-the-art learning based hand pose estimation methods applied to fingertip detection. Significant classification improvements are shown over the baseline implementation. Our framework involves no tracking, kinematic constraints or explicit prior model of the articulated object in hand. To support our approach we also describe a new pipeline for high accuracy magnetic annotation and labeling of objects imaged by a depth camera.

The popularity of low-cost RGB-D scanners is increasing on a daily basis. Nevertheless, existing scanners often cannot capture subtle details in the environment. We present a novel method to enhance the depth map by fusing the intensity and depth information to create more detailed range profiles. The lighting model we use can handle natural scene illumination. It is integrated in a shape from shading like technique to improve the visual fidelity of the reconstructed object. Unlike previous efforts in this domain, the detailed geometry is calculated directly, without the need to explicitly find and integrate surface normals. In addition, the proposed method operates four orders of magnitude faster than the state of the art. Qualitative and quantitative visual and statistical evidence support the improvement in the depth obtained by the suggested method.

Shape recovery based on shading variations of a lighted object was recently revisited with improvements that allow for the photometric stereo approach to serve as a competitive alternative for other shape reconstruction methods. However, most efforts of using photometric stereo tend to ignore some factors that are relevant in practical applications. The approach we consider tackles the photometric stereo reconstruction in the case of near-field imaging which means that both camera and light sources are close to the imaged object. The known challenges that characterize the problem involve perspective viewing geometry, attenuation of light and possibly missing regions. Here, we pay special attention to the question of how to faithfully model these aspects and by the same token design an efficient and robust numerical solver. We present a well-posed mathematical representation that integrates the above assumptions into a single coherent model. The surface reconstruction in our near-field scenario can then be executed efficiently in linear time. The merging strategy of the irradiance equations provided for each light source allows us to consider a characteristic expansion model which enables the direct computation of the surface. We evaluate several types of light attenuation models with nonuniform albedo and noise on synthetic data using four virtual sources. We also demonstrate the proposed method on surface reconstruction of real data using three images, each one taken with a different light source.

Shape from shading and photometric stereo are two fundamental problems in computer vision aimed at reconstructing surface depth given either a single image taken under a known light source or multiple images taken under different illuminations from the same viewing angle. Whereas the former uses partial differential equation techniques to solve the image irradiance equation, the latter can be expressed as a linear system of equations in surface derivatives when three or more images are given. Therefore, it seems that current photometric stereo techniques do not extract all possible depth information from each image by itself. Extending our previous results on this problem, we consider the more realistic perspective projection of surfaces during the photographic process. Under this assumption, there is a unique weak solution (Lipschitz continuous) to the problem at hand, solving the well-known convex/concave ambiguity of the shape from shading problem. The main contribution of this paper is based on a new differential approach for multi-image photometric stereo. Most of the existing works on this topic do not directly address this problem. The common approach is to estimate the gradient field of the surface by minimizing some functional and integrate it afterwards to find the depth and hence the geometry of the object. Our new differential approach allows us to solve the problem directly, while dealing with images having missing parts. The mathematical well-posedness of the new formulation allows a fast numerical algorithm based on a combination of fast marching and fast sweeping methods.

Shape recovery of an object based on shading variations resulting from different light sources has recently been reconsidered. Improvements have been made that allow for the photometric stereo approach to serve as a competitive alternative to other shape reconstruction methods. However, most photometric stereo methods tend to ignore factors that are relevant in practical applications. The setup considered in this paper tackles photometric stereo reconstruction in the case of a specific near-field imaging. This means that both the camera and the light sources are close to the imaged object, where close can be loosely considered as a setup having similar distances between lights, camera, and object. The known challenges that characterize the problem involve perspective viewing geometry, point light sources, and images that may include shadowed regions. Here, we pay special attention to the question of how to faithfully model these aspects and at the same time design an efficient and robust numerical solver. We present a mathematical formulation that integrates the above assumptions into a single coherent model based on quasi-linear PDEs. The well-posedness is proved showing uniqueness of a weak (i.e., Lipschitz continuous) solution. The surface reconstruction in our near-field scenario can then be executed efficiently in linear time. The merging strategy of the irradiance equations provided for each light source allows us to consider a characteristic expansion model which enables the direct computation of the surface. We evaluate several types of light attenuation models with a nonuniform albedo and noise on synthetic data. We also demonstrate the proposed method on surface reconstruction of real data using three images, each one taken with a different light source by a working prototype. We demonstrate the accuracy of the proposed method compared to other methods that ignore the near-field setup and assume distant, parallel beam light sources.

Reconstruction of 3D objects Based on images is useful in many applications. One of the methods Based on multi-image data is the Photometric Stereo technique relying on several photographs of the observed object from the same point of view, each one taken under a different illumination condition. The common approach is to estimate the gradient field of the surface by minimizing a functional, integrating the distance from the camera and thereby obtaining the geometry of the observed object. We propose an alternative method that consists of a novel differential approach for multi-image Photometric Stereo and permits a direct solution of a novel PDE Based model without going through the gradient field while naturally dealing with shadowed regions. The mathematical well-posed ness of the problem in terms of numerical stability yields a fast algorithm that efficiently converges, even for pictures of sizes in the order of several mega pixels affected by noise.