We present a method to estimate an HDR environment map from a narrow field-of-view LDR camera image in real-time. This enables perceptually appealing reflections and shading on virtual objects of any material finish, from mirror to diffuse, rendered into a real environment using augmented reality. Our method is based on our efficient convolutional neural network, EnvMapNet, trained end-to-end with two novel losses, ProjectionLoss for the generated image, and ClusterLoss for adversarial training. Through qualitative and quantitative comparison to state-of-the-art methods, we demonstrate that our algorithm reduces the directional error of estimated light sources by more than 50%, and achieves 3.7 times lower Frechet Inception Distance (FID). We further showcase a mobile application that is able to run our neural network model in under 9 ms on an iPhone XS, and render in real-time, visually coherent virtual objects in previously unseen real-world environments.

Images form a rich information source, which remains underutilized in biomedical document classification. We present here work that uses both image- and text-based features in order to identify articles of interest, in this case, pertaining to cis-regulatory modules in the context of gene-networks. Extending on our new idea, which we have recently introduced, of using OCR-based features to identify DNA contents in images, we combine image and text based classifiers to categorize documents as relevant or irrelevant to cis-regulatory modules. Using a set of hundreds of articles, marked by experts as relevant or irrelevant to cis-regulatory modules, we train/test image and text based classifiers, as well as classifiers integrating both. Our results indicate that the latter show the best performance with Recall, F-measure and Utility measures all above 0.9, demonstrating the significance of incorporating image data, and specifically OCR-based features, into the document categorization process. Moreover, the use of character distribution properties to represent images is directly relevant to other biomedical images containing text (e.g. RNA, proteins). Diagrams and other images containing text are also prevalent outside the biomedical domain, hence the work stands to be applicable and beneficial in other application areas.

In this work, we combine the complementary depth sensors Kinect and stereo image matching to obtain high quality correspondences. Our goal is to obtain a dense disparity map at the spatial and depth resolution of the stereo cameras (4-12 MP). We propose a global optimization scheme, where both the data and smoothness costs are derived using sensor confidences and low resolution geometry from Kinect. A spatially varying search range is used to limit the number of potential disparities at each pixel. The smoothness prior is Based on available low resolution depth from Kinect rather than image gradients, thus performing better in both textured areas with smooth depth and texture-less areas with depth gradient. We also propose a spatially varying smoothness weight to better handle occlusion areas, and the relative contribution of the two energy terms. We demonstrate how the two sensors can be effectively fused to obtain correct scene depth in ambiguous areas, as well as fine structural details in textured areas.

Autonomous systems which learn and utilize a limited visual vocabulary have wide spread applications. Enabling such systems to segment a set of cluttered scenes into objects is a challenging vision problem owing to the non-homogeneous texture of objects and the random configurations of multiple objects in each scene. We present a solution to the following question: given a collection of images where each object appears in one or more images and multiple objects occur in each image, how best can we extract the boundaries of the different objects? The algorithm is presented with a set of stereo images, with one stereo pair per scene. The novelty of our work is the use of both color/texture and structure to refine previously determined object boundaries to achieve segmentation consistent with each of the input scenes presented. The algorithm populates an object library, which consists of a 3D model per object. Since an object is characterized both by texture and structure, for most purposes this representation is both complete and concise.

Computer vision techniques have been applied for rapid and accurate structure recovery in many fields. Most methods perform poorly in areas containing little or no texture and in presence of repetitive patterns. We present a portable, cost-effective pattern projector system powered by the flash of a camera, to aid the reconstruction of such areas. No calibration is required between the camera-projector, projector-scene or pattern. We demonstrate the effectiveness of our system on various representative surfaces like metal, clay, porcelain and natural fibres with different inherent colors/textures. A pipeline is presented to automatically generate textured, true scale metric models, which can be used for quantitative studies or visualization. The practability of our system is explored in the specific area of digital archiving of historically significant objects.

Stereo algorithms for structure reconstruction demand accurate disparities with low mismatch errors and false positives. Mismatch errors in large textureless regions force most accurate algorithms to be sparse, with disparities known only in textured regions. We propose a novel method which uses characteristics of the multi-valued disparity to segregate image regions into unambiguous, occluded but textured and regions with low color variation. The disparity in the unambiguous region is calculated using stable matching with local disparity filtering. The disparity is interpolated into other regions by diffusion using unstructured triangulation and method of finite elements for rapid convergence. The boundary conditions for each of the region are appropriately modified so that accurate discontinuities in the disparity are preserved. A comparison of our method with some existing methods through experiments reveal that this algorithm indeed performs significantly better in producing dense accurate disparities.