High-performance dense stereo is a critical component of computer vision applications like 3D reconstruction, robot navigation, and augmented reality. In this paper, we present a low-power, high performance FPGA implementation of a stereo algorithm suitable for embedded real-time platforms. The design is scalable for higher resolution images and frame rates and supporting different cameras and application requirements. We achieve this by designing highly parallel computation cores with very efficient memory access to the image data. Using a prototype board, we demonstrate real-time stereo processing with 640×480 pixel GigE Vision cameras at 30 frames per second. We show that this FPGA design is 10 times lower power, more scalable and has lower latency, as compared to a GPU based implementation of the same stereo algorithm.

We present a novel computationally efficient approach to obstacle detection that is applicable to both structured (e.g. indoor, road) and unstructured (e.g. off-road, grassy terrain) environments. In contrast to previous works that attempt to explicitly identify obstacles, we explicitly detect scene regions that are traversable - safe for the robot to go to - from its current position. Traversability is defined on a 2D grid of cells. Given 3D points, we map them to individual cells and compute histograms of elevations of the points in each cell. This elevation information is then used in a graph based algorithm to label all traversable cells. In this manner, positive and negative obstacles, as well as unknown regions are implicitly detected and avoided. Our notion of traversability does not make any flat-world assumptions and does not need sensor pitch-roll compensation. It also accounts for overhanging structures like tree branches. We demonstrate that our approach can be used with both lidar and stereo sensors even though the two sensors differ in their resolution and accuracy. We present several results from our real-time implementation on realistic environments using both lidar and stereo.

The range of scene depths that appear focused in an image is known as the depth of field (DOF). Conventional cameras are limited by a fundamental trade-off between depth of field and signal-to-noise ratio (SNR). For a dark scene, the aperture of the lens must be opened up to maintain SNR, which causes the DOF to reduce. Also, today's cameras have DOFs that correspond to a single slab that is perpendicular to the optical axis. In this paper, we present an imaging system that enables one to control the DOF in new and powerful ways. Our approach is to vary the position and/or orientation of the image detector during the integration time of a single photograph. Even when the detector motion is very small (tens of microns), a large range of scene depths (several meters) is captured, both in and out of focus. Our prototype camera uses a micro-actuator to translate the detector along the optical axis during image integration. Using this device, we demonstrate four applications of flexible DOF. First, we describe extended DOF where a large depth range is captured with a very wide aperture (low noise) but with nearly depth-independent defocus blur. Deconvolving a captured image with a single blur kernel gives an image with extended DOF and high SNR. Next, we show the capture of images with discontinuous DOFs. For instance, near and far objects can be imaged with sharpness, while objects in between are severely blurred. Third, we show that our camera can capture images with tilted DOFs (Scheimpflug imaging) without tilting the image detector. Finally, we demonstrate how our camera can be used to realize nonplanar DOFs. We believe flexible DOF imaging can open a new creative dimension in photography and lead to new capabilities in scientific imaging, vision, and graphics.

The field of view of a traditional camera has a fixed shape. This severely restricts how scene elements can be composed into an image. We present a novel imaging system that uses a flexible mirror in conjunction with a camera to overcome this limitation. By deforming the mirror, our system can produce fields of view with a wide range of shapes and sizes. A captured image is typically a multi-perspective view of the scene with spatially varying resolution. As a result, scene objects appear distorted. To minimize these distortions, we have developed an efficient algorithm that maps a captured image to one with almost uniform resolution. To determine this mapping we need to know the shape of the mirror. For this, we have developed a simple calibration method that automatically estimates the mirror shape from its boundary, which is visible in the captured image. We present a number of examples that demonstrate that a flexible field of view imaging system can be used to compose scenes in ways that have not been possible before. This flexibility can be exploited in applications such as video surveillance and monitoring.

In this paper, we present a class of imaging systems, called radial imaging systems , that capture a scene from a large number of view-points within a single image, using a camera and a curved mirror. These systems can recover scene properties such as geometry, reflectance, and texture. We derive analytic expressions that describe the properties of a complete family of radial imaging systems, including their loci of viewpoints, fields of view, and resolution characteristics. We have built radial imaging systems that, from a single image, recover the frontal 3D structure of an object, generate the complete texture map of a convex object, and estimate the parameters of an analytic BRDF model for an isotropic material. In addition, one of our systems can recover the complete geometry of a convex object by capturing only two images. These results show that radial imaging systems are simple, effective, and convenient devices for a wide range of applications in computer graphics and computer vision.

Many events are captured using multiple cameras today. Frames of each video stream have to be synchronized and aligned to a common time axis before processing them. Synchronization of the video streams necessarily needs a hardware based solution that is applied while capturing. The alignment problem between the frames of multiple videos can be posed as a search using traditional measures for image similarity. Multiview relations and constraints developed in Computer Vision recently can provide more elegant solutions to this problem. In this paper, we provide two solutions for the video frame alignment problem using two view and three view constraints. We present solutions to this problem for the case when the videos are taken using affine cameras and for general projective cameras. Excellent experimental results are achieved by our algorithms.