In this paper we present a novel approach to object recognition based on image localization and registration applied to the problem of multi-modal interaction in smart-home environments. Typically such environments contain multiple small devices which need to be controlled from a distance. Thus, a major problem in recognizing a specific object is its small size in the image compounded by typically cluttered backgrounds. We therefore resort to recognizing an intended object by first registering the acquired image within the panorama. An environment map is used to recognize potential objects within the userpsilas field of view. Experimental results of using such a multi-modal input system on a running smart home environment are presented, where the benefits of combining visual and verbal inputs are demonstrated.

Catadioptric imaging systems consisting of conventional lens based cameras and mirrors are widely used in many vision applications. Most of the prior work in this area has focused either on aspects of mirror design or the application it is being used for. In contrast, little attention has been paid to the aspect of sharp image formation. In this paper we study how reflections in curved mirrors affect image formation from the perspective of lens focus. In particular we answer the question "How and where must the lens focus when imaging reflections in curved mirrors?". Using caustics to model the reflections, we show that the space of infinite scene depth is compressed within a finite volume which we call the "caustic volume". Analysis of the caustic volume is presented for common catadioptric systems. We also present a framework to derive the "optimal" focal plane at which to focus the lens for sharp image formation along with experimental verification.

This paper concerns the problem of actively searching for and localizing ground features by a coordinated team of air and ground robotic sensor platforms. The approach taken builds on well known Decentralized Data Fusion (DDF) methodology. In particular, it brings together established representations developed for identification and linearized estimation problems to jointly address feature detection and localization. This provides transparent and scalable integration of sensor information from air and ground platforms. As in previous studies, an Information-theoretic utility measure and local control strategy drive the robots to uncertainty reducing team configurations. Complementary characteristics in terms of coverage and accuracy are revealed through analysis of the observation uncertainty for air and ground on-board cameras. Implementation results for a detection and localization example indicate the ability of this approach to scalably and efficiently realize such collaborative potential.

A framework for analyzing distortions in non-single viewpoint imaging systems is presented. Such systems possess loci of viewpoints called caustics. In general, perspective (or undistorted) views cannot be computed from images acquired with such systems without knowing scene structure. Views computed without scene structure will exhibit distortions, which we call caustic distortions. We first introduce a taxonomy of distortions based on the geometry of imaging systems. Then, we derive a metric to quantify caustic distortions. We present an algorithm to compute minimally distorted views using simple priors on scene structure. These priors are defined as parameterized primitives such as spheres, planes and cylinders with simple uncertainty models for the parameters. To validate our method, we conducted extensive experiments on rendered and real images. In all cases our method produces nearly undistorted views even though the acquired images were strongly distorted. We also provide an approximation of the above method that warps the entire captured image into a quasi single viewpoint representation that can be used by any "viewer" to compute near-perspective views in real-time.

Conventional vision systems and algorithms assume the camera to have a single viewpoint. However, sensors need not always maintain a single viewpoint. For instance, an incorrectly aligned system could cause non-single viewpoints. Also, systems could be designed to specifically deviate from a single viewpoint to trade-off image characteristics such as resolution and field of view. In these cases, the locus of viewpoints forms what is called a caustic. In this paper, we present an in-depth analysis of caustics of catadioptric cameras with conic reflectors. Properties of caustics with respect to field of view and resolution are presented. Finally, we present ways to calibrate conic catadioptric systems and estimate their caustics from known camera motion.

Images taken with wide-angle cameras tend to have severe distortions which pull points towards the optical center. This paper proposes a simple method for recovering the distortion parameters without the use of any calibration objects. Since distortions cause straight lines in the scene to appear as curves in the image, our algorithm seeks to find the distortion parameters that map the image curves to straight lines. The user selects a small set of points along the image curves. Recovery of the distortion parameters is formulated as the minimization of an objective function which is designed to explicitly account for noise in the selected image points. Experimental results are presented for synthetic data as well as real images. We also present the idea of a polycamera which is defined as a tightly packed camera cluster. Possible configurations are proposed to capture very large fields of view. Such camera clusters tend to have a nonsingle viewpoint. We therefore provide analysis of what we call the minimum working distance for such clusters. Finally, we present results for a polycamera consisting of four wide-angle sensors having a minimum working distance of about 4 m. On undistorting the acquired images using our proposed technique, we create real-time high resolution panoramas.

Images taken with wide-angle cameras tend to have severe distortions which pull points towards the optical center. This paper proposes a method for recovering the distortion parameters without the use of any calibration objects. The distortions cause straight lines in the scene to appear as curves in the image. Our algorithm seeks to find the distortion parameters that would map the image curves to straight lines. The user selects a small set of points along the image curves. Recovery of the parameters is formulated as the minimization of an objective function which is designed to explicitly account for noise in the selected image points. Experimental results are presented for synthetic data with different noise levels as well as for real images. Once calibrated, the image streams from these cameras can be undistorted in real time using look up tables. We also present an application of this calibration method for wide-angle camera clusters, which we call polycameras. We apply our distortion correction technique to a polycamera with four wide-angle cameras to create a high resolution 360 degree panorama in real-time.

Conventional video cameras have limited fields of view which make them restrictive in a variety of applications. A catadioptric sensor uses a combination of lenses and mirrors placed in a carefully arranged configuration to capture a much wider field of view. At Columbia University, we have developed a wide range of catadioptric sensors. Some of these sensors have been designed to produce unusually large fields of view. Others have been constructed for the purpose of depth computation. All our sensors perform in real time using just a PC.