Most state-of-the-art optical see-through head-mounted display (OST-HMD) lacks mutual occlusion capability between computer-rendered and real objects so that the virtual view through an OST-HMD appears “ghost-like”, floating in the real world. In this paper, we demonstrated a light-weight, compact OST-HMD with mutual occlusion capability by exploring a highly innovative optical approach based on emerging freeform optical design and fabrication technologies. Our approach enabled us to achieve an occlusion-capable OST-HMD system with a very compelling form factor and high optical performance. The proposed display technology is designed for highly efficient liquid crystal on silicon (LCoS) type spatial light modulator (SLM) and bright Organic LED (OLED) microdisplay, which is capable of working in both indoor and outdoor environments. Our current design offered a 1280×1024 color resolution with a field of view (FOV) of 40 degrees and lightweight optics about 30 grams per eye.

In the paper, we describe an optical system which is capable of providing external access to both the sensor and the lens aperture (i.e., projection center) of a conventional camera. The proposed optical system is attached in front of the camera, and is the equivalent of adding externally accessible intermediate image plane and projection center. The system offers controls of the response of each pixel which could be used to realize many added imaging functions, such as high dynamic range imaging, image modulation, and optical computation. The ability to access the optical center could enable a wide variety of applications by simply allowing manipulation of the geometric properties of the optical center. For instance, panoramic imaging can be implemented by rotating a planar mirror about the camera axis; and small base-line stereo can be implemented by shifting the camera center. We have implemented a bench setup to demonstrate some of these functions. The experimental results are included.

Wide field of view (FOV) and high-resolution image acquisition is highly desirable in many vision-based applications. Several systems have reported the use of reflections off mirror pyramids to capture high-resolution, single-viewpoint, and wide-FOV images. Using a dual mirror pyramid (DMP) panoramic camera as an example, in this paper, we examine how the pyramid geometry, and the selection and placement of imager clusters can be optimized to maximize the overall panoramic FOV, sensor utilization efficiency, and image uniformity. The analysis can be generalized and applied to other pyramid-based designs.

In augmented reality (AR) applications, accurately registering a virtual object with its real counterpart is a challenging problem. The size, depth, and geometry, as well as the physical attributes of the virtual object, have to be rendered precisely relative to a physical reference. This paper presents a systematic calibration process to address the registration challenge in a custom-designed AR system, which is based upon recent head-mounted projective display (HMPD) technology. Following a concise review of the HMPD concept and our system configuration, we first present a computational model of the HMPD viewing system and requirements for the system calibration. Then, we describe, in detail, the calibration procedures to obtain estimates of unknown transformations, summarize the application of the estimates in a customized graphics rendering toolkit, and discuss the evaluation experiments and observations. Finally, the implementation of a testbed to demonstrate a successful registration is briefly described, and experimental results are presented

We propose a novel depth sensing system composed of a single camera, and a transparent plate which is placed in front of the camera and rotates about the optical axis of the camera. The camera takes a sequence of images as the plate rotates, which provide the equivalent of a large number of stereo pairs. Compared with conventional multi-camera stereo systems, the use of a single camera for capturing stereo pairs helps improve the accuracy of detecting correspondences. The availability of the large number of stereo pairs also reduces matching ambiguities, even for objects with low texture. By using both the images and the estimated depth map, we show that the proposed system is also capable of generating super-resolution images. Experimental results on reconstructing 3D structures and recovering highresolution images are presented.

The lack of image brightness is a common problem in optical see-through head-mounted displays (OST-HMD) where a beamsplitter is required to combine views from HMD image source and the direct-view of a real world scene. This problem is further aggregated in a head-mounted projective display (HMPD) due to the fact that light passes through the beamsplitter multiple times. We present a novel design of an ultra-bright polarized head-mounted projective display (p-HMPD). The image brightness observed by a viewer theoretically is four-times brighter than existing designs. We further demonstrate a design with currently available technology that leads to a display in which the observed image is significantly brighter than existing designs. Finally, experimental results from a bench setup are presented.

We have previously developed a collaborative infrastructure called SCAPE - an acronym for Stereoscopic Collaboration in Augmented and Projective Environments - that integrates the traditionally separate paradigms of virtual and augmented reality. In this paper, we extend SCAPE by formalizing its underlying mathematical framework and detailing three augmented Widgets constructed via this framework: CoCylinder, Magnifier, and CoCube. These devices promote intuitive ways of selecting, examining, and sharing synthetic objects, and retrieving associated documentary text. Finally we present a testbed application to showcase SCAPE's capabilities for interaction in large, augmented virtual environments.

Augmented reality (AR) superimposes computer-generated virtual images on the real world to allow users exploring both virtual and real worlds simultaneously. For a successful augmented reality application, an accurate registration of a virtual object with its physical counterpart has to be achieved, which requires precise knowledge of the projection information of the viewing device. The paper proposes a fast and easy off-line calibration strategy based on well-established camera calibration methods. Our method does not need exhausting effort on the collection of world-to-image correspondence data. All the correspondence data are sampled with an image based method and they are able to achieve sub-pixel accuracy. The method is applicable for all AR systems based on optical see-through head-mounted display (HMD), though we took a head-mounted projective display (HMPD) as the example. We first review the calibration requirements for an augmented reality system and the existing calibration methods. Then a new view projection model for optical see through HMD is addressed in detail, and proposed calibration method and experimental result are presented. Finally, the evaluation experiments and error analysis are also included. The evaluation results show that our calibration method is fairly accurate and consistent.

This paper presents a multi-user collaborative infrastructure, SCAPE (an acronym for Stereoscopic Collaboration in Augmented and Projective Environments), which is based on recent advancement in head-mounted projective display (HMPD) technology. The SCAPE mainly consists of a 3'/spl times/5' interactive workbench and a 12'/spl times/12'/spl times/9' room-sized walk-through display environment, multiple head-tracked HMPDs, multi-modality interface devices, and a generic application-programming interface (API) designed to coordinate the components. The infrastructure provides a shared space in which multiple users can simultaneously interact with a 3D synthetic environment from their individual viewpoints. We detail the SCAPE implementation and include an application example that demonstrates major interface and cooperation features.

In augmented reality (AR) applications, registering a virtual object with its real counterpart accurately and comfortably is one of the basic and challenging issues in the sense that the size, depth, geometry, as well as physical attributes of the virtual objects have to be rendered precisely relative to a physical reference, which is well known as the calibration or registration problem. This paper presents a systematic calibration process to address static registration in a custom-designed augmented reality system, which is based upon the recent advancement of head-mounted projective display (HMPD) technology. Following a concise review of the HMPD concept and system configuration, we present in detail a computational model for system calibration, describe the calibration procedures to obtain estimations of the unknown transformations, and include the calibration results, evaluation experiments and results.

A head-mounted projective display (HMPD) consists of a pair of miniature projection lenses, beam splitters and displays mounted on the helmet and retro-reflective sheeting materials placed strategically in the environment. This has recently been proposed as an alternative to existing 3D visualization devices. In this paper, we first briefly review HMPD technology, including its featured capabilities and the recent development in both display implementations and applications. Then the implementation of a testbed for playing a "Go" game with a remote opponent in a 3D augmented environment is described. The testbed not only demonstrates the capabilities of virtual-real augmentation and registration, the natural occlusion of virtual objects by real ones, interaction with augmented environments and networking collaboration, but also embodies part of our long-term objective to develop a collaborative framework in 3D augmented environments. Through the testbed, major calibration issues, such as accommodation/convergence considerations and determination of viewing transformations, are studied and discussed in detail. Both calibration methods and results are included, which are applicable to other applications. Finally, experimental results of the testbed implementation are presented.

Head-mounted projective displays (HMPD) have been recently proposed as an alternative to conventional eyepiece-type head-mounted displays. HMPDs consist of a pair of miniature projection lenses and displays mounted on the helmet and retro-reflective sheeting materials placed strategically in the environment. Its novel concept and properties suggest solutions to part of the problems of state-of-the-art visualization devices and make it extremely suitable for multiple-user collaborative applications and wearable systems. A brief review of conventional visualization techniques is followed by an extensive discussion of HMPD technology, which includes a summary of its features and a comparison with conventional head-mounted displays (HMDs), projection-based displays, and HMPDs. An ultra-light and compact design (i.e. 8g) of a projection lens system using diffractive optical element (DOE) as well as plastic components for a HMPD is presented. Through the usage of fast prototyping technology, a compact stereoscopic head-mounted prototype with weight less than 700 grams was implemented, and optomechanical adjustments and ergonomic considerations are discussed. Finally, the motivated application in multiuser tele-collaboration is described.