Multi-projector displays show significant variation of brightness and black offset. Prior work use single-camera based methods to create a seamless display. However, the limited field-of-view (FOV) of single-camera based solutions limit their applicability to large, non-flat display shapes such as surround displays or domes. Although using a single camera at multiple locations can provide some ad-hoc corrections, it lacks both automation and accuracy. Further, automated black offset seamlessness is arguably still one of the most challenging problems in multi-projector displays. In this paper, we present the first scalable brightness and black offset seamlessness method for multi-projector displays using multiple uncalibrated cameras that scale to a large number of projectors on developable display geometries, and can operate in the presence of camera vignetting effects and varying projector brightness. Using a display partitioning method, we relate the brightness response for every pixel of the display to a common reference via appropriate camera correction factors. Then, we adapt a prior single-camera-based brightness seamlessness method to achieve seamlessness across the entire display. Next, we show that the same technique can be used to provide an automated method to achieve black offset seamlessness across the display. We demonstrate the scalability of our method to displays with upto 32 projectors on complex developable projection surfaces with cameras that have different color properties. We also provide quantitative results to demonstrate the significantly positive impact of the brightness and black offset seamlessness methods on the quality of the display.

Prior works on multi-projector displays have focused primarily on static rigid objects, some focusing on dynamic rigid objects. However, works on projection based displays on deformable dynamic objects have focused only on small scale single projector displays. Tracking a deformable dynamic surface and updating projections precisely in real time on it is a significantly challenging task, even for a single projector system. In this paper, we present the first end-to-end solution for achieving a real-time, seamless display on deformable surfaces using mutliple unsychronized projectors without requiring any prior knowledge of the surface or device parameters. The system first accurately calibrates multiple RGB-D cameras and projectors using the deformable display surface itself, and then using those calibrated devices, tracks the continuous changes in the surface shape. Based on the deformation and projector calibration, the system warps and blends the image content in real-time to create a seamless display on a surface that continuously changes shape. Using multiple projectors and RGB-D cameras, we provide the much desired aspect of scale to the displays on deformable surfaces. Most prior dynamic multi-projector systems assume rigid objects and depend critically on the constancy of surface normals and non-existence of local shape deformations. These assumptions break in deformable surfaces making prior techniques inapplicable. Point-based correspondences become inadequate for calibration, exacerbated with no synchronization between the projectors. A few works address non-rigid objects with several restrictions like targeting semi-deformable surfaces (e.g. human face), or using single coaxial (optically aligned) projector-camera pairs, or temporally synchronized cameras. We break loose from such restrictions and handle multiple projector systems for dynamic deformable fabric-like objects using temporally unsynchronized devices. We devise novel methods using ray and plane-based constraints imposed by the pinhole camera model to address these issues and design new blending methods dependent on 3D distances suitable for deformable surfaces. Finally, unlike all prior work with rigid dynamic surfaces that use a single RGB-D camera, we devise a method that involve all RGB-D cameras for tracking since the surface is not seen completely by a single camera. These methods enable a seamless display at scale in the presence of continuous movements and deformations. This work has tremendous applications on mobile and expeditionary systems where environmentals (e.g. wind, vibrations, suction) cannot be avoided. One can create large displays on tent walls in remote, austere military or emergency operations in minutes to support large scale command and control, mission rehearsal or training operations. It can be used to create displays on mobile and inflatable objects for tradeshows/events and touring edutainment applications.

In a spatially augmented reality system, multiple projectors are tiled on a complex shaped surface to create a seamless display on it. This has several applications in visualization, gaming, education and entertainment. The main challenges in creating seamless and undistorted imagery on such complex shaped surfaces are geometric registration and color correction. Prior methods that provide solutions for the spatial color variation in multi-projector displays assume rectangular overlap regions across the projectors that is possible only on flat surfaces with extremely constrained projector placement. In this article, we present a novel and fully automated method for removing color variations in a multi-projector display on arbitrary shaped smooth surfaces using a general color gamut morphing algorithm that can handle any arbitrarily shaped overlap between the projectors and assures imperceptible color variations across the display surface.

Dynamic projection mapping (DPM) is becoming increasingly popular, enabling viewers to visualize information on moving and deformable surfaces. Examples include large data visualization on the moving walls of tents deployed in austere remote locations during emergency management or defense operations. A DPM system typically comprises a RGB-D camera and a projector. In this paper, we present the first fully functional DPM system that auto-calibrates (without any physical props like planar checkerboard or rigid 3D objects) and creates a comprehensible display in the presence of large and fast movements by managing jitter and occlusion by passing objects.Prior DPM systems need specific calibration props, manual inputs and in order to deliver sub-pixel calibration accuracy. Recalibration in the face of movement or change in system setup becomes a time consuming process where the calibration prop needs to be brought back. When rendering content using DPM, errors in calibration are exacerbated and the noise in the depth camera leads to jitter, making the projection unreadable or incomprehensible. Occlusion may disrupt operations completely by jumbling up even the unoccluded parts of the display.In this paper we propose key hardware-agnostic methods for DPM calibration and rendering to make DPM systems easily deployable, stable and legible. First, we present a novel projector-camera calibration that does not need synchronization of the devices and leverages the moving surface itself, a counter-intuitive proposition. We project ArUCo markers on the moving surface and use corresponding detected features of these markers in the RGB and depth camera over multiple frames to accurately estimate the intrinsics and extrinsics of both the projector and the RGB-D camera. Second, we present a DPM rendering method that uses Kalman filtering models to reduce jitter and predict the surface shape in the presence of short term occlusions by other static objects. This results in the first DPM system, to the best of our knowledge, that can auto-calibrate in minutes and can render high resolution content like high-resolution text or images comprehensible even in the presence of fast movements, deformations and occlusions. We compare and evaluate the accuracy with prior methods and analyze the effect of surface movement on the calibration accuracy.

In recent years, medical research has made extensive use of Augmented Reality (AR) for visualization. These visualizations provide improved 3D understanding and depth perception for surgeons and medical staff during surgical planning, medical training, and procedures. Often, AR in medicine involves impractical and extensive instrumentation in order to provide the precision needed for clinical use. We propose a mobile AR 3D model registration system for use in a practical, non-instrumented hospital setting. Our registration system takes as input a patient-specific model and overlays it on the patient using an accurate pose registration technique that requires a single marker as a point of reference to initialize a point cloud-based pose refinement technique. Our method is automatic, easy to use, and runs in real-time on a mobile phone. We conduct quantitative and qualitative analysis of the registration. The results confirm that our AR pose registration system produces an accurate and visually correct overlay of the medical data in real-time.

In this article we present a completely automated and scalable multi-projector registration system that allows multiple uncalibrated projectors and cameras on arbitrary shape surfaces. Our method estimates the parameters of multiple uncalibrated tiled or superimposed projectors, the extrinsic parameters of the observing cameras (with known intrinsic parameters), the shape of the illuminated 3D geometry and geometrically registers the projectors on it. This is achieved without using any fiducials, even if part of the surface is visible to only one camera. The method uses a completely automatic approach for cross-correlation and cross-validation of the device parameters and the surface geometry resulting in an accurate registration on the arbitrary unknown geometry that does not need an accurate prior calibration of each of the uncalibrated devices using physical patterns or fiducials. Estimating projector parameters allows for quick recalibration of the system in the face of projector movements, by re-estimating only the parameters of the moved projector and not the entire system. Thus, our work can enable easy deployment of spatially augmented reality environments of different sizes (from small table top objects to large immersive environments), different shapes (inside-looking-out or outside-looking in), and different configurations (tiled or superimposed) using the same proposed method.

We present a method for dynamic projection mapping on deformable, stretchable and elastic materials (e.g. cloth) using a time of flight (ToF) depth camera (e.g. Azure Kinect or Pico-Flexx) that come equipped with an IR camera. We use Bezier surfaces to model the projection surface without explicitly modeling the deformation. We devise an efficient tracking method that tracks the boundary of the surface material using the IR-Depth camera. This achieves realistic mapping even in the interior of the surface, with simple markers (e.g. black dots or squares) or without markers entirely, such that the projection appears to be printed on the material. The surface representation is updated in real-time using GPU based computations. Further, we also show that the speed of these updates is limited by the camera frame rate and therefore can be adopted for higher speed cameras as well. This technique can be used to project on several stretchable moving materials to change their appearance.

Abstract Radiometric compensation methods remove the effect of the underlying spatially varying surface reflectance of the texture when projecting on textured surfaces. All prior work sample the surface reflectance dependent radiometric transfer function from the projector to the camera at every pixel that requires the camera to observe tens or hundreds of images projected by the projector. In this paper, we cast the radiometric compensation problem as a sampling and reconstruction of multi‐dimensional radiometric transfer function that models the color transfer function from the projector to an observing camera and the surface reflectance in a unified manner. Such a multi‐dimensional representation makes no assumption about linearity of the projector to camera color transfer function and can therefore handle projectors with non‐linear color transfer functions(e.g. DLP, LCOS, LED‐based or laser‐based). We show that with a well‐curated sampling of this multi‐dimensional function, achieved by exploiting the following key properties, is adequate for its accurate representation: (a) the spectral reflectance of most real‐world materials are smooth and can be well‐represented using a lower‐dimension function; (b) the reflectance properties of the underlying texture have strong redundancies – for example, multiple pixels or even regions can have similar surface reflectance; (c) the color transfer function from the projector to camera have strong input coherence. The proposed sampling allows us to reduce the number of projected images that needs to be observed by a camera by up to two orders of magnitude, the minimum being only two. We then present a new multi‐dimensional scattered data interpolation technique to reconstruct the radiometric transfer function at a high spatial density (i.e. at every pixel) to compute the compensation image. We show that the accuracy of our interpolation technique is higher than any existing methods.

Distortions due to perspective projection is often described under the umbrella term of foreshortening in computer graphics and are treated the same way. However, a large body of literature from artists, perceptual psychologists and perception scientists have shown that the perception of these distortions is different in different situations. While the distortions themselves depend on both the depth and the orientation of the object with respect to the camera image plane, the perception of these distortions depends on other depth cues present in the image. In the absence of any depth cue or prior knowledge about the objects in the scene, the visual system finds it hard to correct the foreshortening automatically and such images need user input and external algorithmic distortion correction.In this paper, we claim that the shape distortion is more perceptible than area distortion, and quantify such perceived foreshortening as the non-uniformity across the image, of the ratio e of the differential areas of an object in the scene and its projection. We also categorize foreshortening into uniform and non-uniform foreshortening. Uniform foreshortening is perceived by our visual system as a distortion, even if e is uniform across the image, only when comparative objects of known sizes are present in the image. Non-uniform foreshortening is perceived when there is no other depth cue in the scene that can help the brain to correct for the distortion. We present a unified solution to correct these distortions in one or more non-occluded foreground objects by applying object-specific segmentation and affine transformation of the segmented camera image plane. Our method also ensures that the background undergoes minimal distortion and preserves background features during this process. This is achieved efficiently by solving Laplace's equations with Dirichlet boundary conditions, assisted by a simple and intuitive user interface.

In this demo we will demonstrate integration of general content delivery from a windows desktop to a multi-projector display of arbitrary, shape, size and resolution automatically calibrated using our calibration methods. We have developed these sophisticated completely automatic geometric and color registration techniques in our lab for deploying seamless multi-projector displays on popular non-planar surfaces (e.g. cylinders, domes, truncated domes). This work has gotten significant attention in both VR and Visualization venues in the past 5 years and this will be the first time such calibration will be integrated with content delivery.

Abstract Many works focus on multi‐spectral capture and analysis, but multi‐spectral display still remains a challenge. Most prior works on multi‐primary displays use ad‐hoc narrow band primaries that assure a larger color gamut, but cannot assure a good spectral reproduction. Content‐dependent spectral analysis is the only way to produce good spectral reproduction, but cannot be applied to general data sets. Wide primaries are better suited for assuring good spectral reproduction due to greater coverage of the spectral range, but have not been explored much. In this paper we explore the use of wide band primaries for accurate spectral reproduction for the first time and present the first content‐independent multi‐spectral display achieved using superimposed projections with modified wide band primaries. We present a content‐independent primary selection method that selects a small set of n primaries from a large set of m candidate primaries where m > n. Our primary selection method chooses primaries with complete coverage of the range of visible wavelength (for good spectral reproduction accuracy), low interdependency (to limit the primaries to a small number) and higher light throughput (for higher light efficiency). Once the primaries are selected, the input values of the different primary channels to generate a desired spectrum are computed using an optimization method that minimizes spectral mismatch while maximizing visual quality. We implement a real prototype of multi‐spectral display consisting of 9‐primaries using three modified conventional 3‐primary projectors, and compare it with a conventional display to demonstrate its superior performance. Experiments show our display is capable of providing large gamut assuring a good visual appearance while displaying any multi‐spectral images at a high spectral accuracy.

Simultaneous modification and navigation of massive 3D models are difficult because repeated data edits affect the data layout and coherency on a secondary storage, which in turn affect the interactive out-of-core rendering performance. In this paper, we propose a novel approach for distributed data management for simultaneous interactive navigation and modification of massive 3D data using the readily available infrastructure of a tiled display. Tiled multi-displays, projection or LCD panel based, driven by a PC cluster, can be viewed as a cluster of storage-compute-display (SCD) nodes. Given a cluster of SCD node infrastructure, we first propose a distributed memory hierarchy for interactive rendering applications. Second, in order to further reduce the latency in such applications, we propose a new data partitioning approach for distributed storage among the SCD nodes that reduces the variance in the data load across the SCD nodes. Our data distribution method takes in a data set of any size, and reorganizes it into smaller partitions, and stores it across the multiple SCD nodes. These nodes store, manage, and coordinate data with other SCD nodes to simultaneously achieve interactive navigation and modification. Specifically, the data is not duplicated across these distributed secondary storage devices. In addition, coherency in data access, due to screen-space adjacency of adjacent displays in the tile, as well as object space adjacency of the data sets, is well leveraged in the design of the data management technique. Empirical evaluation on two large data sets, with different data density distribution, demonstrates that the proposed data management approach achieves superior performance over alternative state-of-the-art methods.

The emergence of pico projectors as a part of future mobile devices presents unique opportunities for collaborative settings, especially in entertainment applications, such as video playback. By aggregating pico projectors from several users, it is possible to enhance resolution, brightness, or frame rate. In this paper we present a camera-based methodology for the alignment and synchronization of multiple projectors. The approach does not require any complicated ad hoc network setup among the mobile devices. A prototype system has been set up and used to test the proposed techniques.

Earlier work has explored enhancing the perceived resolution of a display by shifting multiple different low-resolution images by fractions of a pixel and overlaying them in a temporally multiplexed fashion. This increases the manufacturing cost and also sacrifices the temporal resolution that can compromise other capabilities like 3D active stereo. In this paper we propose a method to achieve the same goal in projectors by performing the pixel shift and superposition optically by introducing a simple and inexpensive optical ensemble of a set of lenses on the projector light path. This does not sacrifice the temporal resolution and is extremely easy to implement in practice. However, instead of overlaying different images, we overlay an image with one or more sub-pixel shifted copies of itself. Therefore, we seek a single n×n image which when shifted and overlaid with itself creates a perceptually closer to a higher resolution 2n × 2n target image. This changes the optimization formulation significantly and requires solving a system of sparse linear equations. We take advantage of this sparsity and design a parallel implementation of this optimization in GPUs for real-time computation of the input image critical for its practical implementation. But, since this system is more constrained that using multiple overlaid images, the enhancement of resolution is compromised. However, since the optical design is very simple and inexpensive, it can be deployed on a variety of low-cost projectors and still offer a significant image quality benefit.

The use of wide angle lens on commodity cameras are becoming increasingly popular. However, this leads to image distortions due to the large difference in relative orientation of the different foreground objects with respect to the camera's image plane. More importantly, these distortions are different in different parts of the image since it depends entirely on the position and orientation of the foreground object with respect to the image plane. Such distortions often manifest themselves as objects in the foreground appearing fatter than they are supposed to be or parts thereof (e.g. hands or head of people) looking disproportionately larger than the rest of them. Though there are earlier works addressing other common distortions in cameras like radial distortions, little attention has been given to this problem. In this paper, we present an effective method to remove such distortions in foreground objects minimally distorting the background. This is achieved efficiently using a mesh-based pixel displacement technique assisted by a simple and intuitive user interface design.

Color is one of the most common ways to convey information in visualization applications. Color vision deficiency (CVD) affects approximately 200 million individuals worldwide and considerably degrades their performance in understanding such contents by creating red-green or blue-yellow ambiguities. While several content-specific methods have been proposed to resolve these ambiguities, they cannot achieve this effectively in many situations for contents with a large variety of colors. More importantly, they cannot facilitate color identification. We propose a technique for using patterns to encode color information for individuals with CVD, in particular for dichromats. We present the first content-independent method to overlay patterns on colored visualization contents that not only minimizes ambiguities but also allows color identification. Further, since overlaying patterns does not compromise the underlying original colors, it does not hamper the perception of normal trichromats. We validated our method with two user studies: one including 11 subjects with CVD and 19 normal trichromats, and focused on images that use colors to represent multiple categories; and another one including 16 subjects with CVD and 22 normal trichromats, which considered a broader set of images. Our results show that overlaying patterns significantly improves the performance of dichromats in several color-based visualization tasks, making their performance almost similar to normal trichromats'. More interestingly, the patterns augment color information in a positive manner, allowing normal trichromats to perform with greater accuracy.

Pico projectors are expected to become increasingly popular in the near future, in particular when embedded in mobile devices such as smart phones, portable media players and digital cameras. Consumers seeking more features and attracted to all in one portable devices, will find such integrated devices very appealing. However the resolution and the brightness provided by integrated mobile projectors are much lower than what standard projectors commonly offer. Our collaborative scheme based on our synchronization technique provides a means of increasing resolution and brightness for the video projected by such mobile devices, significantly enhancing the viewing experience for the user.

We present a system that superimposes multiple projections onto an object of arbitrary shape and color to produce high-resolution appearance changes. Our system produces appearances at an improved resolution compared to prior works and can change appearances at near interactive rates. Three main components are central to our system. First, the problem of computing compensation images is formulated as a constrained optimization which yields high-resolution appearances. Second, decomposition of the target appearance into base and scale images enables fast swapping of appearances on the object by requiring the constrained optimization to be computed only once per object. Finally, to make high-quality appearance edits practical, an elliptical Gaussian is used to model projector pixels and their interaction between projectors. To the best of our knowledge, we build the first system that achieves high-resolution and high-quality appearance edits using multiple superimposed projectors on complex nonplanar colored objects. We demonstrate several appearance edits including specular lighting, subsurface scattering, inter-reflections, and color, texture, and geometry changes on objects with different shapes and colors.

Digital projection technology has improved significantly in recent years. But, the relationship of cost with respect to available resolution in projectors is still super-linear. In this paper, we present a method that uses projector light modulator panels (e.g. LCD or DMD panels) of resolution n X n to create a perceptually close match to a target higher resolution cn X cn image, where c is a small integer greater than 1. This is achieved by enhancing the resolution using smaller pixels at specific regions of interest like edges. A target high resolution image ( cn X cn ) is first decomposed into (a) a high resolution ( cn X cn ) but sparse edge image , and (b) a complementary lower resolution ( n X n ) non-edge image . These images are then projected in a time sequential manner at a high frame rate to create an edge-enhanced image -- an image where the pixel density is not uniform but changes spatially. In 3D ready projectors with readily available refresh rate of 120Hz, such a temporal multiplexing is imperceptible to the user and the edge-enhanced image is perceptually almost identical to the target high resolution image. To create the higher resolution edge image, we introduce the concept of optical pixel sharing . This reduces the projected pixel size by a factor of 1/ c 2 while increasing the pixel density by c 2 at the edges enabling true higher resolution edges. Due to the sparsity of the edge pixels in an image we are able to choose a sufficiently large subset of these to be displayed at the higher resolution using perceptual parameters. We present a statistical analysis quantifying the expected number of pixels that will be reproduced at the higher resolution and verify it for different types of images.

The articles in this special section contain selected papers from the IEEE Virtual Reality Conference (VR).

In this paper, we present the first method for the geometric autocalibration of multiple projectors on a set of CAVE-like immersive display surfaces including truncated domes and 4 or 5-wall CAVEs (three side walls, floor, and/or ceiling). All such surfaces can be categorized as swept surfaces and multiple projectors can be registered on them using a single uncalibrated camera without using any physical markers on the surface. Our method can also handle nonlinear distortion in the projectors, common in compact setups where a short throw lens is mounted on each projector. Further, when the whole swept surface is not visible from a single camera view, we can register the projectors using multiple pan and tilted views of the same camera. Thus, our method scales well with different size and resolution of the display. Since we recover the 3D shape of the display, we can achieve registration that is correct from any arbitrary viewpoint appropriate for head-tracked single-user virtual reality systems. We can also achieve wallpapered registration, more appropriate for multiuser collaborative explorations. Though much more immersive than common surfaces like planes and cylinders, general swept surfaces are used today only for niche display environments. Even the more popular 4 or 5-wall CAVE is treated as a piecewise planar surface for calibration purposes and hence projectors are not allowed to be overlapped across the corners. Our method opens up the possibility of using such swept surfaces to create more immersive VR systems without compromising the simplicity of having a completely automatic calibration technique. Such calibration allows completely arbitrary positioning of the projectors in a 5-wall CAVE, without respecting the corners.

Welcome to IEEE Virtual Reality 2011! Similar to previous years, the conference program is mostly made up of single track presentations. There are fourteen long paper presentations, eighteen short presentations, and three panels. This year, we have some new improvements to the program including the addition of four invited presentations from the highly regraded IEEE Transactions on Visualization and Computer Graphics journal.

Abstract Projection‐based appearances are used in a variety of computer graphics applications to impart different appearances onto physical surfaces using digitally controlled projector light. To achieve a compliant appearance, all points on the physical surface must be altered to the colours of the desired target appearance; otherwise, an incompliant appearance results in a misleading visualization. Previous systems typically assume to operate with compliant appearances or restrict themselves to the simpler case of white surfaces. To achieve compliancy, one may change the physical surface's albedo, increase the amount of projector light radiance available or modify the target appearance's colours. This paper presents an approach to modify a target appearance to achieve compliant appearance editing without altering the physical surface or the projector setup. Our system minimally alters the target appearance's colours while maintaining cues important for perceptual similarity (e.g. colour constancy). First, we discuss how to measure colour compliancy. Next, we describe our approach to partition the physical surface into patches based on the surface's colours and the target appearance's colours. Finally, we describe our appearance optimization process, which computes a compliant appearance that is as perceptually similar as possible to the target appearance's colours. We perform several real‐world projection‐based appearances and compare our results to naïve approaches, which either ignore compliancy or simply reduce the appearance's overall brightness.

Abstract In this paper we present a novel technique for easily calibrating multiple casually aligned projectors on spherical domes using a single uncalibrated camera. Using the prior knowledge of the display surface being a dome, we can estimate the camera intrinsic and extrinsic parameters and the projector to display surface correspondences automatically using a set of images. These images include the image of the dome itself and a projected pattern from each projector. Using these correspondences we can register images from the multiple projectors on the dome. Further, we can register displays which are not entirely visible in a single camera view using multiple pan and tilted views of an uncalibrated camera making our method suitable for displays of different size and resolution. We can register images from any arbitrary viewpoint making it appropriate for a single head‐tracked user in a 3D visualization system. Also, we can use several cartographic mapping techniques to register images in a manner that is appropriate for multi‐user visualization. Domes are known to produce a tremendous sense of immersion and presence in visualization systems. Yet, till date, there exists no easy way to register multiple projectors on a dome to create a high‐resolution realistic visualizations. To the best of our knowledge, this is the first method that can achieve accurate geometric registration of multiple projectors on a dome simply and automatically using a single uncalibrated camera.

We present a camera with switchable primaries using shiftable layers of color filter arrays (CFAs). By layering a pair of CMY CFAs in this novel manner we can switch between multiple sets of color primaries (namely RGB, CMY and RGBCY) in the same camera. In contrast to fixed color primaries (e.g. RGB or CMY), which cannot provide optimal image quality for all scene conditions, our camera with switchable primaries provides optimal color fidelity and signal to noise ratio for multiple scene conditions. Next, we show that the same concept can be used to layer two RGB CFAs to design a camera with switchable low dynamic range (LDR) and high dynamic range (HDR) modes. Further, we show that such layering can be generalized as a constrained satisfaction problem (CSP) allowing to constrain a large number of parameters (e.g. different operational modes, amount and direction of the shifts, placement of the primaries in the CFA) to provide an optimal solution. We investigate practical design options for shiftable layering of the CFAs. We demonstrate these by building prototype cameras for both switchable primaries and switchable LDR/HDR modes. To the best of our knowledge, we present, for the first time, the concept of shiftable layers of CFAs that provides a new degree of freedom in photography where multiple operational modes are available to the user in a single camera for optimizing the picture quality based on the nature of the scene geometry, color and illumination.

In this paper, we present a novel technique to calibrate multiple casually aligned projectors on fiducial-free piecewise smooth vertically extruded surfaces using a single camera. Such surfaces include cylindrical displays and CAVEs, common in immersive virtual reality systems. We impose two priors to the display surface. We assume the surface is a piecewise smooth vertically extruded surface for which the aspect ratio of the rectangle formed by the four corners of the surface is known and the boundary is visible and segmentable. Using these priors, we can estimate the display's 3D geometry and camera extrinsic parameters using a nonlinear optimization technique from a single image without any explicit display to camera correspondences. Using the estimated camera and display properties, the intrinsic and extrinsic parameters of each projector are recovered using a single projected pattern seen by the camera. This in turn is used to register the images on the display from any arbitrary viewpoint making it appropriate for virtual reality systems. The fast convergence and robustness of this method is achieved via a novel dimension reduction technique for camera parameter estimation and a novel deterministic technique for projector property estimation. This simplicity, efficiency, and robustness of our method enable several coveted features for nonplanar projection-based displays. First, it allows fast recalibration in the face of projector, display or camera movements and even change in display shape. Second, this opens up, for the first time, the possibility of allowing multiple projectors to overlap on the corners of the CAVE-a popular immersive VR display system. Finally, this opens up the possibility of easily deploying multiprojector displays on aesthetic novel shapes for edutainment and digital signage applications.

In this paper we present a novel technique to calibrate multiple casually aligned projectors on a fiducial-free cylindrical curved surface using a single camera. We impose two priors to the cylindrical display: (a) cylinder is a vertically extruded surface; and (b) the aspect ratio of the rectangle formed by the four corners of the screen is known. Using these priors, we can estimate the display's 3D surface geometry and camera extrinsic parameters using a single image without any explicit display to camera correspondences. Using the estimated camera and display properties, we design a novel deterministic algorithm to recover the intrinsic and extrinsic parameters of each projector using a single projected pattern seen by the camera which is then used to register the images on the display from any arbitrary viewpoint making it appropriate for virtual reality systems. Finally, our method can be extended easily to handle sharp corners - making it suitable for the common CAVE like VR setup. To the best of our knowledge, this is the first method that can achieve accurate geometric auto-calibration of multiple projectors on a cylindrical display without performing an extensive stereo reconstruction.

In this paper, we present the first method to geometrically register multiple projectors on a swept surface (e.g. a truncated dome) using a single uncalibrated camera without using any physical markers on the surface. Our method can even handle non-linear distortion in projectors common in compact setups where a short throw lens is mounted on each projector. Further, when the whole swept surface is not visible from a single camera view, we can register the projectors using multiple pan and tilted views of the same camera. Thus, our method scales well with different size and resolution of the display. Since we recover the 3D shape of the display, we can achieve registration that is correct from any arbitrary viewpoint appropriate for head-tracked single-user virtual reality systems. We can also achieve wallpapered registration more appropriate for multi-user collaborative explorations. Our method achieves sub-pixel accuracy and the image correction required to achieve the registration runs in real-time on the GPU.

Abstract Recent work have shown that it is possible to register multiple projectors on non‐planar surfaces using a single uncalibrated camera instead of a calibrated stereo pair when dealing with a special class of non‐planar surfaces, vertically extruded surfaces. However, this requires the camera view to contain the entire display surface. This is often an impossible scenario for large displays, especially common in visualization, edutainment, training and simulation applications. In this paper we present a new method that can achieve an accurate geometric registration even when the field‐of‐view of the uncalibrated camera can cover only a part of the vertically extruded display at a time. We pan and tilt the camera from a single point and employ a multi‐view approach to register the projectors on the display. This allows the method to scale easily both in terms of camera resolution and display size. To the best of our knowledge, our method is the first to achieve a scalable multi‐view geometric registration of large vertically extruded displays with a single uncalibrated camera. This method can also handle a different situation of having multiple similarly oriented cameras in different locations, if the camera focal length is known.

Many visualization applications benefit from displaying content on real-world objects rather than on a traditional display (e.g., a monitor). This type of visualization display is achieved by projecting precisely controlled illumination from multiple projectors onto the real-world colored objects. For such a task, the placement of the projectors is critical in assuring that the desired visualization is possible. Using ad hoc projector placement may cause some appearances to suffer from color shifting due to insufficient projector light radiance being exposed onto the physical surface. This leads to an incorrect appearance and ultimately to a false and potentially misleading visualization. In this paper, we present a framework to discover the optimal position and orientation of the projectors for such projection-based visualization displays. An optimal projector placement should be able to achieve the desired visualization with minimal projector light radiance. When determining optimal projector placement, object visibility, surface reflectance properties, and projector-surface distance and orientation need to be considered. We first formalize a theory for appearance editing image formation and construct a constrained linear system of equations that express when a desired novel appearance or visualization is possible given a geometric and surface reflectance model of the physical surface. Then, we show how to apply this constrained system in an adaptive search to efficiently discover the optimal projector placement which achieves the desired appearance. Constraints can be imposed on the maximum radiance allowed by the projectors and the projectors' placement to support specific goals of various visualization applications. We perform several real-world and simulated appearance edits and visualizations to demonstrate the improvement obtained by our discovered projector placement over ad hoc projector placement.

We present the first distributed paradigm for multiple users to interact simultaneously with large tiled rear projection display walls. Unlike earlier works, our paradigm allows easy scalability across different applications, interaction modalities, displays and users. The novelty of the design lies in its distributed nature allowing well-compartmented, application independent, and application specific modules. This enables adapting to different 2D applications and interaction modalities easily by changing a few application specific modules. We demonstrate four challenging 2D applications on a nine projector display to demonstrate the application scalability of our method: map visualization, virtual graffiti, virtual bulletin board and an emergency management system. We demonstrate the scalability of our method to multiple interaction modalities by showing both gesture-based and laser-based user interfaces. Finally, we improve earlier distributed methods to register multiple projectors. Previous works need multiple patterns to identify the neighbors, the configuration of the display and the registration across multiple projectors in logarithmic time with respect to the number of projectors in the display. We propose a new approach that achieves this using a single pattern based on specially augmented QR codes in constant time. Further, previous distributed registration algorithms are prone to large misregistrations. We propose a novel radially cascading geometric registration technique that yields significantly better accuracy. Thus, our improvements allow a significantly more efficient and accurate technique for distributed self-registration of multi-projector display walls.

Multi-projector displays show significant spatial variation in 3D color gamut due to variation in the chromaticity gamuts across the projectors, vignetting effect of each projector and also overlap across adjacent projectors. In this paper we present a new constrained gamut morphing algorithm that removes all these variations and results in true color seamlessness across tiled multiprojector displays. Our color morphing algorithm adjusts the intensities of light from each pixel of each projector precisely to achieve a smooth morphing from one projector's gamut to the other's through the overlap region. This morphing is achieved by imposing precise constraints on the perceptual difference between the gamuts of two adjacent pixels. In addition, our gamut morphing assures a C1 continuity yielding visually pleasing appearance across the entire display.We demonstrate our method successfully on a planar and a curved display using both low and high-end projectors. Our approach is completely scalable, efficient and automatic. We also demonstrate the real-time performance of our image correction algorithm on GPUs for interactive applications. To the best of our knowledge, this is the first work that presents a scalable method with a strong foundation in perception and realizes, for the first time, a truly seamless display where the number of projectors cannot be deciphered.

In this paper, we present the first algorithm to geometrically register multiple projectors in a view-independent manner (i.e. wallpapered) on a common type of curved surface, vertically extruded surface, using an uncalibrated camera without attaching any obtrusive markers to the display screen. Further, it can also tolerate large non-linear geometric distortions in the projectors as is common when mounting short throw lenses to allow a compact set-up. Our registration achieves sub-pixel accuracy on a large number of different vertically extruded surfaces and the image correction to achieve this registration can be run in real time on the GPU. This simple markerless registration has the potential to have a large impact on easy set-up and maintenance of large curved multi-projector displays, common for visualization, edutainment, training and simulation applications.

In order to achieve seamless imagery in a planar multi-projector display, geometric distortions and misalignment of images within and across projectors have to be removed. Camera-based calibration methods are popular for achieving this in an automated fashion. Previous methods for geometric calibration fall into two categories: (a) methods that model the geometric function relating the projectors to cameras using simple linear models, like homography, to calibrate the display. These models assume perfect linear devices and cannot address projector nonlinearities, like lens distortions, which are common in most commodity projectors, (b) methods that use piecewise linear approximations to model the relationship between projectors and cameras. These require a dense sampling of the function space to achieve good calibration. In this paper, we present a new closed-form model that relates projectors to cameras in planar multi-projector displays, using rational Bezier patches. This model overcomes the shortcomings of the previous methods by allowing for projectors with significant lens distortion. It can be further used to develop an efficient and accurate geometric calibration method with a sparse sampling of the function.

In this paper, we present a method for photometric self- calibration of a projector-camera system. In addition to the input transfer functions (commonly called gamma functions), we also reconstruct the spatial intensity fall-off from the center to fringe (commonly called the vignetting effect) for both the projector and camera. Projector-camera systems are becoming more popular in a large number of applications like scene capture, 3D reconstruction, and calibrating multi-projector displays. Our method enables the use of photometrically uncalibrated projectors and cameras in all such applications.

Study of contrast sensitivity of the human eye shows that our suprathreshold contrast sensitivity follows the Weber Law and, hence, increases proportionally with the increase in the mean local luminance. In this paper, we effectively apply this fact to design a contrast-enhancement method for images that improves the local image contrast by controlling the local image gradient with a single parameter. Unlike previous methods, we achieve this without explicit segmentation of the image, either in the spatial (multiscale) or frequency (multiresolution) domain. We pose the contrast enhancement as an optimization problem that maximizes the average local contrast of an image strictly constrained by a perceptual constraint derived directly from the Weber Law. We then propose a greedy heuristic, controlled by a single parameter, to approximate this optimization problem.

Multi-projector displays today are automatically registered, both geometrically and photometrically, using cameras. Existing registration techniques assume pre-calibrated projectors and cameras that are devoid of imperfections such as lens distortion. In practice, however, these devices are usually imperfect and uncalibrated. Registration of each of these devices is often more challenging than the multi-projector display registration itself. To make tiled projection-based displays accessible to a layman user we should allow the use of uncalibrated inexpensive devices that are prone to imperfections. In this paper, we make two important advances in this direction. First, we present a new geometric registration technique that can achieve geometric alignment {\em in the presence of severe projector lens distortion} using a relatively inexpensive low-resolution camera. This is achieved via a closed-form model that relates the projectors to cameras, in planar multi-projector displays, using rational Bezier patches. This enables us to geometrically calibrate a 3000 x 2500 resolution planar multi-projector display made of 3 x 3 array of nine severely distorted projectors using a low resolution (640 x 480) VGA camera. Second, we present a photometric self-calibration technique for a projector-camera pair. This allows us to photometrically calibrate the same display made of nine projectors using a photometrically uncalibrated camera. To the best of our knowledge, this is the first work that allows geometrically imperfect projectors and photometrically uncalibrated cameras in calibrating multi-projector displays.

Centralized techniques have been used until now when automatically calibrating (both geometrically and photometrically) large high-resolution displays created by tiling multiple projectors in a 2D array. A centralized server managed all the projectors and also the camera(s) used to calibrate the display. In this paper, we propose an asynchronous distributed calibration methodology via a display unit called the plug-and-play projector (PPP). The PPP consists of a projector, camera, computation and communication unit, thus creating a self-sufficient module that enables an asynchronous distributed architecture for multi-projector displays. We present a single-program-multiple-data (SPMD) calibration algorithm that runs on each PPP and achieves a truly scalable and reconfigurable display without any input from the user. It instruments novel capabilities like adding/removing PPPs from the display dynamically, detecting faults, and reshaping the display to a reasonable rectangular shape to react to the addition/removal/faults. To the best of our knowledge, this is the first attempt to realize a completely asynchronous and distributed calibration architecture and methodology for multi-projector displays.

Study of contrast sensitivity of the human eye shows that we are more sensitive to brightness differences at low intensity levels than at high intensity levels. We apply this fact effectively to achieve brightness seamless-ness in multi-projector displays. Multi-displays, popularly made of a rectangular array of partially overlapping projectors, show severe spatial variation in brightness. Existing methods achieve brightness uniformity across the display by matching the brightness response of every pixel to the pixel with the most limited contrast leading to severe compression in the contrast of the display. In this paper, we propose a method that allows a constrained variation in brightness guided by the human contrast sensitivity function such that it is imperceptible to the human eye. This achieves a seamless multi-display with a dramatic improvement in the contrast making the display practically usable.

An interesting issue of contemplation amongst researchers working on multi-projector displays is whether spatial superresolution can be achieved by overlapping images from multiple projectors. This paper presents a thorough theoretical analysis to answer this question using signal processing and perturbation theory. Our analysis is supported by results from a simulated overlapping projector display. This analysis shows that achieving spatial super-resolution using overlapping projectors is practically infeasible.

Abstract The concept of tiled displays can be successful only if such displays are made to look like a single display perceptually. The two issues that need to be solved to achieve this goal are geometric correction and color seamlessness of images spanning across tiles. Geometric correction algorithms borrow pin‐hole camera models to model projector display geometry. In this paper, we introduce an abstract modeling function that describes the color seen by a viewer when displayed by a display device. Though this function can be used to model color displayed by any common display device, in this paper, we use it to model color in multiprojector display systems. We use the model to explain the reasons for different types of color variations in a multiprojector display, to compare different color correction algorithms, and to derive such algorithms directly from the model.

Arguably, the most vexing problem remaining for multi-projector displays is that of photometric (brightness) seamlessness within and across different projectors. Researchers have strived for strict photometric uniformity that achieves identical response at every pixel of the display. However, this goal typically results in displays with severely compressed dynamic range and poor image quality. In this article, we show that strict photometric uniformity is not a requirement for achieving photometric seamlessness. We introduce a general goal for photometric seamlessness by defining it as an optimization problem, balancing perceptual uniformity with display quality. Based on this goal, we present a new method to achieve perceptually seamless high quality displays. We first derive a model that describes the photometric response of projection-based displays. Then we estimate the model parameters and modify them using perception-driven criteria. Finally, we use the graphics hardware to reproject the image computed using the modified model parameters by manipulating only the projector inputs at interactive rates. Our method has been successfully demonstrated on three different practical display systems at Argonne National Laboratory, made of 2 × 2 array of four projectors, 2 × 3 array of six, projectors, and 3 × 5 array of fifteen projectors. Our approach is efficient, automatic and scalable---requiring only a digital camera and a photometer. To the best of our knowledge, this is the first approach and system that addresses the photometric variation problem from a perceptual stand point and generates truly seamless displays with high dynamic range.

Multiprojector, large-scale displays are used in scientific visualization, virtual reality, and other visually intensive applications. In recent years, a number of camera-based computer vision techniques have been proposed to register the geometry and color of tiled projection-based display. These automated techniques use cameras to "calibrate" display geometry and photometry, computing per-projector corrective warps and intensity corrections that are necessary to produce seamless imagery across projector mosaics. These techniques replace the traditional labor-intensive manual alignment and maintenance steps, making such displays cost-effective, flexible, and accessible. In this paper, we present a survey of different camera-based geometric and photometric registration techniques reported in the literature to date. We discuss several techniques that have been proposed and demonstrated, each addressing particular display configurations and modes of operation. We overview each of these approaches and discuss their advantages and disadvantages. We examine techniques that address registration on both planar (video walls) and arbitrary display surfaces and photometric correction for different kinds of display surfaces. We conclude with a discussion of the remaining challenges and research opportunities for multiprojector displays.

An important problem in achieving seamless multi-projector displays is the photometric (luminance) variation across the display. Currently, evaluation of the improvement in the quality of the imagery achieved by applying different photometric correction methods and a comparison between them is done by a subjective human user. In this paper, we present a camera-based method for relative evaluation of the luminance properties of the imagery on multi-projector displays, both before correction and after correction using different compensation techniques. To the best of our knowledge, this is the first camera-based method designed to evaluate the different photometric correction methods on a multi-projector display. We show that using our simple and automatic evaluation scheme the optimal photometric compensation parameter for a multi-projector display can be identified, which is not possible by subjective evaluation.

Large-area displays made up of several projectors show significant variation in color. In this paper, we identify different projector parameters that cause the color variation and study their effects on the luminance and chrominance characteristics of the display. This work leads to the realization that luminance varies significantly within and across projectors, while chrominance variation is relatively small, especially across projectors of same model. To address this situation, we present a method to achieve luminance matching across all pixels of a multiprojector display that results in photometrically uniform displays. We use a camera as a measurement device for this purpose. Our method comprises a one-time calibration step that generates a per channel per projector luminance attenuation map (LAM), which is then used to correct any image projected on the display at interactive rates on commodity graphics hardware. To the best of our knowledge, this is the first effort to match luminance across all the pixels of a multiprojector display.

Large-area multi-projector displays show significant spatial variation in color, both within a single projector's field of view and across different projectors. Recent research in this area has shown that the color variation is primarily due to luminance variation. Luminance varies within a single projector's field of view, across different brands of projectors and with the variation in projector parameters. Luminance variation is also introduced by overlap between adjacent projectors. On the other hand, chrominance remains constant throughout a projector's field of view and varies little with the change in projector parameters, especially for projectors of the same brand. Hence, matching luminance response of all the pixels of a multi-projector display should help us to achieve photometric uniformity.In this paper, we present a method to do a per channel per pixel luminance matching. Our method consists of a one-time calibration procedure when a luminance attenuation map (LAM) is generated. This LAM is then used to correct any image to achieve photometric uniformity. In the one-time calibration step, we first use a camera to measure the per channel luminance response of a multi-projector display and find the pixel with the most luminance response. Then, for each projector, we generate a per channel LAM that assigns a weight to every pixel of the projector to scale the luminance response of that pixel to match with the most limited response. This LAM is then used to attenuate any image projected by the projector.This method can be extended to do the image correction in real time on traditional graphics pipeline by using alpha blending and color look-up-tables. To the best of our knowledge, this is the first effort to match luminance across all the pixels of a multi-projector display. Our results show that luminance matching can indeed achieve photometric uniformity.

Large area tiled displays are gaining popularity for use in collaborative immersive virtual environments and scientific visualization. While recent work has addressed the issues of geometric registration, rendering architectures, and human interfaces, there has been relatively little work on photometric calibration in general, and photometric non-uniformity in particular. For example, as a result of differences in the photometric characteristics of projectors, the color and intensity of a large area display varies from place to place. Further, the imagery typically appears brighter at the regions of overlap between adjacent projectors. We analyze and classify the causes of photometric non-uniformity in a tiled display. We then propose a methodology for determining corrections designed to achieve uniformity, that can correct for the photometric variations across a tiled projector display in real time using per channel color look-up-tables (LUT).

This paper presents a new algorithm for immersive teleconferencing, which addresses the problem of registering and blending multiple images together to create a single seamless panorama. In the immersive teleconference paradigm, one frame of the teleconference is a panorama that is constructed from a compound-image sensing device. These frames are rendered at the remote site on a projection surface that surrounds the user, creating an immersive feeling of presence and participation in the teleconference. Our algorithm efficiently creates panoramic frames for a teleconference session that are both geometrically registered and intensity blended. We demonstrate a prototype that is able to capture images from a compound-image sensor, register them into a seamless panoramic frame, and render those panoramic frames on a projection surface at 30 frames per second.