Projection mapping with inexpensive hardware often suffers from calibration errors that lead to visually compromised results. In this paper, we classify common errors that lead to typical visual artifacts. Based on this classification, we present the first content-aware brightness solver. It is tailored for high GPU performance, yet efficiently hides the most common calibration artifacts. Moreover, it is specifically designed to handle both single and larger networked projection mapping setups with minimal latency.

We present a pop-up projection painting system that projects onto an unknown three-dimensional surface, while the user creates the projection content on the fly. The digital paint is projected immediately and follows the object if it is moved. If unexplored surface areas are thereby exposed, an automated trigger system issues new depth recordings that expand and refine the surface estimate. By intertwining scanning and projection painting we scan the exposed surface at the appropriate time and only if needed. Like image-based rendering, multiple automatically recorded depth maps are fused in screen space to synthesize novel views of the object, making projection poses independent from the scan positions. Since the user’s digital paint is also stored in images, we eliminate the need to reconstruct and parametrize a single full mesh, which makes geometry and color updates simple and fast.

Projection mapping augments a real-world object's appearance by projecting digital content on its surface. However, a remaining obstacle to immersive projection mapping is the limitation to white Lambertian surfaces and uniform neutral environment light, if any. Violating one of these assumptions results in a discernible difference between the source material and the appearance of the projected content. For example, some colors may not be visible due to intense environment lighting or pronounced surface colors. We present a system that actively subdues many of those real-world influences, especially environment lighting. Our system supports dynamic (i.e., movable) target objects as well as changing lighting conditions while requiring no prior color calibration of the projector nor any precomputed environment probing. We automatically and continuously estimate these influences during runtime in a real-time feedback-loop and adjust the projected colors accordingly.

Common VR headsets require lenses that increase the field of view and allow the user to focus the display. In most systems, thick lenses are used that generate strong pincushion distortions. To account for this, the content is warped by a corresponding barrel distortion before displaying, resulting in undistorted images for the viewer. This approach assumes that the eye is exactly positioned, typically on the optical axis of the lens. However, in real systems the eye's location deviates from this optimal position even at rest; moreover, the pupil - and thus the optical center of the eye - moves by several millimeters when the user looks around. Thus, eye movement results in additional distortion, which is ignored in current VR headsets. Also in literature on head-mounted displays, the effect is most often not considered, or at least badly documented. The contribution of this paper are experiments that emphasize the importance of this mostly ignored effect. To this end, we have built a simple setup with a camera at variable eye positions in a standard VR headset, that allows us to directly measure the variation of distortion during eye movement. Our experiments show that distortion varies by several dozens of pixels within the full range of eye movements, which emphasizes that the effect is definitely significant. We also demonstrate how in a headset with built-in eye tracker, the knowledge of the eye position can be used to achieve a view-dependent lens correction with only minimal additional effort at run-time.

This paper shows that motion vectors representing the true motion of an object in a scene can be exploited to improve the encoding process of computer generated video sequences. Therefore, a set of sequences is presented for which the true motion vectors of the corresponding objects were generated on a per-pixel basis during the rendering process. In addition to conventional motion estimation methods, it is proposed to exploit the computer generated motion vectors to enhance the rate-distortion performance. To this end, a motion vector mapping method including disocclusion handling is presented. It is shown that mean rate savings of 3.78% can be achieved.

The quality of every dynamic multi-projection mapping system is limited by the quality of the projector to tracking device calibration. Common problems with poor calibration result in noticeable artifacts for the user, such as ghosting and seams. In this work we introduce a new, fully automated calibration algorithm that is tailored to reduce these artifacts, based on consumer-grade hardware. We achieve this goal by repurposing a structured-light scanning setup. A structured-light scanner can generate 3D geometry based on a known intrinsic and extrinsic calibration of its components (projector and RGB camera). We revert this process by providing the resulting 3D model to determine the intrinsic and extrinsic parameters of our setup (including those of a variety of tracking systems). Our system matches features and solves for all parameters in a single pass while respecting the lower quality of our sensory input.

We present a novel approach to mesh deformation that enables simple context sensitive manipulation of 3D geometry. The method is based on locally anisotropic transformations and is extended to global control directions. This allows intuitive directional modeling within an easy to implement framework. The proposed method complements current sculpting paradigms by providing further possibilities for intuitive surface-based editing without the need for additional host geometries. We show the anisotropic deformation to be seamlessly transferable to free boundary parameterization methods, which allows us to solve the hard problem of flattening compression garments in the domain of apparel design.

Projection-based mixed reality is an effective tool to create immersive visualizations on real-world objects. Its wide range of applications includes art installations, education, stage shows, and advertising. In this work, we enhance a multi-projector system for dynamic projection mapping by handling various physical stray-light effects: interreflection, projector black-level, and environmental light in real time for dynamic scenes. We show how all these effects can be efficiently simulated and accounted for at runtime, resulting in significantly improved projection mapping results. By adding a global optimization step, we can further increase the dynamic range of the projection.

Recent publications and art performances demonstrate amazing results using projection mapping. To our knowledge, there exists no multi-projection system that can project onto non-rigid target geometries. This constrains the applicability and quality for live performances with multiple spectators. Given the cost and complexity of current systems, we present a low-cost easy-to-use markerless non-rigid face multi-projection system. It is based on a non-rigid, dense face tracker and a real-time multi-projection solver adapted to imprecise tracking, geometry and calibration. Using this novel system we produce compelling results with only consumer-grade hardware.

Using projection mapping enables us to bring virtual worlds into shared physical spaces. In this paper, we present a novel, adaptable and real-time projection mapping system, which supports multiple projectors and high quality rendering of dynamic content on surfaces of complex geometrical shape. Our system allows for smooth blending across multiple projectors using a new optimization framework that simulates the diffuse direct light transport of the physical world to continuously adapt the color output of each projector pixel. We present a real-time solution to this optimization problem using off-the-shelf graphics hardware, depth cameras and projectors. Our approach enables us to move projectors, depth camera or objects while maintaining the correct illumination, in realtime, without the need for markers on the object. It also allows for projectors to be removed or dynamically added, and provides compelling results with only commodity hardware.