We present a pop-up projection painting system that projects onto an unknown three-dimensional surface, while the user creates the projection content on the fly. The digital paint is projected immediately and follows the object if it is moved. If unexplored surface areas are thereby exposed, an automated trigger system issues new depth recordings that expand and refine the surface estimate. By intertwining scanning and projection painting we scan the exposed surface at the appropriate time and only if needed. Like image-based rendering, multiple automatically recorded depth maps are fused in screen space to synthesize novel views of the object, making projection poses independent from the scan positions. Since the user’s digital paint is also stored in images, we eliminate the need to reconstruct and parametrize a single full mesh, which makes geometry and color updates simple and fast.

Projection mapping augments a real-world object's appearance by projecting digital content on its surface. However, a remaining obstacle to immersive projection mapping is the limitation to white Lambertian surfaces and uniform neutral environment light, if any. Violating one of these assumptions results in a discernible difference between the source material and the appearance of the projected content. For example, some colors may not be visible due to intense environment lighting or pronounced surface colors. We present a system that actively subdues many of those real-world influences, especially environment lighting. Our system supports dynamic (i.e., movable) target objects as well as changing lighting conditions while requiring no prior color calibration of the projector nor any precomputed environment probing. We automatically and continuously estimate these influences during runtime in a real-time feedback-loop and adjust the projected colors accordingly.

The quality of every dynamic multi-projection mapping system is limited by the quality of the projector to tracking device calibration. Common problems with poor calibration result in noticeable artifacts for the user, such as ghosting and seams. In this work we introduce a new, fully automated calibration algorithm that is tailored to reduce these artifacts, based on consumer-grade hardware. We achieve this goal by repurposing a structured-light scanning setup. A structured-light scanner can generate 3D geometry based on a known intrinsic and extrinsic calibration of its components (projector and RGB camera). We revert this process by providing the resulting 3D model to determine the intrinsic and extrinsic parameters of our setup (including those of a variety of tracking systems). Our system matches features and solves for all parameters in a single pass while respecting the lower quality of our sensory input.

Recent publications and art performances demonstrate amazing results using projection mapping. To our knowledge, there exists no multi-projection system that can project onto non-rigid target geometries. This constrains the applicability and quality for live performances with multiple spectators. Given the cost and complexity of current systems, we present a low-cost easy-to-use markerless non-rigid face multi-projection system. It is based on a non-rigid, dense face tracker and a real-time multi-projection solver adapted to imprecise tracking, geometry and calibration. Using this novel system we produce compelling results with only consumer-grade hardware.