Dynamic Facial Projection Mapping (DFPM) overlays computer-generated images onto human faces to create immersive experiences that have been used in the makeup and entertainment industries. In this study, we propose two concepts to reduce the misalignment artifacts between projected images and target faces, which is a persistent challenge for DFPM. Our first concept is a high-speed face-tracking method that exploits temporal information. We first introduce a cropped-area-limited inter/extrapolation-based face detection framework, which allows parallel execution with facial landmark detection. We then propose a novel hybrid facial landmark detection method that combines fast Ensemble of Regression Trees (ERT)-based detections and an auxiliary detection. ERT-based detections rapidly produce results in 0.107 ms using temporal information with the support of auxiliary detection to recover from detection errors. To train the facial landmark detection method, we propose an innovative method for simulating high-frame-rate video annotations to address the lack of publicly available high-frame-rate annotated datasets. Our second concept is a lens-shift co-axial projector-camera setup that maintains a high optical alignment with only a 1.274-pixel error between 1 m and 2 m depth. This setup reduces misalignment by applying the same optical designs to the projector and camera without causing large misalignment as in conventional methods. Based on these concepts, we developed a novel high-speed DFPM system that achieves nearly perfect alignment with human visual perception.

High-speed dynamic facial projection mapping (DFPM) is an advanced technology that aims to create perceptual changes in facial appearance by overlapping images based on facial position and shape. Compared to traditional monitor-based augmented reality systems, DFPM offers a higher level of immersion because users can directly observe digital content on their faces. However, DFPM suffers from misalignment issues owing to a slight temporal delay from sensing to projection, which reduces the level of immersion. To the best of our knowledge, no previous study has established the necessary latency requirements to avoid perceptible misalignment and achieve an immersive experience. Furthermore, conventional DFPM works followed latency requirements that were not reported for the DFPM scenario. Therefore, this study measured the latency that provided a just-noticeable difference (JND) in DFPM under different facial motion conditions, using the weighted up-down two-alternative forced-choice method. The results showed that user-perceptible misalignment was influenced by facial motion types and their velocities. Additionally, it was found that an average latency of 3.87 ms was necessary to avoid perceptible misalignment in the DFPM system when the translation speed was 0.5 m/s, which contradicts the commonly held belief regarding the required latency threshold.

Projection mapping using multiple projectors is promising for spatial augmented reality; however, it is difficult to apply it to dynamic scenes. This is because the conventional method decides all pixel intensities of multiple images simultaneously based on the global optimization method, and it is hard to reduce the latency from motion to projection. To mitigate this, we propose a novel method of controlling the intensity based on a pixel-parallel calculation for each projector in real-time with low latency. This parallel calculation leverages the insight that the projected pixels from different projectors in overlapping areas can be approximated independently if the pixel is sufficiently small relative to the surface structure. Additionally, our pixel-parallel calculation method allows a distributed system configuration, such that the number of projectors can be increased to form a network for high scalability. We demonstrate a seamless mapping into dynamic scenes at 360 fps with a 9.5-ms latency using ten cameras and four projectors.