Dynamic Projection Mapping (DPM) necessitates geometric compensation of the projection image based on the position and orientation of moving objects. Additionally, the projector's shallow depth of field results in pronounced defocus blur even with minimal object movement. Achieving delay-free DPM with high image quality requires real-time implementation of geometric compensation and projector deblurring. To meet this demand, we propose a framework comprising two neural components: one for geometric compensation and another for projector deblurring. The former component warps the image by detecting the optical flow of each pixel in both the projection and captured images. The latter component performs real-time sharpening as needed. Ideally, our network's parameters should be trained on data acquired in an actual environment. However, training the network from scratch while executing DPM, which demands real-time image generation, is impractical. Therefore, the network must undergo pre-training. Unfortunately, there are no publicly available large real datasets for DPM due to the diverse image quality degradation patterns. To address this challenge, we propose a realistic synthetic data generation method that numerically models geometric distortion and defocus blur in real-world DPM. Through exhaustive experiments, we have confirmed that the model trained on the proposed dataset achieves projector deblurring in the presence of geometric distortions with a quality comparable to state-of-the-art methods.

This paper proposes a projection system that optically removes the cast shadow in projection mapping. Specifically, we realize the large-aperture (LA) projection using a large-format Fresnel lens to suppress cast shadows by condensing the projection light from a wide viewing angle. However, the resolution and contrast of the projected results are significantly degraded by defocus blur, veiling glare, and stray light caused by the aberration of an LA Fresnel lens. To solve the technical problems, we employ two different approaches: optical and digital image processing methods. First, we introduce a residual projector with a typical aperture lens on the same optical axis as the LA projector, projecting the residual (i.e., high-frequency) components attenuated in the LA projection. These projectors play different roles in shadow suppression and blur compensation, both achieved by projecting simultaneously. Secondly, we optimize the pair of projection images that can balance the shadow suppression and deblurring performance of our projection system. We implemented a proof-of-concept prototype and validated the above-mentioned techniques through projection experiments and a user study.

This paper presents a shadowless projection mapping system for interactive applications in which a target surface is frequently occluded from a projector with a user's body. We propose a delay-free optical solution for this critical problem. Specifically, as the primary technical contribution, we apply a large format retrotransmissive plate to project images onto the target surface from wide viewing angles. We also tackle technical issues unique to the proposed shadowless principle. First, the retrotransmissive optics inevitably suffer from stray light, which leads to significant contrast degradation of the projected result. We propose to block the stray light by covering the retrotransmissive plate with a spatial mask. Because the mask reduces not only the stray light but the achievable luminance of the projected result, we develop a computational algorithm that determines the shape of the mask to balance the image quality. Second, we propose a touch sensing technique by leveraging the optically bidirectional property of the retrotransmissive plate to support interaction between the user and the projected contents on the target object. We implement a proof-of-concept prototype and validate the above-mentioned techniques through experiments.

Since ancient times, a hug has been one of the most basic ways to express emotions and has played an important role in building relationships between people. On the other hand, social robots that are designed to provide mental health care to patients have been attracting great attention, and hugging between humans and robots is becoming more and more popular. In this study, we propose a huggable robot that allows intimate interactions between humans and robots. Our robot is based on a tensegrity structure, which is composed of rigid elements connected by springs, and the structure allows the robot to flexibly respond to external force from a hugging interaction, and express various emotions through its movements. In addition, we conducted user experiments and explored an interaction design for the affective movement of the proposed robot. Through the experiments, we confirm that a robot with a tensegrity structure can be used for a hug interaction and it has a large possibility for emotional interaction.

Projector deblurring is an important technology for dynamic projection mapping (PM), where the distance between a projector and a projection surface changes in time. However, conventional projector deblurring techniques do not support dynamic PM because they need to project calibration patterns to estimate the amount of defocus blur each time the surface moves. We present a deep neural network that can compensate for defocus blur in dynamic PM. The primary contribution of this paper is a unique network structure that consists of an extractor and a generator. The extractor explicitly estimates a defocus blur map and a luminance attenuation map. These maps are then injected into the middle layers of the generator network that computes the compensation image. We also propose a pseudo-projection technique for synthesizing physically plausible training data, considering the geometric misregistration that potentially happens in actual PM systems. We conducted simulation and actual PM experiments and confirmed that: (1) the proposed network structure is more suitable than a simple, more general structure for projector deblurring; (2) the network trained with the proposed pseudo-projection technique can compensate projection images for defocus blur artifacts in dynamic PM; and (3) the network supports the translation speed of the surface movement within a certain range that covers normal human motions.