Projection mapping (PM) is useful in the product design process, since it seamlessly bridges a physical mockup and its digital twin by allowing designers to interactively explore new textures, colors, and shapes without the need to create new physical mockups. While PM has proven effective for car interior design, previous research focused solely on supporting the design of dashboards and instrument panels, neglecting evaluation in realistic driving scenarios. This paper introduces a self-contained car interior design support system that extends beyond the dashboard to include the A-pillars. Additionally, to enable designers to evaluate their designs in authentic driving conditions, we integrate a driving simulator, complete with a motion platform, into the PM system. Through the construction of a prototype, we demonstrate the feasibility of our proposed system. Finally, through user studies, we derive guidelines for PM-based car interior design to optimize the user experience.

Live streaming, a dynamic medium that merges real-time audiovisual content with interactive text-based chat, presents unique challenges for maintaining viewer engagement and ensuring streamers' well-being.This study introduces a multi-criteria evaluation framework designed to identify response-worthy chats during live streaming.We proposed a system that evaluates chats based on sentiment polarity and intensity, contextual relevance, and topic uniqueness.We also constructed a dataset annotated by human reviewers who validates the framework, demonstrating a closer alignment with human preferences compared to single-criterion baselines.This framework not only supports the development of more responsive and engaging live streaming environments but also contributes to the broader field of dialog systems by highlighting the distinct needs of real-time, large-scale conversational contexts.

Dynamic Projection Mapping (DPM) necessitates geometric compensation of the projection image based on the position and orientation of moving objects. Additionally, the projector's shallow depth of field results in pronounced defocus blur even with minimal object movement. Achieving delay-free DPM with high image quality requires real-time implementation of geometric compensation and projector deblurring. To meet this demand, we propose a framework comprising two neural components: one for geometric compensation and another for projector deblurring. The former component warps the image by detecting the optical flow of each pixel in both the projection and captured images. The latter component performs real-time sharpening as needed. Ideally, our network's parameters should be trained on data acquired in an actual environment. However, training the network from scratch while executing DPM, which demands real-time image generation, is impractical. Therefore, the network must undergo pre-training. Unfortunately, there are no publicly available large real datasets for DPM due to the diverse image quality degradation patterns. To address this challenge, we propose a realistic synthetic data generation method that numerically models geometric distortion and defocus blur in real-world DPM. Through exhaustive experiments, we have confirmed that the model trained on the proposed dataset achieves projector deblurring in the presence of geometric distortions with a quality comparable to state-of-the-art methods.

This paper proposes a projection system that optically removes the cast shadow in projection mapping. Specifically, we realize the large-aperture (LA) projection using a large-format Fresnel lens to suppress cast shadows by condensing the projection light from a wide viewing angle. However, the resolution and contrast of the projected results are significantly degraded by defocus blur, veiling glare, and stray light caused by the aberration of an LA Fresnel lens. To solve the technical problems, we employ two different approaches: optical and digital image processing methods. First, we introduce a residual projector with a typical aperture lens on the same optical axis as the LA projector, projecting the residual (i.e., high-frequency) components attenuated in the LA projection. These projectors play different roles in shadow suppression and blur compensation, both achieved by projecting simultaneously. Secondly, we optimize the pair of projection images that can balance the shadow suppression and deblurring performance of our projection system. We implemented a proof-of-concept prototype and validated the above-mentioned techniques through projection experiments and a user study.

Projection mapping (PM) is a technique that enhances the appearance of real-world surfaces using projected images, enabling multiple people to view augmentations simultaneously, thereby facilitating communication and collaboration. However, PM typically requires a dark environment to achieve high-quality projections, limiting its practicality. In this paper, we overcome this limitation by replacing conventional room lighting with heterogeneous projectors. These projectors replicate environmental lighting by selectively illuminating the scene, excluding the projection target. Our contributions include a distributed projector optimization framework designed to effectively replicate environmental lighting and the incorporation of a large-aperture projector, in addition to standard projectors, to reduce high-luminance emitted rays and hard shadows-undesirable factors for collaborative tasks in PM. We conducted a series of quantitative and qualitative experiments, including user studies, to validate our approach. Our findings demonstrate t hat our projector-based lighting system significantly enhancesthe contrast and realism of PM results even under e nvironmental lighting compared to typical lights. Furthermore, our method facilitates a substantial shift in the perceived color mode from the undesirable aperture-color mode, where observers perceive the projected object as self-luminous, to the surface-color mode in PM.

Accurately simulating and designing the vibro-acoustic performance of 3D printed structures is a challenging task due to the lack of material elastic properties, particularly when designing without specialized measurement instruments. To tackle this issue, we propose a numerical-experimental estimation method based on modal impact sound, which only requires a microphone for measurement. Specifically, we established a modal impact sound simulation pipeline that considers the material anisotropy and estimated the material elastic property by minimizing the residuals between the simulated and recorded modal impact sound features. Our method outperforms the state-of-the-art modal-sound approach in terms of convergence speed and accuracy, and achieves an average modal-frequency residual of 0.66% across two materials and six infills. We also designed and manufactured a xylophone using the estimated material elasticity for further demonstration and verification.

This paper presents a shadowless projection mapping system for interactive applications in which a target surface is frequently occluded from a projector with a user's body. We propose a delay-free optical solution for this critical problem. Specifically, as the primary technical contribution, we apply a large format retrotransmissive plate to project images onto the target surface from wide viewing angles. We also tackle technical issues unique to the proposed shadowless principle. First, the retrotransmissive optics inevitably suffer from stray light, which leads to significant contrast degradation of the projected result. We propose to block the stray light by covering the retrotransmissive plate with a spatial mask. Because the mask reduces not only the stray light but the achievable luminance of the projected result, we develop a computational algorithm that determines the shape of the mask to balance the image quality. Second, we propose a touch sensing technique by leveraging the optically bidirectional property of the retrotransmissive plate to support interaction between the user and the projected contents on the target object. We implement a proof-of-concept prototype and validate the above-mentioned techniques through experiments.

This article proposes HaptoMapping, a projection-based visuo-haptic augmented reality (VHAR) system, that can render visual and haptic content independently and present consistent visuo-haptic sensations on physical surfaces. HaptoMapping controls wearable haptic displays by embedded control signals that are imperceptible to the user in projected images using a pixel-level visible light communication technique. The prototype system is comprised of a high-speed projector and three types of haptic devices-finger worn, stylus, and arm mounted. The finger-worn and stylus devices present vibrotactile sensations to a user's fingertips. The arm-mounted device presents stroking sensations on a user's forearm using arrayed actuators with a synchronized hand projection mapping. We identified that the developed system's maximum latency of haptic from visual sensations was 93.4 ms. We conducted user studies on the latency perception of our VHAR system. The results revealed that the developed haptic devices can present haptic sensations without user-perceivable latencies, and the visual-haptic latency tolerance of our VHAR system was 100, 159, 500 ms for the finger-worn, stylus, and arm-mounted devices, respectively. Another user study with the arm-mounted device discovered that the visuo-haptic stroking system maintained both continuity and pleasantness when the spacing between each substrate was relatively sparse, such as 20 mm, and significantly improved both the continuity and pleasantness at 80 and 150 mm/s when compared to the haptic only stroking system. Lastly, we introduced four potential applications in daily scenes. Our system methodology allows for a wide range of VHAR application design without concern for latency and misalignment effects.

This paper presents a monocular projector-camera (procam) system using modular architecture based on relay optics. Conventional coaxial procam systems cannot support (1) online changes to lens settings (zoom and focus) and (2) wide-angle projection mapping. We develop design guidelines for a proposed procam system that would solve these restrictions and address the proposed system's unique technical issue of crosstalk between the camera and projector pixels. We conducted experiments using prototypes to validate the feasibility of the proposed framework. First, we confirmed that the proposed crosstalk reduction technique worked well. Second, we found our technique could achieve correct alignment of a projected image onto a moving surface while changing the zoom and focus of the objective lens. The monocular procam system also achieved radiometric compensation where a surface texture was visually concealed by pixel-wise control of a projection color based on the captured results of offline color pattern projections. Finally, we demonstrated the high expandability of our modular architecture, through the creation of a high dynamic range projection.

As covid-19 spread worldwide, the growing trend to reduce people’s mobility and the establishment of new lifestyles such as remote work, contactless communication systems have been required. Using a projected hand makes it possible to realize remote communication with a sense of presence without loading a user’s physical body. However, conventional projected hand systems have a problem that the relative brightness of the hand image in a bright living room is low. Therefore, we propose a configuration that combines a video call system and a contour-based projected hand by a laser scanning projector. In addition, we incorporate a haptic feedback system with a tablet PC to improve the sense of contact with remote objects. To enhance the sense of contact in remote areas, we focus on providing the transition between hovering and contact states of the contour-based projected hand and users’ hands. According to the experiment, we confirmed that there is a possibility to express hovering and contact even with simple swinging flat-panel feedback and visual effects of the contour-based projected hand.

We present interiqr, a method that utilizes the infill parameter in the 3D printing process to embed information inside the food that is difficult to recognize with the human eye. Our key idea is to utilize the air space or secondary materials to generate a specific pattern inside the food without changing the model geometry. As a result, our method exploits the patterns that appear as hidden edible tags to store the data and simultaneously adds them to a 3D printing pipeline. Our contribution also includes the framework that connects the user with a data-embedding interface through the food 3D printing process, and the decoding system allows the user to decode the information inside the 3D printed food through backlight illumination and a simple image processing technique. Finally, we evaluate the usability of our method under different settings and demonstrate our method through the example application scenarios.

Abstract Nature evolves structures like honeycombs at optimized performance with limited material. These efficient structures can be artificially created with the collaboration of structural topology optimization and additive manufacturing. However, the extensive computation cost of topology optimization causes low mesh resolution, long solving time, and rough boundaries that fail to match the requirements for meeting the growing personal fabrication demands and printing capability. Therefore, we propose the neural synthesizing topology optimization that leverages a self‐supervised coordinate‐based network to optimize structures with significantly shorter computation time, where the network encodes the structural material layout as an implicit function of coordinates. Continuous solution space is further generated from optimization tasks under varying boundary conditions or constraints for users' instant inference of novel solutions. We demonstrate the system's efficacy for a broad usage scenario through numerical experiments and 3D printing.

Projector deblurring is an important technology for dynamic projection mapping (PM), where the distance between a projector and a projection surface changes in time. However, conventional projector deblurring techniques do not support dynamic PM because they need to project calibration patterns to estimate the amount of defocus blur each time the surface moves. We present a deep neural network that can compensate for defocus blur in dynamic PM. The primary contribution of this paper is a unique network structure that consists of an extractor and a generator. The extractor explicitly estimates a defocus blur map and a luminance attenuation map. These maps are then injected into the middle layers of the generator network that computes the compensation image. We also propose a pseudo-projection technique for synthesizing physically plausible training data, considering the geometric misregistration that potentially happens in actual PM systems. We conducted simulation and actual PM experiments and confirmed that: (1) the proposed network structure is more suitable than a simple, more general structure for projector deblurring; (2) the network trained with the proposed pseudo-projection technique can compensate projection images for defocus blur artifacts in dynamic PM; and (3) the network supports the translation speed of the surface movement within a certain range that covers normal human motions.

Stereoscopic projection mapping (PM) allows a user to see a three-dimensional (3D) computer-generated (CG) object floating over physical surfaces of arbitrary shapes around us using projected imagery. However, the current stereoscopic PM technology only satisfies binocular cues and is not capable of providing correct focus cues, which causes a vergence-accommodation conflict (VAC). Therefore, we propose a multifocal approach to mitigate VAC in stereoscopic PM. Our primary technical contribution is to attach electrically focus-tunable lenses (ETLs) to active shutter glasses to control both vergence and accommodation. Specifically, we apply fast and periodical focal sweeps to the ETLs, which causes the "virtual image" (as an optical term) of a scene observed through the ETLs to move back and forth during each sweep period. A 3D CG object is projected from a synchronized high-speed projector only when the virtual image of the projected imagery is located at a desired distance. This provides an observer with the correct focus cues required. In this study, we solve three technical issues that are unique to stereoscopic PM: (1) The 3D CG object is displayed on non-planar and even moving surfaces; (2) the physical surfaces need to be shown without the focus modulation; (3) the shutter glasses additionally need to be synchronized with the ETLs and the projector. We also develop a novel compensation technique to deal with the "lens breathing" artifact that varies the retinal size of the virtual image through focal length modulation. Further, using a proof-of-concept prototype, we demonstrate that our technique can present the virtual image of a target 3D CG object at the correct depth. Finally, we validate the advantage provided by our technique by comparing it with conventional stereoscopic PM using a user study on a depth-matching task.

Intrinsic projector calibration is essential in projection mapping (PM) applications, especially in dynamic PM. However, due to the shallow depth-of-field (DOF) of a projector, more work is needed to ensure accurate calibration. We aim to estimate the intrinsic parameters of a projector while avoiding the limitation of shallow DOF. As the core of our technique, we present a practical calibration device that requires a minimal working volume directly in front of the projector lens regardless of the projector's focusing distance and aperture size. The device consists of a flat-bed scanner and pinhole-array masks. For calibration, a projector projects a series of structured light patterns in the device. The pinholes directionally decompose the structured light, and only the projected rays that pass through the pinholes hit the scanner plane. For each pinhole, we extract a ray passing through the optical center of the projector. Consequently, we regard the projector as a pinhole projector that projects the extracted rays only, and we calibrate the projector by applying the standard camera calibration technique, which assumes a pinhole camera model. Using a proof-of-concept prototype, we demonstrate that our technique can calibrate projectors with different focusing distances and aperture sizes at the same accuracy as a conventional method. Finally, we confirm that our technique can provide intrinsic parameters accurate enough for a dynamic PM application, even when a projector is placed too far from a projection target for a conventional method to calibrate the projector using a fiducial object of reasonable size.

Personalization of eating such that everyone consumes only what they need allows improving our management of food waste. In this paper, we explore the use of food 3D printing to create perceptual illusions for controlling the level of perceived satiety given a defined amount of calories. We present FoodFab, a system that allows users to control their food intake through modifying a food's internal structure via two 3D printing parameters: infill pattern and infill density. In two experiments with a total of 30 participants, we studied the effect of these parameters on users' chewing time that is known to affect people's feeling of satiety. Our results show that we can indeed modify the chewing time by varying infill pattern and density, and thus control perceived satiety. Based on the results, we propose two computational models and integrate them into a user interface that simplifies the creation of personalized food structures.

The appearance of fabric motion is suggested to affect the human perception of bending stiffness. This study presents a novel spatial augmented reality, or projection mapping, approach that can visually manipulate the perceived bending stiffness of a fabric. Particularly, we proposed a flow enhancement method that can change the apparent fabric motion by using a simple optical flow analysis technique rather than complex physical simulations for interactive applications. Through a psychophysical experiment, we investigated the relationship between the magnification factor of our flow enhancement and the perceived bending stiffness of a fabric. Furthermore, we constructed a prototype application system that allows users to control the stiffness of a fabric without changing the actual physical fabric. By evaluating the prototype, we confirmed that the proposed technique can manipulate the perceived stiffness of various materials (i.e., cotton, polyester, and mixed cotton and linen) at an average accuracy of 90.3 percent.

This paper presents a novel active marker for dynamic projection mapping (PM) that emits a temporal blinking pattern of infrared (IR) light representing its ID. We used a multi-material three dimensional (3D) printer to fabricate a projection object with optical fibers that can guide IR light from LEDs attached on the bottom of the object. The aperture of an optical fiber is typically very small; thus, it is unnoticeable to human observers under projection and can be placed on a strongly curved part of a projection surface. In addition, the working range of our system can be larger than previous marker-based methods as the blinking patterns can theoretically be recognized by a camera placed at a wide range of distances from markers. We propose an automatic marker placement algorithm to spread multiple active markers over the surface of a projection object such that its pose can be robustly estimated using captured images from arbitrary directions. We also propose an optimization framework for determining the routes of the optical fibers in such a way that collisions of the fibers can be avoided while minimizing the loss of light intensity in the fibers. Through experiments conducted using three fabricated objects containing strongly curved surfaces, we confirmed that the proposed method can achieve accurate dynamic PMs in a significantly wide working range.

Aiming at realizing novel vision augmentation experiences, this paper proposes the IlluminatedFocus technique, which spatially defocuses real-world appearances regardless of the distance from the user's eyes to observed real objects. With the proposed technique, a part of a real object in an image appears blurred, while the fine details of the other part at the same distance remain visible. We apply Electrically Focus-Tunable Lenses (ETL) as eyeglasses and a synchronized high-speed projector as illumination for a real scene. We periodically modulate the focal lengths of the glasses (focal sweep) at more than 60 Hz so that a wearer cannot perceive the modulation. A part of the scene to appear focused is illuminated by the projector when it is in focus of the user's eyes, while another part to appear blurred is illuminated when it is out of the focus. As the basis of our spatial focus control, we build mathematical models to predict the range of distance from the ETL within which real objects become blurred on the retina of a user. Based on the blur range, we discuss a design guideline for effective illumination timing and focal sweep range. We also model the apparent size of a real scene altered by the focal length modulation. This leads to an undesirable visible seam between focused and blurred areas. We solve this unique problem by gradually blending the two areas. Finally, we demonstrate the feasibility of our proposal by implementing various vision augmentation applications.

This paper explores how human perceptions, actions, and interactions can be changed through an embodied and active experience of being a smaller person in a real-world environment, which we call an egocentric smaller person experience. We developed a wearable visual translator that provides the perspective of a smaller person by shifting the wearer's eyesight level down to their waist using a head-mounted display and a stereo camera module, while allowing for field of view control through head movements. In this study, we investigated how the developed device can modify the wearer's body representation and experiences based on a field study conducted at a nursing school and museums, and through lab studies. It was observed that the participants changed their perceptions, actions, and interactions because they are considered to have perceived themselves as being smaller. Using this device, designers and teachers can understand the perspectives of other people in an existing environment.

We propose a novel spatial augmented reality (SAR) framework to edit the appearance of physical glossy surfaces. The key idea is utilizing the specular reflection, which was a major distractor in conventional SAR systems. Namely, we spatially manipulate the appearance of an environmental surface, which is observed through the specular reflection. We use a stereoscopic display to present two appearances with disparity on the environmental surface, by which the depth of the specularly reflected visual information corresponds to the glossy surface. We refer to this method as augmented environment mapping (AEM). The paper describes its principle, followed by three different implementation approaches inspired by typical virtual and augmented reality approaches. We confirmed the feasibility of AEM through both quantitative and qualitative experiments using prototype systems.

Presentation of odors is considered to be important as a means for giving a sense of presence. However, the basic odors have not been established to generate any kinds of odors by combining them, like the three primary colors in vision, the basic tastes in gustation. In order to present various kinds of odors, in general, it is necessary to prepare each corresponding odorant. In this research, an odor modulation method is proposed based on a cross-modal effect between olfaction and thermal sensation, which might be able to decrease the number of odorants used to generate odors presented to the user. The experimental results are reported to show that sensation of odors can be modulated by presenting warm/cool air with the odors even if the same odors are presented to the subjects.

Shadowless Projector is projection mapping system in which a shadow (more specifically, umbra) does not suffer the projected result. A typical shadow removal technique used a multiple overlapping projection system. In this paper, we propose a shadow-less projection method with single projector. Inspired by a surgical light system that does not cast shadows on patients' bodies in clinical practice, we apply a special optical system that consists of methodically positioned vertical mirrors. This optical system works as a large aperture lens, it is impossible to block all projected ray by a small object such as a hand. Consequently, only penumbra is caused, which leads to a shadow-less projection.

Car interior design, such as dashboard, broadly consists of two parts. One is shape design, where the processes of 2D drawing, 3D modeling and evaluation with full-scale mockups are iterated, which takes a massive amount of time and cost. The other is material design such as surface property tuning, where designers compare material samples. This way, however, has a limitation on the number of material samples to compare and does not allow applying of the samples of interest to the whole mockups in early phases. In this paper, we apply projection mapping technique to boost the design process by altering the appearance of the surface of projected objects and enabling various shape and material evaluations in early phases. Our proposed system uses multiple projectors, one of which is 4K projector to reproduce fine leather surface. Utilizing physiological and psychological depth cues, the system allows the user to perceive the projected mockup as deformed. Psychological experiments confirm that users perceive deformation and have controlled impression on leather reproduced with certain parameters. In addition, we discuss the usability of the proposed system as a support system of car interior design.

We present fARFEEL, a remote communication system that provides visuo-haptic feedback allows a local user to feel touching distant objects. The system allows the local and remote users to communicate by using the projected virtual hand (VH) for the agency of his/her own hands. The necessary haptic information is provided to the non-manipulating hand of the local user that does not bother the manipulation of the projected VH. We also introduce the possible visual stimulus that could potentially provide the sense of the body ownership over the projected VH.

This paper proposes a learning-based quality evaluation framework for inpainted results that does not require any subjectively annotated training data. Image inpainting, which removes and restores unwanted regions in images, is widely acknowledged as a task whose results are quite difficult to evaluate objectively. Thus, existing learning-based image quality assessment (IQA) methods for inpainting require subjectively annotated data for training. However, subjective annotation requires huge cost and subjects’ judgment occasionally differs from person to person in accordance with the judgment criteria. To overcome these difficulties, the proposed framework generates and uses simulated failure results of inpainted images whose subjective qualities are controlled as the training data. We also propose a masking method for generating training data towards fully automated training data generation. These approaches make it possible to successfully estimate better inpainted images, even though the task is quite subjective. To demonstrate the effectiveness of our approach, we test our algorithm with various datasets and show it outperforms existing IQA methods for inpainting.

This paper presents an approach for non-contact haptic augmentation with spatial augmented reality (SAR). We construct a thermo-visual projection system by combining a standard RGB projector and a fabricated infrared (IR) projector. The primary contribution of this paper is that we conduct thorough psychophysical experiments to investigate a design guideline for spatiotemporal projection patterns for both RGB and IR projectors to render a warm object with high presence. We develop application systems to evaluate the validity of the proposed system and design guideline. The evaluation results demonstrate that the proposed system can render warm objects with significantly higher presence than a standard SAR system.

A block machine has been proposed as effective systems to support attack practice in volleyball. A method for manipulating the system by tablet operation has been established, and the use of the system has been shown to improve practice effectiveness. However, due to the requirement of manual operation, it has been reported that the efficiency of practice decreases due to the error between the block position and the attack position. Therefore, in order to operate the block machine automatically, we propose a method to acquire the player position from monocular video in real time and predict the attack position.

We propose a visual marker embedding method for the pose estimation of a projection surface to correctly map projected images onto the surface. Assuming that the surface is fabricated by a full-color or multi-material three-dimensional (3D) printer, we propose to automatically embed visual markers on the surface with mechanical accuracy. The appearance of the marker is designed such that the marker is detected by infrared cameras even when printed on a non-planar surface while its appearance can be diminished by the projection to be as imperceptible as possible to human observers. The marker placement is optimized using a genetic algorithm to maximize the number of valid viewpoints from which the pose of the object can be estimated correctly using a stereo camera system. We also propose a radiometric compensation technique to quickly diminish the marker appearance. Experimental results confirm that the pose of projection objects are correctly estimated while the appearance of the markers was diminished to an imperceptible level. At the same time, we confirmed the limitations of the current method; only one object can be handled, and pose estimation is not performed at interactive frame rates. Finally, we demonstrate the proposed technique to show that it works successfully for various surface shapes and target textures.

This paper presents a system that consists of three robots to imitate the motion of top volleyball blockers. In a volleyball match, in order to score by spiking, it is essential to improve the spike decision rate of each spiker. To increase the spike decision rates, iterative spiking training with actual blockers is required. Therefore, in this study, a block machine system was developed that can be continuously used in an actual practice field to improve attack practice. In order to achieve the required operating speed and mechanical strength each robot has five degrees of freedom. This robot performs high speed movement on 9 m rails that are arranged in parallel with the volleyball net. In addition, an application with a graphical user interface to enable a coach to manipulate these robots was developed. It enables the coach to control block motions and change the parameters such as the robots' positions and operation timing. Through practical use in the practice field, the effectiveness of this system was confirmed.

This paper presents a novel projected pixel localization principle for online geometric registration in dynamic projection mapping applications. We propose applying a time measurement of a laser projector raster-scanning beam using a photosensor to estimate its position while the projector displays meaningful visual information to human observers. Based on this principle, we develop two types of position estimation techniques. One estimates the position of a projected beam when it directly illuminates a photosensor. The other localizes a beam by measuring the reflection from a retro-reflective marker with the photosensor placed in the optical path of the projector. We conduct system evaluations using prototypes to validate this method as well as to confirm the applicability of our principle. In addition, we discuss the technical limitations of the prototypes based on the evaluation results. Finally, we build several dynamic projection mapping applications to demonstrate the feasibility of our principle.

This paper presents the extended work on LazyNav, a head-free, eyes-free and hands-free mid-air ground navigation control model presented at the IEEE 3D User Interfaces (3DUI) 2015, in particular with a new application to the head-mounted display (HMD). Our mid-air interaction metaphor makes use of only a single pair of the remaining tracked body elements to tailor the navigation. Therefore, the user can navigate in the scene while still being able to perform other interactions with her hands and head, e.g., carrying a bag, grasping a cup of coffee, or observing the content by moving her eyes and locally rotating her head. We design several body motions for navigation by considering the use of non-critical body parts and develop assumptions about ground navigation techniques. Through the user studies, we investigate the motions that are easy to discover, easy to control, socially acceptable, accurate and not tiring. Finally, we evaluate the desired ground navigation features with a prototype application in both a large display (LD) and a HMD navigation scenarios. We highlight several recommendations for designing a particular mid-air ground navigation technique for a LD and a HMD.

We propose a novel inter-reflection compensation technique for immersive projection displays wherein we spatially modulate the reflectance pattern on the screen to improve the compensation performance of conventional methods. As the luminance of light reflected on a projection surface is mathematically represented as the multiplication of the illuminance of incident light and the surface reflectance, we can reduce undesirable intensity elevation because of inter-reflections by decreasing surface reflectance. Based on this principle, we improve conventional inter-reflection compensation techniques by applying reflectance pattern modulation. We realize spatial reflectance modulation of a projection screen by painting it with a photochromic compound, which changes its color (i.e., the reflectance of the screen) when ultraviolet (UV) light is applied and by controlling UV irradiation with a UV LED array placed behind the screen. The main contribution of this paper is a computational model to optimize a reflectance pattern for the accurate reproduction of a target appearance by decreasing the intensity elevation caused by inter-reflection while maintaining the maximum intensity of the target appearance. Through simulation and physical experiments, we demonstrate the feasibility of the proposed model and confirm its advantage over conventional methods.

Many researchers are trying to realize a pseudo-haptic system which can visually manipulate a user's haptic shape perception when touching a physical object. In this paper, we focus on manipulating the perceived surface shape of a curved object when touching it with an index finger, by visually deforming it's surface shape and displacing the visual representation of the user's index finger as like s/he is touching the deformed surface, using spatial augmented reality. Experiments were conducted with a projection system to confirm the effect of the visual feedback for manipulating the perceived shape of curved surface. The results supported the proposed concept.

The ideal AR x-ray vision should enable users to clearly observe and grasp not only occludees, but also occluders. We propose a novel selective visualization method of both occludee and oc-cluder layers with dynamic opacity depending on the user's gaze depth. Using the gaze depth as a trigger to select the layers has a essential advantage over using other gestures or spoken commands in the sense of avoiding collision between user's intentional commands and unintentional actions. Our experiment by a visual paired-comparison task shows that our method has achieved a 20% higher success rate, and significantly reduced 30% of the average task completion time than a non-selective method using a constant and half transparency.

This paper presents a novel technique to reduce the motion blur artifacts of foveal projection in a dynamic focus-plus-context (DF+C) display. The DF+C display is generally configured with multiple projectors and provides a nonuniform spatial resolution that consists of high-resolution (hi-res) regions (foveal projection) and low-resolution regions (peripheral projection). A serious problem of the DF+C display is motion blur, which inevitably occurs when a foveal projection is moved by a pan-tilt mirror or gantry. We propose a solution that reduces the motion blur artifacts, and evaluate how this solution improves the image quality using both qualitative and quantitative experiments. Our proposed method defines an error function to assess the displayed image quality as the difference between an original hi-res image and the displayed image by considering the nonuniform spatial property of human visual acuity. Then, it decides the set of positions and moving techniques of foveal projections so that the sum of errors during a video sequence is minimized. Through experiment, we confirmed that the proposed method can provide a better image quality and significantly improve the motion blur artifacts when compared with a conventional DF+C display.

A simple and cost-efficient method for extending a projector's depth-of-field (DOF) is proposed. By leveraging liquid lens technology, we can periodically modulate the focal length of a projector at a frequency that is higher than the critical flicker fusion (CFF) frequency. Fast periodic focal length modulation results in forward and backward sweeping of focusing distance. Fast focal sweep projection makes the point spread function (PSF) of each projected pixel integrated over a sweep period (IPSF; integrated PSF) nearly invariant to the distance from the projector to the projection surface as long as it is positioned within sweep range. This modulation is not perceivable by human observers. Once we compensate projection images for the IPSF, the projected results can be focused at any point within the range. Consequently, the proposed method requires only a single offline PSF measurement; thus, it is an open-loop process. We have proved the approximate invariance of the projector's IPSF both numerically and experimentally. Through experiments using a prototype system, we have confirmed that the image quality of the proposed method is superior to that of normal projection with fixed focal length. In addition, we demonstrate that a structured light pattern projection technique using the proposed method can measure the shape of an object with large depth variances more accurately than normal projection techniques.

We present SoftAR, a novel spatial augmented reality (AR) technique based on a pseudo-haptics mechanism that visually manipulates the sense of softness perceived by a user pushing a soft physical object. Considering the limitations of projection-based approaches that change only the surface appearance of a physical object, we propose two projection visual effects, i.e., surface deformation effect (SDE) and body appearance effect (BAE), on the basis of the observations of humans pushing physical objects. The SDE visualizes a two-dimensional deformation of the object surface with a controlled softness parameter, and BAE changes the color of the pushing hand. Through psychophysical experiments, we confirm that the SDE can manipulate softness perception such that the participant perceives significantly greater softness than the actual softness. Furthermore, fBAE, in which BAE is applied only for the finger area, significantly enhances manipulation of the perception of softness. We create a computational model that estimates perceived softness when SDE+fBAE is applied. We construct a prototype SoftAR system in which two application frameworks are implemented. The softness adjustment allows a user to adjust the softness parameter of a physical object, and the softness transfer allows the user to replace the softness with that of another object.

Choosing the appropriate soft/hard material is important for designing a product such as sofa or bed, but typically limited by the number of physical materials that the designer owns. Pseudo-haptic feedback is an alternative way that enables designer to roughly simulate material properties (e.g., softness, hardness) by only generating the visual illusion. However, the current technique is limited within video see-through augmented reality, in which the user interact in a real space while looking at a virtual space. This paper explores the possibility to realize pseudo-haptic feedback for touching objects in spatial augmented reality. We investigate and compare effects of visually superimposing a projection graphics onto the surface of a touched object and the fingernail/finger for changing the surface tactile perception. The potential of our method is discussed through a preliminary user study.

In this paper, we propose a novel radiometric compensation method that applies a high-spatial-resolution camera to a projector-camera system to reduce the artifacts around the regions where the reflectance of the projection surface changes steeply. The proposed method measures the reflection in the region of a single projector pixel on a projection surface with multiple camera pixels. From the measurement, it computes multiple color-mixing matrices, each of which represents a color space conversion between each camera and the projector pixels. Using these matrices, we calculate the optimal projection color by applying the linear least squares method, so that the displayed color in the projector pixel region is as close as possible to the target appearance. Through projection experiments, we confirm that our proposed method reduces the artifacts around the regions where the reflectance changes steeply, when compared with other conventional compensation methods.

We introduce a shadow-based interface for interactive tabletops. The proposed interface allows a user to browse graphical information by casting the shadow of his/her body, such as a hand, on a tabletop surface. Central to our technique is a new optical design that utilizes polarization in addition to the additive nature of light so that the desired graphical information is displayed only in a shadow area on a tabletop surface. In other words, our technique conceals the graphical information on surfaces other than the shadow area, such as the surface of the occluder and non-shadow areas on the tabletop surface. We combine the proposed shadow-based interface with a multi-touch detection technique to realize a novel interaction technique for interactive tabletops. We implemented a prototype system and conducted proof-of-concept experiments along with a quantitative evaluation to assess the feasibility of the proposed optical design. Finally, we showed implemented application systems of the proposed shadow-based interface.

In this paper we propose a novel depth measurement method by fusing depth from defocus (DFD) and stereo. One of the problems of passive stereo method is the difficulty of finding correct correspondence between images when an object has a repetitive pattern or edges parallel to the epipolar line. On the other hand, the accuracy of DFD method is inherently limited by the effective diameter of the lens. Therefore, we propose the fusion of stereo method and DFD by giving different focus distances for left and right cameras of a stereo camera with coded apertures. Two types of depth cues, defocus and disparity, are naturally integrated by the magnification and phase shift of a single point spread function (PSF) per camera. In this paper we give the proof of the proportional relationship between the diameter of defocus and disparity which makes the calibration easy. We also show the outstanding performance of our method which has both advantages of two depth cues through simulation and actual experiments.

This research aims at constructing an intrinsic tactile interface for ubiquitous haptic AR. In order to avoid obstruction of natural object manipulation with the fingertip, we focused on electrotactile interface. We proposed a time division multiplexing technique with compensating processes allowing detection and augmentation of touch feeling through shared electrodes. In this paper, we investigated the performance as a sensor and display by a pressing task. First, we obtained an optimized parameter for the multiplexing. Next, the stimulus effect on the sensor performance was tested. Finally, we confirmed that the proposed interface has a potential to modulate user's cutaneous feeling.

This paper shows a random and distinct shape of each pixel improves the performance of super-resolution using multiple input images. Since the spatial light sensitivity distribution in each pixel of an image sensor is rectangular and identical, the process of imaging is equivalent to the point sampling of blurred image which is a result of convolution of a rectangle with the original image. The convolution results in a loss of the high spatial frequency component of the original image, which limits the performance of super-resolution. Thus, we sprayed a fine-grained black powder on an image sensor to give a random code to the spatial light sensitivity distribution in each pixel. This approach was combined with a reconstruction technique based on sparse regularization, which is commonly used in compressed sensing, in an experiment with an actual setup. A high-resolution image was reconstructed from a limited number of input images and the performance of super-resolution was significantly improved.

This study addresses the problem of geometric consistency between displayed images and real scenes in augmented reality using a video see-through hand-held display or tablet. To solve this problem, we present approximated user-perspective images rendered by homography transformation of camera images. Homography approximation has major advantages not only in terms of computational costs, but also in the quality of image rendering. However, it can lead to an inconsistency between the real image and virtual objects. This study also introduces a variety of rendering methods for virtual objects and discusses the differences between them. We implemented two prototypes and designed three types of user studies on matching tasks between real scenes and displayed images. We have confirmed that the proposed method works in real time on an off-the-shelf tablet. Our pilot tests show the potential to improve users' visibility, even in real environments, by using our method.

In this paper, we present a high-speed and high-accuracy method for estimating scene flow with a Kinect. The region where motion objects may exist is detected by calculating the difference between three frames. From this region, the bounding box of the moving object is then determined by segmenting the depth image. We estimate the scene flow by finding the corresponding key-point pairs between frames. In the process of establishing the correspondence between key-points, we use the SURF feature and the depth with variable cuboid search algorithm. We have compared our method with other existing methods by using a common test data set.

This paper presents a new label layout technique for projection-based augmented reality (AR) that determines the placement of each label directly projected onto an associated physical object with a surface that is normally inappropriate for projection (i.e., nonplanar and textured). Central to our technique is a new legibility estimation method that evaluates how easily people can read projected characters from arbitrary viewpoints. The estimation method relies on the results of a psychophysical study that we conducted to investigate the legibility of projected characters on various types of surfaces that deform their shapes, decrease their contrasts, or cast shadows on them. Our technique computes a label layout by minimizing the energy function using a genetic algorithm (GA). The terms in the function quantitatively evaluate different aspects of the layout quality. Conventional label layout solvers evaluate anchor regions and leader lines. In addition to these evaluations, we design our energy function to deal with the following unique factors, which are inherent in projection-based AR applications: the estimated legibility value and the disconnection of the projected leader line. The results of our subjective experiment showed that the proposed technique could significantly improve the projected label layout.

This paper introduces a new high dynamic range (HDR) display system that generates a physical 3D HDR image without using stereoscopic methods. To boost contrast beyond that obtained using either a hardcopy or a projector, we employ a multiprojection system to superimpose images onto a textured solid hardcopy that is output by a 3D printer or a rapid prototyping machine. We introduce two basic techniques for our 3D HDR display. The first technique computes an optimal placement of projectors so that projected images cover the hardcopy's entire surface while maximizing image quality. The second technique allows a user to place the projectors near the computed optimal position by projecting from each projector images that act as visual guides. Through proof-of-concept experiments, we were able to modulate luminance and chrominance with a registration error of less than 3 mm. The physical contrast ratio obtained using our method was approximately 5,000:1, while it was 5:1 in the case of viewing the 3D printout under environmental light and 1,000:1 in the case of using the projectors to project the image on regular screens.

This paper presents a novel dynamic control of multiple focal-plane projections. Our approach multiplexes the projectors' focal-planes so that all the displayed images are focused. The projectors are placed at various locations with various directions in the scene so that there is no occlusion of the projection light even for dynamically moving objects. Our approach compensates for defocus blur, oblique blur (loss of high-spatial-frequency components according to the incidence angle of the projection light), and cast shadows of projected images. This is achieved by selecting an optimal projector that can display the finest image for each point of the object surface by comparing the projectors' point spread functions (PSFs). The proposed approach requires geometric calibration to be performed only once in advance. Our approach can compute PSFs without any additional calibrations even when the surface moves if it is tracked by an external sensor. The method is particularly useful in interactive systems in which the object to be projected is tracked.

Exaggerated defocus can not be created with an ordinary compact digital camera because of its tiny sensor size, so it is hard to take pictures that attract a viewer to the main subject. On the other hand, there are many methods for controlling focus and defocus of previously taken pictures. However, most of these methods require purpose-built equipment such as a camera array to take pictures. Therefore, in this paper, we propose a method to create images focused at any depth with arbitrarily blurred background from the set of images taken by a handheld compact digital camera moved randomly. Using our method, it is possible to produce various aesthetic blurs by changing the size, shape or density of the blur kernel. In addition, we confirm the potential of our method through a subjective evaluation of blurred images created by our system.

In computer vision, background subtraction method is widely used to extract a changing region in a scene. However, it is difficult to simply apply this method to a scene with moving background object, because such object may be extracted as a changing region. Therefore, a method has been proposed to estimate both current background image and occluding object region simultaneously by using eigenspace-based background representation. On the other hand, image completion method using eigenspace have been extended to non-linear subspace using kernel trick, however, such existing method takes large computational cost. Therefore, in this paper, we propose a method for rapid simultaneous estimation of a background image and occluded region in non-linear space, using the kernel trick and iterative projection.

The light transport is one of the useful representation of the optical phenomena inside a scene (Sen et al;, 2005 and Seitz et al., 2005). By using this concept, we can regard any complex scene as a simple linear system between incident light and luminance distribution. However, it is difficult to obtain enough sample under varied lighting condition, because the light source has large degrees of freedom. Therefore, in this paper, we propose a fast and efficient sampling method using characteristics of direct and indirect components. Our method can be divided to three stages: at first, we obtain geometric relationships between projector and camera using Gray code. Then repetive dot patterns are projected to acquire direct reflection component in parallel. Finally, stripe pattern in a small rectangular is projected to model the low-frequency indirect light distribution. The system enables to obtain light transport quickly, and dynamic range of indirect component is also improved. Separated representation of direct and indirect components in light transport allows further applications.

This paper presents a novel mixed reality (MR) system for virtually modifying (e.g., denting, engraving, swelling) shape of real objects by using projection of computer-generated shade. Users of this system, which we call HYPERREAL, perceive as if the real object is actually being deformed when they operate the system to modify the shape of the object while only the illumination pattern of the real object is changed. The authors are aiming to apply this technology to product designing field: designers would be able to evaluate and modify form of their product more efficiently and effectively in an intuitive manner using HYPERREAL than conventional design process (typically, computer aided design, or CAD, systems and solid mock-ups) since the system is able to provide users with actuality/presence of real mock-up and flexibility of shape data on a computer system, such as CAD system, all at once.

Searching of spread documents on a real desk is an arduous task. The chaos on the desk makes users’ searching of the desired documents difficult. In this paper, we propose Limpid Desk which supports a document search on a desk by transparentizing the upper layer of a document stack in projection-based mixed reality environments. In the system, the special pattern light which is calculated to compensate the appearances of the upper layer documents as if they are transparent is projected to the stack, and as a result, users can visually access to the lower layer document. We propose a touch sensing method using a thermal image for the input interface of the system. The method realizes that users’ touch areas on real documents can be detected with no worn or hold devices. Hence, users can intuitively select the stack, which they would like to transparentize, by their simple touch gestures. We present three intuitive interaction techniques which allow users to figure out what the lower layer document is without physically removing its upper documents. Since the searching space, the manipulating space, and the display space are completely unified onto a real desk, users can directly access to the lower documents without PC monitor and interface. We claim that this kind of spatial consistency of the manipulation is the key for realizing the intuitive interaction.

We propose Limpid Desk which supports document search on a real desktop with virtual transparentizing of the upper layer of a document stack in projection-based mixed reality (MR)environments. In the system, users can visually access a lower layer document without physically removing the upper documents. This is accomplished by projecting a special pattern of light that is calculated to compensate the appearances of the upper layer documents as if they are transparent. In addition, we propose two types of visual effects for users to cognize the progress of the transparentizing of real documents and to recognize the layer number of the virtually exposing document, and excecute psychological tests to confirm the intuitiveness of these effects. This paper also presents an intuitive document search interaction in the proposed system.

For the real-time recognition of unspecified gestures by an arbitrary person, a comprehensive framework is presented that addresses two important problems in gesture recognition systems: selective attention and processing frame rate. To address the first problem, we propose the Quadruple Visual Interest Point Strategy. No assumptions are made with regard to scale or rotation of visual features, which are computed from dynamically changing regions of interest in a given image sequence. In this paper, each of the visual features is referred to as a visual interest point, to which a probability density function is assigned, and the selection is carried out. To address the second problem, we developed a selective control method to equip the recognition system with self-load monitoring and controlling functionality. Through evaluation experiments, we show that our approach provides robust recognition with respect to such factors as type of clothing, type of gesture, extent of motion trajectories, and individual differences in motion characteristics. In order to indicate the real-time performance and utility aspects of our approach, a gesture video system is developed that demonstrates full video-rate interaction with displayed image objects.

One of the methods achieving mixed reality (MR) displays is the texture projection method using projectors. Another kind of emerging information environment is a wearable information device, which realizes ubiquitous computing. It is very promising to integrate these technologies. Using this kind of fusion system, two or more users can get the same MR environments without using HMD at the same moment. In this demonstration, we propose a wearable MR system with an on-board projector and introduce some applications with this system.

In the MR world, it needs to overlay virtual objects onto real scene with harmonizing the photometric consistency. We developed a new illumination device "LCD cube" for keeping photometric consistency. In the demonstration, we plan to present the method to display high dynamic range light environments by "LCD cube" for image-based lighting. In addition, we performed light reproduction in real-time.

A range image segmentation contest was organized in conjunction with ICPR'2000. The goal is to continue the effort of experimentally evaluating range image segmentation algorithms initiated by Hoover et al. (1996) and Powell et al. (1998). This paper summarizes the results of the contest.

Proposes a new method for reconstructing complete 3D shapes and colors of outdoor objects, especially of archaeological sites, as a reconstructed 3D image can be useful for archaeological research. The method integrates two kinds of data: (1) partial surfaces (inaccurate object surfaces with color information which are obtained by stereogrammetry, each of which has its local geometry fixed on the roughly calibrated stereo cameras), and (2) surveyed points (sampled points whose 3D coordinate values are represented with the world geometry and accurately measured by a sensor). The authors focused on fitting the partial surfaces to the surveyed points, and adopted adjustment processes which contained two feedback loops. The method is applied to an outdoor 3D measurement of the Kamaato archaeological site.

Aa new structured light projection method for measuring 3D and range objects is proposed. The method projects a moving structured light pattern modulated by a time-compression signal, then the objects are captured as a dynamic and spatio-temporal image by a camera. By applying a matched filter to the spatio-temporal image, the ideal slit light is realized even using an actual light projector. According to the extracted slit light and triangulation principle, 3D geometric information of the objects scene are obtained more accurately. By using separable signals which have no correlation with each other, the method is able to project multiple light patterns at different positions simultaneously. It solves the occlusion problem due to a single projector. Some experimental results show its advantages.

We propose a new method to track 3-D motion of an object in real-time, using range images. A very high speed range sensor "Silicon Range Finder" makes it possible to acquire range images at more than 30 frames per second, and rotated to keep the object in the measurable area. Rough but fast results of range image processing controls the angle of the range sensor, and the object centered range images decide the pose of the object. The 3-D CG rendering hardware of the graphics workstation generates range images from the surface model very fast, so it makes the comparison algorithm simple. Tracking speed and accuracies of estimated motion seem to be feasible for the man-machine cooperative systems.

A new color image acquisition system which can measure continuous spectral pixel data in order to apply continuous spectral images to object recognition. For segmentation of continuous spectral image, object contours can be detected by checking the correlation coefficient between neighboring pixels. Also the continuous spectral images can be used as reference image when selecting the most suitable imaging sensor, such as a gray scale camera, a RGB camera or a spectral camera with narrow band-pass filter. When investigating features in spectral distribution, object recognition should be simplified for color objects.

Presents a range imaging VLSI sensor which gives a range image of a scene with high robustness and high accuracy at a video frame rate. No adjustment is required. Photodiodes and operational circuits are integrated into each pixel of the range imaging VLSI sensor to function as analog/digital micro-rangefinders. The 2D array of the micro-rangefinders, the range imaging VLSI sensor, acquires frames of range data in fully parallel manner as a light stripe is swept across a scene. Complementary pairs of photodiodes in each cell realize reliable capturing of range data without being disturbed by temperature drift, ambient light, manufacturing irregularities or other noise. A prototype system has been built using a 24/spl times/24 VLSI array of sensing elements and gives 24/spl times/24 range images at a video frame rate.>

The authors discuss a method of recovering reflectance properties of a surface from a range image given by a range finder and a brightness image given by a standard TV camera. The Torrance-Sparrow model is used for the reflectance model. The model consists of the Lambertian and specular components: its reflectance properties consist of the relative strength between the Lambertian and specular components and specular sharpness as well as light source direction. An iterative least square fitting method is used to obtain these parameters based on the range and brightness images. An input image is segmented into four different parts using the parameters: Lambertian reflection, specular reflection, interreflection, and shadow part. The authors also reconstruct ideal images that consist of only Lambertian or specular reflection.>

A method is presented for recovering reflectance parameters of optically rough surfaces from a range and a brightness image, both of which are generated by a range-finder. The reflectance model of an optically rough surface consists of two components, known as Lambertian and specular components and respectively represented as a cosine and a Gaussian function, and contains the following three basic parameters: the Lambertian strength, specular strength, and specular sharpness. An iterative least-squares fitting method is used to obtain these parameters derived from range and brightness images. Assuming all pixels only contain the Lambertian component, the authors fit the Lambertian function to all the data points. Based on the fitting of these results, they use a threshold derived from a sensor model of the range-finder and exclude those pixels lying outside the bounds of this threshold. To examine the convergence of the algorithm, the authors implemented this algorithm and applied it to several synthesized images. Several experiments using real images demonstrated the applicability of the algorithm.>

A rule-driven recognition system of a 3-D scene using range images is described. The system is constructed with a production system. A scene is described with simple a surface patch network (planar or quadric) by examining the local curvature of objects. The region and edge network is interpreted with IF-THEN-like heuristic rules. The system can recognize smoothly joined curved surfaces and explain the structure of a scene consisting of plural objects. The experiments have been carried out with real range data.>