Advances in emerging technologies, such as on-body mechanical actuators and electrical muscle stimulation, have allowed computers to take control over our bodies. This presents opportunities as well as challenges, raising fundamental questions about agency and the role of our bodies when interacting with technology. To advance this research field as a whole, we brought together expert perspectives in a week-long seminar to articulate the grand challenges that should be tackled when it comes to the design of computers’ control over our bodies. These grand challenges span technical, design, user, and ethical aspects. By articulating these grand challenges, we aim to begin initiating a research agenda that positions bodily control not only as a technical feature but as a central, experiential, and ethical concern for future human–computer interaction endeavors.

In social virtual reality (VR) platforms, players can embody “non-human” avatars, which are representations whose appearance or skeletal structure diverge from typical human characteristics. This capability fosters the emergence of distinctive cultures of social interaction. This paper reports on interviews with users who employ such avatars, investigating (1) motivations for their adoption, (2) their impact on social interactions, and (3) challenges encountered when employing them in social contexts. Our findings reveal that users adopt “non-human” avatars both to escape the expectations and norms associated with the human body—thereby enabling more relaxed social communication—and to gain access to new forms of embodied experience and creative self-expression. The study also provides empirical evidence and discussion on the cultures of social interaction mediated by alternative embodiments, changes in bodily perception resulting from prolonged use, functional and social challenges related to avatar use, and the design strategies and etiquette practices developed to overcome them.

Wearable supernumerary robotic limbs (SRLs) sit at the intersection of human augmentation and embodied AI, promising to function as extensions of the human body. However, their movements within the intimate near-body space raise unresolved challenges for perceived safety, user control, and trust. In this paper, we present results from a Wizard-of-Oz study (n=18), where participants completed near-body collaboration tasks with SRLs to explore these challenges. We collected qualitative data through think-aloud protocols and semi-structured interviews, complemented by physiological signals and post-task ratings. Findings indicate that greater autonomy did not inherently enhance perceived safety or trust. Instead, participants identified near-body zones and paired them with clear coordination rules. They also expressed expectations for how different arm components should behave, shaping preferences around autonomy, perceived safety, and trust. Building on these insights, we introduce SRL Proxemics, a zone- and segment-level design framework showing that autonomy is not monolithic: perceived safety hinges on spatially calibrated, legible behaviors, not on autonomy level alone.

When language is used as a medium for sensory information—as when one describes a scene in words—a kind of virtual reality emerges: realities projected into the same sentence become virtual/equivalent. We call this Linguistic VR (LVR). LVR is constituted by codes shaped chiefly by majority cultural norms. In today’s world—where AI, largely built on linguistic data and processes, is deeply entangled with the everyday mediation of sensory information—it is necessary to critically re-examine the virtuality of this VR.

Street-level imagery (SLI) has become increasingly relevant in crisis informatics; however, its application in post-disaster contexts remains underexplored. Here, we present the systematic capture and public dissemination of over 640 km of SLI following the 2024 Noto Peninsula earthquake in Japan. Combining a survey of 30 participants, interviews with six stakeholders, and an analysis of SLI view metrics, we examine how different populations, including current residents, former residents, and remote observers, perceive its value and risks. While many participants valued SLI as a tool for connection, virtual inspection, and long-term archiving, concerns were raised regarding privacy, vulnerability, and potential burglary. We conceptualize these tensions through the lens of a boundary object, highlighting how disaster-related SLI is not simply a technical artifact, but an emergent sociotechnical infrastructure shaped by ongoing negotiation among contributors, platforms, and affected communities. Furthermore, we address ethical implications related to privacy protection, release timing, and sustainable archiving that can better support infrastructure among stakeholders and affected communities.

Cooling feedback has been shown to improve user experience in virtual reality. However, many existing methods face limitations, such as requiring skin contact, relying on airflow, having bulky components, or delivering unstable sensations. These issues limit their applicability to wearable systems, especially in scenarios demanding high responsiveness and perceptual consistency. We propose ChillFactor, a highly responsive, non-contact cooling interface for head-mounted displays (HMDs). It uses ultrasonic atomization of a predetermined alcohol solution to generate evaporative cooling near the skin, reducing airflow-induced artifacts and providing relatively consistent cold sensations. To validate the design, we first evaluated the system's cooling performance. At maximum output, it lowered skin temperature by approximately 2.6 °C within five seconds, a rate sufficient to induce an almost immediate perception of cold. We subsequently integrated ChillFactor into an HMD and conducted a user study with 12 participants in immersive VR scenarios. Participants reported perceiving the cold within 0.66 seconds and noted significantly higher levels of presence and realism compared to conditions without cooling feedback, demonstrating the system's effectiveness in enhancing immersion through rapid and adjustable thermal cues. This work provides a practical foundation for non-contact cooling feedback and expands the design space for multisensory interaction in virtual environments.

This study investigates how both the body part used to control a VR avatar and the avatar's appearance affect redirection detection thresholds. We conducted experiments comparing hand and foot manipulation of two types of avatars: a hand-shaped avatar and an abstract spherical avatar. Our results show that, irrespective of the body part used, the redirection detection threshold increased by 21% when using the hand avatar compared to the abstract avatar. Additionally, when the avatar's position was redirected toward the body midline, the detection threshold increased by 49% compared to redirection away from the midline. No significant differences in detection thresholds were observed between the hand and foot manipulations. These findings suggest that avatar appearance and redirection direction significantly influence user perception in VR environments, offering valuable insights for the design of full-body VR interactions and human augmentation systems.

Manipulating a non-humanoid body using a mapping approach that translates human body activity into different structural movements enables users to perform tasks that are difficult with their innate bodies. However, a key challenge is how to design an effective mapping to control non-analogous body parts with the human body. To address this challenge, we designed an articulated virtual arm and investigated the effect of mapping methods on a user’s manipulation experience. Specifically, we developed an unbranched 12-joint virtual arm with an octopus-like appearance. Using this arm, we conducted a user study to compare the effects of several mapping methods with different arrangements on task performance and subjective evaluations of embodiment and user preference. As a result, we identified three important factors in mapping: “Visual and Configurational Similarity”, “Kinematics Suitability for the User”, and “Correspondence with Everyday Actions.” Based on these findings, we discuss a mapping design for non-humanoid body manipulation.

We control the temperature of materials in everyday interactions, recognizing temperature’s important influence on our bodies, minds, and experiences. However, thermal feedback is an under-explored modality in human-computer interaction partly due to its limited temporal (slow) and spatial (small-area and non-moving) capabilities. We introduce hydroptical thermal feedback, a spatial thermal feedback method that works by applying visible lights on body parts in water. Through physical measurements and psychophysical experiments, our results show: (1) Humans perceive thermal sensations when visible lights are cast on the skin under water, and perceived warmth is greater for lights with shorter wavelengths, (2) temporal capabilities, (3) apparent motion (spatial) of warmth and coolness sensations, and (4) hydroptical thermal feedback can support the perceptual illusion that the water itself is warmer. We propose applications, including virtual reality (VR), shared water experiences, and therapies. Overall, this paper contributes hydroptical thermal feedback as a novel method, empirical results demonstrating its unique capabilities, proposed applications, and design recommendations for using hydroptical thermal feedback. Our method introduces controlled, spatial thermal perceptions to water experiences.

When humans generate stimuli voluntarily, they perceive the stimuli more weakly than those produced by others, which is called sensory attenuation (SA). SA has been investigated in various body parts, but it is unclear whether an extended body induces SA. This study investigated the SA of audio stimuli generated by an extended body. SA was assessed using a sound comparison task in a virtual environment. We prepared robotic arms as extended bodies, and the robotic arms were controlled by facial movements. To evaluate the SA of robotic arms, we conducted two experiments. Experiment 1 investigated the SA of the robotic arms under four conditions. The results showed that robotic arms manipulated by voluntary actions attenuated audio stimuli. Experiment 2 investigated the SA of the robotic arm and innate body under five conditions. The results indicated that the innate body and robotic arm induced SA, while there were differences in the sense of agency between the innate body and robotic arm. Analysis of the results indicated three findings regarding the SA of the extended body. First, controlling the robotic arm with voluntary actions in a virtual environment attenuates the audio stimuli. Second, there were differences in the sense of agency related to SA between extended and innate bodies. Third, the SA of the robotic arm was correlated with the sense of body ownership.

Half a century since the concept of a cyborg was introduced, digital cyborgs, enabled by the spread of wearable robotics, are the focus of much research in recent times. We introduce JIZAI ARMS, a supernumerary robotic limb system consisting of a wearable base unit with six terminals and detachable robot arms controllable by the wearer. The system was designed to enable social interaction between multiple wearers, such as an exchange of arm(s), and explore possible interactions between digital cyborgs in a cyborg society. This paper describes the JIZAI ARMS’ design process, an interdisciplinary collaboration between human augmentation researchers, product designers, a system architect, and manufacturers, to realize a technically complex system while considering the aesthetics of a digital cyborg. We also provide an autobiographical report of our first impressions of using the JIZAI ARMS and use our findings to speculate on a model of potential social interactions between digital cyborgs.

Nowadays, many computer vision (CV) techniques require high-precision feature point detection in calibration targets. Checkerboards or circle grids are common patterns used to infer the locations of the points directly from the image. However, such approaches can lead to degradation in sub-pixel accuracy for low-resolution or sub-sampled sensor data (e.g., Bayer pattern). This paper proposes a target-based calibration technique applicable to various conventional CV methods, leveraging the moiré phenomena. We use a moiré pattern that is created between the digital camera’s pixels and the high-frequency texture on the calibration target. Even a tiny movement of the target or camera leads to a significant motion of the moiré, acting as a magnifier of the target texture for estimating accurate location of targets or camera. In contrast to previous moiré approaches that employ frequency domain analyses, we introduce the moiré coordinates and derive them from the difference between the sensor and target texture position. In the moiré coordinate, each moiré pattern is undistorted. Unlike moiré in standard images, each moiré pattern has almost the same scale and shape in these coordinates. Utilizing this property, we propose a straightforward algorithm to detect and handle correspondence in the moiré peaks in image and texture location, treating them as feature points of calibration target. In addition, the moiré vision can magnify pixels, which means the pattern in the moiré coordinate visualizes the pixel sensitivity with high spatial resolution. To validate the proposed approach, we present an empirical evaluation using simulated and real Bayer sensor patterns. We show the advantages of the proposed method in terms of accuracy and identify promising future directions.

This paper describes a case study wherein we realized a novel remote shopping experience by using T-Leap, a wearable telepresence system with an integrated 360° camera, to connect a supermarket in the city center with elderly participants living far from the store. The participants of our case study operated T-Leap via a PC/tablet and remotely joined a surrogate shopper to explore the supermarket and shop for products. In this study, we revealed that a real-time 360° view provides opportunities for accidental, serendipitous purchasing activities, despite being difficult for the elderly participants to control. Reflecting on our findings, we discuss the design of future remote shopping services from an HCI perspective. Our design implications will contribute to assisting people who have difficulties regularly shopping for their daily needs, like our participants, have rich, in-store experiences that have never been provided by existing online shopping services.

Outdoor activities, such as traveling and sightseeing, become challenging to realize when individuals are geographically distant. This paper introduces Syn-Leap, a symmetrical wearable telepresence system designed to allow multiple users to explore outdoor settings while mutually sharing their environments. With Syn-Leap, users wear a 360° camera and AR glasses, allowing them to share their environment through video streaming while simultaneously viewing the videos of other users while walking outdoors. To investigate the experiences and communication generated by this system, we conducted a field study in Kyoto, Japan, focusing on tourism scenarios. The results revealed that participants could achieve a sense of walking together by seamlessly switching between exploring their own environment and viewing the video feeds of other users, occasionally overlaying the AR content from other users onto their own environment.

Wearable robotic limbs (WRLs) augment human capabilities through robotic structures that attach to the user’s body. While WRLs are intensely researched and various device designs have been presented, it remains difficult for non-roboticists to engage with this exciting field. We aim to empower interaction designers and application domain experts to explore novel designs and applications by rapidly prototyping personalized WRLs that are customized for different tasks, different body locations, or different users. In this paper, we present WRLKit, an interactive computational design approach that enables designers to rapidly prototype a personalized WRL without requiring extensive robotics and ergonomics expertise. The body-aware optimization approach starts by capturing the user’s body dimensions and dynamic body poses. Then, an optimized fabricable structure of the WRL is generated for a desired mounting location and workspace of the WRL, to fit the user’s body and intended task. The results of a user study and several implemented prototypes demonstrate the practical feasibility and versatility of WRLKit.

Technological advancement has opened up opportunities for new sports and physical activities. We introduce a concept called machine-mediated teaming, in which a human and a surrogate machine form a team to participate in physical sports games. To understand the experience of machine-mediated teaming and the guidelines for designing the system to achieve the concept, we built a case study system based on tug-of-war. Our system is a sports game played by two against two. One team consists of a player who actually pulls the rope and another player who participates in the physical game by controlling the machine’s actuators. We conducted user studies using this system to investigate the sport experience in this form and to reveal insights to inform future research on machine-mediated teaming. Based on the data obtained from the user studies, we clarified three perspectives, machine stamina, action space, and explicit feedback, that should be considered when designing future machine-mediated teaming systems. The research presented in this paper offers a first step towards exploring how humans and machines can coexist in highly dynamic physical interactions.

Apparent color and pixel color are known to be different due to color constancy and contexts of images. An apparent color dataset for photographs and images will be a basis for developing perception-based color extraction and transfer system. In this study, we collected the apparent color dataset for a given image and extracted position through an online experiment that mimics the situation of using a color extraction system. Additionally, we analyzed 1110 data and verified the difference between apparent color and pixel color. As a result, approximately 80% data has $\Delta E_{\mathrm{NBS}} \gt 6.0$ color difference that is defined “Much” or “Very Much” difference by the national bureau of standards (NBS). Luminance (L*) of the apparent color was on average $\triangle E_{00}=[9.90,11.9]$ (95% confidence interval) greater than that of the pixel color. These results indicate that the color deviations cannot be ignored when apparent colors are extracted in images.

Human environments are physically supported by floors, which prevent people and furniture from gravitational pull. Since our body motions continuously generate vibrations and loads that propagate into the ground, measurement of these expressive signals leads to unobtrusive activity sensing. In this study, we present Flexel, a modular floor interface for room-scale tactile sensing. By paving a room with floor interfaces, our system can immediately begin to infer touch locations, track user locations, recognize foot gestures, and detect object locations. Through a series of exploratory studies, we determined the preferable hardware design that adheres to construction conventions, as well as the optimal sensor density that mediates the trade-off between cost and performance. We summarize our findings into design guidelines that are generalizable to other floor interfaces. Finally, we provide example applications for room-scale tactile sensing enabled by our Flexel system.

While floral tributes are commonly used for the public commemoration of victims of disasters, war, and other accidents, flowers in vases color everyday life. In this research, these features of flowers are intertwined with the recent phenomenon of online memorials to develop a virtual floral tribute concept that includes physical rituals. We designed SenseVase, a smart vase to detect flowers placed in it, and a 3DCG Virtual Memorial that illustrates floral tributes given by people using SenseVases at home. This paper describes how we developed our design concept by reviewing previous literature and social aspects, and presents a video illustrating the concept. To validate the current concept, we interviewed several experts knowledgeable in public commemorations, virtual and online communities, and the floral business. Through a discussion of our findings from the design process and interviews, we propose a new direction for how HCI technology can contribute to public commemoration in addition to personal memorialization.

Funerals are irreplaceable events, especially for bereaved family members and relatives. However, the COVID-19 pandemic has prevented many people worldwide from attending their loved ones’ funerals. The authors had the opportunity to assist one family faced with this predicament by webcasting and recording funeral rites held near Tokyo in June, 2020. Using our original 360-degree Telepresence system and smartphones running Zoom, we enabled the deceased’s elder siblings to remotely attend the funeral and did our utmost to make them feel present in the funeral hall. Despite the webcasting via Zoom contributing more to their remote attendances than our system, we discovered thoughtful findings which could be useful for designing remote funeral attendances. From the findings, we also discuss how HCI designers can contribute to this highly sensitive issue, weaving together knowledge from various domains including techno-spiritual practices, thanato-sensitive designs; and other religious and cultural aspects related to death rituals.

Makeup (i.e., cosmetics) has long been used to transform not only one’s appearance but also their self-representation. Previous studies have demonstrated that visual transformations can induce a variety of effects on self-representation. Herein, we introduce Digital Speech Makeup (DSM), the novel concept of using voice conversion (VC) based auditory feedback to transform human self-representation. We implemented a proof-of-concept system that leverages a state-of-the-art algorithm for near real-time VC and bone-conduction headphones for resolving speech disruptions caused by delayed auditory feedback. Our user study confirmed that conversing for a few dozen minutes using the system influenced participants’ speech ownership and implicit bias. Furthermore, we reviewed the participants’ comments about the experience of DSM and gained additional qualitative insight into possible future directions for the concept. Our work represents the first step towards utilizing VC to design various interpersonal interactions, centered on influencing the users’ psychological state.

In comics, Textual Sound Effects (TE) can describe sounds, but also actions, events, etc. TE could be used in Virtual Environment to efficiently create an easily recognizable scene and add more information to objects at a relatively low design cost. We investigate the impact of TE in a Virtual Environment on objects’ material perception (on category and properties) and on sound perception (on volume [dB] and spatial position). Participants (N=13, repeated measures) categorized metallic and wooden spheres and significantly changed their reaction time depending on the TE congruence with the spheres’ material/sound. They then rated a sphere’s properties (i.e., wetness, warmness, softness, smoothness, and dullness) and significantly changed their rating depending on the TE. When comparing 2 sound volumes, they perceived a sound associated with a shrinking TE as less loud and a sound associated with a growing TE as louder. When locating an audio source location, they located it significantly closer to a TE.

Haptic information significantly improves human awareness of objects in virtual reality. One way of presenting this information is via encountered-type haptic feedback. An advantage of encounter type feedback is that it enables physical interaction with virtual environments without the need for specialized haptic devices on the hand. Additionally, encountered-type haptics are known for being able to provide high quality contact feedback to the user. However, such systems are typically designed to be grounded (i.e., fixed to the floor). As such, they typically have bounded workspace and a limited range of possible applications. In this work, we present a novel, wearable approach to presenting a user with encountered-type haptic feedback. We realize this feedback using a wearable robotic limb that holds a plate where the user might interact with their environment. An appropriate location for the plate is determined by a novel haptic solver while control of the arm is made possible using motion trackers. The system was designed to be stable, for presenting consistent haptic feedback, while also being safe and lightweight for wearability. By making the feedback system wearable, we enable the presentation of stiff feedback while maintaining the spatial freedom and unbounded workspace of natural hand interaction. Herein, we present the design of the novel system, mechanical and safety considerations when designing a wearable encountered-type system, and an evaluation of the system. A technical evaluation of the implemented system showed that the system provides a stiffness over 25 N/m and slant angle errors under 3°. Three user studies show the limitations of haptic slant perception in humans and the quantitative and qualitative effectiveness of the current prototype system. We conclude the paper by discussing various potentialapplications and possible improvements that could be made to the system.

This paper presents an approach for coupling robotic capability with human ability in dynamic motor skills, called "Human-Machine Mutual Actuation (HMMA)." We focus specifically on throwing motions and propose a method to control the release timing computationally. A system we developed achieves our concept, HMMA, by a robotic handheld device that acts as a release controller. We conducted user studies to validate the feasibility of the concept and clarify related technical issues to be tackled. We recognized that the system successfully performs on throwing according to the target while it exploits human ability. These empirical experiments suggest that robotic capability can be embedded into the users' motions without losing their senses of control. Throughout the user study, we also revealed several issues to be tackled in further research contributing to HMMA.

Human-Computer Integration (HInt) is an emerging paradigm in which computational and human systems are closely interwoven. Integrating computers with the human body is not new. however, we believe that with rapid technological advancements, increasing real-world deployments, and growing ethical and societal implications, it is critical to identify an agenda for future research. We present a set of challenges for HInt research, formulated over the course of a five-day workshop consisting of 29 experts who have designed, deployed and studied HInt systems. This agenda aims to guide researchers in a structured way towards a more coordinated and conscientious future of human-computer integration.

A reduced balance ability can lead to falls and critical injuries. To prevent falls, humans use reaction forces and torques generated by swinging their arms. In animals, we can find that a similar strategy is taken using tails. Inspired by these strategies, we propose an approach that utilizes a robotic appendage as a human balance supporter without assistance from environmental contact. As a proof of concept, we developed a wearable robotic appendage that has one actuated degree of freedom and rotates around the sagittal axis of the wearer. To validate the feasibility of our proposed approach, we conducted an evaluation experiment with human subjects. Controlling the robotic appendage we developed improved the subjects' balance ability and enabled the subject to withstand up to 22.8 % larger impulse disturbances on average than in the fixed appendage condition.

This paper describes a field study conducted with our system, T-Leap, a telepresence system connecting one person (the Viewer), situated indoors, with multiple destinations (the Nodes), that roam outdoors. Here, each Node is a person wearing a module that includes a 360-degree camera and a microphone-speaker. Through our study, we demonstrate that T-Leap enables the Viewer to perform various interactions with the Nodes including being helped by them, collaborating with them, and guiding them. These interactions were demonstrated through three studies completing different tasks: 1) Nodes purchasing souvenirs for the Viewer, 2) Nodes finding objects in the park, and 3) Viewer guiding Nodes to purchase things. The studies were primarily conducted with Taiwanese locals and Japanese visitors in Taipei. Throughout the studies, we found that T-Leap worked especially well for mediating communication between a Viewer with local knowledge acting as a guide and several Nodes who were being guided. To conclude the paper, we broadly discuss our findings, the lessons we learned from our field study, and present recommendations for the future development of mobile and wearable telepresence systems.

To efficiently edit first-person videos, manually highlighting important scenes while recording is helpful. However, little study has been performed on how such annotation contributes to video editing and affects user behavior during recording. To elicit fundamental requirements for designing useful record-time annotation techniques, we conducted a study using a set of prototype wearable camera system and a video editing interface that enables users to annotate scenes during recording. We asked participants to perform video recording and editing tasks with two different interface settings. We observed that the participants edited videos more efficiently with detailed annotation techniques, whereas focussing on annotating scenes affected their record-time behavior. We conclude the paper with the design guidelines developed from the findings.

Most AR games focus primarily on static augmentation, leaving out the potential for rich and tangible interactions. Recently, with the creation of the Superhuman Sports genre, new AR games emerged with the goal to extend the sports experience by utilizing technology. With the understanding of various related projects towards physical object interaction in AR and with the rich selection of sensors in mobile devices, we want to present two innovative visions in the direction of AR Sports Games: 1.Floating Ball: A ball shaped object is augmented and while flying in a circular pattern, two players are competing against each other by interacting with it. 2.Augmented Drone: Resulting out of the experiences with the Floating Ball, a competitive game concept with a drone as a versatile augmented game ball will be presented. We want to utilize the results as a step towards a competitive multi-player AR Sports Game based on mobile devices. In both concepts, one device is functioning as a Tangible AR Interface with the task to let the player manipulate the flying path of the physical ball. For this purpose, the ball is incorporated into the augmented world to create a realistic, perceivable playing experience.

This paper presents development on a conceptual method to remap supernumerary limbs using Virtual Reality (VR) as a platform for experimentation. Our VR system allows users to control a third arm through their own limbs such as their head, arms, and feet with the ability to switch between them. To realize and experiment with our remapping method, we used the Oculus Rift in conjunction with OptiTrack to track users in a room-scaled virtual environment. We present some initial findings from a small pilot study and conclude with suggestions for future work.

Humans have one own body. However, we can share a body with two different persons in a virtual environment. We have developed a shared body as an avatar that is controlled by two persons' actions. Movements of two subjects were continuously captured, and integrated into the avatar's motion with the ratios of 0:100,25:75, 50:50, 75:25, 100:0. They were not aware of integration ratios, and asked to reach cubes with the right hand. They felt body ownership and sense of agency to the shared avatar more when the responsible ratio was higher. The reaching path of the avatar's hand was shorter in the shared body condition (75:25) than the single body condition (100:0). These results suggest that we have some of body ownership and sense of agency to the shared body, and the task performance is improved by the shared body.

We propose “ExLeap”, a minimal telepresence system that creates leaping experience. Multiple “nodes” with an omnidirectional camera, mic and speaker transmit the video to clients, and on the client, videos are rendered in 3D space. When moving to another node, by crossfading two videos, the user can feel as if she/he leaps between two places. Also, on each node, the user can talk with people in that place. Each node consists of very simple hardware, so we can put them on multiple places we want to go to. Moreover, because the system can be used 24/7 by multi-user simultaneously and is very easy to use, it creates various types of chances of communications.

Humans can feel as if a fake body is their own body in the illusory body ownership. The illusion can be induced to a full body by a visual-tactile synchronicity or visual-motor synchronicity. In our previous study, illusory full body ownership of invisible body was elicited by synchronous movements of virtual gloves and socks. In this study, we aimed to investigate whether the spatial relationship is necessary for the full body illusion using a stimulus of scrambled body. In the scrambled body, positions of gloves and socks were scrambled. The results suggest that spatial relationship of body parts is necessary for the full body illusion.

With advancements in robotics, systems featuring wearable robotic arms teleoperated by a third party are appearing. An important aspect of these systems is the visual feedback provided to the third party operator. This can be achieved by placing a wearable camera on the robotic arm's “host,” or Main Body Operator (MBO), but such a setup makes the visual feedback dependant on the movements of the main body. Here we introduce a VR system called Parasitic Body to explore a VR shared body concept representative of the wearable robotic arms “host” (the MBO) and of the teleoperator (here called the Parasite Body Operator (PBO)). 2 users jointly operate a shared virtual body with a third arm: The MBO controls the main body and the PBO controls a third arm sticking out from the left shoulder of the main body. We focused here on the perspective dependency of the PBO (indeed, the PBO view is dependant of the movement of the MBO) in a “finding and reaching” task.

Robot assistance can reduce the user's workload of a task. However, the robot needs to be programmed or trained on how to assist the user. Virtual Reality (VR) can be used to train and validate the actions of the robot in a safer and cheaper environment. In this paper, we examine how a robotic arm can be trained using Coloured Petri Nets (CPN) and Partial Least Squares Regression (PLSR). Based upon these algorithms, we discuss the concept of using the user's acceleration and rotation as a sufficient means to train a robotic arm for a procedural task in VR. We present a work-in-progress system for training robotic limbs using VR as a cost effective and safe medium for experimentation. Additionally, we propose PLSR data that could be considered for training data analysis.

We introduce MetaArms, wearable anthropomorphic robotic arms and hands with six degrees of freedom operated by the user's legs and feet. Our overall research goal is to re-imagine what our bodies can do with the aid of wearable robotics using a body-remapping approach. To this end, we present an initial exploratory case study. MetaArms' two robotic arms are controlled by the user's feet motion, and the robotic hands can grip objects according to the user's toes bending. Haptic feedback is also presented on the user's feet that correlate with the touched objects on the robotic hands, creating a closed-loop system. We present formal and informal evaluations of the system, the former using a 2D pointing task according to Fitts' Law. The overall throughput for 12 users of the system is reported as 1.01 bits/s (std 0.39). We also present informal feedback from over 230 users. We find that MetaArms demonstrate the feasibility of body-remapping approach in designing robotic limbs that may help us re-imagine what the human body could do.

Illusory body ownership can be induced by visual-tactile stimulation or visual-motor synchronicity. We aimed to test whether a right thumb could be remapped to a virtual right arm and illusory body ownership of the virtual arm induced through synchronous movements of the right thumb and the virtual right arm. We presented the virtual right arm in synchronization with movements of a participant's right thumb on a head-mounted display (HMD). We found that the participants felt as though their right thumb became the right arm, and that the right arm belonged to their own body.

We propose a drawing method to create large-scale pictures in public space. We use a property of anisotropic reflection to show images on the grass field. We created a prototype of roller type device which can control the angle of grass. We observed that our system entertains people in public exhibition.

This article presents a novel smart eyewear that recognizes the wearer’s facial expressions in daily scenarios. Our device uses embedded photo-reflective sensors and machine learning to recognize the wearer’s facial expressions. Our approach focuses on skin deformations around the eyes that occur when the wearer changes his or her facial expressions. With small photo-reflective sensors, we measure the distances between the skin surface on the face and the 17 sensors embedded in the eyewear frame. A Support Vector Machine (SVM) algorithm is then applied to the information collected by the sensors. The sensors can cover various facial muscle movements. In addition, they are small and light enough to be integrated into daily-use glasses. Our evaluation of the device shows the robustness to the noises from the wearer’s facial direction changes and the slight changes in the glasses’ position, as well as the reliability of the device’s recognition capacity. The main contributions of our work are as follows: (1) We evaluated the recognition accuracy in daily scenes, showing 92.8% accuracy regardless of facial direction and removal/remount. Our device can recognize facial expressions with 78.1% accuracy for repeatability and 87.7% accuracy in case of its positional drift. (2) We designed and implemented the device by taking usability and social acceptability into account. The device looks like a conventional eyewear so that users can wear it anytime, anywhere. (3) Initial field trials in a daily life setting were undertaken to test the usability of the device. Our work is one of the first attempts to recognize and evaluate a variety of facial expressions with an unobtrusive wearable device.

This paper presents a novel smart eyewear that uses embedded photo reflective sensors and machine learning to recognize a wearer's facial expressions in daily life. We leverage the skin deformation when wearers change their facial expressions. With small photo reflective sensors, we measure the proximity between the skin surface on a face and the eyewear frame where 17 sensors are integrated. A Support Vector Machine (SVM) algorithm was applied for the sensor information. The sensors can cover various facial muscle movements and can be integrated into everyday glasses. The main contributions of our work are as follows. (1) The eyewear recognizes eight facial expressions (92.8% accuracy for one time use and 78.1% for use on 3 different days). (2) It is designed and implemented considering social acceptability. The device looks like normal eyewear, so users can wear it anytime, anywhere. (3) Initial field trials in daily life were undertaken. Our work is one of the first attempts to recognize and evaluate a variety of facial expressions in the form of an unobtrusive wearable device.

Reading is a very common learning activity, a lot of people perform it everyday even while standing in the subway or waiting in the doctors office. However, we know little about our everyday reading habits, quantifying them enables us to get more insights about better language skills, more effective learning and ultimately critical thinking. This paper presents a first contribution towards establishing a reading log, tracking how much reading you are doing at what time. We present an approach capable of estimating the words read by a user, evaluate it in an user independent approach over 3 experiments with 24 users over 5 different devices (e-ink reader, smartphone, tablet, paper, computer screen). We achieve an error rate as low as 5% (using a medical electrooculography system) or 15% (based on eye movements captured by optical eye tracking) over a total of 30 hours of recording. Our method works for both an optical eye tracking and an Electrooculography system. We provide first indications that the method works also on soon commercially available smart glasses.

This paper introduces a registration and projection method for directly projecting the tumor region for breast cancer surgery assistance based on the breast procedure of our collaborating doctor. We investigated the steps of the breast cancer procedure of our collaborating doctor and how it can be applied for tumor region projection. We propose a novel way of MRI acquisition so we may correlate the MRI coordinates to the patient in the real world. By calculating the transformation matrix from the MRI coordinates and the coordinates from the markers that is on the patient, we are able to register the acquired MRI data to the patient. Our registration and presentation method of the tumor region was then evaluated by medical doctors.

In this paper we propose an augmented reality system that superimposes MRI onto the patient model. We use a half-silvered mirror and a handheld device to superimpose the MRI onto the patient model. By tracking the coordinates of the patient model and the handheld device using optical markers, we are able to transform the images to the correlated position. Voxel data of the MRI are made so that the user is able to view the MRI from many different angles.

In this paper, we propose an electric power management system for delivering power to home appliances with mobile robots. With our system, the user places an appliance without power cables freely within the home, and a robot provides the appliance with the electric power required for its operation. In addition to providing explanations of a usage scenario and a theoretical analysis, we demonstrate a prototype implementation. The robot of the prototype autonomously locates the target appliance, transfers its battery power to the appliance, and returns to the home position to recharge its battery. The validation study showed our proof-of-concept prototype worked as expected by the theory.

Post-it notes are a popular paper format that serves a multitude of purposes in our daily lives, as they provide excellent affordances for quick capturing of informal notes, and location-sensitive reminding. In this paper, we present Move-it, a system that combines Post-it notes with a technologically enhanced paperclip to demonstrate how a passive piece of paper can be turned into an "active" medium that conveys information through motion. We present two application examples that investigate the applicability of Move-it sticky notes for ambient information awareness. In comparison to existing notification systems, experimental results show that they reduce negative effects of interruptions on emotional state and performance, and provide unique affordances by combining advantages of physical and digital systems into a novel active paper interface.

We present Tracs, a dual-sided see-through display system with controllable transparency. Traditional displays are a constant visual and communication barrier, hindering fast and efficient collaboration of spatially close or facing co-workers. Transparent displays could potentially remove these barriers, but introduce new issues of personal privacy, screen content privacy and visual interference. We therefore propose a solution with controllable transparency to overcome these problems. Tracs consists of two see-through displays, with a transparency-control layer, a backlight layer and a polarization adjustment layer in-between. The transparency-control layer is built as a grid of individually addressable transparency-controlled patches, allowing users to control the transparency overall or just locally. Additionally, the locally switchable backlight layer improves the contrast of LCD screen content. Tracs allows users to switch between personal and collaborative work fast and easily and gives them full control of transparent regions on their display.

We devised a display technology that utilizes the phenomenon whereby the shading properties of fur change as the fibers are raised or flattened. One can erase drawings by first flattening the fibers by sweeping the surface by hand in the fiber's growth direction, and then draw lines by raising the fibers by moving the finger in the opposite direction. These material properties can be found in various items such as carpets in our living environments. We have developed three different devices to draw patterns on a "fur display" utilizing this phenomenon: a roller device, a pen device and pressure projection device. Our technology can turn ordinary objects in our environment into rewritable displays without requiring or creating any non-reversible modifications to them. In addition, it can be used to present large-scale image without glare, and the images it creates require no running costs to maintain.

We present Reality Jockey, a system that confuses the participant's perception of the reality by mixing in a recorded past-reality. The participant will be immersed in a spatialized 3D sound environment that is a mix of sounds from the reality and from the past. The sound environment from the past is augmented with haptic feedback in cross-modality. The haptic feedback is associated with certain sounds such as the vibration in the table when stuff is placed on the table to make the illusion of it happening in live. The seamless transition between live and past creates immersive experience of past events. The blending of live and past allows interactivity. To validate our system, we conducted user studies on 1) does blending live sensations improve such experiences, and 2) how beneficial is it to provide haptic feedbacks in recorded pasts. Potential applications are suggested to illustrate the significance of Reality Jockey.

This paper introduces an input and output device that enables illumination, bi-directional data communication, and position sensing on a soft cloth. This "LightCloth" is woven from diffusive optical fibers. Since the fibers are arranged in parallel, the cloth has one-dimensional position information. Sensor-emitter pairs attached to bundles of contiguous fibers enable bundle-specific light input and output. We developed a prototype system that allows full-color illumination and 8-bit data input by infrared signals. We present as an application a chair with a LightCloth cover whose illumination pattern is specified using an infrared light pen. Here we describe the implementation details of the device and discuss possible interactions using the device.

This paper proposes an interface that helps understanding the correspondence between the patient and medical images. In our proposed method, we have developed an interactive visualizer for tomographic images based on the relative position and orientation of the handheld device and the patient.

How do animals understand the physical world they live in? One answer, due to Gibson, is that their sensory systems are tuned to pick up relevant affordances for behavior, but how is it that the brain and the sensory apparatus become suitably adapted to perform this feat? To cast light on this question we have been investigating active touch sensing in mammals, including humans, and developing biomimetic robots that can help us understand these biological systems whilst also developing useful haptic technologies. An important focus has been on the vibrissal (whisker) system of rodents, and its emergence through evolution and development, which we have investigated through a combination of (i) ethological studies of behaving animals, (ii) computational neuroscience models of the neural circuits involved in vibrissal processing, and (iii) biomimetic robots embodying many of the characteristics of whiskered animals in their design and control. This work has resulted in a series of whiskered robots, the most recent of which, Shrewbot, is able to construct tactile maps of its environment and recognize and track moving objects. We are also studying humanoid touch, focusing on the development of Bayesian strategies for active tactile sensing with robot hands. Here our results have provided the first demonstration of hyperacuity in robot touch whilst also indicating that tactile perception is improved in unstructured environments by appropriate active control. The active sensing framework can also be applied to the development of haptic interfaces for human users that can augment our existing sensory capability. For instance, we are developing a head-mounted “remote touch” system that links distance sensors (ultrasound arrays) with vibrotactile displays. Here an interesting question is how the signals that are delivered through the displays should be modulated to take into account the intentional head and body movements of the user and in order to provide a meaningful and intuitive experience. The talk will present converging lines of evidence, from these different research strands, for the importance of active control in haptics. Our results will also be used to illustrate how experimental, computational, and robotic approaches can operate together to advance our understanding of sensorimotor cognition in behaving systems.

We present a sensing technology and input method that uses skin deformation estimated through a thin band-type device attached to the human body, the appearance of which seems socially acceptable in daily life. An input interface usually requires feedback. SenSkin provides tactile feedback that enables users to know which part of the skin they are touching in order to issue commands. The user, having found an acceptable area before beginning the input operation, can continue to input commands without receiving explicit feedback. We developed an experimental device with two armbands to sense three-dimensional pressure applied to the skin. Sensing tangential force on uncovered skin without haptic obstacles has not previously been achieved. SenSkin is also novel in that quantitative tangential force applied to the skin, such as that of the forearm or fingers, is measured. An infrared (IR) reflective sensor is used since its durability and inexpensiveness make it suitable for everyday human sensing purposes. The multiple sensors located on the two armbands allow the tangential and normal force applied to the skin dimension to be sensed. The input command is learned and recognized using a Support Vector Machine (SVM). Finally, we show an application in which this input method is implemented.

PINOKY is a wireless ring-like device that can be externally attached to any plush toy as an accessory that animates the toy by moving its limbs. A user is thus able to instantly convert any plush toy into a soft robot. The user can control the toy remotely or input the movement desired by moving the plush toy and having the data recorded and played back. Unlike other methods for animating plush toys, PINOKY is non-intrusive, so alterations to the toy are not required. In a user study, 1) the roles of plush toys in the participants' daily lives were examined, 2) how participants played with plush toys without PINOKY was observed, 3) how they played with plush toys with PINOKY was observed, and their reactions to the device were surveyed. On the basis of the results, potential applications were conceptualized to illustrate the utility of PINOKY.

"panavi," a recipe medium, supports cooking experience for domestic users to master professional culinary arts in their kitchens by managing temperature and pan movement properly. Utilizing a sensors-embedded frying pan--providing projected images, LED indications, and vibration--wirelessly connected with a computer system that shows text messages with sounds, the panavi system analyzes sensors' data, recognizes users' conditions, and provides the users with situated instructions. Describing our vision, design process, implementation, and user study that outlines experience of challenging professional cooking, this paper introduces a design framework model of this recipe medium for domestic usage. Throughout revealing the design process--from ideation to the finished research artifact as a whole cooking support system--this research suggests how to design interactive systems responding to human situated actions, for use as daily commodities enriching domestic user experience.

We propose an AR-based painting interface that enables users to design an illumination distribution for a real room using an array of computer-controlled lights. Users specify an illumination distribution of the room by painting on the image obtained by a camera mounted in the room. The painting result is overlaid on the camera image as contour lines of the target illumination intensity. The system runs an optimization interactively to calculate light parameters to deliver the requested illumination condition. In this implementation, we used actuated lights that can change the lighting direction to generate the requested illumination condition more accurately and efficiently than static lights. We built a miniature-scale experimental environment and ran a user study to compare our method with a standard direct manipulation method using widgets. The results showed that the users preferred our method for informal light control.

We present universal earphones that use both a proximity sensor and a skin conductance sensor and we demonstrate several implicit interaction techniques they achieve by automatically detecting the context of use. The universal earphones have two main features. The first involves detecting the left and right sides of ears, which provides audio to either ear, and the second involves detecting the shared use of earphones and this provides mixed stereo sound to both earphones. These features not merely free users from having to check the left and right sides of earphones, but they enable them to enjoy sharing stereo audio with other people.

We have developed a simple skin-like user interface that can be easily attached to curved as well as flat surfaces and used to measure tangential force generated by pinching and dragging interactions. The interface consists of several photoreflectors that consist of an IR LED and a phototransistor and elastic fabric such as stocking and rubber membrane. The sensing method used is based on our observation that photoreflectors can be used to measure the ratio of expansion and contraction of a stocking using the changes in transmissivity of IR light passing through the stocking. Since a stocking is thin, stretchable, and nearly transparent, it can be easily attached to various types of objects such as mobile devices, robots, and different parts of the body as well as to various types of conventional pressure sensors without altering the original shape of the object. It can also present natural haptic feedback in accordance with the amount of force exerted. A system using several such sensors can determine the direction of a two-dimensional force. A variety of example applications illustrated the utility of this sensing system.

This paper presents a new type of optical camouflage system based on the retro-reflective projection technology. Retro-reflective projection is a method used to create augmented reality that combines the virtual world with the real world. The conventional model of an optical camouflage system consists of a retro-reflective screen, a projection source and a beam splitter. In such a setup, the user needs to observe an object covered with the retro-reflective screen through a single viewpoint. This is called a monocular system. In our new setup, our aim is to construct a system that has multiple viewpoints by applying a novel projection array system. We will describe the method with which this projection array system is achieved using one projection source, the configuration of the system, and the trade-offs of the system. In addition, we will describe an application of our system in a car. The installed system makes the backseat virtually transparent, allowing the driver to see the blind spots at the rear when reversing the car.

As various home robots come into homes, the need for efficient robot task management tools is arising. Current tools are designed for controlling individual robots independently, so they are not ideally suitable for assigning coordinated action among multiple robots. To address this problem, we developed a management tool for home robots with a graphical editing interface. The user assigns instructions by selecting a tool from a toolbox and sketching on a bird's-eye view of the environment. Layering supports the management of multiple tasks in the same room. Layered graphical representation gives a quick overview of and access to rich information tied to the physical environment. This paper describes the prototype system and reports on our evaluation of the system.

We present an actuated handheld puppet system for controlling the posture of a virtual character. Physical puppet devices have been used in the past to intuitively control character posture. In our research, an actuator is added to each joint of such an input device to provide physical feedback to the user. This enhancement offers many benefits. First, the user can upload pre-defined postures to the device to save time. Second, the system is capable of dynamically adjusting joint stiffness to counteract gravity, while allowing control to be maintained with relatively little force. Third, the system supports natural human body behaviors, such as whole-body reaching and joint coupling. This paper describes the user interface and implementation of the proposed technique and reports the results of expert evaluation. We also conducted two user studies to evaluate the effectiveness of our method.

Photograph is a very useful tool for describing configurations of real-world objects to others. People immediately understand various pieces of information such as "what is the target object" and "where is the target position" by looking at a photograph, even without verbal descriptions. Our goal was to leverage these features of photographs to enrich human-robot interactions. We propose to use photographs as a front-end between a human and a home robot system. We named this method "Snappy". The user takes a photo to remember the target in a real-world situation involving a task and shows it to the system to make it physically execute the task. We developed a prototype system in which the user took a photo of a dish layout on a table and showed it to the system later to then have robots deliver and arrange the dishes in the same way.

Abstract—We present a method of instructing a sequential task to a household robot using a hand-held augmented reality device. The user decomposes a high-level goal such as “prepare a drink ” into steps such as delivering a mug under a kettle and pouring hot water into the mug. The user takes a photograph of each step using the device and annotates it with necessary information via touch operation. The resulting sequence of annotated photographs serves as a reference for review and reuse at a later time. We created a working prototype system with various types of robots and appliances. I.

We present the FuwaFuwa sensor module, a round, hand-size, wireless device for measuring the shape deformations of soft objects such as cushions and plush toys. It can be embedded in typical soft objects in the household without complex installation procedures and without spoiling the softness of the object because it requires no physical connection. Six LEDs in the module emit IR light in six orthogonal directions, and six corresponding photosensors measure the reflected light energy. One can easily convert almost any soft object into a touch-input device that can detect both touch position and surface displacement by embedding multiple FuwaFuwa sensor modules in the object. A variety of example applications illustrate the utility of the FuwaFuwa sensor module. An evaluation of the proposed deformation measurement technique confirms its effectiveness.

Recommendation systems have become widespread, however these systems only determine information inputted from the customers through a browser, and cannot be used when actually moving around outside. This paper presents TTI Model, a model extracting individual's curiosity level in urban spaces on their spare time by collecting behavior data from sensors. It calculates person's real time curiosity level by analyzing behavior depending on the walking speed within the city, such as window shopping or just hanging around by themselves. This paper evaluates this model with a sensor device prototype, and elaborates possibilities when understanding individuals in detail, by extracting the curiosity predicted from current behaviors using sensors.

We propose a cooking system that operates in an open environment. The system cooks a meal by pouring various ingredients into a boiling pot on an induction heating cooker and adjusts the heating strength according to the user's instructions. We then describe how the system incorporates robotic- and human-specific elements in a shared workspace so as to achieve a cooperative rudimentary cooking capability. First, we use small mobile robots instead of built-in arms to save space, improve flexibility and increase safety. Second, we use detachable visual markers to allow the user to easily configure the real-world environment. Third, we provide a graphical user interface to display detailed cooking instructions to the user. We hope insights obtained in this experiment will be useful for the design of other household systems in the future.

Personal photographs are important media for communication in our daily lives. People take photos to remember things about themselves and show them to others to share the experience. We expect that a photograph can be useful tool for teaching a task to a robot. We propose a novel human-robot interaction using photographs. The user takes a photo to remember the target in a real-world situation involving a task and shows it to the system to make it physically execute the task. We developed a prototype system in which the user took a photo of a dish arrangement on a table and showed it to the system later to then have a small robot deliver and arrange the dishes in the same way.

This paper proposes a physical user interface for a user to teach a robot to navigate a home environment. The user places small devices containing infrared based communication functionality as landmarks in the environment. The robot follows these landmarks to navigate to a goal landmark. Active landmarks communicate with each other to map their spatial relationships. Our method allows the user to start using the system immediately after placing the landmarks without installing any global position sensing system or prior mapping by the robot.

This paper introduces a simple algorithm for non-prehensile object transportation by a pushing robot on a flat surface. We assume that the global position and orientation of the robot and objects are known. The system computes a dipole field around the object and moves the robot along the field. This simple algorithm resolves many subtle issues in implementing reliable pushing behaviors, such as collision avoidance, error recovery, and multi-robot coordination. We verify the effectiveness of the algorithm via several experiments with varying robot and object form factors. Although object delivery by pushing and motion control by a vector field are not new, the proposed algorithm offers easier implementation with fewer parameter adjustments because of its mode-less definition and scale-invariant formulation.

We propose a simple drag-and-drop interface on a touch screen to give instructions to an object-delivery robot. The user uses a finger to drag a target object and drop it to a target location on the screen that shows an image of the floor provided by a ceiling camera. The system then autonomously executes a delivery task without continuous control by the user. We employ user-guided computer vision methods to identify and track an object in a scene without prior registration. We conducted a user study, which showed that the participants completed separate tasks more successfully when using our method than when using a remote controller.

Myglobe is a user generated navigation service that enables users to share each cognitive map with one another. Cognitive map is a personalized map, shape of which is emphasized according to user's preference and activity in the city. It facilitates users to look back on their own city and have a new understanding by using an application in smart phones and physically interacting with a globe shaped device. In this paper, we present Myglobe service for users to achieve a new city experience with cognitive maps.

Numerous robots have been developed, and some of them are already being used in homes, institutions, and workplaces. Despite the development of useful robot functions, the focus so far has not been on user interfaces of robots. General users of robots find it hard to understand what the robots are doing and what kind of work they can do. This paper presents an interface for the commanding home robots by using stroke gestures on a computer screen. This interface allows the user to control robots and design their behaviors by sketching the robot's behaviors and actions on a top-down view from ceiling cameras. To convey a feeling of directly controlling the robots, our interface employs the live camera view. In this study, we focused on a house-cleaning task that is typical of home robots, and developed a sketch interface for designing behaviors of vacuuming robots.

This paper presents a new form of remote active tangible interactions built with the Display-based Measurement and Control System. A prototype system was constructed to demonstrate the concepts of coupled remote tangible objects on rear projected tabletop displays. A user evaluation measuring social presence for two users performing a furniture placement task was performed, to determine a difference between this new system and a traditional mouse.

In this paper, we present a high speed optical motion capture method that can measure three dimensional motion, orientation, and incident illumination at tagged points in a scene. We use tracking tags that work in natural lighting conditions and can be imperceptibly embedded in attire or other objects. Our system supports an unlimited number of tags in a scene, with each tag uniquely identified to eliminate marker reacquisition issues. Our tags also provide incident illumination data which can be used to match scene lighting when inserting synthetic elements. The technique is therefore ideal for on-set motion capture or real-time broadcasting of virtual sets. Unlike previous methods that employ high speed cameras or scanning lasers, we capture the scene appearance using the simplest possible optical devices - a light-emitting diode (LED) with a passive binary mask used as the transmitter and a photosensor used as the receiver. We strategically place a set of optical transmitters to spatio-temporally encode the volume of interest. Photosensors attached to scene points demultiplex the coded optical signals from multiple transmitters, allowing us to compute not only receiver location and orientation but also their incident illumination and the reflectance of the surfaces to which the photosensors are attached. We use our untethered tag system, called Prakash, to demonstrate methods of adding special effects to captured videos that cannot be accomplished using pure vision techniques that rely on camera images.

This paper describes a kind of active camouflage system named optical camouflage. Optical camouflage uses the retro-reflective projection technology, a projection-based augmented-reality system composed of a projector with a small iris and a retro-reflective screen. The object that needs to be made transparent is painted or covered with retro-reflective material. Then a projector projects the background image on it making the masking object virtually transparent.

Previous research on augmented reality has been mainly focused on augmentation of visual or acoustic information. However, humans can receive information not only through vision and acoustics, but also through haptics. Haptic sensation is very intuitive, and some researchers are focusing on making use of haptics in augmented reality systems. While most previous research on haptics is based on static data, such as that generated from CAD, CT, and so on, these systems have difficulty responding to a changing real environment in real time. In this paper, we propose a new concept for the augmented reality of haptics, the SmartTool. The SmartTool responds to the real environment by using real time sensor(s) and a haptic display. The sensor(s) on the SmartTool measure the real environment then send us that information through haptic sensation. Furthermore, we will describe the prototype system we have developed.

We present an autostereoscopic display named TWISTER I (Telexistence Wide-view-angle Immersive STEReoscope, Model I), which is designed for a face-to-face telecommunication system called "mutual telexistence". By rotating display units that consist of LED arrays and a barrier around a viewer, TWISTER I can display panoramic stereoscopic images that can be observed without the use of special eyewear. This "glassless" feature is essential for applying this apparatus to mutual telexistence because eye contact is important in non-verbal communication.

Proposes a novel visuo-haptic display using a head-mounted projector (HMP) with X'tal Vision (Crystal Vision) optics. Our goal is to develop a device which enables an observer to touch a virtual object just as it is seen. We describe in detail the design of an HMP with X'tal Vision, which is very suitable for augmented reality. For instance, the HMP makes the occlusion relationship between the virtual and the real environments nearly correct. Accordingly, the user can observe his/her real hand with the virtual objects. Furthermore, the HMP reduces eye fatigue because of the low inconsistency of accommodation and convergence. Therefore, we applied HMP-model 2 to a visuo-haptic display using a camouflage technique. This technique, called optical camouflage, makes an obstacle object, such as a haptic display, become translucent. With this method, a user can observe a stereoscopic virtual object with a nearly correct occlusion relationship between the virtual and the real environments, and he can actually feel the object.