Contemporary interactive systems often center design around discrete user actions and interface responses. While effective for tasks, this focus can overlook how experience is shaped through perceptual conditions such as light, sound, airflow, temperature, and material presence. This pictorial presents Reduced Perceptual Cues (RPC) as a research-through-design method that shifts attention from interaction events toward the deliberate selection and composition of perceptual cues.

Soft floating robots (SFRs) represent a shift from rigid machines, offering gravity-defying, compliant, and tactile embodiments for indoor cohabitation. However, their development remains fragmented across isolated prototypes, lacking a coherent design vocabulary. Without a systematic understanding of their interactional capabilities, designers struggle to leverage SFRs’ unique affordances, and these systems often remain limited to novelty applications that are difficult to integrate into everyday life. To address this, we propose a design space for interaction with SFRs. Informed by an exploratory study with 12 experts from HCI, Design, and Robotics, we identify ten design dimensions spanning physical, interactive, and behavioral properties, along with a range of application scenarios. We further present proof-of-concept design examples to demonstrate how this design space can support diverse interaction possibilities. This work contributes a structured framework for understanding and designing interactions with SFRs, supporting their integration into everyday indoor environments.

Previous research shows how animacy attribution is shaped by perceptual and cognitive processes based on morphological and behavioural characteristics. Existing evidence is largely focused on passive observation of movement on a screen or scenarios involving predefined interactions. We explore cognitive animacy perceptions of 19 participants during embodied engagement with a spatial-scale kinetic origami surface with no defined interaction modalities. In a mixed-methods study, we collected EEG data, Godspeed Questionnaire evaluations, videos of participants engaging with the surface and semi-structured interviews to understand participants' experiences. Our results show individuals' animacy perceptions are constructed through complex, embodied meaning-making processes, affected by artifacts' morpho-behavioural traits as well as individuals' actions and interpretations. However, widespread brain activations make it hard to pinpoint to specific neurological phenomena. We unpack intrinsic and extrinsic factors shaping animacy perception, articulate the importance of ambiguity in the design of animate' objects and present application scenarios for our prototype.

We present ROomBOT, a novel concept of intelligent and embodied space where the room itself functions as an assistive robotic interface. Most prior research on intelligent spaces has focused on context sensing and home automation, lacking embodied capabilities for directly manipulating everyday physical objects such as books, blankets, and tables. To address this gap, ROomBOT actuates suspended robotic end-effectors that can move the entire room via ceiling-mounted winches, enabling extension of the user’s physical capabilities within the room. This enables assistive scenarios such as allowing wheelchair users to access objects placed at high locations or enabling bedridden individuals to retrieve a water bottle without leaving their bed, transforming ordinary rooms into assistive environments. We propose the interaction concept and design space of ROomBOT, emphasizing safety considerations for real-world deployment, and demonstrate its potential as a dynamic spatial interface that seamlessly augments daily human activities through a range of application scenarios.

We present SonicSlate, a walkable flat-panel speaker interface designed for embodied sound interaction. This low-cost, easy-to-install system integrates conventional modular flooring mechanics, layering auditory information across a space by transmitting sound and vibrations directly from the floor to the body. Unlike traditional 3D audio setups that rely on large or intrusive speaker arrays, SonicSlate delivers an immersive auditory experience without visible hardware, seamlessly extending across rooms through modular flooring. Beyond the interface itself, we explore the interaction design space of SonicSlate, developing a proof-of-concept prototype embedded with a matrix of flat-panel speakers. We provide a brief technical evaluation of the prototype’s spatial audio capabilities and outline unique use cases enabled by its design. We demonstrate how walkable sound interfaces open up new possibilities for spatialized audio experiences that are inherently physical, scalable, and unobtrusive. By integrating sound into floor materials, SonicSlate creates new opportunities for embodied interaction.

Design reflects the human tendency to adapt to and inhabit surroundings, with architectural decisions significantly shaping behavioral and cognitive experiences. This pictorial focuses on the floor as a primary, embodied interface in space. To explore its influence, five floor design typologies (completing, switching, zoning, stimulating, and bending) were identified through twenty hours of expert discussions involving architects, designers, an artist, an engineer, and researchers. A collaborative workshop further defined sub-categories via participatory observations. Fieldwork then informed site selection for an observational study, which confirmed the behavioral and cognitive impacts of the identified typologies. Based on these findings, floor codes were developed by shifting the design focus from visual cues to somatic sensations and applied in design scenarios. This research contributes to understanding human experience in architectural environments. It offers insights for virtual architecture, proposing evidence-based strategies for designing personalized and interactive spaces in virtual and mixed-reality contexts.

To enhance focused eating and dining socialization, previous Human-Food Interaction research has indicated that external devices can support these dining objectives and immersion. However, methods that focus on the food itself and the diners themselves have remained underdeveloped. In this study, we integrated biofeedback with food, utilizing diners' heart rates as a source of the food's appearance to promote focused eating and dining socialization. By employing LED lights, we dynamically displayed diners' real-time physiological signals through the transparency of the food. Results revealed significant effects on various aspects of dining immersion, such as awareness perceptions, attractiveness, attentiveness to each bite, and emotional bonds with the food. Furthermore, to promote dining socialization, we established a "Sharing Bio-Sync Food" dining system to strengthen emotional connections between diners. Based on these findings, we developed tableware that integrates biofeedback into the culinary experience.

Nowadays, touch remains essential for emotional conveyance and interpersonal communication as more interactions are mediated remotely. While many studies have discussed the effectiveness of using haptics to communicate emotions, incorporating affect into haptic design still faces challenges due to individual user tactile acuity and preferences. We assessed the conveying of emotions using a two-channel haptic display, emphasizing individual differences. First, 24 participants generated 187 haptic messages reflecting their immediate sentiments after watching 8 emotionally charged film clips. Afterwards, 19 participants were asked to identify emotions from haptic messages designed by themselves and others, yielding 593 samples. Our findings suggest potential links between haptic message decoding ability and emotional traits, particularly Emotional Competence (EC) and Affect Intensity Measure (AIM). Additionally, qualitative analysis revealed three strategies participants used to create touch messages: perceptive, empathetic, and metaphorical expression.

Social virtual reality (VR) is the experience of a shared virtual space populated by virtual representations of each individual, allowing them to communicate, collaborate and interact with each other not unlike the real world. Conventional avatars generally mirror factors like an individual’s appearance, speech, movement, and so on, yet a VR can offer many more possibilities to represent a person beyond what reality can offer. One such representation is that of emotions and empathy. To that regard, we propose Bioresponsive Avatars, an avatar system that predicts user emotional states and represents them visually via their avatar appearance. To achieve this, we first conducted an avatar design workshop to understand how user’s imagine emotional states to appear on an avatar. Then, we performed an in-the-wild demonstration of a social VR prototype where dyadic users are presented with affective topics to communicate while their avatars adapt based on their predicted emotions. • Developed bioresponsive avatars to map emotions into Valence–Arousal model. • Implemented a real-time avatar using physiological sensing and ML in social VR. • Conducted an in-the-wild study with 55 participants, noting patterns of empathy in VR.

Introspectus AI is a generative AI-based system designed to enhance self-reflection and support positive behavior change. By leveraging multimodal information from users' daily life recordings, it provides personalized and detailed feedback, aiming to deepen self-awareness and facilitate positive behavioral adjustments. This study explores the short-term and long-term impacts of interacting with Introspectus AI, focusing on its potential to enhance reflective practices and improve the acceptance of generative AI tools. Following the user experience was defined through an initial round of workshops with four experts. The resulting system was evaluated through a long-term study involving 64 participants. The results demonstrate that AI-supported interventions significantly improved engagement in self-reflection, the need for reflection, and insight, while also increasing user acceptance of generative AI over time. These findings underscore the potential of generative AI as a practical tool for self-improvement, offering insights into its broader applicability in promoting well-being and personal growth.

Members of the LGBTQIA+ community are more likely to face mental health challenges. However, stigma and the fear of being outed often prevent them from seeking professional support. To address this, we collaborated with mental health professionals and LGBTQIA+ communities in Japan to develop a multi-user Virtual Reality (VR) platform that facilitates access to group therapy sessions. The system allows users to participate using personalized avatars and customized voices, preserving anonymity while enabling them to present themselves as they wish. We conducted a user study with 21 LGBTQIA+ participants and two qualified counselors to evaluate their experiences with VR-based therapy. Findings revealed that the created avatars enabled participants to express their chosen gender identity and increase confidence, acting as protective intermediaries. However, participants also noted how anonymity could affect trust, and suggested that better representation of body language and the introduction of trust-building activities could help compensate for such ambivalence. Overall, the platform fostered a strong sense of co-presence, and both counselors and LGBTQIA+ members felt that, with some ergonomic adjustment to improve the comfort of the headset during longer sessions, VR platforms could offer substantial opportunities for safe and representative access to mental health services.

Neurodivergent individuals, particularly those with Autism and Attention Deficit Hyperactivity Disorder (ADHD), frequently experience anxiety, panic attacks, meltdowns, and emotional dysregulation due to societal pressures and inadequate accommodations. These challenges are especially pronounced for neurodivergent women and non-binary individuals navigating intersecting barriers of neurological differences and gender expectations. This research investigates virtual reality (VR) as a portable safe space for emotional regulation, addressing challenges of sensory overload and motion sickness while enhancing relaxation capabilities. Our mixed-methods approach included an online survey (N=223) and an ideation workshop (N=32), which provided key design elements for creating effective calming VR environments. Based on these findings, we developed and iteratively tested VR prototypes with neurodivergent women and non-binary participants (N=12), leading to a final version offering enhanced adaptability to individual sensory needs. This final prototype underwent a comprehensive evaluation with 25 neurodivergent participants to assess its effectiveness as a regulatory tool. This research contributes to the development of inclusive, adaptive VR environments that function as personalized "portable silent rooms" offering neurodivergent individuals on-demand access to sensory regulation regardless of physical location.

Depending on one’s needs electric wheelchairs can feature different interfaces and driving paradigms with control handed to the user, a remote pilot, or shared. However, these systems have generally been implemented on separate wheelchairs, making comparison difficult. We present the design of an omnidirectional electric wheelchair that can be controlled using two sensing seats detecting changes in the centre of gravity. One of the sensing seats is used by the person on the wheelchair, whereas the other is used as a remote control by a second person. We explore the use of the wheelchair using different control paradigms (independent, remote, and shared) from both the wheelchair and the remote control seat with 5 dyads and 1 triad of participants, including wheelchair users and non. Results highlight key advantages and disadvantages of the SeaHare in different paradigms, with participants’ perceptions affected by their skills and lived experiences, and reflections on how different control modes might suit different scenarios.

Musicians with significant mobility limitations, face unique challenges in being able to use their bodies to interact with fans during live performances. In this paper we present the results of a collaboration between a professional DJ with advanced Amyotrophic Lateral Sclerosis and a group of technologists and researchers culminating in two public live performances leveraging human augmentation technologies to enhance the artist’s stage presence. Our system combines Brain Machine Interface, and accelerometer based trigger, to select pre-programmed moves performed by robotic arms during a live event, as well as for facilitating direct physical interaction during a “Meet the DJ” event. Our evaluation includes ethnographic observations and interviews with the artist and members of the audience. Results show that the system allowed artist and audience to feel a sense of unity, expanded the imagination of creative possibilities, and challenged conventional perceptions of disability in the arts and beyond.

To enhance the dining experience, prior studies in Human-Computer Interaction (HCI) and gastrophysics have demonstrated that modifying the static shape of solid foods can amplify taste perception. However, the exploration of dynamic shape-changing mechanisms in liquid foods remains largely untapped. In the present study, we employ cymatics, a scientific discipline focused on utilizing sound frequencies to generate patterns in liquids and particles—to augment the drinking experience. Utilizing speakers, we dynamically reshaped liquids exhibiting five distinct taste profiles and evaluated resultant changes in taste perception and drinking experience. Our research objectives extend beyond merely augmenting taste from visual to tactile sensations; we also prioritize the experiential aspects of drinking. Through a series of experiments and workshops, we revealed a significant impact on taste perception and overall drinking experience when mediated by cymatics effects. Building upon these findings, we designed and developed tableware to integrate cymatics principles into gastronomic experiences.

Robotic avatars and telepresence technology enable people with disabilities to engage in physical work. Despite the recent popularity of the metaverse, few studies have explored the use of virtual avatars and environments by people with disabilities. In this study, seven disabled participants working in a cafe where remote customer service is provided via robotic avatars, were engaged in the development and use of personalized virtual avatars displayed on a large screen in-situ in combination with existing physical robots, creating a hybrid cyber-physical space. We conducted longitudinal semi-structured interviews to investigate the psychological changes experienced by the participants. The results revealed that mass-produced robotic avatars allowed participants to not disclose their disability if they did not want to, but also backgrounded their identities; by contrast, customized virtual avatars shaped without physical constraints, highlighted their personalities. The combined use of robotic and virtual avatars complemented each other and can support pilots in redefining their identity.

During the grieving process, physical objects often serve as catalysts for remembering and honouring the relationship with departed loved ones. Leveraging a participatory design approach, we created Be.side, a fully customisable multi-modal artefact that incorporates scent, sound, and heartbeat stimulation and acts as a touch-point between the deceased and the bereaved. We conducted a four-week study with three participants to understand how the artefact, continuously attuned to each participant, helped to continue bonds with the deceased. Our results show that Be.side’s bespoke elements helped participants to evoke memories of the deceased. Participants created personalised rituals for remembrance. They sustained bonds by not only interacting with Be.side but also participating in the research. Finally, highlighting that remembrance can both provide comfort and deepen sadness, we discuss future design considerations.

In this paper, we propose a method for utilizing musical artifacts and physiological data as a means for creating a new form of live music experience that is rooted in the physiology of the performers and audience members. By utilizing physiological data (namely Electrodermal Activity (EDA) and Heart Rate Variability (HRV)) and applying this data to musical artifacts including a robotic koto (a traditional 13-string Japanese instrument fitted with solenoids and linear actuators), a Eurorack synthesizer, and Max/MSP software, we aim to develop a new form of semi-improvisational and significantly indeterminate performance practice. It has since evolved into a multi-modal methodology which honors improvisational performance practices and utilizes physiological data which offers both performers and audiences an ever-changing and intimate experience.

Skin temperature is an important physiological factor for human hand dexterity. Leveraging this feature, we engineered an exoskeleton, called DexteriSync, that can dynamically adjust the user’s finger dexterity and induce different thermal perceptions by modulating finger skin temperature. This exoskeleton comprises flexible silicone-copper tube segments, 3D-printed finger sockets, a 3D-printed palm base, a pump system, and a water temperature control with a storage unit. By realising an embodied experience of compromised dexterity, DexteriSync can help product designers understand the lived experience of compromised hand dexterity, such as that of the elderly and/or neurodivergent users, when designing daily necessities for them.

The limited nonverbal cues and spatially distributed nature of remote communication make it challenging for unacquainted members to be expressive during social interactions over video conferencing. Though it enables seeing others' facial expressions, the visual feedback can instead lead to unexpected self-focus, resulting in users missing cues for others to engage in the conversation equally. To support expressive communication and equal participation among unacquainted counterparts, we propose SealMates, a behavior-driven avatar in which the avatar infers the engagement level of the group based on collective gaze and speech patterns and then moves across interlocutors' windows in the video conferencing. By conducting a controlled experiment with 15 groups of triads, we found the avatar's movement encouraged people to experience more self-disclosure and made them perceive everyone was equally engaged in the conversation than when there was no behavior-driven avatar. We discuss how a behavior-driven avatar influences distributed members' perceptions and the implications of avatar-mediated communication for future platforms.

The development of novel computer interfaces has led to the possibility of integrating inputs from multiple individuals into a single avatar, fostering collaboration by combining skills and sharing the cognitive load. However, the collaboration dynamic and its effectiveness may vary depending on the individuals involved. Particularly in scenarios where two individuals remotely control a robotic avatar without the possibility of direct communication, understanding each other’s characteristics can result in enhanced performance. To achieve this, it is essential to ascertain if individuals can discern their partner’s characteristics within the merged embodiment. This paper investigates the accuracy with which participants can distinguish between two different collaborating partners (one attempting to lead and one attempting to follow) when sharing control of a robot arm during a block pick-and-place task. The results suggested that participants who changed their roles according to the different roles of the two partners achieved the highest discrimination rates. Furthermore, participants changed their movements through the trials, adapting their actions to their preferred approach. This research provides insights into the factors determining individuals’ ability to understand partner characteristics during control of collaborative avatars.

Robotic avatars can help disabled people extend their reach in interacting with the world. Technological advances make it possible for individuals to embody multiple avatars simultaneously. However, existing studies have been limited to laboratory conditions and did not involve disabled participants. In this paper, we present a real-world implementation of a parallel control system allowing disabled workers in a café to embody multiple robotic avatars at the same time to carry out different tasks. Our data corpus comprises semi-structured interviews with workers, customer surveys, and videos of café operations. Results indicate that the system increases workers’ agency, enabling them to better manage customer journeys. Parallel embodiment and transitions between avatars create multiple interaction loops where the links between disabled workers and customers remain consistent, but the intermediary avatar changes. Based on our observations, we theorize that disabled individuals possess specific competencies that increase their ability to manage multiple avatar bodies.

Dementia describes a syndrome of cognitive degeneration, and Behavioural and Psychological Symptoms of Dementia (BPSD) is the non-cognitive symptom. BPSD can be improved by care services. To aid better care service, we explore the potential of using Augmented Reality (AR) to support dementia education for medical workers in three steps: (1) We explore medical workers’ perspective on dementia care lived experience and XR, (2) we co-design an educational experience containing an AR-based application and a 5-min activity with medical workers, (3) we evaluate the effectiveness of the system through a mixed method study. Our result shows that the AR experience successfully touches participants, and motivates them to reflect on the provision of care service. On this basis, we discuss the elements and challenges of designing XR-enabled dementia education for users unfamiliar with novel technology, and the potential of using XR in clinical education.

The human arm has seven DoF. If the human arm were more redundant, it would improve workability. However, intuitive manipulation of redundant arms is difficult due to the complexity of the input system or the limitations of the human arm. In this study, we constructed a system in which multiple operators integrate their bodies to intuitively operate a redundant multi-DoF robotic arm. The robotic arm was assembled by connecting two arms, and the roles of each section –distal arm and proximal arm– were assigned to different operators. First, vibrotactile feedback methods to the users based on the operation movement were investigated to compensate for the recognition of the partner’s movement. Three types of feedback methods —the actions of both oneself and one’s partner, the actions of the partner, and none— were compared in order to improve the body awareness. It was shown that feedback of the movement from the partner who operates the proximal robotic arm significantly enhanced the sense of body ownership. Then, the proposed co-operation of the redundant robotic arm was evaluated in the workability. Two types of manipulation methods —a single-operator system that switched the target robotic arm and a multi-DoF body-integration system used by two people— were compared through an object-manipulation task involving obstacle avoidance. It was shown that the two-person co-operation through body integration system to be faster than the single-person operation.

Humans have spatiotemporal limitations. Robotic avatar technology enables the expansion of an individual's innate physical characteristics, physical capabilities, and perceptual and cognitive abilities. Our research group has focused on scalability and co-creativity through co-operation and collaboration between humans, and has developed a system in which two operators are physically integrated into a single robotic arm. In particular, to enable more human-like tasks, such as receiving and delivering, we extended the system to a dual-arm robotic avatar and built a system in which multiple persons can integrate their bodies, with the expectation of further task efficiency and human extension. Specifically, we investigated the workability and operator cognition of the dual-arm robotic avatar when it was operated by one, two, or three individuals. The results showed that stability improved with the three-person operation. Furthermore, it was found that co-operation among multiple persons can overcome the original individual abilities for operation. In the three-person condition, all participants obtained a higher feeling of control than the actual amount of manipulation, which has potential applications in skill transfer. The three-person condition tended to be better, validating the usefulness of multi-person body integration.

The use of wearable sensor technology opens up exciting avenues for both art and HCI research, providing new ways to explore the invisible link between audience and performer. To be effective, such work requires close collaboration between performers and researchers. In this article, we report on the co-design process and research insights from our work integrating physiological sensing and live performance. We explore the connection between the audience’s physiological data and their experience during the performance, analyzing a multi-modal dataset collected from 98 audience members. We identify notable moments based on HRV and EDA, and show how the audience’s physiological responses can be linked to the choreography. The longitudinal changes in HRV features suggest a strong connection to the choreographer’s intended narrative arc, while EDA features appear to correspond with short-term audience responses to dramatic moments. We discuss the physiological phenomena and implications for designing feedback systems and interdisciplinary collaborations.

Tactile sensation is essential for recognizing mechanical interactions between the human body and its environment; it is generally uniquely experienced by individuals. Haptic sharing technologies enable to share of other person’s (or robot’s) tactile sensations, and it is expected to apply for skill learning, collaborative work, and robot operation. While wearable tactile sensors are useful for tactile measurements, they are affected by differences in physical characteristics between individuals. Therefore, to accurately convey the tactile information felt by others and utilize it for work, it is necessary to remove individual differences due to differences in individual physical characteristics. In this study, based on the frequency based intensity of skin vibration of an individual, the vibration detected by the acceleration sensor affixed to the fingertip was modulated to remove individual differences. To confirm the effectiveness of the proposed method, we investigated both physical and perceptual similarities for skin vibrations collected from three persons with three different fine textures. The results showed that vibrations individually modulated with the proposed method was similar in the intensity and power spectrum density than those using the uniform modulation function, and was perceived similar when presenting those with a vibrator.

In stage performances, an invisible wall in front of the stage often weakens the connections between the audience and performers. To amplify this performative connection, we present the concept ”Boiling Mind”. Our design concept is based on streaming sensor data related to heart and electrodermal activities from audience members and integrating this data into staging elements, such as visual projections, music, and lighting. Thus, the internal states of the audience directly influence the staging. Artists can have a more direct perception of the inner reactions of audience members and can create physical expressions in response to them. In this paper, we present the wearable sensing system as well as design considerations of mapping heart and electrodermal activity to changes in the staging elements. We evaluated our design and setup over three live performances.

Tactile sensation contributes to sensory-motor control and facilitates skillful manipulation of objects. If the tactile sensation can be shared between two partners, the state of each partner can be partially observed by the other, and the possibilities for cooperative work may be expanded. Recently, examples of utilizing tactile sensation in human-robot cooperative work have been proposed. For cooperative work between people, it is necessary to investigate the basic ability of humans to identify other person's motions and operations by tactile presentation. To avoid hindering work performed with the hands and fingertips, the sensation must be presented elsewhere. This study investigates the possibility of identifying other person's tool operations by presenting tactile information induced on his/her finger with a wearable vibrator on the arm. A wearable skin vibration sensor was employed to acquire tactile information during an experiment in which five different tool operations were tested. This sensor measures skin vibration while directly touching the target. We proposed a non-linear signal processing function to adjust the intensity of the skin vibration to within the range of human sensitivity for tactile presentation. We compared vibrotactile stimulation between the non-linear and linear corrections, and then conducted experiments on identifying operations. The results showed that the non-linear correction increased small signals and enhanced the variance of large signals, and that operations were significantly identified by tactile presentation to the arm.

Smartwatches enable the wrist to be used as an ideal location to provide always-available haptic notifications as they are constantly worn with direct contact with the skin. With the wrist straps, the haptic feedback can be extended to the full space around the wrist to provide more spatial and enriched feedback. With ThermalBracelet, we investigate thermal feedback as a haptic feedback modality around the wrist. We present three studies that lead to the development of a smartwatch-integratable thermal bracelet that stimulates six locations around the wrist. Our initial evaluation reports on the selection of the thermal module configurations. Secondly, with the selected six-module configuration, we explore its usability in a real-world scenarios such as walking and reading. Thirdly, we investigate its capability of providing spatio temporal feedback while engaged in distracting tasks. Finally we present application scenarios that demonstrates its usability.

This study demonstrates that by opening and closing the human pinna, we can change the direction of sound perceived by humans. Each ear was independently transformed into a 100% open, 50% open, and 100% closed state, and all 9 combinations of these ear transformations were tested to evaluate the perceived direction of the sound output from 7 speakers placed 180 degrees around the subject. We demonstrate that by deforming the pinna, we could change the perception of the direction of sound, or make it illusory. We also found that except for 1 out of 7 speakers (or directions of sound), closing 100% of the ear on the side of the speaker where the sound is coming from and 50% of the ear on the other side of the speaker tends produce the most alteration to the perceived direction of sound.

Gaze-based interactions promise to be fast, intuitive and effective in controlling virtual and augmented environments. Yet, there is still a lack of usable 3D navigation and observation techniques. In this work: 1) We introduce a highly advantageous orbital navigation technique, AnyOrbit, providing an intuitive and hands-free method of observation in virtual environments that uses eye-tracking to control the orbital center of movement; 2) The versatility of the technique is demonstrated with several control schemes and use-cases in virtual/augmented reality head-mounted-display and desktop setups, including observation of 3D astronomical data and spectator sports.

Gaze-based interactions promise to be fast, intuitive and effective in controlling virtual and augmented environments. Yet, there is still a lack of usable 3D navigation and observation techniques. In this work: 1) We introduce a highly advantageous orbital navigation technique, AnyOrbit, providing an intuitive and hands-free method of observation in virtual environments that uses eye-tracking to control the orbital center of movement; 2) The versatility of the technique is demonstrated with several control schemes and use-cases in virtual/augmented reality head-mounted-display and desktop setups, including observation of 3D astronomical data and spectator sports.

Robots have replaced people in many manufacturing production lines but the information they gather from sensors might not be sufficient to autonomously accomplish dexterous manipulation operations. Symbiotic human-robot cooperation appears to be a more realistic near future in industrial scenarios. In this paper we present a configuration of human-robot collaboration in which the robot is sensory-augmented by means of a set of tactile signals coming from the human operator. The incorporation of low-level robot “intelligence” permits the cooperative manipulation of an object while enabling the human operator to stay focused on task itself and carry it out in the most natural way. The effectiveness of this approach is demonstrated in a use case in which a robot helps a human operator to successfully accomplish a writing task. System performance has been evaluated, considering several positions of the tiny vibration sensor in charge of gathering the human perception, by testing it on both the human hand and the co-manipulated object. Results suggest that the sensor provides valuable information for recognizing operator actions when it is placed either on the human hand or on the co-manipulated object. However, the sensor on the finger directly represents the operator's perception, while the output of the sensor attached to the object changes according to the distance between the interaction point and the sensor itself. In addition, in wearing the sensor, neither the object nor the robot need to be instrumented: the operator is free to interact with a large set of objects and collaborate with any existing robot without requiring supplemental equipment.

The purpose of this paper is to introduce using scalable autostereoscopic displays that allow users to interact with floating images publicly, such as for digital signage. By tiling multiple display modules, it is possible to project larger autostereoscopic images in the air than by displaying it using a single display module. By projecting autostereoscopic images, it is possible for the user to understand the contents in better detail compared to a standard 2D public displays. Furthermore, by projecting floating images, it is possible to reduce the spatial divergence between the user's body and the autostereoscopic images, so that the user can interact intuitively with autostereoscopic images. Moreover, since this system has scalability, it is possible to expand the screen size just by adding more display modules. This expandability makes the system easy to move and arrange in public spaces. In this paper, we propose an autostereoscopic display module which can enlarge the screen size in the horizontal direction as the first step to realizing our concept.

We introduce MetaArms, wearable anthropomorphic robotic arms and hands with six degrees of freedom operated by the user's legs and feet. Our overall research goal is to re-imagine what our bodies can do with the aid of wearable robotics using a body-remapping approach. To this end, we present an initial exploratory case study. MetaArms' two robotic arms are controlled by the user's feet motion, and the robotic hands can grip objects according to the user's toes bending. Haptic feedback is also presented on the user's feet that correlate with the touched objects on the robotic hands, creating a closed-loop system. We present formal and informal evaluations of the system, the former using a 2D pointing task according to Fitts' Law. The overall throughput for 12 users of the system is reported as 1.01 bits/s (std 0.39). We also present informal feedback from over 230 users. We find that MetaArms demonstrate the feasibility of body-remapping approach in designing robotic limbs that may help us re-imagine what the human body could do.

Illusory body ownership can be induced by visual-tactile stimulation or visual-motor synchronicity. We aimed to test whether a right thumb could be remapped to a virtual right arm and illusory body ownership of the virtual arm induced through synchronous movements of the right thumb and the virtual right arm. We presented the virtual right arm in synchronization with movements of a participant's right thumb on a head-mounted display (HMD). We found that the participants felt as though their right thumb became the right arm, and that the right arm belonged to their own body.

Head Mounted Displays (HMDs) provide a promising opportunity for providing haptic feedback on the head for an enhanced immersive experience. In ThermoVR, we integrated five thermal feedback modules on the HMD to provide thermal feedback directly onto the user's face. We conducted evaluations with 15 participants using two approaches: Firstly, we provided simultaneously actuated thermal stimulations (hot and cold) as directional cues and evaluated the accuracy of recognition; secondly, we evaluated the overall immersive thermal experience that the users experience when provided with thermal feedback on the face. Results indicated that the recognition accuracy for cold stimuli were of approx. 89.5% accuracy while the accuracy for hot stimuli were 68.6%. Also, participants reported that they felt a higher level of immersion on the face when all modules were simultaneously stimulated (hot and cold). The presented applications demonstrate the ThermoVR's directional cueing and immersive experience.

A head-mounted display (HMD) immerses users in a virtual world, but separates them from outsiders in the real world. We present FrontFace, which is a novel HMD that combines an eye-tracker with a front-facing screen, to lower the communication barrier between HMD users and outsiders. The front-facing screen reveals user attention (e.g., the users eye motions) and user presence in the virtual or real world by displaying the scene in the virtual world or a skin background respectively, enabling eye-contact interactions between the HMD user and the outsiders. FrontFace has the following benefits. Firstly, it communicates the presence of the HMD user to outsiders; secondly, it reveals the player's visual attention by introducing the HMD users originally occluded eye motions, enabling outsiders to make sense of the HMD user's reaction in the virtual world or the real world. Three interactive techniques for the outsiders to initiate communication to HMD users are proposed: they are tap-trigger, hand-gesture trigger, and voice-trigger interactions. A small focus group provided feedback.

This paper presents our preliminary exploration of providing dynamic thermal haptic feedback on the face. We have designed a thermal feedback prototype that integrates Peltier modules directly with a head mounted display which is able to provide thermal output directly on users' faces. We explored the possibility of providing dynamic thermal feedback which could present information and enhance user experience through designing four initial dynamic thermal patterns of our prototype. We also conducted a preliminary user experiment and analysis of the design of different patterns ("dynamic information") to evaluate the design on three aspects, the accuracy of recognition of dynamic stimuli, smoothness of the movement and comfort level. Our results indicate that the accuracy of recognition of dynamic stimuli was approximately 71.84%.

We developed HapticAid, which is a wearable haptic augmentation system. This system enhances the skin vibrations that occur during touch and provide them as haptic feedback on the wrist. We revised HapticAid to be wearable and mobile by using a mobile platform and installed a digital signal processing to control haptic feedback for haptic experiences. This paper describes the implementation of our system, design of digital signal processing the method of wearing our system for haptic experiences.

Here, we propose that the use of unusual crossed-sense-modality couplings can be exploited to engineer non-realistic but potentially beneficial VEs providing high degrees of interaction and presence, and provide new avenues for enhancing the experience and utility of single- and multi-user VE technologies. We report user's reactions to mediated environments in which the sound of their voice is represented via a frequency-to-color-mapped visualization. Users reported a sense of interaction and enjoyment, but results from a preliminary user study have yet to show a clear link with presence. We discuss the next stages of the research which will investigate how mixed-sense modalities contribute to co-presence and collaboration in multi-user environments.

We propose an indirect-vision, video-see-through augmented reality (AR) cockpit that uses telexistence technology to provide an AR enriched, virtually transparent view of the surroundings through monitors instead of windows. Such a virtual view has the potential to enhance driving performance and experience above conventional glass as well as head-up display equipped cockpits by combining AR overlays with images obtained by future image sensors that are superior to human eyes. As a proof of concept, we replaced the front windshield of an experimental car by a large stereoscopic monitor. A robotic stereo camera pair that mimics the driver's head motions provides stereoscopic images with seamless motion parallax to the monitor. Initial driving tests at moderate speeds on roads within our research facility confirmed the illusion of transparency. We will conduct human factors evaluations after implementing AR functions in order to show whether it is possible to achieve an overall benefit over conventional cockpits in spite of possible conceptual issues like latency, shift of viewpoint and short distance between driver and display.

Rubber Hand Illusion (RHI) is known to provide a proprioceptive drift toward the direction where the rubber hand is tapped. On the other hand, Out-of-Body Experiences (OBEs) induce an entire body shift from a third-person Point of View (PoV). These experiences are known to trigger external passive tactile cues. Telexistence allows an entire body shift from a first-person PoV, compared to RHI and OBE. However, the precise origin of the entire localization is not revealed when we actively provide the internal voluntary tactile cue. Therefore, in this paper, we focus on the effects of tactile cues to determine self-localization during the action of self-tapping. Participants viewed and tapped their own body on the back, and evaluated where on their body they perceived the tapping. The experiments show that the localized position tends to shift forward when an individual is tapped by a third-person; in contrast, the localized position tends to shift backward when an individual is tapping his own back. The subjective report indicates that participants perceived themselves as leaving their own bodies, or that someone appeared in front of them suddenly. Thus, we consider that self-produced tactile cue induces telexistence experience.

We present Reality Jockey, a system that confuses the participant's perception of the reality by mixing in a recorded past-reality. The participant will be immersed in a spatialized 3D sound environment that is a mix of sounds from the reality and from the past. The sound environment from the past is augmented with haptic feedback in cross-modality. The haptic feedback is associated with certain sounds such as the vibration in the table when stuff is placed on the table to make the illusion of it happening in live. The seamless transition between live and past creates immersive experience of past events. The blending of live and past allows interactivity. To validate our system, we conducted user studies on 1) does blending live sensations improve such experiences, and 2) how beneficial is it to provide haptic feedbacks in recorded pasts. Potential applications are suggested to illustrate the significance of Reality Jockey.

In this paper, we propose a haptic transmission system for telexistence to improve the ability to sense the presence of remote objects. This system can transmit information about the existence and surface textures of objects in remote locations. The system consists of a conjugated haptic sensor and display. The sensor on the robot's finger detects the pressure, vibration, and temperature of a remote object, and the display provides above information on the operator's finger. Based on this information, the operator can understand what he/she is touching and whether its surface is hard or soft, cold or hot, and smooth or rough. With the use of our system, the operator can recognize the difference between objects such as silk and denim.

This paper focuses on design of a dexterous anthropomorphic robot where the operator can perceive the transferring bodily consciousness to the slave robot during a tele-operation. Accordingly, we propose a telexistence surrogate anthropomorphic robot called “TELESAR V”, which was designed and constructed by development of the following: a 52 DOF slave robot with a torso, upper limbs, hands and head to model the operator's posture on all parts of the upper body and maintain a 6 DOF accuracy in arm endpoint; a HD Head mounted display with 6 DOF point of view accuracy for wide angle stereovision; and a mechanism for sensing and reproducing fingertip haptic and thermal sensation. This paper describes the development of the TELESAR V system, where the effectiveness has been verified through functional experiments.

Telexistence technology enables a highly realistic sensation of existence in a remote place without any actual travel. The concept was originally proposed by the first author in 1980, and its feasibility has been demonstrated through the construction of alter-ego robot systems such as TELESAR & TELESAR V, which were developed under the national large scale project on “Robots in Hazardous Environments” and the “CREST Haptic Telexistence Project.” A mutual telexistence system, such as TELESAR II & IV, capable of generating the sensation of being in a remote place in local space using a combination of an alter-ego robot and retro-reflective projection technology (RPT), has been developed, and the feasibility of mutual telexistence has been demonstrated. Thirty-two years of telexistence development are historically reviewed in this jubilee video.

This paper presents a new type of optical camouflage system based on the retro-reflective projection technology. Retro-reflective projection is a method used to create augmented reality that combines the virtual world with the real world. The conventional model of an optical camouflage system consists of a retro-reflective screen, a projection source and a beam splitter. In such a setup, the user needs to observe an object covered with the retro-reflective screen through a single viewpoint. This is called a monocular system. In our new setup, our aim is to construct a system that has multiple viewpoints by applying a novel projection array system. We will describe the method with which this projection array system is achieved using one projection source, the configuration of the system, and the trade-offs of the system. In addition, we will describe an application of our system in a car. The installed system makes the backseat virtually transparent, allowing the driver to see the blind spots at the rear when reversing the car.

In this paper, we implement an ungrounded pen-shaped kinesthetic display and construct a three-dimensional (3D) haptic interaction system. The ungrounded pen-shaped kinesthetic display provides kinesthetic sensations to a user's fingers without the use of mechanical linkages. Therefore, the user can move his/her hand freely in air and interact with virtual environments with sensation of touching. We verified the ability of the device to provide forces and to represent a virtual surface for 3D input. Then we constructed a 3D haptic interaction system in that the user can touch virtual objects displayed as 3D images directly. We also applied the device to an interface for 3D modeling and constructed a 3D haptic modeling system in that the user can create touchable 3D models by sketching two-dimensional (2D) figures in air.

Our aim is to afford a remote person the opportunity to participate virtually in some event by using a surrogate robot to communicate with local participants while moving about freely at the venue. Accordingly, we propose a mutual telexistence surrogate robot system, called TELESAR4, which was designed and constructed by development of the following: an immersive audiovisual system; an omnidirectionally mobile robot with a robot arm and hand; an omnidirectional stereo camera system (VORTEX); a head with a retroreflective screen for embodiment of the remote participant; and a retroreflective projection system for local participants. This paper describes the development of the TELESAR4 system, whose efficacy has been verified through demonstration experiments.

We propose a three-dimensional (3D) haptic modeling system that enables a user to create 3D models as though he/she is drawing pictures in mid-air and touch the created models. In the proposed system, we use our ungrounded pen-shaped kinesthetic display as the interface. The kinesthetic display device can generate kinesthetic sensations on the user's fingers. Because our system does not use mechanical linkages, the user can freely move his/her hand and feel the sensation of touching virtual objects. In the proposed system, the user can easily and intuitively create various 3D shapes by drawing closed curves in air using the device. The created shapes are generated in a physics-based simulation environment and are displayed as 3D images. Therefore, the user can touch and see the drawn shapes as though they exist in reality.

This paper presents some preliminary results on RemoTouch, a system allowing to perform experiences of remote touch. The system consists of an avatar equipped with an instrumented glove and a user wearing tactile displays allowing to feel the remote tactile interaction. The main features of RemoTouch are that it is a wearable system and that a human avatar is used to collect remote tactile interaction data. New paradigms of tactile communication can be designed around the RemoTouch system. Two simple experiences are reported to show the potential of the proposed remote touch architecture.

We proposed a pen-shaped handheld haptic display that provides kinesthetic sensations to the fingers of a user without the use of mechanical linkages. The user's movements are not restricted since the device does not have mechanical linkages, and the user can enjoy haptic interactions with virtual environments. In order to downsize the device, we designed our device on the basis of a hypothesis that kinesthetic sensations on the user's fingers alone are sufficient to represent the sensations of touch. We implemented a prototype device and performed an experiment to confirm the representational ability of our device. We also developed a prototype haptic augmented reality system, using which the user can see and touch a computer graphics object.

For safety and operability of drivers while operating a vehicle, it is very important to obtain a wide field of vision. However, the space available for setting up windows is limited. Therefore, we propose a ";transparent cockpit,"; in which the image of a blind spot is displayed on the inner wall of the vehicle using a retro-reflective projection technology. In this system, the internal components of the vehicle, such as the doors and floor, are virtually transparent, and the blind spot is clearly visible, as observed from a window. In this paper, we describe the implementation of a prototype of the proposed system and demonstrate its effectiveness by an evaluation experiment.