In office environments, achieving flexible workspace configurations by adjusting the position, height, and curvature of partitions is crucial for accommodating diverse worker needs, but implementing such flexibility typically introduces system complexities. We present OrigamiWalls, a robotic partitioning system that addresses this trade-off by using origami structures to permit diverse configurations without requiring high-degree-of-freedom actuation. Based on expert insights, we explore the OrigamiWalls design space and report on the implementation of the desktop-scale and floor-scale prototypes utilizing the Tachi-Miura polyhedron structure. These prototypes can transition continuously between three distinct forms (no wall, curved wall, and flat wall) with high expansion ratios (up to 660%) using a single reel-type linear actuator. Our preliminary user study with our desktop-scale prototype suggests that origami-based shape changes are perceived to enable task-adaptive workspace configurations and support workers’ task transitions unobtrusively and safely.

Habitual static stretching is recommended in many ergonomics guidelines, but continuing this habit remains challenging to workers. We address two factors causing this difficulty: most workers lack an understanding of the appropriate mechanisms of stretching, and stretching sessions require the interruption of ongoing tasks. We propose Screen-Directed Stretching, a novel technique that supports neck stretching in extended reality (XR) workspaces without interrupting work. Our technique temporarily repositions the virtual screen that the user is focusing on, guiding them to rotate their head in yaw, pitch, and roll directions and to maintain a posture for a specific duration, thus facilitating muscle extension. Through two preliminary user studies, we developed a prototype that seamlessly switches the screen’s coordinate system between world-bound and body-bound frames of reference, balancing practical workability and good stretching guidance. A user study (N = 16) demonstrates that our technique effectively induces stretching movements while minimizing loss of task efficiency.

Eye strain presents a significant challenge in human-information interaction in virtual reality (VR), as prolonged exposure contributes to various eye problems. This study introduces a new gaze redirection method called EyeXRciser, which passively activates eye movements to help prevent eye muscle stiffness during VR reading. This method achieves gaze redirection by slowly shifting the relative position of the text window within the user’s field of view through a head-bound coordinate system. We implemented our method with two different redirection speeds (i.e., unnoticeable speed at 0.03 rad/s; noticeable speed at 0.12 rad/s) and conducted a user study (N = 24) comparing with a baseline using a fixed text window. Results show that both our methods successfully minimized the decline in accommodative ability caused by prolonged reading, without negatively impacting reading comprehension. Results also show that the unnoticeable redirection speed produced less subjective discomfort, eye fatigue, and reading distraction than the noticeable speed.

Current Mixed Reality (MR) headsets are typically controlled by mid-air hand inputs, which require significant arm effort and consequently reduce input speed and accuracy during extended sessions. We propose BlanchTouch, a research prototype that explores an alternative touch input approach for MR by appropriating uninstrumented flat surfaces as interactive areas, with reduced arm effort. It incorporates a CNN model that detects the fingertip’s pressure-induced blanching phenomenon when a firm touch is applied to hard surfaces, serving as a confirmation of selection. Additionally, hand tracking via the headset is employed to determine the fingertip’s position, thereby providing targeting information for virtual buttons or objects. BlanchTouch operates using only the headset’s built-in cameras and sensors, without requiring additional hardware or prior surface calibration, enabling walk-up use in unfamiliar environments. We conducted two user studies to evaluate the system and compare its performance against a baseline (a real touchscreen) and the state-of-the-art mid-air interaction. Results demonstrated that BlanchTouch achieved an average input positional error of 5.33mm and a recall rate of 97.88%. Furthermore, the findings indicate that BlanchTouch enables competitive input speeds, alleviates arm strain, and delivers satisfactory feedback for each touch.

Drawing, writing, and painting on 2D surfaces in VR offers arbitrary interaction with virtual canvases of unlimited size. However, this advantage has not been fully exploited because physical surfaces are needed to support accurate and comfortable user interactions, and the size of such surfaces is limited. We propose Redirected Drawing, a novel methodology that applies visual manipulation to the user’s drawing movement in VR to expand the perceived surface size required for the drawing experience. To this end, this study specifically explores translation gain manipulation, where the displacement of a drawing stroke in reality is simply multiplied by a gain to represent it in VR. We conducted a user study (N = 18) to measure the noticeability, acceptability, and task performance of a simple VR line-tracing task under different gain and input methods (i.e., finger in mid-air, finger on surface, pen on surface). Our results show that pen-based drawing on a physical surface allows for a gain of up to 1.16 without being noticed by the user, which is more suited to expanding a virtual canvas size than finger-based drawing on a physical surface or in mid-air. Furthermore, we show that user acceptability of the gain manipulation could be estimated by a model using the subjective discrepancy caused by the gain. Results for this model suggest that users could accept gains of up to 1.30. Based on our results, we derive guidelines for the effective use of gain manipulation.

Dental cutting is a crucial skill for dental students. However, current VR dental cutting training systems rely on bulky and costly haptic devices, which reduce opportunities for individual practice. Moreover, the limitations imposed by the maximum reaction force of an active haptic device would impact the range of tooth hardness that can be reproduced. We propose a compact XR tooth-cutting training system, VirtuEleDent, that employs a passive haptic approach using a 3D-printed physical teeth model and a three-dimensionally tracked handpiece. Their spatial relationship is accurately rendered in the virtual environment of a mobile head-mounted display (HMD), providing users with realistic haptic sensations during virtual tooth-cutting exercises. Our tracking platform is operated using electromagnetic resonance (EMR) stylus technology and consists of a digitizer (i.e., tracking board) and a handpiece device. A customized EMR stylus unit (i.e., resonance coil) and an inertial measurement unit (IMU) sensor are installed inside the handpiece, allowing for precise measurement of its tip’s 3D position and orientation. This setup enables the learner to physically manipulate dexterous handpieces on the teeth model while experiencing virtual tooth-cutting in the HMD. First, we detail the design and implementation of our initial prototype system, including the tracking platform. We then describe an expert user study conducted with our prototype, where licensed dentists compare VirtuEleDent to a baseline with a conventional haptic device. Finally, the educator’s feedback demonstrates its potential and informs our further improvements and the directions of future development.

We propose SnapSteer, a bimanual 3D manipulation interface using common VR controllers, which allows restriction of motion degrees of freedom (DoFs) as needed. This interface is based on the conventional one-handed 6-DoF manipulation interface called Robot Telekinesis, and assigns the other hand the role of controlling whether and in which direction the DoFs are restricted. This enables users to quickly switch between unconstrained 6-DoF operation and precise 1-DoF operation according to the task. We designed and implemented a prototype of this interface in VR, and conducted a user study (N=12) comparing its performance in a straight 3D steering task with two baseline interfaces (i.e., a 6-DoF individual control interface and Robot Telekinesis). The results showed that our interface outperformed the other two in task efficiency. On the other hand, there was no significant difference in subjective workload or usability compared to Robot Telekinesis, and which derives discussion of improvements to visual feedback during the direction adjustment phase.

Building higher-quality image classification models requires better performance analysis (PA) to help understand their behaviors. We propose ConfusionLens, a dynamic and interactive visualization interface that augments a conventional confusion matrix with focus+context visualization. This interface allows users to seamlessly switch table layouts among three views (overall view, class-level view, and between-class view) while observing all of the instance images in a single screen. We designed and implemented a ConfusionLens prototype that supports hundreds of instances, and then conducted a user study (N \(=\) 14) to evaluate it compared to the conventional confusion matrix with a split view of instances. Results show that ConfusionLens achieved faster task-completion time in observing instance-level performance and higher accuracy in observing between-class confusion. Moreover, we conducted an expert interview (N \(=\) 6) to investigate the applicability of our interface to practical PA tasks, and then implemented several extensions of ConfusionLens based on the feedback. Feedback on these extensions from users experienced in image classification (N \(=\) 5) demonstrated their general usefulness and highlighted their beneficial use in PA tasks.

We introduce InflatableBots, shape-changing inflatable robots for large-scale encountered-type haptics in VR. Unlike traditional inflatable shape displays, which are immobile and limited in interaction areas, our approach combines mobile robots with fan-based inflatable structures. This enables safe, scalable, and deployable haptic interactions on a large scale. We developed three coordinated inflatable mobile robots, each of which consists of an omni-directional mobile base and a reel-based inflatable structure. The robot can simultaneously change its height and position rapidly (horizontal: 58.5 cm/sec, vertical: 10.4 cm/sec, from 40 cm to 200 cm), which allows for quick and dynamic haptic rendering of multiple touch points to simulate various body-scale objects and surfaces in real-time across large spaces (3.5 m x 2.5 m). We evaluated our system with a user study (N = 12), which confirms the unique advantages in safety, deployability, and large-scale interactability to significantly improve realism in VR experiences.

Videos accompanied by documents—document-based videos—enable presenters to share contents beyond videos and audience to use them for detailed content comprehension. However, concurrently exploring multiple channels of information could be taxing. We propose SwapVid, a novel interface for viewing and exploring document-based videos. SwapVid seamlessly integrates a video and a document into a single view and lets the content behaves as both video and a document; it adaptively switches a document-based video to act as a video or a document upon direct manipulation (e.g., scrolling the document, manipulating the video timeline). We conducted a user study with twenty participants, comparing SwapVid to a side-by-side video/document views. Results showed that our interface reduces time and physical workload when exploring slide-based documents based on video referencing. Based on the study findings, we extended SwapVid with additional functionalities and demonstrated that it further extends the practical capabilities.

Traditionally, sports commentators provide viewers with diverse information, encompassing in-game developments and player performances. Yet young adult football viewers increasingly use mobile devices for deeper insights during football matches. Such insights into players on the pitch and performance statistics support viewers’ understanding of game stakes, creating a more engaging viewing experience. Inspired by commentators’ traditional roles and to incorporate information into a single platform, we developed AiCommentator, a Multimodal Conversational Agent (MCA) for embedded visualization and conversational interactions in football broadcast video. AiCommentator integrates embedded visualization, either with an automated non-interactive or with a responsive interactive commentary mode. Our system builds upon multimodal techniques, integrating computer vision and large language models, to demonstrate ways for designing tailored, interactive sports-viewing content. AiCommentator’s event system infers game states based on a multi-object tracking algorithm and computer vision backend, facilitating automated responsive commentary. We address three key topics: evaluating young adults’ satisfaction and immersion across the two viewing modes, enhancing viewer understanding of in-game events and players on the pitch, and devising methods to present this information in a usable manner. In a mixed-method evaluation (n=16) of AiCommentator, we found that the participants appreciated aspects of both system modes but preferred the interactive mode, expressing a higher degree of engagement and satisfaction. Our paper reports on our development of AiCommentator and presents the results from our user study, demonstrating the promise of interactive MCA for a more engaging sports viewing experience. Systems like AiCommentator could be pivotal in transforming the interactivity and accessibility of sports content, revolutionizing how sports viewers engage with video content.

In room-scale virtual reality, providing continuous haptic feedback from touching grounded objects, such as walls and handrails, has been challenging due to the user’s walking range and the required force. In this study, we propose LoopBot, a novel technique to provide continuous haptic feedback from grounded objects using only a single user-following robot. Specifically, LoopBot is equipped with a loop-shaped haptic prop attached to an omnidirectional robot that scrolls to cancel out the robot’s displacement, giving the user the haptic sensation that the prop is actually fixed in place, or “grounded.” We first introduce the interaction design space of LoopBot and, as one of its promising interaction scenarios, implement a prototype for the experience of walking while grasping handrails. A performance evaluation shows that scrolling the prop cancels 77.5% of the robot’s running speed on average. A preliminary user test (N = 10) also shows that the subjective realism of the experience and the sense of the virtual handrails being grounded were significantly higher than when the prop was not scrolled. Based on these findings, we discuss possible further development of LoopBot.

RedirectedDoors is a visuo-haptic door-opening redirection technique in VR, and it has shown promise in its ability to efficiently compress the physical space required for a room-scale VR experience. However, its previous implementation has only supported laboratory experiments with a single door opening at a fixed location. To significantly expand this technique for room-scale VR, we have developed RedirectedDoors+, a robot-based system that permits consecutive door-opening redirection with haptics. Specifically, our system is mainly achieved with the use of three components: (1) door robots, a small number of wheeled robots equipped with a doorknob-like prop, (2) a robot-positioning algorithm that arbitrarily positions the door robots to provide the user with just-in-time haptic feedback during door opening, and (3) a user-steering algorithm that determines the redirection gain for every instance of door opening to keep the user away from the boundary of the play area. Results of simulated VR exploration in six virtual environments reveal our system's performance relative to user walking speed, paths, and number of door robots, from which we derive its usage guidelines. We then conduct a user study ($N=12$) in which participants experience a walkthrough application using the actual system. The results demonstrate that the system is able to provide haptic feedback while redirecting the user within a limited play area.

We examine the effect of auditory cues occurring in reality on redirection. Specifically, we set two hypotheses: the auditory cues emanating from fixed positions in reality (Fixed sound, FS) increase the noticeability of redirection, while the auditory cues whose positions are manipulated consistently with the visual manipulation (Redirected sound, RDS) decrease the noticeability of redirection. To verify these hypotheses, we implemented an experimental environment that virtually reproduced FS and RDS conditions using binaural recording, and then we conducted a user study ( N=18) to investigate the detection thresholds (DTs) for rotational manipulation and the sound localization accuracy of the auditory cues under FS and RDS, as well as the baseline condition without auditory cues (No sound, NS). The results show, against the hypotheses, FS gave a wider range of DTs than NS, while RDS gave a similar range of DTs to NS. Combining these results with those of sound localization accuracy reveals that, rather than the auditory cues affecting the participants' spatial perception in VR, the visual manipulation made their sound localization less accurate, which would be a reason for the increased range of DTs under FS. Furthermore, we conducted a follow-up user study ( N=11) to measure the sound localization accuracy of FS where the auditory cues were actually placed in a real setting, and we found that the accuracy tended to be similar to that of virtually reproduced FS, suggesting the validity of the auditory cues used in this study. Given these findings, we also discuss potential applications.

We propose BirdViewAR, a surroundings-aware remote drone-operation system that provides significant spatial awareness to pilots through an augmented third-person view (TPV) from an autopiloted secondary follower drone. The follower drone responds to the main drone’s motions and directions using our optimization-based autopilot, allowing the pilots to clearly observe the main drone and its imminent destination without extra input. To improve their understanding of the spatial relationships between the main drone and its surroundings, the TPV is visually augmented with AR-overlay graphics, where the main drone’s spatial statuses are highlighted: its heading, altitude, ground position, camera field-of-view (FOV), and proximity areas. We discuss BirdViewAR’s design and implement its proof-of-concept prototype using programmable drones. Finally, we conduct a preliminary outdoor user study and find that BirdViewAR effectively increased spatial awareness and piloting performance.

Room-scale VR has been considered an alternative to physical office workspaces. For office activities, users frequently require planar input methods, such as typing or handwriting, to quickly record annotations to virtual content. However, current off-the-shelf VR HMD setups rely on mid-air interactions, which can cause arm fatigue and decrease input accuracy. To address this issue, we propose UbiSurface, a robotic touch surface that can automatically reposition itself to physically present a virtual planar input surface (VR whiteboard, VR canvas, etc.) to users and to permit them to achieve accurate and fatigue-less input while walking around a virtual room. We design and implement a prototype of UbiSurface that can dynamically change a canvas-sized touch surface's position, height, and pitch and yaw angles to adapt to virtual surfaces spatially arranged at various locations and angles around a virtual room. We then conduct studies to validate its technical performance and examine how UbiSurface facilitates the user's primary mid-air planar interactions, such as painting and writing in a room-scale VR setup. Our results indicate that this system reduces arm fatigue and increases input accuracy, especially for writing tasks. We then discuss the potential benefits and challenges of robotic touch devices for future room-scale VR setups.

We propose WaddleWalls, a room-scale interactive partitioning system using a swarm of robotic partitions that allows occupants to interactively reconfigure workspace partitions to satisfy their privacy and interaction needs. The system can automatically arrange the partitions’ layout designed by the user on demand. The user specifies the target partition’s position, orientation, and height using the controller’s 3D manipulations. In this work, we discuss the design considerations of the interactive partition system and implement WaddleWalls’ proof-of-concept prototype assembled with off-the-shelf materials. We demonstrate the functionalities of WaddleWalls through several application scenarios in an open-planned office environment. We also conduct an initial user evaluation that compares WaddleWalls with conventional wheeled partitions, finding that WaddleWalls allows effective workspace partitioning and mitigates the physical and temporal efforts needed to fulfill ad hoc social and privacy requirements. Finally, we clarify the feasibility, potential, and future challenges of WaddleWalls through an interview with experts.

We propose RedirectedDoors, a novel space-efficient technique for redirection in VR focused on door-opening behavior. This technique manipulates the user’s walking direction by rotating the entire virtual environment (VE) at a certain angular ratio of the door being opened. This ratio is called door rotation gain. At the same time, the virtual door’s position is kept unmanipulated so that a realistic door-opening user experience can be ensured. We designed and implemented the rotational manipulation algorithm and two types of door-opening interfaces; with and without a doorknob-type passive haptic prop. We then conducted a user study (N = 12) to investigate redirection performance and user feedback as we examined three independent variables: door rotation gain, door-opening interface, and door-opening direction (push/pull). From the results, the estimated detection thresholds generally showed a higher space efficiency of redirection with our technique. Our results also showed that providing the haptic feedback led to a higher noticeability of redirection, but at the same time supported a higher subjective sense of realism and less discomfort. Following our results, we discuss which combinations of gain and door-opening direction can jointly provide lower noticeability and higher acceptability.

Jumping onto steps is a promising action for creating an instant height movement in VR, but installing physical steps is impractical. We propose PseudoJumpOn, a novel locomotion technique using a common VR setup that allows the user to experience virtual step-up jumping motion by applying two types of viewpoint-manipulation methods to a physical jump on a flat floor. The core idea is to replicate the physical characteristics of ascending jumps, and thus we designed two viewpoint-manipulation methods: gain manipulation, which differentiates the ascent and descent height, and peak shifting, which delays the peak timing. We conducted a user study asking participants (N = 20) to experience two-legged step-up jumps onto 0.2–0.8-m heights in VR as the two methods were applied in combination (gain manipulation: five conditions where the ascending gain was 1.0–5.0; peak shifting: four conditions where the peak timing in VR was delayed by 0–1.0 ratios of the original timing). The results showed that the participants in most conditions felt positively in terms of reality and naturalness of actually jumping onto steps, even though knowing no physical steps existed. In addition, subsequent analyses also derived practical guidelines for determining the appropriate gains and the potential use of peak shifting to achieve a natural step-up jumping experience.

We propose HandyGaze, a 6-DoF gaze tracking technique for room-scale environments that can be carried out by simply holding a smartphone naturally without installing any sensors or markers in the environment. Our technique simultaneously employs the smartphone’s front and rear cameras: The front camera estimates the user’s gaze vector relative to the smartphone, while the rear camera (and depth sensor, if available) performs self-localization by reconstructing a pre-obtained 3D map of the environment. To achieve this, we implemented a prototype that works on iOS smartphones by running an ARKit-based algorithm for estimating the user’s 6-DoF head orientation. We additionally implemented a novel calibration method that offsets the user-specific deviation between the head and gaze orientations. We then conducted a user study (N=10) that measured our technique’s positional accuracy to the gaze target under four conditions, based on combinations of use with and without a depth sensor and calibration. The results show that our calibration method was able to reduce the mean absolute error of the gaze point by 27%, with an error of 0.53 m when using the depth sensor. We also report the target size required to avoid erroneous inputs. Finally, we suggest possible applications such as a gaze-based guidance application for museums.

We propose a novel phone-case-shaped interface named TetraForce, which enables four types of force input consisting of two force directions (i.e., pressure force and shear force) and two force-applied surfaces (i.e., touch surface and back surface) in a single device. Force detection is achieved using the smartphone's built-in magnetometer (and supplementary accelerometer and gyro sensor) by estimating the displacement of a magnet attached to a 3-DoF passively movable panel at the back. We conducted a user study (N=12) to investigate the fundamental user performance by our interface and demonstrated that the input was detected as intended with a success rate of 97.4% on average for all four input types. We further conducted an ideation workshop with people who were involved in human-computer interaction (N=12) to explore possible applications of this interface, and we obtained 137 ideas for applications using individual input types and 51 possible scenarios using them in combination. Organizing these ideas reveals the advantages of each input type and suggests that our interface is useful for applications that require complex operations and that it can improve intuitiveness through elastic feedback.

Accurate drone positioning is challenging because pilots only have a limited position and direction perception of a flying drone from their perspective. This makes conventional joystick-based speed control inaccurate and more complicated and significantly degrades piloting performance. We propose PinpointFly, an egocentric drone interface that allows pilots to arbitrarily position and rotate a drone using position-control direct interactions on a see-through mobile AR where the drone position and direction are visualized with a virtual cast shadow (i.e., the drone’s orthogonal projection onto the floor). Pilots can point to the next position or draw the drone’s flight trajectory by manipulating the virtual cast shadow and the direction/height slider bar on the touchscreen. We design and implement a prototype of PinpointFly for indoor and visual line of sight scenarios, which are comprised of real-time and predefined motion-control techniques. We conduct two user studies with simple positioning and inspection tasks. Our results demonstrate that PinpointFly makes the drone positioning and inspection operations faster, more accurate, simpler and fewer workload than a conventional joystick interface with a speed-control method.

We propose TiltChair, an actuated office chair that physically manipulates the user’s posture by actively inclining the chair’s seat to address problems associated with prolonged sitting. The system controls the inclination angle and motion speed with the aim of achieving manipulative but unobtrusive posture guidance. To demonstrate its potential, we first built a prototype of TiltChair with a seat that could be tilted by pneumatic control. We then investigated the effects of the seat’s inclination angle and motions on task performance and overall sitting experience through two experiments. The results show that the inclination angle mainly affects the difficulty of maintaining one’s posture, while the motion speed affected the conspicuousness and subjective acceptability of the motion. However, these seating conditions did not affect objective task performance. Based on these results, we propose a design space for facilitating effective seat-inclination behavior using the three dimensions of angle, speed, and continuity. Furthermore, we discuss promising applications.

We propose ModularHMD, a new mobile head-mounted display concept, which adopts a modular mechanism and allows a user to perform ad-hoc peripheral interaction with real-world devices or people during VR experiences. ModularHMD is comprised of a central HMD and three removable module devices installed in the periphery of the HMD cowl. Each module has four main states: occluding, extended VR view, video see-through (VST), and removed/reused. Among different combinations of module states, a user can quickly setup the necessary HMD forms, functions, and real-world visions for ad-hoc peripheral interactions without removing the headset. For instance, an HMD user can see her surroundings by switching a module into the VST mode. She can also physically remove a module to obtain direct peripheral visions of the real world. The removed module can be reused as an instant interaction device (e.g., touch keyboards) for subsequent peripheral interactions. Users can end the peripheral interaction and revert to a full VR experience by re-mounting the module. We design ModularHMD’s configuration and peripheral interactions with real-world objects and people. We also implement a proof-of-concept prototype of ModularHMD to validate its interactions capabilities through a user study. Results show that ModularHMD is an effective solution that enables both immersive VR and ad-hoc peripheral interactions.

We explore BouncyScreen, an actuated 1D display system that enriches indirect interaction with a virtual object by pseudo-haptic feedback mechanics enhanced through the screen's physical movements. We configured a proof-of-concept prototype of BouncyScreen with a flat-screen mounted on a mobile robot. When the user manipulates a virtual object using virtual reality (VR) controllers, the screen moves in accordance with the virtual object. We conducted psychophysical studies to examine how BouncyScreen's physical movements would affect users' pseudo-haptic perceptions and interaction experiences. Our preliminary study using a weight discrimination task for object pushing interaction showed that BouncyScreen offers identical pseudo-force feedback to the vision-based pseudo-haptic technique. We conducted a follow-up psychophysical study using a weight magnitude estimation task for object pushing and bumping interactions. The results reveal that a users' perceived weight magnitude is enhanced by the screen's physical motion under different characteristics depending on interaction styles (i.e., pushing and bumping). Our study also confirmed that the screen's synchronous physical motions significantly enhance the reality of the interaction and the sense of presence. We close this paper with some applications and use suggestions for BouncyScreen in future HMD-free flat-screen 3D user interface systems.

Proximity interaction, such as hugging, plays an essential role in building relationships between parents and children. However, parents and children cannot freely interact in the neonatal intensive care unit due to visiting restrictions imposed by COVID-19. In this study, we develop a system of pseudo-proximity interaction with a remote infant through a VR headset by using an infant-like device that reproduces the haptic feedback features of the hugging sensation, such as weight, body temperature, breathing, softness, and unstable neck.

Drilling teeth is a significant technique for dental learners. However, the existing VR tooth drill training simulators are physically large and costly, which cannot be used at home or classroom for a private study. This work presents a novel low-cost mobile VR dental simulator using off-the-shelf devices, including a mobile HMD and a stylus smartphone. In this system, a 3D-printed physical teeth prop is placed on an EMR stylus smartphone where its stylus tracks the tip position of a physical drill. Unlike existing solutions using haptic/force devices, our approach involving physical contact between the prop and drill tip enables the user’s natural teeth hardness sensation. The use of smartphone stylus would enable significantly more accurate drill position sensing around the teeth than HMD’s accompanying controllers. We also developed VR software to simulate tooth drilling on this setup. This demo will show how our new mobile simulator offers a realistic feeling of drilling teeth.

Head-mounted displays (HMDs) increase immersion into virtual worlds. The problem is that this limits headset users' awareness of bystanders: headset users cannot attend to bystanders' presence and activities. We call this the HMD boundary. We explore how to make the HMD boundary permeable by comparing different ways of providing informal awareness cues to the headset user about bystanders. We adapted and implemented three visualization techniques (Avatar View, Radar and Presence++) that share bystanders' location and orientation with headset users. We conducted a hybrid user and simulation study with three different types of VR content (high, medium, low interactivity) with twenty participants to compare how these visualization techniques allow people to maintain an awareness of bystanders, and how they affect immersion (compared to a baseline condition). Our study reveals that a see-through avatar representation of bystanders was effective, but led to slightly reduced immersion in the VR content. Based on our findings, we discuss how future awareness visualization techniques can be designed to mitigate the reduction of immersion for the headset user.

We propose TuVe, a novel shape-changing display consisting of a flexible tube and fluids, in which the droplets flowing through the tube compose the display medium that represents information. In this system, every colored droplet is flowed by controlling valves and a pump connected to the tube. The display part employs a flexible tube that can be shaped to any structure (e.g., wrapped around a specific object), which is achieved by a calibration made to capture the tube structure using image processing with a camera. A performance evaluation reveals that our prototype succeeds in controlling each droplet with a positional error of 2 mm or less, which is small enough to show such simple characters as alphabetic characters using a 7 × 7-pixel resolution display. We also discuss example applications, such as large public displays and flow-direction visualization, that illustrate the characteristics of the TuVe display.

We focus on the problem of simulating the haptic infrastructure of a virtual environment (i.e. walls, doors). Our approach relies on multiple ZoomWalls---autonomous robotic encounter-type haptic wall-shaped props---that coordinate to provide haptic feedback for room-scale virtual reality. Based on a user's movement through the physical space, ZoomWall props are coordinated through a predict-and-dispatch architecture to provide just-in-time haptic feedback for objects the user is about to touch. To refine our system, we conducted simulation studies of different prediction algorithms, which helped us to refine our algorithmic approach to realize the physical ZoomWall prototype. Finally, we evaluated our system through a user experience study, which showed that participants found that ZoomWalls increased their sense of presence in the VR environment. ZoomWalls represents an instance of autonomous mobile reusable props, which we view as an important design direction for haptics in VR.

We propose ShearSheet, a low-cost, feasible, and simple DIY method that enables tangential (shear) force input on a touchscreen using a rubber-mounted slim transparent sheet. The sheet has tiny conductive material(s) attached to specific positions on the underside so that displacement of the sheet is recognized as touch input(s). This allows the system to distinguish between normal touch input and shear force input depending on whether the sheet moves or not, without requiring any external sensors or power. We first introduce ShearSheet's simple implementation for a smartphone and then discuss several promising interaction techniques that augment shear interaction with one- and two-finger operations. We demonstrate the effectiveness of ShearSheet's smooth transition between position- and rate-based control through a controlled user study using simple scrolling tasks. The results confirm that ShearSheet offers improved performance such as fewer operations and higher subjective preference, suggesting its substantial benefits and potential applications.

We propose Third-Person Piloting, a novel drone manipulation interface that increases situational awareness using an interactive third-person perspective from a second, spatially coupled drone. The pilot uses a controller with a manipulatable miniature drone. Our algorithm understands the relationship between the pilot's eye position and the miniature drone and ensures that the same spatial relationship is maintained between the two real drones in the sky. This allows the pilot to obtain various third-person perspectives by changing the orientation of the miniature drone while maintaining standard primary drone control using the conventional controller. We design and implement a working prototype with programmable drones and propose several representative operation scenarios. We gather user feedback to obtain the initial insights of our interface design from novices, advanced beginners, and experts. Our result suggests that the interactive third-person perspective provided by the second drone offers sufficient potential for increasing situational awareness and supporting their primary drone operations.

Jumping is a fundamental movement in our daily lives that is often used in many video games. However, little research has been done on jumping and its possible use as a redirection technique in virtual reality (VR). In this study we explore Redirected Jumping, a novel redirection technique which enables us to purposefully manipulate the mapping of the user's physical jumping movements (e.g., distance and direction) to movement in the virtual space, allowing richer and more active physical VR experiences within a limited tracking area. To demonstrate the possibilities afforded by Redirected Jumping, we implemented a jump detection algorithm and jumping redirection methods for three basic jumping actions (i.e., horizontal, vertical, and rotational jumps) using common VR devices. We conducted three user studies to investigate the effective manipulation ranges, and the results revealed that our methods can manipulate a user's jumping movements without his/her noticing, similar to walking.

Digital photos and their metadata get explosively growth, requiring better support to browsing them. We propose a dynamic and flexible visualization of digital photos and their metadata. A prototype is designed and implemented based on the algorithm of D-FLIP, our previous work for flexibly displaying a large photo collection. We design various dynamic photo visualizations with up to four-dimensional meta-information by using Bertin's visual variables and three-dimensional representation. It allows users to dynamically and effectively manage photo visualizations by selecting meta-information of user's current needs and interest.

We are pursuing a vision of reactive interior spaces that are aware of people's actions and transform according to changing needs. We envision furniture and walls that act as interactive displays and that shapeshift to the correct physical form, and the appropriate interactive visual content and modality. This paper briefly describes our proposal based on our recent efforts on realizing this vision.

We propose a portable umbrella-shaped device called U-brella as an application of “vibrationizing” information. Vibrationizing is a term coined to mean the transformation of various types of information into a vibration. Funbrella is an earlier version of U-brella that allows users to experience rain with vibrations, while U-brella offers them the ability to feel ultraviolet rays by converting them into vibration. We first conceptualized the implementation of U-brella: improving the mechanism of Funbrella's vibration generator to make it lighter and easier to carry. Then we conducted a user study to confirm the efficiency of our system's vibrationizing and users' impressions of U-brella. The results showed that the participants easily perceived the amount of ultraviolet rays and that they really felt the rays pour down upon their heads with U-brella.

This paper describes Ambient Party Room, which is a room-shaped information environment for party situations. This system estimates the conversation states by measuring participants' nonverbal cues and displays various types of information to stimulate the conversation. In this study, we reconstructed the system architecture from an early prototype and newly examined several types of information stimulus using a floor display, wall displays, and speakers. We conducted an experiment in which six participants talked freely in Ambient Party Room. Results showed that both profile-based information stimulus and feedback information of participants' behaviors contributed to their activated conversations.

People often communicate with others using social touch interactions including hugging, rubbing, and punching. We propose a soft social-touchable interface called “Emoballoon” that can recognize the types of social touch interactions. The proposed interface consists of a balloon and some sensors including a barometric pressure sensor inside of a balloon, and has a soft surface and ability to detect the force of the touch input. We construct the prototype of Emoballoon using a simple configuration based on the features of a balloon, and evaluate the implemented prototype. The evaluation indicates that our implementation can distinguish seven types of touch interactions with 83.5% accuracy.

In this paper, we investigate the effectiveness of the records of user's activities in Twitter, which is a popular microblogging site, for estimating his/her depressive tendency. We construct multiple regression model to estimate user's depressive tendency from the frequencies of words used by the user. We perform experiments to estimate participants' depressive tendencies using the constructed regression model. Our experimental results show that there exists medium positive correlation (correlation coefficient r ≃ 0.45) between the Zung's Self-rating Depression Scale, which is a popular measure for estimating depressive tendency, and estimated score obtained from the regression model.

We propose a room-shaped information environment called Ambient Suite that enhances interpersonal communication. In Ambient Suite, the room itself works as both sensors to estimate the conversation states of participants and displays to present information to stimulate conversation. This paper introduces an implementation assumed standing-party situations as a typical use case of Ambient Suite. From the result of user study using its implementation, we confirmed that our system adequately encouraged participant conversations.

We propose a cup-shaped device called Cup-le for conversational experiment. Cup-le has various sensors to record participants' nonverbal behavior such as utterance. Cup-le saves the work of attaching sensors. Also, since Cup-le appears cup-shaped, it can sense without being aware of the sensor unlike conventional wearable sensors. Moreover, by attaching touch display, Cup-le can show information and can be operated. In this conversational experiment, we use the input-output as method to have a questionnaire during the experiment and capture participants' impression for their conversation. We conducted real conversational experiment with Cup-le, and we confirmed that Cup-le supported sufficiently the experiment without affecting their impression.

We propose a furry and scalable multi-touch display called the “FuSA2Touch Display.” The furry type of tactile sensation of this surface affords various interactions such as stroking or clawing. The system utilizes plastic fiber optic bundles to realize a furry-type texture. The system can show visual feedback by projection and detects multi-touch input. We implemented a 24-inch display, and found that our implemented display encourages users to interact with it in various actions.