Drawing, writing, and painting on 2D surfaces in VR offers arbitrary interaction with virtual canvases of unlimited size. However, this advantage has not been fully exploited because physical surfaces are needed to support accurate and comfortable user interactions, and the size of such surfaces is limited. We propose Redirected Drawing, a novel methodology that applies visual manipulation to the user’s drawing movement in VR to expand the perceived surface size required for the drawing experience. To this end, this study specifically explores translation gain manipulation, where the displacement of a drawing stroke in reality is simply multiplied by a gain to represent it in VR. We conducted a user study (N = 18) to measure the noticeability, acceptability, and task performance of a simple VR line-tracing task under different gain and input methods (i.e., finger in mid-air, finger on surface, pen on surface). Our results show that pen-based drawing on a physical surface allows for a gain of up to 1.16 without being noticed by the user, which is more suited to expanding a virtual canvas size than finger-based drawing on a physical surface or in mid-air. Furthermore, we show that user acceptability of the gain manipulation could be estimated by a model using the subjective discrepancy caused by the gain. Results for this model suggest that users could accept gains of up to 1.30. Based on our results, we derive guidelines for the effective use of gain manipulation.

Dental cutting is a crucial skill for dental students. However, current VR dental cutting training systems rely on bulky and costly haptic devices, which reduce opportunities for individual practice. Moreover, the limitations imposed by the maximum reaction force of an active haptic device would impact the range of tooth hardness that can be reproduced. We propose a compact XR tooth-cutting training system, VirtuEleDent, that employs a passive haptic approach using a 3D-printed physical teeth model and a three-dimensionally tracked handpiece. Their spatial relationship is accurately rendered in the virtual environment of a mobile head-mounted display (HMD), providing users with realistic haptic sensations during virtual tooth-cutting exercises. Our tracking platform is operated using electromagnetic resonance (EMR) stylus technology and consists of a digitizer (i.e., tracking board) and a handpiece device. A customized EMR stylus unit (i.e., resonance coil) and an inertial measurement unit (IMU) sensor are installed inside the handpiece, allowing for precise measurement of its tip’s 3D position and orientation. This setup enables the learner to physically manipulate dexterous handpieces on the teeth model while experiencing virtual tooth-cutting in the HMD. First, we detail the design and implementation of our initial prototype system, including the tracking platform. We then describe an expert user study conducted with our prototype, where licensed dentists compare VirtuEleDent to a baseline with a conventional haptic device. Finally, the educator’s feedback demonstrates its potential and informs our further improvements and the directions of future development.

Building higher-quality image classification models requires better performance analysis (PA) to help understand their behaviors. We propose ConfusionLens, a dynamic and interactive visualization interface that augments a conventional confusion matrix with focus+context visualization. This interface allows users to seamlessly switch table layouts among three views (overall view, class-level view, and between-class view) while observing all of the instance images in a single screen. We designed and implemented a ConfusionLens prototype that supports hundreds of instances, and then conducted a user study (N \(=\) 14) to evaluate it compared to the conventional confusion matrix with a split view of instances. Results show that ConfusionLens achieved faster task-completion time in observing instance-level performance and higher accuracy in observing between-class confusion. Moreover, we conducted an expert interview (N \(=\) 6) to investigate the applicability of our interface to practical PA tasks, and then implemented several extensions of ConfusionLens based on the feedback. Feedback on these extensions from users experienced in image classification (N \(=\) 5) demonstrated their general usefulness and highlighted their beneficial use in PA tasks.

We introduce InflatableBots, shape-changing inflatable robots for large-scale encountered-type haptics in VR. Unlike traditional inflatable shape displays, which are immobile and limited in interaction areas, our approach combines mobile robots with fan-based inflatable structures. This enables safe, scalable, and deployable haptic interactions on a large scale. We developed three coordinated inflatable mobile robots, each of which consists of an omni-directional mobile base and a reel-based inflatable structure. The robot can simultaneously change its height and position rapidly (horizontal: 58.5 cm/sec, vertical: 10.4 cm/sec, from 40 cm to 200 cm), which allows for quick and dynamic haptic rendering of multiple touch points to simulate various body-scale objects and surfaces in real-time across large spaces (3.5 m x 2.5 m). We evaluated our system with a user study (N = 12), which confirms the unique advantages in safety, deployability, and large-scale interactability to significantly improve realism in VR experiences.

Videos accompanied by documents—document-based videos—enable presenters to share contents beyond videos and audience to use them for detailed content comprehension. However, concurrently exploring multiple channels of information could be taxing. We propose SwapVid, a novel interface for viewing and exploring document-based videos. SwapVid seamlessly integrates a video and a document into a single view and lets the content behaves as both video and a document; it adaptively switches a document-based video to act as a video or a document upon direct manipulation (e.g., scrolling the document, manipulating the video timeline). We conducted a user study with twenty participants, comparing SwapVid to a side-by-side video/document views. Results showed that our interface reduces time and physical workload when exploring slide-based documents based on video referencing. Based on the study findings, we extended SwapVid with additional functionalities and demonstrated that it further extends the practical capabilities.

As CDNs facing DDoS attacks targeting some content every day, CDNI, which is designed to allow multiple CDNs to cooperate each other to distribute content more broadly, will also be targeted if CDNI distributes such content, but for small CDNs owned by ISPs or NSPs in CDNI, it is hard to mitigate large DDoS attacks continuously because of financial limitations. Considering that each CDN in CDNI is operated autonomously in terms of system, security, etc., countermeasures against DDoS attacks should be implemented autonomously in each CDN, so that one of possible countermeasures against DDoS attacks in CDNI is to share information among CDNs to cope with DDoS attacks cooperatively. However, there is no previous research about what an architecture are required or what information should be shared through them to handle DDoS attacks. In this study, we propose a system that handles DDoS attacks through information exchange with leveraging capabilities of CDNI, and we show that, through several use cases of this system, information for requesting CDNI operations and information representing the attacks are essential. Especially, in order for the CDNI to effectively respond to attacks and gain the cooperation of more CDNs, two key types of information are needed: the volume of the attack and information that helps mitigate the attack. Finally, we briefly show an example implementation approach of the architecture and a format for sharing this information by using DOTS.

In room-scale virtual reality, providing continuous haptic feedback from touching grounded objects, such as walls and handrails, has been challenging due to the user’s walking range and the required force. In this study, we propose LoopBot, a novel technique to provide continuous haptic feedback from grounded objects using only a single user-following robot. Specifically, LoopBot is equipped with a loop-shaped haptic prop attached to an omnidirectional robot that scrolls to cancel out the robot’s displacement, giving the user the haptic sensation that the prop is actually fixed in place, or “grounded.” We first introduce the interaction design space of LoopBot and, as one of its promising interaction scenarios, implement a prototype for the experience of walking while grasping handrails. A performance evaluation shows that scrolling the prop cancels 77.5% of the robot’s running speed on average. A preliminary user test (N = 10) also shows that the subjective realism of the experience and the sense of the virtual handrails being grounded were significantly higher than when the prop was not scrolled. Based on these findings, we discuss possible further development of LoopBot.

RedirectedDoors is a visuo-haptic door-opening redirection technique in VR, and it has shown promise in its ability to efficiently compress the physical space required for a room-scale VR experience. However, its previous implementation has only supported laboratory experiments with a single door opening at a fixed location. To significantly expand this technique for room-scale VR, we have developed RedirectedDoors+, a robot-based system that permits consecutive door-opening redirection with haptics. Specifically, our system is mainly achieved with the use of three components: (1) door robots, a small number of wheeled robots equipped with a doorknob-like prop, (2) a robot-positioning algorithm that arbitrarily positions the door robots to provide the user with just-in-time haptic feedback during door opening, and (3) a user-steering algorithm that determines the redirection gain for every instance of door opening to keep the user away from the boundary of the play area. Results of simulated VR exploration in six virtual environments reveal our system's performance relative to user walking speed, paths, and number of door robots, from which we derive its usage guidelines. We then conduct a user study ($N=12$) in which participants experience a walkthrough application using the actual system. The results demonstrate that the system is able to provide haptic feedback while redirecting the user within a limited play area.

We examine the effect of auditory cues occurring in reality on redirection. Specifically, we set two hypotheses: the auditory cues emanating from fixed positions in reality (Fixed sound, FS) increase the noticeability of redirection, while the auditory cues whose positions are manipulated consistently with the visual manipulation (Redirected sound, RDS) decrease the noticeability of redirection. To verify these hypotheses, we implemented an experimental environment that virtually reproduced FS and RDS conditions using binaural recording, and then we conducted a user study ( N=18) to investigate the detection thresholds (DTs) for rotational manipulation and the sound localization accuracy of the auditory cues under FS and RDS, as well as the baseline condition without auditory cues (No sound, NS). The results show, against the hypotheses, FS gave a wider range of DTs than NS, while RDS gave a similar range of DTs to NS. Combining these results with those of sound localization accuracy reveals that, rather than the auditory cues affecting the participants' spatial perception in VR, the visual manipulation made their sound localization less accurate, which would be a reason for the increased range of DTs under FS. Furthermore, we conducted a follow-up user study ( N=11) to measure the sound localization accuracy of FS where the auditory cues were actually placed in a real setting, and we found that the accuracy tended to be similar to that of virtually reproduced FS, suggesting the validity of the auditory cues used in this study. Given these findings, we also discuss potential applications.

We propose BirdViewAR, a surroundings-aware remote drone-operation system that provides significant spatial awareness to pilots through an augmented third-person view (TPV) from an autopiloted secondary follower drone. The follower drone responds to the main drone’s motions and directions using our optimization-based autopilot, allowing the pilots to clearly observe the main drone and its imminent destination without extra input. To improve their understanding of the spatial relationships between the main drone and its surroundings, the TPV is visually augmented with AR-overlay graphics, where the main drone’s spatial statuses are highlighted: its heading, altitude, ground position, camera field-of-view (FOV), and proximity areas. We discuss BirdViewAR’s design and implement its proof-of-concept prototype using programmable drones. Finally, we conduct a preliminary outdoor user study and find that BirdViewAR effectively increased spatial awareness and piloting performance.

Room-scale VR has been considered an alternative to physical office workspaces. For office activities, users frequently require planar input methods, such as typing or handwriting, to quickly record annotations to virtual content. However, current off-the-shelf VR HMD setups rely on mid-air interactions, which can cause arm fatigue and decrease input accuracy. To address this issue, we propose UbiSurface, a robotic touch surface that can automatically reposition itself to physically present a virtual planar input surface (VR whiteboard, VR canvas, etc.) to users and to permit them to achieve accurate and fatigue-less input while walking around a virtual room. We design and implement a prototype of UbiSurface that can dynamically change a canvas-sized touch surface's position, height, and pitch and yaw angles to adapt to virtual surfaces spatially arranged at various locations and angles around a virtual room. We then conduct studies to validate its technical performance and examine how UbiSurface facilitates the user's primary mid-air planar interactions, such as painting and writing in a room-scale VR setup. Our results indicate that this system reduces arm fatigue and increases input accuracy, especially for writing tasks. We then discuss the potential benefits and challenges of robotic touch devices for future room-scale VR setups.

We propose WaddleWalls, a room-scale interactive partitioning system using a swarm of robotic partitions that allows occupants to interactively reconfigure workspace partitions to satisfy their privacy and interaction needs. The system can automatically arrange the partitions’ layout designed by the user on demand. The user specifies the target partition’s position, orientation, and height using the controller’s 3D manipulations. In this work, we discuss the design considerations of the interactive partition system and implement WaddleWalls’ proof-of-concept prototype assembled with off-the-shelf materials. We demonstrate the functionalities of WaddleWalls through several application scenarios in an open-planned office environment. We also conduct an initial user evaluation that compares WaddleWalls with conventional wheeled partitions, finding that WaddleWalls allows effective workspace partitioning and mitigates the physical and temporal efforts needed to fulfill ad hoc social and privacy requirements. Finally, we clarify the feasibility, potential, and future challenges of WaddleWalls through an interview with experts.

We propose RedirectedDoors, a novel space-efficient technique for redirection in VR focused on door-opening behavior. This technique manipulates the user’s walking direction by rotating the entire virtual environment (VE) at a certain angular ratio of the door being opened. This ratio is called door rotation gain. At the same time, the virtual door’s position is kept unmanipulated so that a realistic door-opening user experience can be ensured. We designed and implemented the rotational manipulation algorithm and two types of door-opening interfaces; with and without a doorknob-type passive haptic prop. We then conducted a user study (N = 12) to investigate redirection performance and user feedback as we examined three independent variables: door rotation gain, door-opening interface, and door-opening direction (push/pull). From the results, the estimated detection thresholds generally showed a higher space efficiency of redirection with our technique. Our results also showed that providing the haptic feedback led to a higher noticeability of redirection, but at the same time supported a higher subjective sense of realism and less discomfort. Following our results, we discuss which combinations of gain and door-opening direction can jointly provide lower noticeability and higher acceptability.

Jumping onto steps is a promising action for creating an instant height movement in VR, but installing physical steps is impractical. We propose PseudoJumpOn, a novel locomotion technique using a common VR setup that allows the user to experience virtual step-up jumping motion by applying two types of viewpoint-manipulation methods to a physical jump on a flat floor. The core idea is to replicate the physical characteristics of ascending jumps, and thus we designed two viewpoint-manipulation methods: gain manipulation, which differentiates the ascent and descent height, and peak shifting, which delays the peak timing. We conducted a user study asking participants (N = 20) to experience two-legged step-up jumps onto 0.2–0.8-m heights in VR as the two methods were applied in combination (gain manipulation: five conditions where the ascending gain was 1.0–5.0; peak shifting: four conditions where the peak timing in VR was delayed by 0–1.0 ratios of the original timing). The results showed that the participants in most conditions felt positively in terms of reality and naturalness of actually jumping onto steps, even though knowing no physical steps existed. In addition, subsequent analyses also derived practical guidelines for determining the appropriate gains and the potential use of peak shifting to achieve a natural step-up jumping experience.

We propose HandyGaze, a 6-DoF gaze tracking technique for room-scale environments that can be carried out by simply holding a smartphone naturally without installing any sensors or markers in the environment. Our technique simultaneously employs the smartphone’s front and rear cameras: The front camera estimates the user’s gaze vector relative to the smartphone, while the rear camera (and depth sensor, if available) performs self-localization by reconstructing a pre-obtained 3D map of the environment. To achieve this, we implemented a prototype that works on iOS smartphones by running an ARKit-based algorithm for estimating the user’s 6-DoF head orientation. We additionally implemented a novel calibration method that offsets the user-specific deviation between the head and gaze orientations. We then conducted a user study (N=10) that measured our technique’s positional accuracy to the gaze target under four conditions, based on combinations of use with and without a depth sensor and calibration. The results show that our calibration method was able to reduce the mean absolute error of the gaze point by 27%, with an error of 0.53 m when using the depth sensor. We also report the target size required to avoid erroneous inputs. Finally, we suggest possible applications such as a gaze-based guidance application for museums.

We propose a novel phone-case-shaped interface named TetraForce, which enables four types of force input consisting of two force directions (i.e., pressure force and shear force) and two force-applied surfaces (i.e., touch surface and back surface) in a single device. Force detection is achieved using the smartphone's built-in magnetometer (and supplementary accelerometer and gyro sensor) by estimating the displacement of a magnet attached to a 3-DoF passively movable panel at the back. We conducted a user study (N=12) to investigate the fundamental user performance by our interface and demonstrated that the input was detected as intended with a success rate of 97.4% on average for all four input types. We further conducted an ideation workshop with people who were involved in human-computer interaction (N=12) to explore possible applications of this interface, and we obtained 137 ideas for applications using individual input types and 51 possible scenarios using them in combination. Organizing these ideas reveals the advantages of each input type and suggests that our interface is useful for applications that require complex operations and that it can improve intuitiveness through elastic feedback.

Accurate drone positioning is challenging because pilots only have a limited position and direction perception of a flying drone from their perspective. This makes conventional joystick-based speed control inaccurate and more complicated and significantly degrades piloting performance. We propose PinpointFly, an egocentric drone interface that allows pilots to arbitrarily position and rotate a drone using position-control direct interactions on a see-through mobile AR where the drone position and direction are visualized with a virtual cast shadow (i.e., the drone’s orthogonal projection onto the floor). Pilots can point to the next position or draw the drone’s flight trajectory by manipulating the virtual cast shadow and the direction/height slider bar on the touchscreen. We design and implement a prototype of PinpointFly for indoor and visual line of sight scenarios, which are comprised of real-time and predefined motion-control techniques. We conduct two user studies with simple positioning and inspection tasks. Our results demonstrate that PinpointFly makes the drone positioning and inspection operations faster, more accurate, simpler and fewer workload than a conventional joystick interface with a speed-control method.

We propose TiltChair, an actuated office chair that physically manipulates the user’s posture by actively inclining the chair’s seat to address problems associated with prolonged sitting. The system controls the inclination angle and motion speed with the aim of achieving manipulative but unobtrusive posture guidance. To demonstrate its potential, we first built a prototype of TiltChair with a seat that could be tilted by pneumatic control. We then investigated the effects of the seat’s inclination angle and motions on task performance and overall sitting experience through two experiments. The results show that the inclination angle mainly affects the difficulty of maintaining one’s posture, while the motion speed affected the conspicuousness and subjective acceptability of the motion. However, these seating conditions did not affect objective task performance. Based on these results, we propose a design space for facilitating effective seat-inclination behavior using the three dimensions of angle, speed, and continuity. Furthermore, we discuss promising applications.

Although children’s behavioral and mental problems are generally diagnosed in clinical settings, the prediction and awareness of children’s mental wellness in daily settings are getting increased attention. Toy blocks are both accessible in most children’s daily lives and provide physicality as a unique non-verbal channel to express their inner world. In this paper, we propose a toy block approach for predicting a range of behavior problems in young children (4-6 years old) measured by the Child Behavior Checklist (CBCL). We defined and classified a set of quantitative play actions from IMU-embedded toy blocks. Play data collected from 78 preschoolers revealed that specific play actions and patterns indicate total problems, internalizing problems, and aggressive behavior in children. The results align with our qualitative observations, and suggest the potential of predicting the clinical behavior problems of children based on short free-play sessions with sensor-embedded toy blocks.

We propose ModularHMD, a new mobile head-mounted display concept, which adopts a modular mechanism and allows a user to perform ad-hoc peripheral interaction with real-world devices or people during VR experiences. ModularHMD is comprised of a central HMD and three removable module devices installed in the periphery of the HMD cowl. Each module has four main states: occluding, extended VR view, video see-through (VST), and removed/reused. Among different combinations of module states, a user can quickly setup the necessary HMD forms, functions, and real-world visions for ad-hoc peripheral interactions without removing the headset. For instance, an HMD user can see her surroundings by switching a module into the VST mode. She can also physically remove a module to obtain direct peripheral visions of the real world. The removed module can be reused as an instant interaction device (e.g., touch keyboards) for subsequent peripheral interactions. Users can end the peripheral interaction and revert to a full VR experience by re-mounting the module. We design ModularHMD’s configuration and peripheral interactions with real-world objects and people. We also implement a proof-of-concept prototype of ModularHMD to validate its interactions capabilities through a user study. Results show that ModularHMD is an effective solution that enables both immersive VR and ad-hoc peripheral interactions.

We explore BouncyScreen, an actuated 1D display system that enriches indirect interaction with a virtual object by pseudo-haptic feedback mechanics enhanced through the screen's physical movements. We configured a proof-of-concept prototype of BouncyScreen with a flat-screen mounted on a mobile robot. When the user manipulates a virtual object using virtual reality (VR) controllers, the screen moves in accordance with the virtual object. We conducted psychophysical studies to examine how BouncyScreen's physical movements would affect users' pseudo-haptic perceptions and interaction experiences. Our preliminary study using a weight discrimination task for object pushing interaction showed that BouncyScreen offers identical pseudo-force feedback to the vision-based pseudo-haptic technique. We conducted a follow-up psychophysical study using a weight magnitude estimation task for object pushing and bumping interactions. The results reveal that a users' perceived weight magnitude is enhanced by the screen's physical motion under different characteristics depending on interaction styles (i.e., pushing and bumping). Our study also confirmed that the screen's synchronous physical motions significantly enhance the reality of the interaction and the sense of presence. We close this paper with some applications and use suggestions for BouncyScreen in future HMD-free flat-screen 3D user interface systems.

Drilling teeth is a significant technique for dental learners. However, the existing VR tooth drill training simulators are physically large and costly, which cannot be used at home or classroom for a private study. This work presents a novel low-cost mobile VR dental simulator using off-the-shelf devices, including a mobile HMD and a stylus smartphone. In this system, a 3D-printed physical teeth prop is placed on an EMR stylus smartphone where its stylus tracks the tip position of a physical drill. Unlike existing solutions using haptic/force devices, our approach involving physical contact between the prop and drill tip enables the user’s natural teeth hardness sensation. The use of smartphone stylus would enable significantly more accurate drill position sensing around the teeth than HMD’s accompanying controllers. We also developed VR software to simulate tooth drilling on this setup. This demo will show how our new mobile simulator offers a realistic feeling of drilling teeth.

Head-mounted displays (HMDs) increase immersion into virtual worlds. The problem is that this limits headset users' awareness of bystanders: headset users cannot attend to bystanders' presence and activities. We call this the HMD boundary. We explore how to make the HMD boundary permeable by comparing different ways of providing informal awareness cues to the headset user about bystanders. We adapted and implemented three visualization techniques (Avatar View, Radar and Presence++) that share bystanders' location and orientation with headset users. We conducted a hybrid user and simulation study with three different types of VR content (high, medium, low interactivity) with twenty participants to compare how these visualization techniques allow people to maintain an awareness of bystanders, and how they affect immersion (compared to a baseline condition). Our study reveals that a see-through avatar representation of bystanders was effective, but led to slightly reduced immersion in the VR content. Based on our findings, we discuss how future awareness visualization techniques can be designed to mitigate the reduction of immersion for the headset user.

We focus on the problem of simulating the haptic infrastructure of a virtual environment (i.e. walls, doors). Our approach relies on multiple ZoomWalls---autonomous robotic encounter-type haptic wall-shaped props---that coordinate to provide haptic feedback for room-scale virtual reality. Based on a user's movement through the physical space, ZoomWall props are coordinated through a predict-and-dispatch architecture to provide just-in-time haptic feedback for objects the user is about to touch. To refine our system, we conducted simulation studies of different prediction algorithms, which helped us to refine our algorithmic approach to realize the physical ZoomWall prototype. Finally, we evaluated our system through a user experience study, which showed that participants found that ZoomWalls increased their sense of presence in the VR environment. ZoomWalls represents an instance of autonomous mobile reusable props, which we view as an important design direction for haptics in VR.

We propose ShearSheet, a low-cost, feasible, and simple DIY method that enables tangential (shear) force input on a touchscreen using a rubber-mounted slim transparent sheet. The sheet has tiny conductive material(s) attached to specific positions on the underside so that displacement of the sheet is recognized as touch input(s). This allows the system to distinguish between normal touch input and shear force input depending on whether the sheet moves or not, without requiring any external sensors or power. We first introduce ShearSheet's simple implementation for a smartphone and then discuss several promising interaction techniques that augment shear interaction with one- and two-finger operations. We demonstrate the effectiveness of ShearSheet's smooth transition between position- and rate-based control through a controlled user study using simple scrolling tasks. The results confirm that ShearSheet offers improved performance such as fewer operations and higher subjective preference, suggesting its substantial benefits and potential applications.

We propose Third-Person Piloting, a novel drone manipulation interface that increases situational awareness using an interactive third-person perspective from a second, spatially coupled drone. The pilot uses a controller with a manipulatable miniature drone. Our algorithm understands the relationship between the pilot's eye position and the miniature drone and ensures that the same spatial relationship is maintained between the two real drones in the sky. This allows the pilot to obtain various third-person perspectives by changing the orientation of the miniature drone while maintaining standard primary drone control using the conventional controller. We design and implement a working prototype with programmable drones and propose several representative operation scenarios. We gather user feedback to obtain the initial insights of our interface design from novices, advanced beginners, and experts. Our result suggests that the interactive third-person perspective provided by the second drone offers sufficient potential for increasing situational awareness and supporting their primary drone operations.

In this work, we were interested in using smartphone touchscreen keyboard for text typing in virtual environments (VEs) with head-mounted display. We carried out an experiment comparing the smartphone to the ordinary devices: gamepad and HTC Vive Controllers. We represented the touchscreen keyboard in the VE with a virtual interface and the fingertips with tracked green circles. A confirm-on-release paradigm was employed for text typing. Results showed that the smartphone did not fully outperformed the other devices. However, unlike the other devices, smartphones users tended to correct progressively their error while typing thanks to their familiarity with the device.

We propose an approach for flexibly scalable workspace that changes viewer's perceived size of the workspace by simulating a virtually movable wall on the real wall. We conducted an experiment comparing viewers depth estimates between extended workspace virtually and physically. Besides, we investigated walls motion effects; discrete wall motion or continuous wall motion. Distances were estimated verbally and using perceptual matching method. Verbal depth estimates were equivalent in the virtual extended workspace and the physical one, with a high accuracy (estimation errors were less than 5%). However, perceptual matching estimates were signicantly different between virtual and physical scaling. Overall, they were slightly overestimated. No difference of the motion effect was found. Our results clearly suggest the potential of the workspace scaling by just giving such visual impression of extended workspace using perspective projection technique.

The 10th annual IEEE 3DUI Contest focuses on the development of 3D User Interfaces (3DUIs) using passive haptic feedback in Mixed Reality Environments. Material properties of real objects are enhanced with virtual behaviors while used for interaction in a mixed reality scenario. The contest was open to anyone interested in 3DUIs, from researchers to students, enthusiasts, and professionals. The purpose of the contest is to stimulate innovative and creative solutions to challenging 3DUI problems.

Jumping is a fundamental movement in our daily lives that is often used in many video games. However, little research has been done on jumping and its possible use as a redirection technique in virtual reality (VR). In this study we explore Redirected Jumping, a novel redirection technique which enables us to purposefully manipulate the mapping of the user's physical jumping movements (e.g., distance and direction) to movement in the virtual space, allowing richer and more active physical VR experiences within a limited tracking area. To demonstrate the possibilities afforded by Redirected Jumping, we implemented a jump detection algorithm and jumping redirection methods for three basic jumping actions (i.e., horizontal, vertical, and rotational jumps) using common VR devices. We conducted three user studies to investigate the effective manipulation ranges, and the results revealed that our methods can manipulate a user's jumping movements without his/her noticing, similar to walking.

Abstract As the range of handheld, mobile and desktop devices expands and worldwide demand for collaborative application tools increases, there is a growing need for higher speed impromptu cross-device application sharing to keep up with workplace requirements for on-site or remote collaborations. To address this, we have developed CamCutter, a cross-device interaction technique enabling a user to quickly select and share an application running on another screen using the camera of a handheld device. This technique can accurately identify the targeted application on a display using our adapted computer vision algorithm, system architecture and software implementation, allowing impromptu real-time and synchronized application sharing between devices. For desktop and meeting room set-ups, we performed a technical evaluation, measuring accuracy and speed of migration. For a single-user reading task and a collaborative composition task, we carried out a user study comparing our technique with commercial screen sharing applications. The results of this study showed both higher performance and preference for our system. Finally, we discuss CamCutter’s limitations and present insights for future vision-based cross-device application sharing.

This paper describes an unsupervised adaptation method of deep neural networks (DNNs) regarding discriminative sound source localization (SSL). DNNs-based SSL and its unsupervised adaptation fail under different conditions from those during training. The estimations sometimes include incoherent unpredictable errors due to the NN's non-linearity. We propose an eliminative posterior probability constraint using a model-based SSL for unsupervised DNNs adaptation. This constraint forces the probability of “less possible candidates” to become zero to eliminate incoherent errors. The candidates are indicated by a model-based SSL method because it can estimate the azimuth of the sound source with moderate accuracy and explicit reasoning. As a result, the localization performance of adapted DNNs improved more than that of model-based SSL. Experimental results showed that our method improved localization correctness of 1D azimuth and 3D regions by a maximum of 13.3 and 5.9 points compared with the model-based SSL.

Digital photos and their metadata get explosively growth, requiring better support to browsing them. We propose a dynamic and flexible visualization of digital photos and their metadata. A prototype is designed and implemented based on the algorithm of D-FLIP, our previous work for flexibly displaying a large photo collection. We design various dynamic photo visualizations with up to four-dimensional meta-information by using Bertin's visual variables and three-dimensional representation. It allows users to dynamically and effectively manage photo visualizations by selecting meta-information of user's current needs and interest.

Large interactive tabletops are beneficial for various tasks involving exploration and visualization, but regions of the screen far from users can be difficult to reach. We propose AdapTable; a concept and prototype of a flexible multi-display tabletop that can physically reconfigure its layout, allowing for interaction with difficult-to-reach regions. We conducted a design study where we found users preferred to change screen layouts for full-screen interaction, motivated by reduced physical demands and frustration. We then prototyped AdapTable using four actuated tabletop displays each propelled by a mobile robot, and a touch menu was chosen to control the layout. Finally, we conducted a user study to evaluate how well AdapTable addresses the reaching problem compared with a conventional panning technique. Our findings show that AdapTable provides a more efficient method for complex full-screen interaction.

We present a concept-based world building approach, realized in a system called VR Safari Park, which allows users to rapidly create and manipulate a world simulation. Conventional world building tools focus on the manipulation and arrangement of entities to set up the simulation, which is time consuming as it requires frequent view and entity manipulations. Our approach focuses on a far simpler mechanic, where users add virtual blocks which represent world entities (e.g. animals, terrain, weather, etc.) to a World Tree, which represents the simulation. In so doing, the World Tree provides a quick overview of the simulation, and users can easily set up scenarios in the simulation without having to manually perform fine-grain manipulations on world entities. A preliminary user study found that the proposed interface is effective and usable for novice users without prior immersive VR experience.

We are pursuing a vision of reactive interior spaces that are aware of people's actions and transform according to changing needs. We envision furniture and walls that act as interactive displays and that shapeshift to the correct physical form, and the appropriate interactive visual content and modality. This paper briefly describes our proposal based on our recent efforts on realizing this vision.

Many motion tracking systems require solving inverse problem to compute the tracking result from original sensor measurements. For real-time motion tracking, such typical solutions as the Gauss-Newton method for solving their inverse problems need an initial value to optimize the cost function through iterations. A powerful initializer is crucial to generate a proper initial value for every time instance and, for achieving continuous accurate tracking without errors and rapid tracking recovery even when it is temporally interrupted. An improper initial value easily causes optimization divergence, and cannot always lead to reasonable solutions. Therefore, we propose a new initializer based on random-forest to obtain proper initial values for efficient real-time inverse problem computation. Our method trains a random-forest model with varied massive inputs and corresponding outputs and uses it as an initializer for runtime optimization. As an instance, we apply our initializer to IM3D[1], which is a real-time magnetic 3D motion tracking system with multiple tiny, identifiable, wireless, occlusion-free passive markers (LC coils).

We present a concept of dynamically adaptive workspace using MovemenTable, autonomously moving interactive digital table. In this demo, we introduce two interaction scenarios, arranging workspace anywhere and floating interactive surface on large floor screen. In the first scenario, the interactive digital table is automatically arranged anywhere in a room to keep user's workspace efficient and optimal by self-actuated movement of MovemenTable. The second scenario is more advanced one with collaborative use of large floor screen system where the MovemenTable works as a floating interactive surface that allows local data exploration of the global information on the floor screen.

We conducted a user study evaluating five selection techniques for augmented reality in optical see-through head-mounted displays (OST-HMDs). The techniques we studied aim at supporting mobile usage scenarios where the devices do not need external tracking tools or special environments, and therefore we selected techniques that rely solely on tracking technologies built into conventional commercially available OST-HMDs [i.e. gesture trackers, gaze tracking and inertial measurement units (IMUs)]. While two techniques are based on raycasting using built-in IMU sensing, three techniques are based on a hand-controlled 3D cursor using gestural tracking. We compared these techniques in an experiment with 12 participants. Our results show that raycasting using head orientation (i.e. IMU on the headset) was the fastest, fatigueless and the most preferable technique to select spatially arranged objects. We discuss the implications of our findings for design of interaction techniques in mobile OST-HMDs.

Wall displays almost universally assume a mostly flat and static shape. We ask two questions: Would people choose a flat display for a given interaction scenario and, if not, what are the display shapes they actually prefer? We conducted a design study around these two questions. Our results show that participants designed different screen shapes that varied based upon peoples' distance from the display and the content shown. Shapes ranged primarily between flat, separated, concave, L-shape and convex displays. Based on our findings, we designed a dynamic display that changes to these and other configurations. Shape-shifting is controlled either by explicit interaction (where the display responds to hand gestures) or implicitly (where the display infers a shape based both on its content and the sensed positions of the people around it). Overall, we contribute: a study that motivates research on shape-shifting wall displays, and a shape-shifting display system that responds to explicit and implicit controls to match particular activities.

We describe our approach that derives reliable 6-DOF information including the translation and the rotation of a rigid marker in a 3D space from a set of insufficient 5-DOF measurements. As a practical example, we carefully constructed a prototype and its design and evaluated it in our 3D dexterous motion-tracking system, IM6D, which is our novel real-time magnetic 3D motion-tracking system that uses multiple identifiable, tiny, lightweight, wireless, and occlusion-free markers. The system contains two key technologies; a 6-DOF computation algorithm and a marker design for 6D marker. The 6-DOF computation algorithm computes the result of complete 6-DOF information including translation and rotation in 3D space for a single rigid marker that consists of three LC coils. We propose several possible approaches for implementation, including geometric, matrix-based kinematics, and computational approaches. In addition, we introduce workflow to find an optimal marker design for the system to achieve the best compromise between its smallness and accuracy based on the tracking principle. We experimentally compare the performances of some typical marker prototypes with different layouts of LC coils. Finally, we also show another experimental result to prove the effectiveness of the results from the solutions in these two problems.

This paper explores the Boundless Scroll interface that extends the motor space for finger drag motions into the off-screen space using a tracking system around the screen. Seamlessly extended motor spaces enable continuous scroll actions across the on- and off-screens, reducing the chance of clutching and preserving content visibility during long-distance and smooth scrolling actions. We empirically evaluate our new technique, the Boundless Scroll and compare it to ordinary scroll interfaces (Drag and Flick) in two dominant screen devices. The first screen condition uses a large flat touch screen where users manipulate a non-full-screen window while using the surrounding space as an extended motor space. The second condition is a mobile device to examine mid-air finger scroll motions in the off-screen space using a 3D motion sensor. An empirically controlled study demonstrates our Boundless Scroll's many benefits, such as fewer clutching actions, occlusion-less content observations, and efficient and effortless off-screen acquisitions and also provided further design factors, implementations of the Boundless Scroll, and around-device interfaces.

We propose Coupled-clay, a bi-directional 3D collaborative interactive environment that supports the 3D modeling work between groups of users at remote locations. Coupled-clay consists of two network-connected workspaces, the Physical Interaction Space and the Virtual Interaction Space. The physical interaction space allows a user to directly manipulate a physical object whose shape and position are precisely tracked. This tracked 3D information is transferred to the virtual interaction space in real time. The virtual interaction space is made of an interactive multi-user stereoscopic 3D tabletop, or other 3D displays with adequate interaction device. The users at the virtual interaction space observe the virtual 3D object which corresponds to the physical object and manipulate its geometrical attributes (e.g., translation, rotation and scaling). Additionally, they can control the graphical attributes of the virtual object such as color and texture. Information about changes in geometrical and graphical attributes are sent back to the physical interaction space in real time and reflected to the object in the physical interaction space by a robotic arm and a top-mounted projector. Coupled-clay can be used to remotely collaborate on 3D modeling tasks such as between a skilled designer and novice learners. This paper details our Coupled-clay implementation and presents its interaction capabilities.

We propose IM6D, a novel real-time magnetic motion-tracking system using multiple identifiable, tiny, lightweight, wireless and occlusion-free markers. It provides reasonable accuracy and update rates and an appropriate working space for dexterous 3D interaction. Our system follows a novel electromagnetic induction principle to externally excite wireless LC coils and uses an externally located pickup coil array to track each of the LC coils with 5-DOF. We apply this principle to design a practical motion-tracking system using multiple markers with 6-DOF and to achieve reliable tracking with reasonable speed. We also solved the principle's inherent dead-angle problem. Based on this method, we simulated the configuration of parameters for designing a system with scalability for dexterous 3D motion. We implemented an actual system and applied a parallel computation structure to increase the tracking speed. We also built some examples to show how well our system works for actual situations.

We present our study of Calamaro: a robotic platform designed to investigate the impact of emotive motion on people, and its deployment in an extensive field study in a busy public space at the University of Calgary campus. Our paper details the design of the Calamaro robot, the field study conducted with it, including hundreds of observers and 88 participants, and our quantitative and qualitative findings. The paper provides a thorough discussion of the implications of our results on the design of robotic emotive motions, and reflections on the deployment of robotic interfaces in field studies.

This paper describes Ambient Party Room, which is a room-shaped information environment for party situations. This system estimates the conversation states by measuring participants' nonverbal cues and displays various types of information to stimulate the conversation. In this study, we reconstructed the system architecture from an early prototype and newly examined several types of information stimulus using a floor display, wall displays, and speakers. We conducted an experiment in which six participants talked freely in Ambient Party Room. Results showed that both profile-based information stimulus and feedback information of participants' behaviors contributed to their activated conversations.

We propose a room-shaped information environment called Ambient Suite that enhances interpersonal communication. In Ambient Suite, the room itself works as both sensors to estimate the conversation states of participants and displays to present information to stimulate conversation. This paper introduces an implementation assumed standing-party situations as a typical use case of Ambient Suite. From the result of user study using its implementation, we confirmed that our system adequately encouraged participant conversations.

We propose a cup-shaped device called Cup-le for conversational experiment. Cup-le has various sensors to record participants' nonverbal behavior such as utterance. Cup-le saves the work of attaching sensors. Also, since Cup-le appears cup-shaped, it can sense without being aware of the sensor unlike conventional wearable sensors. Moreover, by attaching touch display, Cup-le can show information and can be operated. In this conversational experiment, we use the input-output as method to have a questionnaire during the experiment and capture participants' impression for their conversation. We conducted real conversational experiment with Cup-le, and we confirmed that Cup-le supported sufficiently the experiment without affecting their impression.

We propose a furry and scalable multi-touch display called the “FuSA2Touch Display.” The furry type of tactile sensation of this surface affords various interactions such as stroking or clawing. The system utilizes plastic fiber optic bundles to realize a furry-type texture. The system can show visual feedback by projection and detects multi-touch input. We implemented a 24-inch display, and found that our implemented display encourages users to interact with it in various actions.

In this paper, we present our vision on large-scale and dynamic social network analysis in real environment, which is expected to be enabled by introduction of large-scale heterogeneous sensors in ambient environment. We address challenges toward realization of large-scale dynamic social network analysis in real environment, and discuss several promising applications. We finally present our preliminary experimental results of dynamic social network analysis for six-person social gatherings in real environment.

This paper details the design and evaluation of the Delphian Desktop, a mechanism for online spatial prediction of cursor movements in a Windows-Icons-Menus-Pointers (WIMP) environment. Interaction with WIMP-based interfaces often becomes a spatially challenging task when the physical interaction mediators are the common mouse and a high resolution, physically large display screen. These spatial challenges are especially evident in overly crowded Windows desktops. The Delphian Desktop integrates simple yet effective predictive spatial tracking and selection paradigms into ordinary WIMP environments in order to simplify and ease pointing tasks. Predictions are calculated by tracking cursor movements and estimating spatial intentions using a computationally inexpensive online algorithm based on estimating the movement direction and peak velocity. In testing the Delphian Desktop effectively shortened pointing time to faraway icons, and reduced the overall physical distance the mouse (and user hand) had to mechanically traverse.