In office environments, achieving flexible workspace configurations by adjusting the position, height, and curvature of partitions is crucial for accommodating diverse worker needs, but implementing such flexibility typically introduces system complexities. We present OrigamiWalls, a robotic partitioning system that addresses this trade-off by using origami structures to permit diverse configurations without requiring high-degree-of-freedom actuation. Based on expert insights, we explore the OrigamiWalls design space and report on the implementation of the desktop-scale and floor-scale prototypes utilizing the Tachi-Miura polyhedron structure. These prototypes can transition continuously between three distinct forms (no wall, curved wall, and flat wall) with high expansion ratios (up to 660%) using a single reel-type linear actuator. Our preliminary user study with our desktop-scale prototype suggests that origami-based shape changes are perceived to enable task-adaptive workspace configurations and support workers’ task transitions unobtrusively and safely.

Ikebana, the Japanese art of flower arrangement, offers practitioners significant mental health benefits, yet daily practice is often limited by a lack of guidance and resources. This work explores how augmented reality (AR) in combination with AI can address these accessibility challenges to support ikebana self-practice. Based on a formative study interviewing 14 students and instructors on their ikebana experiences and challenges, we designed HanaARrange, an AR system providing visual and verbal guidance as users arrange real-life ikebana. In a user study with 9 participants, we found that HanaARrange outperformed traditional self-practice in achieving correct basic ikebana forms, while offering comparable mental well-being benefits. We then conducted iterative refinements with 6 more experienced learners, identifying recursive use patterns and future opportunities. We contribute insights into a design framework for supporting intangible cultural heritage practices, expanding the ikebana ecosystem beyond constraints of physical lessons while supporting mental well-being and life qualities.

Habitual static stretching is recommended in many ergonomics guidelines, but continuing this habit remains challenging to workers. We address two factors causing this difficulty: most workers lack an understanding of the appropriate mechanisms of stretching, and stretching sessions require the interruption of ongoing tasks. We propose Screen-Directed Stretching, a novel technique that supports neck stretching in extended reality (XR) workspaces without interrupting work. Our technique temporarily repositions the virtual screen that the user is focusing on, guiding them to rotate their head in yaw, pitch, and roll directions and to maintain a posture for a specific duration, thus facilitating muscle extension. Through two preliminary user studies, we developed a prototype that seamlessly switches the screen’s coordinate system between world-bound and body-bound frames of reference, balancing practical workability and good stretching guidance. A user study (N = 16) demonstrates that our technique effectively induces stretching movements while minimizing loss of task efficiency.

Eye strain presents a significant challenge in human-information interaction in virtual reality (VR), as prolonged exposure contributes to various eye problems. This study introduces a new gaze redirection method called EyeXRciser, which passively activates eye movements to help prevent eye muscle stiffness during VR reading. This method achieves gaze redirection by slowly shifting the relative position of the text window within the user’s field of view through a head-bound coordinate system. We implemented our method with two different redirection speeds (i.e., unnoticeable speed at 0.03 rad/s; noticeable speed at 0.12 rad/s) and conducted a user study (N = 24) comparing with a baseline using a fixed text window. Results show that both our methods successfully minimized the decline in accommodative ability caused by prolonged reading, without negatively impacting reading comprehension. Results also show that the unnoticeable redirection speed produced less subjective discomfort, eye fatigue, and reading distraction than the noticeable speed.

Current Mixed Reality (MR) headsets are typically controlled by mid-air hand inputs, which require significant arm effort and consequently reduce input speed and accuracy during extended sessions. We propose BlanchTouch, a research prototype that explores an alternative touch input approach for MR by appropriating uninstrumented flat surfaces as interactive areas, with reduced arm effort. It incorporates a CNN model that detects the fingertip’s pressure-induced blanching phenomenon when a firm touch is applied to hard surfaces, serving as a confirmation of selection. Additionally, hand tracking via the headset is employed to determine the fingertip’s position, thereby providing targeting information for virtual buttons or objects. BlanchTouch operates using only the headset’s built-in cameras and sensors, without requiring additional hardware or prior surface calibration, enabling walk-up use in unfamiliar environments. We conducted two user studies to evaluate the system and compare its performance against a baseline (a real touchscreen) and the state-of-the-art mid-air interaction. Results demonstrated that BlanchTouch achieved an average input positional error of 5.33mm and a recall rate of 97.88%. Furthermore, the findings indicate that BlanchTouch enables competitive input speeds, alleviates arm strain, and delivers satisfactory feedback for each touch.

In this paper, we present FanType, a text entry technique that enables efficient typing on small virtual keyboards in extended reality (XR), closely resembling thumb-based typing on smartphones and tablets. While virtual environments theoretically provide unlimited space, practical XR usage often occurs in mobile and spatially constrained contexts, such as when using augmented reality (AR) headsets while walking or in confined environments, where large virtual keyboards are impractical, as they occupy substantial visual space and interfere with other virtual or real content. To address this, FanType integrates a fan-shaped interface that groups keys within the thumb's reach and leverages a lift-up gesture for disambiguation, thereby reducing hand movement while preserving natural typing speed. Additionally, an intention-inferring mechanism involving tailored algorithms is implemented to counterbalance touch bias and further ensure input accuracy. A user study with 22 participants evaluated the method on tablet- and smartphone-size portrait keyboards, comparing it with a baseline (touch-only) and a state-of-the-art mid-air technique. Results provide insights into the performance of such a design and highlight the challenges of designing typing methods for small virtual keyboards.

Drawing, writing, and painting on 2D surfaces in VR offers arbitrary interaction with virtual canvases of unlimited size. However, this advantage has not been fully exploited because physical surfaces are needed to support accurate and comfortable user interactions, and the size of such surfaces is limited. We propose Redirected Drawing, a novel methodology that applies visual manipulation to the user’s drawing movement in VR to expand the perceived surface size required for the drawing experience. To this end, this study specifically explores translation gain manipulation, where the displacement of a drawing stroke in reality is simply multiplied by a gain to represent it in VR. We conducted a user study (N = 18) to measure the noticeability, acceptability, and task performance of a simple VR line-tracing task under different gain and input methods (i.e., finger in mid-air, finger on surface, pen on surface). Our results show that pen-based drawing on a physical surface allows for a gain of up to 1.16 without being noticed by the user, which is more suited to expanding a virtual canvas size than finger-based drawing on a physical surface or in mid-air. Furthermore, we show that user acceptability of the gain manipulation could be estimated by a model using the subjective discrepancy caused by the gain. Results for this model suggest that users could accept gains of up to 1.30. Based on our results, we derive guidelines for the effective use of gain manipulation.

Dental cutting is a crucial skill for dental students. However, current VR dental cutting training systems rely on bulky and costly haptic devices, which reduce opportunities for individual practice. Moreover, the limitations imposed by the maximum reaction force of an active haptic device would impact the range of tooth hardness that can be reproduced. We propose a compact XR tooth-cutting training system, VirtuEleDent, that employs a passive haptic approach using a 3D-printed physical teeth model and a three-dimensionally tracked handpiece. Their spatial relationship is accurately rendered in the virtual environment of a mobile head-mounted display (HMD), providing users with realistic haptic sensations during virtual tooth-cutting exercises. Our tracking platform is operated using electromagnetic resonance (EMR) stylus technology and consists of a digitizer (i.e., tracking board) and a handpiece device. A customized EMR stylus unit (i.e., resonance coil) and an inertial measurement unit (IMU) sensor are installed inside the handpiece, allowing for precise measurement of its tip’s 3D position and orientation. This setup enables the learner to physically manipulate dexterous handpieces on the teeth model while experiencing virtual tooth-cutting in the HMD. First, we detail the design and implementation of our initial prototype system, including the tracking platform. We then describe an expert user study conducted with our prototype, where licensed dentists compare VirtuEleDent to a baseline with a conventional haptic device. Finally, the educator’s feedback demonstrates its potential and informs our further improvements and the directions of future development.

We propose SnapSteer, a bimanual 3D manipulation interface using common VR controllers, which allows restriction of motion degrees of freedom (DoFs) as needed. This interface is based on the conventional one-handed 6-DoF manipulation interface called Robot Telekinesis, and assigns the other hand the role of controlling whether and in which direction the DoFs are restricted. This enables users to quickly switch between unconstrained 6-DoF operation and precise 1-DoF operation according to the task. We designed and implemented a prototype of this interface in VR, and conducted a user study (N=12) comparing its performance in a straight 3D steering task with two baseline interfaces (i.e., a 6-DoF individual control interface and Robot Telekinesis). The results showed that our interface outperformed the other two in task efficiency. On the other hand, there was no significant difference in subjective workload or usability compared to Robot Telekinesis, and which derives discussion of improvements to visual feedback during the direction adjustment phase.

Building higher-quality image classification models requires better performance analysis (PA) to help understand their behaviors. We propose ConfusionLens, a dynamic and interactive visualization interface that augments a conventional confusion matrix with focus+context visualization. This interface allows users to seamlessly switch table layouts among three views (overall view, class-level view, and between-class view) while observing all of the instance images in a single screen. We designed and implemented a ConfusionLens prototype that supports hundreds of instances, and then conducted a user study (N \(=\) 14) to evaluate it compared to the conventional confusion matrix with a split view of instances. Results show that ConfusionLens achieved faster task-completion time in observing instance-level performance and higher accuracy in observing between-class confusion. Moreover, we conducted an expert interview (N \(=\) 6) to investigate the applicability of our interface to practical PA tasks, and then implemented several extensions of ConfusionLens based on the feedback. Feedback on these extensions from users experienced in image classification (N \(=\) 5) demonstrated their general usefulness and highlighted their beneficial use in PA tasks.

We introduce InflatableBots, shape-changing inflatable robots for large-scale encountered-type haptics in VR. Unlike traditional inflatable shape displays, which are immobile and limited in interaction areas, our approach combines mobile robots with fan-based inflatable structures. This enables safe, scalable, and deployable haptic interactions on a large scale. We developed three coordinated inflatable mobile robots, each of which consists of an omni-directional mobile base and a reel-based inflatable structure. The robot can simultaneously change its height and position rapidly (horizontal: 58.5 cm/sec, vertical: 10.4 cm/sec, from 40 cm to 200 cm), which allows for quick and dynamic haptic rendering of multiple touch points to simulate various body-scale objects and surfaces in real-time across large spaces (3.5 m x 2.5 m). We evaluated our system with a user study (N = 12), which confirms the unique advantages in safety, deployability, and large-scale interactability to significantly improve realism in VR experiences.

Videos accompanied by documents—document-based videos—enable presenters to share contents beyond videos and audience to use them for detailed content comprehension. However, concurrently exploring multiple channels of information could be taxing. We propose SwapVid, a novel interface for viewing and exploring document-based videos. SwapVid seamlessly integrates a video and a document into a single view and lets the content behaves as both video and a document; it adaptively switches a document-based video to act as a video or a document upon direct manipulation (e.g., scrolling the document, manipulating the video timeline). We conducted a user study with twenty participants, comparing SwapVid to a side-by-side video/document views. Results showed that our interface reduces time and physical workload when exploring slide-based documents based on video referencing. Based on the study findings, we extended SwapVid with additional functionalities and demonstrated that it further extends the practical capabilities.

In room-scale virtual reality, providing continuous haptic feedback from touching grounded objects, such as walls and handrails, has been challenging due to the user’s walking range and the required force. In this study, we propose LoopBot, a novel technique to provide continuous haptic feedback from grounded objects using only a single user-following robot. Specifically, LoopBot is equipped with a loop-shaped haptic prop attached to an omnidirectional robot that scrolls to cancel out the robot’s displacement, giving the user the haptic sensation that the prop is actually fixed in place, or “grounded.” We first introduce the interaction design space of LoopBot and, as one of its promising interaction scenarios, implement a prototype for the experience of walking while grasping handrails. A performance evaluation shows that scrolling the prop cancels 77.5% of the robot’s running speed on average. A preliminary user test (N = 10) also shows that the subjective realism of the experience and the sense of the virtual handrails being grounded were significantly higher than when the prop was not scrolled. Based on these findings, we discuss possible further development of LoopBot.

We present a framework that captures sparse Lagrangian flow information from a volume of real liquid and reconstructs its detailed kinematic information in real time. Our framework can perform flow reconstruction even when the liquid is disturbed by an object of unknown movement and shape. Through a large dataset of liquid moving under external disturbance, an agent is trained using reinforcement learning to reproduce the target flow kinematics with only the captured sparse information as inputs while remaining oblivious to the movement and the shape of the disturbance sources. To ensure that the underlying simulation model faithfully obeys physical reality, we also optimize the viscosity parameters in Smoothed Particle Hydrodynamics (SPH) using classical fluid dynamics knowledge and gradient-based optimization. By quantitatively comparing the reconstruction results against real-world and simulated ground truth, we verified that our reconstruction method is resilient to different agitation patterns.

Importance sampling techniques significantly reduce variance in physically based rendering. In this article, we propose a novel online framework to learn the spatial-varying distribution of the full product of the rendering equation, with a single small neural network using stochastic ray samples. The learned distributions can be used to efficiently sample the full product of incident light. To accomplish this, we introduce a novel closed-form density model, called the Normalized Anisotropic Spherical Gaussian mixture, that can model a complex light field with a small number of parameters and that can be directly sampled. Our framework progressively renders and learns the distribution, without requiring any warm-up phases. With the compact and expressive representation of our density model, our framework can be implemented entirely on the GPU, allowing it to produce high-quality images with limited computational resources. The results show that our framework outperforms existing neural path guiding approaches and achieves comparable or even better performance than state-of-the-art online statistical path guiding techniques.

RedirectedDoors is a visuo-haptic door-opening redirection technique in VR, and it has shown promise in its ability to efficiently compress the physical space required for a room-scale VR experience. However, its previous implementation has only supported laboratory experiments with a single door opening at a fixed location. To significantly expand this technique for room-scale VR, we have developed RedirectedDoors+, a robot-based system that permits consecutive door-opening redirection with haptics. Specifically, our system is mainly achieved with the use of three components: (1) door robots, a small number of wheeled robots equipped with a doorknob-like prop, (2) a robot-positioning algorithm that arbitrarily positions the door robots to provide the user with just-in-time haptic feedback during door opening, and (3) a user-steering algorithm that determines the redirection gain for every instance of door opening to keep the user away from the boundary of the play area. Results of simulated VR exploration in six virtual environments reveal our system's performance relative to user walking speed, paths, and number of door robots, from which we derive its usage guidelines. We then conduct a user study ($N=12$) in which participants experience a walkthrough application using the actual system. The results demonstrate that the system is able to provide haptic feedback while redirecting the user within a limited play area.

We examine the effect of auditory cues occurring in reality on redirection. Specifically, we set two hypotheses: the auditory cues emanating from fixed positions in reality (Fixed sound, FS) increase the noticeability of redirection, while the auditory cues whose positions are manipulated consistently with the visual manipulation (Redirected sound, RDS) decrease the noticeability of redirection. To verify these hypotheses, we implemented an experimental environment that virtually reproduced FS and RDS conditions using binaural recording, and then we conducted a user study ( N=18) to investigate the detection thresholds (DTs) for rotational manipulation and the sound localization accuracy of the auditory cues under FS and RDS, as well as the baseline condition without auditory cues (No sound, NS). The results show, against the hypotheses, FS gave a wider range of DTs than NS, while RDS gave a similar range of DTs to NS. Combining these results with those of sound localization accuracy reveals that, rather than the auditory cues affecting the participants' spatial perception in VR, the visual manipulation made their sound localization less accurate, which would be a reason for the increased range of DTs under FS. Furthermore, we conducted a follow-up user study ( N=11) to measure the sound localization accuracy of FS where the auditory cues were actually placed in a real setting, and we found that the accuracy tended to be similar to that of virtually reproduced FS, suggesting the validity of the auditory cues used in this study. Given these findings, we also discuss potential applications.

We propose BirdViewAR, a surroundings-aware remote drone-operation system that provides significant spatial awareness to pilots through an augmented third-person view (TPV) from an autopiloted secondary follower drone. The follower drone responds to the main drone’s motions and directions using our optimization-based autopilot, allowing the pilots to clearly observe the main drone and its imminent destination without extra input. To improve their understanding of the spatial relationships between the main drone and its surroundings, the TPV is visually augmented with AR-overlay graphics, where the main drone’s spatial statuses are highlighted: its heading, altitude, ground position, camera field-of-view (FOV), and proximity areas. We discuss BirdViewAR’s design and implement its proof-of-concept prototype using programmable drones. Finally, we conduct a preliminary outdoor user study and find that BirdViewAR effectively increased spatial awareness and piloting performance.

Room-scale VR has been considered an alternative to physical office workspaces. For office activities, users frequently require planar input methods, such as typing or handwriting, to quickly record annotations to virtual content. However, current off-the-shelf VR HMD setups rely on mid-air interactions, which can cause arm fatigue and decrease input accuracy. To address this issue, we propose UbiSurface, a robotic touch surface that can automatically reposition itself to physically present a virtual planar input surface (VR whiteboard, VR canvas, etc.) to users and to permit them to achieve accurate and fatigue-less input while walking around a virtual room. We design and implement a prototype of UbiSurface that can dynamically change a canvas-sized touch surface's position, height, and pitch and yaw angles to adapt to virtual surfaces spatially arranged at various locations and angles around a virtual room. We then conduct studies to validate its technical performance and examine how UbiSurface facilitates the user's primary mid-air planar interactions, such as painting and writing in a room-scale VR setup. Our results indicate that this system reduces arm fatigue and increases input accuracy, especially for writing tasks. We then discuss the potential benefits and challenges of robotic touch devices for future room-scale VR setups.

We propose WaddleWalls, a room-scale interactive partitioning system using a swarm of robotic partitions that allows occupants to interactively reconfigure workspace partitions to satisfy their privacy and interaction needs. The system can automatically arrange the partitions’ layout designed by the user on demand. The user specifies the target partition’s position, orientation, and height using the controller’s 3D manipulations. In this work, we discuss the design considerations of the interactive partition system and implement WaddleWalls’ proof-of-concept prototype assembled with off-the-shelf materials. We demonstrate the functionalities of WaddleWalls through several application scenarios in an open-planned office environment. We also conduct an initial user evaluation that compares WaddleWalls with conventional wheeled partitions, finding that WaddleWalls allows effective workspace partitioning and mitigates the physical and temporal efforts needed to fulfill ad hoc social and privacy requirements. Finally, we clarify the feasibility, potential, and future challenges of WaddleWalls through an interview with experts.

We propose RedirectedDoors, a novel space-efficient technique for redirection in VR focused on door-opening behavior. This technique manipulates the user’s walking direction by rotating the entire virtual environment (VE) at a certain angular ratio of the door being opened. This ratio is called door rotation gain. At the same time, the virtual door’s position is kept unmanipulated so that a realistic door-opening user experience can be ensured. We designed and implemented the rotational manipulation algorithm and two types of door-opening interfaces; with and without a doorknob-type passive haptic prop. We then conducted a user study (N = 12) to investigate redirection performance and user feedback as we examined three independent variables: door rotation gain, door-opening interface, and door-opening direction (push/pull). From the results, the estimated detection thresholds generally showed a higher space efficiency of redirection with our technique. Our results also showed that providing the haptic feedback led to a higher noticeability of redirection, but at the same time supported a higher subjective sense of realism and less discomfort. Following our results, we discuss which combinations of gain and door-opening direction can jointly provide lower noticeability and higher acceptability.

Jumping onto steps is a promising action for creating an instant height movement in VR, but installing physical steps is impractical. We propose PseudoJumpOn, a novel locomotion technique using a common VR setup that allows the user to experience virtual step-up jumping motion by applying two types of viewpoint-manipulation methods to a physical jump on a flat floor. The core idea is to replicate the physical characteristics of ascending jumps, and thus we designed two viewpoint-manipulation methods: gain manipulation, which differentiates the ascent and descent height, and peak shifting, which delays the peak timing. We conducted a user study asking participants (N = 20) to experience two-legged step-up jumps onto 0.2–0.8-m heights in VR as the two methods were applied in combination (gain manipulation: five conditions where the ascending gain was 1.0–5.0; peak shifting: four conditions where the peak timing in VR was delayed by 0–1.0 ratios of the original timing). The results showed that the participants in most conditions felt positively in terms of reality and naturalness of actually jumping onto steps, even though knowing no physical steps existed. In addition, subsequent analyses also derived practical guidelines for determining the appropriate gains and the potential use of peak shifting to achieve a natural step-up jumping experience.

We propose HandyGaze, a 6-DoF gaze tracking technique for room-scale environments that can be carried out by simply holding a smartphone naturally without installing any sensors or markers in the environment. Our technique simultaneously employs the smartphone’s front and rear cameras: The front camera estimates the user’s gaze vector relative to the smartphone, while the rear camera (and depth sensor, if available) performs self-localization by reconstructing a pre-obtained 3D map of the environment. To achieve this, we implemented a prototype that works on iOS smartphones by running an ARKit-based algorithm for estimating the user’s 6-DoF head orientation. We additionally implemented a novel calibration method that offsets the user-specific deviation between the head and gaze orientations. We then conducted a user study (N=10) that measured our technique’s positional accuracy to the gaze target under four conditions, based on combinations of use with and without a depth sensor and calibration. The results show that our calibration method was able to reduce the mean absolute error of the gaze point by 27%, with an error of 0.53 m when using the depth sensor. We also report the target size required to avoid erroneous inputs. Finally, we suggest possible applications such as a gaze-based guidance application for museums.

We propose a novel phone-case-shaped interface named TetraForce, which enables four types of force input consisting of two force directions (i.e., pressure force and shear force) and two force-applied surfaces (i.e., touch surface and back surface) in a single device. Force detection is achieved using the smartphone's built-in magnetometer (and supplementary accelerometer and gyro sensor) by estimating the displacement of a magnet attached to a 3-DoF passively movable panel at the back. We conducted a user study (N=12) to investigate the fundamental user performance by our interface and demonstrated that the input was detected as intended with a success rate of 97.4% on average for all four input types. We further conducted an ideation workshop with people who were involved in human-computer interaction (N=12) to explore possible applications of this interface, and we obtained 137 ideas for applications using individual input types and 51 possible scenarios using them in combination. Organizing these ideas reveals the advantages of each input type and suggests that our interface is useful for applications that require complex operations and that it can improve intuitiveness through elastic feedback.

We propose IM3D+, a novel approach to reconstructing 3D motion data from a flexible magnetic flux sensor array using deep learning and a structure-aware temporal bilateral filter. Computing the 3D configuration of markers (inductor-capacitor (LC) coils) from flux sensor data is difficult because the existing numerical approaches suffer from system noise, dead angles, the need for initialization, and limitations in the sensor array's layout. We solve these issues with deep neural networks to learn the regression from the simulation flux values to the LC coils' 3D configuration, which can be applied to the actual LC coils at any location and orientation within the capture volume. To cope with the influence of system noise and the dead-angle limitation caused by the characteristics of the hardware and sensing principle, we propose a structure-aware temporal bilateral filter for reconstructing motion sequences. Our method can track various movements, including fingers that manipulate objects, beetles that move inside a vivarium with leaves and soil, and the flow of opaque fluid. Since no power supply is needed for the lightweight wireless markers, our method can robustly track movements for a very long time, making it suitable for various types of observations whose tracking is difficult with existing motion-tracking systems. Furthermore, the flexibility of the flux sensor layout allows users to reconfigure it based on their own applications, thus making our approach suitable for a variety of virtual reality applications.

Accurate drone positioning is challenging because pilots only have a limited position and direction perception of a flying drone from their perspective. This makes conventional joystick-based speed control inaccurate and more complicated and significantly degrades piloting performance. We propose PinpointFly, an egocentric drone interface that allows pilots to arbitrarily position and rotate a drone using position-control direct interactions on a see-through mobile AR where the drone position and direction are visualized with a virtual cast shadow (i.e., the drone’s orthogonal projection onto the floor). Pilots can point to the next position or draw the drone’s flight trajectory by manipulating the virtual cast shadow and the direction/height slider bar on the touchscreen. We design and implement a prototype of PinpointFly for indoor and visual line of sight scenarios, which are comprised of real-time and predefined motion-control techniques. We conduct two user studies with simple positioning and inspection tasks. Our results demonstrate that PinpointFly makes the drone positioning and inspection operations faster, more accurate, simpler and fewer workload than a conventional joystick interface with a speed-control method.

We propose TiltChair, an actuated office chair that physically manipulates the user’s posture by actively inclining the chair’s seat to address problems associated with prolonged sitting. The system controls the inclination angle and motion speed with the aim of achieving manipulative but unobtrusive posture guidance. To demonstrate its potential, we first built a prototype of TiltChair with a seat that could be tilted by pneumatic control. We then investigated the effects of the seat’s inclination angle and motions on task performance and overall sitting experience through two experiments. The results show that the inclination angle mainly affects the difficulty of maintaining one’s posture, while the motion speed affected the conspicuousness and subjective acceptability of the motion. However, these seating conditions did not affect objective task performance. Based on these results, we propose a design space for facilitating effective seat-inclination behavior using the three dimensions of angle, speed, and continuity. Furthermore, we discuss promising applications.

Although children’s behavioral and mental problems are generally diagnosed in clinical settings, the prediction and awareness of children’s mental wellness in daily settings are getting increased attention. Toy blocks are both accessible in most children’s daily lives and provide physicality as a unique non-verbal channel to express their inner world. In this paper, we propose a toy block approach for predicting a range of behavior problems in young children (4-6 years old) measured by the Child Behavior Checklist (CBCL). We defined and classified a set of quantitative play actions from IMU-embedded toy blocks. Play data collected from 78 preschoolers revealed that specific play actions and patterns indicate total problems, internalizing problems, and aggressive behavior in children. The results align with our qualitative observations, and suggest the potential of predicting the clinical behavior problems of children based on short free-play sessions with sensor-embedded toy blocks.

We propose ModularHMD, a new mobile head-mounted display concept, which adopts a modular mechanism and allows a user to perform ad-hoc peripheral interaction with real-world devices or people during VR experiences. ModularHMD is comprised of a central HMD and three removable module devices installed in the periphery of the HMD cowl. Each module has four main states: occluding, extended VR view, video see-through (VST), and removed/reused. Among different combinations of module states, a user can quickly setup the necessary HMD forms, functions, and real-world visions for ad-hoc peripheral interactions without removing the headset. For instance, an HMD user can see her surroundings by switching a module into the VST mode. She can also physically remove a module to obtain direct peripheral visions of the real world. The removed module can be reused as an instant interaction device (e.g., touch keyboards) for subsequent peripheral interactions. Users can end the peripheral interaction and revert to a full VR experience by re-mounting the module. We design ModularHMD’s configuration and peripheral interactions with real-world objects and people. We also implement a proof-of-concept prototype of ModularHMD to validate its interactions capabilities through a user study. Results show that ModularHMD is an effective solution that enables both immersive VR and ad-hoc peripheral interactions.

We explore BouncyScreen, an actuated 1D display system that enriches indirect interaction with a virtual object by pseudo-haptic feedback mechanics enhanced through the screen's physical movements. We configured a proof-of-concept prototype of BouncyScreen with a flat-screen mounted on a mobile robot. When the user manipulates a virtual object using virtual reality (VR) controllers, the screen moves in accordance with the virtual object. We conducted psychophysical studies to examine how BouncyScreen's physical movements would affect users' pseudo-haptic perceptions and interaction experiences. Our preliminary study using a weight discrimination task for object pushing interaction showed that BouncyScreen offers identical pseudo-force feedback to the vision-based pseudo-haptic technique. We conducted a follow-up psychophysical study using a weight magnitude estimation task for object pushing and bumping interactions. The results reveal that a users' perceived weight magnitude is enhanced by the screen's physical motion under different characteristics depending on interaction styles (i.e., pushing and bumping). Our study also confirmed that the screen's synchronous physical motions significantly enhance the reality of the interaction and the sense of presence. We close this paper with some applications and use suggestions for BouncyScreen in future HMD-free flat-screen 3D user interface systems.

Motion tracking systems with viewpoint concerns or whose marker data include unreliable states have proven difficult to use despite many impactful benefits. We propose a technique inspired by active vision and using a customized hill-climbing approach to control a robot-sensor setup and apply it to a magnetic induction system capable of occlusion-free motion tracking. Our solution reduces the impact of displacement and orientation issues for markers which inherently present a dead-angle range that disturbs usability and accuracy. The resulting interface is successful in stabilizing previously unexploitable data while preventing sub-optimal states for up to hundreds of occurrences per recording and featuring an approximate 40% decrease in tracking error.

Drilling teeth is a significant technique for dental learners. However, the existing VR tooth drill training simulators are physically large and costly, which cannot be used at home or classroom for a private study. This work presents a novel low-cost mobile VR dental simulator using off-the-shelf devices, including a mobile HMD and a stylus smartphone. In this system, a 3D-printed physical teeth prop is placed on an EMR stylus smartphone where its stylus tracks the tip position of a physical drill. Unlike existing solutions using haptic/force devices, our approach involving physical contact between the prop and drill tip enables the user’s natural teeth hardness sensation. The use of smartphone stylus would enable significantly more accurate drill position sensing around the teeth than HMD’s accompanying controllers. We also developed VR software to simulate tooth drilling on this setup. This demo will show how our new mobile simulator offers a realistic feeling of drilling teeth.

Head-mounted displays (HMDs) increase immersion into virtual worlds. The problem is that this limits headset users' awareness of bystanders: headset users cannot attend to bystanders' presence and activities. We call this the HMD boundary. We explore how to make the HMD boundary permeable by comparing different ways of providing informal awareness cues to the headset user about bystanders. We adapted and implemented three visualization techniques (Avatar View, Radar and Presence++) that share bystanders' location and orientation with headset users. We conducted a hybrid user and simulation study with three different types of VR content (high, medium, low interactivity) with twenty participants to compare how these visualization techniques allow people to maintain an awareness of bystanders, and how they affect immersion (compared to a baseline condition). Our study reveals that a see-through avatar representation of bystanders was effective, but led to slightly reduced immersion in the VR content. Based on our findings, we discuss how future awareness visualization techniques can be designed to mitigate the reduction of immersion for the headset user.

We focus on the problem of simulating the haptic infrastructure of a virtual environment (i.e. walls, doors). Our approach relies on multiple ZoomWalls---autonomous robotic encounter-type haptic wall-shaped props---that coordinate to provide haptic feedback for room-scale virtual reality. Based on a user's movement through the physical space, ZoomWall props are coordinated through a predict-and-dispatch architecture to provide just-in-time haptic feedback for objects the user is about to touch. To refine our system, we conducted simulation studies of different prediction algorithms, which helped us to refine our algorithmic approach to realize the physical ZoomWall prototype. Finally, we evaluated our system through a user experience study, which showed that participants found that ZoomWalls increased their sense of presence in the VR environment. ZoomWalls represents an instance of autonomous mobile reusable props, which we view as an important design direction for haptics in VR.

We propose ShearSheet, a low-cost, feasible, and simple DIY method that enables tangential (shear) force input on a touchscreen using a rubber-mounted slim transparent sheet. The sheet has tiny conductive material(s) attached to specific positions on the underside so that displacement of the sheet is recognized as touch input(s). This allows the system to distinguish between normal touch input and shear force input depending on whether the sheet moves or not, without requiring any external sensors or power. We first introduce ShearSheet's simple implementation for a smartphone and then discuss several promising interaction techniques that augment shear interaction with one- and two-finger operations. We demonstrate the effectiveness of ShearSheet's smooth transition between position- and rate-based control through a controlled user study using simple scrolling tasks. The results confirm that ShearSheet offers improved performance such as fewer operations and higher subjective preference, suggesting its substantial benefits and potential applications.

We propose Third-Person Piloting, a novel drone manipulation interface that increases situational awareness using an interactive third-person perspective from a second, spatially coupled drone. The pilot uses a controller with a manipulatable miniature drone. Our algorithm understands the relationship between the pilot's eye position and the miniature drone and ensures that the same spatial relationship is maintained between the two real drones in the sky. This allows the pilot to obtain various third-person perspectives by changing the orientation of the miniature drone while maintaining standard primary drone control using the conventional controller. We design and implement a working prototype with programmable drones and propose several representative operation scenarios. We gather user feedback to obtain the initial insights of our interface design from novices, advanced beginners, and experts. Our result suggests that the interactive third-person perspective provided by the second drone offers sufficient potential for increasing situational awareness and supporting their primary drone operations.

In this work, we were interested in using smartphone touchscreen keyboard for text typing in virtual environments (VEs) with head-mounted display. We carried out an experiment comparing the smartphone to the ordinary devices: gamepad and HTC Vive Controllers. We represented the touchscreen keyboard in the VE with a virtual interface and the fingertips with tracked green circles. A confirm-on-release paradigm was employed for text typing. Results showed that the smartphone did not fully outperformed the other devices. However, unlike the other devices, smartphones users tended to correct progressively their error while typing thanks to their familiarity with the device.

We propose an approach for flexibly scalable workspace that changes viewer's perceived size of the workspace by simulating a virtually movable wall on the real wall. We conducted an experiment comparing viewers depth estimates between extended workspace virtually and physically. Besides, we investigated walls motion effects; discrete wall motion or continuous wall motion. Distances were estimated verbally and using perceptual matching method. Verbal depth estimates were equivalent in the virtual extended workspace and the physical one, with a high accuracy (estimation errors were less than 5%). However, perceptual matching estimates were signicantly different between virtual and physical scaling. Overall, they were slightly overestimated. No difference of the motion effect was found. Our results clearly suggest the potential of the workspace scaling by just giving such visual impression of extended workspace using perspective projection technique.

We present “Be Bait!”, a unique virtual fishing experience that allows the user to become a bait by lying on the hammocks, instead of holding a fishing rod. We implement the hammock-based locomotion method with haptic feedback mechanisms. In our demonstration, users can enjoy exploring a virtual underwater world and fighting with fishes in direct and intuitive ways.

This paper proposes an interactive rear-projection tabletop glasses-free 3D display using a novel wireless magnetic motion capture system. Our tracking system employs an electromagnetic field generator and 16 magnetic detectors. It detects the 3D positions of several small markers in the generated electromagnetic field. The detectors are arranged in a ring around the rim of the conical screen of the 3D display to avoid occluding the reproduced 360-degree-viewable 3D images. For the proposed configuration, our experimental results reveal that a toroidal area around a hemispherical 3D image display area allows the 3D position to be measured with sufficient accuracy. We implemented an application to demonstrate real-time interaction with virtual 3D objects displayed on the table using markers attached to a physical object like a stick or finger.

Abstract As the range of handheld, mobile and desktop devices expands and worldwide demand for collaborative application tools increases, there is a growing need for higher speed impromptu cross-device application sharing to keep up with workplace requirements for on-site or remote collaborations. To address this, we have developed CamCutter, a cross-device interaction technique enabling a user to quickly select and share an application running on another screen using the camera of a handheld device. This technique can accurately identify the targeted application on a display using our adapted computer vision algorithm, system architecture and software implementation, allowing impromptu real-time and synchronized application sharing between devices. For desktop and meeting room set-ups, we performed a technical evaluation, measuring accuracy and speed of migration. For a single-user reading task and a collaborative composition task, we carried out a user study comparing our technique with commercial screen sharing applications. The results of this study showed both higher performance and preference for our system. Finally, we discuss CamCutter’s limitations and present insights for future vision-based cross-device application sharing.

This paper describes an unsupervised adaptation method of deep neural networks (DNNs) regarding discriminative sound source localization (SSL). DNNs-based SSL and its unsupervised adaptation fail under different conditions from those during training. The estimations sometimes include incoherent unpredictable errors due to the NN's non-linearity. We propose an eliminative posterior probability constraint using a model-based SSL for unsupervised DNNs adaptation. This constraint forces the probability of “less possible candidates” to become zero to eliminate incoherent errors. The candidates are indicated by a model-based SSL method because it can estimate the azimuth of the sound source with moderate accuracy and explicit reasoning. As a result, the localization performance of adapted DNNs improved more than that of model-based SSL. Experimental results showed that our method improved localization correctness of 1D azimuth and 3D regions by a maximum of 13.3 and 5.9 points compared with the model-based SSL.

Digital photos and their metadata get explosively growth, requiring better support to browsing them. We propose a dynamic and flexible visualization of digital photos and their metadata. A prototype is designed and implemented based on the algorithm of D-FLIP, our previous work for flexibly displaying a large photo collection. We design various dynamic photo visualizations with up to four-dimensional meta-information by using Bertin's visual variables and three-dimensional representation. It allows users to dynamically and effectively manage photo visualizations by selecting meta-information of user's current needs and interest.

Large interactive tabletops are beneficial for various tasks involving exploration and visualization, but regions of the screen far from users can be difficult to reach. We propose AdapTable; a concept and prototype of a flexible multi-display tabletop that can physically reconfigure its layout, allowing for interaction with difficult-to-reach regions. We conducted a design study where we found users preferred to change screen layouts for full-screen interaction, motivated by reduced physical demands and frustration. We then prototyped AdapTable using four actuated tabletop displays each propelled by a mobile robot, and a touch menu was chosen to control the layout. Finally, we conducted a user study to evaluate how well AdapTable addresses the reaching problem compared with a conventional panning technique. Our findings show that AdapTable provides a more efficient method for complex full-screen interaction.

We present a concept-based world building approach, realized in a system called VR Safari Park, which allows users to rapidly create and manipulate a world simulation. Conventional world building tools focus on the manipulation and arrangement of entities to set up the simulation, which is time consuming as it requires frequent view and entity manipulations. Our approach focuses on a far simpler mechanic, where users add virtual blocks which represent world entities (e.g. animals, terrain, weather, etc.) to a World Tree, which represents the simulation. In so doing, the World Tree provides a quick overview of the simulation, and users can easily set up scenarios in the simulation without having to manually perform fine-grain manipulations on world entities. A preliminary user study found that the proposed interface is effective and usable for novice users without prior immersive VR experience.

We are pursuing a vision of reactive interior spaces that are aware of people's actions and transform according to changing needs. We envision furniture and walls that act as interactive displays and that shapeshift to the correct physical form, and the appropriate interactive visual content and modality. This paper briefly describes our proposal based on our recent efforts on realizing this vision.

Many motion tracking systems require solving inverse problem to compute the tracking result from original sensor measurements. For real-time motion tracking, such typical solutions as the Gauss-Newton method for solving their inverse problems need an initial value to optimize the cost function through iterations. A powerful initializer is crucial to generate a proper initial value for every time instance and, for achieving continuous accurate tracking without errors and rapid tracking recovery even when it is temporally interrupted. An improper initial value easily causes optimization divergence, and cannot always lead to reasonable solutions. Therefore, we propose a new initializer based on random-forest to obtain proper initial values for efficient real-time inverse problem computation. Our method trains a random-forest model with varied massive inputs and corresponding outputs and uses it as an initializer for runtime optimization. As an instance, we apply our initializer to IM3D[1], which is a real-time magnetic 3D motion tracking system with multiple tiny, identifiable, wireless, occlusion-free passive markers (LC coils).

We present a concept of dynamically adaptive workspace using MovemenTable, autonomously moving interactive digital table. In this demo, we introduce two interaction scenarios, arranging workspace anywhere and floating interactive surface on large floor screen. In the first scenario, the interactive digital table is automatically arranged anywhere in a room to keep user's workspace efficient and optimal by self-actuated movement of MovemenTable. The second scenario is more advanced one with collaborative use of large floor screen system where the MovemenTable works as a floating interactive surface that allows local data exploration of the global information on the floor screen.

We conducted a user study evaluating five selection techniques for augmented reality in optical see-through head-mounted displays (OST-HMDs). The techniques we studied aim at supporting mobile usage scenarios where the devices do not need external tracking tools or special environments, and therefore we selected techniques that rely solely on tracking technologies built into conventional commercially available OST-HMDs [i.e. gesture trackers, gaze tracking and inertial measurement units (IMUs)]. While two techniques are based on raycasting using built-in IMU sensing, three techniques are based on a hand-controlled 3D cursor using gestural tracking. We compared these techniques in an experiment with 12 participants. Our results show that raycasting using head orientation (i.e. IMU on the headset) was the fastest, fatigueless and the most preferable technique to select spatially arranged objects. We discuss the implications of our findings for design of interaction techniques in mobile OST-HMDs.

Wall displays almost universally assume a mostly flat and static shape. We ask two questions: Would people choose a flat display for a given interaction scenario and, if not, what are the display shapes they actually prefer? We conducted a design study around these two questions. Our results show that participants designed different screen shapes that varied based upon peoples' distance from the display and the content shown. Shapes ranged primarily between flat, separated, concave, L-shape and convex displays. Based on our findings, we designed a dynamic display that changes to these and other configurations. Shape-shifting is controlled either by explicit interaction (where the display responds to hand gestures) or implicitly (where the display infers a shape based both on its content and the sensed positions of the people around it). Overall, we contribute: a study that motivates research on shape-shifting wall displays, and a shape-shifting display system that responds to explicit and implicit controls to match particular activities.

We describe our approach that derives reliable 6-DOF information including the translation and the rotation of a rigid marker in a 3D space from a set of insufficient 5-DOF measurements. As a practical example, we carefully constructed a prototype and its design and evaluated it in our 3D dexterous motion-tracking system, IM6D, which is our novel real-time magnetic 3D motion-tracking system that uses multiple identifiable, tiny, lightweight, wireless, and occlusion-free markers. The system contains two key technologies; a 6-DOF computation algorithm and a marker design for 6D marker. The 6-DOF computation algorithm computes the result of complete 6-DOF information including translation and rotation in 3D space for a single rigid marker that consists of three LC coils. We propose several possible approaches for implementation, including geometric, matrix-based kinematics, and computational approaches. In addition, we introduce workflow to find an optimal marker design for the system to achieve the best compromise between its smallness and accuracy based on the tracking principle. We experimentally compare the performances of some typical marker prototypes with different layouts of LC coils. Finally, we also show another experimental result to prove the effectiveness of the results from the solutions in these two problems.

This paper explores the Boundless Scroll interface that extends the motor space for finger drag motions into the off-screen space using a tracking system around the screen. Seamlessly extended motor spaces enable continuous scroll actions across the on- and off-screens, reducing the chance of clutching and preserving content visibility during long-distance and smooth scrolling actions. We empirically evaluate our new technique, the Boundless Scroll and compare it to ordinary scroll interfaces (Drag and Flick) in two dominant screen devices. The first screen condition uses a large flat touch screen where users manipulate a non-full-screen window while using the surrounding space as an extended motor space. The second condition is a mobile device to examine mid-air finger scroll motions in the off-screen space using a 3D motion sensor. An empirically controlled study demonstrates our Boundless Scroll's many benefits, such as fewer clutching actions, occlusion-less content observations, and efficient and effortless off-screen acquisitions and also provided further design factors, implementations of the Boundless Scroll, and around-device interfaces.

We propose Coupled-clay, a bi-directional 3D collaborative interactive environment that supports the 3D modeling work between groups of users at remote locations. Coupled-clay consists of two network-connected workspaces, the Physical Interaction Space and the Virtual Interaction Space. The physical interaction space allows a user to directly manipulate a physical object whose shape and position are precisely tracked. This tracked 3D information is transferred to the virtual interaction space in real time. The virtual interaction space is made of an interactive multi-user stereoscopic 3D tabletop, or other 3D displays with adequate interaction device. The users at the virtual interaction space observe the virtual 3D object which corresponds to the physical object and manipulate its geometrical attributes (e.g., translation, rotation and scaling). Additionally, they can control the graphical attributes of the virtual object such as color and texture. Information about changes in geometrical and graphical attributes are sent back to the physical interaction space in real time and reflected to the object in the physical interaction space by a robotic arm and a top-mounted projector. Coupled-clay can be used to remotely collaborate on 3D modeling tasks such as between a skilled designer and novice learners. This paper details our Coupled-clay implementation and presents its interaction capabilities.

We propose IM6D, a novel real-time magnetic motion-tracking system using multiple identifiable, tiny, lightweight, wireless and occlusion-free markers. It provides reasonable accuracy and update rates and an appropriate working space for dexterous 3D interaction. Our system follows a novel electromagnetic induction principle to externally excite wireless LC coils and uses an externally located pickup coil array to track each of the LC coils with 5-DOF. We apply this principle to design a practical motion-tracking system using multiple markers with 6-DOF and to achieve reliable tracking with reasonable speed. We also solved the principle's inherent dead-angle problem. Based on this method, we simulated the configuration of parameters for designing a system with scalability for dexterous 3D motion. We implemented an actual system and applied a parallel computation structure to increase the tracking speed. We also built some examples to show how well our system works for actual situations.

We present our study of Calamaro: a robotic platform designed to investigate the impact of emotive motion on people, and its deployment in an extensive field study in a busy public space at the University of Calgary campus. Our paper details the design of the Calamaro robot, the field study conducted with it, including hundreds of observers and 88 participants, and our quantitative and qualitative findings. The paper provides a thorough discussion of the implications of our results on the design of robotic emotive motions, and reflections on the deployment of robotic interfaces in field studies.

We propose a furry and scalable multi-touch display called the “FuSA2Touch Display.” The furry type of tactile sensation of this surface affords various interactions such as stroking or clawing. The system utilizes plastic fiber optic bundles to realize a furry-type texture. The system can show visual feedback by projection and detects multi-touch input. We implemented a 24-inch display, and found that our implemented display encourages users to interact with it in various actions.

We establish that two-part models of pointing performance (Welford’s model) describe pointing on a computer display significantly better than traditional one-part models (Fitts’s Law). We explore the space of pointing models and describe how independent contributions of movement amplitude and target width to pointing time can be captured in a parameter k . Through a reanalysis of data from related work we demonstrate that one-part formulations are fragile in describing pointing performance, and that this fragility is present for various devices and techniques. We show that this same data can be significantly better described using two-part models. Finally, we demonstrate through further analysis of previous work and new experimental data that k increases linearly with gain. Our primary contribution is the demonstration that Fitts’s Law is more limited in applicability than previously appreciated, and that more robust models, such as Welford’s formulation, should be adopted in many cases of practical interest.

We propose a system that allows multiple groups of users at remote locations to naturally share a 3D image of real objects. All users can interactively observe a 3D stereoscopic image without distortion from their own viewpoints. The system basically consists of a combination of systems: imaging and display. The imaging system captures the real object's images from certain number of viewpoints which are used to generate stereoscopic images from arbitrary viewpoints based on an image-based rendering technique implemented on GPU. The display system interactively generates a 3D virtual image of the real object based on the IllusionHole technique. Users at one place just put the real object on their imaging system to capture a set of its images from sparse viewpoints around it; other groups of multiple users at remote places connected by networks observe its virtual image from arbitrary viewpoints on their display systems.

We propose a system that allows multiple groups of users at remote locations to naturally share a 3D image of real objects. All users can interactively observe a 3D stereoscopic image without distortion from their own viewpoints. The system basically consists of a combination of subsystems: imaging and display. The imaging subsystem generates images of the real object observed from arbitrary viewpoints based on image-based rendering technique implemented on GPU. The display system generates a 3D virtual image of the real object to be interactively observed by multiple people around the tabletop display. People at one place just put the real object on their imaging system to capture a set of its images from sparse viewpoints around it; other groups of multiple persons at remote places connected by networks observe its virtual image from arbitrary viewpoints, as if there is a copy of the real object, on their display systems. This paper describes details of the system configurations and algorithms. Then discussions are made based on experimental results.

"MADO Interface" is a tangible user interface consisting of a compact touch-screen display and physical blocks. "MADO" means "window" in Japanese, and MADO Interface is utilized as the real window into the virtual world. Users construct a physical object by simply combining electrical blocks. Then, by connecting MADO Interface to the physical object, they can watch the virtual model corresponding to the physical block configuration (shape, color, etc.) The size and the viewpoint of the virtual model seen by the user depend on the position of MADO Interface, maintaining the consistency between the physical and virtual worlds. In addition, users can interact with the virtual model by touching the display on MADO Interface. These features enable users to explore the virtual world intuitively and powerfully.

We propose a novel interface called Tearable that allows users to continuously experience the real sense of tearing paper. To provide such a real sense, we measured the actual vibration data of tearing a piece of real paper and analyzed them. Based on this data, we utilized hook-and-loop fasteners and a DC motor for representing the sense of tearing. We compared the force given by Tearable with that by a piece of real paper and recommended its reproducibility and usability. In addition, we evaluated Tearable with questionnaires after user experiences.

Abstract We solve the light transport problem by introducing a novel unbiased Monte Carlo algorithm called replica exchange light transport, inspired by the replica exchange Monte Carlo method in the fields of computational physics and statistical information processing. The replica exchange Monte Carlo method is a sampling technique whose operation resembles simulated annealing in optimization algorithms using a set of sampling distributions. We apply it to the solution of light transport integration by extending the probability density function of an integrand of the integration to a set of distributions. That set of distributions is composed of combinations of the path densities of different path generation types: uniform distributions in the integral domain, explicit and implicit paths in light (particle/photon) tracing, indirect paths in bidirectional path tracing, explicit and implicit paths in path tracing, and implicit caustics paths seen through specular surfaces including the delta function in path tracing. The replica‐exchange light transport algorithm generates a sequence of path samples from each distribution and samples the simultaneous distribution of those distributions as a stationary distribution by using the Markov chain Monte Carlo method. Then the algorithm combines the obtained path samples from each distribution using multiple importance sampling. We compare the images generated with our algorithm to those generated with bidirectional path tracing and Metropolis light transport based on the primary sample space. Our proposing algorithm has better convergence property than bidirectional path tracing and the Metropolis light transport, and it is easy to implement by extending the Metropolis light transport.

We present a novel approach for interactive multimedia content creation that establishes an interactive environment in cyberspace in which users interact with autonomous agents generated from video images of real-world creatures. Each agent has autonomy, personality traits, and behaviors that reflect the results of various interactions determined by an emotional model with fuzzy logic. After an agent's behavior is determined, a sequence of video images that best match the determined behavior is retrieved from the database in which a variety of video image sequences of the real creature's behaviors are stored. The retrieved images are successively displayed on the cyberspace to make it responsive. Thus the autonomous agent behaves continuously. In addition, an explicit sketch-based method directly initiate the reactive behavior of the agent without involving the emotional process. This paper describes the algorithm that establishes such an interactive system. First, an image processing algorithm to generate a video database is described. Then the process of behavior generation using emotional models and sketch-based instruction are introduced. Finally, two application examples are demonstrated: video agents with humans and goldfish.

Multi-display environments compose displays that can be at different locations from and different angles to the user; as a result, it can become very difficult to manage windows, read text, and manipulate objects. We investigate the idea of perspective as a way to solve these problems in multi-display environments. We first identify basic display and control factors that are affected by perspective, such as visibility, fracture, and sharing. We then present the design and implementation of E-conic, a multi-display multi-user environment that uses location data about displays and users to dynamically correct perspective. We carried out a controlled experiment to test the benefits of perspective correction in basic interaction tasks like targeting, steering, aligning, pattern-matching and reading. Our results show that perspective correction significantly and substantially improves user performance in all these tasks.

In this study, we discuss a display table suitable for collaborative work environments for medical use. Using an interactive stereoscopic display system allows simultaneous observation of accurate stereoscopic images generated from volume data. We further investigate all requirements for design guidelines of the display system, including hardware configuration, rendering software to generate the stereoscopic images, and the interface system to operate the displayed images

We have proposed the IllusionHole system, which allows three or more people to simultaneously observe stereoscopic images from dynamically changing individual viewpoints. With a simple configuration, this system provides intelligible 3D stereoscopic images free of flicker and distortion. Based on the IllusionHole concept, we have built a prototype system using two liquid crystal projectors and polarizing filters. This paper describes the details of this prototype. We investigate how the image formation position of the stereographic image is affected by sources of errors such as measurements of the user's interpupillary distance and observing position. We also measure and discuss the variation of brightness and hue with viewing direction.

This paper details the design and evaluation of the Delphian Desktop, a mechanism for online spatial prediction of cursor movements in a Windows-Icons-Menus-Pointers (WIMP) environment. Interaction with WIMP-based interfaces often becomes a spatially challenging task when the physical interaction mediators are the common mouse and a high resolution, physically large display screen. These spatial challenges are especially evident in overly crowded Windows desktops. The Delphian Desktop integrates simple yet effective predictive spatial tracking and selection paradigms into ordinary WIMP environments in order to simplify and ease pointing tasks. Predictions are calculated by tracking cursor movements and estimating spatial intentions using a computationally inexpensive online algorithm based on estimating the movement direction and peak velocity. In testing the Delphian Desktop effectively shortened pointing time to faraway icons, and reduced the overall physical distance the mouse (and user hand) had to mechanically traverse.

This paper proposes a virtual chopsticks system using force feedback and object deformation with FEM (finite element model). The force feedback model is established by using a leverage based on the correct chopsticks handling manner, and the force is applied to the index and middle finger. The object deformation is obtained in real-time by calculating inverse stiffness matrix beforehand. We performed experiments to compare the hardness of virtual objects. As a result, we found that a recognition rate of almost 100% can be achieved between virtual objects where the logarithmic difference in hardness is 0.4 or more, while lower recognition rates are obtained when the difference in hardness is smaller than this.

We present a novel method for interactive retrieval of 3D shapes using ysical objects. Our method is based on simple ysical 3D interaction with a set of tangible blocks. As the user connects blocks, the system automatically recognizes the shape of the constructed ysical structure and picks similar 3D shape models from a preset model database, in real time. Our system fully supports interactive retrieval of 3D shape models in an extremely simple fashion, which is completely non-verbal and cross-cultural. These advantages make it an ideal interface for inexperienced users, previously barred from many applications that include 3D shape retrieval tasks.

Start of the above-titled section of the conference proceedings record.

We propose a method to extract motion individuality from the form of a sport expert and to apply it to figures that have different segment lengths. For this purpose, a standard motion is established by removing the subjective motion from an actor who conducts a sport skillfully, and the surface features (angular velocity and torque of the joint) are extracted for each joint angle difference in each animation frame between the original motion of the sports expert and the standard motion. The surface features and relative timing in the deep structure are transmitted to the characters having different segment lengths by applying them to a standard motion of the characters, which is also made by removing subjective motion in advance.

In this paper, we propose a new technique based on recent neuroimaging studies as a tool for the assessment of interactive systems. For this purpose, we analyze the mental process that takes place while human subjects learn to use new tools by using two different approaches. One is an experiment on task performance based on the conventional direct testing method of a user interface, and the other is an indirect method based on recent neuroimaging studies that indirectly estimate the process of interaction through the observation of the human brain activities. The results obtained from the direct experiment on performance evaluation are compared with those from the indirect analysis of the human brain activity, which is measured by a non-invasive neuroimaging measuring method. The process of acquisition of internal models while subjects learn to use new tools is also discussed.

We propose a real-time interactive system that enables users to generate, manipulate and edit the shape model of a tree based on growth simulation by directly indicating its global and spatial information. For this purpose, three-dimensional (3D) spatial information is introduced to the well-known L-system as an attribute of the growth simulation. Moreover, we propose an efficient data structure of L-strings in order to speed up the process.

Assessments of spatial, constructional ability are used widely in cognitive research and in clinical diagnosis of disease or injury. Some believe that three-dimensional (3D) forms of these assessments would be particularly sensitive, but difficulties with consistency in administration and scoring have limited their use. We describe Cognitive Cubes, a novel computerized tool for 3D constructional assessment that increases consistency and promises improvements in flexibility, reliability, sensitivity and control. Cognitive Cubes makes use of ActiveCube, a novel tangible user interface for describing 3D shape. In testing, Cognitive Cubes was sensitive to differences in cognitive ability and task, and correlated well to a standard paper-and-pencil 3D spatial assessment

This paper describes a method for the consolidated manipulation of virtual and real objects using a "magnetic metaphor". The method reduces the behavioral differences between the virtual and real objects. A limited number of physical laws are selected and simulated for the virtual objects; at the same time, limitations are placed on the physical laws for the real objects. Accordingly, a compromise can be found between the physical laws that operate on virtual objects and those which operate on real objects. Therefore, a system with this method enables a user to manipulate virtual and real objects in a similar manner by expecting the same responses and behaviors according to the same physical laws. Experimental results show that the proposed method improves the task performance in the manipulation of virtual and real objects existing in the same environment simultaneously.

The paper describes a retrieval technique for video databases that is based on gestures. The proposed technique gives users a method to create various video content by intuitive interaction. This database system can extract feature parameters from video, perform automatic indexing, retrieve an appropriate result, and replace a presented object with the retrieval result in real time. Furthermore, we implement the fundamental functions of the proposal database. The retrieval accuracy and computational cost of the system are also described.

An ideal stereoscopic display system for multiple users is proposed. It allows three or more people to simultaneously observe individual stereoscopic image pairs from their own viewpoints. The system tracks the head positions of all of the users and generates distortion-free images for each eye of each user. The system consists of a normal display and a display mask, which has a hole in its center. The display mask is placed over the display surface at a suitable distance from it. By controlling the position of the image drawing area for each user according to the corresponding user's viewpoint, each user can observe the stereoscopic image pairs shown in an individual area of the display system with shutter glasses. On the other hand, no user is able to see the image drawing areas of the other users because these areas are adequately occluded by the display mask. Accordingly, the display system can simultaneously provide intelligible 3D stereoscopic images for three or more moving observers without flicker or distortion.

This paper presents a method of interactively generating natural hand gesture animation using reduced dimensionality from multiple captured data sequences of finger motions conducting specific tasks. This method is achieved by introducing an estimation with multiple regression analysis. Even when the skeletal structure of the user who inputs the motion is different from that of the shape model in the computer, the motion that a user imagines is generated. Experimental results obtained from the interface applied to virtual object manipulation showed that the proposed method generates animation naturally, just as users would expect. This method enables us to make input devices that require minimal user training and computer calibration, and helps to make the user interface intuitive and easy to use.

Methods of interactive image generation using processed video contents, and its 3D model capture are proposed; we mainly stress the key technique of the system and the implementation of moving 3D object extraction. In order to reflect a user's intention for generating new multimedia contents, we discuss the methods of storage, retrieval, and presentation. This system is an example of the novel creation of multimedia contents by using the technologies of computer vision and virtual reality. The user is able to generate novel multimedia contents without shape modeling by a human. The proposed low-cost extraction and interaction algorithm can track objects at 8 frames/sec with sufficient accuracy.

ActiveCube is a bi-directional user interface that supports a person's creative work through constructing and interacting with 3D virtual objects by using actual physical cubes. With this flexible interface, the user can easily construct various 3D structures in a virtual environment by simply combining the cubes. All of the faces of the cubes are the same and can be connected to any face of any other cube. A computer recognizes the constructed 3D structure in real time, so consistency is always maintained between the real environment and its corresponding representation in a virtual environment. Each cube is equipped with a processor for autonomous simulation and communication functions between cubes. A cube also utilizes a sensor for detecting the operational intention of the user and/or a display/actuator used as an output channel. Users can interact with ActiveCube in various ways by using the sensors, and the response to the interaction is also shown by displays/actuators installed on the cube. Each ActiveCube is equipped with both input and output devices, which makes the interface intuitive and helps to clarify the causal relationship between the input of the user's operational intention and the output of simulation results.

This paper describes an interactive manipulation technique for objects extracted from video images by an efficient algorithm in order to produce multimedia contents. An object contained in a video image must be able to be extracted at a reasonable speed and a moving object able to be tracked with sufficient accuracy. Moreover, real object interactions that meet such criteria must be established using techniques for manipulating objects in virtual worlds. The proposed technique has potential to generate novel multimedia contents without using shape modeling. An interactive digital fishtank is described as one example application of the proposed technique. In this application, the user is able to obtain various video sequences of fishtanks from the original video sequence.

A technique is proposed for object manipulation with a virtual tool using multiple exact interactions. Exact test is introduced that uses real-time collision detection for both hand-and-tool and tool-and-object interactions. Chopsticks are adopted for one of the trials of our technique; although they have a very simple shape, they do have multiple functions. Here, a virtual object is manipulated by the virtual chopsticks, which are used by the motion of a user's hand captured by a hand gesture input device. Exact interaction is applied between the hand and chopsticks based on correct table manners. Experimental results demonstrate the effectiveness of the proposed multiple exact interactions, especially for precise object alignment tasks.

Describes an interactive manipulation technique for objects in video images extracted by an efficient algorithm. An object contained in a video image has to be able to be extracted at a reasonable speed and a moving object is able to be tracked within a sufficient accuracy. Moreover, any interaction method for real objects with such functions has to be established using a technique for manipulating objects in virtual worlds. An interactive digital fishtank is described as one example application of the proposed technique. In this application, the user is able to obtain various video sequences of fishtanks from any video sequence.

We propose a manipulation technique to intuitively control the "bird's eye" overview displav of an entire large-scale virtual environment in a display system that present a user with both overviews (global views) and a life-size virtual environment (local view) simultaneously.It enables efficient navigation even in enormous and complicated environments using both global and local views.The motion of the bird's eye viewpoint interlocks with the relative motion of the user's viewpoint and his/her hand, therefore, the user can control the "bird's eve" viewpoint by intuitive manipulation.Sophisticated display techniques are obtained based on the proposed method by introducing some constraints on the parameters of the "bird's eye" viewpoint.Experimental results show a combination of the bird's eye overview image and the life-size local image is displayed to the user by reflecting his/her intuitive manipulation.1

We propose a manipulation environment with virtual and real objects by providing a consolidated manipulation manner for both objects.This approach assists the user and improves the task performance of object manipulation (e.g., speed and accuracy).A limited kinds of physical laws are selected and simulated for virtual objects; at the same time, limitations on the physical laws for real objects are introduced.Accordingly, we reached a compromise between the physical laws that operate on both virtual and real objects.The proposed environment enables a user to manipulate virtual and real objects in the same manner by expecting the same response according to the same physical laws.In the environment, real objects are treated as "substantial" as they are, without using insubstantial images taken by cameras or graphical images generated from geometric shape representations of objects.It is possible to bring a real object into a virtual world on demand, according to the user's irregular and sudden requirements, without using troublesome procedures to reconstruct the object shapes.The configuration of an object manipulation environment using a "magnetic metaphor" is shown as an example.This environment is suitable .forobject manipulation in a combined virtual/real world.

We propose a parallel algorithm that efficiently generates an octree from a polyhedral shape representation. The algorithm is simple and can apply to general (non-convex) objects. The parallelized algorithm is based on two ideas for the assignment of multiple processors. The algorithm is implemented and tested on MIMD multi-processors having a shared memory. Experimental results, which demonstrate the effectiveness and efficiency of the proposed method are given (e.g. 2.5 seconds are required to generate an octree shape representation from a polyhedral object having 148 polygons). Through investigation of the performance of the proposed parallel algorithm, the features of the parallelism are discussed.

We propose a real-time algorithm for path planning in a dynamically changing 3-D environment where the motion of the obstacles cannot be predicted. The method represents everything in the environment (an arbitrarily shaped robot and static/dynamic obstacles) with an octree that makes no distinction of the movability of objects. A collision free path for a robot with arbitrary motion (translation and rotation) in a dynamic 3-D environment is efficiently sought by using the real-time A* algorithm and the potential field generated from each black cell of the octree. Parallel computation is used to speed up the generation of the potential field. Finally, the experimental results demonstrate the efficiency of the proposed path planning method for a dynamically-changing 3-D environment with arbitrary motion.

A method is proposed to help a user manipulate an object in a virtual environment. The method does not give special properties to the object in advance, and does not use special hardware. Instead, it uses only visual constraints among object features that are dynamically selected while the user manipulates the object. It uses face-to-face, edge-to-face, and vertex-to-face constraints; therefore, it provides a natural and intuitive environment for virtual object manipulation that replicates the task in a real environment. The constraint is transferred with a smooth motion. Experimental results show the effectiveness of this method in providing the user with a natural impression of motion in a virtual environment equipped with 3-D/6-D input devices and a conventional graphic workstation to finish all procedures within the cycle time of the human perceptual processor for objects with complicated shapes.

Proposes an efficient and simple method for finding a collision-free path and orientation for a rigid robot in a dynamic observable 3-D environment for unmanned aerial vehicles (UAVs). The method uses an octree for representing every object (robot and static/dynamic obstacles) in the environment without any distinction of the movability of objects; therefore, all objects are dealt with in the same way. The path of a robot from its starting position to the given goal with arbitrary motion (i.e., translation and rotation) in a 3-D environment is efficiently searched for in successive adjoining regions (octree white cells) by using the potential field generated from each black cell of the octree. The algorithm is simple, so it can easily be accelerated by using parallelization techniques. Experimental results obtained under several conditions, such as a point shaped robot and arbitrarily shaped robots in static/dynamic environments, are reported.

We propose an efficient method for detecting potential collisions among multiple objects with arbitrary motion (translation and rotation) in 3D space. The method is useful for online monitoring and path planning in a 3D environment in which there are multiple independently-moving objects. The method consists of two main stages: 1) the coarse stage, an approximate test is performed to identify interfering objects in the entire workspace using octree representation of object shapes; and 2) the fine stage, polyhedral representation of object shapes is used to more accurately identify any object parts that might cause interference and collisions. For this purpose, specific pairs of faces belonging to any of the interfering objects found in the first stage are tested, thus performing detailed computation on a reduced amount of data. Experimental results, which demonstrate the efficiency of the proposed collision detection method, are given.

An expert system for algorithm design for industrial machine vision is presented. Its knowledge base includes human experts' knowledge about image processing techniques, and it is capable of solving given vision problems. As a problem domain, the authors have chosen vision algorithms for a parts feeder, which determines the attitude of mechanical parts on a conveyor belt and rejects parts with inappropriate attitudes. The expert system for a parts feeder is designed to consist of three components: a feature selection expert, an image processing expert, and a decision tree generator. The knowledge necessary for the design of a vision algorithm for determining the attitudes of parts is discussed. A framework for representing the knowledge needed in order to find solutions to pattern classification problems is established.>