Emotional states influence a wide range of cognitive processes, such as attention, memory, and decision-making, and play an increasingly recognized role in human-computer interaction (HCI). Although most prior research has focused on high-level effects of emotion, the impact of incidental emotional states on fine-grained motor behaviors such as pointing and selection remains understudied. Addressing this gap, the present study examines how affective priming with validated emotional stimuli modulates user performance in a Fitts’ Law-based pointing paradigm. By systematically varying task difficulty, we quantitatively assess the effects of emotion on core performance measures: movement time, reaction time, error rate, and throughput. Our findings demonstrate that emotion can modulate low-level motor interactions in digital environments, with important implications for the design of adaptive, emotion-aware interfaces. These results advance the theoretical understanding of the interplay between emotion, cognition, and motor control and offer actionable insights to develop more robust and personalized interactive systems.

Managing multiple activities in virtual reality (VR) is often hindered by fragmented workflows and disruptive application switching. We present JourneyVR, a metaphorical interaction model designed for spatial and sequential workflows that reframes tasks as continuous journeys rather than isolated sessions. Users construct a Journey Map as a layout of islands (tasks) and bridges (transitions), which expands into an immersive world where activities unfold as a coherent, embodied narrative. Through a formative study, a controlled comparison, and an expert evaluation, JourneyVR was shown to enhance experiential continuity, intention to use, and overall satisfaction compared to using a conventional app launcher. Participants highlighted how the metaphor fosters motivation and achievement, while we identify boundaries regarding task type, scalability, and flexibility. Our findings demonstrate that framing sequential activities as navigable journeys can transform fragmented tasks into meaningful narratives, offering concrete guidelines for sustained engagement and more flexible workflows in immersive environments.

Navigation is a core task in VR that must be well supported. Interface choices shape task performance (e.g., search), viewing experience, and the sense of presence and immersion. Mini-maps are widely recommended and often help with these goals. However, the mini-map may also increase mental and physical workload and may even cause discomfort, exacerbating the already-likely-to-exist cybersickness. In this work, we compare two virtual navigation conditions, one with the mini-map and the other without, to investigate and reexamine their various effects, especially on cybersickness, an important and overlooked aspect of the VR experience. While the experiment reconfirmed the previously known fact that the map could enhance search task/navigational performance, it also caused or worsened cybersickness symptoms, especially for those possessing relatively lesser spatial abilities. The results indicate the need to revise the interaction design guidelines in VR regarding the pros and cons of using mini-maps for navigation and search.

Cybersickness has been a hindrance to the widespread adoption of virtual reality. As cybersickness is dynamic, with its manifestation changing constantly, predicting and mitigating it in real time is key. However, previous research has utilized predictive models trained mostly with datasets whose sickness levels were measured and labeled only after experiencing the content. In addition, many datasets rely on physiological signals as input, which makes them difficult to apply in actual VR usage. This makes such timely predictions unreliable or practically infeasible. We have created a publicly available dataset where the ground truth sickness levels were densely marked every 0.5 seconds and adjusted/updated on-demand using the FMS, and purposely excluded the difficult-to-collect physiological data. We demonstrate that predictive models trained with such a dataset, comprising just the content motion profile and FMS data, can still produce comparably reliable sickness prediction, and more so, when user-specific parameters (e.g., age, gender) are added. The created dataset is available at https://github.com/pvtryan1024/2025_Cybersickness_dataset.

Virtual reality (VR) offers a compelling and immersive experience; however, cybersickness (or VR sickness) stands as a significant obstacle to its widespread adoption. When a user experiences cybersickness, one’s physical condition deteriorates with various symptoms, often accompanied by an increased and destabilized heart rate and even altered perception of one’s state. In this paper, we propose to provide “False but Stable Heart rate (FSH)” feedback through auditory and vibrotactile stimulation to reversely induce a stably perceived heart rate and, thereby, alleviate cybersickness while navigating a sickness-inducing VR content. The validation of the human experiment confirmed the intended effect in a statistically significant way. Furthermore, it was found that the lesser compatible FSH feedback had a more substantial sickness reduction effect but distracted the user with the reduced immersive experience. The compatible FSH feedback still showed moderate sickness reduction with the maintained sense of presence and immersion.

This study investigates how different virtual agent (VA) behaviors influence subjects’ perceptions and group decision-making. Participants carried out experimental group discussions with a VA exhibiting varying levels of engagement and affective behavior. Engagement refers to the VA’s focus on the group task, whereas affective behavior reflects the VA’s emotional state. The findings revealed that VA’s engagements effectively captured participants’ attention even in the group setting and enhanced group synergy, thereby facilitating more in-depth discussion and producing better consensus. On the other hand, VA’s affective behavior negatively affected the perceived social presence and trustworthiness. Consequently, in the context of group discussion, participants preferred the engaged and non-affective VA to the non-engaged and affective VA. The study provides valuable insights for improving the VA’s behavioral design as a team member for collaborative tasks.

While controllers can support many generic 3D interactions in virtual reality (VR), it alone may fall short of eliciting the core experience for certain actions. One such task is the “object throwing’’, ubiquitous in many sports contents, involving intricately timed actions of aiming, arm swinging, and object releasing. With the increasing availability of eye tracking, we propose to combine gaze-based targeting with the controller swing gesture to model the object throw. The target is aimed/locked by gaze, and the throw is enacted by the controller swing/button press with the object let-gone by the button release. We compare and evaluate the proposed interface against the conventional controller-only based interaction through two typical baseball tasks – Pitcher and Outfielder. The findings indicated that the task accuracy was similar, but the gaze-based targeting allowed for faster completion. More importantly, the gaze-based method showed significantly higher usability and richer VR experience.

Employing airflow feedback is an effective means of enriching the virtual reality experience multimodally. Most interestingly, there have been conflicting reports of its effect on cybersickness. For example, some research has found that the airflow can be felt as refreshment and ease cybersickness, while others seen it as promoting the sense of vection and worsening the sickness symptoms. We present a comprehensive experimental study to clarify and investigate the deeper mechanism of the effect of directional airflow on cybersickness and user experience in virtual navigation. Our findings indicated that the airflow consistent with the navigation direction can strengthen the sense of vection, while no effect was observed for the inconsistent. On the other hand, regardless of the direction, the refreshing effect of the airflow was significant, resulting in the overall decreased level of cybersickness. Such a result can help us design the virtual navigation experience with higher dynamism and lesser extent of sickness symptoms.

The virtual reality (VR) headset is still relatively heavy, causing a significant physical and mental burden and negatively affecting the VR user experience, particularly during extended periods of use. In this paper, we present a prototype design of the “Lifter,” which utilizes a counterbalanced wire-pulley mechanism to partially relieve the weight of the VR headset (between 50% and 85%). The human subject study has confirmed that the Lifter relieved not only physical fatigue but also significantly improved mental burden, sense of immersion, presence, and flow (perception of time passing) during prolonged usage (30 minutes or more).

Locomotion is an important task in many virtual reality (VR) applications. Locomotion requires two different directional settings for: (1) translation to a target, and (2) body rotation. Conventional locomotion methods mostly make use of the hand-held controller for such directional control. The controller-based directional setting may not be natural, as in real life, humans set directions with the gaze, often together with the head/body rotation. Note that the view direction is independently and naturally controlled by the head (or gaze) direction. In this paper, we propose to use the peripheral gaze for body rotation, seamlessly together with the foveal gaze for the view control, so that the two tasks practically do not interfere with one another. In the comparative study, the peripheral gaze showed similar task performance to the controller-based rotation but with reduced VR sickness and improved usability.

Vital signals tend to become destabilized and generally increase when one’s physical condition is not well. For example, experiencing virtual reality (VR) sickness brings about a deteriorated physical condition, often accompanied by an increased heart rate. Several research have shown that providing feedback of false heart rate can induce various altered perceptions, such as increased effort and anxiety. In this poster, we propose to provide false but “stable” heart rate feedback through sound and vibration while navigating a sickness-inducing VR scene. We hypothesize that the false but stable heart rate feedback will have an induced effect of calming the user down (even stabilizing the heart rate itself) and reducing the unpleasant VR sickness symptoms. A pilot study was conducted to compare three conditions, namely viewing a sickness eliciting VR content, (1) as is, (2) with the false but stable heart rate feedback through sound, and (3) with the false but stable heart rate feedback through vibration. Results showed that the level of sickness was significantly reduced by sound and vibration feedback, respectively.

While there are several theories of virtual reality (VR) sickness causes and pertinent methods suggested for mitigation, it remains an important problem. One possible solution might be to prescribe measures for just relieving the immediate symptoms (vs. addressing the very root causes). Understanding that the severity of the sickness may affect individuals differently, we examined three methods: (1) reducing the weight of the headset (using a suspension mechanism); (2) refreshing the user with a gentle breeze of wind (using a fan); (3) accompanying the VR viewing experience with mindful breathing. We assess the relative sickness reduction effect, if any, of these three measures through a comparative pilot experiment and individual case analysis. The preliminary results point to rather the importance of system usability and how it affects the relationship between the perceived immersion and the extent of sickness. The initial proposition to enhance the user’s physical condition as a way to better withstand VR sickness symptoms could not be established.

Eye fatigue and unpleasant symptoms caused by the vergence and accommodation conflict with stereoscopic rendering pose a substantial usability issue in virtual reality. To address this problem, dual rendering of the stereoscopic imagery is proposed, aimed to alleviate such stress on the user. First, the scene is divided into two regions: the front proximal zone and the rest behind it. The back layer is rendered first for objects in the rest, with the conventional stereoscopic viewing parameter values. Then, remaining objects in the proximal zone are rendered using viewing parameters adjusted to reduce the VAC. The validation experiment confirmed that the proposed approach significantly reduced the overall eye fatigue without compromising the user’s depth perception ability to manipulate objects in the parameters-altered proximal zone.

One way to alleviate VR sickness is to reduce the sensory mismatch between the visual and vestibular organ regarding the motion perception. Mixing in the motion trail in the reverse direction to the original has been suggested as one such method. However, as such visual feedback can be content intrusive, we consider supplementing it by the non-visual multimodal reverse flow. In particular, we have devised methods to supply sound effects as if heard from the reverse direction, and vibration and air flow likewise. Our validation experiment has shown that the multimodal feedback was effective in significantly reducing the sickness, but its direction (reverse or not) did not have an effect as hypothesized.

It is well-known that employing the Walking-in-Place (WIP) interface can significantly reduce VR sickness in addition to promoting the sense of presence, immersion, and natural interaction. In this poster, we re-examine the conditions for which WIP will effectively reduce VR sickness. In particular, we investigate and compare the cases of applying WIP to navigating on flat terrain vs. up-and-down ramps with respect to the sickness reduction effect. We point out that naively designed WIP/navigation content has the possibility of actually worsening the VR sickness due to the sensory and reality mismatch between the flat real operating environment and the inclined virtual terrain.

Drones are controlled remotely through the on-board live camera often using a headset to immersively (without external distraction) situate the drone operator with a first-person view. The highly dynamic imagery of drone piloting can elicit significant VR sickness. In this poster, we demonstrate the application of mixing the reverse optical flow pattern into the drone piloting imagery for mitigating VR sickness, using only the features of the static landmarks in the scene. We compare it to the cases of applying no mitigation technique (baseline) and mixing in the optical flow pattern from “all” object features. The results show that the suggested method was significantly more effective in reducing sickness than when considering all object features, and had a higher preference for improved visibility and controllability.

The extent of virtual reality (VR) sickness varies widely among users, as each user has different sensitivities to diverse causes of VR sickness. This can make prescribing a single particular reduction technique difficult and ineffective. In this poster, we present preliminary work examining the more effective and preferred sickness-reduction techniques for a given user under varied sickness-inducing motions. Based on the user-specific information collected using VR content, a customized strategy is developed for a given user and applied to the same VR content. We report the experimental results for testing the effectiveness of the customized reduction technique, comparing it to a single particular reduction method.

Simulator sickness has been one of the major obstacles toward making virtual reality (VR) widely accepted and used. For example, virtual navigation produces vection, which is the illusion of self-motion as one perceives bodily motion despite no movement actually occurs. This, in turn, causes a sensory conflict between visual and actual (or vestibular) motion and sickness. In this study, we explore a method to reduce simulator sickness by visually mixing the optical flow patterns that are in the reverse direction of the virtual visual motion. As visual motion is mainly detected and perceived by the optical flow, artificial mixing in the reverse flow is hypothesized to induce a cancellation effect, thereby reducing the degree of the conflict with the vestibular sense and sickness. To validate our hypothesis, we developed a real-time algorithm to visualize the reverse optical flow and conducted experiments by comparing the before and after sickness levels in seven virtual navigation conditions. The experimental results confirmed the proposed method was effective for reducing the simulator sickness in a statistically significant manner. However, no dependency to the motion type or degrees of freedom were found. Significant distraction and negative influence to the sense of presence and immersion were observed only when the the artificially added reverse optical flow patterns were rather visually marked with high contrast to the background content.

In this paper, we propose to take advantage of the digital twinned environment to interact more efficiently in the large and complex AR space in spite of the limited sized and clipped FOV of the AR display. Using the digital twin of the target environment, “magical” VR interaction techniques can be applied, as visualized and overlaid through the small window, while still maintaining the spatial association to the augmented real world. First we consider the use of amplified movement within the corresponding VR twinned space to help the user search, plan, navigate and explore efficiently by providing an effectively larger view and thereby better spatial understanding of the same AR space with less amount of physical movements. Secondly, we also apply the amplified movement and in addition, the stretchable arm to interact with relatively large objects (or largely spaced objects) which cannot be seen in their entirety at a time with the small FOV glass. The results of the experiment with the proposed methods have showed advantages with regards to the interaction performance as the scene became more complex and task more difficult. The work illustrates the concept of and potential for XR based interaction where the user can leverage the advantages of both VR and AR mode operations.

This paper proposes a novel generative model called PUGAN, which progressively synthesizes high-quality audio in a raw waveform. Progressive upsampling GAN (PUGAN) leverages the progressive generation of higher-resolution output by stacking multiple encoder-decoder architectures. Compared to an existing state-of-the-art model called WaveGAN, which uses a single decoder architecture, our model generates audio signals and converts them to a higher resolution in a progressive manner, while using a significantly smaller number of parameters, e.g., 3.17x smaller for 16 kHz output, than WaveGAN. Our experiments show that the audio signals can be generated in real time with a comparable quality to that of WaveGAN in terms of the inception scores and human perception.

Using virtual reality systems with the head-mounted display can incur interaction difficulties and safety problems because of the user’s view being isolated from the real world operating space. One possible solution is to super-impose the real world objects or environment information onto the virtual scene. A variety of such visualization methods have been proposed, all in hopes of minimizing the negative effects of introducing foreign elements to the original virtual scene. In this poster, we propose to apply the neural style transfer technique to blend in the real world operating environment in the style of the given virtual space to make the super-imposed resulting image as natural as possible, maintaining the sense of immersion with the least level of distraction. Our pilot experimental study has shown that the stylization obscured the clear presentation of the environment and worsened or did not improve the safe user performance, and was neither considered sufficiently natural.

We explore a method to reduce motion sickness and allow people to use virtual reality while moving in vehicles. We put forth a usage scenario where the target VR content is based on constant road navigation so that the actual motion can enhance the VR experience. The method starts with a virtual scene and objects around an infinitely straight road. The motion of the vehicle is sensed by the GPS and IMU module. The sensed motion is reflected in a way that the virtual scene is navigated according to the vehicle motion, and its pathways distorted such that the virtual motion has a near-identical optical flow pattern to the actual. This would align the user’s visual and vestibular sense and reduce the effect of vection and motion sickness. We ran an pilot experiment to validate our approach, comparing the before and after sickness levels with the VR content (1) not aligned to the motion of the vehicle and (2) aligned by our method. Our preliminary results have shown the sickness was reduced significantly (but not eliminated to a negligible level yet) with our approach.

In this paper, we applied the concept of diminished reality to remove content-irrelevant pedestrian (i.e., real object) in the context of handheld augmented reality (AR). We prepared three view conditions: in Transparent (TP) condition, we removed the pedestrian entirely; in Semi-transparent (STP) condition, the pedestrian became semi-transparent; lastly, in Default (DF) condition, the pedestrian appeared as is. We conducted a user study to compare the effects of the three conditions on users’ engagement and perception of a virtual pet in the AR content. Our findings revealed that users felt less distracted to the AR content in TP and STP conditions, compared to the DF condition. Furthermore, users felt the virtual pet as more life-like, its behavior more plausible, and felt a higher spatial presence in the real environment, in the TP condition.

We investigate the effects of different ways of visualizing the virtual gait of the avatar in the context of Walk-in-Place (WIP) based navigation in a virtual environment (VE). In Study 1, participants navigated through a VE using the WIP method while inhabiting an avatar. We varied the visualization of the avatar’s leg motion while performing the WIP gesture: (1) Fixed Body: the legs stood still; (2) Pre-recorded Animation: the legs moved in a fixed predetermined pace (plausible but not in accordance to that of the user in general); (3) Synchronized Motion the legs moved according (synchronized) to those of the user. Our results indicated that the sense of presence and body ownership improved significantly when the leg motion was rendered synchronized to that of the user (Synchronized Motion). In addition, we developed a deep neural network (DNN) that predicted the users’ leg postures only with the head position tracking, eliminating the need for any external sensors. We carried out Study 2, to assess the effects of different gait visualizations, under two new factors: (1) virtual gait seen directly by the user looking down, or already visible by one’s shadow (i.e., no need to look down); and (2) playing a pre-recorded animation, or pre-recorded animation whose playback speed was adjusted to match with pace of the users’ actual leg motions as predicted by the DNN. The results of Study 2 showed that the virtual gait temporally synchronized with that of the user greatly improved the sense of body ownership, whether it was witnessed directly or indirectly with the shadow. However, the effect of virtual gait on presence was less marked when indirectly observed. We discuss our findings and the implications for representing the avatar locomotion in immersive virtual environments.

Virtual reality (VR) has been applied as a complimentary way to conventional treatment for mental disorders successfully. On the other hand, it has not been clearly shown what type of immersive media such as VR can directly affect one’s physiological parameters, associated with the state of mindfulness. We sought to assess how being subjected to differently designed VR contents can affect and modulate one’s anxiety both psychologically and more importantly physiologically. We empirically tested the comparative effects of two polarizing VR content types to this effect: (1) “calm/soothing” content and (2) “disturbing”. Twenty-five adults participated and their mental state, anxiety level and physiological signals were measured before and after experiencing the respective VR content type. The experiment found a statistically significant effect of the content type to the changes in these measures and confirmed that the “calm” content was helpful for one to self-regulate to lower heart rate and blood pressure, stable GSR, and the “disturbing” content in the opposite way. We applied this result to calm down and stabilize vital signs of patients during actual coronary angiography and catheterization operations. We were able to observe the same effect with positive comments from the patients and operating team.

In this work, we propose to use an approximate 2D map of the environment generated from the latest environment modeling technology and enhance the object manipulation performance for the touch based mobile augmented reality. We validated the advantage of the proposed interface through a pilot experiment and confirmed that the use of the 2D map helps reduce the task completion time almost 2 times and improve the usability as well.

We demonstrate PillowVR, virtual reality framework that integrates the smartphone, magnifier and sensors into a pillow/cushion for immersive VR experience in bed. PillowVR is applied for presenting immersive bed time stories to children to help them go to sleep and therefore its interaction was designed to minimize excessive bodily movements – only simple back-of-the head pressure events are used to browse the content. PillowVR illustrates shows how VR can be more woven into our daily lives inexpensively and naturally by customizing the set up and interaction for the specific task and experience.

In this paper, we propose to use the deep learning technique to estimate and predict the torso direction from the head movements alone. The prediction allows to implement the walk-in-place navigation interface without additional sensing of the torso direction, and thereby improves the convenience and usability. We created a small dataset and tested our idea by training an LSTM model and obtained a 3-class prediction rate of about 90%, a figure higher than using other conventional machine learning techniques. While preliminary, the results show the possible inter-dependence between the viewing and torso directions, and with richer dataset and more parameters, a more accurate level of prediction seems possible.

Onomatopoeia refers to a word that phonetically imitates, resembles the sound, or depict an event at hand. In languages like Korean and Japanese, it is used in everyday conversation to emphasize certain situation and enrich the prose. In this poster, we explore if the use of onomatopoeia, visualized and added to the usual sound feedback, could be taken advantage to increase or alter the perceived realism of the sound feedback itself, and furthermore of the situation at hand in virtual reality. A pilot experiment was run to compare the user's subjective perceived realism and experience under four test conditions of presenting a simple physical interaction, accompanying it with: (1) just the "as-is" sound (baseline), (2) "as-is" sound and onomatopoeia, (3) a representative sound sample (e.g. one for all different collision conditions), and (4) a representative sound sample and onomatopoeia. Our pilot study has found that the use of onomatopoeia can alter and add on to the perceived realism/naturalness of the virtual situation such that the experiences of the single representative sound added with the onomatopoeia and "as-is" sound were deemed similar.

Onomatopoeia refers to a word that phonetically imitates the sound. It is often used, in comics or video, in caption as a way to dramatize, emphasize, exaggerate and draw attention the situation. In this paper we explore if the use of onomatopoeia could also bring about similar effects and improve the user experience in virtual reality. We present an experiment comparing the user's subjective experiences and attentive performance in two virtual worlds, each configured in two test conditions: (1) sound feedback with no onomatopoeia and (2) sound feedback with it. Our experiment has found that the moderate and strategic use of onomatopoeia can indeed help direct user attention, offer object affordance and thereby enhance user experience and even the sense of presence and immersion.

The field of information visualization studies the interactive visual representations of data to reinforce human cognition, thereby facilitate the discovery of new tacit knowledge and even amplify human intelligence. Augmented reality (AR) shares the same objective and it can be treated as one particular form of information visualization where the data are both the real objects and the augmentations for them. As such, it presents a unique set of problems within the general requirements for an effective information visualization method. In this talk, I will first outline and put forth four main requirements for AR visualization, namely, (1) naturalness, (2) visibility (3) persistance/stability and (4) glass/hmd ergonomics. Then I will present a short survey of the existing AR visualization techniques and characterize them by their attributes and categorize them in terms of how they satisfy or address the proposed requirements. Finally, I will also introduce some of my own on-going research work in this area, specifically, for real time contrast adjustment for mobile augmented reality, usability with glasses, augmentation data organization, and multimodal AR data presentation. I hope that this work can instigate and shed some light on the future directions for further research in AR visualization techniques.

In this paper we report on an investigation of the effects of a self-avatar's visual similarity to a user's actual appearance, on their perceptions of the avatar in an immersive virtual reality (IVR) experience. We conducted a user study to examine the participant's sense of body ownership, presence and visual realism under three levels of avatar-owner visual similarity: (L1) an avatar reconstructed from real imagery of the participant's appearance, (L2) a cartoon-like virtual avatar created by a 3D artist for each participant, where the avatar shoes and clothing mimic that of the participant, but using a low-fidelity model, and (L3) a cartoon-like virtual avatar with a pre-defined appearance for the shoes and clothing. Surprisingly, the results indicate that the participants generally exhibited the highest sense of body ownership and presence when inhabiting the cartoon-like virtual avatar mimicking the outft of the participant (L2), despite the relatively low participant similarity. We present our experiment and main findings, also, discuss the potential impact of a self-avatar's visual differences on human perceptions in IVR.

Mobile virtual reality (M-VR) uses an inexpensive and light headset into which the smartphone is inserted to conveniently experience immersive contents. Even so the headset is bulky and difficult to carry around, and makes the smartphone inaccessible. Recently, an alternative form of M-VR has appeared in the market in which the magnifying lenses are simply clipped on the smartphone (dubbed "EasyVR"). Despite being open and the user'speripheral view not shut from the outside world, it still gives a good level of immersion with a wide magnified field of view. EasyVR has the added advantages of quick switch between with the regular smartphone usage mode and access to the touch screen for the seamless interaction. In this paper, we examine and compare the level of presence and immersion as provided by three different display configurations of M-VR: (1) EasyVR, (2) the usual headset which completely isolates the user from the outer-world with the vignetted view (ClosedVR), and (3) completely open hand-held smartphone view (OpenVR). We also control the environment condition, static or dynamic, as seen and perceived through the peripheral view and possibly having an effect on the level of presence and immersion in the respective display configuration. Our findings first show the easily expected, namely, both ClosedVR and EasyVR clearly exhibiting a much higher level of presence and immersion than OpenVR. The results also show that even though there is a substantial extent within the peripheral view showing the outer environment, the level of presence and immersion of EasyVR is nearly comparable to that of ClosedVR. Only when the environment was dynamic (as visible in the peripheral view ends), EasyVR showed a lower level of presence and immersion than ClosedVR, but still significantly higher than OpenVR. Therefore, EasyVR is a very attractive alternative to the usual ClosedVR, especially as a "use-anywhere" VR considering its clearly improved convenience and sufficient level of immersion beyond just for casual purposes.

In this paper, we present a virtual reality based content/system called the "Drop the beat" designed to help the mindfulness and train one to overcome panic disorder. The two main elements of the proposed system are the (1) use of 360-degree video for presenting the panic inducing situation and, (2) facilitation of the mindfulness through a compelling scenario and immersive experience with multimodal feedback. In particular, we hypothesized that the direct observance and tangibly feeling for the beating heart in one's hand would help the user train to rationalize and overcome the situation (and e.g. effectively bring down one's heart rate back to a normal level). We conducted a small pilot study, administering the proposed VR content to five panic disorder patients and report the interim results.

Funneling and saltation are the two major illusory feedback techniques employed by vibrotactile feedback. They elicit the sensation of a vibrotactile stimulus outside the user's body, originating from an externally held object that visually extends the body. This paper examines the synergy of associating the out-of-body illusory tactile sensation with different visual feedback to improve the user experience for interacting with the augmented virtual objects. There are two important types of visual feedback: the rendering of a “body-extending” object (that appears attached to and connecting the two fingertips to create the illusion) and “interaction” with the object itself (with which the user interacts). Two experiments were performed, for funneling and saltation, assessing the perceptual effects under four associated visual feedback conditions: with 1) no visual feedback, 2) a body-extending virtual object, 3) a virtual interaction object (rendered at the illusion target location) and 4) both the body-extending and interaction virtual objects. We hypothesized that rendering a body-extending object will maintain an important role in eliciting the illusion itself, while showing the actual interaction object will improve user performance and experience through multimodal integration. Our findings indicated that the effect of the interaction object was much stronger than that of the body extension. In the case of funneling, the visual body extension was not even necessary to elicit the out-of-body sensation. The effect of the body extension was marginal for funneling. These findings can be applied to tactile interaction design using only few actuators on a variety of media platforms including augmented content.

In this paper, we present an application of rotational gain to horizontal panning for viewing 360 degree videos. Rotational gain refers to the ratio between the rotation velocity in the virtual (video) space to that of the physical, and it allows the user to rotate one's head less quickly than actually needed (without the user noticing such adjustment to some degree). As such it can bring about convenience, and less physical movement, fatigue and possibly even sickness. We implemented a 360 video panning system with both a constant and dynamic gain, and compared user behavior and subjective usability. Our pilot study showed promising results in that with the proper gain value and control scheme, the user will unknowingly use less physical movement than needed, yet maintain reasonable spatial understanding with higher usability.

In this paper, we present an AR-based tele-coaching system for fast-paced tasks, applied to the game of tennis, and explore for interface design guidelines through a user study. We have evaluated the player's performance for instruction understanding in three different modalities of augmentation: (1) AR --- visual only, (2) Sound --- aural only and (3) Multimodal --- both visual and aural. The augmented instructions were useful even in the stringent temporal conditions, and most effective with the visual only augmentation due to its modal capability of encoding and presenting several information compactly at once.

We propose a text entry method for VR, using the smartphone and its hovering function, called the HoVR-Type. The hovering function effectively acts as the finger tracking sensor thereby allowing the user to type in the virtual space. When added with the additional phase to correct the initial touch input and having the final key entered upon the finger release, the proposed method showed competitive performance to that of the conventional "aim-and-shoot" method and also exhibited much higher usability. Overall, the use of the smartphone leverages on the already established mobile user experience and can be further extended to other VR interaction techniques so that one can use the common smartphone as an all-purpose VR interaction device.

In this paper, we propose a 1st person view and body motion based control method realized on a smart phone/pad called the Sprint (Smart Phone/Pad and Robot for Teleoperation and Tele-presence). The smart phone/pad is chosen, aside from its ubiquity and wide usage, for its mobility and rich media (sensor/display) capabilities. The key idea of Sprint is to have the operator control and interact by "posing" as the robot in the remote area in hopes of eliciting a high sense of tele-presence and thereby promoting the spatial task performance and the overall interactive experience. In particular, we demonstrate a remote robot viewing control method based on this design principle and validate that the proposed approach does improve the sense of tele-presence and spatial understanding.

In this work, we present a technique based on image saliency analysis to improve the conspicuity of the foreground augmentation to the background real world medium by adjusting the local brightness contrast. The proposed technique is implemented on a mobile platform considering the usage nature of AR. The saliency computation is carried out for the augmented object's representative color rather than all the pixels, and searching and adjusting over only a discrete number of brightness levels to produce the highest contrast saliency, thereby making near-real time computation possible. Thus, while the resulting imagery may not be optimal due to such a simplification, our tests showed that the visibility was still significantly improved without much difference to the "optimal" ground truth in terms of correctly perceiving and recognizing the augmented information

The "Internet of Things (IoT)" is one of the most promising emerging technologies today. IoT refers to the concept of, if not all, many everyday objects possessing information processing power and network connectivity from which personal and context based services can be derived collectively [Perera et al, 2014]. Any useful and successful service will require on an effective and usable means for interaction, and Augmented Reality (AR) has been touted as one such ideal method. One difficulty with the proliferation of the AR technology (and associated services) is with the provision of large scale and usable contents. Useful contents and services often require recognition and tracking of a very high number of objects, and as the target objects and respective services increase, so will data redundancy and error.

Remote live coaching/collaboration requires an effective means of communication between the two involved parties. In this poster, we consider the problem of how to augment 3D information to a target object in the remote environment in the context of a tele-coaching scenario. Figure 1 illustrates a typical "tele-augmentation" situation for a coaching purpose. User A wants to set up a new PC, looking into the scene with a video see-through HMD and a camera. User B, the domain expert in a remote site, has a smart phone to which the live video of User A's view is fed. User B makes annotations on the transmitted view (to coach and instruct User A as how to set up the PC) and the annotations are immediately visible to User A for him to follow. It would be desirable if the support system can operate without pre-registered targets or artificial markers.

In this poster, we propose "proprioceptive" pointing (and selection) in which we use the finger and the sense of proprioception, without focusing on the finger, to point and select an object in the real world for the purpose of further interaction. The assumption is that this will reduce user fatigue since the user will switch one's focus less frequently, while still being able to point effectively through the proprioceptive sense. Figure 1 illustrates the concept and how such a technique could be effectively used e.g. for object query system using a see-through glass. In this typical scenario, the user selects an object in the real world (by proprioceptive pointing/designation), which in turn is captured by the on-glass camera and then identified and recognized for final augmentation. Note the "blurry" fingertip used for aiming to the target object.

In this article, we present a design and an evaluation of a hand-held, trackball-based, touch-haptic interface, named ‘Touchball’, for conveying directional cues. By the nature of the trackball mechanism, the device provides flexibility in terms of directional degrees of freedom compared with conventional vibro-tactile devices. It also has the advantage of a direct transfer of force feedback through frictional touch (leveraging on human's high skin sensitivity), thus requiring only a relatively small amount of inertia by the actuator. This leads to a compact hand-held design appropriate for mobile and 3D interactive applications. We first conducted psychophysical experiments to discover the detection thresholds and measure the perceived amount of directional force feedback. In addition, we conducted a performance and usability test, comparing the Touchball to a vibro-tactile (another touch-based) and aural (another modality) feedback device for an actual application, namely a navigational guide. Experimental results have shown that while both the Touchball and vibro-tactile devices produced comparable task completion time to that with aural feedback, their accuracies were significantly lower. The usability, in most categories, was higher in the order of the sound feedback, Touchball, then vibro-tactile feedback with statistically significant differences. To convey more than just on–off information through tactility such as directions, the vibro-tactile device requires relatively many vibrators and contact points, whereas the Touchball is a more compact and equally effective but with higher usability (e.g. mental load, ease of use, naturalness, etc.). Thus, Touchball offers a viable or even better alternative to a single vibrator or grid-based tactile feedback device especially as a supplementary modality device (to the visual or aural centric interface).

Abstract With the continued technology innovation in object sensing and human motion tracking, the traditional two‐dimensional video‐based tele‐conference systems are projected to evolve into the three‐dimensional immersive and augmented reality (AR) based on which one can communicate with the teleported remote other as if present, moving and interacting naturally in the same location. One technical hurdle to this vision is the need to resolve the environmental differences and the resulting teleported avatar motion anomaly between the remote and local sites. This paper presents a novel method to first establish a spatial and object‐level match between the remote and local sites and adapts the position and motion of the teleported avatar into the local AR space according to the matched information. This results in a natural looking and spatially correct rendering of the remote user in the local augmented space and a significantly improved tele‐conference experience and communication performance. Copyright © 2015 John Wiley & Sons, Ltd.

ABSTRACT Whole‐body interaction is an effective way to promote the level of presence and immersion in virtual reality systems. In this paper, we introduce “G‐Bar,” a grounded isometric interaction device that naturally induces whole‐body interaction without complicated sensing and active haptic feedback apparatus. G‐Bar takes advantage of the significant passive reaction force feedback sensed throughout the body to produce an enhanced level of presence/immersion and possibly even task performance. For detailed investigation in the contributing factors, two experiments were carried out to assess the comparative effectiveness of G‐Bar to the following: (1) grounded but isotonic device (with force feedback and without); and (2) nongrounded handheld devices (both isotonic and isometric). The results showed that the G‐Bar induced significantly higher presence and competitive task performance (fixed velocity navigation) than the isotonic (grounded or handheld) and nongrounded isometric interfaces. Compared with the grounded isometric device with active force feedback, G‐Bar produced competitive performance. In particular, the analysis of the subjective evaluation revealed a high correlation between the level of presence and whole‐body interaction. On the other hand, whole‐body experience was not induced as much with just the active force‐feedback devices. Thus, for appropriate tasks, the grounded isometric interface can be a viable alternative to expensive and mechanically limiting active force‐feedback devices in enhancing user experience. Copyright © 2013 John Wiley & Sons, Ltd.

Funneling and saltation are two major illusory feedback techniques for vibration-based tactile feedback. They are often put into practice e.g. to reduce the number of vibrators to be worn on the body and thereby build a less cumbersome feedback device. Recently, these techniques have been found to be applicable to eliciting "out of the body" experiences as well (e.g. through user-held external objects). This paper examines the possibility of applying this phenomenon to interacting with virtual objects. Two usability experiments were run to test the effects of funneling and saltation respectively for perceiving tactile sensation from a virtual object in an augmented reality setting. Experimental results have shown solid evidences for phantom sensations from virtual objects with funneling, but mixed results for saltation.

In this poster, we present an experiment that compares three forms of interaction to study the user behavior with regards to the effects of camera view for in-situ robot control. We compared three hand-held interfaces with: (1) no camera view (Nominal), (2) a camera view/aim is always fixed toward the robot (Fixed) and (3) a camera view with user controlled aim (Free). The three approaches represent different balances between information availability, interface accessibility and the amount of induced attentional shifts. Experiment results have shown that all three interaction models exhibited similar task performance even though the Fixed type induced much less attentional shifts. On the other hand, the users much preferred the Nominal and Free type. Users mostly ignored the camera view despite having to shift one's attention excessively, due to the lack of visual quality, realistic scale and depth information.

Participating in a play is one of integral curriculum for young children at nurseries and kindergartens. At the same time, it is not very easy to successfully run and manage a play for young children due to their low age and immaturity. Scripts are difficult to memorize and children's attention span is quite short. We are exploring the use of a robot and augmented reality (AR) technology to assist the nursery teachers in hopes to alleviate the difficult and complicated task of running the play, and also as a way to increase the learning effect by promoting the concentration and immersion (by the presence of the robot and novelty of the augmented display) [1, 2, 3]. For this purpose, we have devised a semi-autonomous remote-controlled projector robot with the capabilities of background projection and control, generating the synthesized augmented view, camera/movement control, producing story narration and various special effects. We have recently deployed the robot assistant for a play ('Three Little Pigs') at an actual nursery to observe and investigate various aspects of human robot interaction. For instance, the robot interacts with the actors on stage, leading and guiding them by showing (with small display on the robot) the synthesized augmented view, script guidance, and putting forth and changing the backdrop projection. It also assumes the role of the camera man and may instigate minute interplay with the actor as it zooms in and out on actors (by remote control). Our initial observation indicated that the use of the robot and AR indeed exhibited very high potential in drawing the attention of the children and enhancing the educational effect, but required the right amount of autonomy and external control and an intuitive interface.

With the advent of portable projectors (also embedded in a smart phone), projection based augmented reality (AR) will be an attractive form of AR as the augmentation is made directly in real space (instead of on the video screen). Several interaction methods for “Procam” systems, also applicable to projection based AR, have been developed, but their comparative usability has not been studied in depth. In this paper, we compare the usability of four representative interaction methods, applied to the menu selection task, for the hand-held projection based AR. The four menu selection methods studied are formed by combinations of two types of cursor control (projector cursor vs. on-device touch screen), and two types of item selection (explicit click vs. crossing). Experimental results have shown that the menu selection task was most efficient, usable and preferred when the projector cursor with the crossing widget was used. Furthermore, the task performance was not statistically different between using the dominant, non-dominant hand and even both hands.

There are growing needs for the 3D environment. The 3D devices, such like 3D television, have already started to be sold in markets to the general public. In other words, it means that 3D environment is becoming accustomed to everything. In the same manner, the ways for the 3D manipulation are also getting more important, and there are lots of ways for the 3D manipulation on large screen, which has been used so far, such like 6DOF tracking devices. The problem, however, is that they require high cost, and appropriate place with good environment should be prepared. In this paper, another easier and cheaper way for the 3D manipulation is introduced. The method which is introduced is a way using a mobile device instead of the expensive trackers. Using the method that this paper shows, it is expected that this method is accustomed to everybody and can be used easily.

In this paper, we compare the efficiency in information transfer and user acceptance between the traditional humanoid robot and the projector robot.

In this poster, we present an authoring tool for mixed reality (MR) contents, called k-MART. The tool is comprehensive to accommodate many different types of mixed reality applications and platforms, being able to mix and match different types of contexts and behaviors for content creation. The content is intuitively modeled as a collection of “context-behavior” pairs and saved in a declarative manner to be “played” by a content browser. The tool is designed to minimize user programming and offers many abstractions of functionalities with only important details to be filled in by the user using graphical user interfaces. We believe that our approach has a good potential for the wide dissemination and sharing of MR contents.

This article explores the synergistic interaction among the geometric field of view (GFOV), proprioceptive interaction and multimodal feedback in a virtual environment. We hypothesise that, with proprioceptive interaction and multimodal feedback, the GFOV can be set larger than its reference value originally set to give similar distance perception when viewed with a given physical field of view of a head mounted display. This in turn would help a user carry out a task more effectively with the increased visibility. We tested our hypothesis with an experiment to measure the user's change in distance perception according to different values of egocentric GFOV and task/feedback conditions (15 subjects). Our experimental results have shown that, when coupled with proprioceptive interaction, the GFOV could be increased up to 170% of the reference without introducing change in user's distance perception. When additional visual and tactile feedback was introduced, the GFOV could be increased up to 200%. The results offer a useful guideline for designing multimodal interfaces for close range spatial tasks in virtual environments. In addition, it demonstrates one way to overcome the inadequacy of the narrow physical fields of view often found in most general purpose head mounted displays.

Abstract Hand‐held devices are also becoming computationally more powerful and being equipped with special sensors and non‐traditional displays for diverse applications aside from just making phone calls. As such, it raises the question of whether realizing virtual reality, providing a minimum level of immersion and presence, might be possible on a hand‐held device capable of only relatively “small” display. In this paper, we propose that motion based interaction can widen the perceived field of view (FOV) more than the actual physical FOV, and in turn, increase the sense of presence and immersion up to a level comparable to that of a desktop or projection display based VR systems. We have implemented a prototype hand‐held VR platform and conducted two experiments to verify our hypothesis. Our experimental study has revealed that when a motion based interaction was used, the FOV perceived by the user for the small hand held device was significantly greater than (around 50%) the actual. Other larger display platforms using the conventional button or mouse/keyboard interface did not exhibit such a phenomenon. In addition, the level of user felt presence in the hand‐held platform was higher than or comparable to those in VR platforms with larger displays. We hypothesize that this phenomenon is related to and analogous to the way the human vision system compensates for differences in acuity resolution in the eye/retina through the saccadic activity. The paper demonstrates the distinct possibility of realizing reasonable virtual reality even with devices with a small visual FOV and limited processing power. Copyright © 2010 John Wiley & Sons, Ltd.

In this paper, we present a design and an evaluation of a hand-held trackball based touch-haptic interface, named "TouchBall." Using a trackball mechanism, the device provides flexibility in terms of directional degrees of freedom. It also has an advantage of a direct transfer of force feedback through frictional touch (with high sensitivity), thus requiring only relatively small amount of inertia. This leads to a compact hand-held design appropriate for mobile and 3D interactive applications. The device is evaluated for the detection thresholds for directions of the force feedback and the perceived amount of directional force. The refined directionality information should combine with other modalities with less sensory conflict, enriching the user experience for a given application.

Current forms of Human Robot Interaction (HRI) pursue mostly anthropomorphism and direct interaction. That is, the interaction paradigm is based on imitating how "people" interact with one another (e.g. using spoken language, gestures, facial expression, etc.). However, "direct" interaction/contact with the "robot," often causes significant inconvenience and usability problems. In this paper, we present an alternative to the anthropomorphic interface using a projected display and indirect, yet already familiar GUI based interaction. That is, the projector (on the moving robot) projects information on the nearby surface and provides a relatively large area through which indirect GUI based interaction can occur. As an instance of such an HRI paradigm, we present a moving robot kiosk that projects displays around itself and serve and interact with multiple people at once. We report our on-going development efforts and a pilot experimental study that compares it to the typical touch screen based direct HRI.

In this paper, we investigated the effect of relative difference between the virtual object and tangible prop in its size and shape in terms of usability. We have found the obvious fact that in general that manipulation is more efficient with equally sized/shaped prop and virtual objects. In addition, when decoupled, the size difference factor did not matter. While an additional experiment (in progress) is needed to confirm the true effect of shape difference, we posit that prop design should concentrate on representing the critical shape features for a given class of objects the prop is to represent. Humans are adept at recognizing and identifying objects even if there are shown at different scales and in different angles. This ability, called ldquoconstancyrdquo is weaker in the dimension of shapes, e.g. compared to sizes, which is another reason to suspect shape to be a more critical factor for effective prop design. Another significant factor, in the design of props (not treated in this paper) is the prop to virtual object alignment (which can affect the task performance in terms of finding and feeling for a stable grasp). For instance, possible choices for spatially registering a prop to a virtual object (or vice versa) can be about their respective center of gravity, about the chosen surface, etc. Future experiments and prop design method will have to take this into account as well.

This paper presents a real-time framework for computationally tracking objects visually attended by the user while navigating in interactive virtual environments. In addition to the conventional bottom-up (stimulus-driven) saliency map, the proposed framework uses top-down (goal-directed) contexts inferred from the user's spatial and temporal behaviors, and identifies the most plausibly attended objects among candidates in the object saliency map. The computational framework was implemented using GPU, exhibiting high computational performance adequate for interactive virtual environments. A user experiment was also conducted to evaluate the prediction accuracy of the tracking framework by comparing objects regarded as visually attended by the framework to actual human gaze collected with an eye tracker. The results indicated that the accuracy was in the level well supported by the theory of human cognition for visually identifying single and multiple attentive targets, especially owing to the addition of top-down contextual information. Finally, we demonstrate how the visual attention tracking framework can be applied to managing the level of details in virtual environments, without any hardware for head or eye tracking.

This article presents a real-time GPU-based post-filtering method for rendering acceptable depth-of-field effects suited for virtual reality. Blurring is achieved by nonlinearly interpolating mipmap images generated from a pinhole image. Major artifacts common in the post-filtering techniques such as bilinear magnification artifact, intensity leakage, and blurring discontinuity are practically eliminated via magnification with a circular filter, anisotropic mipmapping, and smoothing of blurring degrees. The whole framework is accelerated using GPU programs for constant and scalable real-time performance required for virtual reality. We also compare our method to recent GPU-based methods in terms of image quality and rendering performance.

This paper proposes to generate and provide wide field of view (FOV) augmented reality (AR) imagery by mosaicing images from smaller fields of moving views in "desktop" tangible AR (DTAR) environments. AR systems usually offer a limited FOV into the interaction space, constrained by the FOV of the camera and/or the display, which causes serious usability problems especially when the interaction space is large and many tangible props/markers are used. This problem is more apparent in DTAR environments in which an upright frontal display is used, instead of a head mounted display. This can be solved partly by placing the camera at a relatively far location or by using multiple cameras and increasing the working FOV. However, as for the former solution, the large distance between the interaction space and the fixed camera decreases the tracking and recognition reliability of the tangible markers, and the latter solution introduces significant additional set-up, cost, and computational load. Thus, we propose to use a mosaiced image to provide wide FOV AR imagery. We experimentally compare our solution, i.e. to offer the entire view of the interaction space at once, to other nominal AR set-ups. The experimental results show that, despite some amounts of visual artifacts due to the imperfect mosaicing, the proposed solution can improve task performance and usability for a typical DTAR system. Our findings should contribute to making AR systems more practical and usable for the mass.

Abstract We present a real‐time method for rendering a depth‐of‐field effect based on the per‐pixel layered splatting where source pixels are scattered on one of the three layers of a destination pixel. In addition, the missing information behind foreground objects is filled with an additional image of the areas occluded by nearer objects. The method creates high‐quality depth‐of‐field results even in the presence of partial occlusion, without major artifacts often present in the previous real‐time methods. The method can also be applied to simulating defocused highlights. The entire framework is accelerated by GPU, enabling real‐time post‐processing for both off‐line and interactive applications.

This article reports two human experiments to investigate the effects of visual cues and sustained attention on spatial presence over a period of prolonged exposure in virtual environments. Inspired by the two functional subsystems subserving spatial and object vision in the human brain, visual cues and sustained attention were each classified into spatial and object cues, and spatial and non-spatial attention, respectively. In the first experiment, the effects of visual cues on spatial presence were examined when subjects were exposed to virtual environments configured with combinations of spatial and object cues. It was found that both types of visual cues enhanced spatial presence with saturation over a period of prolonged exposure, but the contribution of spatial cues became more relevant with longer exposure time. In the second experiment, subjects were asked to carry out two tasks involving sustained spatial attention and sustained non-spatial attention. We observed that spatially directed attention improved spatial presence more than non-spatially directed attention did. Furthermore, spatial attention had a positive interaction with detailed object cues.

The International Conference on Computer Animation and Social Agents (CASA) organized by the Computer Graphics Society (CGS) is one of the premier academic conferences in the field of computer animation, specializing in character/object modeling, animation, and behavior simulation of social agents. This year, the conference was held in Seoul, Korea marking its 21st occurrence. The CASA international program committee has selected 32 full papers from 100 submissions around the world based on comments and scores from at least four reviewers assigned for each submitted paper. It is our great pleasure to have those distinguished work to be presented in this special issue of the Computer Animation and Virtual Worlds Journal (CAVW). The 32 papers presented in this journal are nicely categorized into: (1) character animation, (2) facial animation, (3) crowd modeling and simulation, (4) synthesis and 2D methods, (5) general modeling and deformation techniques, (6) physical simulation, and (7) applications. The Program Co-Chairs would like to thank the international program committee for spending their valuable time and effort in the reviewing process and the selected authors for their contribution.

In this paper, we have addressed the usability issue of augmented reality systems. In order to overcome the lack of view into the interaction space in nominal AR systems, we proposed to create and use a mosaiced image as a cost effective and practical solution. One of the obvious problem is that even if the view span is increased by the mosaicing, because of the fixed and limited resolution/size of the display(either HMD or upright display), the content must be viewed in smaller scale. Yang et al. has found out that increasing the geometric field of view can be tolerated to some degree without much degradation in task performance [3]. In our pilot experiment, a comparison of our mosaiced AR display to the nominal set up improved task performance when the task involved manipulation of many tangible props in a relatively wide area. Our future work is to conduct a more formal usability study and address other AR usability issues.

This paper presents a real-time framework for computationally tracking objects visually attended by the user while navigating in interactive virtual environments. In addition to the conventional bottom-up (stimulus-driven) features, the framework also uses topdown (goal-directed) contexts to predict the human gaze. The framework first builds feature maps using preattentive features such as luminance, hue, depth, size, and motion. The feature maps are then integrated into a single saliency map using the center-surround difference operation. This pixel-level bottom-up saliency map is converted to an object-level saliency map using the item buffer. Finally, the top-down contexts are inferred from the user's spatial and temporal behaviors during interactive navigation and used to select the most plausibly attended object among candidates produced in the object saliency map. The computational framework was implemented using the GPU and exhibited extremely fast computing performance (5.68 msec for a 256X256 saliency map), substantiating its adequacy for interactive virtual environments. A user experiment was also conducted to evaluate the prediction accuracy of the visual attention tracking framework with respect to actual human gaze data. The attained accuracy level was well supported by the theory of human cognition for visually identifying a single and multiple attentive targets, especially due to the addition of top-down contextual information. The framework can be effectively used for perceptually based rendering without employing an expensive eye tracker, such as providing the depth-of-field effects and managing the level-of-detail in virtual environments.

Hand-held computing devices are ubiquitous and have become part of our lives these days. Moreover, hand-held devices are also increasingly being equipped with special sensors and non-traditional displays. As such, it raises the question of whether such a "small" and "reduced" device could serve as an effective virtual reality (VR) platform and provide sufficient immersion and presence, e.g. through multimodal interaction. In this paper, we address this question by comparing the perceived field of view (FOV) and level of immersion and presence among the users' of VR platforms, varied in the sizes of physical/software FOV and in styles of interaction. In particular, we consider a motion based interaction, a style of interaction uniquely suitable for the "hand-held" devices. Our experimental study has revealed that when a motion based interaction was used, the FOV perceived by the user for the small hand held device was significantly greater than (around 50%) the actual. Other displays using the button or mouse/keyboard interface did not exhibit such a phenomenon. In addition, the level of user felt presence was higher than even that from a large projection based VR platform. The paper demonstrates the distinct possibility of realizing reasonable virtual reality even with devices with a small visual field of view and limited processing power.

Hand-held devices possess a large potential as an interaction device for their today's ubiquity, and present us with an opportunity to devise new and unique ways of interaction as a smart device with multi-modal sensing and display capabilities. This paper introduces user interaction techniques (for selection, translation, scaling and rotation of objects) using a camera-equipped hand-held device such as a mobile phone or a PDA for large shared environments. We propose three intuitive interaction techniques for 2D and 3D objects in such an environment. The first approach uses the motion flow information to estimate the relative motion of the hand-held device and interact with the large display. The marker-object and marker-cursor approaches both use software markers on the interaction object or on the cursor for the various interactive tasks. The proposed interaction techniques can be further combined with many auxiliary functions and wireless services (of the hand-held devices) for seamless information sharing and exchange among multiple users. A formal usability analysis is currently on-going.

Virtual studios have long been used in commercial broadcasting. However, most virtual studios are based on "blue screen" technology, and its two-dimensional (2-D) nature restricts the user from making natural three-dimensional (3-D) interactions. Actors have to follow prewritten scripts and pretend as if directly interacting with the synthetic objects. This often creates an unnatural and seemingly uncoordinated output. In this paper, we introduce an improved virtual-studio framework to enable actors/users to interact in 3-D more naturally with the synthetic environment and objects. The proposed system uses a stereo camera to first construct a 3-D environment (for the actor to act in), a multiview camera to extract the image and 3-D information about the actor, and a real-time registration and rendering software for generating the final output. Synthetic 3-D objects can be easily inserted and rendered, in real time, together with the 3-D environment and video actor for natural 3-D interaction. The enabling of natural 3-D interaction would make more cinematic techniques possible including live and spontaneous acting. The proposed system is not limited to broadcast production, but can also be used for creating virtual/augmented-reality environments for training and entertainment

In the future ubiquitous computing living environment, humans will interact with computers (knowingly or unknowingly) any where by various means other than through the traditional desktop interface (e.g. keyboard and mouse). Interacting with a "smart" table could very well be one such probable scenario. In this paper, we outline and classify various possible techniques for interacting with the smart table under reasonable assumptions about the operating environment. We considered the primitive tasks of object selection, object manipulation and menu control. The classification should serve as a basis for experimenting, designing, selecting, composing and implementing a specific interaction technique for a given task to achieve high usability.

In this paper, we propose to use P-VoT (POSTECH Virtual reality system development Tool), a component based interactive authoring toolkit for virtual environments, to quickly build a virtual ubiquitous computing environment. We have created four sets of components in P-VoT for this purpose: (1) sensors, (2) displays, (3) processing objects, and (4) data collection objects. The first three types of objects are used for quickly prototyping and planting ubiquitous computing subsystems in the virtual environment. They are easily reusable within P-VoT and can be adapted as needed: its function or behavior can be altered easily at different levels of abstraction. The data collection objects are used for collecting usability data such as performance measurements and survey answers. Such a virtual ubiquitous computing environment can be effectively used for system architects to walk through and present various scenarios, to validate interaction usability and simply as software testing platforms.

Published in: IEEE Proceedings. VR 2005. Virtual Reality, 2005

In this paper, we suggest a new approach for authoring tangible augmented reality applications, called 'immersive authoring.' The approach allows the user to carry out the authoring tasks within the AR application being built, so that the development and testing of the application can be done concurrently throughout the development process. We describe the functionalities and the interaction design for the proposed authoring system that are specifically targeted for intuitive specification of scenes and various object behaviors. Several cases of applications developed using the authoring system are presented. A small pilot user study was conducted to compare the proposed method to a non-immersive approach, and the results have shown that the users generally found it easier and faster to carry out authoring tasks in the immersive environment.

Presence is one of the goals of many virtual reality systems. Historically, in the context of virtual reality, the concept of presence has been associated much with spatial perception (bottom up process) as its informal definition of "feeling of being there" suggests. However, recent studies in presence have challenged this view and attempted to widen the concept to include psychological immersion, thus linking more high level elements (processed in a top down fashion) to presence such as story and plots, flow, attention and focus, identification with the characters, emotion, etc. In this paper, we experimentally studied the relationship between two content elements, each representing the two axis of the presence dichotomy, perceptual cues for spatial presence and sustained attention for (psychological) immersion. Our belief was that spatial perception or presence and a top down processed concept such as voluntary attention have only a very weak relationship, thus our experimental hypothesis was that sustained attention would positively affect spatial presence in a virtual environment with impoverished perceptual cues, but have no effect in an environment rich in them. In order to confirm the existence of the sustained attention in the experiment, fMRI of the subjects were taken and analyzed as well. The experimental results showed that that attention had no effect on spatial presence, even in the environment with impoverished spatial cues.

One of the goals in the design of virtual environments (VE) is to give the user the feeling of existence within the VE, known as presence. Employing multimodality is one way to increase presence, and as such, numerous multimodal input and output devices have been used in the context of virtual reality (VR). However, the simulation and investigation into the effects of the wind (or air flow) has not been treated much in the VR research community. In this paper, we introduce a wind display system, called the "WindCube," designed for virtual reality applications. The WindCube consists of a number of small fans attached to a cubical structure in which a VR system user interacts with the VE. We first discuss the design parameters of the proposed display device such as the type of the fan used, and the appropriate number, locations and directions of the fans in relation to providing the minimum level of the wind effects and enhanced presence. In order to simulate the effects of the wind, a wind field is first specified within the virtual environment. We describe how the specified wind field is rendered to the user through the proposed device. Finally, we investigate the effects of the proposed wind display to user felt presence through an experiment. It is our belief that wind display is very important and cost effective modality to consider and employ, because it involves "air," a medium that makes the VE felt more "livable," in contrast to many VE's that looks vaccum.

One of the goals and means of realizing virtual reality is through multimodal interfaces, leveraging on the many sensory organs that humans possess. Among them, the tactile sense is important and useful for close range interaction and manipulation tasks. In this paper, we explore this possibility using a vibro-tactile device on the whole body for simulating collision between the user and virtual environment. We first experimentally verify the effect of enhanced user felt presence by employing localized vibration feedback alone on collision, and further investigate how to effectively provide the sense of collision using the vibro-tactile display in different ways. In particular, we test the effects of using a vibration feedback model (for simulating collision with different object materials), saltation, and simultaneous use of 3D sound toward spatial presence and perceptual realism. The results have shown that employing the proposed vibro-tactile interface did enhance the sense of presence, especially when combined with 3D sound. Furthermore, the use of saltation also helped the user detect and localize the point of contact more correctly. The use of the vibration feedback model was not found to be significantly effective, and sometimes even hindered the correct sense of collision primarily due to the limitation of the vibrotactile display device.

One of the goals and defining characteristics of virtual reality systems is to create "presence" and fool the user into believing that one is, or is doing something "in" the synthetic environment. Most research and papers on presence to date have been directed toward coming up with the definitions of presence, and based on them, identifying key elements that affect presence. We carried out an elaborate experiment in which presence levels were measured (with subjective questionnaire) in test virtual worlds configured with different combinations of six visual presence elements.

This research explores different ways to use features of one's own body for interacting with computers. Such "body-based" interfaces may find good uses in wearable computing or virtual reality systems as part of a 3D multi-modal interface in the future, freeing the user from holding interaction devices. Four types of body-based interfaces have been identified: Body-inspired metaphor (BIM); Body-as-interaction-surface (BAIS); Mixed mode (MM); and Object mapping (OM). These four body-based interfaces were applied to a few different applications (and associated tasks) and were tested for their performance and preference. It was generally found that, among the four, the BIM exhibited low error rates, but produced relatively longer task completion times and significant fatigue. The BAIS method had the contrasting character of higher error rates, but shorter task completion times and lower intuitiveness. The OM method exhibited high error rates, longer completion times, and much fatigue. Overall, the MM was superior in terms of both performance and preference as it combined the merits of the above three methods. Thus, it is expected, for applications with many associated tasks, a careful division of tasks among those that have natural semantic links to body parts and those that do not, is necessary to design the most performing body-based interface.

We address the problem of teleoperating a mobile robot using shared autonomy: an on-board controller performs obstacle avoidance while the operator uses the manipulandum of a haptic probe to designate the desired speed and rate of turn. Sensors on the robot are used to measure obstacle range information. We describe a strategy to convert such range information into forces, which are reflected to the operator's hand, via the haptic probe. This haptic information provides feedback to the operator in addition to imagery from a front-facing camera mounted on the mobile robot. Extensive experiments with a user population show that the added haptic feedback significantly improves operator performance in several ways (reduced collisions, increased minimum distance between the robot and obstacles) without a significant increase in navigation time.

Development of virtual reality systems requires iterations of specification, implementation and evaluation. Since correct evaluations of immersive VR systems require the tedious process of wearing many devices, there exist both temporal and spatial gaps between the implementation and evaluation stage, and this usually causes delay and inefficiency in the development process. In order to overcome this gap, there have been several approaches to constructing or modeling the physical aspects of the virtual world (or objects) within the virtual environment. However, modeling their behaviors is still carried out in conventional (2D) programming environments.This paper proposes an interaction model, and interfaces for specifying (and modifying) object behavior, within the virtual environment, based on an underlying virtual object model. The interaction model follows the concept of programming by demonstration, and based on it, we have built a system called the PiP (Programming virtual object behavior in virtual reality Program) "in" which a user can create, modify, test, and save object behaviors. We illustrate examples of interactive virtual worlds constructed using the PiP, and discuss its merits and shortcomings as a content development platform.

Usability has become one of the key ingredients in making virtual reality (VR) systems work, and a big part of a usable VR system is the design of effective interface/interaction schemes. We investigate the usability of various menu presentation and multi-modal selection schemes in immersive virtual environments. We have identified five major menu displays and 13 possible menu selection methods; among them, we have completed the usability testing for two interaction methods across the five display methods: (1) position control by tracking and command-giving by button-pressing, and (2) both position control and command-giving by gesture recognition. Other interaction methods, including by voice, remain to be investigated.

In this paper, we present "PODIUM (POstech Distributed virtual Music environment)", a distributed virtual environment that allows users to participate in a shared space and play music with other participants in a collaborative manner. In addition to playing virtual instruments, users can communicate and interact in various ways to enhance the collaboration and, thus, the quality of the music played together. Musical messages are generated note by note through interaction with the keyboard, mouse, and other devices, and transmitted through an IP-multicasting network among participants. In addition to such note-level information, additional messages for visualization, and interaction are supported. Real world based visualization has been chosen, against, for instance, abstract music world based visualization, to promote "co-presence" (e.g. recognize and interact with other players), which is deemed important for collaborative music production. In addition to the entertainment purpose, we hope that DVME will find great use in casual practice sessions for even professional performers/orchestras/bands.Since even a slight interruption in the flow of the music or out - of-synch graphics and sound would dramatically decrease utility of the system, we employ various techniques to minimize the network delay. An adapted server-client architecture and UDP' s are used to ensure fast packet deliveries and reduce the data bottleneck problem. Time-critical messages such as MIDI messages are multicasted among clients, and the less time-critical and infrequently updated messages are sent through the server. Predefined animations of avatars are invoked by interpreting the musical messages. Using the latest graphics and sound processing hardware, and by maintaining an appropriate scene complexity, and a frame rate sufficiently higher than the fastest note duration, the time constraint for graphics and sound synchronization can be met. However, we expect the network delay could cause considerable problems when the system is scaled up for many users and processing simultaneous notes (for harmony). To assess the scalability, we carried out a performance analysis of our system model to derive the maximum number of simultaneous participants. For example, according to our data, about 50 participants should be able to play together without significant disruption, each using one track with five simultaneous notes and for playing a musical piece at a speed of 16 ticks per second in a typical PC/LAN environment.In hopes of enhancing the feeling of "co-presence" among participants, a simple sound localization technique is used to compute panning and relative volumes from positions and orientations of participants. This reduced sound localization model is used also in order to minimize the computational cost and the network traffic. Participants can send predefined messages by interacting with the keyboard, mouse, and other input devices. All of the predefined messages are mapped into simple avatar motions, such as playing various types of instruments (players), making applause (audience), and conducting gestures (conductors). We believe that for coordinated music performance, indirect interaction will be the main interaction method, for example, exchanging particular gestures, signals, and voice commands to synchronize music, confirming and reminding expression of the upcoming portion of the music, and just exchanging glances to enjoy each others' emotion. In this view, there would be mainly three groups of participants: conductor, players, and the audience, playing different roles, but creating co-presence together through mutual recognition. We ran a simple experiment comparing the music performance of two groups of participants, one provided with co-presence cues and the other without, and found no performance edge by the group with the co-presence cues. Such a result can serve as one guideline for building music-related VR applications.

For real-time performance, virtual reality systems often employ various performance optimization techniques. One of the most popular methods is using geometric models with different “Levels of Detail (LOD)”. In a previous paper by this author [1], we have proposed to use software engineering principles such as the concept of hierarchical and incremental modeling, and simultaneous consideration of form, function and behavior for modeling VR objects. Each refinement stage driven by such a modeling philosophy produces step-by-step form, function and behavior specifications of a VR object. We can make good use of these by-products as LOD for adaptive display and simulation by additionally specifying conditions for LOD switching. These specifications can be simulated and analyzed in advance for an estimation of performance for a given VR execution environment. Such an engineering process deals with behavior and geometry together; different geometric LOD may possess different behaviors and vice versa. A certain function or behavior might dictate an inclusion of a particular geometric feature that may not be possible to preserve, if the geometric LOD were to created in a bottom-up fashion (e.g. using the mesh simplification algorithms). We demonstrate our approach by modeling an automobile object with three levels of geometric and behavior detail in a top-down manner, simulate their instances in a small virtual town, and based on the simulation result, make predictions to the maximum allowable number of vehicles that will maintain an acceptable frame rate if executed in a faster simulation environment. We believe that our approach combines the idea of hierarchical refinement of virtual objects and the use of LOD in a very natural and intuitive manner.

Specification and validation of a real-time system are often based on making simplistic predictions and assumptions about relevant behavior of the external environment and the controlled device interacting with it. However, in many cases, real-time systems physically interact with other external objects in a complex manner in a dynamically changing world and thus, their form (e.g. physical properties such as shape, mass, material, and configuration) can play a critical role in producing a correct specification and obtaining realistic simulation output. We present ASADAL/PROTO, a specification and simulation tool for real-time systems, that takes form into account in addition to the function and behavior handled by ASADAL/SIM, its predecessor. Simulation of the control system specification runs in conjunction with the environmental simulation, and the resulting interactive behavior of the controlled system is observed visually for analysis. Our vision is to incrementally model, simulate and analyze all three views (behavior, function, and form) of real-time system specification for its increased level of confidence at early prototyping stage.

Developing a virtual reality application is a dificult task.FOT one thing, it requires an in-depth knowledge and understanding in many different disciplines.However, the most dificult part lies in the effective management of simultaneous design of form, function and behavior of virtual objects.Up to now, only little focus has been given to methods and tools for rapid prototyping VR software and modeling virtual objects.Most VR applications are developed by modeling the needed objects on conventional CAD systems first, then by programming their functions and behavior using low-level graphics libraries OT high level simulation packages.In this paper, we advocate for a structural and pedagogical approach to developing VR applications, and add the component of "form" to an existing real-time system development methodology so that the resulting methodology (and the prototyping environment) supports simultaneous and hierarchical engineering of form, function and behavior.Using the proposed methodology and supporting tool, it is our hope that VR maintained effectively.

We report on the development of a knowledge representation model, which is based on the SHARED object model reported in Shared Workspaces for Computer-Aided Collaborative Engineering (Wong, A. and Sriram, D., Technical Report, IESL 93-06, Intelligent Engineering Systems Laboratory, Department of Civil Engineering, MIT, March, 1993) and Research in Engineering Design (Wong, A. and Sriram, D., SHARED: An Information Model for Cooperative Product Development, 1993, Fall, 21-39). Our current model is implemented as a layered scheme, that incorporates both an evolving artifact and its associated design process. To represent artifacts as they evolve, we define objects recursively without a pre-defined granularity on this recursive decomposition. This eliminates the need for translations between levels of abstraction in the design process. The SHARED model extends traditional OOP in three ways: by allowing explicit relationship classes with inheritance hierarchies; by permitting constraints to be associated with objects and relationships; and by comparing `similar' objects at three different levels (form, function and behavior).

Despite considerable advancement in software engineering methods during the past three decades, requirements engineering of large and complex software systems still remains a difficult and active research problem. One such difficulty lies in developing correct and useful methods for the validation and verification of real-time software specifications. One way of analyzing and validating/verifying software specifications is to mathematically derive or prove desired system properties based on formal specification languages. A full scale system analysis using such formal methods is limited in practice because of the required mathematical skills and computational costs. Formal methods are often used to check only a few very critical real-time properties. Simulation is a complementary approach to testing various system characteristics and validating user requirements. It is especially good for providing a rough picture of final system behavior. This paper presents ASADAL/SIM, a tool for multi-level simulation and analysis of real-time software specifications. It is a subsystem of a larger computer-aided real-time software development environment called ASADAL, and complements ASADAL/PROVER, another subsystem of ASADAL which is a formal verification module.1. With ASADAL/SIM, simulation primitives can be added to evolving specifications in order to assign stochastic behaviors to external entities and internal processes, and to build a simulation model. ASADAL/SIM can execute the model and, at the same time, demonstrate the final system behavior by graphically showing internal workings of the system; catch undesirable system behaviors with breakpoints; and present various analytical results and system statistics ASADAL/SIM, following ASADAL's philosophies of hierarchical system modeling and early system validation, allows users to simulate ‘evolving’ specifications at different, mixed, and wide levels of detail. In particular, algorithmic details may be specified for low level behavioral blocks, and simulated with abstract entities yet to be refined to such a level. This facilitates the tracking of critical data values at the specification level, and eases the next transformation into code level implementation. With ASADAL/SIM, ASADAL becomes an effective and comprehensive supporting tool for various existing software engineering approaches, particularly top-down refinement and incremental development practices. © 1998 John Wiley & Sons, Ltd.

The main objective of assembly planning is to determine a sequence of assembling a product with respect to its geometric and resource constraints. Recent strides toward concurrent engineering have called for a tighter integration of assembly planning with design, because often during assembly planning, one realizes that a significant amount of assembly cost can be cut by redesigning the product itself. Instead of "retroactively" redesigning a product upon assembly planning, a concurrent engineering platform must allow its users to evaluate designs, and "proactively" explore different alternatives at various levels of abstractions so that one can zoom in on "promising" design paths and design right within the first few design cycles. This paper presents such an integrated system in which assembly planning becomes an essential part of the design exploration process. An algorithm for assembly planning during design for such an environment is developed. Designing for assembly is more effective and correct because, for a particular sequence of assembly, different design alternatives can be created and evaluated.

The authors introduce a quantitative shape metric for planar parts, called feedability, based on a stochastic parts feeding algorithm reported by K. Goldberg et al., (1990,1991). Although feedability is only one possible metric for design-for-assembly (DFA), it relates one aspect of assembly cost directly to part geometry. Therefore, based on feedability, a set of geometric redesign strategies can be produced. The application of feedability to DFA is illustrated and compared with other DFA methods.>