Personal computers and handheld devices provide keyboard shortcuts and swipe gestures to enable users to efficiently switch between applications, whereas today’s virtual reality (VR) systems do not. In this work, we present an exploratory study on user interface aspects to support efficient switching between worlds in VR. We created eight interfaces that afford previewing and selecting from the available virtual worlds, including methods using portals and worlds-in-miniature (WiMs). To evaluate these methods, we conducted a controlled within-subjects empirical experiment (N=22) where participants frequently transitioned between six different environments to complete an object collection task. Our quantitative and qualitative results show that WiMs supported rapid acquisition of high-level spatial information while searching and were deemed most efficient by participants while portals provided fast pre-orientation. Finally, we present insights into the applicability, usability, and effectiveness of the VR world switching methods we explored, and provide recommendations for their application and future context/world switching techniques and interfaces.

In most cases, retaining memories of things we have experienced in the past is desirable, but in some cases, we want to clear our minds so that we may focus completely on subsequent activities. When someone switches from one task to another, they commonly incur some “cognitive residue” where some of their cognitive resources such as working memory and attention remain devoted to their previous task even after they try to switch their focus to their new task. This residue could have a negative impact on their performance in the next task, and in such circumstances, it is important to reduce that residue. In this paper, we explore the concept of cognitive residue in the context of switching between virtual reality (VR) environments. We conducted a human-subject experiment (N=24) with a spatial recall task to investigate how different visual transitions might reduce participants’ spatial cognitive residue. In this instance, more errors on the recall task corresponds to less spatial cognitive residue. We found that transitions that lasted one minute successfully reduced spatial cognitive residue: they significantly reduced participants’ abilities to recall the positions of objects in their previous VE compared to an instantaneous cut transition. Additionally, for transitions that showed a nature scene, greater head movement significantly correlated with more spatial memory errors (i.e., less spatial cognitive residue). We discuss how these findings can be applied to support users transitioning between virtual tasks and environments in VR task switching scenarios.

Augmented reality (AR) head-mounted displays (HMDs) provide users with a view in which digital content is blended spatially with the outside world. However, one critical issue faced with such display technologies is misperception, i.e., perceptions of computer-generated content that differs from our human perception of other real-world objects or entities. Misperception can lead to mistrust in these systems and negative impacts in a variety of application fields. Although there is a considerable amount of research investigating either size, distance, or speed misperception in AR, far less is known about the relationships between these aspects. In this paper, we present an outdoor AR experiment (N = 20) using a HoloLens 2 HMD. Participants estimated size, distance, and speed of Familiar and Unfamiliar outdoor animals at three distances (30, 60, 90 meters). To investigate whether providing information about one aspect may influence another, we divided our experiment into three phases. In Phase I, participants estimated the three aspects without any provided information. In Phase II, participants were given accurate size information, then asked to estimate distance and speed. In Phase III, participants were given accurate distance and size information, then asked to estimate speed. Our results show that estimates of speed in particular of the Unfamiliar animals benefited from provided size information, while speed estimates of all animals benefited from provided distance information. We found no support for the assumption that distance estimates benefited from provided size information.

It is our great pleasure to welcome you to the 31st IEEE Conference on Virtual Reality and 3D User Interfaces, the premier international conference focused on the latest research in these domains. We are delighted to host IEEE VR 2024 as the first fully in-person VR since the 2020 COVID-19 pandemic and look forward to seeing, learn, share, and hang out with our vibrant community after these challenging years.

It is not uncommon for science fiction movies to portray futuristic user interfaces that can only be realized decades later with state-of-the-art technology. In this work, we present a prototypical augmented reality (AR) installation that was inspired by the movie The Time Machine (2002). It consists of a transparent screen that acts as a window through which users can see the stereoscopic projection of a three-dimensional virtual human (VH). However, there are some key differences between the vision of this technology and the way VHs on these displays are actually perceived. In particular, the additive light model of these displays causes darker VHs to appear more transparent, while light in the physical environment further increases transparency, which may affect the way VHs are perceived, to what degree they are trusted, and the distances one maintains from them in a spatial setting. In this paper, we present a user study in which we investigate how transparency in the scope of transparent AR screens affects the perception of a VH’s appearance, social presence with the VH, and the social space around users as defined by proxemics theory. Our results indicate that appearances are comparatively robust to transparency, while social presence improves in darker physical environments, and proxemic distances to the VH largely depend on one’s distance from the screen but are not noticeably affected by transparency. Overall, our results suggest that such transparent AR screens can be an effective technology for facilitating social interactions between users and VHs in a shared physical space.

The expression of human emotion is integral to social interaction, and in virtual reality it is increasingly common to develop virtual avatars that attempt to convey emotions by mimicking these visual and aural cues, i.e. the facial and vocal expressions. However, errors in (or the absence of) facial tracking can result in the rendering of incorrect facial expressions on these virtual avatars. For example, a virtual avatar may speak with a happy or unhappy vocal inflection while their facial expression remains otherwise neutral. In circumstances where there is conflict between the avatar's facial and vocal expressions, it is possible that users will incorrectly interpret the avatar's emotion, which may have unintended consequences in terms of social influence or in terms of the outcome of the interaction. In this paper, we present a human-subjects study (N = 22) aimed at understanding the impact of conflicting facial and vocal emotional expressions. Specifically we explored three levels of emotional valence (unhappy, neutral, and happy) expressed in both visual (facial) and aural (vocal) forms. We also investigate three levels of head scales (down-scaled, accurate, and up-scaled) to evaluate whether head scale affects user interpretation of the conveyed emotion. We find significant effects of different multimodal expressions on happiness and trust perception, while no significant effect was observed for head scales. Evidence from our results suggest that facial expressions have a stronger impact than vocal expressions. Additionally, as the difference between the two expressions increase, the less predictable the multimodal expression becomes. For example, for the happy-looking and happy-sounding multimodal expression, we expect and see high happiness rating and high trust, however if one of the two expressions change, this mismatch makes the expression less predictable. We discuss the relationships, implications, and guidelines for social applications that aim to leverage multimodal social cues.

Speech perception is optimal in quiet environments, but noise can impair comprehension and increase errors. In these situations, lip reading can help, but it is not always possible, such as during an audio call or when wearing a face mask. One approach to improve speech perception in these situations is to use an artificial visual lip reading aid. In this paper, we present a user study (N = 17) in which we compared three levels of audio stimuli visualizations and two levels of modulating the appearance of the visualization based on the speech signal, and we compared them against two control conditions: an audio-only condition, and a real human speaking. We measured participants’ speech reception thresholds (SRTs) to understand the effects of these visualizations on speech perception in noise. These thresholds indicate the decibel levels of the speech signal that are necessary for a listener to receive the speech correctly 50% of the time. Additionally, we measured the usability of the approaches and the user experience. We found that the different artificial visualizations improved participants’ speech reception compared to the audio-only baseline condition, but they were significantly poorer than the real human condition. This suggests that different visualizations can improve speech perception when the speaker’s face is not available. However, we also discuss limitations of current plug-and-play lip sync software and abstract representations of the speaker in the context of speech perception.

Various reasons exist why humans desire to magnify portions of our visually perceived surroundings, e.g., because they are too far away or too small to see with the naked eye. Different technologies are used to facilitate magnification, from telescopes to microscopes using monocular or binocular designs. In particular, modern digital cameras capable of optical and/or digital zoom are very flexible as their high-resolution imagery can be presented to users in real-time with displays and interfaces allowing control over the magnification. In this paper, we present a novel design space of intuitive augmented reality (AR) magnifications where an AR head-mounted display is used for the presentation of real-time magnified camera imagery. We present a user study evaluating and comparing different visual presentation methods and AR interaction techniques. Our results show different advantages for unimanual, bimanual, and situated AR magnification window interfaces, near versus far vergence distances for the image presentation, and five different user interfaces for specifying the scaling factor of the imagery.

Extended reality (XR) technologies, such as virtual reality (VR) and augmented reality (AR), provide users, their avatars, and embodied agents a shared platform to collaborate in a spatial context. Although traditional face-to-face communication is limited by users’ proximity, meaning that another human’s non-verbal embodied cues become more difficult to perceive the farther one is away from that person, researchers and practitioners have started to look into ways to accentuate or amplify such embodied cues and signals to counteract the effects of distance with XR technologies. In this article, we describe and evaluate the Big Head technique, in which a human’s head in VR/AR is scaled up relative to their distance from the observer as a mechanism for enhancing the visibility of non-verbal facial cues, such as facial expressions or eye gaze. To better understand and explore this technique, we present two complimentary human-subject experiments in this article. In our first experiment, we conducted a VR study with a head-mounted display to understand the impact of increased or decreased head scales on participants’ ability to perceive facial expressions as well as their sense of comfort and feeling of “uncannniness” over distances of up to 10 m. We explored two different scaling methods and compared perceptual thresholds and user preferences. Our second experiment was performed in an outdoor AR environment with an optical see-through head-mounted display. Participants were asked to estimate facial expressions and eye gaze, and identify a virtual human over large distances of 30, 60, and 90 m. In both experiments, our results show significant differences in minimum, maximum, and ideal head scales for different distances and tasks related to perceiving faces, facial expressions, and eye gaze, and we also found that participants were more comfortable with slightly bigger heads at larger distances. We discuss our findings with respect to the technologies used, and we discuss implications and guidelines for practical applications that aim to leverage XR-enhanced facial cues.

Human speech perception is generally optimal in quiet environments, however it becomes more difficult and error prone in the presence of noise, such as other humans speaking nearby or ambient noise. In such situations, human speech perception is improved by speech reading, i.e., watching the movements of a speaker's mouth and face, either consciously as done by people with hearing loss or subconsciously by other humans. While previous work focused largely on speech perception of two-dimensional videos of faces, there is a gap in the research field focusing on facial features as seen in head-mounted displays, including the impacts of display resolution, and the effectiveness of visually enhancing a virtual human face on speech perception in the presence of noise. In this paper, we present a comparative user study ( N=21) in which we investigated an audio-only condition compared to two levels of head-mounted display resolution ( 1832×1920 or 916×960 pixels per eye) and two levels of the native or visually enhanced appearance of a virtual human, the latter consisting of an up-scaled facial representation and simulated lipstick (lip coloring) added to increase contrast. To understand effects on speech perception in noise, we measured participants' speech reception thresholds (SRTs) for each audio-visual stimulus condition. These thresholds indicate the decibel levels of the speech signal that are necessary for a listener to receive the speech correctly 50% of the time. First, we show that the display resolution significantly affected participants' ability to perceive the speech signal in noise, which has practical implications for the field, especially in social virtual environments. Second, we show that our visual enhancement method was able to compensate for limited display resolution and was generally preferred by participants. Specifically, our participants indicated that they benefited from the head scaling more than the added facial contrast from the simulated lipstick. We discuss relationships, implications, and guidelines for applications that aim to leverage such enhancements.

In a future of pervasive augmented reality (AR), AR systems will need to be able to efficiently draw or guide the attention of the user to visual points of interest in their physical-virtual environment. Since AR imagery is overlaid on top of the user's view of their physical environment, these attention guidance techniques must not only compete with other virtual imagery, but also with distracting or attention-grabbing features in the user's physical environment. Because of the wide range of physical-virtual environments that pervasive AR users will find themselves in, it is difficult to design visual cues that "pop out" to the user without performing a visual analysis of the user's environment, and changing the appearance of the cue to stand out from its surroundings. In this article, we present an initial investigation into the potential uses of dichoptic visual cues for optical see-through AR displays, specifically cues that involve having a difference in hue, saturation, or value between the user's eyes. These types of cues have been shown to be preattentively processed by the user when presented on other stereoscopic displays, and may also be an effective method of drawing user attention on optical see-through AR displays. We present two user studies: one that evaluates the saliency of dichoptic visual cues on optical see-through displays, and one that evaluates their subjective qualities. Our results suggest that hue-based dichoptic cues or "Forbidden Colors" may be particularly effective for these purposes, achieving significantly lower error rates in a pop out task compared to value-based and saturation-based cues.

Social priming is the idea that observations of a virtual human (VH) engaged in short social interactions with a real or virtual human bystander can positively influence users’ subsequent interactions with that VH. In this paper we investigate the question of whether the positive effects of social priming are limited to interactions with humanoid entities. For instance, virtual dogs offer an attractive candidate for non-humanoid entities, as previous research suggests multiple positive effects. In particular, real human dog owners receive more positive attention from strangers than non-dog owners.To examine the influence of such social priming we carried out a human-subjects experiment with four conditions: three social priming conditions where a participant initially observed a VH interacting with one of three virtual entities (another VH, a virtual pet dog, or a virtual personal robot), and a non-social priming condition where a VH (alone) was intently looking at her phone as if reading something. We recruited 24 participants and conducted a mixed-methods analysis. We found that a VH’s prior social interactions with another VH and a virtual dog significantly increased participants’ perceptions of the VHs’ affective attraction. Also, participants felt more inclined to interact with the VH in the future in all of the social priming conditions. Qualitatively, we found that the social priming conditions resulted in a more positive user experience than the non-social priming condition. Also, the virtual dog and the virtual robot were perceived as a source of positive surprise, with participants appreciating the non-humanoid interactions for various reasons, such as the avoidance of social anxieties sometimes associated with humans.

When medical caregivers transfer patients to another person’s care (a patient handoff), it is essential they effectively communicate the patient’s condition to ensure the best possible health outcomes. Emergency situations caused by mass casualty events (e.g., natural disasters) introduce additional difficulties to handoff procedures such as environmental noise. We created a projected mixed reality simulation of a handoff scenario involving a medical evacuation by air and tested how low, medium, and high levels of helicopter noise affected participants’ handoff experience, handoff performance, and behaviors. Through a human-subjects experimental design study (N = 21), we found that the addition of noise increased participants’ subjective stress and task load, decreased their self-assessed and actual performance, and caused participants to speak louder. Participants also stood closer to the virtual human sending the handoff information when listening to the handoff than they stood to the receiver when relaying the handoff information. We discuss implications for the design of handoff training simulations and avenues for future handoff communication research.

The closed design of virtual reality (VR) head-mounted displays substantially limits users’ awareness of their real-world surroundings. This presents challenges when another person in the same physical space needs to interrupt the VR user for a brief conversation. Such interruptions, e.g., tapping a VR user on the shoulder, can cause a disruptive break in presence (BIP), which affects their place and plausibility illusions, and may cause a drop in performance of their virtual activity. Recent findings related to the concept of diegesis, which denotes the internal consistency of an experience/story, suggest potential benefits of integrating registered virtual representations for physical interactors, especially when these appear internally consistent in VR. In this paper, we present a human-subject study we conducted to compare and evaluate five different diegetic and non-diegetic methods to facilitate cross-reality interruptions in a virtual office environment, where a user’s task was briefly interrupted by a physical person. We created a Cross-Reality Interaction Questionnaire (CRIQ) to capture the quality of the interaction from the VR user’s perspective. Our results show that the diegetic representations afforded reasonably high senses of co-presence, the highest quality interactions, the highest place illusions, and caused the least disruption of the participants’ virtual experiences. We discuss our findings as well as implications for practical applications that aim to leverage virtual representations to ease cross-reality interruptions.

Previous research on distance estimation in virtual reality (VR) has well established that even for geometrically accurate virtual objects and environments users tend to systematically mis-estimate distances. This has implications for Social VR, where it introduces variables in personal space and proxemics behavior that change social behaviors compared to the real world. One yet unexplored factor is related to the trend that avatars' embodied cues in Social VR are often scaled, e.g., by making one's head bigger or one's voice louder, to make social cues more pronounced over longer distances. In this paper we investigate how the perception of avatar distance is changed based on two means for scaling embodied social cues: visual head scale and verbal volume scale. We conducted a human-subject study employing a mixed factorial design with two Social VR avatar representations (full-body, head-only) as a between factor as well as three visual head scales and three verbal volume scales (up-scaled, accurate, down-scaled) as within factors. For three distances from social to far-public space, we found that visual head scale had a significant effect on distance judgments and should be tuned for Social VR, while conflicting verbal volume scales did not, indicating that voices can be scaled in Social VR without immediate repercussions on spatial estimates. We discuss the interactions between the factors and implications for Social VR.

Light-on-dark color schemes, so-called “Dark Mode,” are becoming more and more popular over a wide range of display technologies and application fields. Many people who have to look at computer screens for hours at a time, such as computer programmers and computer graphics artists, indicate a preference for switching colors on a computer screen from dark text on a light background to light text on a dark background due to perceived advantages related to visual comfort and acuity, specifically when working in low-light environments. In this article, we investigate the effects of dark mode color schemes in the field of optical see-through head-mounted displays (OST-HMDs), where the characteristic “additive” light model implies that bright graphics are visible but dark graphics are transparent . We describe two human-subject studies in which we evaluated a normal and inverted color mode in front of different physical backgrounds and different lighting conditions. Our results indicate that dark mode graphics displayed on the HoloLens have significant benefits for visual acuity and usability, while user preferences depend largely on the lighting in the physical environment. We discuss the implications of these effects on user interfaces and applications.

In this article, we investigate the effects of the physical influence of a virtual human (VH) in the context of face-to-face interaction in a mixed reality environment. In Experiment 1, participants played a tabletop game with a VH, in which each player takes a turn and moves their own token along the designated spots on the shared table. We compared two conditions as follows: the VH in the virtual condition moves a virtual token that can only be seen through augmented reality (AR) glasses, while the VH in the physical condition moves a physical token as the participants do; therefore the VH's token can be seen even in the periphery of the AR glasses. For the physical condition, we designed an actuator system underneath the table. The actuator moves a magnet under the table which then moves the VH's physical token over the surface of the table. Our results indicate that participants felt higher co-presence with the VH in the physical condition, and participants assessed the VH as a more physical entity compared to the VH in the virtual condition. We further observed transference effects when participants attributed the VH's ability to move physical objects to other elements in the real world. Also, the VH's physical influence improved participants' overall experience with the VH. In Experiment 2, we further looked into the question how the physical-virtual latency in movements affected the perceived plausibility of the VH's interaction with the real world. Our results indicate that a slight temporal difference between the physical token reacting to the virtual hand's movement increased the perceived realism and causality of the mixed reality interaction. We discuss potential explanations for the findings and implications for future shared mixed reality tabletop setups.

360-degree experiences such as cinematic virtual reality and 360-degree videos are becoming increasingly popular. In most examples, viewers can freely explore the content by changing their orientation. However, in some cases, this increased freedom may lead to viewers missing important events within such experiences. Thus, a recent research thrust has focused on studying mechanisms for guiding viewers' attention while maintaining their sense of presence and fostering a positive user experience. One approach is the utilization of diegetic mechanisms, characterized by an internal consistency with respect to the narrative and the environment, for attention guidance. While such mechanisms are highly attractive, their uses and potential implementations are still not well understood. Additionally, acknowledging the user in 360-degree experiences has been linked to a higher sense of presence and connection. However, less is known when acknowledging behaviors are carried out by attention guiding mechanisms. To close these gaps, we conducted a within-subjects user study with five conditions of no guide and virtual arrows, birds, dogs, and dogs that acknowledge the user and the environment. Through our mixed-methods analysis, we found that the diegetic virtual animals resulted in a more positive user experience, all of which were at least as effective as the non-diegetic arrow in guiding users towards target events. The acknowledging dog received the most positive responses from our participants in terms of preference and user experience and significantly improved their sense of presence compared to the non-diegetic arrow. Lastly, three themes emerged from a qualitative analysis of our participants' feedback, indicating the importance of the guide's blending in, its acknowledging behavior, and participants' positive associations as the main factors for our participants' preferences.

Virtual reality (VR) technologies provide a shared platform for collaboration among users in a spatial context. To enhance the quality of social signals during interaction between users, researchers and practitioners started augmenting users’ interpersonal space with different types of virtual embodied social cues. A prominent example is commonly referred to as the "Big Head" technique, in which the head scales of virtual interlocutors are slightly increased to leverage more of the display’s visual space to convey facial social cues. While beneficial in improving interpersonal social communication, the benefits and thresholds of human perception of facial cues and comfort in such Big Head environments are not well understood, limiting their usefulness and subjective experience.In this paper, we present a human-subject study that we conducted to understand the impact of an increased or decreased head scale in social VR on participants’ ability to perceive facial expressions as well as their sense of comfort and feeling of "uncanniness." We explored two head scaling methods and compared them with respect to perceptual thresholds and user preferences. We further show that the distance to interlocutors has an important effect on the results. We discuss implications and guidelines for practical applications that aim to leverage VR-enhanced social cues.

Past work in augmented reality has shown that temperature-associated AR stimuli can induce warming and cooling sensations in the user, and prior work in psychology suggests that a person’s body temperature can influence that person’s sense of subjective perception of duration. In this paper, we present a user study to evaluate the relationship between temperature-associated virtual stimuli presented on an AR-HMD and the user’s sense of subjective perception of duration and temperature. In particular, we investigate two independent variables: the apparent temperature of the virtual stimuli presented to the participant, which could be hot or cold, and the location of the stimuli, which could be in direct contact with the user, in indirect contact with the user, or both in direct and indirect contact simultaneously. We investigate how these variables affect the users’ perception of duration and perception of body and environment temperature by having participants make prospective time estimations while observing the virtual stimulus and answering subjective questions regarding their body and environment temperatures. Our work confirms that temperature-associated virtual stimuli are capable of having significant effects on the users’ perception of temperature, and highlights a possible limitation in the current augmented reality technology in that no secondary effects on the users’ perception of duration were observed.

Current virtual reality (VR) head-mounted displays (HMDs) are characterized by a low angular resolution that makes it difficult to make out details, leading to reduced legibility of text and increased visual fatigue. Light-on-dark graphics modes, so-called "dark mode" graphics, are becoming more and more popular over a wide range of display technologies, and have been correlated with increased visual comfort and acuity, specifically when working in low-light environments, which suggests that they might provide significant advantages for VR HMDs.In this paper, we present a human-subject study investigating the correlations between the color mode and the ambient lighting with respect to visual acuity and fatigue on VR HMDs. We compare two color schemes, characterized by light letters on a dark background (dark mode), or dark letters on a light background (light mode), and show that the dark background in dark mode provides a significant advantage in terms of reduced visual fatigue and increased visual acuity in dim virtual environments on current HMDs. Based on our results, we discuss guidelines for user interfaces and applications.

Collaboration in a group has the potential to achieve more effective solutions for challenging problems, but collaboration per se is not an easy task, rather a stressful burden if the collaboration partners do not communicate well with each other. While Intelligent Virtual Assistants (IVAs), such as Amazon Alexa, are becoming part of our daily lives, there are increasing occurrences in which we collaborate with such IVAs for our daily tasks. Although IVAs can provide important support to users, the limited verbal interface in the current state of IVAs lacks the ability to provide effective non-verbal social cues, which is critical for improving collaborative performance and reducing task load.In this paper, we investigate the effects of IVA embodiment on collaborative decision making. In a within-subjects study, participants performed a desert survival task in three conditions: (1) performing the task alone, (2) working with a disembodied voice assistant, and (3) working with an embodied assistant. Our results show that both assistant conditions led to higher performance over when performing the task alone, but interestingly the reported task load with the embodied assistant was significantly lower than with the disembodied voice assistant. We discuss the findings with implications for effective and efficient collaborations with IVAs while also emphasizing the increased social presence and richness of the embodied assistant.

While research and development related to robotics has been going on for decades, the past decade in particular has seen a marked increase in related efforts, in part due to technological advances, increased technological accessibility and reliability, and increased commercial availability. What have come to be known as social robots are now being used to explore novel forms of human-robot interaction, to understand social norms, and to test expectations and human responses. To capture the contributions of these research efforts, identify the current trends, and future directions, we systematically review 10 years of research in the field of social robotics between 2008 and 2018, which includes 86 publications with 70 user studies. We classify the past work based on the research topics and application areas, and provide information about the publications, their user studies, and the capabilities of the social robots utilized. We also discuss selected papers in detail and outline overall trends. Based on these findings, we identify some areas of potential future research.

Human gaze awareness is important for social and collaborative interactions. Recent technological advances in augmented reality (AR) displays and sensors provide us with the means to extend collaborative spaces with real-time dynamic AR indicators of one's gaze, for example via three-dimensional cursors or rays emanating from a partner's head. However, such gaze cues are only as useful as the quality of the underlying gaze estimation and the accuracy of the display mechanism. Depending on the type of the visualization, and the characteristics of the errors, AR gaze cues could either enhance or interfere with collaborations. In this paper, we present two human-subject studies in which we investigate the influence of angular and depth errors, target distance, and the type of gaze visualization on participants' performance and subjective evaluation during a collaborative task with a virtual human partner, where participants identified targets within a dynamically walking crowd. First, our results show that there is a significant difference in performance for the two gaze visualizations ray and cursor in conditions with simulated angular and depth errors: the ray visualization provided significantly faster response times and fewer errors compared to the cursor visualization. Second, our results show that under optimal conditions, among four different gaze visualization methods, a ray without depth information provides the worst performance and is rated lowest, while a combination of a ray and cursor with depth information is rated highest. We discuss the subjective and objective performance thresholds and provide guidelines for practitioners in this field.

Modern augmented reality (AR) head-mounted displays comprise a multitude of sensors that allow them to sense the environment around them. We have extended these capabilities by mounting two heat-wavelength infrared cameras to a Microsoft HoloLens, facilitating the acquisition of thermal data and enabling stereoscopic thermal overlays in the user's augmented view. The ability to visualize live thermal information opens several avenues of investigation on how that thermal awareness may affect a user's thermoception. We present a human-subject study, in which we simulated different temperature shifts using either heat vision overlays or 3D AR virtual effects associated with thermal cause-effect relationships (e.g., flames burn and ice cools). We further investigated differences in estimated temperatures when the stimuli were applied to either the user's body or their environment. Our analysis showed significant effects and first trends for the AR virtual effects and heat vision, respectively, on participants' temperature estimates for their body and the environment though with different strengths and characteristics, which we discuss in this paper.

Domestic animals have a long history of enriching human lives physically and mentally by filling a variety of different roles, such as service animals, emotional support animals, companions, and pets. Despite this, technological realizations of such animals in augmented reality (AR) are largely underexplored in terms of their behavior and interactions as well as effects they might have on human users' perception or behavior. In this paper, we describe a simulated virtual companion animal, in the form of a dog, in a shared AR space. We investigated its effects on participants' perception and behavior, including locomotion related to proxemics, with respect to their AR dog and other real people in the environment. We conducted a 2 by 2 mixed factorial human-subject study, in which we varied (i) the AR dog's awareness and behavior with respect to other people in the physical environment and (ii) the awareness and behavior of those people with respect to the AR dog. Our results show that having an AR companion dog changes participants' locomotion behavior, proxemics, and social interaction with other people who can or can not see the AR dog. We also show that the AR dog's simulated awareness and behaviors have an impact on participants' perception, including co-presence, animalism, perceived physicality, and dog's perceived awareness of the participant and environment. We discuss our findings and present insights and implications for the realization of effective AR animal companions.

Embodied virtual agents serving as patient simulators are widely used in medical training scenarios, ranging from physical patients to virtual patients presented via virtual and augmented reality technologies. Physical-virtual patients are a hybrid solution that combines the benefits of dynamic visuals integrated into a human-shaped physical form that can also present other cues, such as pulse, breathing sounds, and temperature. Sometimes in simulation the visuals and shape do not match. We carried out a human-participant study employing graduate nursing students in pediatric patient simulations comprising conditions associated with matching/non-matching of the visuals and shape.

This paper presents the implementation and evaluation of a 50,000-pose-sample-per-second, 6-degree-of-freedom optical head tracking instrument with motion-to-pose latency of 28μs and dynamic precision of 1-2 arcminutes. The instrument uses high-intensity infrared emitters and two duo-lateral photodiode-based optical sensors to triangulate pose. This instrument serves two purposes: it is the first step towards the requisite head tracking component in sub- 100μs motion-to-photon latency optical see-through augmented reality (OST AR) head-mounted display (HMD) systems; and it enables new avenues of research into human visual perception - including measuring the thresholds for perceptible real-virtual displacement during head rotation and other human research requiring high-sample-rate motion tracking. The instrument's tracking volume is limited to about 120×120×250 but allows for the full range of natural head rotation and is sufficient for research involving seated users. We discuss how the instrument's tracking volume is scalable in multiple ways and some of the trade-offs involved therein. Finally, we introduce a novel laser-pointer-based measurement technique for assessing the instrument's tracking latency and repeatability. We show that the instrument's motion-to-pose latency is 28μs and that it is repeatable within 1-2 arcminutes at mean rotational velocities (yaw) in excess of 500°/sec.

Presents the preface to the 2019 Virtual Reality Conference.

Redirected and amplified head movements have the potential to provide more natural interaction with virtual environments (VEs) than using controller-based input, which causes large discrepancies between visual and vestibular self-motion cues and leads to increased VR sickness. However, such amplified head movements may also exacerbate VR sickness symptoms over no amplification. Several general methods have been introduced to reduce VR sickness for controller-based input inside a VE, including a popular vignetting method that gradually reduces the field of view.

Intelligent Virtual Agents (IVAs) are becoming part of our everyday life, thanks to artificial intelligence technology and Internet of Things devices. For example, users can control their connected home appliances through natural voice commands to the IVA. However, most current-state commercial IVAs, such as Amazon Alexa, mainly focus on voice commands and voice feedback, and lack the ability to provide non-verbal cues which are an important part of social interaction. Augmented Reality (AR) has the potential to overcome this challenge by providing a visual embodiment of the IVA. In this paper we investigate how visual embodiment and social behaviors influence the perception of the IVA. We hypothesize that a user's confidence in an IVA's ability to perform tasks is improved when imbuing the agent with a human body and social behaviors compared to the agent solely depending on voice feedback. In other words, an agent's embodied gesture and locomotion behavior exhibiting awareness of the surrounding real world or exerting influence over the environment can improve the perceived social presence with and confidence in the agent. We present a human-subject study, in which we evaluated the hypothesis and compared different forms of IVAs with speech, gesturing, and locomotion behaviors in an interactive AR scenario. The results show support for the hypothesis with measures of confidence, trust, and social presence. We discuss implications for future developments in the field of IVAs.

Conventional Intelligent Virtual Agents (IVAs) focus primarily on the visual and auditory channels for both the agent and the interacting human: the agent displays a visual appearance and speech as output, while processing the human's verbal and non-verbal behavior as input. However, some interactions, particularly those between a patient and healthcare provider, inherently include tactile components. We introduce an Intelligent Physical-Virtual Agent (IPVA) head that occupies an appropriate physical volume; can be touched; and via human-in-the-loop control can change appearance, listen, speak, and react physiologically in response to human behavior. Compared to a traditional IVA, it provides a physical affordance, allowing for more realistic and compelling human-agent interactions. In a user study focusing on neurological assessment of a simulated patient showing stroke symptoms, we compared the IPVA head with a high-fidelity touch-aware mannequin that has a static appearance. Various measures of the human subjects indicated greater attention, affinity for, and presence with the IPVA patient, all factors that can improve healthcare training.

The field of intelligent virtual agents (IVAs) has evolved immensely over the past 15 years, introducing new application opportunities in areas such as training, health care, and virtual assistants. In this survey paper, we provide a systematic review of the most influential user studies published in the IVA conference from 2001 to 2015 focusing on IVA development, human perception, and interactions. A total of 247 papers with 276 user studies have been classified and reviewed based on their contributions and impact. We identify the different areas of research and provide a summary of the papers with the highest impact. With the trends of past user studies and the current state of technology, we provide insights into future trends and research challenges.

While research in the field of augmented reality (AR) has produced many innovative human-computer interaction techniques, some may produce physical and visual perceptions with unforeseen negative impacts on user performance. In a controlled human-subject study we investigated the effects of mismatched physical and visual perception on cognitive load and performance in an AR touching task by varying the physical fidelity (matching vs. non-matching physical shape) and visual mechanism (projector-based vs. HMD-based AR) of the representation. Participants touched visual targets on four corresponding physical-visual representations of a human head. We evaluated their performance in terms of touch accuracy, response time, and a cognitive load task requiring target size estimations during a concurrent (secondary) counting task. After each condition, participants completed questionnaires concerning mental, physical, and temporal demands; stress; frustration; and usability. Results indicated higher performance, lower cognitive load, and increased usability when participants touched a matching physical head-shaped surface and when visuals were provided by a projector from underneath.

The 2018 Virtual Reality Technical Achievement Award goes to Gregory F. Welch of the University of Central Florida, USA, in recognition for his contributions to human motion tracking and to mixed reality applications in medicine and training. Professor Welch has been working in Virtual Reality and Augmented Reality for more than 25 years. His method for single- constraint-at-a-time tracking with continuous automatic calibration has been cited as the reason UNC Chapel Hill’s HiBall tracking system was the fastest, lowest latency, most precise, and most accurate wide-area tracking system. Among his many publications, his “Introduction to the Kalman Filter” report has been cited over 8,000 times. Professor Welch co-developed the ideas and methods for unified image-based modeling and spatially immersive displays, Spatial Augmented Reality, and Shader Lamps. He later applied these methods to humanoid forms to achieve physical-virtual avatars, and groundbreaking patient simulators for training healthcare practitioners. The IEEE VGTC is pleased to award Gregory F. Welch the 2018 Virtual Reality Technical Achievement Award.

In this paper, we investigate the effects of the physical influence of a virtual human (VH) in the context of face-to-face interaction in augmented reality (AR). In our study, participants played a tabletop game with a VH, in which each player takes a turn and moves their own token along the designated spots on the shared table. We compared two conditions as follows: the VH in the virtual condition moves a virtual token that can only be seen through AR glasses, while the VH in the physical condition moves a physical token as the participants do; therefore the VH's token can be seen even in the periphery of the AR glasses. For the physical condition, we designed an actuator system underneath the table. The actuator moves a magnet under the table which then moves the VH's physical token over the surface of the table. Our results indicate that participants felt higher co-presence with the VH in the physical condition, and participants assessed the VH as a more physical entity compared to the VH in the virtual condition. We further observed transference effects when participants attributed the VH's ability to move physical objects to other elements in the real world. Also, the VH's physical influence improved participants' overall experience with the VH. We discuss potential explanations for the findings and implications for future shared AR tabletop setups.

Immersive computer-generated environments (aka virtual reality, VR ) are limited by the physical space around them, e.g., enabling natural walking in VR is only possible by perceptually-inspired locomotion techniques such as redirected walking (RDW). We introduce a completely new approach to imperceptible position and orientation redirection that takes advantage of the fact that even healthy humans are functionally blind for circa ten percent of the time under normal circumstances due to motor processes preventing light from reaching the retina (such as eye blinks) or perceptual processes suppressing degraded visual information (such as blink-induced suppression). During such periods of missing visual input, change blindness occurs, which denotes the inability to perceive a visual change such as the motion of an object or self-motion of the observer. We show that this phenomenon can be exploited in VR by synchronizing the computer graphics rendering system with the human visual processes for imperceptible camera movements, in particular to implement position and orientation redirection. We analyzed human sensitivity to such visual changes with detection thresholds, which revealed that commercial off-the-shelf eye trackers and head-mounted displays suffice to translate a user by circa 4 -- 9 cm and rotate the user by circa 2 -- 5 degrees in any direction, which could be accumulated each time the user blinks. Moreover, we show the potential for RDW, whose performance could be improved by approximately 50% when using our technique.

In 2008, Zhou et al. presented a survey paper summarizing the previous ten years of ISMAR publications, which provided invaluable insights into the research challenges and trends associated with that time period. Ten years later, we review the research that has been presented at ISMAR conferences since the survey of Zhou et al., at a time when both academia and the AR industry are enjoying dramatic technological changes. Here we consider the research results and trends of the last decade of ISMAR by carefully reviewing the ISMAR publications from the period of 2008-2017, in the context of the first ten years. The numbers of papers for different research topics and their impacts by citations were analyzed while reviewing them-which reveals that there is a sharp increase in AR evaluation and rendering research. Based on this review we offer some observations related to potential future research areas or trends, which could be helpful to AR researchers and industry members looking ahead.

We are pleased to present the technical papers for IEEE VR 2018: the 25th IEEE Conference on Virtual Reality and 3D User Interfaces, held March 18-22, 2018 in Reutlingen, Germany. IEEE VR 2018 features two categories of submissions: (i) VR Journal Papers, and (ii) VR Conference Papers. Both categories have their own program committees, submission processes, and review processes. This year, 178 submissions were submitted to the Journal track from which 29 were accepted as articles to IEEE TVCG (16.3%). All of these will be presented at the IEEE VR 2018 conference, along with 6 additional papers in the VR area that were published in IEEE TVCG during the past year. Furthermore, 6 submission (3.4%) were recommended for a regular issue of TVCG with major revisions with reviewer continuity. Each of the papers in this special issue went through a rigorous two-round review process. All accepted journal papers are published in a special issue of IEEE Transactions on Visualization and Computer Graphics (TVCG). The Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces contains all of the accepted conference papers and the poster abstracts.

In this paper, we investigate factors and issues related to human locomotion behavior and proxemics in the presence of a real or virtual human in augmented reality (AR). First, we discuss a unique issue with current-state optical see-through head-mounted displays, namely the mismatch between a small augmented visual field and a large unaugmented periphery, and its potential impact on locomotion behavior in close proximity of virtual content. We discuss a potential simple solution based on restricting the field of view to the central region, and we present the results of a controlled human-subject study. The study results show objective benefits for this approach in producing behaviors that more closely match those that occur when seeing a real human, but also some drawbacks in overall acceptance of the restricted field of view. Second, we discuss the limited multimodal feedback provided by virtual humans in AR, present a potential improvement based on vibrotactile feedback induced via the floor to compensate for the limited augmented visual field, and report results showing that benefits of such vibrations are less visible in objective locomotion behavior than in subjective estimates of co-presence. Third, we investigate and document significant differences in the effects that real and virtual humans have on locomotion behavior in AR with respect to clearance distances, walking speed, and head motions. We discuss potential explanations for these effects related to social expectations, and analyze effects of different types of behaviors including idle standing, jumping, and walking that such real or virtual humans may exhibit in the presence of an observer.

We investigate the effect of vibrotactile feedback delivered to one's feet in an immersive virtual environment (IVE). In our study, participants observed a virtual environment where a virtual human (VH) walked toward the participants and paced back and forth within their social space. We compared three conditions as follows: participants in the “Sound” condition heard the footsteps of the VH; participants in the “Vibration” condition experienced the vibration of the footsteps along with the sounds; while participants in the “Mute” condition were not exposed to sound nor vibrotactile feedback. We found that the participants in the “Vibration” condition felt a higher social presence with the VH compared to those who did not feel the vibration. The participants in the “Vibration” condition also exhibited greater avoidance behavior while facing the VH and when the VH invaded their personal space.

We discuss the design and results of an experiment investigating Plausibility Illusion in virtual human (VH) interactions, in particular, the coherence of conversation with a VH. This experiment was performed in combination with another experiment evaluating two display technologies. As that aspect of the study is not relevant to this poster, it will be mentioned only in the Materials section. Participants who interacted with a low-coherence VH looked around the room markedly more than participants interacting with a high-coherence VH, demonstrating that the level of coherence of VHs can have a detectable effect on user behavior and that head and gaze behavior can be used to evaluate the quality of a VH interaction.

In this paper we report on an investigation of the effects of a self-avatar's visual similarity to a user's actual appearance, on their perceptions of the avatar in an immersive virtual reality (IVR) experience. We conducted a user study to examine the participant's sense of body ownership, presence and visual realism under three levels of avatar-owner visual similarity: (L1) an avatar reconstructed from real imagery of the participant's appearance, (L2) a cartoon-like virtual avatar created by a 3D artist for each participant, where the avatar shoes and clothing mimic that of the participant, but using a low-fidelity model, and (L3) a cartoon-like virtual avatar with a pre-defined appearance for the shoes and clothing. Surprisingly, the results indicate that the participants generally exhibited the highest sense of body ownership and presence when inhabiting the cartoon-like virtual avatar mimicking the outft of the participant (L2), despite the relatively low participant similarity. We present our experiment and main findings, also, discuss the potential impact of a self-avatar's visual differences on human perceptions in IVR.

Augmented reality (AR) enables the illusion of computer-generated virtual objects and humans co-existing with us in the real world. Virtual humans (VHs) in AR can further induce an illusion of physicality in the real world due to their form of presentation and their behavior, such as showing awareness of their surroundings. However, certain behaviors can cause a conflict that breaks this illusion, for example, when we see a VH passing through a physical object.

Abstract In augmented reality, people can feel the illusion of virtual humans (VHs) integrated into a real (physical) space. However, affordances of the real world and virtual contents might conflict, for example, when the VHs and real objects “collide” by occupying the same space. This implausible conflict can cause a break in presence in real–virtual human interactions. In this paper, we address an effort to avoid this conflict by maintaining the VH's spatial and behavioral coherence with respect to the physical objects or events (e.g., natural occlusions and appropriate help‐requesting behaviors to avoid implausible physical–virtual collisions). We present a human subject experiment examining the effects of the physical–virtual coherence on human perceptions, such as social/copresence and behaviors with the VH. The basic ideas, experimental design, and results supporting the benefit of the VH's spatial and behavioral coherence are presented and discussed.

We explore whether a peripheral observation of apparent mutual social presence between a real human (RH) and a virtual human (VH) can in turn increase a subject's sense of social presence with the VH. In other words, we explore whether social presence can “transfer” from one RH-VH interaction to another. Specifically, we carried out an experiment where human subjects were asked to play a game with a VH. As they entered the game room, approximately half of the subjects were exposed to a brief but apparently engaging conversation between an RH and the VH. The subjects who were exposed to the brief RH-VH interaction had significantly higher measures of both emotional connection and the attentional allocation dimension of social presence for the VH, compared to those who were not. We describe the motivation, the experiment, and the results.

While performing everyday interactions, we often incidentally touch and move objects in subtle ways. These objects are not necessarily directly related to the task at hand, and the movement of an object might even be entirely unintentional. If another person is touching the object at the same time, the movement can transfer through the object and be experienced — however subtly — by the other person. For example, when one person hands a drink to another, at some point both individuals will be touching the glass, and consequently exerting small (often unnoticed) forces on the other person. Despite the frequency of such subtle incidental movements of shared objects in everyday interactions, few have examined how these movements affect human-virtual human (VH) interaction. We ran an experiment to assess how presence and social presence are affected when a person experiences subtle, incidental movement through a shared real-virtual object. We constructed a real-virtual room with a table that spanned the boundary between the real and virtual environments. The participant was seated on the real side of the table, which visually extended into the virtual world via a projection screen, and the VH was seated on the virtual side of the table. The two interacted by playing a game of “Twenty Questions,” where one player asked the other a series of 20 yes/no questions to deduce what object the other player was thinking about. During the game, the “wobbly” group of subjects experienced subtle incidental movements of the real-virtual table: the entire real-virtual table tilted slightly away/toward the subject when the virtual/real human leaned on it. The control group also played the same game, except the table did not wobble. Results indicate that the wobbly group had higher presence and social presence with the virtual human in general, with statistically significant increases in presence, co-presence, and attentional allocation. We present the experiment and results, and discuss some potential implications for virtual human systems and some potential future studies.

We demonstrate a generalizable method for unified multitouch detection and response on a human head-shaped surface with a rear-projection animated 3D face. The method helps achieve hands-on touch-sensitive training with dynamic physical-virtual patient behavior. The method, which is generalizable to other non-parametric rear-projection surfaces, requires one or more infrared (IR) cameras, one or more projectors, IR light sources, and a rear-projection surface. IR light reflected off of human fingers is captured by cameras with matched IR pass filters, allowing for the localization of multiple finger touch events. These events are tightly coupled with the rendering system to produce auditory and visual responses on the animated face displayed using the projector(s), resulting in a responsive, interactive experience. We illustrate the applicability of our physical prototype in a medical training scenario.

In Augmented Reality (AR), visible misregistration can be caused by many inherent error sources, such as errors in tracking, calibration, and modeling. In this paper we present a novel pixel-wise closed-loop registration framework that can automatically detect and correct registration errors using a reference model comprised of the real scene model and the desired virtual augmentations. Registration errors are corrected in both global world space via camera pose refinement, and local screen space via pixel-wise corrections, resulting in spatially accurate and visually coherent registration. Specifically we present a registration-enforcing model-based tracking approach that weights important image regions while refining the camera pose estimates (from any conventional tracking method) to achieve better registration, even in the case of modeling errors. To deal with remaining errors, which can be rigid or non-rigid, we compute the optical flow between the camera image and the real model image rendered with the refined pose, enabling direct screen-space pixel-wise corrections to misregistration. The estimated flow field can be applied to improve registration in two distinct ways: (1) forward warping of modeled on-real-object-surface augmentations (e.g., object re-texturing) into the camera image, leading to surface details that are not present in the virtual object; and (2) backward warping of the camera image into the real scene model, preserving the full use of the dense geometry buffer (depth in particular) provided by the combined real-virtual model for registration, leading to pixel accurate real-virtual occlusion. We discuss the trade-offs between, and different use cases of, forward and backward warping with model-based tracking in terms of specific properties for registration. We demonstrate the efficacy of our approach with both simulated and real data.

Large networks of cameras have been increasingly employed to capture dynamic events for tasks such as surveillance and training. When using active cameras to capture events distributed throughout a large area, human control becomes impractical and unreliable. This has led to the development of automated approaches for online camera control. We introduce a new automated camera control approach that consists of a stochastic performance metric and a constrained optimization method . The metric quantifies the uncertainty in the state of multiple points on each target. It uses state-space methods with stochastic models of target dynamics and camera measurements. It can account for occlusions, accommodate requirements specific to the algorithms used to process the images, and incorporate other factors that can affect their results. The optimization explores the space of camera configurations over time under constraints associated with the cameras, the predicted target trajectories, and the image processing algorithms. The approach can be applied to conventional surveillance tasks (e.g., tracking or face recognition), as well as tasks employing more complex computer vision methods (e.g., markerless motion capture or 3D reconstruction).

Welcome to the 21st IEEE Virtual Reality Conference, the premier international conference and exhibition on virtual reality!We are honored to host IEEE VR 2014 in Minneapolis, Minnesota, a vibrant center of culture, business, learning, and research in the upper midwest of the United States.The highlight of our outstanding technical program is the eighteen full-length research papers that are being published as a special issue of the IEEE Transactions on Visualization and Computer Graphics.We are indebted to the TVCG editor-in-chief, Ming Lin, for her vision and leadership in continuing to make this collaboration possible.Complementing these journal papers, our conference proceedings include ten short papers and fortyfour abstracts presenting an exciting collection of groundbreaking technical contributions.We thank this year's program chairs, Sabine Coquillart, Kiyoshi Kiyokawa, J. Edward Swan II, and Doug A. Bowman, for their dedicated leadership in soliciting papers, managing the multi-phase review process, and arranging the paper presentations at the conference.In addition to the long and short paper presentations, the week's technical events include the presentation of three panels, five workshops, three tutorials, thirty-one posters, thirteen research demonstrations, ten exhibits, and eight videos, bookended by keynote and capstone talks from VR visionaries Henry Fuchs and Hunter Hoffman.Accompanying these activities is the second annual Virtual Reality Doctoral Consortium, which with the generous support of the US National Science Foundation is providing a valuable mentoring opportunity to twelve promising PhD students working in the areas of virtual and augmented reality and 3D user interaction.We are further honored to again have the IEEE Symposium on 3D User Interfaces (3DUI) held jointly with the VR conference.The 9th installment of 3DUI features an outstanding program of papers, technotes, posters, and

The 20th IEEE Virtual Reality Conference (VR 2013) held March 16th - 20th spans five days, with the first two days including five workshops and three tutorials. The following three days includes a strong program of papers, panels, posters, demos, and videos. Of special note is that this year featured both a high number of, and a higher than usual quality of, submissions. This results in a packed program that did require the panel sessions to coincide with a few paper sessions.

The typical registration process in augmented reality CAR) consists of three independent consecutive stages: static calibration, dynamic tracking, and graphics overlay. The result is that the real-virtual registration is “open loop”-inaccurate calibration or tracking leads to misregistration that is seen by the users but not the system. To cope with this, we propose a general approach to “close the loop” in the displayed appearance by using the visual feedback of registration for pose tracking to achieve accurate registration. Specifically, a model-based method is introduced to simultaneously track and augment real objects in a closed-loop fashion, where the model is comprised of the combination of the real object to be tracked and the virtual object to be rendered. This method is applicable to paradigms including video-based AR, projector-based AR, and diminished reality. Both qualitative and quantitative experiments are presented to demonstrate the feasibility and effectiveness of our approach.

This paper presents an architecture to control avatars and virtual characters in remote interaction environments. A human-in-the-loop (interactor) metaphor provides remote control of multiple virtual characters, with support for multiple interactors and multiple observers. Custom animation blending routines and a gesture-based interface provide interactors with an intuitive digital puppetry paradigm. This paradigm reduces the cognitive and physical loads on the interactor while supporting natural bi-directional conversation between a user and the virtual characters or avatar counterparts. A multi-server-client architecture, based on a low-demand network protocol, connects the user environment, interactor station(s) and observer station(s). The associated system affords the delivery of personalized experiences that adapt to the actions and interactions of individual users, while staying true to each virtual character's personality and backstory. This approach has been used to create experiences designed for training, education, rehabilitation, remote presence and other-related applications.

Tracking and augmentation are usually handled in independent consecutive stages in augmented reality (AR). The result is that the real-virtual registration is “open loop”-inaccurate tracking leads to misregistration that is seen by the users but not the system. We propose a general approach to “close the loop” in the displayed appearance by using the visual feedback of registration for tracking. Specifically, a model-based method is introduced to simultaneously track and augment real objects in a closed-loop fashion, where the model is comprised of the combination of the real object to be tracked and the virtual object to be rendered. This method is applicable to paradigms including video-based AR, projector-based AR, and diminished reality.

The S&T Program Chairs are delighted to welcome you to ISMAR 2011, the 10th symposium on Mixed and Augmented Reality! This year's symposium continues a long tradition of ISMAR meetings, a series that itself followed a related series of IWAR, ISMR, and ISAR meetings. This year's Science and Technology (S&T) track comprises a mix of highly selective mixed and augmented reality research and related work. Specifically, in the S&T program this year you will find 26 papers, 27 posters, as well as a stimulating mixture of keynote talks, demonstrations, tutorials, workshops and the tracking competition. All of these elements of the program are the result of dedicated hard work by members of various conference committees and additional volunteers, and we would like to thank all of them for their generous efforts.

We introduce a general technique for blending imagery from multiple projectors on a tracked, moving, non-planar object. Our technique continuously computes visibility of pixels over the surfaces of the object and dynamically computes the per-pixel weights for each projector. This approach supports smooth transitions between areas of the object illuminated by different number of projectors, down to the illumination contribution of individual pixels within each polygon. To achieve real-time performance, we take advantage of graphics hardware, implementing much of the technique with a custom dynamic blending shader program within the GPU associated with each projector. We demonstrate the technique with some tracked objects being illuminated by three projectors.

Applications such as telepresence and training involve the display of real or synthetic humans to multiple viewers. When attempting to render the humans with conventional displays, non-verbal cues such as head pose, gaze direction, body posture, and facial expression are difficult to convey correctly to all viewers. In addition, a framed image of a human conveys only a limited physical sense of presence - primarily through the display's location. While progress continues on articulated robots that mimic humans, the focus has been on the motion and behavior of the robots. We introduce a new approach for robotic avatars of real people: the use of cameras and projectors to capture and map the dynamic motion and appearance of a real person onto a humanoid animatronic model. We call these devices Animatronic Shader Lamps Avatars (SLA).We present a proof-of-concept prototype comprised of a camera, a tracking system, a digital projector, and a life-sized styrofoam head mounted on a pan-tilt unit. The system captures imagery of a moving, talking user and maps the appearance and motion onto the animatronic SLA, delivering a dynamic, real-time representation of the user to multiple viewers.

We present a novel calibration framework for multi-projector displays that achieves continuous geometric calibration by estimating and refining the poses of all projectors in an ongoing fashion during actual display use. Our framework provides scalability by operating as a distributed system of "intelligent" projector units: projectors augmented with rigidly-mounted cameras, and paired with dedicated computers. Each unit interacts asynchronously with its peers, leveraging their combined computational power to cooperatively estimate the poses of all of the projectors. In cases where the projection surface is static, our system is able to continuously refine all of the projector poses, even when they change simultaneously.

Abstract We conducted an experiment with a posttest, between‐subjects design to evaluate the potential of emerging 3D telepresence technology to support collaboration in emergency health care. 3D telepresence technology has the potential to provide richer visual information than do current 2D video conferencing techniques. This may be of benefit in diagnosing and treating patients in emergency situations where specialized medical expertise is not locally available. The experimental design and results concerning information behavior are presented in the article “Exploring the Potential of Video Technologies for Collaboration in Emergency Medical Care: Part I. Information Sharing” (Sonnenwald et al., this issue). In this article, we explore paramedics' task performance during the experiment as they diagnosed and treated a trauma victim while working alone or in collaboration with a physician via 2D videoconferencing or via a 3D proxy. Analysis of paramedics' task performance shows that paramedics working with a physician via a 3D proxy performed the fewest harmful interventions and showed the least variation in task performance time. Paramedics in the 3D proxy condition also reported the highest levels of self‐efficacy. Interview data confirm these statistical results. Overall, the results indicate that 3D telepresence technology has the potential to improve paramedics' performance of complex medical tasks and improve emergency trauma health care if designed and implemented appropriately.

Abstract We are investigating the potential of 3D telepresence, or televideo, technology to support collaboration among geographically separated medical personnel in trauma emergency care situations. 3D telepresence technology has the potential to provide richer visual information than current 2D videoconferencing techniques. This may be of benefit in diagnosing and treating patients in emergency situations where specialized medical expertise is not locally available. The 3D telepresence technology does not yet exist, and there is a need to understand its potential before resources are spent on its development and deployment. This poses a complex challenge. How can we evaluate the potential impact of a technology within complex, dynamic work contexts when the technology does not yet exist? To address this challenge, we conducted an experiment with a posttest, between‐subjects design that takes the medical situation and context into account. In the experiment, we simulated an emergency medical situation involving practicing paramedics and physicians, collaborating remotely via two conditions: with today's 2D videoconferencing and a 3D telepresence proxy. In this article, we examine information sharing between the attending paramedic and collaborating physician. Postquestionnaire data illustrate that the information provided by the physician was perceived to be more useful by the paramedic in the 3D proxy condition than in the 2D condition; however, data pertaining to the quality of interaction and trust between the collaborating physician and paramedic show mixed results. Postinterview data help explain these results.

We introduce a novel approach to on-line structure from motion, using a pipelined pair of extended Kalman filters to improve accuracy with a minimal increase in computational cost. The two filters, a leading and a following filter, run concurrently on the same measurements in a synchronized producer-consumer fashion, but offset from each other in time. The leading filter estimates structure and motion using all of the available measurements from an optical flow based 2D tracker, passing the best 3D feature estimates, covariances, and associated measurements to the following filter, which runs several steps behind. This pipelined arrangement introduces a degree of noncausal behavior, effectively giving the following filter the benefit of decisions and estimates made several steps ahead. This means that the following filter works with only the best features, and can begin full 3D estimation from the very start of the respective 2D tracks. We demonstrate a reduction of more than 50% in mean reprojection errors using this approach on real data.

The appearance of a scene is a function of the scene contents, the lighting, and the camera pose. A set of n-pixel images of a non-degenerate scene captured from different perspectives lie on a 6D nonlinear manifold in Rn. In general, this nonlinear manifold is complicated and numerous samples are required to learn it globally. In this paper, we present a novel method and some preliminary results for incrementally tracking camera motion through sampling and linearizing the local appearance manifold. At each frame time, we use a cluster of calibrated and synchronized small baseline cameras to capture scene appearance samples at different camera poses. We compute a first-order approximation of the appearance manifold around the current camera pose. Then, as new cluster samples are captured at the next frame time, we estimate the incremental camera motion using a linear solver. By using intensity measurements and directly sampling the appearance manifold, our method avoids the commonly-used feature extraction and matching processes, and does not require 3D correspondences across frames. Thus it can be used for scenes with complicated surface materials, geometries, and view-dependent appearance properties, situations where many other camera tracking methods would fail.

Emergency trauma is a major health problem worldwide. To evaluate the potential of emerging 3D telepresence technology for facilitating paramedic - physician collaboration while providing emergency medical trauma care we conducted a between-subjects post-test experimental lab study. During a simulated emergency situation 60 paramedics diagnosed and treated a trauma victim while working alone or in collaboration with a physician via 2D video or a 3D proxy. Analysis of paramedics' task performance shows that the fewest harmful procedures occurred in the 3D proxy condition. Paramedics in the 3D proxy condition also reported higher levels of self-efficacy. These results indicate 3D telepresence technology has potential to improve paramedics' performance of complex emergency medical tasks and improve emergency trauma health care when designed appropriately.

Most multi-camera vision applications assume a single common color response for all cameras. However different cameras - even of the same type - can exhibit radically different color responses, and the differences can cause significant errors in scene interpretation. To address this problem we have developed a robust system aimed at inter-camera color consistency. Our method consists of two phases: an iterative closed-loop calibration phase that searches for the per-camera hardware register settings that best balance linearity and dynamic range, followed by a refinement phase that computes the per-camera parametric values for an additional software-based color mapping

S.313

We introduce a general method for evaluating and comparing the expected performance of sensing systems for interactive computer graphics. Example applications include head tracking systems for virtual environments, motion capture systems for movies, and even multi-camera 3D vision systems for image-based visual hulls.Our approach is to estimate the asymptotic position and/or orientation uncertainty at many points throughout the desired working volume, and to visualize the results graphically. This global performance estimation can provide both a quantitative assessment of the expected performance, and intuition about the type and arrangement of sources and sensors, in the context of the desired working volume and expected scene dynamics.

We present two extensions to the space carving framework. The first is a progressive scheme to better reconstruct surfaces lacking sufficient textures. The second is a novel photo-consistency measure that is valid for both specular and diffuse surfaces, under unknown lighting conditions.

We introduce and present preliminary results for a hybrid display system combining head-mounted and projector-based displays. Our work is motivated by a surgical training application, where it is necessary to simultaneously provide both a high-fidelity view of a central close-up task (the surgery) and visual awareness of objects and events in the surrounding environment In particular, for trauma surgeons it would be valuable to learn to work in an environment that is realistically filled with both necessary and distracting objects and events. In this paper, we motivate the use of a hybrid display system, discuss previous work, describe a prototype along with methods for geometric calibration, and present results from a controlled human subject experiment.

Abstract We present a novel use of commodity graphics hardware that effectively combines a plane‐sweeping algorithm with view synthesis for real‐time, online 3D scene acquisition and view synthesis. Using real‐time imagery from a few calibrated cameras, our method can generate new images from nearby viewpoints, estimate a dense depth map from the current viewpoint, or create a textured triangular mesh. We can do each of these without any prior geometric information or requiring any user interaction, in real time and online. The heart of our method is to use programmable Pixel Shader technology to square intensity differences between reference image pixels, and then to choose final colors (or depths) that correspond to the minimum difference, i.e. the most consistent color. In this paper we describe the method, place it in the context of related work in computer graphics and computer vision, and present some results. ACM CSS: I.3.3 Computer Graphics—Bitmap and framebuffer operations, I.4.8 Image Processing and Computer Vision—Depth cues, Stereo

We present a novel use of commodity graphics hardware that effectively combines a plane-sweeping algorithm with view synthesis for real-time, on-line 3D scene acquisition and view synthesis. Using real-time imagery from a few calibrated cameras, our method can generate new images from nearby viewpoints, estimate a dense depth map from the current viewpoint, or create a textured triangular mesh. We can do this without prior geometric information or requiring any user interaction, in real time and on line. The heart of our method is using programmable pixel shader technology to square intensity differences between reference image pixels, and then to choose final colors (or depths) that correspond to the minimum difference, i.e. the most consistent color. In this paper we describe the method, place it in the context of related work in computer graphics and computer vision, and present results.

We introduce an idea and some preliminary results for a new projector-based approach to re-creating real and imagined sites. Our goal is to achieve re-creations that are both visually and spatially realistic, providing a small number of relatively unencumbered users with a strong sense of immersion as they jointly walkaround the virtual site.Rather than using head-mounted or general-purpose projector-based displays, our idea builds on previous projector-based work on spatially-augmented realityand shader lamps. Using simple white building blocks we construct a static physical model that approximates the size, shape, and spatial arrangementof the site. We then project dynamic imagery onto the blocks, transforming the lifeless physical model into a visually faithful reproduction of the actual site. Some advantages of this approach include wide field-of-view imagery, real walking around the site, reduced sensitivity to tracking errors, reduced sensitivity to system latency, auto-stereoscopic vision, the natural addition of augmented virtualityand the provision of haptics.In addition to describing the major challenges to (and limitations of) this vision, in this paper we describe some short-term solutions and practical methods, and we present some proof-of-concept results.

In 1998 we introduced the idea for a project we call the Office of the Future. Our long-term vision is to provide a better every-day working environment, with high-fidelity scene reconstruction for life-sized 3D tele-collaboration. In particular, we want a true sense of presence with our remote collaborator and their real surroundings. The challenges related to this vision are enormous and involve many technical tradeoffs. This is true in particular for scene reconstruction. Researchers have been striving to achieve real-time approaches, and while they have made respectable progress, the limitations of conventional technologies relegate them to relatively low resolution in a restricted volume. We present a significant step toward our ultimate goal, via a slightly different path. In lieu of low-fidelity dynamic scene modeling we present an exceedingly high fidelity reconstruction of a real but static office. By assembling the best of available hardware and software technologies in static scene acquisition, modeling algorithms, rendering, tracking and stereo projective display, we are able to demonstrate a portal to a real office, occupied today by a mannequin, and in the future by a real remote collaborator. We now have both a compelling sense of just how good it could be, and a framework into which we will later incorporate dynamic scene modeling, as we continue to head toward our ultimate goal of 3D collaborative telepresence.

Large area tiled displays are gaining popularity for use in collaborative immersive virtual environments and scientific visualization. While recent work has addressed the issues of geometric registration, rendering architectures, and human interfaces, there has been relatively little work on photometric calibration in general, and photometric non-uniformity in particular. For example, as a result of differences in the photometric characteristics of projectors, the color and intensity of a large area display varies from place to place. Further, the imagery typically appears brighter at the regions of overlap between adjacent projectors. We analyze and classify the causes of photometric non-uniformity in a tiled display. We then propose a methodology for determining corrections designed to achieve uniformity, that can correct for the photometric variations across a tiled projector display in real time using per channel color look-up-tables (LUT).

Conventional projector-based display systems are typically designed around precise and regular configurations of projectors and display surfaces. While this results in rendering simplicity and speed, it also means painstaking construction and ongoing maintenance. In previously published work, we introduced a vision of projector-based displays constructed from a collection of casually-arranged projectors and display surfaces. In this paper, we present flexible yet practical methods for realizing this vision, enabling low-cost mega-pixel display systems with large physical dimensions, higher resolution, or both. The techniques afford new opportunities to build personal 3D visualization systems in offices, conference rooms, theaters, or even your living room. As a demonstration of the simplicity and effectiveness of the methods that we continue to perfect, we show in the included video that a 10-year old child can construct and calibrate a two-camera, two-projector, head-tracked display system, all in about 15 minutes.

Our HiBall Tracking System generates over 2000 head-pose estimates per second with less than one millisecond of latency, and less than 0.5 millimeters and 0.02 degrees of position and orientation noise, everywhere in a 4.5 by 8.5 meter room. The system is remarkably responsive and robust, enabling VR applications and experiments that previously would have been difficult or even impossible.

We introduce ideas, proposed technologies, and initial results for an office of the future that is based on a unified application of computer vision and computer graphics in a system that combines and builds upon the notions of the CAVE™, tiled display systems, and image-based modeling .The basic idea is to use real-time computer vision techniques to dynamically extract per-pixel depth and reflectance information for the visible surfaces in the office including walls, furniture, objects, and people, and then to either project images on the surfaces, render images of the surfaces , or interpret changes in the surfaces.In the first case, one could designate every-day (potentially irregular) real surfaces in the office to be used as spatially immersive display surfaces, and then project high-resolution graphics and text onto those surfaces.In the second case, one could transmit the dynamic image-based models over a network for display at a remote site.Finally, one could interpret dynamic changes in the surfaces for the purposes of tracking, interaction, or augmented reality applications.To accomplish the simultaneous capture and display we envision an office of the future where the ceiling lights are replaced by computer controlled cameras and "smart" projectors that are used to capture dynamic image-based models with imperceptible structured light techniques, and to display high-resolution images on designated display surfaces.By doing both simultaneously on the designated display surfaces, one can dynamically adjust or autocalibrate for geometric, intensity, and resolution variations resulting from irregular or changing display surfaces, or overlapped projector images.Our current approach to dynamic image-based modeling is to use an optimized structured light scheme that can capture per-pixel depth and reflectance at interactive rates.Our system implementation is not yet imperceptible, but we can demonstrate the approach in the laboratory.Our approach to rendering on the designated (potentially irregular) display surfaces is to employ a two-pass projective texture scheme to generate images that when projected onto the surfaces appear correct to a moving headtracked observer.We present here an initial implementation of the overall vision, in an office-like setting, and preliminary demonstrations of our dynamic modeling and display techniques.

We present a promising new mathematical method for tracking a user's pose (position and orientation) for interactive computer graphics.The method, which is applicable to a wide variety of both commercial and experimental systems, improves accuracy by properly assimilating sequential observations, filtering sensor measurements, and by concurrently autocalibrating source and sensor devices.It facilitates user motion prediction, multisensor data fusion, and higher report rates with lower latency than previous methods.Tracking systems determine the user's pose by measuring signals from low-level hardware sensors.For reasons of physics and economics, most systems make multiple sequential measurements which are then combined to produce a single tracker report.For example, commercial magnetic trackers using the SPASYN ( Space Synchro ) system sequentially measure three magnetic vectors and then combine them mathematically to produce a report of the sensor pose.Our new approach produces tracker reports as each new lowlevel sensor measurement is made rather than waiting to form a complete collection of observations.Because single observations under-constrain the mathematical solution, we refer to our approach as single-constraint-at-a-time or SCAAT tracking.The key is that the single observations provide some information about the user's state, and thus can be used to incrementally improve a previous estimate.We recursively apply this principle, incorporating new sensor data as soon as it is measured.With this approach we are able to generate estimates more frequently, with less latency, and with improved accuracy.We present results from both an actual implementation, and from extensive simulations.