In many complex tasks, a remote subject-matter expert may need to assist a local user to guide actions on objects in the local user's environment. However, effective spatial referencing and action demonstration in a remote physical environment can be challenging. We introduce two approaches that use Virtual Reality (VR) or Augmented Reality (AR) for the remote expert, and AR for the local user, each wearing a stereo head-worn display. Both approaches allow the expert to create and manipulate virtual replicas of physical objects in the local environment to refer to parts of those physical objects and to indicate actions on them. This can be especially useful for parts that are occluded or difficult to access. In one approach, the expert points in 3D to portions of virtual replicas to annotate them. In another approach, the expert demonstrates actions in 3D by manipulating virtual replicas, supported by constraints and annotations. We performed a user study of a 6DOF alignment task, a key operation in many physical task domains, comparing both approaches to an approach in which the expert uses a 2D tablet-based drawing system similar to ones developed for prior work on remote assistance. The study showed the 3D demonstration approach to be faster than the others. In addition, the 3D pointing approach was faster than the 2D tablet in the case of a highly trained expert.

Many tasks in real or virtual environments require users to view a target object or location from one of a set of strategic viewpoints to see it in context, avoid occlusions, or view it at an appropriate angle or distance. We introduce ParaFrustum, a geometric construct that represents this set of strategic viewpoints and viewing directions. ParaFrustum is inspired by the look-from and look-at points of a computer graphics camera specification, which precisely delineate a location for the camera and a direction in which it looks. We generalize this approach by defining a ParaFrustum in terms of a look-from volume and a look-at volume, which establish constraints on a range of acceptable locations for the user's eyes and a range of acceptable angles in which the user's head can be oriented. Providing tolerance in the allowable viewing positions and directions avoids burdening the user with the need to assume a tightly constrained 6DoF pose when it is not required by the task. We describe two visualization techniques for virtual or augmented reality that guide a user to assume one of the poses defined by a ParaFrustum, and present the results of a user study measuring the performance of these techniques. The study shows that the constraints of a tightly constrained ParaFrustum (e.g., approximating a conventional camera frustum) require significantly more time to satisfy than those of a loosely constrained one. The study also reveals interesting differences in participant trajectories in response to the two techniques.

We introduce an augmented reality referencing technique for shared environments that is designed to improve the accuracy with which one user can point out a real physical object to another user. Our technique, GARDEN (Gesturing in an Augmented Reality Depth-mapped ENvironment), is intended for use in otherwise unmodeled environments in which objects in the environment, and the hand of the user performing a selection, are interactively observed by a depth camera, and users wear tracked see-through displays. We present the results of a user study that compares GARDEN against existing augmented reality referencing techniques, as well as the use of a physical laser pointer. GARDEN performed significantly more accurately than all the comparison techniques when the participating users have sufficiently different views of the scene, and significantly more accurately than one of these techniques when the participating users have similar perspectives.

How can multiple different display and interaction devices be used together to create an effective augmented reality environment? We explore the design of several prototype hybrid user interfaces that combine a 2D multi-touch tabletop display with a 3D head-tracked video-see-through display. We describe a simple modeling application and an urban visualization tool in which the information presented on the head-worn display supplements the information displayed on the tabletop, using a variety of approaches to track the head-worn display relative to the tabletop. In all cases, our goal is to allow users who can see only the tabletop to interact effectively with users wearing head-worn displays.

In a multi-user augmented reality application for a shared physical environment, it is possible for users to interfere with each other. For example, in a multi-player game in which each player holds a display whose tracked position and orientation affect the outcome, one player may physically block another player's view or physically contact another player. We explore software techniques intended to avoid such interference. These techniques modify what a user sees or hears, and what interaction capabilities they have, when their display gets too close to another user's display. We present Redirected Motion, an effective, yet nondistracting, interference avoidance technique for hand-held AR, which transforms the 3D space in which the user moves their display, to direct the display away from other displays. We conducted a within-subject, formal user study to evaluate the effectiveness and distraction level of Redirected Motion compared to other interference avoidance techniques. The study is based on an instrumented, two-player, first-person-shooter, augmented reality game, in which each player holds a 6DOF-tracked ultra-mobile computer. Comparison conditions include an unmanipulated control condition and three other software techniques for avoiding interference: dimming the display, playing disturbing sounds, and disabling interaction capabilities. Subjective evaluation indicates that Redirected Motion was unnoticeable, and quantitative analysis shows that the mean distance between users during Redirected Motion was significantly larger than for the comparison conditions.

Consumers are increasingly relying on Web-based social content, such as product reviews, prior to making to a purchase. Recent surveys in the retail industry confirm that social content is indeed the #1 aid in a buying decision. Currently, accessing or adding to this valuable Web-based social content repository is mostly limited to computers far removed from the site of the shopping experience itself. We present a mobile augmented reality application, which extends such social content from the computer monitor into the physical world through mobile phones, providing consumers with in situ information on products right when and where they need to make buying decisions.

We present a set of mobile augmented reality interaction techniques for authoring situated media: multimedia and hypermedia that are embedded within the physical environment. Our techniques are designed for use with a tracked hand-held tablet display with an attached camera, and rely on "freezing" the frame for later editing.