Despite steady progress, text entry in Extended Reality (XR) often remains slower and more effortful than typing on a physical keyboard or touchscreen. We explore a simple idea: use gaze to swipe through a virtual keyboard for the fast, low-effort where and a manual pinch held throughout the swipe for the when, extending and validating it through a series of user studies. We first show that a basic version including a low-latency decoder with spatiotemporal Dynamic Time Warping and fixation filtering outperforms selecting individual keys sequentially, either by finger tapping each or gazing at each while pinching. We then add mid-swipe prediction and in-gesture cancellation, improving words per minute (WPM) without hurting accuracy. We show that this approach is faster and more preferred than previous gaze-swipe approaches, finger tapping with prediction, or hand swiping with the same additions. Furthermore, a seven-day, 30-session study demonstrates sustained learning, with peak performance reaching 64.7 WPM.

We investigate hybrid user interfaces (HUIs), aiming to establish a cohesive understanding and to adopt consistent terminology for this nascent research area. HUIs combine heterogeneous devices in complementary roles, leveraging the distinct benefits of each. Our work focuses on cross-device interaction between 2D devices and mixed reality environments, which are particularly compelling, leveraging the familiarity of traditional 2D platforms while providing spatial awareness and immersion. Although prior work has prominently explored such HUIs in the context of mixed reality, we still lack a cohesive understanding of the unique design possibilities and challenges of such combinations, resulting in a fragmented research landscape. We conducted a systematic survey and present a taxonomy of HUIs that combine conventional display technology and mixed reality environments. Based on this, we discuss past and current challenges, the evolution of definitions, and prospective opportunities to tie together the past 30 years of research with our vision of future HUIs.

We present a prototype virtual reality user interface for robot teleoperation that supports high-level specification of 3D object positions and orientations in remote assembly tasks. Users interact with virtual replicas of task objects. They asynchronously assign multiple goals in the form of 6DoF destination poses without needing to be familiar with specific robots and their capabilities, and manage and monitor the execution of these goals. The user interface employs two different spatiotemporal visualizations for assigned goals: one represents all goals within the user’s workspace (Aggregated View), while the other depicts each goal within a separate world in miniature (Timeline View). We conducted a user study of the interface without the robot system to compare how these visualizations affect user efficiency and task load. The results show that while the Aggregated View helped the participants finish the task faster, the participants preferred the Timeline View.

The inferior alveolar nerve block (IANB) is a dental anesthetic injection that is critical to the performance of many dental procedures. Dental students typically learn to administer an IANB through videos and practice on silicone molds and, in many dental schools, on other students. This causes significant stress for both the students and their early patients. To reduce discomfort and improve clinical outcomes, we created an anatomically informed virtual reality headset-based educational system for the IANB. It combines a layered 3D anatomical model, dynamic visual guidance for syringe position and orientation, and active force feedback to emulate syringe interaction with tissue. A companion mobile augmented reality application allows students to step through a visualization of the procedure on a phone or tablet. We conducted a user study to determine the advantages of preclinical training with our IANB simulator. We found that in comparison to dental students who were exposed only to traditional supplementary study materials, dental students who used our IANB simulator were more confident administering their first clinical injections, had less need for syringe readjustments, and had greater success in numbing patients.

Three-dimensional geospatial thinking is an important skillset used by earth scientists and students to analyze and interpret data [1]. This method of inquiry is useful in glaciology, where traditional geophysical survey techniques have been adapted to map three-dimensional (3D) ice sheet structures and inform studies of ice flow, mass change, and history in both Greenland and Antarctica. Ice-penetrating radar images the ice in two-dimensional (2D) cross-sections from the surface to the base. Analysis of this data often requires visual inspection and 3D interpretation, but is hindered by data visualization tools and techniques that rarely transcend the two-dimensionality of the computer screen [2]. Recent advances in Augmented Reality (AR) and Virtual Reality (VR), together referred to as Extended Reality (XR), offer a glimpse into the future of 3D ice-sheet data analysis [3]. These technologies offer users an immersive experience where 3D geospatial datasets can be understood more immediately than with 2D maps, and gestural user interfaces can enhance understanding. Here we present Pol-XR, an XR application that supports both visualization and interpretation of ice-penetrating radar in Antarctica and Greenland.

Augmented reality (AR) has been used to guide users in multi-step tasks, providing information about the current step (cueing) or future steps (precueing). However, existing work exploring cueing and precueing a series of rigid-body transformations requiring rotation has only examined one-degree-of-freedom (DoF) rotations alone or in conjunction with 3DoF translations. In contrast, we address sequential tasks involving 3DoF rotations and 3DoF translations. We built a testbed to compare two types of visualizations for cueing and precueing steps. In each step, a user picks up an object, rotates it in 3D while translating it in 3D, and deposits it in a target 6DoF pose. Action-based visualizations show the actions needed to carry out a step and goal-based visualizations show the desired end state of a step. We conducted a user study to evaluate these visualizations and the efficacy of precueing. Participants performed better with goal-based visualizations than with action-based visualizations, and most effectively with goal-based visualizations aligned with the Euler axis. However, only a few of our participants benefited from precues, most likely because of the cognitive load of 3D rotations.

We explore Spatial Augmented Reality (SAR) precues (predictive cues) for procedural tasks within and between workspaces and for visualizing multiple upcoming steps in advance. We designed precues based on several factors: cue type, color transparency, and multi-level (number of precues). Precues were evaluated in a procedural task requiring the user to press buttons in three surrounding workspaces. Participants performed fastest in conditions where tasks were linked with line cues with different levels of color transparency. Precue performance was also affected by whether the next task was in the same workspace or a different one.

In virtual reality, VR sickness resulting from continuous locomotion via controllers or joysticks is still a significant problem. In this article, we present a set of algorithms to mitigate VR sickness that dynamically modulate the user's field of view by modifying the contrast of the periphery based on movement, color, and depth. In contrast with previous work, this vision modulator is a shader that is triggered by specific motions known to cause VR sickness, such as acceleration, strafing, and linear velocity. Moreover, the algorithm is governed by delta velocity, delta angle, and average color of the view. We ran two experiments with different washout periods to investigate the effectiveness of dynamic modulation on the symptoms of VR sickness, in which we compared this approach against a baseline and pitch-black field-of-view restrictors. Our first experiment made use of a just-noticeable-sickness design, which can be useful for building experiments with a short washout period.

Studies in robot teleoperation have been centered around action specifications-from continuous joint control to discrete end-effector pose control. However, these “robot-centric” interfaces often require skilled operators with extensive robotics expertise. To make teleoperation accessible to nonexpert users, we propose the framework “Scene Editing as Teleoperation” (SEaT), where the key idea is to transform the traditional “robot-centric” interface into a “scene-centric” interface-instead of controlling the robot, users focus on specifying the task's goal by manipulating digital twins of the real-world objects. As a result, a user can perform teleoperation without any expert knowledge of the robot hardware. To achieve this goal, we utilize a category-agnostic scene-completion algorithm that translates the real-world workspace (with unknown objects) into a manipulable virtual scene representation and an action-snapping algorithm that refines the user input before generating the robot's action plan. To train the algorithms, we procedurely generated a large-scale, diverse kit-assembly dataset that contains object-kit pairs that mimic real-world object-kitting tasks. Our experiments in simulation and on a real-world system demonstrate that our framework improves both the efficiency and success rate for 6DoF kit-assembly tasks. A user study demonstrates that SEaT framework participants achieve a higher task success rate and report a lower subjective workload compared to an alternative robot-centric interface.

Work on cueing performance in AR and VR has focused on sequential tasks in which each step must be completed in order before the user can proceed to the next. However, for unordered tasks such as putting books back on a library shelf, the user may be able to perform multiple steps concurrently without needing to follow a specific order. In such situations, giving the user multiple cues for potentially concurrent steps may improve performance time. To investigate this, we built a bimanual VR testbed in which the user needs to move objects to designated destinations, guided by different numbers of cues. The user can decide the order to perform the cued steps and, in some conditions, can affect which cues are shown.In a formal user study, we found that in most conditions, participants perform fastest with three cues. Dynamically updating the set of displayed cues based on hand proximity improves performance, and updating the set based on eye gaze improves performance even more. Finally, for both the hand-proximity and eye-gaze mechanisms, performance can be further improved by locking the cues for objects predicted to be moved next based on hand distance.

Augmented reality has been used to improve sequential-task performance by cueing information about a current task step and precueing information about future steps. Existing work has shown the benefits of precueing movement (translation) information. However, rotation is also a major component in many real-life tasks, such as turning knobs to adjust parameters on a console. We developed an AR testbed to investigate whether and how much precued rotation information can improve user performance. We consider two unimanual tasks: one requires a user to make sequential rotations of a single object, and the other requires the user to move their hand between multiple objects to rotate them in sequence.

When a user is performing a manual task, AR or VR can provide information about the current subtask (cueing) and upcoming subtasks (precueing) that makes them easier and faster to complete. Previous research on cueing and precueing in AR and VR has focused on path-following tasks requiring simple actions at each of a series of locations, such as pushing a button or just visiting. We consider a more complex task, whose subtasks involve moving to and picking up an item, moving that item to a designated place while rotating it to a specific angle, and depositing it. We conducted two user studies to examine how people accomplish this task while wearing an AR headset, guided by different visualizations that cue and precue movement and rotation. Participants performed best when given movement information for two successive subtasks and rotation information for a single subtask. In addition, participants performed best when the rotation visualization was split across the manipulated object and its destination.

Work on VR and AR task interaction and visualization paradigms has typically focused on providing information about the current step (a cue) immediately before or during its performance. Some research has also shown benefits to simultaneously providing information about the next step (a precue). We explore whether it would be possible to improve efficiency by precueing information about multiple upcoming steps before completing the current step. To accomplish this, we developed a remote VR user study comparing task completion time and subjective metrics for different levels and styles of precueing in a path-following task. Our visualizations vary the precueing level (number of steps precued in advance) and style (whether the path to a target is communicated through a line to the target, and whether the place of a target is communicated through graphics at the target). Participants in our study performed best when given two to three precues for visualizations using lines to show the path to targets. However, performance degraded when four precues were used. On the other hand, participants performed best with only one precue for visualizations without lines, showing only the places of targets, and performance degraded when a second precue was given. In addition, participants performed better using visualizations with lines than ones without lines.

For our society to be more inclusive and accessible, the more than 2.2 billion people worldwide with limited vision should be considered more frequently in design decisions, such as architectural planning. To help architects in evaluating their designs and give medical personnel some insight on how patients experience cataracts, we worked with ophthalmologists to develop the first medically-informed, pilot-studied simulation of cataracts in eye-tracked augmented reality (AR). To test our methodology and simulation, we conducted a pilot study with cataract patients between surgeries of their two cataract-affected eyes. Participants compared the vision of their corrected eye, viewing through simulated cataracts, to that of their still affected eye, viewing an unmodified AR view. In addition, we conducted remote experiments via video call, live adjusting our simulation and comparing it to related work, with participants who had cataract surgery a few months before. We present our findings and insights from these experiments and outline avenues for future work.

Many AR and VR task domains involve manipulating virtual objects; for example, to perform 3D geometric transformations. These operations are typically accomplished with tracked hands or hand-held controllers. However, there are some activities in which the user's hands are already busy with another task, requiring the user to temporarily stop what they are doing to perform the second task, while also taking time to disengage and reengage with the original task (e.g., putting down and picking up tools). To avoid the need to overload the user's hands this way in an AR system for guiding a physician performing a surgical procedure, we developed a hands-free approach to performing 3D transformations on patient-specific virtual organ models. Our approach uses small head motions to accomplish first-order and zero-order control, in conjunction with voice commands to establish the type of transformation. To show the effectiveness of this approach for translating, scaling, and rotating 3D virtual models, we conducted a within-subject study comparing the hands-free approach with one based on conventional manual techniques, both running on a Microsoft HoloLens and using the same voice commands to specify transformation type. Independent of any additional time to transition between tasks, users were significantly faster overall using the hands-free approach, significantly faster for hands-free translation and scaling, and faster (although not significantly) for hands-free rotation.

Navigating stairs is a dangerous mobility challenge for people with low vision, who have a visual impairment that falls short of blindness. Prior research contributed systems for stair navigation that provide audio or tactile feedback, but people with low vision have usable vision and don't typically use nonvisual aids. We conducted the first exploration of augmented reality (AR) visualizations to facilitate stair navigation for people with low vision. We designed visualizations for a projection-based AR platform and smartglasses, considering the different characteristics of these platforms. For projection-based AR, we designed visual highlights that are projected directly on the stairs. In contrast, for smartglasses that have a limited vertical field of view, we designed visualizations that indicate the user's position on the stairs, without directly augmenting the stairs themselves. We evaluated our visualizations on each platform with 12 people with low vision, finding that the visualizations for projection-based AR increased participants' walking speed. Our designs on both platforms largely increased participants' self-reported psychological security.

Vision impairments, such as cataracts, affect the way many people interact with their environment, yet are rarely considered by architects and lighting designers because of a lack of design tools. To address this, we present a method to simulate vision impairments, in particular cataracts, graphically in virtual reality (VR), using eye tracking for gaze-dependent effects. We also conduct a VR user study to investigate the effects of lighting on visual perception for users with cataracts. In contrast to existing approaches, which mostly provide only simplified simulations and are primarily targeted at educational or demonstrative purposes, we account for the user's vision and the hardware constraints of the VR headset. This makes it possible to calibrate our cataract simulation to the same level of degraded vision for all participants. Our study results show that we are able to calibrate the vision of all our participants to a similar level of impairment, that maximum recognition distances for escape route signs with simulated cataracts are significantly smaller than without, and that luminaires visible in the field of view are perceived as especially disturbing due to the glare effects they create. In addition, the results show that our realistic simulation increases the understanding of how people with cataracts see and could therefore also be informative for health care personnel or relatives of cataract patients.

In the real world, we are surrounded by potentially important data. For example, military personnel and first responders may need to understand the layout of an environment, including the locations of designated assets, specified in latitude and longitude. However, many augmented reality (AR) systems cannot associate absolute geographic coordinates with the coordinate system in which they track. We describe a simple approach for developing a wide-area outdoor wearable AR system that uses RTK GNSS position tracking to align together and georegister multiple smaller maps from an existing SLAM tracking system.

Many optical-see-through displays have a relatively narrow field of view. However, a limited field of view can constrain how information can be presented and searched through. To understand these constraints, we present a series of experiments that address the interrelationships between field of view, information density, and search performance. We do so by simulating various fields of view using two approaches: limiting the field of view presented on a Microsoft HoloLens optical-see-through head-worn display and dynamically changing the portion of a large tiled-display wall on which information is presented, for head-tracked users in both cases. Our results indicate a significant effect of information density and field of view on search performance, with potential search performance benefits of using a larger FOV between ca. 7-28%. Furthermore, while grids guided visual search, they did not significantly affect performance.

In recent years, the entity--component--system pattern has become a fundamental feature of the software architectures of game-development environments such as Unity and Unreal, which are used extensively in developing 3D user interfaces. In these systems, UI components typically respond to events, requiring programmers to write application-specific callback functions. In some cases, components are organized in a hierarchy that is used to propagate events among vertically connected components. When components need to communicate horizontally, programmers must connect those components manually and register/unregister events as needed. Moreover, events and callback signatures may be incompatible, making modular UIs cumbersome to build and share within or across applications. To address these problems, we introduce a messaging framework, Mercury, to facilitate communication among components. We provide an overview of Mercury, outline its underlying protocol and how it propagates messages to responders using relay nodes, describe a reference implementation in Unity, and present example systems built using Mercury to explain its advantages.

Map users often need to interact repetitively with multiple important locations. For example, a traveler may frequently check her hotel or a train station on a map, use them to localize an unknown location, or investigate routes involving them. Ironically, these location-centric tasks cannot be performed using locations directly; users must instead pan and zoom the map or use a menu to access locations. We propose SpaceTokens, interactive widgets that act as clones of locations, and which users can create and place on map edges like virtual whiteboard magnets. SpaceTokens make location a first-class citizen of map interaction. They empower users to rapidly perform location-centric tasks directly using locations: users can select combinations of on-screen locations and SpaceTokens to control the map window, or connect them to create routes. Participants in a study overwhelmingly preferred a SpaceTokens prototype over Google Maps on identical smartphones for the majority of tasks.

Vascular interventions are minimally invasive surgical procedures in which a physician navigates a catheter through a patient's vasculature to a desired destination in the patient's body. Since perception of relevant patient anatomy is limited in procedures of this sort, virtual reality and augmented reality systems have been developed to assist in 3D navigation. These systems often require user interaction, yet both of the physician's hands may already be busy performing the procedure. To address this need, we demonstrate hands-free interaction techniques that use voice and head tracking to allow the physician to interact with 3D virtual content on a head-worn display while making both hands available intraoperatively. Our approach supports rotation and scaling of 3D anatomical models that appear to reside in the surrounding environment through small head rotations using first-order control, and rigid body transformation of those models using zero-order control. This allows the physician to easily manipulate a model while it stays close to the center of their field of view.

We demonstrate an interaction technique that allows a user to point at a world-in-miniature representation of a city-scale virtual environment and perform efficient and precise teleportation by pre-orienting an avatar. A preview of the post-teleport view of the full-scale virtual environment updates interactively as the user adjusts the position, yaw, and pitch of the avatar's head with a pair of 6DoF-tracked controllers. We describe design decisions and contrast with alternative approaches to virtual travel.

Maps on mobile/wearable devices often make it difficult to determine the location of a point of interest (POI). For example, a POI may exist outside the map or on a background with no meaningful cues. To address this issue, we present Personalized Compass, a self-contained compact graphical location indicator. Personalized Compass uses personal a priori POIs to establish a reference frame, within which a POI in question can then be localized. Graphically, a personalized compass combines a multi-needle compass with an abstract overview map. We analyze the characteristics of Personalized Compass and the existing Wedge technique, and report on a user study comparing them. Personalized Compass performs better for four inference tasks, while Wedge is better for a locating task. Based on our analysis and study results, we suggest the two techniques are complementary and offer design recommendations.

OBJECTIVE: Prior studies of computing applications that support patients' medication knowledge and self-management offer valuable insights into effective application design, but do not address inpatient settings. This study is the first to explore the design and usefulness of patient-facing tools supporting inpatient medication management and tracking. MATERIALS AND METHODS: We designed myNYP Inpatient, a custom personal health record application, through an iterative, user-centered approach. Medication-tracking tools in myNYP Inpatient include interactive views of home and hospital medication data and features for commenting on these data. In a two-phase pilot study, patients used the tools during cardiothoracic postoperative care at Columbia University Medical Center. In Phase One, we provided 20 patients with the application for 24-48 h and conducted a closing interview after this period. In Phase Two, we conducted semi-structured interviews with 12 patients and 5 clinical pharmacists who evaluated refinements to the tools based on the feedback received during Phase One. RESULTS: Patients reported that the medication-tracking tools were useful. During Phase One, 14 of the 20 participants used the tools actively, to review medication lists and log comments and questions about their medications. Patients' interview responses and audit logs revealed that they made frequent use of the hospital medications feature and found electronic reporting of questions and comments useful. We also uncovered important considerations for subsequent design of such tools. In Phase Two, the patients and pharmacists participating in the study confirmed the usability and usefulness of the refined tools. CONCLUSIONS: Inpatient medication-tracking tools, when designed to meet patients' needs, can play an important role in fostering patient participation in their own care and patient-provider communication during a hospital stay.

Assembly or repair tasks often require objects to be held in specific orientations to view or fit together. Research has addressed the use of AR to assist in these tasks, delivered as registered overlaid graphics on stereoscopic head-worn displays. In contrast, we are interested in using monoscopic head-worn displays, such as Google Glass. To accommodate their small monoscopic field of view, off center from the user's line of sight, we are exploring alternatives to registered overlays. We describe four interactive rotation guidance visualizations for tracked objects intended for these displays.

Wearable computers and Generation-5 Digital Eye Glass easily recognize a user's own gestures, forming the basis for shared AR (Augmediated Reality). This Studio-workshop presents the latest in wearable AR, plus an historical perspective with new insights. Participants will sculpt 3D objects using hand gestures and create Unity 3D art+game objects using computational lightpainting.

In many complex tasks, a remote subject-matter expert may need to assist a local user to guide actions on objects in the local user's environment. However, effective spatial referencing and action demonstration in a remote physical environment can be challenging. We introduce two approaches that use Virtual Reality (VR) or Augmented Reality (AR) for the remote expert, and AR for the local user, each wearing a stereo head-worn display. Both approaches allow the expert to create and manipulate virtual replicas of physical objects in the local environment to refer to parts of those physical objects and to indicate actions on them. This can be especially useful for parts that are occluded or difficult to access. In one approach, the expert points in 3D to portions of virtual replicas to annotate them. In another approach, the expert demonstrates actions in 3D by manipulating virtual replicas, supported by constraints and annotations. We performed a user study of a 6DOF alignment task, a key operation in many physical task domains, comparing both approaches to an approach in which the expert uses a 2D tablet-based drawing system similar to ones developed for prior work on remote assistance. The study showed the 3D demonstration approach to be faster than the others. In addition, the 3D pointing approach was faster than the 2D tablet in the case of a highly trained expert.

We introduce WeARHand, which allows a user to manipulate virtual 3D objects with a bare hand in a wearable augmented reality (AR) environment. Our method uses no environmentally tethered tracking devices and localizes a pair of near-range and far-range RGB-D cameras mounted on a head-worn display and a moving bare hand in 3D space by exploiting depth input data. Depth perception is enhanced through egocentric visual feedback, including a semi-transparent proxy hand. We implement a virtual hand interaction technique and feedback approaches, and evaluate their performance and usability. The proposed method can apply to many 3D interaction scenarios using hands in a wearable AR environment, such as AR information browsing, maintenance, design, and games.

Many tasks in real or virtual environments require users to view a target object or location from one of a set of strategic viewpoints to see it in context, avoid occlusions, or view it at an appropriate angle or distance. We introduce ParaFrustum, a geometric construct that represents this set of strategic viewpoints and viewing directions. ParaFrustum is inspired by the look-from and look-at points of a computer graphics camera specification, which precisely delineate a location for the camera and a direction in which it looks. We generalize this approach by defining a ParaFrustum in terms of a look-from volume and a look-at volume, which establish constraints on a range of acceptable locations for the user's eyes and a range of acceptable angles in which the user's head can be oriented. Providing tolerance in the allowable viewing positions and directions avoids burdening the user with the need to assume a tightly constrained 6DoF pose when it is not required by the task. We describe two visualization techniques for virtual or augmented reality that guide a user to assume one of the poses defined by a ParaFrustum, and present the results of a user study measuring the performance of these techniques. The study shows that the constraints of a tightly constrained ParaFrustum (e.g., approximating a conventional camera frustum) require significantly more time to satisfy than those of a loosely constrained one. The study also reveals interesting differences in participant trajectories in response to the two techniques.

OBJECTIVE: To systematically review existing literature regarding patient engagement technologies used in the inpatient setting. METHODS: PubMed, Association for Computing Machinery (ACM) Digital Library, Institute of Electrical and Electronics Engineers (IEEE) Xplore, and Cochrane databases were searched for studies that discussed patient engagement ('self-efficacy', 'patient empowerment', 'patient activation', or 'patient engagement'), (2) involved health information technology ('technology', 'games', 'electronic health record', 'electronic medical record', or 'personal health record'), and (3) took place in the inpatient setting ('inpatient' or 'hospital'). Only English language studies were reviewed. RESULTS: 17 articles were identified describing the topic of inpatient patient engagement. A few articles identified design requirements for inpatient engagement technology. The remainder described interventions, which we grouped into five categories: entertainment, generic health information delivery, patient-specific information delivery, advanced communication tools, and personalized decision support. CONCLUSIONS: Examination of the current literature shows there are considerable gaps in knowledge regarding patient engagement in the hospital setting and inconsistent use of terminology regarding patient engagement overall. Research on inpatient engagement technologies has been limited, especially concerning the impact on health outcomes and cost-effectiveness.

Goal-oriented visual search is performed when a person intentionally seeks a target in the visual environment. In augmented reality (AR) environments, visual search can be facilitated by augmenting virtual cues in the person's field of view. Traditional use of explicit AR cues can potentially degrade visual search performance due to the creation of distortions in the scene. An alternative to explicit cueing, known as subtle cueing, has been proposed as a clutter-neutral method to enhance visual search in video-see-through AR. However, the effects of subtle cueing are still not well understood, and more research is required to determine the optimal methods of applying subtle cueing in AR. We performed two experiments to investigate the variables of scene clutter, subtle cue opacity, size, and shape on visual search performance. We introduce a novel method of experimentally manipulating the scene clutter variable in a natural scene while controlling for other variables. The findings provide supporting evidence for the subtlety of the cue, and show that the clutter conditions of the scene can be used both as a global classifier, as well as a local performance measure.

Goal-oriented visual search in augmented reality can be facilitated by using visual cues to call attention to a target. However, traditional use of explicit cues can degrade visual search performance due to scene distortion, occlusion and addition of visual clutter. In contrast, Subtle Cueing has been previously proposed as an alter-native to explicit cueing, but little is known about how well it works for head-tracked head worn displays (HWDs). We investigated the effect of Subtle Cueing for head-tracked head worn displays, using visual search research methods in simulated augmented reality environments. Our user study found that Subtle Cueing improves visual search performance, and serves as a feasible cueing mechanism for AR environments using HWDs.

Eye contact plays a crucial role in our everyday social interactions. The ability of a device to reliably detect when a person is looking at it can lead to powerful human-object interfaces. Today, most gaze-based interactive systems rely on gaze tracking technology. Unfortunately, current gaze tracking techniques require active infrared illumination, calibration, or are sensitive to distance and pose. In this work, we propose a different solution-a passive, appearance-based approach for sensing eye contact in an image. By focusing on gaze *locking* rather than gaze tracking, we exploit the special appearance of direct eye gaze, achieving a Matthews correlation coefficient (MCC) of over 0.83 at long distances (up to 18 m) and large pose variations (up to ±30° of head yaw rotation) using a very basic classifier and without calibration. To train our detector, we also created a large publicly available gaze data set: 5,880 images of 56 people over varying gaze directions and head poses. We demonstrate how our method facilitates human-object interaction, user analytics, image filtering, and gaze-triggered photography.

This paper presents a content structure to build mobile AR applications in HTML5 to achieve a clean separation of mobile AR contents from their application logic to scale like the web. By extending POIs (Point of Interest) to objects and places with Uniform Resource Identifier (URI), we could build objects of interest for mobile AR application as DOM (Document Object Model) elements and control their behavior and user interactions through DOM events. Using our content structure, a mobile AR applications can be developed as normal HTML documents seamlessly under current web eco-system.

Handheld optical projectors provide a simple way to overcome the limited screen real-estate on mobile devices. We present virtual projection (VP), an interaction metaphor inspired by how we intuitively control the position, size, and orientation of a handheld optical projector's image. VP is based on tracking a handheld device without an optical projector and allows selecting a target display on which to position, scale, and orient an item in a single gesture. By relaxing the optical projection metaphor, we can deviate from modeling perspective projection, for example, to constrain scale or orientation, create multiple copies, or offset the image. VP also supports dynamic filtering based on the projection frustum, creating overview and detail applications, and selecting portions of a larger display for zooming and panning. We show exemplary use cases implemented using our optical feature-tracking framework and present the results of a user study demonstrating the effectiveness of VP in complex interactions with large displays.

Already 3D visualization environments provide a large design space not being investigated to the same extent as traditional WIMP-spaces. When using this design space in combination with AR, the design space even further grows. Information can not only be presented in a 3D space, AR also puts virtual information in relation to real objects, locations or events. The different properties of presentation in AR need to be investigated to develop a comprehensive set of dimensions of presentation principles.

Visual search in augmented reality environments is an important task that can be facilitated through different cueing methods. Current cueing methods rely on explicit cueing, which can potentially reduce visual search performance. In comparison, this paper proposes a subtle cueing method that improves visual search performance while being clutter-neutral. Two empirical user studies were conducted to evaluate our subtle cueing method in outdoor scenes. The results show that subtle cueing functions well within a narrow Feature Congestion range, and could be a feasible alternative to explicit cueing.

We introduce an augmented reality referencing technique for shared environments that is designed to improve the accuracy with which one user can point out a real physical object to another user. Our technique, GARDEN (Gesturing in an Augmented Reality Depth-mapped ENvironment), is intended for use in otherwise unmodeled environments in which objects in the environment, and the hand of the user performing a selection, are interactively observed by a depth camera, and users wear tracked see-through displays. We present the results of a user study that compares GARDEN against existing augmented reality referencing techniques, as well as the use of a physical laser pointer. GARDEN performed significantly more accurately than all the comparison techniques when the participating users have sufficiently different views of the scene, and significantly more accurately than one of these techniques when the participating users have similar perspectives.

Magic-lens style augmented reality applications allow users to control camera pose easily by manipulating a portable hand-held device and provide immediate visual feedback. However, strategic vantage points must often be revisited repeatedly, adding time and error and taxing memory. We describe a new approach that allows users to take snapshots of augmented scenes that can be virtually revisited at later times. The system stores still images of scenes along with camera poses, so that augmentations remain dynamic and interactive. Users can manipulate virtual objects while viewing snapshots, instead of moving to real-world views. We present a study comparing performance in snapshot and live mode conditions in a task in which a virtual object must be aligned with two pairs of physical objects. Proper alignment requires sequentially visiting two viewpoints. Participants completed the alignment task significantly faster and more accurately using snapshots than when using the live mode. Moreover, participants preferred manipulating virtual objects using snapshots to the live mode.

The articles in this special issue contain the full paper proceedings of the IEEE Virtual Reality Conference 2012 (IEEE VR 2012), held March 4-8, 2012 in Orange County, California.

In augmented reality, it is often necessary to draw the user's attention to particular objects in the real world without distracting her from her task. We explore the effectiveness of directing a user's attention by imperceptibly modifying existing features of a video. We present three user studies of the effects of applying a saliency modulation technique to video; evaluating modulation awareness, attention, and memory. Our results validate the saliency modulation technique as an alternative means to convey information to the user, suggesting attention shifts and influencing recall of selected regions without perceptible changes to visual input.

How can multiple different display and interaction devices be used together to create an effective augmented reality environment? We explore the design of several prototype hybrid user interfaces that combine a 2D multi-touch tabletop display with a 3D head-tracked video-see-through display. We describe a simple modeling application and an urban visualization tool in which the information presented on the head-worn display supplements the information displayed on the tabletop, using a variety of approaches to track the head-worn display relative to the tabletop. In all cases, our goal is to allow users who can see only the tabletop to interact effectively with users wearing head-worn displays.

While there is significant recent progress in technologies supporting augmented reality for small indoor environments, there is still much work to be done for large outdoor environments. This workshop focuses primarily on research that enables high-quality outdoor Mixed Reality (MR) and Augmented Reality (AR) applications. These research topics include, but are not restricted to: — 3D geo-referenced data (images, point clouds, and models) — Algorithms for object recognition from large databases of geo-referenced data — Algorithms for object tracking in outdoor environment — Multi-cue fusion to achieve improved performance of object detection and tracking — Novel representation schemes to facilitate large-scale content distribution — 3D reasoning to support intelligent augmentation — Novel and improved mobile capabilities for data capture (device sensors), processing, and display — Applications, experiences, and user interface techniques. The workshop will also showcase existing prototypes of applications enabled by these technologies: mirror worlds, high-fidelity virtual environments, applications of panoramic imagery, and user studies relating to these media types. This workshop aims to bring together academic and industrial researchers and to foster discussion amongst participants on the current state of the art and future directions for technologies that enable large-scale outdoor MR and AR applications. The workshop will start with a session in which position statements and overviews of the state of the art are presented. In the afternoon, we will follow up with discussion sessions and a short closing session.

Procedural tasks are common to many domains, ranging from maintenance and repair, to medicine, to the arts. We describe and evaluate a prototype augmented reality (AR) user interface designed to assist users in the relatively under-explored psychomotor phase of procedural tasks. In this phase, the user begins physical manipulations, and thus alters aspects of the underlying task environment. Our prototype tracks the user and multiple components in a typical maintenance assembly task, and provides dynamic, prescriptive, overlaid instructions on a see-through head-worn display in response to the user's ongoing activity. A user study shows participants were able to complete psychomotor aspects of the assembly task significantly faster and with significantly greater accuracy than when using 3D-graphics-based assistance presented on a stationary LCD. Qualitative questionnaire results indicate that participants overwhelmingly preferred the AR condition, and ranked it as more intuitive than the LCD condition.

We present a novel bimanual body-directed travel technique, PenguFly (PF), and compare it with two standard travel-by-pointing techniques by conducting a between-subject experiment in a CAVE. In PF, the positions of the user's head and hands are projected onto the ground, and travel direction and speed are computed based on direction and magnitude of the vector from the midpoint of the projected hand positions to the projected head position. The two baseline conditions both use a single hand to control the direction, with speed controlled discretely by button pushes with the same hand in one case, and continuously by the distance between the hands in the other case. Users were asked to travel through a simple virtual world and collect virtual coins within a set time. We found no significant differences between travel conditions for reported presence or usability, but a significant increase in nausea with PF. Total travel distance was significantly higher for the baseline condition with discrete speed selection, whereas travel accuracy in terms of coin-to-distance ratio was higher with PF.

We explore the development of an experimental augmented reality application that provides benefits to professional mechanics performing maintenance and repair tasks in a field setting. We developed a prototype that supports military mechanics conducting routine maintenance tasks inside an armored vehicle turret, and evaluated it with a user study. Our prototype uses a tracked headworn display to augment a mechanic's natural view with text, labels, arrows, and animated sequences designed to facilitate task comprehension, localization, and execution. A within-subject controlled user study examined professional military mechanics using our system to complete 18 common tasks under field conditions. These tasks included installing and removing fasteners and indicator lights, and connecting cables, all within the cramped interior of an armored personnel carrier turret. An augmented reality condition was tested against two baseline conditions: the same headworn display providing untracked text and graphics and a fixed flat panel display representing an improved version of the laptop-based documentation currently employed in practice. The augmented reality condition allowed mechanics to locate tasks more quickly than when using either baseline, and in some instances, resulted in less overall head movement. A qualitative survey showed that mechanics found the augmented reality condition intuitive and satisfying for the tested sequence of tasks.

We describe fieldwork in which we studied hospital ICU physicians and their strategies and documentation aids for composing patient progress notes. We then present a clinical documentation prototype, activeNotes, that supports the creation of these notes, using techniques designed based on our fieldwork. ActiveNotes integrates automated, context-sensitive patient data retrieval, and user control of automated data updates and alerts via tagging, into the documentation process. We performed a qualitative study of activeNotes with 15 physicians at the hospital to explore the utility of our information retrieval and tagging techniques. The physicians indicated their desire to use tags for a number of purposes, some of them extensions to what we intended, and others new to us and unexplored in other systems of which we are aware. We discuss the physicians' responses to our prototype and distill several of their proposed uses of tags: to assist in note content management, communication with other clinicians, and care delivery.

This paper provides a classification of perceptual issues in augmented reality, created with a visual processing and interpretation pipeline in mind. We organize issues into ones related to the environment, capturing, augmentation, display, and individual user differences. We also illuminate issues associated with more recent platforms such as handhelds or projector-camera systems. Throughout, we describe current approaches to addressing these problems, and suggest directions for future research.

Many tasks require a user to move between various locations within an environment to get different perspectives. This can take significant time and effort, especially when the user must switch among those viewpoints repeatedly. We explore augmented reality interaction techniques that involve taking still pictures of a physical scene using a tracked hand-held magic lens and seamlessly switching between augmenting either the live view or one of the still views, without needing to physically revisit the snapshot locations. We describe our optical-marker-tracking-based implementation and how we represent and switch among snapshots. To determine the effectiveness of our techniques, we developed a test application that lets its user view physical and virtual objects from different viewpoints.

Opportunistic Controls are a class of user interaction techniques that we have developed for augmented reality (AR) applications to support gesturing on, and receiving feedback from, otherwise unused affordances already present in the domain environment. By leveraging characteristics of these affordances to provide passive haptics that ease gesture input, Opportunistic Controls simplify gesture recognition, and provide tangible feedback to the user. In this approach, 3D widgets are tightly coupled with affordances to provide visual feedback and hints about the functionality of the control. For example, a set of buttons can be mapped to existing tactile features on domain objects. We describe examples of Opportunistic Controls that we have designed and implemented using optical marker tracking, combined with appearance-based gesture recognition. We present the results of two user studies. In the first, participants performed a simulated maintenance inspection of an aircraft engine using a set of virtual buttons implemented both as Opportunistic Controls and using simpler passive haptics. Opportunistic Controls allowed participants to complete their tasks significantly faster and were preferred over the baseline technique. In the second, participants proposed and demonstrated user interfaces incorporating Opportunistic Controls for two domains, allowing us to gain additional insights into how user interfaces featuring Opportunistic Controls might be designed.

Urban designers and urban planners often conduct site visits prior to a design activity to search for patterns or better understand existing conditions. We introduce SiteLens, an experimental system and set of techniques for supporting site visits by visualizing relevant virtual data directly in the context of the physical site, which we call situated visualization. We address alternative visualization representations and techniques for data collection, curation, discovery, comparison, manipulation, and provenance. A real use scenario is presented and two iterations of evaluation with faculty and students from the Columbia University Graduate School of Architecture, Planning and Preservation provide directions and insight for further investigation.

We present the design, implementation, and user testing of a prototype augmented reality application to support military mechanics conducting routine maintenance tasks inside an armored vehicle turret. Our prototype uses a tracked head-worn display to augment a mechanic's natural view with text, labels, arrows, and animated sequences designed to facilitate task comprehension, location, and execution. A within-subject controlled user study examined professional military mechanics using our system to complete 18 common tasks under field conditions. These tasks included installing and removing fasteners and indicator lights, and connecting cables, all within the cramped interior of an armored personnel carrier turret. An augmented reality condition was tested against two baseline conditions: an untracked headworn display with text and graphics and a fixed flat panel display representing an improved version of the laptop-based documentation currently employed in practice. The augmented reality condition allowed mechanics to locate tasks more quickly than when using either baseline, and in some instances, resulted in less overall head movement. A qualitative survey showed mechanics found the augmented reality condition intuitive and satisfying for the tested sequence of tasks.

In a multi-user augmented reality application for a shared physical environment, it is possible for users to interfere with each other. For example, in a multi-player game in which each player holds a display whose tracked position and orientation affect the outcome, one player may physically block another player's view or physically contact another player. We explore software techniques intended to avoid such interference. These techniques modify what a user sees or hears, and what interaction capabilities they have, when their display gets too close to another user's display. We present Redirected Motion, an effective, yet nondistracting, interference avoidance technique for hand-held AR, which transforms the 3D space in which the user moves their display, to direct the display away from other displays. We conducted a within-subject, formal user study to evaluate the effectiveness and distraction level of Redirected Motion compared to other interference avoidance techniques. The study is based on an instrumented, two-player, first-person-shooter, augmented reality game, in which each player holds a 6DOF-tracked ultra-mobile computer. Comparison conditions include an unmanipulated control condition and three other software techniques for avoiding interference: dimming the display, playing disturbing sounds, and disabling interaction capabilities. Subjective evaluation indicates that Redirected Motion was unnoticeable, and quantitative analysis shows that the mean distance between users during Redirected Motion was significantly larger than for the comparison conditions.

Menus play an important role in both information presentation and system control. We explore the design space of shake menus, which are intended for use in tangible augmented reality. Shake menus are radial menus displayed centered on a physical object and activated by shaking that object. One important aspect of their design space is the coordinate system used to present menu options. We conducted a within-subjects user study to compare the speed and efficacy of several alternative methods for presenting shake menus in augmented reality (world-referenced, display-referenced, and object-referenced), along with a baseline technique (a linear menu on a clipboard). Our findings suggest tradeoffs amongst speed, efficacy, and flexibility of interaction, and point towards the possible advantages of hybrid approaches that compose together transformations in different coordinate systems. We close by describing qualitative feedback from use and present several illustrative applications of the technique.

While touch-screen displays are becoming increasingly popular, many factors affect user experience and performance. Surface quality, parallax, input resolution, and robustness, for instance, can vary with sensing technology, hardware configurations, and environmental conditions.

Time-series graphs are often used to visualize phenomena that change over time. Common tasks include comparing values at different points in time and searching for specified patterns, either exact or approximate. However, tools that support time-series graphs typically separate query specification from the actual search process, allowing users to adapt the level of similarity only after specifying the pattern. We introduce relaxed selection techniques, in which users implicitly define a level of similarity that can vary across the search pattern, while creating a search query with a single-gesture interaction. Users sketch over part of the graph, establishing the level of similarity through either spatial deviations from the graph, or the speed at which they sketch (temporal deviations). In a user study, participants were significantly faster when using our temporally relaxed selection technique than when using traditional techniques. In addition, they achieved significantly higher precision and recall with our spatially relaxed selection technique compared to traditional techniques.

We introduce two families of techniques, rubbing and tapping, that use zooming to make possible precise interaction on passive touch screens, and describe examples of each. Rub-Pointing uses a diagonal rubbing gesture to integrate pointing and zooming in a single-handed technique. In contrast, Zoom-Tapping is a two-handed technique in which the dominant hand points, while the non-dominant hand taps to zoom, simulating multi-touch functionality on a single-touch display. Rub-Tapping is a hybrid technique that integrates rubbing with the dominant hand to point and zoom, and tapping with the non-dominant hand to confirm selection. We describe the results of a formal user study comparing these techniques with each other and with the well-known Take-Off and Zoom-Pointing selection techniques. Rub-Pointing and Zoom-Tapping had significantly fewer errors than Take-Off for small targets, and were significantly faster than Take-Off and Zoom-Pointing. We show how the techniques can be used for fluid interaction in an image viewer and in Google Maps.

We present Opportunistic Controls, a class of user interaction techniques for augmented reality (AR) applications that support gesturing on, and receiving feedback from, otherwise unused affordances already present in the domain environment. Opportunistic Controls leverage characteristics of these affordances to provide passive haptics that ease gesture input, simplify gesture recognition, and provide tangible feedback to the user. 3D widgets are tightly coupled with affordances to provide visual feedback and hints about the functionality of the control. For example, a set of buttons is mapped to existing tactile features on domain objects. We describe examples of Opportunistic Controls that we have designed and implemented using optical marker tracking, combined with appearance-based gesture recognition. We present the results of a user study in which participants performed a simulated maintenance inspection of an aircraft engine using a set of virtual buttons implemented both as Opportunistic Controls and using simpler passive haptics. Opportunistic Controls allowed participants to complete their tasks significantly faster and were preferred over the baseline technique.

Biological research in the field is constrained by the speed and difficulty of species determination, as well as by access to relevant information about the species encountered. However, recent work on vision-based algorithms raises the promise of rapid botanical species identification. The potential for mobile vision-based identification provides opportunities for new user interface techniques. To explore these issues, we present LeafView, a Tablet-PC-based user interface for an electronic field guide that supports automated identification of botanical species in the field. We describe a user interface design based on an ethnographic study of botanists, field tests of working prototypes by botanists at the Smithsonian Institution on Plummers Island, Maryland, and observations at an internal exhibition at the Smithsonian at which other staff members tried the prototypes. We present functionality specific to mobile identification and collection in the electronic field guide and use this to motivate discussion of mobile identification in general.

Warping the pointer across monitor bezels has previously been demonstrated to be both significantly faster and preferred to the standard mouse behavior when interacting across displays in homogeneous multi-monitor configurations. Complementing this work, we present a user study that compares the performance of four pointer-warping strategies, including a previously untested frame-memory placement strategy, in heterogeneous multi-monitor environments, where displays vary in size, resolution, and orientation. Our results show that a new frame-memory pointer warping strategy significantly improved targeting performance (up to 30% in some cases). In addition, our study showed that, when transitioning across screens, the mismatch between the visual and the device space has a significantly bigger impact on performance than the mismatch in orientation and visual size alone. For mouse operation in a highly heterogeneous multi-monitor environment, all our participants strongly preferred using pointer warping over the regular mouse behavior.

Tangible augmented reality (AR) systems imbue physical objects with the ability to act and respond in new ways. In particular, physical objects and gestures made with them gain meaning that does not exist outside the tangible AR environment. The existence of this new set of possible actions and outcomes is not always apparent, making it necessary to learn new movements or gestures. Addressing this opportunity, we present visual hints, which are graphical representations in AR of potential actions and their consequences in the augmented physical world.Visualhintsenable discovery, learning, and completion of gestures and manipulation in tangible AR. Here, we discuss our investigation of a variety of representations of visual hints and methods for activating them. We then describe a specific implementation that supports gestures developed for a tangible AR user interface to an electronic field guide for botanists, and present results from a pilot study.

Accessing information on mobile and wearable devices often requires the user's visual attention, and the precise operation of virtual or physical widgets. However, these interactions may sometimes be too time-consuming and socially inappropriate. To address this, we introduce a novel input/output device that is based on the manipulation of a retractable string in a polar coordinate frame. Depending on how the user pulls the string from its enclosure--to a particular length, at a particular angle--various system features may be directly accessed. Furthermore, we present our concept for a 1D pixel array, embedded in the string that may be used as a secondary 1D display. Since it is possible to unwind the display itself and trigger functionality with a single pull, information may be accessed and presented quickly, and perceived at a glance. We present scenarios for how the string input/output device may be used in conjunc-tion with the mobile device's primary 2D display and describe our augmented reality proof-of-concept prototype.

We present a set of techniques that enable mobile users to visualize and navigate complex hypermedia structures embedded in the real world, through augmented reality or virtual reality. Situating hypermedia in the 3D physical environment makes it possible to represent information about users' surroundings in context. However, it requires addressing a new set of problems beyond those of visualizing hypermedia on a 2D display: Nodes and links can potentially be distributed across large distances, and may be occluded by other objects, both real and virtual. Our techniques address these issues by enabling mobile users to select and manipulate portions of the hypermedia structure by tilting, lifting and shifting them, to view more clearly links and nodes that would otherwise be occluded or ambiguously connected.

We present a set of mobile augmented reality interaction techniques for authoring situated media: multimedia and hypermedia that are embedded within the physical environment. Our techniques are designed for use with a tracked hand-held tablet display with an attached camera, and rely on "freezing" the frame for later editing.

Scrolling is used to navigate large information spaces on small screens, but is often too restrictive or cumbersome to use for particular types of content, such as multi-page, multi-column documents. To address this problem, we introduce content-aware scrolling (CAS), an approach that takes into account various characteristics of document content to determine scrolling direction, speed, and zoom. We also present the CAS widget, which supports scrolling through a content-aware path using traditional scrolling methods, demonstrating the advantages of making a traditional technique content-aware.

Stereo and bi-ocular head-mounted displays (HMDs) require the user to fuse two images into a coherent picture of the threedimensional world. The human visual system performs this task constantly, but when the input images contain both real and graphical depictions, the problem becomes more difficult. A vertical disparity in the graphics causes diplopia for users trying to fuse the real and virtual objects simultaneously. We implement three methods to measure and correct this disparity and assess them with a collection of a single model of optical see-through HMD.

Information in hybrid user interfaces can be spread over a variety of different, but complementary, displays, with which users interact through a potentially equally varied range of interaction devices. Since the exact configuration of these displays and devices may not be known in advance, it is desirable for users to be able to reconfigure at runtime the dataflow between interaction devices and objects on the displays. To make this possible, we present the design and implementation of a prototype mixed reality system that allows users to immersively reconfigure a running hybrid user interface.

We describe several approaches for using prosodic features of speech and audio localization to control interactive applications. This information can be applied to parameter control, as well as to speech disambiguation. We discuss how characteristics of spoken sentences can be exploited in the user interface; for example, by considering the speed with which a sentence is spoken and the presence of extraneous utterances. We also show how coarse audio localization can be used for low-fidelity gesture tracking, by inferring the speaker's head position.

We present a set of cross-dimensional interaction techniques for a hybrid user interface that integrates existing 2D and 3D visualization and interaction devices. Our approach is built around one-and two-handed gestures that support the seamless transition of data between co-located 2D and 3D contexts. Our testbed environment combines a 2D multi-user, multi-touch, projection surface with 3D head-tracked, see-through, head-worn displays and 3D tracked gloves to form a multi-display augmented reality. We address some of the ways in which we can interact with private data in a collaborative, heterogeneous workspace. We also report on a pilot usability study to evaluate the effectiveness and ease of use of the cross-dimensional interactions.

We present VITA (visual interaction tool for archaeology), an experimental collaborative mixed reality system for offsite visualization of an archaeological dig. Our system allows multiple users to visualize the dig site in a mixed reality environment in which tracked, see-through, head-worn displays are combined with a multi-user, multi-touch, projected table surface, a large screen display, and tracked hand-held displays. We focus on augmenting existing archaeological analysis methods with new ways to organize, visualize, and combine the standard 2D information available from an excavation (drawings, pictures, and notes) with textured, laser range-scanned 3D models of objects and the site itself. Users can combine speech, touch, and 3D hand gestures to interact multimodally with the environment. Preliminary user tests were conducted with archaeology researchers and students, and their feedback is presented here.

Layout refers to the process of determining the size and position of the visual objects in an information presentation. We introduce the WeightMap, a bitmap representation of the visual weight of a presentation. In addition, we present algorithms that use WeightMaps to allow an automated layout system to evaluate the effectiveness of its layouts. Our approach is based on the concepts of visual weight and visual balance, which are fundamental to the visual arts. The objects in the layout are each assigned a visual weight, and a WeightMap is created that encodes the visual weight of the layout. Image-processing techniques, including pyramids and edge detection, are then used to efficiently analyze the WeightMap for balance. In addition, derivatives of the sums of the rows and columns are used to generate suggestions for how to improve the layout.

We present content-aware free-space transparency, an approach to viewing and manipulating the otherwise hidden content of obscured windows through unimportant regions of overlapping windows. Traditional approaches to interacting with otherwise obscured content in a window system render an entire window uniformly transparent. In contrast, content-aware free-space transparency uses opaque-to-transparent gradients and image-processing filters to minimize the interference from overlapping material, based on properties of that material. By increasing the amount of simultaneously visible content and allowing basic interaction with otherwise obscured content, without modifying window geometry, we believe that free-space transparency has the potential to improve user productivity.

Our field and the world have changed greatly in the ten years since the first VRST was held in Singapore in 1994. Computers have grown smaller, faster, and cheaper, while polygon counts, frame rates, and display resolutions have increased impressively, true to the promise of Moore's Law. But, what comes next?This talk will sketch some of the directions in which I feel virtual reality is (or should be) heading. I will discuss the potential for taking virtual reality outside, through wearable and mobile computing; for bring the outside in, by capturing the real world; and for accommodating large numbers of displays, users, and tasks, by embedding them in a fluid and collaborative augmented environment.

We describe an approach to 3D multimodal interaction in immersive augmented and virtual reality environments that accounts for the uncertain nature of the information sources. The resulting multimodal system fuses symbolic and statistical information from a set of 3D gesture, spoken language, and referential agents. The referential agents employ visible or invisible volumes that can be attached to 3D trackers in the environment, and which use a time-stamped history of the objects that intersect them to derive statistics for ranking potential referents. We discuss the means by which the system supports mutual disambiguation of these modalities and information sources, and show through a user study how mutual disambiguation accounts for over 45% of the successful 3D multimodal interpretations. An accompanying video demonstrates the system in action.

We introduce a set of statistical geometric tools designed to identify the objects being manipulated through speech and gesture in a multimodal augmented reality system. SenseShapes are volumetric regions of interest that can be attached to parts of the user's body to provide valuable information about the user's interaction with objects. To assist in object selection, we generate a rich set of statistical data and dynamically choose which data to consider based on the current situation.

Virtual reality has long been an indoor affair. Whether constrained by stationary computers ordisplays, or by the limitations of our tracking technologies, researchers typically build virtualenvironments that work within a single physical room or a portion of a room. Even distributedvirtual reality systems usually interconnect two or more such indoor spaces. Meanwhile, ascomputers grow ever smaller and faster, mobile computing is becoming an increasingly importantpart of our daily lives, accompanying us wherever we go, outdoors, as well as indoors.What will it take for virtual reality to move outdoors and finally see the light of day? And, whyshould we care? Within the virtual reality research community, work on augmented reality hasalready begun to explore outdoor environments-tracking using computer vision, gyroscopes,accelerometers, compasses, and GPS; and experimenting with (barely) wearable testbeds. I willdiscuss why virtual reality (especially in the form of augmented reality) and mobile computing are asynergistic combination, and will provide an overview of the research problems that must beaddressed for mobile augmented reality systems to play a major role in our future.Among the issues that I will review are overcoming physical and aesthetic barriers to mobilityand wearability; tracking and registration of heads, hands, bodies, and other objects; renderingvirtual objects in the real world; and developing sufficiently high quality displays. Equallyimportant is the design of head-tracked user interfaces that are well suited to mobility. Wearablesystems will need to support collaboration among mobile users, facile interaction with real andvirtual objects, and coordination across a wide range of heterogeneous displays and devices. Keyhere is the volatile nature of mobile interactions-users continually move into and out of thepresence of other users, devices, and objects, and rapidly change tasks. Furthermore, augmentedreality makes it possible for real and virtual objects to share the same display space, creating thepotential for a variety of visually confusing relationships as objects overlap and occlude each other.Avoiding these problems will require that the virtual world be redesigned and laid out on the fly, tomaintain desired visual relationships between virtual objects and other real and virtual objects.

We describe an automated layout system called AIL that generates the user interface for the PERSIVAL digital library project. AIL creates a layout based on a variety of content components and associated meta-data information provided by the PERSIVAL generation and retrieval modules. By leveraging semantic links between the content components, the layout that AIL provides is both context and user-model aware. In addition, AIL is capable of interacting intelligently with the natural language generation components of PERSIVAL to tailor the length of the text content for a given layout.

We present a situation-awareness aid for augmented reality systems based on an annotated "world in miniature." Our aid is designed to provide users with an overview of their environment that allows them to select and inquire about the objects that it contains. Two key capabilities are discussed that are intended to address the needs of mobile users. The aid's position, scale, and orientation are controlled by a novel approach that allows the user to inspect the aid without the need for manual interaction. As the user alternates their attention between the physical world and virtual aid, popup annotations associated with selected objects can move freely between the objects' representations in the two models.

Network and server-centric computing paradigms are quickly returning to being the dominant methods by which we use computers. Web applications are so prevalent that the role of a PC today has been largely reduced to a terminal for running a client or viewer such as a Web browser. Implementers of network-centric applications typically rely on the limited capabilities of HTML, employing proprietary "plug ins" or transmitting the binary image of an entire application that will be executed on the client. Alternatively, implementers can develop without regard for remote use, requiring users who wish to run such applications on a remote server to rely on a system that creates a virtual frame buffer on the server, and transmits a copy of its raster image to the local client.We review some of the problems that these current approaches pose, and show how they can be solved by developing a distributed user interface toolkit. A distributed user interface toolkit applies techniques to the high level components of a toolkit that are similar to those used at a low level in the X Window System. As an example of this approach, we present RemoteJFC, a working distributed user interface toolkit that makes it possible to develop thin-client applications using a distributed version of the Java Foundation Classes.

We describe a view-management component for interactive 3D user interfaces. By view management, we mean maintaining visual constraints on the projections of objects on the view plane, such as locating related objects near each other, or preventing objects from occluding each other. Our view-management component accomplishes this by modifying selected object properties, including position, size, and transparency, which are tagged to indicate their constraints. For example, some objects may have geometric properties that are determined entirely by a physical simulation and which cannot be modified, while other objects may be annotations whose position and size are flexible.We introduce algorithms that use upright rectangular extents to represent on the view plane a dynamic and efficient approximation of the occupied space containing the projections of visible portions of 3D objects, as well as the unoccupied space in which objects can be placed to avoid occlusion. Layout decisions from previous frames are taken into account to reduce visual discontinuities. We present augmented reality and virtual reality examples to which we have applied our approach, including a dynamically labeled and annotated environment.

OBJECTIVE: The authors present a system that scans electronic records from cardiac surgery and uses inference rules to identify and classify abnormal events (e.g., hypertension) that may occur during critical surgical points (e.g., start of bypass). This vital information is used as the content of automatically generated briefings designed by MAGIC, a multimedia system that they are developing to brief intensive care unit clinicians on patient status after cardiac surgery. By recognizing patterns in the patient record, inferences concisely summarize detailed patient data. DESIGN: The authors present the development of inference rules that identify important information about patient status and describe their implementation and an experiment they carried out to validate their correctness. The data for a set of 24 patients were analyzed independently by the system and by 46 physicians. MEASUREMENTS: The authors measured accuracy, specificity, and sensitivity by comparing system inferences against physician judgments, in cases where all three physicians agreed and against the majority opinion in all cases. RESULTS: For laboratory inferences, evaluation shows that the system has an average accuracy of 98 percent (full agreement) and 96 percent (majority model). An analysis of interrater agreement, however, showed that physicians do not agree on abnormal hemodynamic events and could not serve as a gold standard for evaluating hemodynamic events. Analysis of discrepancies reveals possibilities for system improvement and causes of physician disagreement. CONCLUSIONS: This evaluation shows that the laboratory inferences of the system have high accuracy. The lack of agreement among physicians highlights the need for an objective quality-assurance tool for hemodynamic inferences. The system provides such a tool by implementing inferencing procedures established in the literature.

We present a general approach to the dynamic representation of 2D space that is well suited for userinterface layout.We partition space into two distinct categories: full and empty.The user can explicitly specify a set of possibly overlapping upright rectangles that represent the objects of interest.These full-space rectangles are processed by the system to create a representation of the remaining empty space.This representation makes it easy for users to develop customized spatial allocation strategies that avoid overlapping the full-space rectangles.We describe the representation; provide efficient incremental algorithms for adding and deleting full-space rectangles, and for querying the empty-space representation; and show several allocation strategies that the representation makes possible.We present two testbed applications that incorporate an implementation of the algorithm; one shows the utility of our representation for window management tasks; the other applies it to the layout of components in a 3D user interface, based on the upright 2D bounding boxes of their projections.

We describe an experimental mobile augmented reality system (MARS) testbed that employs different user interfaces to allow outdoor and indoor users to access and manage information that is spatially registered with the real world. Outdoor users can experience spatialized multimedia presentations that are presented on a head-tracked, see-through, head-worn display used in conjunction with a hand-held pen-based computer. Indoor users can get an overview of the outdoor scene and communicate with outdoor users through a desktop user interface or a head- and hand-tracked immersive augmented reality user interface.

To develop a comprehensive and systematic approach to the automated design of visual discourse, we introduce a visual task taxonomy that interfaces high-level presentation intents with low-level visual techniques.In our approach, visual tasks describe presentation intents through their visual accomplishments, and suggest desired visual techniques through their visual implications.Therefore, we can characterize visual tasks by their visual accomplishments and implications.Through this characterization, visual tasks can guide the visual discourse synthesis process by specifying what presentation intents can be achieved and how to achieve them.

We present Repo-3D, a general-purpose, object-oriented library for developing distributed, interactive 3D graphics applications across a range of heterogeneous workstations.Repo-3D is designed to make it easy for programmers to rapidly build prototypes using a familiar multi-threaded, object-oriented programming paradigm.All data sharing of both graphical and non-graphical data is done via general-purpose remote and replicated objects, presenting the illusion of a single distributed shared memory.Graphical objects are directly distributed, circumventing the "duplicate database" problem and allowing programmers to focus on the application details.Repo-3D is embedded in Repo, an interpreted, lexically-scoped, distributed programming language, allowing entire applications to be rapidly prototyped.We discuss Repo-3D's design, and introduce the notion of local variations to the graphical objects, which allow local changes to be applied to shared graphical structures.Local variations are needed to support transient local changes, such as highlighting, and responsive local editing operations.Finally, we discuss how our approach could be applied using other programming languages, such as Java.

No abstract available.

Communication can be more effective when several media (such as text, speech, or graphics) are integrated and coordinated to present information. This changes the nature of media-specific generation (e.g., language or graphics generation), which must take into account the multimedia context in which it occurs. This paper presents work on coordinating and integrating speech, text, static and animated three-dimensional graphics, and stored images, as part of several systems we have developed at Columbia University. A particular focus of our work has been on the generation of presentations that brief a user on information of interest

A visual discourse is a series of connected visual displays. A coherent visual discourse is characterized by smooth transitions between displays, consistent design within and across displays, and successful integration of new information into existing displays. We use a topdown, hierarchical-decomposition, partial order planner to efficiently construct a visual discourse from scratch, taking advantage of parametrized primitive visual objects that serve as building blocks in the design process. Visual representations are modeled as visual objects, graphical techniques are employed as planning operators, and design policies are encoded as constraints. This approach not only improves computational efficiency compared to search-based approaches, but also facilitates knowledge encoding, and ensures global coherency.

A visual discourse is a series of connected visual displays. A coherent visual discourse requires smooth transitions between displays, consistent design within and across displays, and successful integration of new information into existing displays. We present an approach for automatically designing a coherent visual discourse. A top-down, hierarchical-decomposition partial-order planner is used to efficiently plan the visual discourse. Visual representations are modelled as visual objects, graphical techniques are employed as planning operators, and design policies are encoded as constraints. This approach not only improves the computational efficiency compared to search-based approaches, but also facilitates knowledge encoding, and ensures global coherency. Keywords: Top-down hierarchical planning, automated graphics generation, knowledge-based user interfaces.

We argue that runtime program transformation, partial evaluation, and dynamic compilation are essential tools for automated generation of flexible, highly interactive graphical interfaces. In particular, these techniques help bridge the gap between a high-level, functional description and an efficient implementation. To support our claim, we describe our application of these techniques to a functional implementation of n-Vision, a real-time visualization system that represents multivariate relations as nested 3D interactors, and to Auto Visual, a rule-based system that designs n-Vision visualizations from high-level task specifications. n-Vision visualizations are specified using a simple functional language. These programs are transformed into a cached dataflow graph. A partial evaluator is used on particular computation-intensive function applications, and the results are compiled to native code. The functional representation simplifies generation of correct code, and the program transformations ensure good performance. We demonstrate why these transformations improve performance and why they cannot be done at compile time.

The conference this year is the tenth anniversary of UIST.The keynote talk discusses the history of UIST over the last ten years; this panel looks into the future of the field over the next ten.Each of the panelists will describe a scenario for what life will be like when we meet for UIST'O'I, ten years from now.They will also have a chance to challenge or question each others' scenarios and to participate in open discussion with the audience.

Most of the virtual world systems that have been so well publicized over the past ten years use opaque head mounted displays that block off the wearer from the surrounding real world, effectively immersing her within a synthesized environment. These systems hold tremendous promise for certain applications ranging from fantasy games to scientific research. In contrast, we believe that the most powerful and commonplace virtual worlds of the future will not replace the real world, but will rather augment it with additional information. This approach is called augmented reality and was pioneered by Ivan Sutherland, who, over a quarter century ago, developed the first see-through head mounted display (I. Sutherland, 1968). When completed, his system presented graphics to the user on a pair of stereo displays, worn on the user's head. The image produced by the displays was combined with the user's view of the world using mirror beam splitters. A 3D tracking system determined the position and orientation of the user's head. This enabled the system to change the view, based on the direction in which the wearer was looking.

Creating high-quality multimedia presentations requires much skill, time, and effort.This is particularly true when temporal media, such as speech and animation, are involved.We describe the design and implementation of a knowledge-based system that generates customized temporal multimedia presentations.We provide art overview of the system's architecture, and explain how speech, written text, and graphics are generated and coordinated.Our emphasis is on how temporal media are coordinated by the system through a multi-stage negotiation process.In negotiation, media-specific generation components interact with a novel coordination component that solves temporal constraints provided by the generators.We illustrate our work with a set of examples generated by the system in a testbed application intended to update hospital caregivers on the status of patients who have undergone a cardiac bypass operation.

We describe COTERIE, a toolkit that provides languagelevel support for building distributed virtual environments. COTERIE is based on the distributed data-object paradigm for distributed shared memory. Any data object in COTE-RIE can be declared to be a Shared Object that is replicated fully in any process that is interested in it. These Shared Objects support asynchronous data propagation with atomic serializable updates, and asynchronous notification of updates. COTERIE is built in Modula-3 and uses existing Modula-3 packages that support an integrated interpreted language, multithreading, and 3D animation. Unlike other VE toolkits, COTERIE is based on a set of general-purpose parallel and distributed language concepts designed with the needs of virtual environments in mind. We summarize the

No abstract available.

No abstract available.

Just as broadband networks will enable user-to-user communications to extend from textural services to those employing multimedia, they will also enable a management environment that can take advantage of increased bandwidth and multimedia technology. The fundamental advances incorporated in such an environment can provide efficient solutions to the problem of information management. To establish this environment, the authors tackle the fundamental problems of observability and controllability of broadband networks. A virtual world provides a next-generation network management interface through which a user can observe and interact with the network directly in real time. The system that the authors are developing uses a 3D virtual world as the user interface for managing a large gigabit ATM network. It provides the capability for experimentation in all aspects of network transport, control and management.>

We describe the design and implementation of a prototype heads-up window system intended for use in a 3D environment.Our system includes a see-through head-mounted display that runs a full X server whose image is overlaid on the user's view of the physical world.The user's head is tracked so that the display indexes into a large X bitmap, effectively placing the user inside a display space that is mapped onto part of a surrounding virtual sphere.By tracking the user's body, and interpreting head motion relative to it, we create a portable information surround that envelopes the user as they move about.We support three kinds of windows implemented on top of the X server: windows fixed to the head-mounted display, windows fixed to the information surround, and windows fixed to locations and objects in the 3D world.Objects can also be tracked, allowing windows to move with them.To demonstrate the utility of this model, we describe a small hypermedia system that allows links to be made between windows and windows to be attached to objects.Thus, our hypermedia system can forge links between any combination of physical objects and virtual windows.

Many graphic tasks, such as the manipulation of graphical objects and the construction of user-interface widgets, can be facilitated by geometric constraints. However, the difficulty of specifying constraints by traditional methods forms a barrier to their widespread use. In order to make constraints easier to declare, we have developed a method of specifying constraints implicitly, through multiple examples. Snapshots are taken of an initial scene configuration, and one or more additional snapshots are taken after the scene has been edited into other valid configurations. The constraints that are satisfied in all of the snapshots are then applied to the scene objects. We discuss an efficient algorithm for inferring constraints from multiple snapshots. The algorithm has been incorporated into the Chimera editor, and several examples of its use are discussed.

We describe enhancements to graphical search and replace that allow users to extend the capabilities of a graphical editor. Interactive constraint-based search and replace can search for objects that obey user-specified sets of constraints and automatically apply other constraints to modify these objects. We show how an interactive tool that employs this technique makes it possible for users to define sets of constraints graphically that modify existing illustrations or control the creation of new illustrations. The interace uses the same visual language as the editor and allows users to understand and create powerful rules without conventional programming. Rules can be saved and retrieved for use alone or in combination. Examples, generated with a working implementation, demonstrate applications to drawing beautification and transformation.

Virtual worlds are computer-generated environments created by coupling 3-D displays and interaction devices to powerful graphics workstations. The author describes work in the design of virtual worlds being carried out by Columbia's Computer Graphics and User Interfaces Group Two of the main themes of this group's research are exploiting true 3-D interaction and display devices to visualize and manipulate rich information spaces, and using artificial-intelligence techniques to automate the generation of effective graphics. The projects discussed address virtual worlds for visualizing multivariate data, hybrid user interfaces that merge 2-D and 3-D displays and interaction devices, and augmented realities in which the surrounding physical world is annotated with knowledge-based 3-D graphics.>

This paper introduces an efficient object-precision shadow generation algorithm for static polygonal environments directly illuminated by convex area light sources.Penumbra and umbra regions are calculated analytically and represented as a pair of BSP trees for each light source.As the trees are built, convex scene polygons are filtered down the trees, and split into fragments that are wholly lit, in penumbra, or in umbra.The illumination due to the light source is calculated at selected points within the wholly lit and penumbra regions by contour integration with the visible parts of the light source.We use a fast analytic algorithm to compute the fragments of the area light source visible from a point in penumbra.Rendering is done using hardwaresupported linear interpblated shading on a 3D graphics workstation.Because the scene itself is represented as a BSP tree, visible-surface determination may be performed by using either workstation-supported hardware (e.g., a z-buffer) or software BSP-tree traversal.We provide sample images created by our implementation, including timings and polygon counts.

Many tasks performed using computer interfaces are very repetitive. While programmers can write macros or procedures to automate these repetitive tasks, this requires special skills. Demonstrational systems make macro building accessible to all users, but most provide either no visual representation of the macro or only a textual representation. We have developed a history-based visual representation of commands in a graphical user interface. This representation supports the definition of macros by example in several novel ways. At any time, a user can open a history window, review the commands executed in a session, select operations to encapsulate into a macro, and choose objects and their attributes as arguments. The system has facilities to generalize the macro automatically, save it for future use, and edit it.

Interactive visualization systems provide a powerful means to explore complex data, especially when coupled with 3-D interaction and display devices to produce virtual worlds. While designing a quality static 2-D visualization is already a difficult task for most users, designing an interactive 3-D one is even more challenging. To address this problem, AutoVisual, a research system that designs interactive virtual worlds for visualizing and exploring multivariate relations of arbitrary arity, is being developed. AutoVisual uses worlds within worlds, an interactive visualization technique that exploits nested, heterogeneous coordinate systems to map multiple variables onto each spatial dimension. AutoVisual's designs are guided by user-specified visualization tasks, and by a catalog of design principles encoded using a rule-based language.>

No abstract available.

No abstract available.

This paper describes an automated intent-based approach to illustration. An illustrution is a picture that is designed to fulfill a communicative intent such as showing the location of an object or showing how an object is manipulated. An illustration is generated by implementing a set of stylistic decisions, ranging from determining the way in which an individual object is lit, to deciding the general composition of the illustration. The design of an illustration is treated as a goal-driven process within a system of constraints. The goal is to achieve communicative intent; the constraints are the illustrative techniques an illustrator can apply.We have developed IBIS (Intent-Based Illustration System), a system that puts these ideas into practice. IBIS designs illustrations using a generate-and-test approach, relying upon a rule-based system of methods and evaluators. Methods are rules that specify how to accomplish visual effects, while evaluators are rules that specify how to determine how well a visual effect is accomplished in an illustration. Examples of illustrations designed by IBIS are included.

While virtual worlds offer a compelling alternative to conventional interfaces, the technologies these systems currently use do not provide sufficient resolution and accuracy to support detailed work such as text editing.We describe a pragmatic approach to interface design that provides users with a large virtual world in which such high-resolution work can be performed.Our approach is based on combining heterogeneous display and interaction device technologies to produce a hybrid user interface.Display and interaction technologies that have relatively low resolution, but which cover a wide (visual and interactive) field are used to form an information surround.Display and interaction technologiesthat have relatively high resolution over a limited visual and interaction range are used to present concentrated information in one or more selected portions of the surround. These highresolution fields are embedded within the low-resolution surround by choosing and coordinating complementarydevices that permit the user to see and interact with both simultaneously.This allows each embedded high-resolution interface to serve as a "sweet spot" within which intonation may be preferentially processed,We have developed a preliminary implementation, described in this paper, that uses a Reflection Technology Private Eye display and a Polhemus sensor to provide the secondary lowresohttion surround, and a flat-panel display and mouse to provide the primary high-resolution interface.

Article Free Access Share on A nose gesture interface device: extending virtual realities Authors: Tyson R. Henry Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZ Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZView Profile , Scott E. Hudson Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZ Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZView Profile , Andrey K. Yeatts Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZ Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZView Profile , Brad A. Myers School of Computer Science, Carnegie Mellon University, Pittsburgh, PA School of Computer Science, Carnegie Mellon University, Pittsburgh, PAView Profile , Steven Feiner Department of Computer Science, Columbia University, New York, NY Department of Computer Science, Columbia University, New York, NYView Profile Authors Info & Claims UIST '91: Proceedings of the 4th annual ACM symposium on User interface software and technologyNovember 1991 Pages 65–68https://doi.org/10.1145/120782.120789Published:11 November 1991Publication History 5citation995DownloadsMetricsTotal Citations5Total Downloads995Last 12 Months76Last 6 weeks3 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

No abstract available.

1 Introduction One common problem in graphical user interface design has been n-Vision is a testbed for exploring n-dimensional worlds the need to manipulate and view 3D environments using containing functions of an arbitrary number of variables. inherently 2D interaction devices and displays. Although Although our interaction devices and display hardware are graphics researchers have long been developing true 3D inherently 3D, we demonstrate how they can be used to support interaction and display devices [SUTH65; VICK70; KILP76], it is interaction with these higher-dimensional objects. We introduce a only over the past decade that high-performance 3D graphics new interaction metaphor developed for the system, which we call workstations have been coupled with commercially available 3D "worlds within worlds": nested heterogeneous coordinate devices such as polarized liquid crystal shutters for stereo viewing systems that allow the user to view and manipulate functions. [TEKT87; STER89]...

This paper describes an object-space shadow generation algorithm for static polygonal environments illuminated by movable point light sources. The algorithm can be easily implemented on any graphics system that provides fast polygon scan-conversion and achieves near real-time performance for environments of modest size. It combines elements of two kinds of current shadow generation algorithms: two-pass object-space approaches and shadow volume approaches. For each light source a Binary Space Partitioning (BSP) tree is constructed that represents the shadow volume of the polygons facing it. As each polygon's contribution to a light source's shadow volume is determined, the polygon's shadowed and lit fragments are computed by filtering it down the shadow volume BSP tree. The polygonal scene with its computed shadows can be rendered with any polygon-based visible-surface algorithm. Since the shadow volumes and shadows are computed in object space, they can be used for further analysis of the scene. Pseudocode is provided, along with pictures and timings from an interactive implementation.

We describe the design and prototype implementation of Scope, a system that generates graphical user interfaces for applications programmed in C++. The programmer chooses application data objects and functions that define the capabilities of the interface. At runtime, an interface design component, implemented as a set of production system rules, transforms this semantic specification into an interface built using a window system, an associated user interface toolkit, and the hardware input devices available on the system. The rules match application requirements against a semantic description of the toolkit, selecting virtual devices for input, output, and layout. Thus, Scope uses design rules to create interfaces from high-level programming semantics that are customized both for the application and the run-time environment.

IBIS (Intent-Based Illustration System) generates illustrations automatically, guided by communicative goals. Communicative goals specify that particular properties of objects, such as their color, size, or location are to be conveyed in the illustration. IBIS is intended to be part of an interactive multimedia explanation generation system. It has access to a knowledge base that contains a collection of objects, including information about their geometric properties, material, and location. As the goals are interpreted by a rule-based control component, the system generates a precise definition of the final illustration. If IBIS determines that a set of goals cannot be satisfied in a single picture, then it attempts to create a composite illustration that has multiple viewports. For example, a composite illustration may contain a nested inset illustration showing an object in greater detail than is possible in the parent picture. Each component illustration is defined by its placement, size, viewing specification, lighting specification, and list of objects to be displayed and their graphical style.

An experimental system is described for the design, development, and presentation of computer-based documents that combine pictures and text on a high-resolution color raster display.Such documents can be used, for example, for maintenance and repair tasks, videotex databases, or computer-aided instruction.Documents are directed graphs whose nodes we refer to as pages, in analogy to the pages of a paper book.A page includes a set of simultaneously displayed pictures, actions (procedures and processes), and indexing information.Pages may be nested arbitrarily deeply in chapters that serve much the same organizing function as those of conventional books.The system is comprised of separate programs for laying out and drawing pictures, for graphically specifying the contents of pages, chapters, and their interconnections, and for displaying the document for user interaction.Examples are given from a prototype maintenance and repair manual in which emphasis was placed on designing actions that allow simple real-time animation and assist in finding one's way around the document.

An experimental system is described for the design, development, and presentation of computer-based documents that combine pictures and text on a high-resolution raster color display. Such documents can be used, for example, for maintenance and repair tasks or computer-aided instruction. Documents are directed graphs whose nodes we refer to as pages, in analogy to the pages of a paper book. A page includes a set of simultaneously displayed pictures, actions (procedures and processes) triggered when the page is accessed or when pickable picture elements on it are selected, and indexing information. Pages may be nested arbitrarily deeply in chapters that serve much the same organizing function as those of conventional books. The system is comprised of separate programs for lay-ing out and drawing pictures, for graphically specifying the contents of pages, chapters, and their interconnections, and for displaying the document for user interaction. Examples are given from a prototype document for the maintenance and repair of computerized numerical control equipment. Emphasis was placed on designing actions for simple realtime animation (both by color table techniques and by transforming named primitives and manipulating their attributes), and for finding one's way around the document (displays include: a "timeline " of recently visited pages, immediate predecessor and successor pages, sibling pages and their interconnections, and those pages satisfying key-word retrieval requests).