Information in hybrid user interfaces can be spread over a variety of different, but complementary, displays, with which users interact through a potentially equally varied range of interaction devices. Since the exact configuration of these displays and devices may not be known in advance, it is desirable for users to be able to reconfigure at runtime the dataflow between interaction devices and objects on the displays. To make this possible, we present the design and implementation of a prototype mixed reality system that allows users to immersively reconfigure a running hybrid user interface.

We present a situation-awareness aid for augmented reality systems based on an annotated "world in miniature." Our aid is designed to provide users with an overview of their environment that allows them to select and inquire about the objects that it contains. Two key capabilities are discussed that are intended to address the needs of mobile users. The aid's position, scale, and orientation are controlled by a novel approach that allows the user to inspect the aid without the need for manual interaction. As the user alternates their attention between the physical world and virtual aid, popup annotations associated with selected objects can move freely between the objects' representations in the two models.

We describe a view-management component for interactive 3D user interfaces. By view management, we mean maintaining visual constraints on the projections of objects on the view plane, such as locating related objects near each other, or preventing objects from occluding each other. Our view-management component accomplishes this by modifying selected object properties, including position, size, and transparency, which are tagged to indicate their constraints. For example, some objects may have geometric properties that are determined entirely by a physical simulation and which cannot be modified, while other objects may be annotations whose position and size are flexible.We introduce algorithms that use upright rectangular extents to represent on the view plane a dynamic and efficient approximation of the occupied space containing the projections of visible portions of 3D objects, as well as the unoccupied space in which objects can be placed to avoid occlusion. Layout decisions from previous frames are taken into account to reduce visual discontinuities. We present augmented reality and virtual reality examples to which we have applied our approach, including a dynamically labeled and annotated environment.

We present a general approach to the dynamic representation of 2D space that is well suited for userinterface layout.We partition space into two distinct categories: full and empty.The user can explicitly specify a set of possibly overlapping upright rectangles that represent the objects of interest.These full-space rectangles are processed by the system to create a representation of the remaining empty space.This representation makes it easy for users to develop customized spatial allocation strategies that avoid overlapping the full-space rectangles.We describe the representation; provide efficient incremental algorithms for adding and deleting full-space rectangles, and for querying the empty-space representation; and show several allocation strategies that the representation makes possible.We present two testbed applications that incorporate an implementation of the algorithm; one shows the utility of our representation for window management tasks; the other applies it to the layout of components in a 3D user interface, based on the upright 2D bounding boxes of their projections.