In this paper, we present a case study that explores how children could learn to interact with programmable matter. Flying drone swarms enable physical visualizations of complex data and simulation of physical objects and processes, e.g., planetary movements. The swarms can be digitally controlled as an ensemble as a form of (sparse) "programmable matter." We worked with the toy company LEGO®, to design and evaluate a "build and fly" experience with 240 children in a public exhibition. The children decorated a bendable handheld controller with LEGO® bricks and then used this controller to animate the flight of a 10-drone swarm. Results indicate that children enjoyed the constructive play and performance aspects of the system. Four main patterns of player behavior emerged, which we discuss in relation to possible improvements to the system. We provide implications for design of programmable matter systems for supporting child play experiences.

LightBee is a novel "hologrammatic" telepresence system featuring a self-levitating light field display. It consists of a drone that flies a projection of a remote user's head through 3D space. The movements of the drone are controlled by the remote user's head movements, offering unique support for non-verbal cues, especially physical proxemics. The light field display is created by a retro-reflective sheet that is mounted on the cylindrical quadcopter. 45 smart projectors, one per 1.3 degrees, are mounted in a ring, each projecting a video stream rendered from a unique perspective onto the retroreflector. This creates a light field that naturally provides motion parallax and stereoscopy without requiring any headset nor stereo glasses. LightBee allows multiple local users to experience their own unique and correct perspective of the remote user's head. The system is currently one-directional: 2 small cameras mounted on the drone allow the remote user to observe the local scene.

For telepresence to support the richness of multiparty conversations, it is important to convey motion parallax and stereoscopy without head-worn apparatus. TeleHuman2 is a "hologrammatic" telepresence system that conveys full-body 3D video of interlocutors using a human-sized cylindrical light field display. For rendering, the system uses an array of projectors mounted above the heads of participants in a ring around a retroreflective cylinder. Unique angular renditions are calculated from streaming depth video captured at the remote location. Projected images are retro-reflected into the eyes of local participants, at 1.3º intervals providing angular renditions simultaneously for left and right eyes of all onlookers, which conveys motion parallax and stereoscopy without head-worn apparatus or head tracking. Our technical evaluation of the angular accuracy of the system demonstrates that the error in judging the angle of a remote arrow object represented in TeleHuman2 is within 1 degree, and not significantly different from similar judgments of a collocated arrow object.

We present MagicScroll, a rollable tablet with 2 concatenated flexible multitouch displays, actuated scrollwheels and gestural input. When rolled up, MagicScroll can be used as a rolodex, smartphone, expressive messaging interface or gestural controller. When extended, it provides full access to its 7.5" high-resolution multitouch display, providing the display functionality of a tablet device. We believe that the cylindrical shape in the rolled-up configuration facilitates gestural interaction, while its shape changing and input capabilities allow the navigation of continuous information streams and provide focus plus context functionality. We investigated the gestural affordances of MagicScroll in its rolled-up configuration by means of an elicitation study.

We present GridDrones, a self-levitating programmable matter platform that can be used for representing 2.5D voxel grid relief maps capable of rendering unsupported structures and 3D transformations. GridDrones consists of cube-shaped nanocopters that can be placed in a volumetric 1xnxn mid-air grid, which is demonstrated here with 15 voxels. The number of voxels and scale is only limited by the size of the room and budget. Grid deformations can be applied interactively to this voxel lattice by manually selecting a set of voxels, then assigning a continuous topological relationship between voxel sets that determines how voxels move in relation to each other and manually drawing out selected voxels from the lattice structure. Using this simple technique, it is possible to create unsupported structures that can be translated and oriented freely in 3D. Shape transformations can also be recorded to allow for simple physical shape morphing animations. This work extends previous work on selection and editing techniques for 3D user interfaces.

In this paper, we report on a study investigating a novel haptic illusion for altering the perception of 3D shapes using a non-planar screen and vibrotactile friction. In our study, we presented an image of a rectangular prism on a cylindrical and a flat display. Participants were asked to move their index finger horizontally along the surface of the displays towards the edge of the rectangular prism. Participants were asked whether they were experiencing a flat, cylindrical or rectangular shape. In one condition, a vibrotactile stimulus simulated increasing friction towards the visible edge of the rectangular prism, with a sudden drop-off when this edge was crossed by the finger. Results suggest that presenting an image of a rectangular prism, and applying vibrotactile friction, particularly on a cylindrical display, significantly increased participant ratings stating that they were experiencing a physical rectangular shape.

In this paper, we report on an investigation of Fitts' law using flexible displays. Participants performed a one-dimensional targeting task as described by the ISO 9421-9 standard. In the experiment, we compared two methods of bend input: position control and rate control of a cursor. Participants performed the task with three levels of device stiffness. Results show that bend input is highly correlated with Fitts' law for both position and rate control. Position control produced significantly higher throughput values than rate control. Our experiment also revealed that, when the amount of force applied was controlled, device stiffness did not have a significant effect on performance.

We present BitDrones, a toolbox for building interactive real reality 3D displays that use nano-quadcopters as self-levitating tangible building blocks. Our prototype is a first step towards interactive self-levitating programmable matter, in which the user interface is represented using Catomic structures. We discuss three types of BitDrones: PixelDrones, equipped with an RGB LED and a small OLED display; ShapeDrones, augmented with an acrylic mesh spun over a 3D printed frame in a larger geometric shape; and DisplayDrones, fitted with a thin-film 720p touchscreen. We present a number of unimanual and bimanual input techniques, including touch, drag, throw and resize of individual drones and compound models, as well as user interface elements such as self-levitating cone trees, 3D canvases and alert boxes. We describe application scenarios and depict future directions towards creating high-resolution self-levitating programmable matter.

In this paper, we explore how shape changing interfaces might be used to communicate emotions. We present two studies, one that investigates which shapes users might create with a 2D flexible surface, and one that studies the efficacy of the resulting shapes in conveying a set of basic emotions. Results suggest that shape parameters are correlated to the positive or negative character of an emotion, while parameters related to movement are correlated with arousal level. In several cases, symbolic shape expressions based on clear visual metaphors were used. Results from our second experiment suggest participants were able to recognize emotions given a shape with a good accuracy within 28% of the dimensions of the Circumplex Model. We conclude that shape and shape changes of a 2D flexible surface indeed appear able to convey emotions in a way that is worthy of future exploration.

ReFlex is a flexible smartphone with bend input and active haptic feedback. ReFlex's features allow the introduction of sensations such as friction or resistance. We report results from an experiment using ReFlex in a targeting task, as well as initial users' reactions to the prototype. We explore both absolute and relative tactile haptic feedback, paired with two types of bend input mappings: position-control and rate-control. We observed that position-controlled cursors paired well with relative bend feedback, while rate-controlled cursors paired well with absolute bend feedback to indicate targets. We also explored an eyes-free condition. Results suggest that while eyes-free, haptic feedback conditions were more error-prone than visual-only conditions, the size of the error was relatively small, and users were able to complete the task in all cases. We present two application scenarios that take advantage of the unique input and output modalities of ReFlex and discuss its potential for within document navigation.

We present HoloFlex, a 3D flexible smartphone featuring a light-field display consisting of a high-resolution P-OLED display and an array of 16,640 microlenses. HoloFlex allows mobile users to interact with 3D images featuring natural visual cues such as motion parallax and stereoscopy without glasses or head tracking. Its flexibility allows the use of bend input for interacting with 3D objects along the z axis. Images are rendered into 12-pixel wide circular blocks-pinhole views of the 3D scene-which enable ~80 unique viewports at an effective resolution of 160 × 104. The microlens array distributes each pixel from the display in a direction that preserves the angular information of light rays in the 3D scene. We present a preliminary study evaluating the effect of bend input vs. a vertical touch screen slider on 3D docking performance. Results indicate that bend input significantly improves movement time in this task. We also present 3D applications including a 3D editor, a 3D Angry Birds game and a 3D teleconferencing system that utilize bend input.

Tablet computers aim to bridge the gap between portability and productivity, reducing the need for users to carry multiple devices. However, despite increases in resolution, their displays are limited in size. This commonly results in sequential rather than parallel options for screen navigation, a significant drawback when multitasking. In this paper, we present DisplayCover, a tablet cover that integrates a physical keyboard as well as a touch and stylus sensitive thin-film e-ink display. We developed example applications to demonstrate the ability to dynamically alter the cover display content based on usage context, as well as concurrent access to multiple applications, stylus annotation, gestures and trackpad interactions.

Mobile devices can provide people with contextual information. This information may benefit a primary activity, assuming it is easily accessible. In this paper, we present DisplaySkin, a pose-aware device with a flexible display circling the wrist. DisplaySkin creates a kinematic model of a user's arm and uses it to place information in view, independent of body pose. In doing so, DisplaySkin aims to minimize the cost of accessing information without being intrusive. We evaluated our pose-aware display with a rotational pointing task, which was interrupted by a notification on DisplaySkin. Results show that a pose-aware display reduces the time required to respond to notifications on the wrist.

In this paper, we investigate the use of shape changes in a multi-segmented mobile device for triggering viewport transformations in its graphical interface. We study PaperFold, a foldable device with reconfigurable thin-film electrophoretic display tiles. PaperFold enables users to attach, reorient and fold displays in a mobile form factor that is thin and lightweight even when fully collapsed. We discuss how our design was informed by a participatory study that resulted in 14 preferred shape changes. In a subsequent study, we asked users to rank the utility of shape changes for triggering common view operations in map and text editing applications. Results suggest participants were able to attribute specific view operations as automated responses to folding, attaching, reorienting or detaching displays. Collated or full screen views were preferred when users collocated two displays. When adding a third display, alternative views such as toolbars or a list of apps were suggested. Showing 3D views was strongly associated with folding PaperFold segments into a three dimensional structure.

We present GoonQuad, an emotive quadruped capable of expressing emotional behaviours as a response to human touch. GoonQuad comprises five prerecorded states: angry, cheerful, sleepy, confused and a baseline breathing state. Each state is triggered by human touch in areas specified by the eyebrows and a tattoo, painted with conductive ink. Moreover, GoonQuad is capable of recording and replaying movements via direct user manipulation. To enable the robot to record and replay new motions, analog feedback servos were embedded in the 3D printed structure. Our aim was to develop a system where users can interact with a robot naturally and the robot can adapt to this natural interaction.

We present Mood Fern: digital flora which responds to touch. Depending on the length and intensity of the touch a subset of leaves physically react. The leaves respond on a spectrum of slight oscillation, imitating the effects of swaying in a slight breeze, to complete deformation, as if they were physically trying to respond in a similar manner. Mood Fern's reference to nature highlights its appeal to calm computing. Painted capacitive sensors mimic the appearance of leaf veins and Flexinol SMA wire is used to actuate The Mood Fern's paper structures.

In early design, instrumenting an object with touch sensing capability, especially one with complex surface geometry, can be problematic. In this paper, we show how resistive graph patterns--or resigraphs--can be used to quickly fabricate multi-touch sensors tailored to an object's shape. In very early ideation, resigraphs can be drawn using conductive ink. In later refinements they can be silk-screened or laser cut from off-the-shelf materials. A resigraph uses a commonly available microprocessor (e.g. Arduino), re-quires only three wires, and enables touch input on non-planar and non-developable surfaces.

We present MorePhone, an actuated flexible smartphone with a thin-film E Ink display. MorePhone uses shape memory alloys to actuate the entire surface of the display as well as individual corners. We conducted a participatory study to determine how users associate urgency and notification type with full screen, 1 corner, 2 corner and 3 corner actuations of the smartphone. Results suggest that with the current prototype, actuated shape notifications are useful for visual feedback. Urgent notifications such as alarms and voice calls were best matched with actuation of the entire display surface, while less urgent notifications, such as software notifications were best matched to individual corner bends. While different corner actuations resulted in significantly different matches between notification types, medium urgency notification types were treated as similar, and best matched to a single corner bend. A follow-up study suggested that users prefer to dedicate each corner to a specific type of notification. Users would like to personalize the assignment of corners to notification type. Animation of shape actuation significantly increased the perceived urgency of any of the presented shapes.

We present FlexView, a set of interaction techniques for Z-axis navigation on touch-enabled flexible mobile devices. FlexView augments touch input with bend to navigate through depth-arranged content. To investigate Z-axis navigation with FlexView, we measured document paging efficiency using touch, against two forms of bend input: bending the side of the display (leafing) and squeezing the display (squeezing). In addition to moving through the Z-axis, the second experiment added X-Y navigation in a pan-and-zoom task. Pinch gestures were compared to squeezing and leafing for zoom operations, while panning was consistently performed using touch. Our experiments demonstrate that bend interaction is comparable to touch input for navigation through stacked content. Squeezing to zoom recorded the fastest times in the pan-and-zoom task. Overall, FlexView allows users to easily browse depth arranged information spaces without sacrificing traditional touch interactions.

We present Unifone, a prototype mobile device that explores the use of auxiliary finger input in one-handed mobile interaction. Using force-distributed pressure sensing along the side of device, we examine how squeeze-based gestures impact four common mobile interactions: scrolling, map navigating, text formatting, and application switching. We evaluated the use of Unifone in these tasks using one-handed interactions by the non-dominant hand. Our user study shows that one-handed isometric gestures perform best when they augment rather than restrict or alter the primary pointing action of the thumb and, generally, are suitable for coarse isometric pressure input.

Until recently, the ways in which we interacted with computers were limited by the flat, rigid and rectangular form factors of cathode-ray tubes and liquid crystal displays. Technical limitations forced user interfaces into boxes, limiting interactions to 2D pointing and keyboard input. With technological advances in flexible sensor and display technologies, we are experiencing a new revolution in human–computer interaction (HCI): one in which user interfaces can be worn on the body as if they were cloth, used in the office as if they were paper and used in architecture as if they were wallpapers (Co and Pashenkov, 2008; Buechley and Mellis, 2010; Lahey et al., 2011). Rather than being rigid and static, the user interfaces of tomorrow will be able to have a shape that accommodates the user’s context and fits the data on display. For instance, if a user wants to explore geographic information, they can use a spherical display that does not require distortion of the earth projection (Stevenson, 2010). When reading an interactive map, users can extend their mobile’s screen real estate by unfolding their pocket-size flexible e-paper display.

With the emergence of flexible display technologies, it will be necessary for interface designers to move beyond flat interfaces and to contextualize interaction in an object's physical shape. Grounded in early explorations of organic user interfaces (OUIs), this paper examines the evolving relationship between industrial and interaction designs and examines how not only what we design is changing, but how we design too. First, we discuss how (and why) to better support the design of OUIs: how supporting sketching, a fundamental activity of many design fields, is increasingly critical and why a ‘hypercontextualized’ approach to their design can reduce the drawbacks met when everyday objects become interactive. Finally, underlying both these points is the maturation of technology to that of a computational material; when interactive hardware is seamlessly melded into an object's shape, the ‘computer’ disappears and is better seen as a basic design material that, incidentally, happens to have interactive behavior.

Stacking physical documents is one of the main forms of spatio-temporal organization of information. We present DisplayStacks, a system that enables physical stacking of digital documents via piles of flexible E Ink displays. With a conductive dot pattern sensor attached to the flexible display, we dynamically track the position and orientation of these displays in relation to one another. We introduce mechanisms for interacting with these physical stacks for access and manipulation of information using asymmetric bi-manual interactions, such as providing contextual overviews. Initial user experiences indicate a preference for linear overlaps as a stacking configuration.

In this paper, we present TeleHuman, a cylindrical 3D display portal for life-size human telepresence. The TeleHuman 3D videoconferencing system supports 360 degree motion parallax as the viewer moves around the cylinder and optionally, stereoscopic 3D display of the remote person. We evaluated the effect of perspective cues on the conveyance of nonverbal cues in two experiments using a one-way telecommunication version of the system. The first experiment focused on how well the system preserves gaze and hand pointing cues. The second experiment evaluated how well the system conveys 3D body postural information. We compared 3 perspective conditions: a conventional 2D view, a 2D view with 360 degree motion parallax, and a stereoscopic view with 360 degree motion parallax. Results suggest the combined presence of motion parallax and stereoscopic cues significantly improved the accuracy with which participants were able to assess gaze and hand pointing cues, and to instruct others on 3D body poses. The inclusion of motion parallax and stereoscopic cues also led to significant increases in the sense of social presence and telepresence reported by participants.

While large flat vertical displays may facilitate persistent public sharing of work, they may do so at a cost of limited personal display space when everyone can see each other's activity. By contrast, new form factors, such as spherical displays, support sharing display space by limiting the user's view to at most one hemisphere. In this paper, we investigate how different interactive large display form factors can support differences in sharing of information during competitive and cooperative task conditions. We implemented three different large display types: spherical, flat, and a flat display with divider. Results show that task performance of the flat display with divider did not differ significantly from that of the spherical display. Additionally, we implemented and compared three peeking techniques that facilitated sharing of information. Results show participants peeked significantly more in competitive tasks than they did in cooperative tasks. Usage of peeking techniques between the spherical display and the flat display with divider were similar, and distinct from that of the flat display. Not surprisingly, results show that the affordance of easily glancing at a partner's work on the flat display provided a significant advantage in cooperative tasks.

The Hum is an immersive art installation filled with hundreds of suspended furred catkins surrounding a cocoon. As visitors enter the space, catkins twitch, shiver and hum. In The Hum, we explore the idea of computers that communicate ephemerally through alterations of room and space.

In this paper we describe A Flock of Birds, an interactive, robotic origami art installation. The art installation explores folding paper as a fusion of input, output and computation while simultaneously providing its audience with a fun and exciting experience.

With WristFlicker, we detect wrist movement through sets of stretch sensors embedded in clothing. Our system supports wrist rotation (pronation/supination), and both wrist tilts (flexion/extension and ulnar/radial deviation). Each wrist movement is measured by two opposing stretch sensors, mimicking the counteracting movement of muscles. We discuss interaction techniques that allow a user to control a music player through this lightweight input.

Programmers routinely use source code snippets to increase their productivity. However, locating and adapting code snippets to the current context still takes time: for example, variables must be renamed, and dependencies included. We believe that when programmers decide to invest time in creating a new code snippet from scratch, they would also be willing to spend additional effort to make that code snippet configurable and easy to integrate. To explore this insight, we built SnipMatch, a plug-in for the Eclipse IDE. SnipMatch introduces a simple markup that allows snippet authors to specify search patterns and integration instructions. SnipMatch leverages this information, in conjunction with current code context, to improve snippet search and parameterization. For example, when a search query includes local variables, SnipMatch suggests compatible snippets, and automatically adapts them by substituting in these variables. In the lab, we observed that participants integrated snippets faster when using SnipMatch than when using standard Eclipse. Findings from a public deployment to 93 programmers suggest that SnipMatch has become integrated into the work practices of real users.

In this paper, we present a study of the effects of structural holds and rigidity of a flexible display on touch pointing and dragging performance. We discuss an observational study in which we collected common holds used when pointing on a mockup paper display. We also measured the force patterns each hold generated within the display surface. We analyzed this data to produce 3 force zones in the display for each of the four most frequently observed holds: the grip zone, rigid zone, and the flexible zone. We report on an empirical evaluation in which we compared the efficiency of pointing and dragging operations between holds, and between structural zones within holds, using a real flexible Lumalive display. Results suggest that structural force distributions in a flexible display affect the Index of Performance of both pointing and dragging tasks, irrespective of hold, with rigid parts of the display yielding a 12% average performance gain over flexible areas.

Flexible displays potentially allow for interaction styles that resemble those used in paper documents. Bending the display, e.g., to page forward, shows particular promise as an interaction technique. In this paper, we present an evaluation of the effectiveness of various bend gestures in executing a set of tasks with a flexible display. We discuss a study in which users designed bend gestures for common computing actions deployed on a smartphone-inspired flexible E Ink prototype called PaperPhone. We collected a total of 87 bend gesture pairs from ten participants and their appropriateness over twenty actions in five applications. We identified six most frequently used bend gesture pairs out of 24 unique pairs. Results show users preferred bend gestures and bend gesture pairs that were conceptually simpler, e.g., along one axis, and less physically demanding. There was a strong agreement among participants to use the same three pairs in applications: (1) side of display, up/down (2) top corner, up/down (3) bottom corner, up/down. For actions with a strong directional cue, we found strong consensus on the polarity of the bend gestures (e.g., navigating left is performed with an upwards bend gesture, navigating right, downwards). This implies that bend gestures that take directional cues into account are likely more natural to users.

It is important for users to understand the fundamental tradeoff between sharing information and preserving privacy in collaboration. The more information is shared about one's actions, the less privacy is preserved. Sharing information may, in fact, counter-intuitively result in increased social stress in some cases. Maintaining privacy while allowing for the sharing of information is an important consideration for successful collaboration and we believe display form factor matters for this tradeoff. New form factors such as spherical displays support privacy naturally, by limiting a user's view to at most one hemisphere. In this video we show how different types of interactive large display form factor can provide a balance between privacy and the sharing of information in a cooperative game.

Advances in display, sensor and actuator technology are changing the field of TEI, and opening new research areas. While modern interfaces have been designed for traditional planar and static display devices, next-generation UI allow digital objects to change their shape and embed displays anywhere. Fitting into the paradigm of Organic User Interfaces, these developments require us to reexamine and reevaluate some of the basic design principles and interaction styles currently used. This Second International Workshop on Organic User Interfaces will bring together experts to discuss, brainstorm and prototype next generation of user interfaces.

In this paper, we present a user interface for a textile computer display. It allows users to interact in ways that flow from the natural properties of cloth, with an interface that physically conforms to the shape of the object on which a task is performed. While recent work on flexible interfaces has shown promising results, physical properties such as the rigidity of the display remain a barrier to interaction scenarios that are truly physically flexible. We discuss interaction techniques for our cloth user interface, which include gestures such as pinching, draping, stretching and squeezing. Our interaction techniques employ the unique physical characteristics of cloth, including flexibility and shape-taking. We reflect on the system and examine potential directions for future work.

In this paper, we introduce a multi-touch display surface that can be dynamically deformed from a flat circular display to a convex or concave hemispherical display. A rubber latex material is used for the display surface allowing it to inflate or deflate as air is pumped into or out of an airtight container. The elasticity of the display surface also allows users to deform the surface by varying the pressure exerted on it. This deformation may be detected by the device and provides a z-axis input in addition to the typical x,y-axis inputs of flat-screen displays.

This paper introduces DisplayObjects, a rapid prototyping workbench that allows functional interfaces to be projected onto real 3D physical prototypes. DisplayObjects uses a Vicon motion capture system to track the location of physical models. 3D software renditions of the 3D physical model are then texture-mapped with interactive behavior and projected back onto the physical model to allow real- time interactions with the object. We discuss the implementation of the system, as well as a selection of one and two-handed interaction techniques for DisplayObjects. We conclude with a design case that comments on some of the early design experiences with the system.

In this paper, we present a tiled display system made out of hexagonal cardboard screens with no visible bezel. Use of a bezel-less hexagonal form factor allows users to build larger multiform displays out of smaller tiles. Individual display tiles can be picked up to allow tangible interactions with physics simulations that are rendered onto the individual tiles. The corners of each hexagon are marked with invisible infrared retro-reflective dots. Computer vision is used to track the 3D location and orientation of these tiles. Our prototype projects back images onto each individual display. This allows for a seamless interaction experience that anticipates wireless Organic LED technology. We discuss a number of applications and interaction techniques for compound cardboard displays, which include tilting, rotating, moving and touching of tiles.

We present a Fitts' Law evaluation of a number of eye tracking and manual input devices in the selection of large visual targets. We compared performance of two eye tracking techniques, manual click and dwell time click, with that of mouse and stylus. Results show eye tracking with manual click outperformed the mouse by 16%, with dwell time click 46% faster. However, eye tracking conditions suffered a high error rate of 11.7% for manual click and 43% for dwell time click conditions. After Welford correction eye tracking still appears to outperform manual input, with IPs of 13.8 bits/s for dwell time click, and 10.9 bits/s for manual click. Eye tracking with manual click provides the best tradeoff between speed and accuracy, and was preferred by 50% of participants. Mouse and stylus had IPs of 4.7 and 4.2 respectively. However, their low error rate of 5% makes these techniques more suitable for refined target selection.

This paper presents Foldable User Interfaces (FUI), a combination of a 3D GUI with windows imbued with the physics of paper, and Foldable Input Devices (FIDs). FIDs are sheets of paper that allow realistic transformations of graphical sheets in the FUI. Foldable input devices are made out of construction paper augmented with IR reflectors, and tracked by computer vision. Window sheets can be picked up and flexed with simple movements and deformations of the FID. FIDs allow a diverse lexicon of one-handed and two-handed interaction techniques, including folding, bending, flipping and stacking. We show how these can be used to ease the creation of simple 3D models, but also for tasks such as page navigation.

In this paper, we present an attentive windowing technique that uses eye tracking, rather than manual pointing, for focus window selection. We evaluated the performance of 4 focus selection techniques: eye tracking with key activation, eye tracking with automatic activation, mouse and hotkeys in a typing task with many open windows. We also evaluated a zooming windowing technique designed specifically for eye-based control, comparing its performance to that of a stan-dard tiled windowing environment. Results indicated that eye tracking with automatic activation was, on average, about twice as fast as mouse and hotkeys. Eye tracking with key activation was about 72% faster than manual conditions, and preferred by most participants. We believe eye input performed well because it allows manual input to be provided in parallel to focus selection tasks. Results also suggested that zooming windows outperform static tiled windows by about 30%. Furthermore, this performance gain scaled with the number of windows used. We conclude that eye-controlled zooming windows with key activation pro-vides an efficient and effective alternative to current focus window selection techniques.

In this paper, we present Paper Windows, a prototype windowing environment that simulates the use of digital paper displays. By projecting windows on physical paper, Paper Windows allows the capturing of physical affordances of paper in a digital world. The system uses paper as an input device by tracking its motion and shape with a Vicon Motion Capturing System. We discuss the design of a number of interaction techniques for manipulating information on paper displays.

This paper presents Latent Semantic Googling, a variant of Landauer's Latent Semantic Indexing that uses the Google search engine to judge the semantic closeness of sets of words and phrases. This concept is implemented via Ambient Google, a system for augmenting conversations through the classification of discussed topics. Ambient Google uses a speech recognition engine to generate Google keyphrase queries directly from conversations. These queries are used to analyze the semantics of the conversation, and infer related topics that have been discussed. Conversations are visualized using a spring-model algorithm representing common topics. This allows users to browse their conversation as a contextual relationship between discussed topics, and augment their discussion through the use of related websites discovered by Google. An evaluation of Ambient Google is presented, discussing user reaction to the system.

One of the problems with mobile media devices is that they may distract users during critical everyday tasks, such as navigating the streets of a busy city. We addressed this issue in the design of eyeLook: a platform for attention sensitive mobile computing. eyeLook appliances use embedded low cost eyeCONTACT sensors (ECS) to detect when the user looks at the display. We discuss two eyeLook applications, seeTV and seeTXT, that facilitate courteous media consumption in mobile contexts by using the ECS to respond to user attention. seeTV is an attentive mobile video player that automatically pauses content when the user is not looking. seeTXT is an attentive speed reading application that flashes words on the display, advancing text only when the user is looking. By making mobile media devices sensitive to actual user attention, eyeLook allows applications to gracefully transition users between consuming media, and managing life.

We introduce ViewPointer, a wearable eye contact sensor that detects deixis towards ubiquitous computers embedded in real world objects. ViewPointer consists of a small wearable camera no more obtrusive than a common Bluetooth headset. ViewPointer allows any real-world object to be augmented with eye contact sensing capabilities, simply by embedding a small infrared (IR) tag. The headset camera detects when a user is looking at an infrared tag by determining whether the reflection of the tag on the cornea of the user's eye appears sufficiently central to the pupil. ViewPointer not only allows any object to become an eye contact sensing appliance, it also allows identification of users and transmission of data to the user through the object. We present a novel encoding scheme used to uniquely identify ViewPointer tags, as well as a method for transmitting URLs over tags. We present a number of scenarios of application as well as an analysis of design principles. We conclude eye contact sensing input is best utilized to provide context to action.

No abstract available.

No abstract available.

We present ECSGlasses: wearable eye contact sensing glasses that detect human eye contact. ECSGlasses report eye contact to digital devices, appliances and EyePliances in the user's attention space. Devices use this attentional cue to engage in a more sociable process of turn taking with users. This has the potential to reduce inappropriate intrusions, and limit their disruptiveness. We describe new prototype systems, including the Attentive Messaging Service (AMS), the Attentive Hit Counter, the first person attentive camcorder eyeBlog, and an updated Attentive Cell Phone. We also discuss the potential of these devices to open new windows of interaction using attention as a communication modality. Further, we present a novel signal-encoding scheme to uniquely identify EyePliances and users wearing ECSGlasses in multiparty scenarios.

As ubiquitous computing becomes more prevalent, greater consideration will have to be taken on how devices interrupt us and vie for our attention. This paper describes Auramirror, an interactive art piece that raises questions of how computers use our attention. By measuring attention and visualizing the results for the audience in real-time, Auramirror brings the subject matter to the forefront of the audience's consideration. Finally, some ways of using the Auramirror system to help in the design of attention sensitive devices are discussed.

GAZE-2 is a novel group video conferencing system that uses eye-controlled camera direction to ensure parallax-free transmission of eye contact. To convey eye contact, GAZE-2 employs a video tunnel that allows placement of cameras behind participant images on the screen. To avoid parallax, GAZE-2 automatically directs the cameras in this video tunnel using an eye tracker, selecting a single camera closest to where the user is looking for broadcast. Images of users are displayed in a virtual meeting room, and rotated towards the participant each user looks at. This way, eye contact can be conveyed to any number of users with only a single video stream per user. We empirically evaluated whether eye contact perception is affected by automated camera direction, which causes angular shifts in the transmitted images. Findings suggest camera shifts do not affect eye contact perception, and are not considered highly distractive.

We present an experiment examining effects of gaze on speech during three-person conversations. Understanding such effects is crucial for the design of teleconferencing systems and Collaborative Virtual Environments (CVEs). Previous findings suggest subjects take more turns when they experience more gaze. We evaluated whether this is because more gaze allowed them to better observe whether they were being addressed. We compared speaking behavior between two conditions: (1) in which subjects experienced gaze synchronized with conversational attention, and (2) in which subjects experienced random gaze. The amount of gaze experienced by subjects was a covariate. Results show subjects were 22% more likely to speak when gaze behavior was synchronized with conversational attention. However, covariance analysis showed these results were due to differences in amount of gaze rather than synchronization of gaze, with correlations of .62 between amount of gaze and amount of subject speech. Task performance was 46% higher when gaze was synchronized. We conclude it is commendable to use synchronized gaze models when designing CVEs, but depending on task situation, random models generating sufficient amounts of gaze may suffice.

In this paper, we propose a tentative framework for the classification of Attentive Interfaces, a new category of user interfaces. An Attentive Interface is a user interface that dynamically prioritizes the information it presents to its users, such that information processing resources of both user and system are optimally distributed across a set of tasks. The interface does this on the basis of knowledge – consisting of a combination of measures and models – of the past, present and future state of the user’s attention, given the availability of system resources. We will show how the Attentive Interface provides a natural extension to the windowing paradigm found in Graphical User Interfaces. Our taxonomy of Attentive Interfaces allows us to identify classes of user interfaces that would benefit most from the ability to sense, model and optimize the user’s attentive state. In particular, we show how systems that influence user workflow in concurrent task situations, such as those involved with management of multiparty communication, may benefit from such facilities.

In recent years, there has been a resurgence of interest in the use of eye tracking systems for interactive purposes. However, it is easy to be fooled by the interactive power of eye tracking. When first encountering eye based interaction, most people are genuinely impressed with the almost magical window into the mind of the user that it seems to provide. There are two reasons why this belief may lead to subsequent disappointment. Firstly, although current eye tracking equipment is far superior to that used in the seventies and early eighties, it is by no means perfect. For example, there is still the tradeoff between the use of an obtrusive head-based system or a desk-based system with limited head movement. Such technical problems continue to limit the usefulness of eye tracking as a generic form of input. Secondly, there are real methodological problems regarding the interpretation of eye input for use in graphical user interfaces. One example, the "Midas Touch" problem, is observed in systems that use eye movements to directly control a mouse cursor. When does the system decide that a user is interested in a visual object? Systems that implement dwell time for this purpose run the risk of disallowing visual scanning behavior, requiring users to control their eye movements for the purposes of output, rather than input. However, difficulties in the interpretation of visual interest remain even when systems use another input modality for signaling intent. Another classic methodological problem is exemplified by the application of eye movement recording in usability studies. Although eye fixations provide some of the best measures of visual interest, they do not provide a measure of cognitive interest. It is one thing to determine whether a user has observed certain visual information, but quite another to determine whether this information has in fact been processed or understood. Some of our technological problems can and will be solved. However, we believe that our methodological issues point to a more fundamental problem: What is the nature of the input information conveyed by eye movements and to what interactive functions can this information provide added value?

In multi-agent, multi-user environments, users as well as agents should have a means of establishing who is talking to whom. In this paper, we present an experiment aimed at evaluating whether gaze directional cues of users could be used for this purpose. Using an eye tracker, we measured subject gaze at the faces of conversational partners during four-person conversations. Results indicate that when someone is listening or speaking to individuals, there is indeed a high probability that the person looked at is the person listened (p=88%) or spoken to (p=77%). We conclude that gaze is an excellent predictor of conversational attention in multiparty conversations. As such, it may form a reliable source of input for conversational systems that need to establish whom the user is speaking or listening to. We implemented our findings in FRED, a multi-agent conversational system that uses eye input to gauge which agent the user is listening or speaking to.

In this paper, we discuss why, in designing multiparty mediated systems, we should focus first on providing non-verbal cues which are less redundantly coded in speech than those normally conveyed by video. We show how conveying one such cue, gaze direction, may solve two problems in multiparty mediated communication and collaboration: knowing who is talking to whom, and who is talking about what. As a candidate solution, we present the GAZE Groupware System, which combines support for gaze awareness in multiparty mediated communication and collaboration with small and linear bandwidth requirements. The system uses an advanced, desk- mounted eyetracker to metaphorically convey gaze awareness in a 3D virtual meeting room and within shared documents.

We present the SensOrg, a musical cyberinstrument designed as a modular assembly of input/output devices and musical software, mapped and arranged according to functional characteristics of the man-instrument system. We discuss how the cognitive ergonomics of non-verbal and symbolic task modalities influenced the design of our hardware interface for asynchronous as well as synchronous task situations. Using malleable atoms and tangible bits, we externally represented the musical functionality in a physical interface which is totally flexible yet completely freezable.

The representation and manipulation of sound within multimedia systems is an important and currently under-researched area. The paper gives an overview of the authors' work on the direct manipulation of audio information, and describes a solution based upon the navigation of four-dimensional scaled timbre spaces. Three hardware input devices were experimentally evaluated for use in a timbre space navigation task: the Apple Standard Mouse, Gravis Advanced Mousestick II joystick (absolute and relative) and the Nintendo Power Glove. Results show that the usability of these devices significantly affected the efficacy of the system, and that conventional low-cost, low-dimensional devices provided better performance than the low-cost, multidimensional dataglove.