Reinforcement Learning from Human Feedback (RLHF) is a methodology that aligns agent behavior with human preferences by integrating human feedback into the agent's training process. We introduce a possible framework that employs passive Brain-Computer Interfaces (BCI) to guide agent training from implicit neural signals. We present and release a novel dataset of functional near-infrared spectroscopy (fNIRS) recordings collected from 25 human participants across three domains: a Pick-and-Place Robot, Lunar Lander, and Flappy Bird. We train classifiers to predict levels of agent performance (optimal, sub-optimal, or worst-case) from windows of preprocessed fNIRS feature vectors, achieving an average F1 score of 67% for binary classification and 46% for multi-class models averaged across conditions and domains. We also train regressors to predict the degree of deviation between an agent's chosen action and a set of near-optimal policies, providing a continuous measure of performance. We evaluate cross-subject generalization and demonstrate that fine-tuning pre-trained models with a small sample of subject-specific data increases average F1 scores by 17% and 41% for binary and multi-class models, respectively. Our work demonstrates that mapping implicit fNIRS signals to agent performance is feasible and can be improved, laying the foundation for future brain-driven RLHF systems.

Brain-Computer Interfaces (BCIs) using functional near-infrared spectroscopy (fNIRS) hold promise for future interactive user interfaces due to their ease of deployment and declining cost. However, they typically require a separate calibration process for each user and task, which can be burdensome. Machine learning helps, but faces a data scarcity problem. Due to inherent inter-user variations in physiological data, it has been typical to create a new annotated training dataset for every new task and user. To reduce dependence on such extensive data collection and labeling, we present an adaptive learning framework, NIRS-X, to harness more easily accessible unlabeled fNIRS data. NIRS-X includes two key components: NIRSiam and NIRSformer. We use the NIRSiam algorithm to extract generalized brain activity representations from unlabeled fNIRS data obtained from previous users and tasks, and then transfer that knowledge to new users and tasks. In conjunction, we design a neural network, NIRSformer, tailored for capturing both local and global, spatial and temporal relationships in multi-channel fNIRS brain input signals. By using unlabeled data from both a previously released fNIRS2MW visual n-back dataset and a newly collected fNIRS2MW audio n-back dataset, NIRS-X demonstrates its strong adaptation capability to new users and tasks. Results show comparable or superior performance to supervised methods, making NIRS-X promising for real-world fNIRS-based BCIs.

In this art project, we create Affective Color Fields: an interactive artifact that takes in a user's narrative of their emotional experiences and dynamically transforms it into Rothkoesque color fields through emotion classification. Inspired by Mark Rothko's abstract depiction of human emotions and Merleau-Ponty's phenomenological inquiry, we wish to establish an intimate relationship between interactive art and the subject by employing user's own interpretation and framing of life events. Through the performative and improvisational art-making process, users can playfully appropriate our artifact for a rich and personal aesthetic experience.

Brain-computer interfaces (BCI) are an emerging technology with many potential applications. Functional near-infrared spectroscopy (fNIRS) can provide a convenient and unobtrusive real time input for BCI. fNIRS is especially promising as a signal that could be used to automatically classify a user’s current cognitive workload. However, the data needed to train such a classifier is currently not widely available, difficult to collect, and difficult to interpret due to noise and cross-subject variation. A further challenge is the need for significant user-specific calibration. To address these issues, we introduce a new dataset gathered from 15 subjects and a new multi-stage supervised machine learning pipeline. Our approach learns from both observed data and augmented data derived from multiple subjects in its early stages, and then fine-tunes predictions to an individual subject in its last stage. We show promising gains in accuracy in a standard “n-back” cognitive workload classification task compared to baselines that use only subject-specific data or only group-level data, even when our approach is given much less subject-specific data. Even though these experiments analyzed the data retrospectively, we carefully removed anything from our process that could not have been done in real time, because our process is targeted at future real-time operation. This paper contributes a new dataset, a new multi-stage training pipeline, results showing significant improvement compared to alternative pipelines, and discussion of the implications for user interface design. Our complete dataset and software are publicly available at https://tufts-hci-lab.github.io/code_and_datasets/. We hope these results make fNIRS-based interactive brain input easier for a wide range of future researchers and designers to explore.

Frameworks such as Direct Manipulation or Instrumental Interaction have been an important force in HCI research. Evaluating the impact of frameworks can identify whether and how a framework was used, how it has evolved, and what trends have developed over time. However, studying the impact of such theoretical contributions requires consideration of various perspectives and level of impact. As a case study for investigating the impact of theoretical work in HCI, we present our evaluation of the impact of the Reality Based Interaction (RBI) framework, introduced by the authors in 2008. We provide our findings about the impact of the framework both on contemporary research, through content-based citation analysis, and in HCI education, through a survey we conducted on emerging interaction frameworks. The article contributes a comprehensive methodology for evaluating the impact of frameworks through our twofold approach: content-based citation analysis, including the design of a new citation typology; and a survey on the use of frameworks in education using a taxonomy of learning goals. We also consider the role of frameworks in HCI as well as the future of the RBI framework.

We present Brain Automated Chorales (BACh), an adaptive brain-computer system that dynamically increases the levels of difficulty in a musical learning task based on pianists' cognitive workload measured by functional near-infrared spectroscopy. As users' cognitive workload fell below a certain threshold, suggesting that they had mastered the material and could handle more cognitive information, BACh automatically increased the difficulty of the learning task. We found that learners played with significantly increased accuracy and speed in the brain-based adaptive task compared to our control condition. Participant feedback indicated that they felt they learned better with BACh and they liked the timings of the level changes. The underlying premise of BACh can be applied to learning situations where a task can be broken down into increasing levels of difficulty.

Emerging ultra-small wearables like smartwatches pose a design challenge for touch-based text entry. This is due to the "fat-finger problem," wherein users struggle to select elements much smaller than their fingers. To address this challenge, we developed DriftBoard, a panning-based text entry technique where the user types by positioning a movable qwerty keyboard on an interactive area with respect to a fixed cursor point. In this paper, we describe the design and implementation of DriftBoard and report results of a user study on a watch-size touchscreen. The study compared DriftBoard to two ultra-small keyboards, ZoomBoard (tapping-based) and Swipeboard (swiping-based). DriftBoard performed comparably (no significant difference) to ZoomBoard in the major metrics of text entry speed and error rate, and outperformed Swipeboard, which suggests that panning-based typing is a promising input method for text entry on ultra-small touchscreens.

Physiological data in its different dimensions—bioelectrical, biomechanical, biochemical, or biophysical—and collected through existing sensors or specialized biomedical devices, image capture, or other sources is pushing the boundaries of physiological computing for human-computer interaction (HCI). Although physiological computing shows the potential to enhance the way in which people interact with digital content, systems remain challenging to design and build. The aim of this special issue is to present outstanding work related to use of physiological data in HCI, setting additional bases for next-generation computer interfaces and interaction experiences. Topics covered in this issue include methods and methodologies, human factors, the use of devices, and applications for supporting the development of emerging interfaces.

A growing body of recent work has shown the feasibility of brain and body sensors as input to interactive systems. However, the interaction techniques and design decisions for their effective use are not well defined. We present a conceptual framework for considering implicit input from the brain, along with design principles and patterns we have developed from our work. We also describe a series of controlled, offline studies that lay the foundation for our work with functional near-infrared spectroscopy (fNIRS) neuroimaging, as well as our real-time platform that serves as a testbed for exploring brain-based adaptive interaction techniques. Finally, we present case studies illustrating the principles and patterns for effective use of brain data in human--computer interaction. We focus on signals coming from the brain, but these principles apply broadly to other sensor data and in domains such as aviation, education, medicine, driving, and anything involving multitasking or varying cognitive workload.

Dynamic difficulty adjustments can be used in human-computer systems in order to improve user engagement and performance. In this paper, we use functional near-infrared spectroscopy (fNIRS) to obtain passive brain sensing data and detect extended periods of boredom or overload. From these physiological signals, we can adapt a simulation in order to optimize workload in real-time, which allows the system to better fit the task to the user from moment to moment. To demonstrate this idea, we ran a laboratory study in which participants performed path planning for multiple unmanned aerial vehicles (UAVs) in a simulation. Based on their state, we varied the difficulty of the task by adding or removing UAVs and found that we were able to decrease error by 35% over a baseline condition. Our results show that we can use fNIRS brain sensing to detect task difficulty in real-time and construct an interface that improves user performance through dynamic difficulty adjustment.

The bubble cursor is a promising cursor expansion technique, improving a user's movement time and accuracy in pointing tasks. We introduce a brain-based target expansion system, which improves the efficacy of bubble cursor by increasing the expansion of high importance targets at the optimal time based on brain measurements correlated to a particular type of multitasking. We demonstrate through controlled experiments that brain-based target expansion can deliver a graded and continuous level of assistance to a user according to their cognitive state, thereby improving task and speed-accuracy metrics, even without explicit visual changes to the system. Such an adaptation is ideal for use in complex systems to steer users toward higher priority goals during times of increased demand.

Creativity is a crucial skill in today's knowledge economy, and creativity support tools are a valuable and important area of human-computer interaction research. Passive affective priming has been shown to be an effective means for enhancing creativity. Based on music cognition research, we propose music improvisation as an active and participatory cognitive prime for boosting creative ability. To make music improvisation accessible to all individuals regardless of music expertise, we present a novel instrument with an adaptive interface that facilitates music creation. We demonstrate that improvising music with our adaptive instrument provides an immediate boost to creative ability, even for people without musical training. We quantify the efficacy of interface adaptation techniques for enabling creative expression.

We show how brain sensing can lend insight to the evaluation of visual interfaces and establish a role for fNIRS in visualization. Research suggests that the evaluation of visual design benefits by going beyond performance measures or questionnaires to measurements of the user's cognitive state. Unfortunately, objectively and unobtrusively monitoring the brain is difficult. While functional near-infrared spectroscopy (fNIRS) has emerged as a practical brain sensing technology in HCI, visual tasks often rely on the brain's quick, massively parallel visual system, which may be inaccessible to this measurement. It is unknown whether fNIRS can distinguish differences in cognitive state that derive from visual design alone. In this paper, we use the classic comparison of bar graphs and pie charts to test the viability of fNIRS for measuring the impact of a visual design on the brain. Our results demonstrate that we can indeed measure this impact, and furthermore measurements indicate that there are not universal differences in bar graphs and pie charts.

This paper describes the Brainput system, which learns to identify brain activity patterns occurring during multitasking. It provides a continuous, supplemental input stream to an interactive human-robot system, which uses this information to modify its behavior to better support multitasking. This paper demonstrates that we can use non-invasive methods to detect signals coming from the brain that users naturally and effortlessly generate while using a computer system. If used with care, this additional information can lead to systems that respond appropriately to changes in the user's state. Our experimental study shows that Brainput significantly improves several performance metrics, as well as the subjective NASA-Task Load Index scores in a dual-task human-robot activity.

We propose a new diagrammatic notation system for Goals, Operators, Methods, Selection rules (GOMS), called Codein, with extensions to support the evaluation of Reality Based Interaction Styles. The proposed notation gives added power to GOMS to model and evaluate the task completion time of parallel actions during the performance of a task, something that was previously only possible using CPM-GOMS, which is far more complicated to use. Codein's evaluative power is verified through an experiment. The first condition of the experiment compares the completion time predicted by a GOMSL model, a Codein model, and the actual completion time of participants in a direct manipulation task. The second compares the completion time of participants in a Tangible User Interface task with predictions by a GOMSL model and a Codein model. Predicted task times by Codein in both conditions come close to the actual experimental results.

This project represents a first step towards bridging the gap between HCI and cognition research. Using functional near-infrared spectroscopy (fNIRS), we introduce tech-niques to non-invasively measure a range of cognitive workload states that have implications to HCI research, most directly usability testing. We present a set of usability experiments that illustrates how fNIRS brain measurement provides information about the cognitive demands placed on computer users by different interface designs.

Multitasking has become an integral part of work environments, even though people are not well-equipped cognitively to handle numerous concurrent tasks effectively. Systems that support such multitasking may produce better performance and less frustration. However, without understanding the user's internal processes, it is difficult to determine optimal strategies for adapting interfaces, since all multitasking activity is not identical. We describe two experiments leading toward a system that detects cognitive multitasking processes and uses this information as input to an adaptive interface. Using functional near-infrared spectroscopy sensors, we differentiate four cognitive multitasking processes. These states cannot readily be distinguished using behavioral measures such as response time, accuracy, keystrokes or screen contents. We then present our human-robot system as a proof-of-concept that uses real-time cognitive state information as input and adapts in response. This prototype system serves as a platform to study interfaces that enable better task switching, interruption management, and multitasking.

This paper is a short introduction to a special ICMI session on brain-computer interaction. During this paper, we first discuss problems, solutions, and a five-year view for brain-computer interaction. We then talk further about unique issues with multimodal and hybrid brain-computer interfaces, which could help address many current challenges. This paper presents some potentially controversial views, which will hopefully inspire discussion about the different views on brain-computer interfacing, how to embed brain-computer interfacing in a multimodal and multi-party context, and, more generally, how to look at brain-computer interfacing from an ambient intelligence point of view.

A well designed user interface (UI) should be transparent, allowing users to focus their mental workload on the task at hand. We hypothesize that the overall mental workload required to perform a task using a computer system is composed of a portion attributable to the difficulty of the underlying task plus a portion attributable to the complexity of operating the user interface. In this regard, we follow Shneiderman's theory of syntactic and semantic components of a UI. We present an experiment protocol that can be used to measure the workload experienced by users in their various cognitive resources while working with a computer. We then describe an experiment where we used the protocol to quantify the syntactic workload of two user interfaces. We use functional near infrared spectroscopy, a new brain imaging technology that is beginning to be used in HCI. We also discuss extensions of our techniques to adaptive interfaces.

Much of the work done in the field of tangible interaction has focused on creating tools for learning; however, in many cases, little evidence has been provided that tangible interfaces offer educational benefits compared to more conventional interaction techniques. In this paper, we present a study comparing the use of a tangible and a graphical interface as part of an interactive computer programming and robotics exhibit that we designed for the Boston Museum of Science. In this study, we have collected observations of 260 museum visitors and conducted interviews with 13 family groups. Our results show that visitors found the tangible and the graphical systems equally easy to understand. However, with the tangible interface, visitors were significantly more likely to try the exhibit and significantly more likely to actively participate in groups. In turn, we show that regardless of the condition, involving multiple active participants leads to significantly longer interaction times. Finally, we examine the role of children and adults in each condition and present evidence that children are more actively involved in the tangible condition, an effect that seems to be especially strong for girls.

Because functional near-infrared spectroscopy (fNIRS) eases many of the restrictions of other brain sensors, it has potential to open up new possibilities for HCI research. From our experience using fNIRS technology for HCI, we identify several considerations and provide guidelines for using fNIRS in realistic HCI laboratory settings. We empirically examine whether typical human behavior (e.g. head and facial movement) or computer interaction (e.g. keyboard and mouse usage) interfere with brain measurement using fNIRS. Based on the results of our study, we establish which physical behaviors inherent in computer usage interfere with accurate fNIRS sensing of cognitive state information, which can be corrected in data analysis, and which are acceptable. With these findings, we hope to facilitate further adoption of fNIRS brain sensing technology in HCI research.

Tangible interaction shows promise to significantly enhance computer-mediated support for activities such as learning, problem solving, and design. However, tangible user interfaces are currently considered challenging to design and build. Designers and developers of these interfaces encounter several conceptual, methodological, and technical difficulties. Among others, these challenges include: the lack of appropriate interaction abstractions, the shortcomings of current user interface software tools to address continuous and parallel interactions, as well as the excessive effort required to integrate novel input and output technologies. To address these challenges, we propose a specification paradigm for designing and implementing Tangible User Interfaces (TUIs), that enables TUI developers to specify the structure and behavior of a tangible user interface using high-level constructs which abstract away implementation details. An important benefit of this approach, which is based on User Interface Description Language (UIDL) research, is that these specifications could be automatically or semi-automatically converted into concrete TUI implementations. In addition, such specifications could serve as a common ground for investigating both design and implementation concerns by TUI developers from different disciplines. Thus, the primary contribution of this article is a high-level UIDL that provides developers from different disciplines means for effectively specifying, discussing, and programming a broad range of tangible user interfaces. There are three distinct elements to this contribution: a visual specification technique that is based on Statecharts and Petri nets, an XML-compliant language that extends this visual specification technique, as well as a proof-of-concept prototype of a Tangible User Interface Management System (TUIMS) that semi-automatically translates high-level specifications into a program controlling specific target technologies.

introduction Share on Introduction to the special issue on UIDL for next-generation user interfaces Editors: Orit Shaer Wellesley College Wellesley CollegeView Profile , Robert J. K. Jacob Tufts University Tufts UniversityView Profile , Mark Green University of Ontario Institute of Technology University of Ontario Institute of TechnologyView Profile , Kris Luyten Hasselt University Hasselt UniversityView Profile Authors Info & Claims ACM Transactions on Computer-Human InteractionVolume 16Issue 4Article No.: 16pp 1–3https://doi.org/10.1145/1614390.1614391Published:27 November 2009Publication History 8citation870DownloadsMetricsTotal Citations8Total Downloads870Last 12 Months3Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

In this paper we describe the design and initial evaluation of a tangible computer programming exhibit for children on display at the Boston Museum of Science. We also discuss five design considerations for tangible interfaces in science museums that guided our development and evaluation. In doing so, we propose the notion of passive tangible interfaces. Passive tangibles serve as a way to address practical issues involving tangible interaction in public settings and as a design strategy to promote reflective thinking. Results from our evaluation indicate that passive tangibles can preserve many of the benefits of tangible interaction for informal science learning while remaining cost-effective and reliable.

We are in the midst of an explosion of emerging human-computer interaction techniques that redefine our understanding of both computers and interaction. We propose the notion of Reality-Based Interaction (RBI) as a unifying concept that ties together a large subset of these emerging interaction styles. Based on this concept of RBI, we provide a framework that can be used to understand, compare, and relate current paths of recent HCI research as well as to analyze specific interaction designs. We believe that viewing interaction through the lens of RBI provides insights for design and uncovers gaps or opportunities for future research.

We created a tangible user interface that allows children to create musical compositions through constructive play. Our Marble Track Audio Manipulator (MTAM) is an augmented marble tower construction kit where marbles represent sound clips and tracks represent different sound effects. To create musical compositions, children collaboratively build a marble tower and then play their compositions by dropping marbles into the tower. As marbles roll through the tower children can interact with the marbles and thus improvise and alter their musical compositions. By augmenting a popular toy, physically representing sound clips and effects as well as allowing improvisation, the MTAM system provides children with a creative, playful, and engaging encounter with music.

introduction Introduction to special issue on the aesthetics of interaction Editors: Marianne Graves Petersen University of Aarhus University of AarhusView Profile , Lars Hallnäs The Swedish School of Textiles, University of Borås, Chalmers University of Technology The Swedish School of Textiles, University of Borås, Chalmers University of TechnologyView Profile , Robert J.K. Jacob Tufts University Tufts UniversityView Profile Authors Info & Claims ACM Transactions on Computer-Human InteractionVolume 15Issue 3November 2008 Article No.: 10pp 1–5https://doi.org/10.1145/1453152.1453153Published:11 December 2008Publication History 13citation762DownloadsMetricsTotal Citations13Total Downloads762Last 12 Months15Last 6 weeks2 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

introduction Share on Introduction to special issue on the aesthetics of interaction Editors: Marianne Graves Petersen University of Aarhus University of AarhusView Profile , Lars Hallnäs The Swedish School of Textiles, University of Borås, Chalmers University of Technology The Swedish School of Textiles, University of Borås, Chalmers University of TechnologyView Profile , Robert J.K. Jacob Tufts University Tufts UniversityView Profile Authors Info & Claims ACM Transactions on Computer-Human InteractionVolume 15Issue 4Article No.: 14pp 1–5https://doi.org/10.1145/1460355.1460356Published:11 December 2008Publication History 14citation988DownloadsMetricsTotal Citations14Total Downloads988Last 12 Months23Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

We created Smart Blocks, an augmented mathematical manipulative that allows users to explore the concepts of volume and surface area of 3-dimensional (3D) objects. This interface supports physical manipulation for exploring spatial relationships and it provides continuous feedback for reinforcing learning. By leveraging the benefits of physicality with the advantages of digital information, this tangible interface provides an engaging environment for learning about surface area and volume of 3D objects.

This paper describes a new technique for implementing educational programming languages using tangible interface technology. It emphasizes the use of inexpensive and durable parts with no embedded electronics or power supplies. Students create programs in offline settings—on their desks or on the floor—and use a portable scanning station to compile their code. We argue that languages created with this approach offer an appealing and practical alternative to text-based and visual languages for classroom use. In this paper we discuss the motivations for our project and describe the design and implementation of two tangible programming languages. We also describe an initial case study with children and outline future research goals. Author Keywords Tangible UIs, education, children, programming languages

In this paper we introduce the Tangible Video Editor (TVE), a multi-user, tangible interface for sequencing digital video. We present a new approach to tabletop interaction by using multiple handheld computers embedded in plastic tokens. Drawing from the rich physical experience of tradition film editing techniques we designed the TVE to engage multiple users in a collaborative process and encourage the exploration of narrative ideas. We used active tokens to provide a malleable interface, enabling users to organize the interface components in unspecified ways. Our implementation improves upon common projection-based tabletop interfaces in a number of ways including a design for use beyond dedicated two dimensional spaces and a naturally scaling screen resolution.

Reality-based interfaces (RBIs) offer the promise of natural interfaces that are intuitive to use. By allowing users to leverage their innate skills and knowledge about the real-physical world, RBIs could benefit disadvantage populations that rely on physical I/O modalities and tangible representations. We present our research which is aimed at simplifying the development of RBIs by providing a technology-independent visual language for modeling and programming these interfaces. Such a language will enable developers to systematically analyze and compare interaction designs as part of the development process while addressing issues such as users' skills and needs. Ultimately, we expect this research to lower the barriers for developing RBIs

We identify and present a major interaction approach for tangible user interfaces based upon systems of tokens and constraints. In these interfaces, tokens are discrete physical objects which represent digital information. Constraints are confining regions that are mapped to digital operations. These are frequently embodied as structures that mechanically channel how tokens can be manipulated, often limiting their movement to a single degree of freedom. Placing and manipulating tokens within systems of constraints can be used to invoke and control a variety of computational interpretations.We discuss the properties of the token+constraint approach; consider strengths that distinguish them from other interface approaches; and illustrate the concept with eleven past and recent supporting systems. We present some of the conceptual background supporting these interfaces, and consider them in terms of Bellotti et al.'s [2002] five questions for sensing-based interaction. We believe this discussion supports token+constraint systems as a powerful and promising approach for sensing-based interaction.

The task of organizing information is typically performed either by physically manipulating note cards or sticky notes or by arranging icons on a computer with a graphical user interface. We present a new tangible interface platform for manipulating discrete pieces of abstract information, which attempts to combine the benefits of each of these two alternatives into a single system. We developed interaction techniques and an example application for organizing conference papers. We assessed the effectiveness of our system by experimentally comparing it to both graphical and paper interfaces. The results suggest that our tangible interface can provide a more effective means of organizing, grouping, and manipulating data than either physical operations or graphical computer interaction alone

In this paper, we introduce DLoVe, a new paradigm for designing and implementing distributed and nondistributed virtual reality applications, using one-way constraints. DLoVe allows programs written in its framework to be executed on multiple computers for improved performance. It also allows easy specification and implementation of multi-user interfaces. DLoVe hides all the networking aspects of message passing among the machines in the distributed environment and performs the needed network optimizations. As a result, a user of DLoVe does not need to understand parallel and distributed programming to use the system; he or she needs only be able to use the serial version of the user interface description language. Parallelizing the computation is performed by DLoVe, without modifying the interface description.

Compared to conventional interfaces, Virtual reality (VR) interfaces contain a richer variety and more complex types of objects, behaviors, interactions and communications. Therefore, designers of VR interfaces face significant conceptual and methodological challenges in: a) thinking comprehensively about the overall design of the VR interface; b) decomposing the design task into smaller, conceptually distinct, and easier tasks; and c) communicating the structure of the design to software developers. To help designers to deal with these challenges, we propose a Virtual Reality Interface Design (VRID) Model, and an associated VRID methodology.

Eye gaze interaction can provide a convenient and natural addition to user-computer dialogues. We have previously reported on our interaction techniques using eye gaze [10]. While our techniques seemed useful in demonstration, we now investigate their strengths and weaknesses in a controlled setting. In this paper, we present two experiments that compare an interaction technique we developed for object selection based on a where a person is looking with the most commonly used selection method using a mouse. We find that our eye gaze interaction technique is faster than selection with a mouse. The results show that our algorithm, which makes use of knowledge about how the eyes behave, preserves the natural quickness of the eye. Eye gaze interaction is a reasonable addition to computer interaction and is convenient in situations where it is important to use the hands for other tasks. It is particularly beneficial for the larger screen workspaces and virtual environments of the future, and it will become increasingly practical as eye tracker technology matures.

Eye movement-based interaction offers the potential of easy, natural, and fast ways of interacting in virtual environments. However, there is little empirical evidence about the advantages or disadvantages of this approach. We developed a new interaction technique for eye movement interaction in a virtual environment and compared it to more conventional 3-D pointing. We conducted an experiment to compare performance of the two interaction types and to assess their impacts on spatial memory of subjects and to explore subjects' satisfaction with the two types of interactions. We found that the eye movement-based interaction was faster than pointing, especially for distant objects. However, subjects' ability to recall spatial information was weaker in the eye condition than the pointing one. Subjects reported equal satisfaction with both types of interactions, despite the technology limitations of current eye tracking equipment.

We present a software model and language for describing and programming the fine-grained aspects of interaction in a non-WIMP user interface, such as a virtual environment. Our approach is based on our view that the essence of a non-WIMP dialogue is a set of continuous relationships—most of which are temporary. The model combines a data-flow or constraint-like component for the continuous relationships with an event-based component for discrete interactions, which can enable or diable individual continuous relationships. To demonstrate our approach, we present the PMIW user interface management system for non-WIMP interactions, a set of examples running under it, a visual editor for our user interface description language, and a discussion of our implemantation and our restricted use of constraints for a performance-driven interactive situation. Our goal is to provide a model and language that captures the formal structure of non-WIMP interactions in the way that various previous techniques have captured command-based, textual, and event-based styles and to suggest that using it need and not compromise real-time performance.

The conference this year is the tenth anniversary of UIST.The keynote talk discusses the history of UIST over the last ten years; this panel looks into the future of the field over the next ten.Each of the panelists will describe a scenario for what life will be like when we meet for UIST'O'I, ten years from now.They will also have a chance to challenge or question each others' scenarios and to participate in open discussion with the audience.

Pre-screen projection is a new interaction technique that allows a user to pan and zoom integrally through a scene simply by moving his or her head relative to the screen. The underlying concept is based on real-world visual perception, namely, the fact that a person’s view changes as the head moves. Pre-screen projection tracks a user’s head in three dimensions and alters the display on the screen relative to head position, giving a natural perspective effect in response to a user’s head movements. Specifically, projection of a virtual scene is calculated as if the scene were in front of the screen. As a result, the visible scene displayed on the physical screen expands (zooms) dramatically as a user moves nearer. This is analogous to the real world, where the nearer an object is, the more rapidly it visually expands as a person moves toward it. Further, with pre-screen projection a user can navigate (pan and zoom) around a scene integrally, as one unified activity, rather than performing panning and zooming as separate tasks. This paper describes the technique, the real-world metaphor on which it is conceptually based, issues involved in iterative development of the technique, and our approach to its empirical evaluation in a realistic application testbed.

Current input device taxonomies and other frameworks typically emphasize the mechanical structure of input devices. We suggest that selecting an appropriate input device for an interactive task requires looking beyond the physical structure of devices to the deeper perceptual structure of the task, the device, and the interrelationship between the perceptual structure of the task and the control properties of the device. We affirm that perception is key to understanding performance of multidimensional input devices on multidimensional tasks. We have therefore extended the theory of processing of percetual structure to graphical interactive tasks and to the control structure of input devices. This allows us to predict task and device combinations that lead to better performance and hypothesize that performance is improved when the perceptual structure of the task matches the control structure of the device. We conducted an experiment in which subjects performed two tasks with different perceptual structures, using two input devices with correspondingly different control structures, a three-dimensional tracker and a mouse. We analyzed both speed and accuracy, as well as the trajectories generated by subjects as they used the unconstrained three-dimensional tracker to perform each task. The result support our hypothesis and confirm the importance of matching the perceptual structure of the task and the control structure of the input device.

Plus que les performances d'un systeme technologique, ce sont les possibilites de communication et leurs resultats qui peuvent etre ameliores. L'auteur developpe ici des mecanismes d'entree ou de sortie des donnees qui permettraient une meilleure communication ordinateur-utilisateur.

Concepts such as the logical device, taxonomies, and other descriptive frameworks have improved understanding of input devices but ignored or else treated informally their pragmatic qualities, which are fundamental to selection of input devices for tasks. We seek the greater leverage of a predictive theoretical framework by basing our investigation of three-dimensional vs. two-dimensional input devices on Garner's theory of processing of perceptual structure in multidimensional tasks. Two three-dimensional tasks may seem equivalent, but if they involve different types of perceptual spaces, they should be assigned correspondingly different input devices. Our experiment supports this hypothesis and thus both indicates when to use three-dimensional input devices and gives credence to our theoretical basis for this indication.

In seeking hitherto-unused methods by which users and computers can comrnumcate, we investigate the usefulness of eye movements as a fast and convenient auxiliary user-to-computer H. 5,2 [Information Interfaces and presentation]:

In seeking hitherto-unused methods by which users and computers can communicate, we investigate the usefulness of eye movements as a fast and convenient auxiliary user-to-computer communication mode. The barrier to exploiting this medium has not been eye-tracking technology but the study of interaction techniques that incorporate eye movements into the user-computer dialogue in a natural and unobtrusive way. This paper discusses some of the human factors and technical considerations that arise in trying to use eye movements as an input medium, describes our approach and the first eye movement-based interaction techniques that we have devised and implemented in our laboratory, and reports our experiences and observations on them.

Current expert systems are typically difficult to change once they are built. The authors introduce a method for developing more easily maintainable rule-based expert systems, which is based on dividing the rules into groups and focusing attention on those facts that carry information between rules in different groups. They describe a new algorithm for grouping the rules of a knowledge base automatically and a notation set of software tools for the proposed method. The approach is supported by a study of the connectivity of rules and facts in rule-based systems; it is found that they indeed have the latent structure necessary for the programming methodology. Recent experimental results also support the approach. In contrast to the homogeneous way in which the facts of a rule-based system are usually viewed, this approach shows that certain facts are more important than others with regard to future modifications of the rules.>

A direct-manipulation user interface presents a set of visual representations on a display and a repertoire of manipulations that can be performed on any of them. Such representations might include screen buttons, scroll bars, spreadsheet cells, or flowchart boxes. Interaction techniques of this kind were first seen in interactive graphics systems; they are now proving effective in user interfaces for applications that are not inherently graphical. Although they are often easy to learn and use, these interfaces are also typically difficult to specify and program clearly. Examination of direct-manipulation interfaces reveals that they have a coroutine-like structure and, despite their surface appearance, a peculiar, highly moded dialogue. This paper introduces a specification technique for direct-manipulation interfaces based on these observations. In it, each locus of dialogue is described as a separate object with a single-thread state diagram, which can be suspended and resumed, but retains state. The objects are then combined to define the overall user interface as a set of coroutines, rather than inappropriately as a single highly regular state transition diagram. An inheritance mechanism for the interaction objects is provided to avoid repetitiveness in the specifications. A prototype implementation of a user-interface management system based on this approach is described, and example specifications are given.

It is useful to be able to specify a proposed human-computer interface formally before building it, particularly if a mockup suitable for testing can be obtained directly from the specification. A specification technique for user interfaces, based on state transition diagrams, is introduced and then demonstrated for a secure message system application. An interpreter that executes the resulting specification is then described. Some problems that arise in specifying a user interface are addressed by particular features of the technique: To reduce the complexity of the developer's task, a user interface is divided into the semantic, syntactic, and lexical levels, and a separate executable specification is provided for each. A process of stepwise refinement of the syntactic specification, leading from an informal specification to an executable one is also presented. Since the state diagram notation is based on a non-deterministic model, constraints necessary to realize the system with a deterministic interpreter are given.

This paper surveys specification techniques that can be applied to human-computer interfaces, provides examples of specifications, and presents some conclusions drawn from the author's experience using the techniques for specifying the user interface of the message system.