With robotics rapidly advancing, more effective human–robot interaction is increasingly needed to realize the full potential of robots for society. While spoken language must be part of the solution, our ability to provide spoken language interaction capabilities is still very limited. In this article, based on the report of an interdisciplinary workshop convened by the National Science Foundation, we identify key scientific and engineering advances needed to enable effective spoken language interaction with robotics. We make 25 recommendations, involving eight general themes: putting human needs first, better modeling the social and interactive aspects of language, improving robustness, creating new methods for rapid adaptation, better integrating speech and language with other communication modalities, giving speech and language components access to rich representations of the robot’s current knowledge and state, making all components operate in real time, and improving research infrastructure and resources. Research and development that prioritizes these topics will, we believe, provide a solid foundation for the creation of speech-capable robots that are easy and effective for humans to work with.

This article describes an emerging direction in the intersection between human–computer interaction and cognitive science: the use of cognitive models to give insight into the challenges of cybersecurity (cyber-SA). The article gives a brief overview of work in different areas of cyber-SA where cognitive modeling research plays a role, with regard to direct interaction between end users and computer systems and with regard to the needs of security analysts working behind the scenes. The problem of distinguishing between human users and automated agents (bots) interacting with computer systems is introduced, as well as ongoing efforts toward building Human Subtlety Proofs (HSPs), persistent and unobtrusive windows into human cognition with direct application to cyber-SA. Two computer games are described, proxies to illustrate different ways in which cognitive modeling can potentially contribute to the development of HSPs and similar cyber-SA applications.

This note describes two studies of the use of a performance modeling tool, CogTool, for making recommendations to improve a user interface. The first study replicates findings by Bonnie John [7]: the rates at which novice modelers made correct recommendations (88.1%) and supported them (68.2%) are close to the values in John's study (91.7% and 75.1%, respectively). A follow-on study of novice modelers on the same task without CogTool produced sig-nificantly lower values. CogTool improves the UI design recommendations made by novices.

CAVIAR is designed to aid people with vision impairment in locating, identifying, and acquiring objects in their peripersonal space. A mobile phone, worn on the chest, captures video in front of the user; the computer vision component locates the user's hand and objects in the video stream. The auditory component informs the user about the presence of objects. On user confirmation, the reaching component sends signals to vibrotactile actuators on the user's wristband, guiding the hand to a specific object. This paper describes an end-to-end prototype of CAVIAR and its formative evaluation.

This paper describes a glove with which users enter input by tapping fingertips with the thumb or by rubbing the thumb over the palmar surfaces of the middle and index fingers. The glove has been informally tested as the controller for two semi-autonomous robots in a a 3D simulation environment. A preliminary evaluation of the glove's performance is presented.

In this paper we analyze pointing techniques for simple remote control of nearby and distant objects in an outdoor environment, using a mobile phone. In an experiment we determine the accuracy of pointing at targets from a few meters to a few hundred meters away, either by focusing the phone's camera on a target or holding the phone at waist level in the direction of the target. We describe a simulated network application in which users can activate and control one or more responsive objects using either interaction technique.

This paper describes the integration of perceptual guidelines from human vision with an AI-based mixed-initiative search strategy. The result is a visualization assistant called ViA, a system that collaborates with its users to identify perceptually salient visualizations for large, multidimensional datasets. ViA applies knowledge of low-level human vision to: (1) evaluate the effectiveness of a particular visualization for a given dataset and analysis tasks; and (2) rapidly direct its search towards new visualizations that are most likely to offer improvements over those seen to date. Context, domain expertise, and a high-level understanding of a dataset are critical to identifying effective visualizations. We apply a mixed-initiative strategy that allows ViA and its users to share their different strengths and continually improve ViA's understanding of a user's preferences. We visualize historical weather conditions to compare ViA's search strategy to exhaustive analysis, simulated annealing, and reactive tabu search, and to measure the improvement provided by mixed-initiative interaction. We also visualize intelligent agents competing in a simulated online auction to evaluate ViA's perceptual guidelines. Results from each study are positive, suggesting that ViA can construct high-quality visualizations for a range of real-world datasets.

The longer a fault remains in the code from the time it was injected, the more time it will take to fix the fault. Increasingly, automated fault detection (AFD) tools are providing developers with prompt feedback on recently-introduced faults to reduce fault fix time. If however, the frequency and content of this feedback does not match the developer's goals and/or workflow, the developer may ignore the information. We conducted a controlled study with 18 developers to explore what factors are used by developers to decide whether or not to address a fault when notified of the error. The findings of our study lead to several conjectures about the design of AFD tools to effectively notify developers of faults in the coding phase. The AFD tools should present fault information that is relevant to the primary programming task with accurate and precise descriptions. The fault severity and the specific timing of fault notification should be customizable. Finally, the AFD tool must be accurate and reliable to build trust with the developer.

We describe concepts to support the analysis of cell phone menu hierarchies, based on cognitive models of users and easy-to-use optimization techniques. We present an empirical study of user performance on five simple tasks of menu traversal on an example cell phone. Two of the models applied to these tasks, based on GOMS and ACT-R, give good predictions of behavior. We use the empirically supported models to create an effective evaluation and improvement process for menu hierarchies. Our work makes three main contributions: a novel and timely study of a new, very common HCI task; new versions of existing models for accurately predicting performance; and a search procedure to generate menu hierarchies that reduce traversal time, in simulation studies, by about a third.

Models of users are a way to understand and improve the usability of computer interfaces. We present here a model in ACT-R cognitive-modeling language that interacts with a publicly available driving simulation as a simple analog for robot interfaces. The model interacts with the unmodified Java interface by incorporating a novel use of bitmap parsing. The model's structure starts to describe the knowledge a human operator of a robot must have. The model also indicates some of the aspects of the task will be difficult for the operator. For example, the model's performance makes quantitative predictions about how robot speed will influence navigation quality, correlating well to human performance. While the model does not cover all aspects of human-robot interaction, it illustrates how providing user models access to an interface through its bitmap can lead to more accurate and more widely applicable model users.

Cell phone interfaces are now ubiquitous. In this paper, we describe concepts to support the analysis of cell phone menu hierarchies. We present an empirical study of user performance on five simple tasks of menu traversal on a cell phone. Two models we tested, based on GOMS and ACT-R, give very good predictions of behavior. We use the study results to motivate an effective evaluation process for menu hierarchies. Our work makes several contributions: a novel and timely study of a new, very common HCI task; new models for accurately predicting performance; novel development tools to support such modeling; and a search procedure to generate menu hierarchies that reduce traversal time, in simulation studies, by about a third.

This paper describes a novel approach to human-robot interaction, in which a user modifies a robot's environment to constrain its actions, rather than programming its controller. An HRI simulation of a maze navigation task is presented. An empirical evaluation shows that for this task, users prefer an environment modification strategy rather than a programming strategy as the difficulty of the task increases. Further, user alternation between the two types of strategy follows a clear pattern. A preliminary model extending the HRI simulation, one which allows the specification of more general navigation environments, is also presented

Making an interface more efficient, in a task analysis sense, can make it more difficult for an automated reasoning system to infer user goals, by eliminating some user actions, by presenting information without requiring overt user selection, and so forth. We call the extent to which a system can make such inferences interpretability. In this paper we describe the tradeoff between interpretability and efficiency. We give some general heuristics for improving interpretability in a system and explain how they apply in an implemented system, an assistant for exploratory statistical analysis. Increased interpretability in the system is provided by navigation techniques for data exploration and a data mountain for organizing results; a formative evaluation illustrates some of the potential benefits of applying interpretability heuristics to an intelligent user interface

This paper describes a drawing system that incorporates two novel interaction techniques based on analogies to physical tools. Power tools add limited autonomy in the form of rotators and movers for automated circular, linear, and Bezier-curve movement. Composite tools are user-constructed combinations of existing tools to create new functionality.

This paper describes a design tool, under development, that identifies efficient low-level action sequences in the user interface, accounting for the relationships between the physical properties of an input device and the requirements of primitive tasks. A device representation and a task taxonomy are presented. These, along with a library of efficiency measures for specific devices, provide information to steer search through the space of action sequences.

This paper describes a system for exploring and selecting entries from a music database through a visualization interface. The system is designed for deployment in situations in which the user's attention is a tightly limited resource. The system combines research topics in intelligent user interfaces, visualization techniques, and cognitive modeling. Informal evaluation of the system has given us useful insights into the design tradeoffs that developers may face when building visual interfaces for off-the-desktop applications.

A user interface softbot is a software agent that controls an interactive system through the graphical user interface, relying on visual information from the system rather than an application programming interface or access to source code. Interface softbots have acted as autonomous agents in applications such as drawing and data recording, and the core vision processing algorithms have been incorporated into cognitive models for simple problem-solving tasks. Building interface softbots is still a time-consuming task, unfortunately, requiring experience with complex program components as well as the details of the visual interface. We have developed a prototype development environment that facilitates the development of interface softbots, streamlining the programming process and making it more accessible to new developers.

The ease with which a user interface can be navigated strongly contributes to its usability. In this paper we describe preliminary results of a project aimed at making the evaluation of user interfaces from this perspective more routine. We have designed a system to carry out an autonomous, exploratory navigation through the graphical user interface of interactive, off-the-shelf software applications. The system is not a robust tool, but rather a proof of concept that can exhibit interesting behaviors. The traversal process generates a representation of the connectivity of the user interface, as well as navigational paths to specific commands. The reasoning component of the system is based on the ACT-R architecture, while the perceptual and motor components of the system are built on top of the SegMan perception/action substrate. We present the design of the system and its use in exploring a simple user interface.

This paper describes a set of visualization techniques for interactive planning in a physical force simulation called AFS. We have developed a 3D environment in which textures are overlaid on a simulated landscape to convey information about environmental properties, agent actions, and possible strategies. Scenes are presented, via automated camera planning, such that some simple agent goals can be induced visually with little effort. These two areas of visualization functionality in AFS exploit properties of human low-level and intermediate-level vision, respectively. This paper presents AFS, its visualization environment, and studies we have run to explore the relationship between AFS visualizations and the high-level planning process.

The VisMap system provides for visual manipulation of arbitrary off-the-shelf applications, through an applications graphical user interface. VisMaps API-independent control has advantages for tasks that can benefit from direct access to the functions of the user interface. We describe the design goals and architecture of the system, and we discuss two applications, a user-controlled visual scripting program and an autonomous solitaire-playing program, which together demonstrate some of the capabilities and limitations of the approach.

This paper takes a first step toward formalizing the concept of afhordance in user interfaces.Using a simple example d an AI planning domain, we show how different types of affordance can be described in terms of the costs associated with plan execution.We identify a number of similarities between executing plans and interacting with a graphical user interface, and argue that affordances for planning environments apply equally well to user interface environments.We support our argument with examples of common user interface mechanisms, described in alfordance terms.

This paper describes an ibot, a specialized software agent that exists in the environment of the user interface. Such an agent interacts with applications through the same medium as a human user. Its sensors process screen contents and mouse/keyboard events to monitor the user’s actions and the responses of the environment, while its effecters can generate such events for its own contributions to the interaction. We describe the architecture of our agent and * its algorithms for image processing, event management, and state representation. We illustrate the use of the agent with a small feasibility study in the area of software logging; results are promising for future progress.

This article shows how the concept of affordance in the user interface fits into a well-understood artificial intelligence (AI) model of acting in an environment. In this model AI planning research is used to interpret affordances in terms of the costs associated with the generation and execution of operators in a plan. This article motivates the approach with a brief survey of the affordance literature and its connections to the planning literature and then explores its implications through examples of common user interface mechanisms described in affordances terms. Despite its simplicity, the modeling approach ties together several different threads of practical and theoretical work on affordance into a single conceptual framework.

Modern user interfaces make extensive use of navigation, a metaphor based on wayfinding in a physical space.Navigation can be an effective solution for many problems in understanding and manipulating a complex information space.Our work is in the area of intelligent assistance for decision support environments, where assistance may take the form of automatically exploring decision alternatives, recording justifications for actions, testing global consistency of local results, and generating audit trails, among other activities.This paper describes an experimental evaluation of assisted search in an artificial environment using navigation as a communication medium, under conditions that vary the quality of assistance.

Exploratory data analysis (EDA) plays an increasingly important role in statistical analysis. EDA is difficult, however, even with the help of modern statistical software. We have developed an assistant for data exploration, based on AI planning techniques, that addresses some of the strategic shortcomings of conventional software. This paper illustrates the behavior of the system, gives a high level description of its design, and discusses its experimental evaluation.

Envelopes are a form of decision rule for monitoring plan execution. We describe one type, the DP (dynamic programming) envelope, that draws its decisions from a look-up table computed off-line by dynamic programming. Based on an abstract model of agent progress, DP envelopes let a developer approach execution monitoring as a problem that is independent of issues in agent design. We discuss the application of DP envelopes to a small transportation planning simulation, and discuss the issues that arise in an empirical analysis of the results.

Statistical causal modeling techniques allow us to develop models of program behavior, but these techniques tend to be limited in what they can model: either continuing, repetitive influences or causal influences without cycles, but not both as appear in many environments. The paper describes how two statistical modeling techniques can be combined to suggest and test specific hypotheses about how the environment and the AI planner's design causally influence the planner's behavior. One technique, dependency detection, is designed to identify relationships (dependencies) between particular failures, the methods that repair them and the occurrence of failures downstream. Another method, path analysis, builds causal models of correlational data. Dependency detection operates over a series of events, and path analysis models within a temporal snapshot. We explain the integration of the techniques and demonstrate it on execution data from the Phoenix planner.>