We investigated the effects of example-based explanations for a machine learning classifier on end users' appropriate trust. We explored the effects of spatial layout and visual representation in an in-person user study with 33 participants. We measured participants' appropriate trust in the classifier, quantified the effects of different spatial layouts and visual representations, and observed changes in users' trust over time. The results show that each explanation improved users' trust in the classifier, and the combination of explanation, human, and classification algorithm yielded much better decisions than the human and classification algorithm separately. Yet these visual explanations lead to different levels of trust and may cause inappropriate trust if an explanation is difficult to understand. Visual representation and performance feedback strongly affect users' trust, and spatial layout shows a moderate effect. Our results do not support that individual differences (e.g., propensity to trust) affect users' trust in the classifier. This work advances the state-of-the-art in trust-able machine learning and informs the design and appropriate use of automated systems.

The power grid is a complex, interconnected system that requires the coordination of multiple entities to ensure the reliability of the system. System operators must be aware of the impacts of various natural and man-made contingencies through telemetry from their control systems, and they must also be aware of the integrity and availability of the cyber assets on their systems. A team of researchers from Pacific Northwest National Laboratory and Idaho National Laboratory have conducted a series of observations and interviews at power utilities to identify what information needs to be available for both control room dispatchers and cyber security analysts to maintain situation awareness of cyber security to effectively monitor the cyber assets on utility control systems. The researchers interviewed and observed control room dispatchers and spoke with regional control system engineers and cyber security analysts to define the context in which such information would be used. The results of this work will inform the design of a tool that will enhance cyber situation awareness and help facilitate effective communication between the various personnel and organizations with utility and other regional authorities during contingencies.

Abstract Evaluating interactive question answering (QA) systems with real users can be challenging because traditional evaluation measures based on the relevance of items returned are difficult to employ since relevance judgments can be unstable in multi-user evaluations. The work reported in this paper evaluates, in distinguishing among a set of interactive QA systems, the effectiveness of three questionnaires: a Cognitive Workload Questionnaire (NASA TLX), and Task and System Questionnaires customized to a specific interactive QA application. These Questionnaires were evaluated with four systems, seven analysts, and eight scenarios during a 2-week workshop. Overall, results demonstrate that all three Questionnaires are effective at distinguishing among systems, with the Task Questionnaire being the most sensitive. Results also provide initial support for the validity and reliability of the Questionnaires.

A formal interaction modeling technique known as Goals, Operators, Methods, and Selection rules (GOMS) is well-established in human-computer interaction as a cost-effective way of evaluating designs without the participation of end users. This paper explores the use of GOMS for evaluating human-robot interaction. We provide a case study in the urban search-and-rescue domain and raise issues for developing GOMS models that have not been previously addressed. Further, we provide rationale for selecting different types of GOMS modeling techniques to help the analyst model human-robot interfaces.

This paper discusses the development of a test bed to evaluate the combined performance of the human operator and an explosive ordnance disposal robot. We have other means of evaluating the capabilities of the robots but for the robots to be truly useful it is necessary to understand how effectively and efficiently operators will be able to use these robots in critical situations. In this paper we discuss the tasks developed for the test bed and how we are going about development of the metrics for assessing the human-robot performance and, more specifically, the human-robot user interface.

This paper describes an effort to identify common metrics for task-oriented human-robot interaction (HRI). We begin by discussing the need for a toolkit of HRI metrics. We then describe the framework of our work and identify important biasing factors that must be taken into consideration. Finally, we present suggested common metrics for standardization and a case study. Preparation of a larger, more detailed toolkit is in progress.

This tutorial presents the current status of research in interactions with robots, including adaptive robots/interfaces, speech, gestures, virtual reality, and social interactions. Different user interface designs will be shown and discussed during the tutorial. Human-robot interaction (HRI) guidelines, evaluation methodologies and metrics currently used by the community will be presented. Research needs will also be discussed. Participants will work in small groups to design a robotic application as well as an evaluation plan.

The Peer-To-Peer Human-Robot Interaction (P2P-HRI) project is developing techniques to improve task coordination and collaboration between human and robot partners. Our work is motivated by the need to develop effective human-robot teams for space mission operations. A central element of our approach is creating dialogue and interaction tools that enable humans and robots to flexibly support one another. In order to understand how this approach can influence task performance, we recently conducted a series of tests simulating a lunar construction task with a human-robot team. In this paper, we describe the tests performed, discuss our initial results, and analyze the effect of intervention on task performance.

Performance assessment and enhancement strategies have often focused upon the user or the intelligent technology rather than the interaction between the two. Where system interaction has occurred, we have often settled for co-operation rather than collaboration or symbiosis. This special issue is devoted to the topic of the symbiotic relationship that people have with intelligent systems, and nontraditional computer support for design and user evaluation of this relationship. An objective in coordinating this issue was to encourage a range of papers that addressed the nature of the relationship between human and intelligent agents, the ways in which performance could be augmented, measured and improved, and the need for additional and nontraditional measures of performance for evaluating the anticipated progress as people interact with intelligent computer systems. The potential benefit of such measurement tools and techniques has been recognized in numerous domains including education, information visualization, digital libraries, and computer supported cooperative work. The summaries from several conferences and workshops on human-centered systems contain many references to this subject. However, many questions still remain on how and what to measure for interactive symbiotic systems. Currently, if systems are tested, designers rely on users to test their systems. While this constitutes an acceptable metric it also raises many issues including comparison of skills of users, standardization of test situations, amount of training given, help given, and time allotted for the evaluation. In addition, the influence of the interactions on the users’ performance is rarely studied.

Society today has a wealth of information available due to information technology. The challenge facing researchers working in information access is how to help users easily locate the information needed. Evaluation methodologies and metrics are important tools to assess progress in human information interaction (HII). To properly evaluate these systems, evaluations need to consider the performance of the various components, the usability of the system, and the impact of the system on the end user. Current usability metrics are adequate for evaluating the efficiency, effectiveness, and user satisfaction of such systems. Performance measures for new intelligent technologies will have to be developed. Regardless of how well the systems are and how usable the systems are, it is critical that impact measures are developed. For HII systems to be useful, we need to assess how well information analysts work with the systems. This evaluation needs to go beyond technical performance metrics and usability metrics. What are the metrics for evaluating utility? This paper describes research efforts focused on developing metrics for the intelligence community that measure the impact of new software to facilitate information interaction.

In the past four years, we have worked with several research programs that were developing intelligent software for use by intelligence analysts. Our involvement in these programs was to develop the metrics and methodologies for assessing the impact on users; in this case, on intelligence analysts. In particular, we focused on metrics to evaluate how much the intelligent systems contribute to the users’ tasks and what the cost is to the user in terms of workload and process deviations. In this paper, we describe the approach used. We started with two types of preliminary investigations – first, collecting and analyzing data from analysts working in an instrumented environment for a period of 2 years, and second, developing and conducting formative evaluations of research software. The long-term studies informed our ideas about the processes that analysts use and provided potential metrics in an environment without intelligent software tools. The formative evaluations helped us to define sets of application-specific metrics. Finally, we conducted assessments during and after technology insertions. We describe the metrics and methodologies used in each of these activities, along with the lessons learned.

In this paper, we outline a methodology for evaluating the situation awareness (SA) provided by a supervisory interface for an autonomous on-road vehicle. Our goal is to be able to use the evaluations to compare interface designs with respect to how well each facilitates the users' acquisition of SA. We used Endsely's Situation Awareness Global Assessment Technique (SAGAT) (Endsely, 1988) and developed scenarios and assessment questions appropriate for supervisors of autonomous on-road driving vehicles. We describe the results of two experiments used to refine our SA assessment implementation. In a third experiment, we applied the refined implementation to a graphical user interface we developed to test the sensitivity of our SAGAT implementation. We discuss the results of this experiment and implications for applying the SAGAT methodology to supervisory user interfaces for autonomous vehicles.

This paper introduces an evaluation method that provides the capability of comparing results of like-structured evaluations that occur over time and with changing toolsets or environmental conditions. This makes use of the framework ideal for comparison of collaboration tools. The framework helps to structure evaluations by mapping system goals to evaluation objectives, metrics, and measures. The upper-most levels of the framework are conceptual in nature, while the bottom level is implementation-specific, i.e., evaluation-specific. Careful attention during construction of the conceptual elements for an evaluation template allows for its reuse in a series of like-structured evaluations and comparison of those results.

In this paper we report on the analysis of critical incidents during an urban search and rescue robot competition where critical incidents are defined as a situation where the robot could potentially cause damage to itself, the victim, or the environment. We look at the features present in the human-robot interface that contributed to success in different tasks needed in urban search and rescue and present guidelines for human-robot interaction design.

As part of a Defense Advanced Research Projects Agency/National Science Foundation study on human-robot interaction (HRI), over sixty representatives from academia, government, and industry participated in an interdisciplinary workshop, which allowed roboticists to interact with psychologists, sociologists, cognitive scientists, communication experts and human-computer interaction specialists to discuss common interests in the field of HRI, and to establish a dialogue across the disciplines for future collaborations. We include initial work that was done in preparation for the workshop, links to keynote and other presentations, and a summary of the findings, outcomes, and recommendations that were generated by the participants. Findings of the study include-the need for more extensive interdisciplinary interaction, identification of basic taxonomies and research issues, social informatics, establishment of a small number of common application domains, and field experience for members of the HRI community. An overall conclusion of the workshop was expressed as the following-HRI is a cross-disciplinary area, which poses barriers to meaningful research, synthesis, and technology transfer. The vocabularies, experiences, methodologies, and metrics of the communities are sufficiently different that cross-disciplinary research is unlikely to happen without sustained funding and an infrastructure to establish a new HRI community.

This paper provides a set of definitions that form a framework for describing the types of awareness that humans have of robot activities and the knowledge that robots have of the commands given them by humans. As a case study, we applied this human-robot interaction (HRI) awareness framework to our analysis of the HRI approaches used at an urban search and rescue competition. We determined that most of the critical incidents (e.g., damage done by robots to the test arena) were directly attributable to lack of one or more kinds of HRI awareness.

Various methods can be used for evaluating human-robot interaction. The appropriateness of those evaluation methodologies depends on the roles that people assume in interacting with robots. In this paper we focus on developing an evaluation strategy for the bystander role. In this role, the person has no training in interacting with the robot and must develop a mental model to co-exist in the same environment with the robot.

Universal Usability will be met when affordable, useful, and usable technology accommodates the vast majority of the global population: this entails addressing challenges of technology variety, user diversity, and gaps in user knowledge in ways only beginning to be acknowledged by educational, corporate, and government agencies. (Ben Shneiderman, Communications of the ACM, 2000). Universal usability serves as both an ideal and a research agenda. Users, in their variety of characteristics and contexts, face a corresponding variety of barriers and impediments to the use of interfaces. These variations include age, location, culture, education, range of cognitive and physical abilities, and economic status. Problems with interfaces that many might find trivial can, for others, present significant issues of usability. As more information becomes available (and eventually available only) on-line, as e-government and e-commerce increase in importance, and as our lives become increasingly information-centered, these issues become correspondingly critical. Researchers in the field of human-computer interaction are now addressing this functional aspect of the digital divide.

Article Free Access Share on Design: no job too small Authors: Jean C. Scholtz UserWorks, Inc., 1738 Elton Rd., Suite 138, Silver Spring, MD UserWorks, Inc., 1738 Elton Rd., Suite 138, Silver Spring, MDView Profile , Pete Lockhart Beta Base, Inc., 805 SE 10lst Avenue, Vancouver, WA Beta Base, Inc., 805 SE 10lst Avenue, Vancouver, WAView Profile , Tony Salvador Intel Corporation, JF3-210, 5200 NE Elam Young Parkway, HilIsboro, OR Intel Corporation, JF3-210, 5200 NE Elam Young Parkway, HilIsboro, ORView Profile , James Newbery University of Nottingham, Department of Psychology, Nottingham, UK NG7 1NG University of Nottingham, Department of Psychology, Nottingham, UK NG7 1NGView Profile Authors Info & Claims CHI '97: Proceedings of the ACM SIGCHI Conference on Human factors in computing systemsMarch 1997 Pages 447–454https://doi.org/10.1145/258549.258837Published:27 March 1997Publication History 1citation490DownloadsMetricsTotal Citations1Total Downloads490Last 12 Months4Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my Alerts New Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

The paper begins by describing a model of the influences operating when an experienced programmer learns to program in an unfamiliar language. It then reports on an empirical study which investigated how a change of programming language affects experienced programmers. Programmers solved a familiar problem using two unfamiliar languages. It was found that programmers used knowledge from past experience in solving the problem, but often had to adapt the knowledge to take good advantages of the new languages. They were only partly successful in doing this, and overall performance was poor. The solution process of the programmers was disrupted, as shown by more plan changes and more extensive backward planning. The two unfamiliar languages had impacts at different levels in the planning process.

Learning second and subsequent programming languages is easier than learning a first programming language because many concepts and constructs are shared. However, it is still a hard task. In this protocol analysis of moderately experienced programmers transferring to a new programming language, we classified episodes by whether they involved the syntactic, semantic, or planning level of programming knowledge. We discovered that most episodes involve planning and that in solving a given subproblem there are typically many cycles of language‐independent tactical planning followed by language‐dependent implementation planning. On the other hand, programmers have relatively minor problems with the syntax and semantics of a new language. Our subjects’ protocols and their final programs revealed that the plans they develop are strongly influenced by their knowledge of what would be convenient and appropriate in other languages they know. This prevents them from taking full advantage of the capabilities of the new language.