The drive for increasing adoption of HRI technolo-gies is evident through research and development of manufac-turing, social, medical, and service robot solutions. However, novel methods and metrics are required to overcome the barrier between fundamental HRI research and its adoption in real-world environments. Hence, the fourth installment of the annual workshop, 'Test Methods and Metrics for Effective HRI,’ seeks to identify novel and emerging test methods and metrics for the holistic assessment and assurance of HRI performance. Specifically, the focus is on identifying innovative methods for the evaluation of HRI performance and to advance the growth of the HRI community based on the principles of collaboration, data sharing, and repeatability. The goal of this workshop is to break the boundaries between the development and adoption of HRI technologies through the promotion of robust experimental design, test methods, and metrics for assessing interaction and interface designs. This workshop will have participants from var-ious sectors in the HRI research community including academia, industry, and government in order to accomplish its aims.

This special issue of the Transactions on Human-Robot Interaction highlights, documents, and explores the metrics, test methods, and artifacts used in human-robot interaction (HRI) research. This collection of articles brings to attention the commonalities between the application of measurement science for the assessment and assurance of human-centric robotics in a variety of application domains, including industry, education, and defense. This special issue draws specific attention to the use and impact of metrology toward the advancement of HRI technologies and algorithms, and it promotes the application of measurement science toward the benchmarking and replication of HRI research. Special attention is given to the use cases, data sets, test methodologies, measurement techniques, metrics, and statistical analyses used to evaluate system performance.

We present a comprehensive framework and test methodology for the evaluation of human-machine interfaces (HMI) and human-robot interactions (HRI) in collaborative manufacturing applications. An overview of the challenges that face current- and next-generation collaborative robot systems is presented, specifically focused on the interactions between man and machine, and a series of objectively quantitative and subjectively qualitative metrics are given to guide the development and assessment of interfaces and interactions. A generalized set of guidelines for the design of HMI is also proposed to address these challenges and thereby enable effective and intuitive diagnostics and error corrections when process failures occur. These guidelines are aimed at aiding researchers in developing effective interface and interaction technologies, maximizing operator situation awareness in human-robot collaborative manufacturing teams, promoting effective process and system diagnostics reporting, and enabling faster responses to equipment or application errors.

Verified and validated test methods, being necessary to measure the performance of complex systems, are important tools for driving innovation, benchmarking and improving performance, and establishing trust in collaborative human-robot teams. This full-day workshop aims to explore the metrology necessary for repeatably and independently assessing the collaborative performance of robotic systems in real-world human-robot interaction (HRI) scenarios. This workshop aims to bridge the gaps between the theory and applications of HRI in industry, accelerating the adoption of cutting edge technologies as the industry state-of-practice. The interest in collaborative HRI is evident in the current market as well as standards efforts toward manufacturing, social, medical, and service robot solutions. Though these domains have been considered separate for many years, recent technological and scientific advancements show that, while their applications may differ, the underlying principles of HRI performance impact each identically. As such, this workshop seeks to identify test methods and metrics for the holistic assessment and assurance of collaborative HRI performance. The focus is on identifying the key performance indicators of these seemingly disparate sectors, and additionally to establish a community based on the principles of transparency, repeatability, & establishing trust in the assessment of collaborative HRI. The goal is to aid in the advancement of HRI technologies through the development of experimental scenarios, protocols, test methods, & metrics for the verification and validation of interaction solutions and interface designs.

This paper discusses the development of a test bed to evaluate the combined performance of the human operator and an explosive ordnance disposal robot. We have other means of evaluating the capabilities of the robots but for the robots to be truly useful it is necessary to understand how effectively and efficiently operators will be able to use these robots in critical situations. In this paper we discuss the tasks developed for the test bed and how we are going about development of the metrics for assessing the human-robot performance and, more specifically, the human-robot user interface.

The Peer-To-Peer Human-Robot Interaction (P2P-HRI) project is developing techniques to improve task coordination and collaboration between human and robot partners. Our work is motivated by the need to develop effective human-robot teams for space mission operations. A central element of our approach is creating dialogue and interaction tools that enable humans and robots to flexibly support one another. In order to understand how this approach can influence task performance, we recently conducted a series of tests simulating a lunar construction task with a human-robot team. In this paper, we describe the tests performed, discuss our initial results, and analyze the effect of intervention on task performance.

In this paper, we outline a methodology for evaluating the situation awareness (SA) provided by a supervisory interface for an autonomous on-road vehicle. Our goal is to be able to use the evaluations to compare interface designs with respect to how well each facilitates the users' acquisition of SA. We used Endsely's Situation Awareness Global Assessment Technique (SAGAT) (Endsely, 1988) and developed scenarios and assessment questions appropriate for supervisors of autonomous on-road driving vehicles. We describe the results of two experiments used to refine our SA assessment implementation. In a third experiment, we applied the refined implementation to a graphical user interface we developed to test the sensitivity of our SAGAT implementation. We discuss the results of this experiment and implications for applying the SAGAT methodology to supervisory user interfaces for autonomous vehicles.