There are a multitude of robotic application domains that touch on the field of human-robot interaction (HRI). From modern manufacturing involving human-robot teams, to personal care robots assisting the elderly, the roles that robots are being tasked with and the nature of interactions with humans are constantly shifting. Even the nature of interaction has changed to incorporate wearable technologies such as exoskeletons to enhance human capabilities, and advanced prosthetics to restore those abilities that have been lost. With this ever-evolving spectrum of HRI, the capacity of measurement science to evaluate, assess, and assure performance and safety struggles to keep up. Building on our previous five-workshop series on Test Methods and Metrics for Effective HRI, NIST presents a new series on evaluative methodologies for accelerating the pipeline from cutting-edge HRI research to state-of-practice. This workshop will address issues regarding 1) data collection and reporting for replicability and system validation, 2) test design and execution for performance verification, and 3) cross-modality artifact design for real-world application-adjacent technology transfer. The goal of this workshop is to accelerate and accommodate accessibility to HRI research results, and address the specific key performance indicators that would establish end-user trust and acceptance of emerging HRI technologies.

The drive for increasing adoption of HRI technolo-gies is evident through research and development of manufac-turing, social, medical, and service robot solutions. However, novel methods and metrics are required to overcome the barrier between fundamental HRI research and its adoption in real-world environments. Hence, the fourth installment of the annual workshop, 'Test Methods and Metrics for Effective HRI,’ seeks to identify novel and emerging test methods and metrics for the holistic assessment and assurance of HRI performance. Specifically, the focus is on identifying innovative methods for the evaluation of HRI performance and to advance the growth of the HRI community based on the principles of collaboration, data sharing, and repeatability. The goal of this workshop is to break the boundaries between the development and adoption of HRI technologies through the promotion of robust experimental design, test methods, and metrics for assessing interaction and interface designs. This workshop will have participants from var-ious sectors in the HRI research community including academia, industry, and government in order to accomplish its aims.

Standardized metrics and methods are critical to-wards wider adoption of HRI technologies in real-world applications. However, the interdisciplinary nature of HRI creates an inherently decentralized research paradigm that limits the use of standardized metrics for baseline comparisons among studies. This limitation restricts both the real-world adoption and academic replicability of HRI solutions developed by the research community. To identify specific opportunities for reuse of metrics and methods in HRI, this paper presents a comprehensive survey of 1464 papers from the ACM/IEEE International Conference on Human-Robot Interaction (HRI) and the IEEE International Conference on Robot and Human Interactive Communication (Ro-Man) over seven years. By providing a holistic perspective of the metrological tools leveraged in the current state-of-practice of HRI research, we find that a significant portion of HRI studies use custom surveys, thus limiting baseline comparison. Hence, the analysis in this work aims to advance the field of HRI by identifying specific barriers to adoption of HRI technologies in addition to proposing solutions to overcome existing limitations in the context of metrics and methodologies.

This special issue of the Transactions on Human-Robot Interaction highlights, documents, and explores the metrics, test methods, and artifacts used in human-robot interaction (HRI) research. This collection of articles brings to attention the commonalities between the application of measurement science for the assessment and assurance of human-centric robotics in a variety of application domains, including industry, education, and defense. This special issue draws specific attention to the use and impact of metrology toward the advancement of HRI technologies and algorithms, and it promotes the application of measurement science toward the benchmarking and replication of HRI research. Special attention is given to the use cases, data sets, test methodologies, measurement techniques, metrics, and statistical analyses used to evaluate system performance.

We present a comprehensive framework and test methodology for the evaluation of human-machine interfaces (HMI) and human-robot interactions (HRI) in collaborative manufacturing applications. An overview of the challenges that face current- and next-generation collaborative robot systems is presented, specifically focused on the interactions between man and machine, and a series of objectively quantitative and subjectively qualitative metrics are given to guide the development and assessment of interfaces and interactions. A generalized set of guidelines for the design of HMI is also proposed to address these challenges and thereby enable effective and intuitive diagnostics and error corrections when process failures occur. These guidelines are aimed at aiding researchers in developing effective interface and interaction technologies, maximizing operator situation awareness in human-robot collaborative manufacturing teams, promoting effective process and system diagnostics reporting, and enabling faster responses to equipment or application errors.

Verified and validated test methods, being necessary to measure the performance of complex systems, are important tools for driving innovation, benchmarking and improving performance, and establishing trust in collaborative human-robot teams. This full-day workshop aims to explore the metrology necessary for repeatably and independently assessing the collaborative performance of robotic systems in real-world human-robot interaction (HRI) scenarios. This workshop aims to bridge the gaps between the theory and applications of HRI in industry, accelerating the adoption of cutting edge technologies as the industry state-of-practice. The interest in collaborative HRI is evident in the current market as well as standards efforts toward manufacturing, social, medical, and service robot solutions. Though these domains have been considered separate for many years, recent technological and scientific advancements show that, while their applications may differ, the underlying principles of HRI performance impact each identically. As such, this workshop seeks to identify test methods and metrics for the holistic assessment and assurance of collaborative HRI performance. The focus is on identifying the key performance indicators of these seemingly disparate sectors, and additionally to establish a community based on the principles of transparency, repeatability, & establishing trust in the assessment of collaborative HRI. The goal is to aid in the advancement of HRI technologies through the development of experimental scenarios, protocols, test methods, & metrics for the verification and validation of interaction solutions and interface designs.

The ability to calculate rigid-body transformations between arbitrary coordinate systems (i.e., registration) is an invaluable tool in robotics. This effort builds upon previous work by investigating strategies for improving the registration accuracy between a robotic arm and an extrinsic coordinate system with relatively inexpensive parts and minimal labor. The framework previously presented is expanded with a new test methodology to characterize the effects of strategies that improve registration performance. In addition, statistical analyses of physical trials reveal that leveraging more data and applying machine learning are two major components for significantly reducing registration error. One-shot peg-in-hole tests are conducted to show the application-level performance gains obtained by improving registration accuracy. Trends suggest that the maximum translation positioning error (postregistration) is a good, albeit not perfect, indicator for peg insertion performance. NOTE TO PRACTITIONERS: In a dynamic robotic workcell environment where robots may be frequently relocated or may need to collaborate with other robots, it is simpler and more robust to program robots in an external or unifying reference frame. The process of robot registration involves finding the location of a robot with respect to another reference frame. For instance, if parts are in known locations on a table and a robot can locate itself with respect to the table (an external reference frame), then the robot will also know the location of the parts. Furthermore, if two or more robots can locate themselves with respect to the table, then each robot will not only know the location of the parts, but also the location of every other robot. This knowledge facilitates the coordination of robot motions and robot collaboration and eases the integration of additional robots into the workcell. Since robot registration is critically necessary and occurs frequently, its process needs to be inexpensive, fast, and accurate. This paper details the requirements for a relatively inexpensive and fast robot registration experience, along with detailing strategies that incur significant improvements to registered robot positioning accuracy with minimal overhead. A quantitative verification process is presented to evaluate the performance impacts of these strategies. Peg-in-hole experiments are conducted to validate the notion that more accurate robot registration translates to more reliable task-level performance.

This paper presents planning algorithms for robotic cleaning of stains on nonplanar surfaces. Access to different portions of the stain may require frequent repositioning and reorienting of the object. Some portions with prominent stain may require multiple passes to remove the stain completely. Two robotic arms have been used in the experiments. The object is immobilized with one arm and the cleaning tool is manipulated with the other. The algorithm generates a sequence of reorientation and repositioning moves required to clean the part after analyzing the stain. The plan is generated by accounting for the kinematic constraints of the robot. Our algorithm uses a depth-first branch-and-bound search to generate setup plans. Cleaning trajectories are generated and optimal cleaning parameters are selected by the algorithm. We have validated our approach through numerical simulations and robotic cleaning experiments with two KUKA robots.

A new methodology for describing the safety of human-robot collaborations is presented. Taking a task-based perspective, a risk assessment of a collaborative robot system safety can be evaluated offline during the initial design stages. This risk assessment factors in such elements as tooling, the nature and duration of expected contacts, and any amortized transfer of pressures and forces onto a human operator. Risk assessments of example tasks are provided for illustrative purposes.

This paper proposes a neurobiology-based extension of integrate-and-fire models of Radial Basis Function Neural Networks (RBFNN) that adapts to novel stimuli by means of dynamic restructuring of the network's structural parameters. The new architecture automatically balances synapses modulation, re-centers hidden Radial Basis Functions (RBFs), and stochastically shifts parameter-space decision planes to maintain homeostasis. Example results are provided throughout the paper to illustrate the effects of changes to the RBFNN model.

A set of metrics is proposed that evaluates speed and separation monitoring efficacy in industrial robot environments in terms of the quantification of safety and the effects on productivity. The collision potential is represented by separation metrics and sensor uncertainty based on perceived noise and bounding region radii. In the event of a bounding region collision between a robot and an obstacle during algorithm evaluation, the severity of the separation failure is reported as a percentage of volume penetration.

We present here a framework for the generation, application, and assessment of assistive models for the purpose of aiding automated robotic parameter optimization methods. Our approach represents an expansion of traditional machine learning implementations by employing models to predict the performances of input parameter sequences and then filter a potential population of inputs prior to evaluation on a physical system. We further provide a basis for numerically qualifying these models to determine whether or not they are of sufficient quality to be capable of fulfilling their predictive responsibilities. We demonstrate the effectiveness of this approach using an industrial robotic testbed on a variety of mechanical assemblies, each requiring a different strategy for completion.

Various “biologically-inspired” models of computation have been developed over the years. Though their inception may have been inspired by biology, most are not biologically plausible. The concepts of training neural networks by back propagation and global observers occurs nowhere in nature. In this paper, a novel variation on a Radial-Basis Function (RBF) network is proposed that is biologically plausible as supported by the literature. A case study is presented that demonstrate the efficacy of this method in producing functional approximations of difficult problems.

This work investigates what makes a robotic assembly process “learnable” for the explicit purpose of improving the performance of that process. It has been observed that even stochastic search methods like Genetic Algorithms (GA) can benefit from advanced models of the assembly task. Models built from the results of random samplings of a parameter space have been used previously to predict the performances of parameter sequences not yet evaluated, but the question of what properties of the models actually benefit the optimization remained. A quantitative analysis algorithm is derived and tested on physical assemblies for validation. Results are provided that illustrate the efficacy of the analysis algorithm for prediction-based performance enhancement when such models are used.