In this paper we describe the intervention framework of ATLAS, an artificial socially intelligent agent that advises teams. The framework treats interventions as atomic components, and manages the lifecycle of each intervention through presentation, as well as followups to interventions. The key benefit of this framework is that it allows for rapid development of scenario-specific Interventions that leverage scenario-agnostic team models. The implementation of this framework is reported for three player teams in a Search and Rescue task simulated in Minecraft. Low competence teams advised by ATLAS improved more between first and second trials than those with a human advisor while the reverse was found for high competence. Four times as many interventions were proposed as were presented. 15 % of advice was withheld to avoid repetitive advice, excessive rate of advice, and needlessly advising high performing teams, while a Theory of Mind model and delay for confirmation mechanism filtered out other unnecessary advice.

In this paper, we propose a novel personalized decision support system that combines Theory of Mind (ToM) modeling and explainable Reinforcement Learning (XRL) to provide effective and interpretable interventions. Our method leverages DRL to provide expert action recommendations while incorporating ToM modeling to understand users' mental states and predict their future actions, enabling appropriate timing for intervention. To explain interventions, we use counterfactual explanations based on RL's feature importance and users' ToM model structure. Our proposed system generates accurate and personalized interventions that are easily interpretable by end-users. We demonstrate the effectiveness of our approach through a series of crowd-sourcing experiments in a simulated team decision-making task, where our system outperforms control baselines in terms of task performance. Our proposed approach is agnostic to task environment and RL model structure, therefore has the potential to be generalized to a wide range of applications.

Theory of Mind (ToM) refers to the ability to make inferences about other’s mental states. Such ability is fundamental for human social activities such as empathy, teamwork, and communication. As intelligent agents come to be involved in diverse human-agent teams, they will also be expected to be socially intelligent in order to become effective teammates. In this paper, we describe a computational ToM model which observes team behaviors and infers their mental states in a urban search and rescue (US&R) task. Our modular ToM model approximates human inference by explicitly representing beliefs, belief updates, and action prediction/generation using Deep Neural Networks (DNNs). To validate our model we compare its performance to the gold standard of human observers asked to make the same inferences. The ToM model proved superior to the average judgments of human observers on all four tests of inference and better than 90th percentile observers on three of the four. While the learning bias provided by modularizing belief and prediction proved sufficient for the simple inferences tested, substantial refinement will be needed to replicate the complex nuanced chains of inference observed in human social interaction.

Leader-follower navigation is a popular class of multi-robot algorithms where a leader robot leads the follower robots in a team. The leader has specialized capabilities or mission critical information (e.g. goal location) that the followers lack, and this makes the leader crucial for the mission’s success. However, this also makes the leader a vulnerability -an external adversary who wishes to sabotage the robot team’s mission can simply harm the leader and the whole robot team’s mission would be compromised. Since robot motion generated by traditional leader-follower navigation algorithms can reveal the identity of the leader, we propose a defense mechanism of hiding the leader’s identity by ensuring the leader moves in a way that behaviorally camouflages it with the followers, making it difficult for an adversary to identify the leader. To achieve this, we combine Multi-Agent Reinforcement Learning, Graph Neural Networks and adversarial training. Our approach enables the multi-robot team to optimize the primary task performance with leader motion similar to follower motion, behaviorally camouflaging it with the followers. Our algorithm outperforms existing work that tries to hide the leader’s identity in a multi-robot team by tuning traditional leader-follower control parameters with Classical Genetic Algorithms. We also evaluated human performance in inferring the leader’s identity and found that humans had lower accuracy when the robot team used our proposed navigation algorithm.

Neural agents trained in reinforcement learning settings can learn to communicate among themselves via discrete tokens, accomplishing as a team what agents would be unable to do alone. However, the current standard of using one-hot vectors as discrete communication tokens prevents agents from acquiring more desirable aspects of communication such as zero-shot understanding. Inspired by word embedding techniques from natural language processing, we propose neural agent architectures that enables them to communicate via discrete tokens derived from a learned, continuous space. We show in a decision theoretic framework that our technique optimizes communication over a wide range of scenarios, whereas one-hot tokens are only optimal under restrictive assumptions. In self-play experiments, we validate that our trained agents learn to cluster tokens in semantically-meaningful ways, allowing them communicate in noisy environments where other techniques fail. Lastly, we demonstrate both that agents using our method can effectively respond to novel human communication and that humans can understand unlabeled emergent agent communication, outperforming the use of one-hot communication.

To build an agent providing assistance to human rescuers in an urban search and rescue task, it is crucial to understand not only human actions but also human beliefs that may influence the decision to take these actions. Developing data-driven models to predict a rescuer’s strategies for navigating the environment and triaging victims requires costly data collection and training for each new environment of interest. Transfer learning approaches can be used to mitigate this challenge, allowing a model trained on a source environment/task to generalize to a previously unseen target environment/task with few training examples. In this paper, we investigate transfer learning (a) from a source environment with smaller number of types of injured victims to one with larger number of victim injury classes and (b) from a smaller and simpler environment to a larger and more complex one for navigation strategy. Inspired by hierarchical organization of human spatial cognition, we used graph division to represent spatial knowledge, and Transfer Learning Diffusion Convo-lutional Recurrent Neural Network (TL-DCRNN), a spatial and temporal graph-based recurrent neural network suitable for transfer learning, to predict navigation. To abstract the rescue strategy from a rescuer’s field-of-view stream, we used attention-based LSTM networks. We experimented on various transfer learning scenarios and evaluated the performance using mean average error. Results indicated our assistant agent can improve predictive accuracy and learn target tasks faster when equipped with transfer learning methods.

The ability to collaborate with previously unseen human teammates is crucial for artificial agents to be effective in human-agent teams (HATs). Due to individual differences and complex team dynamics, it is hard to develop a single agent policy to match all potential teammates. In this article, we study both human-human and HAT in a dyadic cooperative task, Team Space Fortress. Results show that the team performance is influenced by both players’ individual skill level and their ability to collaborate with different teammates by adopting complementary policies. Based on human-human team results, we propose an adaptive agent that identifies different human policies and assigns a complementary partner policy to optimize team performance. The adaptation method relies on a novel similarity metric to infer human policy and then selects the most complementary policy from a pretrained library of exemplar policies. We conducted human-agent experiments to evaluate the adaptive agent and examine mutual adaptation in HAT. Results show that both human adaptation and agent adaptation contribute to team performance.

Human behaviors are often prohibited, or permitted by social norms. Therefore, if autonomous agents interact with humans, they also need to reason about various legal rules, social and ethical social norms, so they would be trusted and accepted by humans. Inverse Reinforcement Learning (IRL) can be used for the autonomous agents to learn social norm-compliant behavior via expert demonstrations. However, norms are context-sensitive, i.e. different norms get activated in different contexts. For example, the privacy norm is activated for a domestic robot entering a bathroom where a person may be present, whereas it is not activated for the robot entering the kitchen. Representing various contexts in the state space of the robot, as well as getting expert demonstrations under all possible tasks and contexts is extremely challenging. Inspired by recent work on Modularized Normative MDP (MNMDP) and early work on context-sensitive RL, we propose a new IRL framework, Context-Sensitive Norm IRL (CNIRL). CNIRL treats states and contexts separately, and assumes that the expert determines the priority of every possible norm in the environment, where each norm is associated with a distinct reward function. The agent chooses the action to maximize its cumulative rewards. We present the CNIRL model and show that its computational complexity is scalable in the number of norms. We also show via two experimental scenarios that CNIRL can handle problems with changing context spaces.

The reported study compares groups of 120 participants each, from the United States (U.S.), Taiwan (TW), and Turkey (TK), interacting with versions of an automated path planner that vary in transparency and degree of automation. The nationalities were selected in accordance with the theory of cultural syndromes as representatives of Dignity (U.S.), Face (TW), and Honor (TK) cultures, and were predicted to differ in readiness to trust automation, degree of transparency required to use automation, and willingness to use systems with high degrees of automation. Three experimental conditions were tested. In the first, highlight, path conflicts were highlighted leaving rerouting to the participant. In the second, replanner made requests for permission to reroute when a path conflict was detected. The third combined condition increased transparency of the replanner by combining highlighting with rerouting to make the conflict on which decision was based visible to the user. A novel framework relating transparency, stages of automation, and trust in automation is proposed in which transparency plays a primary role in decisions to use automation but is supplemented by trust where there is insufficient information otherwise. Hypothesized cultural effects and framework predictions were confirmed.

In this paper, we study human trust and its computational models in supervisory control of swarm robots with varied levels of autonomy (LOA) in a target foraging task. We implement three LOAs: manual, mixed-initiative (MI), and fully autonomous LOA. While the swarm in the MI LOA is controlled by a human operator and an autonomous search algorithm collaboratively, the swarms in the manual and autonomous LOAs are fully directed by the human and the search algorithm, respectively. From user studies, we find that humans tend to make their decisions based on physical characteristics of the swarm rather than its performance since the task performance of swarms is not clearly perceivable by humans. Based on the analysis, we formulate trust as a Markov decision process whose state space includes the factors affecting trust. We develop variations of the trust model for different LOAs. We employ an inverse reinforcement learning algorithm to learn behaviors of the operator from demonstrations where the learned behaviors are used to predict human trust. Compared to an existing model, our models reduce the prediction error by at most 39.6%, 36.5%, and 28.8% in the manual, MI, and auto-LOA, respectively.

As domestic service robots become more common and widespread, they must be programmed to efficiently accomplish tasks while aligning their actions with relevant norms. The first step to equip domestic robots with normative reasoning competence is understanding the norms that people apply to the behavior of robots in specific social contexts. To that end, we conducted an online survey of Chinese and United States participants in which we asked them to select the preferred normative action a domestic service robot should take in a number of scenarios. The paper makes multiple contributions. Our extensive survey is the first to: (a) collect data on attitudes of people on normative behavior of domestic robots, (b) across cultures and (c) study relative priorities among norms for this domain. We present our findings and discuss their implications for building computational models for robot normative reasoning.

Swarm robots are coordinated via simple control laws to generate emergent behaviors such as flocking, rendezvous, and deployment. Human-swarm teaming has been widely proposed for scenarios, such as human-supervised teams of unmanned aerial vehicles (UAV) for disaster rescue, UAV and ground vehicle cooperation for building security, and soldier-UAV teaming in combat. Effective cooperation requires an appropriate level of trust, between a human and a swarm. When an UAV swarm is deployed in a real-world environment, its performance is subject to real-world factors, such as system reliability and wind disturbances. Degraded performance of a robot can cause undesired swarm behaviors, decreasing human trust. This loss of trust, in turn, can trigger human intervention in UAVs' task executions, decreasing cooperation effectiveness if inappropriate. Therefore, to promote effective cooperation we propose and test a trust-repairing method (Trust-repair) restoring performance and human trust in the swarm to an appropriate level by correcting undesired swarm behaviors. Faulty swarms caused by both external and internal factors were simulated to evaluate the performance of the Trust-repair algorithm in repairing swarm performance and restoring human trust. Results show that Trust-repair is effective in restoring trust to a level intermediate between normal and faulty conditions.

In recent years, there has been increasing interest in transparency in Deep Neural Networks. Most of the works on transparency have been done for image classification. In this paper, we report on work of transparency in Deep Reinforcement Learning Networks (DRLNs). Such networks have been extremely successful in learning action control in Atari games. In this paper, we focus on generating verbal (natural language) descriptions and explanations of deep reinforcement learning policies. Successful generation of verbal explanations would allow better understanding by people (e.g., users, debuggers) of the inner workings of DRLNs which could ultimately increase trust in these systems. We present a generation model which consists of three parts: an encoder on feature extraction, an attention structure on selecting features from the output of the encoder, and a decoder on generating the explanation in natural language. Four variants of the attention structure full attention, global attention, adaptive attention and object attention - are designed and compared. The adaptive attention structure performs the best among all the variants, even though the object attention structure is given additional information on object locations. Additionally, our experiment results showed that the proposed encoder outperforms two baseline encoders (Resnet and VGG) on the capability of distinguishing the game state images.

Autonomous AI systems will be entering human society in the near future to provide services and work alongside humans. For those systems to be accepted and trusted, the users should be able to understand the reasoning process of the system, i.e. the system should be transparent. System transparency enables humans to form coherent explanations of the system's decisions and actions. Transparency is important not only for user trust, but also for software debugging and certification. In recent years, Deep Neural Networks have made great advances in multiple application areas. However, deep neural networks are opaque. In this paper, we report on work in transparency in Deep Reinforcement Learning Networks (DRLN). Such networks have been extremely successful in accurately learning action control in image input domains, such as Atari games. In this paper, we propose a novel and general method that (a) incorporates explicit object recognition processing into deep reinforcement learning models, (b) forms the basis for the development of "object saliency maps", to provide visualization of internal states of DRLNs, thus enabling the formation of explanations and (c) can be incorporated in any existing deep reinforcement learning framework. We present computational results and human experiments to evaluate our approach.

Swarm search and service (SSS) missions require large swarms to simultaneously search an area while servicing jobs as they are encountered. Jobs must be immediately serviced and can be one of several different job types - each requiring a different service time and number of vehicles to complete its service successfully. After jobs are serviced, vehicles are returned to the swarm and become available for reallocation. As part of SSS mission planning, human operators must determine the number of vehicles needed to achieve this balance. The complexities associated with balancing vehicle allocation to multiple as yet unknown tasks with returning vehicles makes this extremely difficult for humans. Previous work assumes that all system jobs are known ahead of time or that vehicles move independently of each other in a multi-agent framework. We present a dynamic vehicle routing (DVR) framework whose policies optimally allocate vehicles as jobs arrive. By incorporating time constraints into the DVR framework, an M/M/k/k queuing model can be used to evaluate overall steady state system performance for a given swarm size. Using these estimates, operators can rapidly compare system performance across different configurations, leading to more effective choices for swarm size. A sensitivity analysis is performed and its results are compared with the model, illustrating the appropriateness of our method to problems of plausible scale and complexity.

Autonomous robots are envisioned to increasingly become part of our lives in the house, restaurants, hospitals and offices. Additionally, self-driving cars will be soon appearing in city streets and highways and they will have to interact with cars driven by humans as well as other self-driving cars. In these settings the robots not only need to efficiently perform their tasks but also be able to interact with humans in socially appropriate ways. To accomplish this, robots must be able to reason not only on how to perform their tasks, but also incorporate societal values, social norms and legal rules so they can gain human acceptability and trust. Moreover, interactions with these robots will be long term. Long-term human interaction with robots as well as robot combined reasoning about both tasks and social norms generate multiple modeling and computational challenges. In this paper, we address one of the most important of these challenges, namely what is an appropriate and scalable computational framework that enables simultaneous task and normative reasoning. In particular, we report on our work on a novel computational framework, Modular Normative Markov Decision Processes (MNMDP) that integrates reasoning for domain tasks and normative reasoning for long-term autonomy. The MNMDP framework applies normative reasoning considering only the norms that are activated in appropriate contexts, rather than considering the full set of norms, thus significantly reducing computational complexity. The model modularity is also advantageous for long-term human-robot interaction. We present computational experiments that show significant computational improvements as compared with a base Normative Markov Decision Process (MDP) framework that includes the full set of norms.

As part of swarm search and service (SSS) missions, robots are tasked with servicing jobs as they are sensed. This requires small sub-swarm teams to leave the swarm for a specified amount of time to service the jobs. In doing so, fewer robots are required to change motion than if the whole swarm were diverted, thereby minimizing the job's overall effect on the swarm's main goal. We explore the problem of removing the required number of robots from the swarm, while maintaining overall swarm connectivity. By preserving connectivity, robots are able to successfully rejoin the swarm upon completion of their assigned job. These robots are then made available for reallocation. We propose a decentralized and asynchronous method for breaking off sub-swarm groups and rejoining them with the main swarm using the swarm's communication graph topology. Both single and multiple job site cases are explored. The results are compared against a full swarm movement method. Simulation results show that the proposed method outperforms a full swarm method in the average number of messages sent per robot in each step, as well as, the distance traveled by the swarm.

The interaction between swarm robots and human operators is significantly different from the traditional humanrobot interaction due to unique characteristics of the system, such as high cognitive complexity and difficulties in state estimation. In this paper, we concentrated on the method of conveying input from the operator to the swarm. Previous research has shown that control through switching between behaviors offers the greatest flexibility but is particularly difficult for human operators. A recently developed method for finding optimal sequences for composing behaviors offered a potential tool for aiding human operators controlling swarms through behavior switching. This paper compared participants performing a navigation task with and without the availability of the optimal sequencing aid. Results showed that the task of preplanning a sequence of behaviors and durations appeared more difficult for participants than switching between executing behaviors to navigate. Users who used the aid frequently was found to create shorter paths than infrequent users and the control group. In the trails that the aid was used, participants tended to generate more complicated sequences and achieve the first attempt more rapidly, compared to the trails that the aid was not used.

In this paper, we study trust-related human factors in supervisory control of swarm robots with varied levels of autonomy (LOA) in a target foraging task. We compare three LOAs: manual, mixed-initiative (MI), and fully autonomous LOA. In the manual LOA, the human operator chooses headings for a flocking swarm, issuing new headings as needed. In the fully autonomous LOA, the swarm is redirected automatically by changing headings using a search algorithm. In the mixed-initiative LOA, if performance declines, control is switched from human to swarm or swarm to human. The result of this work extends the current knowledge on human factors in swarm supervisory control. Specifically, the finding that the relationship between trust and performance improved for passively monitoring operators (i.e., improved situation awareness in higher LOAs) is particularly novel in its contradiction of earlier work. We also discover that operators switch the degree of autonomy when their trust in the swarm system is low. Last, our analysis shows that operator's preference for a lower LOA is confirmed for a new domain of swarm control.

Trust in automation has become a topic of intensive study since the late 1990s and is of increasing importance with the advent of intelligent interacting systems. While the earliest trust experiments involved human interventions to correct failures/errors in automated control systems, a majority of subsequent studies have investigated information acquisition and analysis decision aiding tasks such as target detection for which automation reliability is more easily manipulated. Despite the high level of international dependence on automation in industry, almost all current studies have employed Western samples primarily from the U.S. The present study addresses these gaps by running a large sample experiment in three (U.S., Taiwan, and Turkey) diverse cultures using a “trust sensitive task” consisting of both automated control and target detection subtasks. This article presents results for the target detection subtask for which reliability and task load were manipulated. The current experiments allow us to determine whether reported effects are universal or specific to Western culture, vary in baseline or magnitude, or differ across cultures. Results generally confirm consistent effects of manipulations across the three cultures as well as cultural differences in initial trust and variation in effects of manipulations consistent with 10 cultural hypotheses based on Hofstede's Cultural Dimensions and Leung and Cohen's theory of Cultural Syndromes. These results provide critical implications and insights for correct trust calibration and to enhance human trust in intelligent automation systems across cultures. Additionally, our results would be useful in designing intelligent systems for users of different cultures. Our article presents the following contributions: First, to the best of our knowledge, this is the first set of studies that deal with cultural factors across all the cultural syndromes identified in the literature by comparing trust in the Honor, Face, Dignity cultures. Second, this is the first set of studies that uses a validated cross-cultural trust measure for measuring trust in automation. Third, our experiments are the first to study the dynamics of trust across cultures.

In this paper, we study the model of human trust where an operator controls a robotic swarm remotely for a search mission. Existing trust models in human-in-the-loop systems are based on task performance of robots. However, we find that humans tend to make their decisions based on physical characteristics of the swarm rather than its performance since task performance of swarms is not clearly perceivable by humans. We formulate trust as a Markov decision process whose state space includes physical parameters of the swarm. We employ an inverse reinforcement learning algorithm to learn behaviors of the operator from a single demonstration. The learned behaviors are used to predict the trust level of the operator based on the features of the swarm.

We present a method to validate a cognitive model, based on the cognitive architecture ACT-R, in dynamic human-automation systems with discrete human input. We are inspired by the general problem of K-choice games as a proxy for many decision making applications in dynamical systems. We model the human as a Markovian controller based on gathered experimental data, that is, a non-deterministic control input with known likelihoods of control actions associated with certain configurations of the state-space. We use reachability analysis to predict the outcome of the resulting discrete-time stochastic hybrid system, in which the outcome is defined as a function of the system trajectory. We suggest that the resulting expected outcomes can be used to validate the cognitive model against actual human subject data. We apply our method to a two-choice game in which the human is tasked with maximizing net coverage of a robotic swarm that can operate under rendezvous or deployment dynamics. We validate the corresponding ACTR cognitive model generated with the data from eight human subjects. The novelty of this work is (1) a method to compute expected outcome in a hybrid dynamical system with a Markov chain model of the human's discrete choice, and (2) application of this method to validation of cognitive models with a database of actual human subject data.

The use of autonomous systems has been rapidly increasing in recent decades. To improve human-automation interaction, trust has been closely studied. Research shows trust is critical in the development of appropriate reliance on automation. To examine how trust mediates the human-automation relationships across cultures, the present study investigated the influences of cultural factors on trust in automation. Theoretically guided empirical studies were conducted in the U.S., Taiwan and Turkey to examine how cultural dynamics affect various aspects of trust in automation. The results found significant cultural differences in human trust attitude in automation.

Human swarm interaction (HSI) involves operators gathering information about a swarm's state as it evolves, and using it to make informed decisions on how to influence the collective behavior of the swarm. In order to determine the proper input, an operator must have an accurate representation and understanding of the current swarm state, including what emergent behavior is currently happening. In this paper, we investigate how human operators perceive three types of common, emergent swarm behaviors: rendezvous, flocking, and dispersion. Particularly, we investigate how recognition of these behaviors differ from each other in the presence of background noise. Our results show that, while participants were good at recognizing all behaviors, there are indeed differences between the three, with rendezvous being easier to recognize than flocking or dispersion. Furthermore, differences in recognition are also affected by viewing time for flocking. Feedback from participants was also especially insightful for understanding how participants went about recognizing behaviors-allowing for potential avenues of research in future studies.

Large teams of robots that operate collectively, whose behavior emerges from local interactions with neighbors, are known as swarms. While significant progress has been made improving the hardware, communication capabilities, and autonomous operation of these swarms, we still have much to learn about how human operators control and interact with them. This research is necessary if real world swarms are to be deployed in the future. The study presented here investigates different methods of displaying information about the swarm state to operators, and asks them to make predictions about the swarm's future state. In the study, participants are shown swarms performing one of three different behaviors, and are asked to use the information available from the display to make their predictions. Results show that summarizing the swarm's current state to just an average position and bounding ellipse allowed predictions as accurate as those made when full state information was shown. Furthermore, two leader-based methods were used, whereby the operators were shown only a small subset of the swarm. However, such display methods were inferior for prediction than either the summary center and ellipse or full information methods. With these results, and with participant feedback about the helpfulness of the four display types, we hope future studies can make more informed decision about interface design when it comes to the control of swarms.

Recent advances in technology are delivering robots of reduced size and cost. A natural outgrowth of these advances are systems comprised of large numbers of robots that collaborate autonomously in diverse applications. Research on effective autonomous control of such systems, commonly called swarms, has increased dramatically in recent years and received attention from many domains, such as bioinspired robotics and control theory. These kinds of distributed systems present novel challenges for the effective integration of human supervisors, operators, and teammates that are only beginning to be addressed. This paper is the first survey of human-swarm interaction (HSI) and identifies the core concepts needed to design a human-swarm system. We first present the basics of swarm robotics. Then, we introduce HSI from the perspective of a human operator by discussing the cognitive complexity of solving tasks with swarm systems. Next, we introduce the interface between swarm and operator and identify challenges and solutions relating to human-swarm communication, state estimation and visualization, and human control of swarms. For the latter, we develop a taxonomy of control methods that enable operators to control swarms effectively. Finally, we synthesize the results to highlight remaining challenges, unanswered questions, and open problems for HSI, as well as how to address them in future works.

Robotic swarms are distributed systems whose members interact via local control laws to achieve a variety of behaviors, such as flocking. In many practical applications, human operators may need to change the current behavior of a swarm from the goal that the swarm was going towards into a new goal due to dynamic changes in mission objectives. There are two related but distinct capabilities needed to supervise a robotic swarm. The first is comprehension of the swarm's state and the second is prediction of the effects of human inputs on the swarm's behavior. Both of them are very challenging. Prior work in the literature has shown that inserting the human input as soon as possible to divert the swarm from its original goal towards the new goal does not always result in optimal performance (measured by some criterion such as the total time required by the swarm to reach the second goal). This phenomenon has been called Neglect Benevolence, conveying the idea that in many cases it is preferable to neglect the swarm for some time before inserting human input. In this paper, we study how humans can develop an understanding of swarm dynamics so they can predict the effects of the timing of their input on the state and performance of the swarm. We developed the swarm configuration shape-changing Neglect Benevolence Task as a Human Swarm Interaction (HSI) reference task allowing comparison between human and optimal input timing performance in control of swarms. Our results show that humans can learn to approximate optimal timing and that displays which make consensus variables perceptually accessible can enhance performance.

Many researchers have employed some form of teleoperated leader to influence a robotic swarm; however, the way in which this influence is conveyed has not been well studied. Some researchers employ designated leaders that are known to be leaders by other members of the swarm and hence followed. Others do not impose a leader/follower distinction on the swarm's algorithms and instead choose to influence the swarm indirectly through controlling one or more of its members. Because the robustness of swarm behavior arises from its many distributed interactions, influence through designated leaders might render it susceptible to noise or disrupt its coherence by overriding these mechanisms. Conversely, limiting human influence to indirect control through the local effects of a leader might prove too sluggish to allow effective human control. This paper compares leader-based methods of each type, designated as Tacit leadership via consensus (no explicit leader/follower distinction) and Explicit leadership via flooding (influence propagating from leader takes precedence). These methods were compared in simulation and in human experiments finding that explicit leadership led to faster convergence in simulation and better performance in the experiments. Effects of noise were slightly more pronounced for Explicit leaders and cohesion slightly poorer.

Controlling a swarm of robots after deployment is difficult, due to the unpredictable and emergent behavior of swarm algorithms. Past work has focused on influencing the swarm via statically selected leaders—swarm members that the operator directly controls—that are pre-selected and remain leaders throughout the scenario execution. This paper investigates the use of dynamically selected leaders that are directly controlled by the human operator to guide the rest of the swarm, which is operating under a flocking-style algorithm. The goal of the operator is to move the swarm to goal regions that arise dynamically in the environment. We experimentally investigated (a) the effect of density of leaders on the ease of human control and system performance, and (b) how restriction of information communicated to the human operator affects the ability to guide the swarm to goal regions. The density of leaders is computed based on an extension of the random competition clustering (RCC) algorithm used in wireless sensor networks to select cluster heads. In particular, we studied the effect of different guarantees of the maximum number of hops in the communication graph from any robot to the nearest leader. Increasing the maximum hop guarantee effectively lowers the density of leaders in the swarm. Our results show that, while there was a large drop in the number of goals reached when moving from a 1-hop to a 2-hop guarantee, the difference between a 2-hop and 3-hop guarantee was not statistically significant. Furthermore, we found that performance was just as good when the information returned to the operator was restricted, showing that operators can still navigate a swarm even when they have imperfect information.

In this seminar we will consider how concepts of information fusion, crowdsourcing and mobility complement each other and accelerate novel advanced research directions in mobile data management. We will elaborate on each of those concepts and explore their synergy under a prominent scenario of situation assessment in multi-robot search and rescue missions.

The study of human control of robotic swarms involves designing interfaces and algorithms for allowing a human operator to influence a swarm of robots. One of the main difficulties, however, is determining how to most effectively influence the swarm after it has been deployed. Past work has focused on influencing the swarm via statically selected leaders-swarm members that the operator directly controls. This paper investigates the use of a small subset of the swarm as leaders that are dynamically selected during the scenario execution and are directly controlled by the human operator to guide the rest of the swarm, which is operating under a flocking-style algorithm. The goal of the operator in this study is to move the swarm to goal regions that arise dynamically in the environment.We experimentally investigated three different aspects of dynamic leader-based swarm control and their interactions: leader density (in terms of guaranteed hops to a leader), sensing error, and method of information propagation from leaders to the rest of the swarm. Our results show that, while there was a large drop in the number of goals reached when moving from a 1-hop to a 2-hop guarantee, the difference between a 2-hop, 3-hop, and 4-hop guarantee was not statistically significant. Furthermore, we found that sensing error impacted the explicit information-propagation method more than the tacit method conditions, and caused participants more trouble the lower the density of leaders, although the explicit method performed better overall.

In this paper, we introduce an automatic information fusion method that exploits the collective intelligence of mobile robots to efficiently “crowdsource” victim detection tasks. We reduce the load on the operators requiring them to acknowledge only presence of the victim in an image (to annotate the image). The task of finding victim location is performed via automatic fusion of annotated images from the image queue.

A robotic swarm is a decentralized group of robots which overcome failure of individual robots with robust emergent behaviors based on local interactions. These behaviors are not well built for accomplishing complex tasks, however, because of the changing assumptions required in various applications and environments. A new movement in the research field is to add human input to influence the swarm in order to help make the robots goal directed and overcome these problems. This research in Human Swarm Interaction (HSI) focuses on different control laws and ways to integrate the human intent with local control laws of the robots. Previous studies have all used visual feedback through a computer interface to give the user the swarm state information. This study adapted swarm control algorithms to give the operator hap tic feedback as well as visual feedback. The study shows the benefits of the additional feedback in a target searching class. Researchers in multi-robot systems have shown benefits of hap tic feedback in obstacle navigation before, but this study is a novel method because of the decentralized formation of the robotic swarm. In most environments, operators were able to cover significantly more area, increasing the chance of finding more targets. The other environment found no significant difference, showing that the hap tic feedback does not degrade performance in any of the tested environments. This supports our hypothesis that hap tic feedback is useful in HSI and requires further research to maximize its potential.

As swarms are used in increasingly more complex scenarios, further investigation is needed to determine how to give human operators the best tools to properly influence the swarm after deployment. Previous research has focused on relaying influence from the operator to the swarm, either by broadcasting commands to the entire swarm or by influencing the swarm through the teleoperation of a leader. While these methods each have their different applications, there has been a lack of research into how the influence should be propagated through the swarm in leader-based methods. This paper focuses on two simple methods of information propagation-flooding and consensus-and compares the ability of operators to maneuver the swarm to goal points using each, both with and without sensing error. Flooding involves each robot explicitly matching the speed and direction of the leader (or matching the speed and direction of the first neighboring robot that has already done so), and consensus involves each robot matching the average speed and direction of all the neighbors it senses. We discover that the flooding method is significantly more effective, yet the consensus method has some advantages at lower speeds, and in terms of overall connectivity and cohesion of the swarm.

In this paper we present the first study of human-swarm interaction comparing two fundamental types of interaction, coined intermittent and environmental. These types are exemplified by two control methods, selection and beacon control, made available to a human operator to control a foraging swarm of robots. Selection and beacon control differ with respect to their temporal and spatial influence on the swarm and enable an operator to generate different strategies from the basic behaviors of the swarm. Selection control requires an active selection of groups of robots while beacon control exerts an influence on nearby robots within a set range. Both control methods are implemented in a testbed in which operators solve an information foraging problem by utilizing a set of swarm behaviors. The robotic swarm has only local communication and sensing capabilities. The number of robots in the swarm range from 50 to 200. Operator performance for each control method is compared in a series of missions in different environments with no obstacles up to cluttered and structured obstacles. In addition, performance is compared to simple and advanced autonomous swarms. Thirty-two participants were recruited for participation in the study. Autonomous swarm algorithms were tested in repeated simulations. Our results showed that selection control scales better to larger swarms and generally outperforms beacon control. Operators utilized different swarm behaviors with different frequency across control methods, suggesting an adaptation to different strategies induced by choice of control method. Simple autonomous swarms outperformed human operators in open environments, but operators adapted better to complex environments with obstacles. Human controlled swarms fell short of task-specific benchmarks under all conditions. Our results reinforce the importance of understanding and choosing appropriate types of human-swarm interaction when designing swarm systems, in addition to choosing appropriate swarm behaviors.

In this paper we investigate principles of swarm control that enable a human operator to exert influence on and control large swarms of robots. We present two principles, coined selection and beacon control, that differ with respect to their temporal and spatial persistence. The former requires active selection of groups of robots while the latter exerts a passive influence on nearby robots. Both principles are implemented in a testbed in which operators exert influence on a robot swarm by switching between a set of behaviors ranging from trivial behaviors up to distributed autonomous algorithms. Performance is tested in a series of complex foraging tasks in environments with different obstacles ranging from open to cluttered and structured. The robotic swarm has only local communication and sensing capabilities with the number of robots ranging from 50 to 200. Experiments with human operators utilizing either selection or beacon control are compared with each other and to a simple autonomous swarm with regard to performance, adaptation to complex environments, and scalability to larger swarms. Our results show superior performance of autonomous swarms in open environments, of selection control in complex environments, and indicate a potential for scaling beacon control to larger swarms.

A wide class of multirobot control tasks involves operator interactions with individual robots. Where the robots' actions are independent, as for example in some foraging tasks, the operator can interact with robots sequentially in a round robin fashion. If the need for interaction can be detected by the robot through self-reflection, the robot could communicate its need for interaction to the operator. The resulting human-robot system would form a queuing system in which the operator is the server and the queue of robots requesting interaction, the jobs. As a queuing system, performance could be optimized using standard techniques, providing the operator's attention could be appropriately directed. An earlier study found that Human-Robot Interaction (HRI) performance was improved by communicating requests for interaction to the operator, however, a first-in-first-out (FIFO) aid showing a single request at a time led to poorer performance than one showing the entire (Open) queue. The current experiment compared Open-queue and FIFO conditions from the first experiment with a Priority-queue using a shortest job first (SJF) discipline known to maximize throughput. Performance in the Priority-queue condition was statistically indistinguishable from the best performance for all measures except those for missed victims where it was intermediate between FIFO (best) and Open-queue. Both of the other conditions produced poorest performance on some measures. The results suggest that operator attention can be effectively scheduled allowing the use of scheduling algorithms to improve the efficiency of HRI.

The present study investigates the effect of imperfect automation in a human multi-robot controlled environment with different principles for scheduling an operator's attention in a foraging task. The experiment compared a SJF-queue (shortest job first) presenting a single alarm at a time with an Open-queue which showed all current alarms. Two levels of automation reliability, high (90%) and low (50%), were examined in the study. Performance for the queue mechanisms was equivalent confirming that operator attention can be effectively directed to improve performance. Additionally, the higher reliability condition raised an operator's success rate for resolving robot failures and assisted the operator in allocating attention to emergent events in a timely manner. Although the more frequent alerts contributed to better performance operators experienced increased levels of workload.

Swarm robots use simple local rules to create complex emergent behaviors. The simplicity of the local rules allows for large numbers of low-cost robots in deployment, but the same simplicity creates difficulties when deploying in many applicable environments. These complex missions sometimes require human operators to influence the swarms towards achieving the mission goals. Human swarm interaction (HSI) is a young field with few user studies exploring operator behavior. These studies all assume perfect information between the operator and the swarm, which is unrealistic in many applicable scenarios. Indoor search and rescue or underwater exploration may present environments where radio limitations restrict the bandwidth of the robots. This study explores this bandwidth restriction in a user study. Three levels of bandwidth are explored to determine what amount of information is necessary to accomplish a swarm foraging task. The lowest bandwidth condition performs poorly, but the medium and high bandwidth condition both perform well. The medium bandwidth condition does so by aggregating useful swarm information to compress the state information. Further, the study shows operators preferences that should have hindered task performance, but operator adaptation allowed for error correction.

Autonomous swarm algorithms have been studied extensively in the past several years. However, there is little research on the effect of injecting human influence into a robot swarm-whether it be to update the swarm's current goals or reshape swarm behavior. While there has been growing research in the field of human-swarm interaction (HSI), no previous studies have investigated how humans interact with swarms under communication latency.We investigate the effects of latency both with and without a predictive display in a basic swarm foraging task to see if such a display can help mitigate the effects of delayed feedback of the swarm state. Furthermore, we introduce a new concept called neglect benevolence to represent how a human operator may need to give time for swarm algorithms to stabilize before issuing new commands, and we investigate it with respect to task performance. Our study shows that latency did affect a user's ability to control a swarm to find targets in the foraging task, and that the predictive display helped to remove these effects. We also found evidence for neglect benevolence, and that operators exploited neglect benevolence in different ways, leading to two different, but equally successful strategies in the target-searching task.

In this paper, we present an asynchronous display method, coined image queue, which allows operators to search through a large amount of data gathered by autonomous robot teams. We discuss and investigate the advantages of an asynchronous display for foraging tasks with emphasis on Urban Search and Rescue. The image queue approach mines video data to present the operator with a relevant and comprehensive view of the environment in order to identify targets of interest such as injured victims. It fills the gap for comprehensive and scalable displays to obtain a network-centric perspective for UGVs. We compared the image queue to a traditional synchronous display with live video feeds and found that the image queue reduces errors and operator's workload. Furthermore, it disentangles target detection from concurrent system operations and enables a call center approach to target detection. With such an approach we can scale up to very large multi-robot systems gathering huge amounts of data that is then distributed to multiple operators.

This paper presents a queueing model that ad dresses robot self-assessment in human-robot-interaction systems. We build the model based on a game theoretic queueing approach, and analyze four issues: 1) individual differences in operator skills/capabilities, 2) differences in difficulty of presenting tasks, 3) trade-off between human interaction and performance and 4) the impact of task heterogeneity in the optimal service decision-making and system performance. The subsequent analytical and numerical exploration helps under stand the way the decentralized decision-making scheme is affected by various service environments.

We address the problem of searching for moving targets in large outdoor environments represented by height maps. To solve the problem we present a complete system that computes from an annotated height map a graph representation and search strategies based on worst-case assumptions about all targets. These strategies are then used to compute a schedule and task assignment for all agents. We improve the graph construction from previous work and for the first time present a method that computes a schedule to minimize the execution time. For this we consider travel times of agents determined by a path planner on the height map. We demonstrate the entire system in a real environment with an area of 700,000m2in which eight human agents search for two intruders using mobile computing devices (iPads). To the best of our knowledge this is the first demonstration of a search system applied to such a large environment.

Controlling a team of Unmanned Aerial Vehicles (UAV) requires the operator to perform continuous surveillance and path planning. The operator's situation awareness is likely to degrade as an increasing number of surveillance videos must be viewed and integrated. The Picture-in-Picture display (PiP) provides one solution for integrating multiple UAV camera video by allowing the operator to view the video feed in the context of surrounding terrain. The experimental SUAVE (Simple Unmanned Areal Vehicle Environment) display extends PiP methods by sampling imagery from the video stream to texture a 3D map of the terrain. The operator can then inspect this imagery using world in miniature (WIM) or fly-through methods. We investigate the properties and advantages of SUAVE in the context of a search mission with 11 UAVs finding a strong advantage for finding targets While performance is expected to improve with increasing numbers of UAVs we did not find differences in performance between models generated by 11 UAVs and those employing 22 UAVs.

Using classical methods the time course of workload over a work session is rarely observed. Subjective measures of necessity are for the entire session rather than instantaneous. While this distinction makes little difference to researchers wanting to compare task difficulty, it is important to designers of systems that wish to schedule operator attention because they must predict workload. The present study provides an experimental setting in which intervals of continuous work and uninterrupted rest can be precisely controlled. No differences in probability of error or latency of response were found for the length of break/rest intervals or their location in a sequence of intervals. The local utilization within a break/work interval (ratio of work/(work + rest)), however, was significant, suggesting that utilization effects are strictly local and that simple algorithms could be used to optimize human-system performance for such tasks.

Exploration of unknown environments has numerous applications in the domains of search and rescue, pursuit-evasion, map building, and military espionage and intelligence gathering. While it is ideal to have autonomous robots search intelligently based on global patterns and features of each environment, such learning algorithms should also be informed by, and perform at least as well as, human exploration. Most autonomous exploration algorithms focus on local and greedy choices and are less concerned with learning to recognize structural features of environments in order to improve global exploration performance. If the strategies people use during exploration can be determined, they can be used as a basis for creating good utility functions for autonomous robots to be optimized by machine learning techniques. Here, we perform a study where human participants explore both random and patterned environments, in two different time-sensitive scenarios, in an attempt to determine what strategies people use to maximize area explored under a variety of cases. We found that participants in a controlled study did in fact adapt their exploration strategies in time-sensitive scenarios and also exploited features of the environments in order to achieve better global exploration performance.

In this paper we present an approach for a pursuit-evasion problem that considers a 2.5d environment represented by a height map. Such a representation is particularly suitable for large-scale outdoor pursuit-evasion, captures some aspects of 3d visibility and can include target heights. In our approach we construct a graph representation of the environment by sampling strategic locations and computing their detection sets, an extended notion of visibility. From the graph we compute strategies using previous work on graph-searching. These strategies are used to coordinate the robot team and to generate paths for all robots using an appropriate classification of the terrain. In experiments we investigate the performance of our approach and provide examples including a sample map with multiple loops and elevation plateaus and two realistic maps, a village and a mountain range. To the best of our knowledge the presented approach is the first viable solution to 2.5d pursuit-evasion with height maps.

Technology for multirobot systems has advanced to the point where we can consider their use in a variety of important domains, including urban search and rescue. A key to the practical usefulness of multirobot systems is the ability to have a large number of robots effectively controlled by small numbers of operators. In this paper, two modalities for controlling a team of 24 robots in a foraging task in an urban search and rescue environment are compared. In both modalities, multiple operators must monitor video streams from the robots to detect and mark victims on a map as well as teleoperating robots that cannot get themselves out of difficult situations. In the first modality, the operators must also provide waypoints for the robots to explore, using both video and a partially completed map to choose appropriate waypoints. In the second modality, the robots autonomously plan their paths, allowing operators to focus on monitoring the video, but without being able to interpret video streams to guide exploration. Experimental results show that significantly better overall performance is achieved with autonomous path planning, although the reduction in operator workload is not significant.

In this paper we explore the effects of service level differentiation on a multi-robot control system. We examine the premise that although long interaction time between robots and operators hurts the efficiency of the system, as it generates longer waiting time for robots, it provides robots with longer neglect time and better performance benefiting the system. In the paper we address the problem of how to choose the optimal service level for an operator in a system through a service level differentiation model. Experimental results comparing system performance for different values of system parameters show that a mixed strategy is a general way to get optimal system performance for a large variety of system parameter settings and in all cases is no worse than a pure strategy.

We are developing a theory for human control of robot teams based on considering how control varies across different task allocations. Our current work focuses on domains such as foraging in which robots perform largely independent tasks. The present study addresses the interaction between automation and organization of human teams in controlling large robot teams performing an Urban Search and Rescue (USAR) task. We identify three subtasks: perceptual search-visual search for victims, assistance-teleoperation to assist robot, and navigation-path planning and coordination. For the studies reported here, navigation was selected for automation because it involves weak dependencies among robots making it more complex and because it was shown in an earlier experiment to be the most difficult. Two possible ways to organize operators were identified as assignment of robots to particular operators or as a shared pool in which operators service robots from the population as needed. The experiment compares two member teams of operators controlling teams of 12 robots each in the assigned robots conditions or sharing control of 24 robots in the shared pool conditions using either waypoint control or autonomous path planning. We identify three self organizing team strategies in the shared pool condition: joint control operators share full authority over robots, mixed control in which one operator takes primary control while the other acts as an assistant, and split control in which operators divide the robots with each controlling a subteam. Automating path planning improved system performance. Effects of team organization favored operator teams who shared authority for the pool of robots.

The present study investigates the effect of the number of controlled robots on performance of an urban search and rescue (USAR) task using a realistic simulation. Participants controlled either 4, 8, or 12 robots. In the fulltask control condition participants both dictated the robots' paths and controlled their cameras to search for victims. In the exploration condition, participants directed the team of robots in order to explore as wide an area as possible. In the perceptual search condition, participants searched for victims by controlling cameras mounted on robots following predetermined paths selected to match characteristics of paths generated under the other two conditions. By decomposing the search and rescue task into exploration and perceptual search subtasks the experiment allows the determination of their scaling characteristics in order to provide a basis for tentative task allocations among humans and automation for controlling larger robot teams. In the fulltask control condition task performance increased in going from four to eight controlled robots but deteriorated in moving from eight to twelve. Workload increased monotonically with number of robots. Performance per robot decreased with increases in team size. Results are consistent with earlier studies suggesting a limit of between 8-12 robots for direct human control.

Camera guided teleoperation has long been the preferred mode for controlling remote robots with other modes such as asynchronous control only used when unavoidable. Because controlling multiple robots places additional demands on the operator we hypothesized that removing the forced pace for reviewing camera video might reduce workload and improve performance. In an earlier experiment participants operated four teams performing a simulated urban search and rescue (USAR) task using a conventional streaming video plus map interface or an experimental interface without streaming video but with the ability to store panoramic images on the map to be viewed at leisure. Operators were more accurate in marking victims on maps using the conventional interface; however, ancillary measures suggested that the asynchronous interface succeeded in reducing temporal demands for switching between robots. This raised the possibility that the asynchronous interface might perform better if teams were larger. In this experiment we evaluate the usefulness of asynchronous video for teams of 4, 8, or 12 robots. Operators in the two conditions were equally successful in finding victims, however, the streaming video maintained its advantage for accuracy in locating victims.

We are developing an architecture for controlling robot teams based on considering how control difficulty for different tasks grows with increases in team size. Our analysis suggests that assignments of persons to commander (single commands to entire robot team), operator (commands to individual robots), and coordinator (control of interdependent robots) roles can lead to the most efficient organization. The ability to assign tasks within or between operators makes scheduling these interactions an important factor in team performance. Two possible ways to organize operators are through Individual Assignments of robots or as a Call Center in which operators service robots from the population as needed. In recent experiments we have found that participants performing an Urban Search And Rescue (USAR) foraging task using waypoint control were at or over their limits when controlling 12 robots. The present study uses the same robots, environment, and level of autonomy but with teams of two operators assigned to control 24 robots. These operators controlled teams of 12 robots in the Individual Assignment condition. In the Call Center condition operators shared control of the 24 robots. For this task and level of robot autonomy Individual Assignment participants performed marginally better searching larger regions but without finding more victims.

We discuss an experiment involving visual path planning for multiple, remote robots in a partially visible building, with a partial 2D map available. Participants in the experiment defined waypoints for each robot to circumnavigate obstacles and explore the building. A cognitively plausible model of visual planning is evaluated using a normalized metric of the fit between model and subject itineraries. We discuss variation in the data and model fit, indicating individual differences in strategies to cope with task demands.

Camera guided teleoperation has long been the preferred mode for controlling remote robots, with other modes such as asynchronous control only used when unavoidable. In this experiment we evaluate the usefulness of asynchronous operation for a multirobot search task. Because controlling multiple robots places additional demands on the operator, removing the forced pace for reviewing camera video might reduce workload and improve performance. In the reported experiment participants operated four robot teams performing a simulated urban search and rescue (USAR) task using either conventional streaming video plus a map interface or an experimental interface without streaming video but with the ability to store panoramic images on the map to be viewed at leisure. Search performance was somewhat better using the conventional interface, however, ancillary measures suggest that the asynchronous interface succeeded in reducing temporal demands for switching between robots.

Human control of multiple robots has been characterized by the average demand of single robots on human attention. While this matches situations in which independent robots are controlled sequentially it does not capture aspects of demand associated with coordinating dependent actions among robots. This paper presents an extension of Crandall's neglect tolerance model intended to accommodate both coordination demands (CD) and heterogeneity among robots. The reported experiment attempts to manipulate coordination demand by varying the proximity needed to perform a joint task in two conditions and by automating coordination within subteams in a third. Team performance and the process measure CD were assessed for each condition. Automating cooperation reduced CD and improved performance. We discuss the utility of process measures such as CD to analyze and improve control performance.

The present study investigates the effect of the number of controlled robots on performance of an urban search and rescue (USAR) task using a realistic simulation. Task performance increased in going from four to eight controlled robots but deteriorated in moving from eight to twelve. Workload increased monotonically with number of robots. Performance per robot decreased with increases in team size. Results are consistent with earlier studies suggesting a limit of between 8-12 robots for direct human control. This study demonstrates that these findings generalize to a more realistic setting and complex task.

Human control of multiple robots has been characterized by the average demand of single robots on human attention or the distribution of demands from multiple robots. When robots are allowed to cooperate autonomously, however, demands on the operator should be reduced by the amount previously required to coordinate their actions. The present experiment compares control of small robot teams in which cooperating robots explored autonomously, were controlled independently by an operator or through mixed initiative as a cooperating team. Mixed initiative teams found more victims and searched wider areas than either fully autonomous or manually controlled teams. Operators who switched attention between robots more frequently were found to perform better in both manual and mixed initiative conditions.

This paper presents USARSim, an open source high fidelity robot simulator that can be used both for research and education. USARSim offers many characteristics that differentiate it from most existing simulators. Most notably, it constitutes the simulation engine used to run the virtual robots competition within the Robocup initiative. We describe its general architecture, describe examples of utilization, and provide a comprehensive overview for those interested in robot simulations for education, research and competitions.

Conventional models of multirobot control assume independent robots and tasks. This allows an additive model in which the operator controls robots sequentially neglecting each until its performance deteriorates sufficiently to require new operator input. This paper presents a model and experiment intended to extend the neglect tolerance model to situations in which robots must cooperate to perform dependent tasks. In the experiment operators controlled 2 robot teams to perform a box pushing task under high cooperation demand (teleoperation), moderate demand (waypoint control/heterogeneous robots), and low demand (waypoint control/homogeneous robots) conditions. Measured demand and performance were consistent with the model’s predictions.

Attitude control refers to controlling the pitch and roll of a mobile robot. As environments become more complex and cues to a robot's pose become sparser, it becomes easy for a teleoperator using an egocentric (camera) display to lose situational awareness. Reported difficulties with teleoperated robots frequently involve rollovers and sometimes even failure to realize that a robot has rolled over. Attitude information has conventionally been displayed separately from the camera view using an artificial horizon graphic or individual indicators for pitch and roll. Information from separate attitude displays may be difficult to integrate with an ongoing navigation task and may lead to errors. In this paper, we report an experiment in which a simulated robot is maneuvered over rough exterior and interior terrains to compare a gravity-referenced view with a separated attitude indication. Results show shorter task times and better path choices for users of the gravity-referenced view but no reduction in rollovers

This paper describes an effort to identify common metrics for task-oriented human-robot interaction (HRI). We begin by discussing the need for a toolkit of HRI metrics. We then describe the framework of our work and identify important biasing factors that must be taken into consideration. Finally, we present suggested common metrics for standardization and a case study. Preparation of a larger, more detailed toolkit is in progress.

Collaborative control systems have shown great promise in offloading tasks to an automated peer, allowing the human operators to focus their efforts on perceptual judgments and decision-making that exceed the current capability of automation. This paper initiates a discussion about exposing camera orientation to the collaborative control interaction model. It may be the case that the robot can work with the human operator to automatically direct attention to critical features in an environment by systematically influencing the orientation of the camera. The results of a user evaluation suggest that this kind of interface has potential, but there are several additional interaction constraints that will need to be considered in a successful design.

Human judgment is an integral part of the teleoperation process that is often heavily influenced by a single video feed returned from the remote environment. Poor camera placement, narrow field of view, and other camera properties can significantly impair the operator's perceptual link to the environment, inviting cognitive mistakes and general disorientation. These faults may be enhanced or muted, depending on the camera mountings and control opportunities that are at the disposal of the operator. These issues form the basis for two user studies that assess the effectiveness of existing and potential teleoperation controls. Findings suggest that providing a camera that is controlled independently from the orientation of the vehicle may yield significant benefits. Moreover, there is evidence to support the use of separate cameras for different navigational subtasks. Third, the use of multiple cameras can also be used to provide assistance without encroaching on the operator's desired threshold for control.

A video stream from a single camera is often the foundation for situational awareness in teleoperation activities. Poor camera placement, narrow field-of-view and other camera properties can significantly impair the operator's perceptual link to the environment, inviting cognitive mistakes and general disorientation. This paper provides a brief overview of viewpoint control research for 3D virtual environments (VE) to motivate a user study that evaluates the effectiveness of viewpoint controls on a simulated robotic vehicle. Findings suggest that providing a camera that is controlled independently from the orientation of the vehicle may facilitate wayfinding tasks. Moreover, there is evidence to support the use of separate cameras and interfaces for different navigational subtasks.

This present study examined the effects of animated characters and presenting voices on comprehension and attention performance in learning from a multimedia presentation. This study also investigated the effect of introversion/extroversion in learning performance and rating affected by animated character. Comprehension was affected by neither animated characters nor presenting voices. Results show that an animated character may retain learner's attention. The presence of animated character did not increase positive attitude of participants toward the multimedia presentation.

Human judgment is an integral part of the teleoperation process that is often heavily influenced by a single video feed returned from the remote environment. This limitation on the perceptual links to the environment leaves the operator prone to cognitive mistakes and general disorientation. These faults may be enhanced or muted, depending on the camera mountings and control opportunities that are at the disposal of the operator. These issues form the basis for an experiment to assess the effectiveness of existing and potential teleoperation controls. Findings suggest that providing a camera that is controlled independently from the orientation of the vehicle may yield significant benefits.

Attitude control refers to controlling the pitch and roll of a mobile robot. As environments grow more complex and cues to a robot's pose sparser it becomes easy for a teleoperator to lose situational awareness. Information from separated attitude displays may be difficult to integrate with an ongoing navigation task and lead to errors. In this paper we report an experiment comparing a gravity referenced display (GRV) with a standard fixed camera with separated attitude Indication. Results show shorter task times and better path choices for users of the GRV.

Rapid advances in consumer electronics have led to the anomaly that consumer off the shelf (COTS) gaming hardware and software now provide better interactive graphics than military and other specialized systems costing orders of magnitude more. UTSAF is bridging software written to take advantage of the power of gaming systems by allowing them to participate in distributed simulations with military simulators such as OneSAF which use the DIS protocol. UTSAF parses DIS PDUs and uses the information gained to control entities within the Unreal game engine using the Game-Bots modification. This paper describes the advantages of game engine based simulation and the UTSAF bridging and control architecture. Our main contribution is to build a simulation bridge that enables affordable high-quality 3-D viewers for military simulations.

We are developing interactive simulations of the National Institute of Standards and Technology (NIST) Reference Test Facility for Autonomous Mobile Robots (Urban Search and Rescue). The NIST USAR Test Facility is a standardized disaster environment consisting of three scenarios of progressive difficulty: Yellow, Orange, and Red arenas. The USAR task focuses on robot behaviors, and physical interaction with standardized but disorderly rubble filled environments. The simulation will be used to test and evaluate designs for teleoperation interfaces and robot sensing and cooperation that will subsequently be incorporated into experimental robots. This paper describes our novel simulation approach using an inexpensive game engine to rapidly construct a visually and dynamically accurate simulation for both individual robots and robot teams.

Abstract Although many different visual information retrieval systems have been proposed, few have been tested, and where testing has been performed, results were often inconclusive. Further, there is very little evidence of benchmarking systems against a common standard. An approach for testing novel interfaces is proposed that uses bottom‐up, stepwise testing to allow evaluation of a visualization, itself, rather than restricting evaluation to the system instantiating it. This approach not only makes it easier to control variables, but the tests are also easier to perform. The methodology will be presented through a case study, where a new visualization technique is compared to more traditional ways of presenting data.

Gaining an accurate mental representation of real environments and realistic virtual environments is a gradual process. Significant aspects of an environment may be obvious to a trained expert, but not to the novice trainee. If a user does not know where to look, he or she may concentrate on irrelevant objects. This detracts from learning the locations of truly prominent landmarks. The paper explores attentive camera navigation, a technique that guides the user to focus on certain objects through automatic gaze redirection. Results of a user study suggest that this technique can help filter out unnecessary objects, and allow users to quickly understand the configuration of a selected subset of landmarks.

Describes the interactions between task characteristics and human agent interfaces in a team rendezvous route-planning task. The agents include an interface agent and two different task agents that perform similar tasks. The MokSAF (Mock modular Semi-Automated Forces) interface agent links an artificial intelligence (AI) route planning agent to a geographic information system (GIS). Through this agent, the user specifies a start and an end point, and describes the composition and characteristics of a military platoon. Two aided conditions and one non-aided condition were examined. In the first aided condition, an autonomous route-planning agent (RPA) determines a minimum-cost path between the specified end points. The user is allowed to define additional "intangible" constraints that describe situational or social information that should be considered when determining the route. In the second aided condition, a different agent, a cooperative RPA, uses the same knowledge of the terrain and cost functions available to the autonomous RPA, but restricts its search to paths within regions drawn by the user. In the unaided condition (the naive RPA), the user draws the route manually, then submits it to be tested against the terrain and cost functions for feasibility. Both aided conditions are superior to the control but differ in their relative effectiveness by scenario. In this paper, we examine the varieties of challenges faced by commanders in two scenarios and relate them to the differential effectiveness of the agents.

HTML pages are designed to convey semantic information to human users through visual emphases, demarcations, spatial cues and repeating patterns which act as "perceptual markup". This human-centric syntax is not easy for machines to identify. Naturally-occurring HTML, especially the machine-generated variety, rarely follows strict markup rules and provides no semantic cues. The visual cues humans use to extract information from a Web page, however, must be reflected in the page's markup. If a human could convey the relationship between visual cues, available to the program as markup patterns, and semantic categories, passed to the program as user-supplied labels, the program would have been instructed in "how to extract information from that page". Scriptor is a program which, run in tandem with a Web browser, allows a user to interactively design a data extraction script for the Web site. It is intended for highly structured repetitive information such as is found in classified listings, online stores, tables for weather, stock or airline schedules, course listings, and other similar sources. Scriptor interleaves a variety of learning methods to allow the specification of extraction rules using extremely simple methods. These consist of repeating pattern recognition, supervised learning, deictics through highlighting, and dialogs in which the user selects the desired result for a set of possible extraction rules. Learning is augmented by direct instructions such as: "label text following '/spl sim/' as 'Author' ". Performance data for the authors and naive subjects are presented for a collection of Web pages showing the potential of this form of highly interactive instruction. Our results demonstrate that very simple programming by example techniques can generate effective parse rules in highly repetitive domains.

Our research program is investigating how to effectively incorporate intelligent agents into human teams, specifically to support team performance. There are several ways to incorporate agents into teams; our approach is to use agents to support the team as a whole (facilitating communication, allocation of tasks, coordination among the human agents, and improving attention focus). In our initial experiments, we investigated the effects of trust and calibration of trust that allows human decision-makers to assess the reliability and meaning of communications from software and human agents. Currently, we are focusing on the team and the different ways of deploying agents to support multiperson teams: 1) supporting the individual by tracking the collected information; 2) supporting communication among team members by automatically passing information to the relevant person; and 3) supporting task prioritization and coordination by providing a shared checklist. Only performance data have been reported for this paper. Aiding with agents was found to be different from the control condition. Learning effects and effects for the target difficulty were found.

Information retrieval has a long history of dealing with printed materials. More recent work has involved the development of experimental visual interfaces to support users' attempts to access appropriate documents. This research has matured to the point that usability studies and evaluation of approaches to information visualization are needed to guide further development. The reported studies examine the use of alternative document visualizations in tightly controlled settings. Five types of interface representations were defined, including ordered text, ordered icons, a table format, a x-y graph format and a novel spring based visualization. To assess the relative utility of the various interfaces, we have chosen to apply two kinds of measures: performance on information retrieval tasks and user preference rankings of the interfaces. The results show that performance is strikingly different across the range of interface types with the ordered icon list and text list producing the best results. Users' preferences however, indicated that the textual format was the least desirable, while both of the visualization methods, i.e., icon list and spring based visual, were preferred. We conclude that performance is more easily and accurately measured and that preferences of users can not be used alone to determine the utility of interfaces.