While much research exists on resolving spatial natural language references to known locations, little work deals with handling references to unknown locations. In this paper we introduce and evaluate algorithms integrated into a cognitive architecture which allow an agent to learn about its environ-ment while resolving references to both known and unknown locations. We also describe how multiple components in the architecture jointly facilitate these capabilities.

Introspection mechanisms are employed in agent architectures toimprove agent performance. However, there is currently no approach tointrospection that makes automatic adjustments at multiple levels inthe implemented agent system. We introduce our novel multi-levelintrospection framework that can be used to automatically adjustarchitectural configurations based on the introspection results at theagent, infrastructure and component level. We demonstrate the utilityof such adjustments in a concrete implementation on a robot where thehigh-level goal of the robot is used to automatically configure thevision system in a way that minimizes resource consumption whileimproving overall task performance.

This paper describes the Brainput system, which learns to identify brain activity patterns occurring during multitasking. It provides a continuous, supplemental input stream to an interactive human-robot system, which uses this information to modify its behavior to better support multitasking. This paper demonstrates that we can use non-invasive methods to detect signals coming from the brain that users naturally and effortlessly generate while using a computer system. If used with care, this additional information can lead to systems that respond appropriately to changes in the user's state. Our experimental study shows that Brainput significantly improves several performance metrics, as well as the subjective NASA-Task Load Index scores in a dual-task human-robot activity.

Robots are currently being used in and developed for critical HRI applications such as search and rescue. In these scenarios, humans operating under changeable and high-stress conditions must communicate effectively with autonomous agents, necessitating that such agents be able to respond quickly and effectively to rapidly-changing conditions and expectations. We demonstrate a robot planner that is able to utilize new information, specifically information originating in spoken input produced by human operators.

Hybrid reactive-deliberative architectures in robotics combine reactive sub-policies for fast action execution with goal sequencing and deliberation. The need for replanning, however, presents a challenge for reactivity and hinders the potential for guarantees about the plan quality. In this paper, we argue that one can integrate abstract planning provided by symbolic dynamic programming in first order logic into a reactive robotic architecture, and that such an integration is in fact natural and has advantages over traditional approaches. In particular, it allows the integrated system to spend off-line time planning for a policy, and then use the policy reactively in open worlds, in situations with unexpected outcomes, and even in new environments, all by simply reacting to a state change executing a new action proposed by the policy. We demonstrate the viability of the approach by integrating the FODD-Planner with the robotic DIARC architecture showing how an appropriate interface can be defined and that this integration can yield robust goal-based action execution on robots in open worlds.

Efficient collaborations between interacting agents, be they humans, virtual or embodied agents, require mutual recognition of the goal, appropriate sequencing and coordination of each agent's behavior with others, and making predictions from and about the likely behavior of others. Moment-by-moment eye gaze plays an important role in such interaction and collaboration. In light of this, we used a novel experimental paradigm to systematically investigate gaze patterns in both human-human and human-agent interactions. Participants in the study were asked to interact with either another human or an embodied agent in a joint attention task. Fine-grained multimodal behavioral data were recorded including eye movement data, speech, first-person view video, which were then analyzed to discover various behavioral patterns. Those patterns show that human participants are highly sensitive to momentary multimodal behaviors generated by the social partner (either another human or an artificial agent) and they rapidly adapt their gaze behaviors accordingly. Our results from this data-driven approach provide new findings for understanding micro-behaviors in human-human communication which will be critical for the design of artificial agents that can generate human-like gaze behaviors and engage in multimodal interactions with humans.

Multitasking has become an integral part of work environments, even though people are not well-equipped cognitively to handle numerous concurrent tasks effectively. Systems that support such multitasking may produce better performance and less frustration. However, without understanding the user's internal processes, it is difficult to determine optimal strategies for adapting interfaces, since all multitasking activity is not identical. We describe two experiments leading toward a system that detects cognitive multitasking processes and uses this information as input to an adaptive interface. Using functional near-infrared spectroscopy sensors, we differentiate four cognitive multitasking processes. These states cannot readily be distinguished using behavioral measures such as response time, accuracy, keystrokes or screen contents. We then present our human-robot system as a proof-of-concept that uses real-time cognitive state information as input and adapts in response. This prototype system serves as a platform to study interfaces that enable better task switching, interruption management, and multitasking.

Natural language interactions between humans and robots are currently limited by many factors, most notably by the robot's concept representations and action repertoires. We propose a novel algorithm for learning meanings of action verbs through dialogue-based natural language descriptions. This functionality is deeply integrated in the robot's natural language subsystem and allows it to perform the actions associated with the learned verb meanings right away without any additional help or learning trials. We demonstrate the effectiveness of the algorithm in a scenario where a human explains to a robot the meaning of an action verb unknown to the robot and the robot is subsequently able to carry out the instructions involving this verb.

In this paper, we present an integrated planning and robotic architecture that actively directs an agent engaged in an urban search and rescue (USAR) scenario. We describe three salient features that comprise the planning component of this system, namely (1) the ability to plan in a world open with respect to objects, (2) execution monitoring and replanning abilities, and (3) handling soft goals, and detail the interaction of these parts in representing and solving the USAR scenario at hand. We show that though insufficient in an individual capacity, the integration of this trio of features is sufficient to solve the scenario that we present. We test our system with an example problem that involves soft and hard goals, as well as goal deadlines and action costs, and show that the planner is capable of incorporating sensing actions and execution monitoring in order to produce goal-fulfilling plans that maximize the net benefit accrued.

Natural human-robot interaction requires different and more robust models of language understanding (NLU) than non-embodied NLU systems. In particular, architectures are required that (1) process language incrementally in order to be able to provide early backchannel feedback to human speakers; (2) use pragmatic contexts throughout the understanding process to infer missing information; and (3) handle the underspecified, fragmentary, or otherwise ungrammatical utterances that are common in spontaneous speech. In this paper, we describe our attempts at developing an integrated natural language understanding architecture for HRI, and demonstrate its novel capabilities using challenging data collected in human-human interaction experiments.

Complex robots with many interacting components in their control architectures are subject to component failures from which neither the control architecture nor the implementing infrastructure can recover. Moreover, the operating conditions for these components might be at odds with goals the robot might have adopted (e.g., through external commands or in the course of the execution of the current task). We argue that the best (if not the only) way to resolve any difficulties that arise from the different requirements at the agent, component and infrastructure levels is to use a common formal logical goal representation for all three layers. We discuss how these representations can be integrated into a complex robotic architecture and demonstrate in an experimental evaluation on a robot how the architecture can recover from a failure situation that it would not have been able to handle without explicit multi-level unified goal representations and their associated monitoring and reasoning processes.

As the number of applications for human-robot teaming continue to rise, there is an increasing need for planning technologies that can guide robots in such teaming scenarios. In this article, we focus on adapting planning technology to Urban Search And Rescue (USAR) with a human-robot team. We start by showing that several aspects of state-of-the-art planning technology, including temporal planning, partial satisfaction planning, and replanning, can be gainfully adapted to this scenario. We then note that human-robot teaming also throws up an additional critical challenge, namely, enabling existing planners, which work under closed-world assumptions, to cope with the open worlds that are characteristic of teaming problems such as USAR. In response, we discuss the notion of conditional goals, and describe how we represent and handle a specific class of them called open world quantified goals. Finally, we describe how the planner, and its open world extensions, are integrated into a robot control architecture, and provide an empirical evaluation over USAR experimental runs to establish the effectiveness of the planning components.

Robot autonomy is of high relevance for HRI, in particular for interactions of humans and robots in mixed human-robot teams. In this paper, we investigate empirically the extent to which autonomy based on independent decision making and acting by the robot can affect the objective task performance of a mixed human-robot team while being subjectively acceptable to humans. The results demonstrate that humans not only accept robot autonomy in the interest of the team, but also view the robot more as a team member and find it easier to interact with, despite a very minimalist graphical/speech interface. Moreover, we find evidence that dynamic autonomy reduces human cognitive load.

Robots that can be given instructions in spoken language need to be able to parse a natural language utterance quickly, determine its meaning, generate a goal representation from it, check whether the new goal conflicts with existing goals, and if acceptable, produce an action sequence to achieve the new goal (ideally being sensitive to the existing goals). In this paper, we describe an integrated robotic architecture that can achieve the above steps by translating natural language instructions incrementally and simultaneously into formal logical goal description and action languages, which can be used both to reason about the achievability of a goal as well as to generate new action scripts to pursue the goal. We demonstrate the implementation of our approach on a robot taking spoken natural language instructions in an office environment.

Many hive-based agents, such as some bees, rely on memory and communication to aid in foraging. The benefits of these abilities seem obvious, but it is unlikely that they are beneficial in every environment. In this paper, we present the results of experiments examining the effect that environmental structure can have on the utility of communication and memory for hive-based agents, finding that there are some environments in which they do not contribute substantially to the agents' ability to survive.

There is recent evidence that males and females view robots differently, from the way robots are conceptualized, to the way humans respond when they interact with them. In this paper, we further explore gender-based differences in human-robot interaction. Moreover, we provide the first available evidence for sex-related differences in reactions to gendered synthetic voices that are either disembodied or physically embodied within a robot. Results indicate that physical embodiment and perceived entity gender may interact with human sex-related characteristics and pre-experimental attitudes in determining how people respond to artificial entities.

Autonomous robots that operate in real-world domains face multiple challenges that make planning and goal selection difficult. Not only must planning and execution occur in real time, newly acquired knowledge can invalidate previous plans, and goals and their utilities can change during plan execution. However, these events can also provide opportunities, if the architecture is designed to react appropriately. We present here an architecture that integrates the SapaReplan planner with the DIARC robot architecture, allowing the architecture to react dynamically to changes in the robot's goal structures.

Social-psychological processes in humans will play an important role in long-term human-robot interactions. This study investigates people's perceptions of social presence in robots during (relatively) short interactions. Findings indicate that males tend to think of the robot as more human-like and accordingly show some evidence of "social facilitation" on an arithmetic task as well as more socially desirable responding on a survey administered by a robot. In contrast, females saw the robot as more machine-like, exhibited less socially desirable responding to the robot's survey, and were not socially facilitated by the robot while engaged in the arithmetic tasks. Various alternative accounts of these findings are explored and the implications of these results for future work are discussed.

The performance of embodied multi-agent systems depends, in addition to the agent architectures of the employed agents, on their physical characteristics (e.g., sensory range, speed, etc.) and group properties (e.g., number of agents, types of agents, etc.). Consequently, it is difficult to evaluate the performance of a multi-agent system based on the performance of an agent architecture alone, even in homogeneous teams. In this paper, we propose a method for analyzing the performance of multi-agent systems based on the notion of "performance-cost-tradeoff," which attempts to determine the relations among different cost-dimensions by performing a performance sampling of these dimensions and comparing them relative to their associated costs. Specifically, we investigate the performance-cost tradeoffs of four candidate architectures for a multi-agent territory exploration task in which a group of agents is required to visit a set of checkpoints randomly placed in an environment in the shortest time possible. Performance tradeoffs between three dimensions (sensory range, group size, and prediction) are then used to illustrate the cost-benefit analyses performed to determine the best agent configurations for different practical settings.

We describe the tight integration of incremental natural language understanding, goal management, and action processing in a complex robotic architecture, which is required for natural interactions between robots and humans. Specifically, the natural language components need to process utterances while they are still spoken to be able to initiate feedback actions in a timely fashion, while the action manager might need information at various points during action execution that must be obtained from humans. We argue that a finer- grained integration provides much more natural human-robot interactions and much more reasonable multitasking.

Complex, autonomous robots integrate a large set of sometimes very diverse algorithms across at least three levels of system organization: the agent architecture, the implementation environment, and the hardware devices. Insofar as a distinction is maintained between them, the levels serve different purposes and thus exhibit different characteristic strengths and weaknesses. Exchanging information among organizational levels can be used to mitigate the shortcomings of one level by making use of the strengths of another. In this paper, we highlight the roles, characteristics, and relations between the infrastructure and the architecture of complex robots, describing a novel form of integration that results from enabling the exchange of information between these two levels, which otherwise is maintained internally. The information from the infrastructure is especially amenable for use by the architecture to achieve a higher level of robustness and system awareness. We demonstrate the functionality and utility of the proposed mechanisms in a set of experiments in which failures of architectural components are induced on an actual robot engaged in a joint human-robot team task.

Recognizing and responding to human affect is important in collaborative tasks in joint human-robot teams. In this paper we present an integrated affect and cognition architecture for HRI and report results from an experiment with this architecture that shows that expressing affect and responding to human affect with affect expressions can significantly improve team performance in a joint human-robot task.

Evolutionary investigations are often very expensive in terms of the required computational resources and many general questions regarding the utility of a feature F of an agent (e.g., in competitive environments) or the likelihood of F evolving (or not evolving) are therefore typically difficult, if not practically impossible to answer. We propose and demonstrate in extensive simulations a methodology that allows us to answer such questions in setups where good predictors of performance in a task T are available. These predictors evaluate the performance of an agent kind A in a task T*, which can then transformed by including costs and additional factors to make predictions about the performance of A in T.

This paper is an examination of communication in hive-based swarms in the biological setting, focusing on the effect environmental factors have on the utility of communication. Our investigation utilizes a generational survival task to measure the benefit of communication in a biological setting. Swarm members forage for food, consuming energy in the process, and returning to a central "hive" to contribute any surplus. The resources of the hive determine when reproduction is possible, so it is in the best interest of the population to maximize the efficiency of foraging. The measure of performance is the size of the swarm surviving at the end of a simulation run. Each simulation begins with a swarm of fixed size (5 agents), making contributions to the hive (and subsequent procreation) necessary for good performance.

We present a physically realizable UAV model for locating and tracking chemical clouds. Simulation results are presented for implementations of this model with two configurations, one that is faster and requires more space to avoid collisions, and one that is slower and can cover an area more densely. Heterogeneous swarms of agents are shown to have better performance than homogeneous swarms of similar size because they take advantage of the strengths of each configuration.