Embodied agents, such as robots, will need to interact in situated environments where successful communication often depends on reasoning over social norms: shared expectations that constrain what actions are appropriate in context. A key capability in such settings is norm-based reference resolution (NBRR), where interpreting referential expressions requires inferring implicit normative expectations grounded in physical and social context. Yet it remains unclear whether Large Language Models (LLMs) can support this kind of reasoning. In this work, we introduce SNIC (Situated Norms in Context), a human-validated diagnostic testbed designed to probe how well state-of-the-art LLMs can extract and utilize normative principles relevant to NBRR. SNIC emphasizes physically grounded norms that arise in everyday tasks such as cleaning, tidying, and serving. Across a range of controlled evaluations, we find that even the strongest LLMs struggle to consistently identify and apply social norms—particularly when norms are implicit, underspecified, or in conflict. These findings reveal a blind spot in current LLMs and highlight a key challenge for deploying language-based systems in socially situated, embodied settings.

We present IntelliProof, an interactive system for analyzing argumentative essays through LLMs. IntelliProof structures an essay as an argumentation graph, where claims are represented as nodes, supporting evidence is attached as node properties, and edges encode supporting or attacking relations. Unlike existing automated essay scoring systems, IntelliProof emphasizes the user experience: each relation is initially classified and scored by an LLM, then visualized for enhanced understanding. The system provides justifications for classifications and produces quantitative measures for essay coherence. It enables rapid exploration of argumentative quality while retaining human oversight. In addition, IntelliProof provides a set of tools for a better understanding of an argumentative essay and its corresponding graph in natural language, bridging the gap between the structural semantics of argumentative essays and the user's understanding of a given text.

Effective human teams excel at maintaining a consistent shared mental model (SMM) that reflects the shared understanding of individual team members about the task and what remains to be done. We present a novel, two-step framework that leverages large language models (LLMs) both as (1) annotators of team dialogues to track the team’s SMM and (2) as automated discrepancy detectors among individuals’ mental states as they are represented in individual mental models. We define an SMM coherence evaluation framework for this use case and apply it to six dialogues in a previously published team corpus, ultimately producing a dataset of human and LLM SMM annotations, a reproducible evaluation framework for SMM coherence, and an empirical assessment of LLM-based discrepancy detection. Our results reveal that while LLMs exhibit apparent coherence on straightforward natural-language annotation tasks, they systematically err in scenarios requiring spatial reasoning or disambiguation of transcription-level disfluencies.

LLMs are being increasingly integrated into embodied robotic systems. A useful capability that the LLMs bring to robots is translating noisy spoken human natural language instructions into executable robot actions. However, these integrations are somewhat ad hoc and understudied as they tend to not consider the gamut of syntactic, semantic, as well as pragmatic aspects of embodied human communication. What is missing is a characterization of the different paradigms for integrating LLMs into robotic architectures as well as a set of evaluation metrics that capture whether an LLM-equipped robot can correctly understand these different aspects of human instruction. In this article, we present a suite of evaluation metrics together with data augmentation techniques for evaluating these architectures, using concepts from the cognitive science and human communication literature. To illustrate an application of these metrics and augmentation techniques, we conduct experiments to compare two integration methods: LLMs as pre-processing components that map human instructions into more constrained versions to be processed by the architecture’s natural language understanding (NLU) subsystem, or LLMs as a wholesale replacement for the NLU’s parser. We provide experimental evaluations and a robotic implementation to show the inherent tradeoffs between the methods. Our results suggest that while they offer increased explainability, traditional parsing tools coupled with LLMs do not perform as well as an LLM that replaces a parser entirely. The proposed evaluation metrics together with the characterization of different LLM integration approaches offer the promise of systematically evaluating LLMs as natural language interfaces to robotic systems as well as tackle the important tradeoff between explainability/verifiability/interpretability and robustness to noisy input and broad language understanding in an open-world embodied setting.

Learning to manipulate objects efficiently, particularly those involving sustained contact (e.g., pushing, sliding) and articulated parts (e.g., drawers, doors), presents significant challenges. Traditional methods, such as robot-centric reinforce-ment learning (RL), imitation learning, and hybrid techniques, require massive training and often struggle to generalize across different objects and robot platforms. We propose a novel framework for learning object-centric manipulation policies in force space, decoupling the robot from the object. By directly applying forces to selected regions of the object, our method simplifies the action space, reduces unnecessary exploration, and decreases simulation overhead. This approach, trained in simulation on a small set of representative objects, captures ob-ject dynamics—such as joint configurations—allowing policies to generalize effectively to new, unseen objects. Decoupling these policies from robot-specific dynamics enables direct transfer to different robotic platforms (e.g., Kinova, Panda, URS) with-out retraining. Our evaluations demonstrate that the method significantly outperforms baselines, achieving over an order of magnitude improvement in training efficiency compared to other state-of-the-art methods. Additionally, operating in force space enhances policy transferability across diverse robot plat-forms and object types. We further showcase the applicability of our method in a real-world robotic setting. Link: https://tufts-ai-robotics-group.github.io/FLEX/

Adapting quickly to dynamic, uncertain environments—often called “open worlds” —remains a major challenge in robotics. Traditional Task and Motion Planning (TAMP) approaches struggle to cope with unforeseen changes, are data-inefficient when adapting, and do not leverage world models during learning. We address this issue with a hybrid planning and learning system that integrates two models: a low-level neural network-based model that learns stochastic transitions and drives exploration via an Intrinsic Curiosity Module (ICM), and a high-level symbolic planning model that captures abstract transitions using operators, enabling the agent to plan in an “imaginary” space and generate reward machines. Our evaluation in a robotic manipulation domain with sequential novelty injections demonstrates that our approach converges faster and outperforms state-of-the-art hybrid methods.

Human-robot interaction requires robots to process language incrementally, adapting their actions in real-time based on evolving speech input. Existing approaches to language-guided robot motion planning typically assume fully specified instructions, resulting in inefficient stop-and-replan behavior when corrections or clarifications occur. In this paper, we introduce a novel reasoning-based incremental parser which integrates an online motion planning algorithm within the cognitive architecture. Our approach enables continuous adaptation to dynamic linguistic input, allowing robots to update motion plans without restarting execution. The incremental parser maintains multiple candidate parses, leveraging reasoning mechanisms to resolve ambiguities and revise interpretations when needed. By combining symbolic reasoning with online motion planning, our system achieves greater flexibility in handling speech corrections and dynamically changing constraints. We evaluate our framework in real-world human-robot interaction scenarios, demonstrating online adaptions of goal poses, constraints, or task objectives. Our results highlight the advantages of integrating incremental language understanding with real-time motion planning for natural and fluid human-robot collaboration. The experiments are demonstrated in the accompanying video at www.acin.tuwien.ac.at/42d5.

Practical and ethical dataset collection remains a challenge blocking many empirical methods in natural language processing, resulting in a lack of benchmarks or data on which to test hypotheses. We propose a solution to some of these areas by presenting a pipeline to reduce the research burden of producing image and text datasets when datasets may not exist. Our approach, with accompanying software tools, involves (1) generating text with LLMs; (2) creating accompanying image vignettes with text–to–image transformers; and (3) low-cost human validation. Based on existing literature that has struggled with quantitative evaluation (due to difficulty of data collection), we present the creation of 3 relevant datasets, and conduct a user study that demonstrates this approach is able to aid researchers in obtaining previously-challenging datasets. We provide sample data generated with this technique, the source code used to produce it, and discuss applicability and limitations.

Open-world robotic tasks such as autonomous driving pose significant challenges to robot control due to unknown and unpredictable events that disrupt task performance. Neural network-based reinforcement learning (RL) techniques (like DQN, PPO, SAC, etc.) struggle to adapt in large domains and suffer from catastrophic forgetting. Hybrid planning and RL approaches have shown some promise in handling environmental changes but lack efficiency in accommodation speed. To address this limitation, we propose an enhanced hybrid system with a nested hierarchical action abstraction that can utilize previously acquired skills to effectively tackle unexpected novelties. We show that it can adapt faster and generalize better compared to state-of-the-art RL and hybrid approaches, significantly improving robustness when multiple environmental changes occur at the same time.

In dynamic open-world environments, agents continually face new challenges due to sudden and unpredictable novelties, hindering Task and Motion Planning (TAMP) in autonomous systems. We introduce a novel TAMP architecture that integrates symbolic planning with reinforcement learning to enable autonomous adaptation in such environments, operating without human guidance. Our approach employs symbolic goal representation within a goal-oriented learning framework, coupled with planner-guided goal identification, effectively managing abrupt changes where traditional reinforcement learning, re-planning, and hybrid methods fall short. Through sequential novelty injections in our experiments, we assess our method’s adaptability to continual learning scenarios. Extensive simulations conducted in a robotics domain corroborate the superiority of our approach, demonstrating faster convergence to higher performance compared to traditional methods. The success of our framework in navigating diverse novelty scenarios within a continuous domain underscores its potential for critical real-world applications.

Reinforcement learning techniques are widely used when robots have to learn new tasks but they typically operate on action spaces defined by the joints of the robot. We present a contrasting approach where actions spaces are the trajectories of objects in the environment, requiring robots to discover events such as object changes and behaviors that must occur to accomplish the task. We show that this allows robots to learn faster, to learn semantic representations that can be communicated to humans, and to learn in a manner that does not depend on the robot itself, enabling low-cost policy transfer between different types of robots. Our demonstrations can be replicated using provided source code1.

This paper presents a novel semantics for the mA* epistemic action language that takes into consideration dynamic per-agent observability of events. Different from the original mA* semantics, the observability of events is defined locally at the level of possible worlds, giving a new method for compiling event models. Locally defined observability represents agents' uncertainty and false-beliefs about each others' ability to observe events. This allows for modeling second-order false-belief tasks where one agent does not know the truth about another agent's observations and resultant beliefs. The paper presents detailed constructions of event models for ontic, sensing, and truthful announcement action occurrences and proves various properties relating to agents' beliefs after the execution of an action. It also shows that the proposed approach can model second order false-belief tasks and satisfies the robustness and faithfulness criteria discussed by Bolander (2018, https://doi.org/10. 1007/978-3-319-62864-6_8).

Natural language instructions are effective at tasking autonomous robots and for teaching them new knowledge quickly. Yet, human instructors are not perfect and are likely to make mistakes at times and will correct themselves when they notice errors in their own instructions. In this article, we introduce a complete system for robot behaviors to handle such corrections, during both task instruction and action execution. We then demonstrate its operation in an integrated cognitive robotic architecture through spoken language in two tasks: a navigation and retrieval task and a meal assembly task. Verbal corrections occur before, during, and after verbally taught sequences of tasks, demonstrating that the proposed methods enable fast corrections not only of the semantics generated from the instructions but also of overt robot behavior in a manner shown to be reasonable when compared to human behavior and expectations.

Domain generalization aims to learn a model with good generalization ability, that is, the learned model should not only perform well on several seen domains but also on unseen domains with different data distributions. State-of-the-art domain generalization methods typically train a representation function followed by a classifier jointly to minimize both the classification risk and the domain discrepancy. However, when it comes to model selection, most of these methods rely on traditional validation routines that select models solely based on the lowest classification risk on the validation set. In this paper, we theoretically demonstrate a trade-off between minimizing classification risk and mitigating domain discrepancy, i.e., it is impossible to achieve the minimum of these two objectives simultaneously. Motivated by this theoretical result, we propose a novel model selection method suggesting that the validation process should account for both the classification risk and the domain discrepancy. We validate the effectiveness of the proposed method by numerical results on several domain generalization datasets.

Probabilistic "if A then B" rules are typically formalized as Bayesian conditionals P(B|A), as many (e.g., Pearl) have argued that Bayesian conditionals are the correct way to think about such rules. However, there are challenges with standard inferences such as modus ponens and modus tollens that might make probabilistic material implication a better candidate at times for rule-based systems employing forward-chaining; and arguably material implication is still suitable when information about prior or conditional probabilities is not available at all. We investigate a generalization of probabilistic material implication and Bayesian conditionals that combines the advantages of both formalisms in a systematic way and prove basic properties of the generalized rule, in particular, for inference chains in graphs.

We present a novel integration between a computational framework for modeling attention-driven perception and cognition (ARCADIA) with a cognitive robotic architecture (DIARC), demonstrating how this integration can be used to drive the gaze behavior of a robotic platform. Although some previous approaches to controlling gaze behavior in robots during human-robot interactions have relied either on models of human visual attention or human cognition, ARCADIA provides a novel framework with an attentional mechanism that bridges both lower-level visual and higher-level cognitive processes. We demonstrate how this approach can produce more natural and human-like robot gaze behavior. In particular, we focus on how our approach can control gaze during an interactive object learning task. We present results from a pilot crowdsourced evaluation that investigates whether the gaze behavior produced during this task increases confidence that the robot has correctly learned each object.

Assessing the cognitive workload of human interactants in mixed-initiative teams is a critical capability for autonomous interactive systems to enable adaptations that improve team performance. Yet, it is still unclear, due to diverging evidence, which sensing modality might work best for the determination of human workload. In this paper, we report results from an empirical study that was designed to answer this question by collecting eye gaze and electroencephalogram (EEG) data from human subjects performing an interactive multi-modal driving task. Different levels of cognitive workload were generated by introducing secondary tasks like dialogue, braking events, and tactile stimulation in the course of driving. Our results show that pupil diameter is a more reliable indicator for workload prediction than EEG. And more importantly, none of the five different machine learning models combining the extracted EEG and pupil diameter features were able to show any improvement in workload classification over eye gaze alone, suggesting that eye gaze is a sufficient modality for assessing human cognitive workload in interactive, multi-modal, multi-task settings.

Domain generalization (DG) is a branch of transfer learning that aims to train the learning models on several seen domains and subsequently apply these pre-trained models to other unseen (unknown but related) domains. To deal with challenging settings in DG where both data and label of the unseen domain are not available at training time, the most common approach is to design the classifiers based on the domain-invariant representation features, i.e., the latent representations that are unchanged and transferable between domains. Contrary to popular belief, we show that designing classifiers based on invariant representation features alone is necessary but insufficient in DG. Our analysis indicates the necessity of imposing a constraint on the reconstruction loss induced by representation functions to preserve most of the relevant information about the label in the latent space. More importantly, we point out the trade-off between minimizing the reconstruction loss and achieving domain alignment in DG. Our theoretical results motivate a new DG framework that jointly optimizes the reconstruction loss and the domain discrepancy. Both theoretical and numerical results are provided to justify our approach.

Invariance-principle-based methods such as Invariant Risk Minimization (IRM), have recently emerged as promising approaches for Domain Generalization (DG). Despite promising theory, such approaches fail in common classification tasks due to mixing of true invariant features and spurious invariant features1. To address this, we propose a framework based on the conditional entropy minimization (CEM) principle to filter-out the spurious invariant features leading to a new algorithm with a better generalization capability. We show that our proposed approach is closely related to the well-known Information Bottleneck (IB) framework and prove that under certain assumptions, entropy minimization can exactly recover the true invariant features. Our approach provides competitive classification accuracy compared to recent theoretically-principled state-of-the-art alternatives across several DG datasets.

Humans use natural language, vision, and context to resolve referents in their environment.While some situated reference resolution is trivial, ambiguous cases arise when the language is underspecified or there are multiple candidate referents.This study investigates how pragmatic modulators external to the linguistic content are critical for the correct interpretation of referents in these scenarios.In particular, we demonstrate in a human subjects experiment how the social norms applicable in the given context influence the interpretation of referring expressions.Additionally, we highlight how current coreference tools in natural language processing fail to handle these ambiguous cases.We also briefly discuss the implications of this work for assistive robots which will routinely need to resolve referents in their environment.

Robots operating in unexplored environments with human teammates will need to learn unknown concepts on the fly.To this end, we demonstrate a novel system that combines a computational model of question generation with a cognitive robotic architecture.The model supports dynamic production of backand-forth dialogue for concept learning given observations of an environment, while the architecture supports symbolic reasoning, action representation, one-shot learning and other capabilities for situated interaction.The system is able to learn about new concepts including objects, locations, and actions, using an underlying approach that is generalizable and scalable.We evaluate the system by comparing learning efficiency to a human baseline in a collaborative reference resolution task and show that the system is effective and efficient in learning new concepts, and that it can informatively generate explanations about its behavior. * Work performed during a summer position at the Army Research Laboratory.1 The term 'concept' in this paper refers to any entity in the task domain, including objects, locations, and actions.. . .

With robotics rapidly advancing, more effective human–robot interaction is increasingly needed to realize the full potential of robots for society. While spoken language must be part of the solution, our ability to provide spoken language interaction capabilities is still very limited. In this article, based on the report of an interdisciplinary workshop convened by the National Science Foundation, we identify key scientific and engineering advances needed to enable effective spoken language interaction with robotics. We make 25 recommendations, involving eight general themes: putting human needs first, better modeling the social and interactive aspects of language, improving robustness, creating new methods for rapid adaptation, better integrating speech and language with other communication modalities, giving speech and language components access to rich representations of the robot’s current knowledge and state, making all components operate in real time, and improving research infrastructure and resources. Research and development that prioritizes these topics will, we believe, provide a solid foundation for the creation of speech-capable robots that are easy and effective for humans to work with.

Transparent task-based communication between human instructors and robot instructees requires robots to be able to determine whether a human instruction can and should be carried out, i.e., whether the human is authorized, and whether the robot can and should do it. If the instruction is not appropriate, the robot needs to be able to reject it in a transparent manner by including its reasons for the rejection. In this article, we provide a brief overview of our work on natural language understanding and transparent communication in the Distributed Integrated Affect Reflection Cognition (DIARC) architecture and demonstrate how the robot can perform different inferences based on context to determine whether it should reject a human instruction. Specifically, we discuss four task-based dialogues and show videos of the interactions with fully autonomous robots that are able to reject human commands and provide succinct explanations and justifications for their rejection. The proposed approach can form the basis of further algorithmic developments for adapting the robot’s level of transparency for different interlocutors and contexts.

Potential applications of robots in private and public human spaces have prompted the design of so-called “social robots” that can interact with humans in social settings and potentially cause humans to attach to the robots. The focus of this article is an analysis of possible benefits and challenges arising from such human-robot attachment as reported in the HRI literature, followed by guidelines for the use and the design of robots that might elicit attachment bonds. We start by analyzing the potential benefits for humans becoming attached to robots, which might include increased natural interaction, effectiveness and acceptance of the robot, social companionship, and well-being for the human. Turning to the potential risks associated with human-robot attachment, we discuss the possibly suboptimal use of the robot in the most benign cases, but also the potential formation of unidirectional emotional bonds, and the potential for deception and subconscious influence of the robot on the person in more severe cases. The upshot of the analysis then is a recommendation to reconceptualize relationships with social robots in an attempt to retain potential benefits of human-robot attachment, while mitigating (to the extent possible) its downsides.

As development of robots with the ability to self-assess their proficiency for accomplishing tasks continues to grow, metrics are needed to evaluate the characteristics and performance of these robot systems and their interactions with humans. This proficiency-based human-robot interaction (HRI) use case can occur before, during, or after the performance of a task. This article presents a set of metrics for this use case, driven by a four-stage cyclical interaction flow: (1) robot self-assessment of proficiency (RSA), (2) robot communication of proficiency to the human (RCP), (3) human understanding of proficiency (HUP), and (4) robot perception of the human’s intentions, values, and assessments (RPH). This effort leverages work from related fields including explainability, transparency, and introspection, by repurposing metrics under the context of proficiency self-assessment. Considerations for temporal level (a priori, in situ, and post hoc) on the metrics are reviewed, as are the connections between metrics within or across stages in the proficiency-based interaction flow. This article provides a common framework and language for metrics to enhance the development and measurement of HRI in the field of proficiency self-assessment.

Much research effort in HRI has focused on how to enable robots to learn new skills from observations, demonstrations, and instructions. Less work, however, has focused on how skills can be corrected if they were learned incorrectly, adapted to changing circumstances, or generalized/specialized to different contexts. In this paper, a skill modification framework is introduced that allows users to modify a robot’s stored skills quickly through instructions to (1) reduce inefficiencies, (2) fix errors, and (3) enable generalizations, all in a way for modified skills to be immediately available for task performance. A thorough evaluation of the implemented framework shows the operation of the algorithms integrated in a cognitive robotic architecture on different fully autonomous robots in various HRI case studies. An additional online HRI user study verifies that subjects prefer to quickly modify robot knowledge in the way we proposed in the framework.

Trust in human-robot interactions (HRI) is measured in two main ways: through subjective questionnaires and through behavioral tasks. To optimize measurements of trust through questionnaires, the field of HRI faces two challenges: the development of standardized measures that apply to a variety of robots with different capabilities, and the exploration of social and relational dimensions of trust in robots (e.g., benevolence). In this paper we look at how different trust questionnaires (Lyons & Guznov, 2019; Schaefer, 2016; Ullman & Malle, 2018) fare given these challenges that pull in different directions (being general vs. being exploratory) by studying whether people think the items in these questionnaires are applicable to different kinds of robots and interactions. In Study 1 we show that after being presented with a robot (non-humanoid) and an interaction scenario (fire evacuation), participants rated multiple questionnaire items such as "This robot is principled" as "Non-applicable to robots in general" or "Non-applicable to this robot." In Study 2 we show that the frequency of these ratings change (indeed, even for items rated as N/A to robots in general) when a new scenario is presented (game playing with a humanoid robot). Finally, while overall trust scores remained robust to N/A ratings, our results revealed potential fallacies in the way these scores are commonly interpreted. We conclude with recommendations for the development, use and results-reporting of trust questionnaires for future studies, as well as theoretical implications for the field of HRI.

Previous work has shown that people provide different moral judgments of robots and humans in the case of moral dilemmas. In particular, robots are blamed more when they fail to intervene in a situation in which they can save multiple lives but must sacrifice one person's life. Previous studies were all conducted with U.S. participants; the present two experiments provide a careful comparison of moral judgments among Japanese and U.S. participants. The experiments assess multiple ways in which cross-cultural differences in moral evaluations may emerge: in the willingness to treat robots as moral agents; the norms that are imposed on robots' behaviors; and the degree of blame that accrues to them when they violate the imposed norms. Even though Japanese and U.S. participants differ to some extent in their treatment of robots as moral agents and in the particular norms they impose on them, the two cultures show parallel patterns of greater blame for robots who fail to intervene in moral dilemmas.

Dialogue agents that interact with humans in situated environments need to manage referential ambiguity across multiple modalities and ask for help as needed. However, it is not clear what kinds of questions such agents should ask nor how the answers to such questions can be used to resolve ambiguity. To address this, we analyzed dialogue data from an interactive study in which participants controlled a virtual robot tasked with organizing a set of tools while engaging in dialogue with a live, remote experimenter. We discovered a number of novel results, including the distribution of question types used to resolve ambiguity and the influence of dialogue-level factors on the reference resolution process. Based on these empirical findings we: (1) developed a computational model for clarification requests using a decision network with an entropy-based utility assignment method that operates across modalities, (2) evaluated the model, showing that it outperforms a slot-filling baseline in environments of varying ambiguity, and (3) interpreted the results to offer insight into the ways that agents can ask questions to facilitate situated reference resolution.

Regular irradiation of indoor environments with ultraviolet C (UVC) light has become a regular task for many in-door settings as a result of COVID-19, but current robotic systems attempting to automate it suffer from high costs and inefficient irradiation. In this paper, we propose a purpose-made inexpensive robotic platform with off-the-shelf components and standard navigation software that, with a novel algorithm for finding optimal irradiation locations, addresses both shortcomings to offer affordable and efficient solutions for UVC irradiation. We demonstrate in simulations the efficacy of the algorithm and show a prototypical run of the autonomous integrated robotic system in an indoor environment. In our sample instances, our proposed algorithm reduces the time needed by roughly 30% while it increases the coverage by a factor of 35% (when compared to the best possible placement of a static light).

Intelligent agents that are confronted with novel concepts in situated environments will need to ask their human teammates questions to learn about the physical world.To better understand this problem, we need data about asking questions in situated task-based interactions.To this end, we present the Human-Robot Dialogue Learning (HuRDL) Corpus -a novel dialogue corpus collected in an online interactive virtual environment in which human participants play the role of a robot performing a collaborative tool-organization task.We describe the corpus data and a corresponding annotation scheme to offer insight into the form and content of questions that humans ask to facilitate learning in a situated environment.We provide the corpus as an empirically-grounded resource for improving question generation in situated intelligent agents.

Explainability has emerged as a critical AI research objective, but the breadth of proposed methods and application domains suggest that criteria for explanation vary greatly. In particular, what counts as a good explanation, and what kinds of explanation are computationally feasible, has become trickier in light of oqaque “black box” systems such as deep neural networks. Explanation in such cases has drifted from what many philosophers stipulated as having to involve deductive and causal principles to mere “interpretation,” which approximates what happened in the target system to varying degrees. However, such post hoc constructed rationalizations are highly problematic for social robots that operate interactively in spaces shared with humans. For in such social contexts, explanations of behavior, and, in particular, justifications for violations of expected behavior, should make reference to socially accepted principles and norms. In this article, we show how a social robot’s actions can face explanatory demands for how it came to act on its decision, what goals, tasks, or purposes its design had those actions pursue and what norms or social constraints the system recognizes in the course of its action. As a result, we argue that explanations for social robots will need to be accurate representations of the system’s operation along causal, purposive, and justificatory lines. These explanations will need to generate appropriate references to principles and norms—explanations based on mere “interpretability” will ultimately fail to connect the robot’s behaviors to its appropriate determinants. We then lay out the foundations for a cognitive robotic architecture for HRI, together with particular component algorithms, for generating explanations and engaging in justificatory dialogues with human interactants. Such explanations track the robot’s actual decision-making and behavior, which themselves are determined by normative principles the robot can describe and use for justifications.

We perform a corpus analysis to develop a representation of the knowledge and reasoning used to interpret indirect speech acts.An indirect speech act (ISA) is an utterance whose intended meaning is different from its literal meaning.We focus on those speech acts in which slight changes in situational or contextual information can switch the dominant intended meaning of an utterance from direct to indirect or vice-versa.We computationalize how various contextual features can influence a speaker's beliefs, and how these beliefs can influence the intended meaning and choice of the surface form of an utterance.We axiomatize the domain-general patterns of reasoning involved, and implement a proof-of-concept architecture using Answer Set Programming.Our model is presented as a contribution to cognitive science and psycholinguistics, so representational decisions are justified by existing theoretical work.

It has been claimed that a main advantage of cognitive architectures (compared to other types of specialized robotic architectures) is that they are task-general and can thus learn to perform any task as long as they have the right perceptual and action primitives. In this paper, we provide empirical evidence for this claim by directly comparing a high-performing custom robotic architecture developed for the standardized robotic "FetchIt!" challenge task to a hybrid cognitive robotic architecture that allows for online one-shot task learning and task modifications through natural language instructions. The results show that there is no disadvantage of running the hybrid architecture (i.e., no significant difference in overall performance or computational overhead compared to the custom architecture) while adding the flexibility of online one-shot task instruction and modification not available in the custom architecture.

We propose a novel approach to the problem of false belief revision in epistemic planning. Our state representations are pointed Kripke models with two binary relations over possible worlds: one representing agents' necessarily true knowledge, and one representing agents' possibly false beliefs. State transition functions maintain S5n properties in the knowledge relation and KD45n properties in the belief relation. When new information contradicts an agent's beliefs, belief revision draws new possible worlds from the agent's knowledge relation. Our method also improves upon prior work by accommodating false announcements. We develop our system as an extension to the mA* action language, presenting transition functions for ontic, sensing, and announcement actions.

Resolving Indirect Speech Acts (ISAs), in which the intended meaning of an utterance is not identical to its literal meaning, is essential to enabling the participation of intelligent systems in peoples’ everyday lives. Especially challenging are those cases in which the interpretation of such ISAs depends on context. To test a system’s ability to perform ISA resolution we need a corpus, but developing such a corpus is difficult, especialy given the contex-dependent requirement. This paper addresses the difficult problems of constructing a corpus of ISAs, taking inspiration from relevant work in using corpora for reasoning tasks. We present a formal representation of ISA Schemas required for such testing, including a measure of the difficulty of a particular schema. We develop an approach to authoring these schemas using corpus analysis and crowdsourcing, to maximize realism and minimize the amount of expert authoring needed. Finally, we describe several characteristics of collected data, and potential future work.

Turn-entry timing is an important requirement for conversation, and one that few spoken dialogue systems consider.In this paper we introduce a computational framework, based on work from psycholinguistics, which is aimed at achieving proper turn-entry timing for situated agents.Our approach involves incremental processing and lexical prediction of the turn in progress, which allows a situated dialogue agent to start its turn and initiate actions earlier than would otherwise be possible.We evaluate the framework by integrating it within a cognitive robotic architecture and testing performance on a corpus of situated, task-oriented human-robot directives.We demonstrate that: 1) the system is superior to a non-incremental system in terms of faster responses, reduced gap between turns, and the ability to perform actions early, 2) the system can time its turn to come in immediately at a turn transition, or earlier to produce several types of overlap, and 3) the system is robust to various forms of disfluency in the input.Overall, this domainindependent framework can be integrated into existing dialogue systems to improve responsiveness, and is another step toward more natural and fluid turn-taking behavior.

Assistive robots are becoming an increasingly important application platform for research in robotics, AI, and HRI, as there is a pressing need to develop systems that support the elderly and people with disabilities, with a clear path to market. Yet, what remains unclear is whether current autonomous systems are already up to the task or whether additional HRI work is needed to make these systems acceptable and useful. In this article, we report our efforts of developing and evaluating an architecture for a fully autonomous robot designed to assist older adults with Parkinson’s disease (PD) in sorting their medications. The main goal for the robot is to aid users in a manner that maintains the autonomy of the user by providing cognitive and social support with varying levels of assistance. We first evaluated the robot with subjects drawn from a pool of university students, which is common practice in experimental work in psychology and HRI. As the results were very positive, we followed up with an evaluation using people with Parkinson’s disease, who surprisingly had mostly negative outcomes. We thus report our analysis of the differences in the evaluations and discuss the challenges for HRI posed by the sources of the negative evaluations: (1) designing a robot to adapt to the many routines the participants use at home, (2) unique needs of participants with PD not present in student participants, and (3) the role of familiar technologies in designing and evaluating a new technology. While it is unlikely, given the current state of technology, that fully autonomous assistive robots for older adults will be available in the near term, we believe that our work exposes a critical need in HRI to involve the target population as early as possible in the design process.

Human behavior is frequently guided by social and moral norms, and no human community can exist without norms. Robots that enter human societies must therefore behave in norm-conforming ways as well. However, currently there is no solid cognitive or computational model available of how human norms are represented, activated, and learned. We provide a conceptual and psychological analysis of key properties of human norms and identify the demands these properties put on any artificial agent that incorporates norms-demands on the format of norm representations, their structured organization, and their learning algorithms.

Robots are machines and as such do not have gender. However, many of the gender-related perceptions and expectations formed in human-human interactions may be inadvertently and unreasonably transferred to interactions with social robots. In this paper, we investigate how gender effects in people's perception of robots and humans depend on their emotional intelligence (EI), a crucial component of successful human social interactions. Our results show that participants perceive different levels of EI in robots just as they do in humans. Also, their EI perceptions are affected by gender-related expectations both when judging humans and when judging robots with minimal gender markers, such as voice or even just a name. We discuss the implications for human-robot interactions (HRI) and propose further explorations of EI for future HRI studies.

Past work on acquisition of word-object associations in robots has focused on either fast instruction-based methods which accept highly constrained input or gradual cross-situational learning methods, but not a mixture of both. In this paper, we present an integrated robotic system which allows for a combination of these methods to contribute to the task of learning the labels of objects in AI agents. We demonstrate the expanded word learning capabilities in the outcome system and how learning from both human-human and human-robot dialogues can be achieved in one integrated system.

We present an approach to generating natural language justifications of decisions derived from norm-based reasoning.Assuming an agent which maximally satisfies a set of rules specified in an object-oriented temporal logic, the user can ask factual questions (about the agent's rules, actions, and the extent to which the agent violated the rules) as well as "why" questions that require the agent comparing actual behavior to counterfactual trajectories with respect to these rules.To produce natural-sounding explanations, we focus on the subproblem of producing natural language clauses from statements in a fragment of temporal logic, and then describe how to embed these clauses into explanatory sentences.We use a human judgment evaluation on a testbed task to compare our approach to variants in terms of intelligibility, mental model and perceived trust.

HRI researchers have made major strides in developing robotic architectures that are capable of reading a limited set of social cues and producing behaviors that enhance their likeability and feeling of comfort amongst humans. However, the cues in these models are fairly direct and the interactions largely dyadic. To capture the normative qualities of interaction more robustly, we propose “consent” as a distinct, critical area for HRI research. Convening important insights in existing HRI work around topics like touch, proxemics, gaze, and moral norms, the notion of consent reveals key expectations that can shape how a robot acts in social spaces. Consent need not be limited to just an explicit permission given in ethically charged or normatively risky scenarios. Instead, it is a richer notion, one that covers even implicit acquiescence in scenarios that otherwise seem normatively neutral. By sorting various kinds of consent through social and legal doctrine, we delineate empirical and technical questions to meet consent challenges faced in major application domains and robotic roles. Attention to consent could show, for example, how extraordinary, norm-violating actions can be justified by agents and accepted by those around them. We argue that operationalizing ideas from legal scholarship can better guide how robotic systems might cultivate and sustain proper forms of consent.

Artificial agents will need to be aware of human moral and social norms, and able to use them in decision-making. In particular, artificial agents will need a principled approach to managing conflicting norms, which are common in human social interactions. Existing logic-based approaches suffer from normative explosion and are typically designed for deterministic environments; reward-based approaches lack principled ways of determining which normative alternatives exist in a given environment. We propose a hybrid approach, using Linear Temporal Logic (LTL) representations in Markov Decision Processes (MDPs), that manages norm conflicts in a systematic manner while accommodating domain stochasticity. We provide a proof-of-concept implementation in a simulated vacuum cleaning domain.

It has been suggested that early human word learning occurs across learning situations and is bootstrapped by syntactic regularities such as word order. Simulation results from ideal learners and models assuming prior access to structured syn-tactic and semantic representations suggest that it is possible to jointly acquire word order and meanings and that learning is improved as each language capability bootstraps the other.We first present a probabilistic framework for early syntactic bootstrapping in the absence of advanced structured representations, then we use our framework to study the utility of joint acquisition of word order and word referent and its onset, in a memory-limited incremental model. Comparing learning results in the presence and absence of joint acquisition of word order in different ambiguous contexts, improvement in word order results showed an immediate onset, starting in early trials while being affected by context ambiguity. Improvement in word learning results on the other hand, was hindered in early trials where the acquired word order was imperfect,while being facilitated by word order learning in future trials as the acquired word order improved. Furthermore, our results showed that joint acquisition of word order and word referent facilitates one-shot learning of new words as well as inferring intentions of the speaker in ambiguous contexts.

The challenge of training AI systems to perform responsibly and beneficially has inspired different approaches for teaching a system what people want and how it is acceptable to attain that in the world. In this paper we compare work in reinforcement learning, in particular inverse reinforcement learning, with our norm inference approach. We test those two systems and present results. Using the idea of the "intentional stance", we explain how a norm inference approach can work even when another agent is acting strictly according to reward functions. In this way norm inference presents itself as a promising, more explicitly accountable approach with which to design AI systems from the start.

In previous work we provided a "norm conflict resolution" algorithm allowing agents in stochastic domains (represented by Markov Decision Processes) to "maximally satisfy" a set of moral or social norms, where such norms are represented by statements in linear temporal logic (LTL). This required the agent designer to provide weights specifying the relative importance of each norm. In this paper, we propose an "inverse norm conflict resolution'' algorithm for learning these weights from demonstration. This approach minimizes a cost function based on the relative entropy between a policy encoding the observed behavior and a policy representing optimal norm-following behavior. We demonstrate the effectiveness of the algorithm in a simple GridWorld domain.

We present a variation of the incremental and memory-limited algorithm in (Sadeghi et al., 2017) for Bayesian cross-situational word learning and evaluate the model in terms of its functional performance and its sensitivity to input order. We show that the functional performance of our sub-optimal model on corpus data is close to that of its optimal counterpart (Frank et al., 2009), while only the sub-optimal model is capable of predicting the input order effects reported in experimental studies.

Naturally interacting robots must be able to understand natural human speech. As such, recent work has sought to allow robots to infer the intentions behind commonly used non-literal utterances such as indirect speech acts (ISAs). However, it is still unclear to what extent ISAs will actually be used in task-based human-robot dialogue, and to what extent robots could function without the ability to understand ISAs. In this paper, we present the results of a Wizard-of-Oz experiment that examined human ISA use in scenarios that did or did not have conventionalized social norms, and analyzed both ISA use and perceptions of robots when robots were or were not capable of understanding ISAs. Our results suggest that (1) ISAs are commonly used in task-based human-robot dialogues, even when robots show themselves unable to understand ISAs; (2) ISA use is more common in contexts with conventionalized social norms; and (3) a robot's inability to understand ISAs harms both the robot's task performance and human perception of the robot.

The complex role of touch is an increasingly appreciated horizon for HRI research. The explicit and implicit registers of touch, both human-to-robot and robot-to-human, have opened up pressing questions in design and HRI ethics about embodiment, communication, care, and human affection. In this paper we present results of an MTurk survey about robot-initiated touch in a social context. We examine how a positive or negative attitude from the robot, as well as whether the robot touches an interactant, affects how a robot is judged as a worker and teammate. Our findings confirm previous empirical support for the idea of touch as enhancing social appraisals of a robot, though the extent of that positive tactile role was complicated and tempered by the survey responses» gender effects.

Learning new knowledge from single instructions and being able to apply it immediately is highly desirable for artificial agents. We provide the first demonstration of spoken instruction-based one-shot object and action learning in a cognitive robotic architecture and briefly discuss the architectural modifications required to enable such fast learning, demonstrating the new capabilities on a fully autonomous robot.

Speech overlap is a common phenomenon in natural conversation and in taskoriented interactions.As human-robot interaction (HRI) becomes more sophisticated, the need to effectively manage turntaking and resolve overlap becomes more important.In this paper, we introduce a computational model for speech overlap resolution in embodied artificial agents.The model identifies when overlap has occurred and uses timing information, dialogue history, and the agent's goals to generate context-appropriate behavior.We implement this model in a Nao robot using the DIARC cognitive robotic architecture.The model is evaluated on a corpus of task-oriented human dialogue, and we find that the robot can replicate many of the most common overlap resolution behaviors found in the human data.

Part of speech (POS) taggers and dependency parsers tend to work well on homogeneous datasets but their performance suffers on datasets containing data from different genres.In our current work, we investigate how to create POS tagging and dependency parsing experts for heterogeneous data by employing topic modeling.We create topic models (using Latent Dirichlet Allocation) to determine genres from a heterogeneous dataset and then train an expert for each of the genres.Our results show that the topic modeling experts reach substantial improvements when compared to the general versions.For dependency parsing, the improvement reaches 2 percent points over the full training baseline when we use two topics.

HRI research has yielded intriguing empirical results connected to ethics and how we act in social contexts with robots, even though much of this work has focused on task-based, one-on-one interaction. In this paper, we point to the need to investigate a wider range of ethically relevant dynamics that interaction with robots carries with it -- individually and in groups, with a single robot or more. We specifically examine three areas: 1) the primacy and implicit dynamics of bodily perception, 2) the competing interests at work in a single robot-human interaction, and 3) the social intricacy of multiple agents -- robots and humans -- communicating and making decisions. While these areas are not exhaustive by any means, we find they yield concrete directions for how HRI can contribute to a widening, intensifying set of ethical debates with critical empirical insight, starting to explore more of the ethical landscape in HRI.

Zero resource spoken term discovery in continuous speech is the discovery of repeated patterns in acoustic signals without any higher level linguistic information. These patterns are then combined to define the compositional units of that speech. We describe and implement an algorithm that tags similar subsequences among sequences of acoustic features. We then discuss the use of this algorithm as part of a complete spoken term discovery system. Our implementation leverages parallelization via modern GPUs, allowing many independent comparisons to be executed concurrently. This parallelization enables the described system to analyze large data sets in tractable time frames. The accuracy and performance of our approach are compared to existing approaches as well as human transcriptions on two corpora of continuous natural speech. Our system improved on published results for multiple metrics.

When uttering referring expressions in situated task descriptions, humans naturally use verbal and non-verbal channels to transmit information to their interlocutor. To develop mechanisms for robot architectures capable of resolving object references in such interaction contexts, we need to better understand the multi-modality of human situated task descriptions. In current computational models, mainly pointing gestures, eye gaze, and objects in the visual field are included as non-verbal cues, if any. We analyse reference resolution to objects in an object manipulation task and find that only up to 50% of all referring expressions to objects can be resolved including language, eye gaze and pointing gestures. Thus, we extract other non-verbal cues necessary for reference resolution to objects, investigate the reliability of the different verbal and non-verbal cues, and formulate lessons for the design of a robot’s natural language understanding capabilities.

For situated agents to effectively engage in natural-language interactions with humans, they must be able to refer to entities such as people, locations, and objects. While classic referring expression generation (REG) algorithms like the Incremental Algorithm (IA) assume perfect, complete, and accessible knowledge of all referents, this is not always possible. In this work, we show how a previously presented consultant framework (which facilitates reference resolution when knowledge is uncertain, heterogeneous and distributed) can be used to extend the IA to produce DIST-PIA, a domain-independent algorithm for REG under uncertain, heterogeneous, and distributed knowledge. We also present a novel framework that can be used to evaluate such REG algorithms without conflating the performance of the algorithm with the performance of classifiers it employs.

As the linguistic capabilities of interactive robots advance, it becomes increasingly important to understand how humans will instruct robots through natural language. What is more, with the increased use of teleoperated humanoid robots, it is important to recognize whether any differences between instructions given to humans and to robots are due to the physical embodiment or to the perceived autonomy of the instructee. In this paper, we present the results of a human-subject experiment in which participants interacted in a collaborative, task-based setting with both a human and a suit-based, teleoperated humanoid robot said to be either autonomous or teleoperated. Our results suggest that humans will use politeness strategies equally with human, autonomous robotic, and teleoperated robotic teammates, reinforcing recent findings that autonomous robots must comprehend and appropriately respond to human utterances that follow such strategies. Our results also suggest variations in how different teammates were perceived. Specifically, our results suggest that human-teleoperated robots were perceived as less intelligent than human teammates; a finding with serious implications for human-robot team dynamics.

Humans often use indirect speech acts (ISAs) when issuing directives. Much of the work in handling ISAs in computational dialogue architectures has focused on correctly identifying and handling the underlying non-literal meaning. There has been less attention devoted to how linguistic responses to ISAs might differ from those given to literal directives and how to enable different response forms in these computational dialogue systems. In this paper, we present ongoing work toward developing dialogue mechanisms within a cognitive, robotic architecture that enables a richer set of response strategies to non-literal directives.

An important open problem for enabling truly taskable robots is the lack of task-general natural language mechanisms within cognitive robot architectures that enable robots to understand typical forms of human directives and generate appropriate responses. In this paper, we first provide experimental evidence that humans tend to phrase their directives to robots indirectly, especially in socially conventionalized contexts. We then introduce pragmatic and dialogue-based mechanisms to infer intended meanings from such indirect speech acts and demonstrate that these mechanisms can handle all indirect speech acts found in our experiment as well as other common forms of requests.

We present a domain-independent approach to reference resolution that allows a robotic or virtual agent to resolve references to entities (e.g., objects and locations) found in open worlds when the information needed to resolve such references is distributed among multiple heterogeneous knowledge bases in its architecture. An agent using this approach can combine information from multiple sources without the computational bottleneck associated with centralized knowledge bases. The proposed approach also facilitates “lazy constraint evaluation”, i.e., verifying properties of the referent through different modalities only when the information is needed. After specifying the interfaces by which a reference resolution algorithm can request information from distributed knowledge bases, we present an algorithm for performing open-world reference resolution within that framework, analyze the algorithm’s performance, and demonstrate its behavior on a simulated robot.

Situated dialogue systems that interact with humans as part of a team (e.g., robot teammates) need to be able to use information from communication channels to gauge the coordination level and effectiveness of the team. Currently, the feasibility of this end goal is limited by several gaps in both the empirical and computational literature. The purpose of this paper is to address those gaps in the following ways: (1) investigate which properties of task-oriented discourse correspond with effective performance in human teams, and (2) discuss how and to what extent these properties can be utilized in spoken dialogue systems. To this end, we analyzed natural language data from a unique corpus of spontaneous, task-oriented dialogue (CReST corpus), which was annotated for disfluencies and conversational moves. We found that effective teams made more self-repair disfluencies and used specific communication strategies to facilitate grounding and coordination. Our results indicate that truly robust and natural dialogue systems will need to interpret highly disfluent utterances and also utilize specific collaborative mechanisms to facilitate grounding. These data shed light on effective communication in performance scenarios and directly inform the development of robust dialogue systems for situated artificial agents.

In three studies we found further evidence for a previously discovered Human-Robot (HR) asymmetry in moral judgments: that people blame robots more for inaction than action in a moral dilemma but blame humans more for action than inaction in the identical dilemma (where inaction allows four persons to die and action sacrifices one to save the four). Importantly, we found that people's representation of the “robot” making these moral decisions appears to be one of a mechanical robot. For when we manipulated the pictorial display of a verbally described robot, people showed the HR asymmetry only when making judgments about a mechanical-looking robot, not a humanoid robot. This is the first demonstration that robot appearance affects people's moral judgments about robots.

Sex robots are gaining a remarkable amount of attention in current discussions about technology and the future of human relationships. To help understand what kinds of relationships people will have with these robots, empirical data about people's views of sex robots is needed. We report the results of the first systematic survey that asks about the appropriateness and value of sex robots, acceptable forms they can take, and the degree to which using them counts as sex. The results show a consistent difference in the uses for which women and men found sex robots to be appropriate, with women less and men more inclined to consider them socially useful. We also found convergences on what sex robots are like and how sex with them is to be classified, suggesting that larger views about relationships and society, not just understandings of sex robots themselves, should be a matter for more research and thus frame future work on the ethics of sex robots.

A robot participating in natural dialogue with a human interlocutor may need to discuss, reason about, or initiate actions concerning dialogue-referenced entities. To do so, the robot must first identify or create new representations for those entities, a capability known as reference resolution. We previously presented algorithms for resolving references occurring in definite noun phrases. In this paper we present GH-POWER: an algorithm for resolving references occurring in a wider array of linguistic forms, by making novel extensions to the Givenness Hierarchy, and evaluate GH-POWER on natural task-based human-human and human-robot dialogues.

The ultimate goal of human natural language interaction is to communicate intentions. However, these intentions are often not directly derivable from the semantics of an utterance (e.g., when linguistic modulations are employed to convey polite-ness, respect, and social standing). Robotic architectures withsimple command-based natural language capabilities are thus not equipped to handle more liberal, yet natural uses of linguistic communicative exchanges. In this paper, we propose novel mechanisms for inferring in-tentions from utterances and generating clarification requests that will allow robots to cope with a much wider range of task-based natural language interactions. We demonstrate the potential of these inference algorithms for natural human-robot interactions by running them as part of an integrated cognitive robotic architecture on a mobile robot in a dialogue-based instruction task.

Moral judgements are a complex phenomenon that have gained a renewed interest in the research community. Many have proposed explanations for moral judgements, including utilitarian accounts and the Principle of Double Effect. Some also advocate for the critical role of emotional processes like empathy. However, developing a computational model of moral judgements is rare perhaps due in part to the numerous influences on it. We present here a computational model of moral judgements based on moral expectation and the Principle of Double Effect. We then extend this model to provide a plausible explanation for the effect of empathy on these judgements. We evaluate these models using results from recent studies with human participants.

People tend to anthropomorphize agents that look and/or act human, and further, they tend to evaluate such agents more positively. This, in turn, has motivated the development of robotic agents that are humanlike in appearance and/or behavior. Yet, some agents -- often those with highly humanlike appearances -- have been found to elicit the opposite, wherein they are evaluated more negatively than their less humanlike counterparts. These trends are captured by Masahiro Mori's uncanny valley hypothesis, which describes a (uncanny) valley in emotional responding - a switch from affinity to dislike - elicited by agents that are ``too humanlike'. However, while the valley phenomenon has been repeatedly observed via subjective measures, it remains unknown as to whether such evaluations reflect a potential impact to a person's behavior (i.e., aversion). We attempt to address this gap in the literature via a novel experimental paradigm employing both traditional subjective ratings, as well as measures of peoples' behavioral and phsyiological responding. The results show that not only do people rate highly humanlike robots as uncanny, but moreover, they exhibit greater avoidance of such encounters than encounters with less humanlike and human agents. Thus, the findings not only support Mori's hypothesis, but further, they indicate the valley should be taken as a serious consideration for peoples' interactions with humanlike agents.

Facial masking is a symptom of Parkinson's disease (PD) in which humans lose the ability to quickly create refined facial expressions. This difficulty of people with PD can be mistaken for apathy or dishonesty by their caregivers and lead to a breakdown in social relationships. We envision future "robot mediators" that could ease tensions in these caregiver-client relationships by intervening when interactions go awry. However, it is currently unknown whether people with PD would even accept a robot as part of their healthcare processes. We thus conducted a first human-robot interaction study to assess the extent to which people with PD are willing to discuss their health status with a robot. We specifically compared a robot interviewer to a human interviewer in a within-subjects design that allowed us to control for individual differences of the subjects with PD caused by their individual disease progression. We found that participants overall reacted positively to the robot, even though they preferred interactions with the human interviewer. Importantly, the robot performed at a human level at maintaining the participants' dignity, which is critical for future social mediator robots for people with PD.

Moral norms play an essential role in regulating human interaction. With the growing sophistication and proliferation of robots, it is important to understand how ordinary people apply moral norms to robot agents and make moral judgments about their behavior. We report the first comparison of people's moral judgments (of permissibility, wrongness, and blame) about human and robot agents. Two online experiments (total N = 316) found that robots, compared with human agents, were more strongly expected to take an action that sacrifices one person for the good of many (a "utilitarian" choice), and they were blamed more than their human counterparts when they did not make that choice. Though the utilitarian sacrifice was generally seen as permissible for human agents, they were blamed more for choosing this option than for doing nothing. These results provide a first step toward a new field of Moral HRI, which is well placed to help guide the design of social robots.

Recently there has been a lot of focus on human robot co-habitation issues that are often orthogonal to many aspects of human-robot teaming; e.g. on producing socially acceptable behaviors of robots and de-conflicting plans of robots and humans in shared environments. However, an interesting offshoot of these settings that has largely been overlooked is the problem of planning for serendipity - i.e. planning for stigmergic collaboration without explicit commitments on agents in co-habitation. In this paper we formalize this notion of planning for serendipity for the first time, and provide an Integer Programming based solution for this problem. Further, we illustrate the different modes of this planning technique on a typical Urban Search and Rescue scenario and show a real-life implementation of the ideas on the Nao Robot interacting with a human colleague.

The problem of uniquely identifying an entity described in natural language, known as reference resolution, has become recognized as a critical problem for the field of robotics, as it is necessary in order for robots to be able to discuss, reason about, or perform actions involving any people, locations, or objects in their environments. However, most existing algorithms for reference resolution are domain-specific and limited to environments assumed to be known a priori. In this paper we present an algorithm for reference resolution which is both domain independent and designed to operate in an open world. We call this algorithm POWER: Probabilistic Open-World Entity Resolution. We then present the results of an empirical study demonstrating the success of POWER both in properly identifying the referents of referential expressions and in properly modifying the world model based on such expressions.

We propose that moral competence consists of five distinct but related elements: (1) having a system of norms; (2) mastering a moral vocabulary; (3) exhibiting moral cognition and affect; (4) exhibiting moral decision making and action; and (5) engaging in moral communication. We identify some of the likely triggers that may convince people to (justifiably) ascribe each of these elements of moral competence to robots. We suggest that humans will treat robots as moral agents (who have some rights, obligations, and are targets of blame) if they perceive them to have at least elements (1) and (2) and one or more of elements (3)-(5).

Autonomous social robots embedded in human societies have to be sensitive to human social interactions and thus to moral norms and principles guiding these interactions. Actions that violate norms can lead to the violator being blamed. Robots thus need to be able to anticipate possible norm violations and attempt to prevent them while they execute actions. If norm violations cannot be prevented (e.g., in a moral dilemma situation in which every action leads to a norm violation), then the robot needs to be able to justify the action to address any potential blame. In this paper, we present a first attempt at an action execution system for social robots that can (a) detect (some) norm violations, (b) consult an ethical reasoner for guidance on what to do in moral dilemma situations, and (c) it can keep track of execution traces and any resulting states that might have violated norms in order to produce justifications.

As future human-robot teams are envisioned for a variety of application domains, researchers have begun to investigate how humans and robots can communicate effectively and naturally in the context of human-robot team tasks. While a growing body of work is focused on human-robot communication and human perceptions thereof, there is currently little work on human perceptions of robot-robot communication. Understanding how robots should communicate information to each other in the presence of human teammates is an important open question for human-robot teaming. In this paper, we present two human-robot interaction (HRI) experiments investigating the human perception of verbal and silent robot-robot communication as part of a human-robot team task. The results suggest that silent communication of task-dependent, human-understandable information among robots is perceived as creepy by cooperative, co-located human teammates. Hence, we propose that, absent specific evidence to the contrary, robots in cooperative human-robot team settings need to be sensitive to human expectations about overt communication, and we encourage future work to investigate possible ways to modulate such expectations.

Being able to quickly and naturally teach robots new knowledge is critical for many future open-world human-robot interaction scenarios. In this paper we present a novel approach to using natural language context for one-shot learning of visual objects, where the robot is immediately able to recognize the described object. We describe the architectural components and demonstrate the proposed approach on a robotic platform in a proof-of-concept evaluation.

Recent proposals for how robots should talk to people when they give advice suggest that the same strategies humans employ with other humans are effective for robots as well. However, the evidence is exclusively based on people's observation of robot giving advice to other humans. Hence, it is not clear whether the results still apply when people actually participate in real interactions with robots. We address this shortcoming in a novel systematic mixed-methods study where we employ both survey-based subjective and brain-based objective measures (using functional near infrared spectroscopy). The results show that previous results from observation conditions do not transfer automatically to interaction conditions, and that robot appearance and interaction distance are important modulators of human perceptions of robot behavior in advice-giving contexts.

Prior approaches to politeness modulation in natural language generation (NLG) of-ten focus on manipulating factors such as the directness of requests that pertain to preserving the autonomy of the addressee (negative face threats), but do not have a systematic way of understanding potential impoliteness from inadvertently critical or blame-oriented communications (positive face threats). In this paper, we discuss on-going work to integrate a computational model of blame to prevent inappropriate threats to positive face. 1

It is well-known that a robot's appearance and its observable behavior can affect a human interactant's perceptions of the robot's capabilities and propensities in settings where humans and robots are co-located; for remote interactions the specific effects are less clear. Here, we use a remote interaction setting to investigate possible effects of simulated versus real first-person robot video feeds. The first experiment uses subject-level comparisons of the two video conditions in a multi-robot setting while the second and third experiments focus on a single robot, single video condition using a larger population (via Amazon Mechanical Turk) to study between-subjects effects. The latter experiments also probe the effects of robot appearance, video feed type, and stake humans have in the task. We observe a complex interplay between interaction, robot appearance, and video feed type as they affect perceived collaboration, utility, competence, and warmth of the robot.

Beliefs play an important role in human-robot teaming scenarios, where the robots must reason about other agents' intentions and beliefs in order to inform their own plan generation process, and to successfully coordinate plans with the other agents. In this paper, we cast the evolving and complex structure of beliefs, and inference over them, as a planning and plan recognition problem. We use agent beliefs and intentions modeled in terms of predicates in order to create an automated planning problem instance, which is then used along with a known and complete domain model in order to predict the plan of the agent whose beliefs are being modeled. Information extracted from this predicted plan is used to inform the planning process of the modeling agent, to enable coordination. We also look at an extension of this problem to a plan recognition problem. We conclude by presenting an evaluation of our technique through a case study implemented on a real robot.

People will eventually be exposed to robotic agents that may protest their commands for a wide range of reasons. We present an experiment designed to determine whether a robot's appearance has a significant effect on the amount of agency people ascribed to it and its ability to dissuade a human operator from forcing it to carry out a specific command. Participants engage in a human-robot interaction (HRI) with either a small humanoid or non-humanoid robot that verbally protests a command. Initial results indicate that humanoid appearance does not significantly affect the behavior of human operators in the task. Agency ratings given to the robots were also not significantly affected.

The Uncanny Valley Hypothesis (UVH) describes the sudden change in a person's affect from affinity to aversion that is evoked by robots that border a human-like appearance. The portion of the human-likeness spectrum in which such aversion is posited to occur is referred to as the “uncanny valley”. However, evidence in support of the UVH is primarily based on subjectively assessed evaluations. Thus it remains an open question as to whether there are behavioral or neurophysiological manifestations of uncanny valley effects. To address this gap in literature, we investigated the activation of the anterior prefrontal cortex (PFC) - a region of the brain associated with emotion regulation - in response to a series of robots with varying human-likeness. We hypothesized that highly human-like robots - which have been found to receive negative subjective attributions - will also elicit increased activity in the PFC versus humans or robots with lesser degrees of human-likeness in accordance with the UVH. Our results show a “valley” in brain activity in the PFC corresponding to the valley observed via subjective measures alone, thus suggesting one neural manifestation (the PFC) of uncanny valley effects and further supporting the affective response (aversion) posited to occur by the UVH. However, the results also reveal a second “uncanny valley” in prefrontal hemodynamics, which suggests that the effects (and the contributing factors) are more complex than previously understood.

Recent research indicates that other factors in addition to appearance may contribute to the “Uncanny Valley” effect, and it is possible that “uncanny actions” such as “robot telepathy” - the nonverbal exchange of information among multiple robots - could be one such factor. We thus specifically examine whether humans are negatively affected by displays of nonverbal robot-robot communication through a disaster relief scenario in which one robot must relay information from a human participant to another robot in order to successfully complete a task. Our results showed no significant difference between the verbal and nonverbal communication strategies, thus suggesting that “telepathic information transmission” is acceptable. However, we also found several unexplained robot-specific effects, prompting future follow-up studies to determine their causes and the extent to which these effects might impact human perception and acceptance of robot communication strategies.

Current approaches to handling indirect speech acts (ISAs) do not account for their sociolinguistic underpinnings (i.e., politeness strategies). Deeper understanding and appropriate generation of indirect acts will require mechanisms that integrate natural language (NL) understanding and generation with social information about agent roles and obligations,which we introduce in this paper. Additionally, we tackle the problem of understanding and handling indirect answers that take the form of either speech acts or physical actions, which requires an inferential, plan-reasoning approach. In order to enable artificial agents to handle an even wider-variety of ISAs, we present a hybrid approach, utilizing both the idiomatic and inferential strategies. We then demonstrate our system successfully generating indirect requests and handling indirect answers, and discuss avenues of future research.

While much research exists on resolving spatial natural language references to known locations, little work deals with handling references to unknown locations. In this paper we introduce and evaluate algorithms integrated into a cognitive architecture which allow an agent to learn about its environ-ment while resolving references to both known and unknown locations. We also describe how multiple components in the architecture jointly facilitate these capabilities.

The prefrontal cortex (PFC) has been investigated extensively with functional magnetic resonance imaging (fMRI) and identified as a neural correlate of emotion regulation and decision-making, particularly in the context of moral utilitarian dilemmas. However, there are two limitations of previous work: (1) fMRI requires strict constraints on the physical experimental environment and (2) experimental manipulations have yet to consider the role of agency on the dilemma outcome and the corresponding neural activity. In this paper, we extend previous work by first evaluating an alternative neuroimaging technique, functional near infrared spectroscopy (NIRS), for observing decision-making processes in a less-constrained environment. We then examine the role of agency in deciding emotional (moral) and non-emotional dilemmas through a 2-part, 20-subject preliminary investigation. Our findings are two-fold: they suggest (1) NIRS is a potential alternative to fMRI in this decision-making context and (2) agency shows some influence on prefrontal neural activity, making NIRS a promising method for objective evaluation of agency and emotional value in human-agent interactions.

The strategies employed by animals to choose mates can have significant consequences for individual fitness and also profoundly influence evolutionary processes. Female choice of mates has been a research focus, but males can also influence outcomes in many situations. We have developed an agent-based model to explore how internal and external variables may interact in treefrogs to alter the mating task and the ultimate quality of mates that both males and females find. In this paper, we investigate specifically strategies males may use to place themselves in the landscape and the effectiveness of those strategies in attracting females. Our simulated swamp environment contained stationary calling male treefrog agents, wandering male agents (searching for territories near other males), and female treefrog agents (searching for calling male mates). Wanderers and females used one of two possible search strategies to find males to mate with or settle by: a closest-above-a-minimum-threshold (min-threshold) strategy or a best-of-closest-S (best-of-n) strategy. We found that the mean quality of mated pairs is highest when both males and females use the best-of-n strategy, despite using it for different purposes. When the initial proportion of wanderers in the population is high, females see significant benefits in male mate quality when they are using the min-threshold strategy, compared to when the proportion is low. As well, the number of agents that mate at all falls sharply when male agents are using the min-threshold strategy with many initial wanderers. Thus, we find a complex interaction between individual internal variables (strategy choice) and external variables (behavior of conspecifics) that can lead to exciting new empirical studies on treefrogs in nature.

Functional near-infrared spectroscopy (fNIRS) is a promising new tool for research in human-robot interaction (HRI). The technology has already been used for brain-robot interfaces to affect robots' behaviors and as an evaluation tool for assessing brain activity during interactions. In this survey, we provide a comprehensive literature review of published research on fNIRS from various communities to assess its utility in HRI. We discuss four exemplary applications in more detail and also list several challenges that need to be overcome for fNIRS to be an effective tool in realistic HRI settings.

Introspection mechanisms are employed in agent architectures toimprove agent performance. However, there is currently no approach tointrospection that makes automatic adjustments at multiple levels inthe implemented agent system. We introduce our novel multi-levelintrospection framework that can be used to automatically adjustarchitectural configurations based on the introspection results at theagent, infrastructure and component level. We demonstrate the utilityof such adjustments in a concrete implementation on a robot where thehigh-level goal of the robot is used to automatically configure thevision system in a way that minimizes resource consumption whileimproving overall task performance.

This paper describes the Brainput system, which learns to identify brain activity patterns occurring during multitasking. It provides a continuous, supplemental input stream to an interactive human-robot system, which uses this information to modify its behavior to better support multitasking. This paper demonstrates that we can use non-invasive methods to detect signals coming from the brain that users naturally and effortlessly generate while using a computer system. If used with care, this additional information can lead to systems that respond appropriately to changes in the user's state. Our experimental study shows that Brainput significantly improves several performance metrics, as well as the subjective NASA-Task Load Index scores in a dual-task human-robot activity.

Robots are currently being used in and developed for critical HRI applications such as search and rescue. In these scenarios, humans operating under changeable and high-stress conditions must communicate effectively with autonomous agents, necessitating that such agents be able to respond quickly and effectively to rapidly-changing conditions and expectations. We demonstrate a robot planner that is able to utilize new information, specifically information originating in spoken input produced by human operators.

Hybrid reactive-deliberative architectures in robotics combine reactive sub-policies for fast action execution with goal sequencing and deliberation. The need for replanning, however, presents a challenge for reactivity and hinders the potential for guarantees about the plan quality. In this paper, we argue that one can integrate abstract planning provided by symbolic dynamic programming in first order logic into a reactive robotic architecture, and that such an integration is in fact natural and has advantages over traditional approaches. In particular, it allows the integrated system to spend off-line time planning for a policy, and then use the policy reactively in open worlds, in situations with unexpected outcomes, and even in new environments, all by simply reacting to a state change executing a new action proposed by the policy. We demonstrate the viability of the approach by integrating the FODD-Planner with the robotic DIARC architecture showing how an appropriate interface can be defined and that this integration can yield robust goal-based action execution on robots in open worlds.

Humans are deeply affective beings that expect other human-like agents to be sensitive to and express their own affect. Hence, complex artificial agents that are not capable of affective communication will inevitably cause humans harm, which suggests that affective artificial agents should be developed. Yet, affective artificial agents with genuine affect will then themselves have the potential for suffering, which leads to the “Affect Dilemma for Artificial Agents,” and more generally, artificial systems. In this paper, we discuss this dilemma in detail and argue that we should nevertheless develop affective artificial agents; in fact, we might be morally obligated to do so if they end up being the lesser evil compared to (complex) artificial agents without affect. Specifically, we propose five independent reasons for the utility of developing artificial affective agents and also discuss some of the challenges that we have to address as part of this endeavor.

Efficient collaborations between interacting agents, be they humans, virtual or embodied agents, require mutual recognition of the goal, appropriate sequencing and coordination of each agent's behavior with others, and making predictions from and about the likely behavior of others. Moment-by-moment eye gaze plays an important role in such interaction and collaboration. In light of this, we used a novel experimental paradigm to systematically investigate gaze patterns in both human-human and human-agent interactions. Participants in the study were asked to interact with either another human or an embodied agent in a joint attention task. Fine-grained multimodal behavioral data were recorded including eye movement data, speech, first-person view video, which were then analyzed to discover various behavioral patterns. Those patterns show that human participants are highly sensitive to momentary multimodal behaviors generated by the social partner (either another human or an artificial agent) and they rapidly adapt their gaze behaviors accordingly. Our results from this data-driven approach provide new findings for understanding micro-behaviors in human-human communication which will be critical for the design of artificial agents that can generate human-like gaze behaviors and engage in multimodal interactions with humans.

Multitasking has become an integral part of work environments, even though people are not well-equipped cognitively to handle numerous concurrent tasks effectively. Systems that support such multitasking may produce better performance and less frustration. However, without understanding the user's internal processes, it is difficult to determine optimal strategies for adapting interfaces, since all multitasking activity is not identical. We describe two experiments leading toward a system that detects cognitive multitasking processes and uses this information as input to an adaptive interface. Using functional near-infrared spectroscopy sensors, we differentiate four cognitive multitasking processes. These states cannot readily be distinguished using behavioral measures such as response time, accuracy, keystrokes or screen contents. We then present our human-robot system as a proof-of-concept that uses real-time cognitive state information as input and adapts in response. This prototype system serves as a platform to study interfaces that enable better task switching, interruption management, and multitasking.

In many contexts, one is confronted with the problem of extract ing information from large amounts of different types soft data (e.g., text) and hard data (from e.g., physics-based sensing systems). In handling hard data, signal and data processing offers a wealth of methods related to modeling, estimation, tracking, and inference tasks. However, soft data present several challenges that necessitate the development of new data processing methods. For example, with suitable statistical natural language processing (NLP) methods, text can be converted into logic statements that are associated with various forms of associated uncertainty related to the credibility of the statement, the reliability of the text source, and so forth. In combining or fusing soft data with either soft or hard data, one must deploy methods that can suitably preserve and update the uncertainty associated with the data, thereby providing uncertainty bounds related to any inferences regarding semantics. Since standard Bayesian probabilistic approaches have problems with suitably handling uncertain logic statements, there is an emerging need for new methods for processing heterogeneous data. In this paper, we describe a framework for fusing soft and hard data based on the Dempster-Shafer (DS) belief theoretic approach which is well-suited to the task of capturing the types of models and uncertain rules that are more typical of soft data. Since the effectiveness of traditional DS methods has been hampered by high computational requirements, we base the processing framework on our new conditional approach to DS theoretic evidence updating and fusion. We address the issue of laying the foundation for a theoretically justifiable, and computationally efficient framework for fusing soft and hard data taking into account the inherent data uncertainty such as reliability and credibility. Moreover, we present an illustrative ex ample that highlights the potential for the DS conditional approach for fusing heterogeneous data.

Natural language interactions between humans and robots are currently limited by many factors, most notably by the robot's concept representations and action repertoires. We propose a novel algorithm for learning meanings of action verbs through dialogue-based natural language descriptions. This functionality is deeply integrated in the robot's natural language subsystem and allows it to perform the actions associated with the learned verb meanings right away without any additional help or learning trials. We demonstrate the effectiveness of the algorithm in a scenario where a human explains to a robot the meaning of an action verb unknown to the robot and the robot is subsequently able to carry out the instructions involving this verb.

In this paper, we present an integrated planning and robotic architecture that actively directs an agent engaged in an urban search and rescue (USAR) scenario. We describe three salient features that comprise the planning component of this system, namely (1) the ability to plan in a world open with respect to objects, (2) execution monitoring and replanning abilities, and (3) handling soft goals, and detail the interaction of these parts in representing and solving the USAR scenario at hand. We show that though insufficient in an individual capacity, the integration of this trio of features is sufficient to solve the scenario that we present. We test our system with an example problem that involves soft and hard goals, as well as goal deadlines and action costs, and show that the planner is capable of incorporating sensing actions and execution monitoring in order to produce goal-fulfilling plans that maximize the net benefit accrued.

Natural human-robot interaction requires different and more robust models of language understanding (NLU) than non-embodied NLU systems. In particular, architectures are required that (1) process language incrementally in order to be able to provide early backchannel feedback to human speakers; (2) use pragmatic contexts throughout the understanding process to infer missing information; and (3) handle the underspecified, fragmentary, or otherwise ungrammatical utterances that are common in spontaneous speech. In this paper, we describe our attempts at developing an integrated natural language understanding architecture for HRI, and demonstrate its novel capabilities using challenging data collected in human-human interaction experiments.

Complex robots with many interacting components in their control architectures are subject to component failures from which neither the control architecture nor the implementing infrastructure can recover. Moreover, the operating conditions for these components might be at odds with goals the robot might have adopted (e.g., through external commands or in the course of the execution of the current task). We argue that the best (if not the only) way to resolve any difficulties that arise from the different requirements at the agent, component and infrastructure levels is to use a common formal logical goal representation for all three layers. We discuss how these representations can be integrated into a complex robotic architecture and demonstrate in an experimental evaluation on a robot how the architecture can recover from a failure situation that it would not have been able to handle without explicit multi-level unified goal representations and their associated monitoring and reasoning processes.

As the number of applications for human-robot teaming continue to rise, there is an increasing need for planning technologies that can guide robots in such teaming scenarios. In this article, we focus on adapting planning technology to Urban Search And Rescue (USAR) with a human-robot team. We start by showing that several aspects of state-of-the-art planning technology, including temporal planning, partial satisfaction planning, and replanning, can be gainfully adapted to this scenario. We then note that human-robot teaming also throws up an additional critical challenge, namely, enabling existing planners, which work under closed-world assumptions, to cope with the open worlds that are characteristic of teaming problems such as USAR. In response, we discuss the notion of conditional goals, and describe how we represent and handle a specific class of them called open world quantified goals. Finally, we describe how the planner, and its open world extensions, are integrated into a robot control architecture, and provide an empirical evaluation over USAR experimental runs to establish the effectiveness of the planning components.

Robot autonomy is of high relevance for HRI, in particular for interactions of humans and robots in mixed human-robot teams. In this paper, we investigate empirically the extent to which autonomy based on independent decision making and acting by the robot can affect the objective task performance of a mixed human-robot team while being subjectively acceptable to humans. The results demonstrate that humans not only accept robot autonomy in the interest of the team, but also view the robot more as a team member and find it easier to interact with, despite a very minimalist graphical/speech interface. Moreover, we find evidence that dynamic autonomy reduces human cognitive load.

Robots that can be given instructions in spoken language need to be able to parse a natural language utterance quickly, determine its meaning, generate a goal representation from it, check whether the new goal conflicts with existing goals, and if acceptable, produce an action sequence to achieve the new goal (ideally being sensitive to the existing goals). In this paper, we describe an integrated robotic architecture that can achieve the above steps by translating natural language instructions incrementally and simultaneously into formal logical goal description and action languages, which can be used both to reason about the achievability of a goal as well as to generate new action scripts to pursue the goal. We demonstrate the implementation of our approach on a robot taking spoken natural language instructions in an office environment.

Many hive-based agents, such as some bees, rely on memory and communication to aid in foraging. The benefits of these abilities seem obvious, but it is unlikely that they are beneficial in every environment. In this paper, we present the results of experiments examining the effect that environmental structure can have on the utility of communication and memory for hive-based agents, finding that there are some environments in which they do not contribute substantially to the agents' ability to survive.

There is recent evidence that males and females view robots differently, from the way robots are conceptualized, to the way humans respond when they interact with them. In this paper, we further explore gender-based differences in human-robot interaction. Moreover, we provide the first available evidence for sex-related differences in reactions to gendered synthetic voices that are either disembodied or physically embodied within a robot. Results indicate that physical embodiment and perceived entity gender may interact with human sex-related characteristics and pre-experimental attitudes in determining how people respond to artificial entities.

Autonomous robots that operate in real-world domains face multiple challenges that make planning and goal selection difficult. Not only must planning and execution occur in real time, newly acquired knowledge can invalidate previous plans, and goals and their utilities can change during plan execution. However, these events can also provide opportunities, if the architecture is designed to react appropriately. We present here an architecture that integrates the SapaReplan planner with the DIARC robot architecture, allowing the architecture to react dynamically to changes in the robot's goal structures.

Social-psychological processes in humans will play an important role in long-term human-robot interactions. This study investigates people's perceptions of social presence in robots during (relatively) short interactions. Findings indicate that males tend to think of the robot as more human-like and accordingly show some evidence of "social facilitation" on an arithmetic task as well as more socially desirable responding on a survey administered by a robot. In contrast, females saw the robot as more machine-like, exhibited less socially desirable responding to the robot's survey, and were not socially facilitated by the robot while engaged in the arithmetic tasks. Various alternative accounts of these findings are explored and the implications of these results for future work are discussed.

Robots that interact with humans face-to-face using natural language need to be responsive to the way humans use language in those situations. We propose a psychologically-inspired natural language processing system for robots which performs incremental semantic interpretation of spoken utterances, integrating tightly with the robot's perceptual and motor systems.

Complex robots that interact naturally with humans require the integration, coordination and maintenance of many diverse software components and algorithms. An architecture that incorporates explicit knowledge about the relationships among these components and the overall system state can be used for introspection and consequently to reason about the best configurations of the computing environment under changing conditions; potential uses include maintaining the system's integrity, promoting its health, and providing the ability to dynamically reconfigure system components (e.g., after component failure). In this paper, we describe a rudimentary reasoning system, part of our distributed integrated affect reflection cognition (DIARC) architecture for human-robot interaction, that can autonomously perform failure detection, failure recovery, and system reconfiguration of distributed architectural components to ensure sustained operation and interactions. We demonstrate the functionality and utility of the proposed mechanisms on a robot, where architectural components are forcefully removed by hand and automatically recovered by the system while the robot is continuing its interactions with humans as part of a joint human-robot task.

The performance of embodied multi-agent systems depends, in addition to the agent architectures of the employed agents, on their physical characteristics (e.g., sensory range, speed, etc.) and group properties (e.g., number of agents, types of agents, etc.). Consequently, it is difficult to evaluate the performance of a multi-agent system based on the performance of an agent architecture alone, even in homogeneous teams. In this paper, we propose a method for analyzing the performance of multi-agent systems based on the notion of "performance-cost-tradeoff," which attempts to determine the relations among different cost-dimensions by performing a performance sampling of these dimensions and comparing them relative to their associated costs. Specifically, we investigate the performance-cost tradeoffs of four candidate architectures for a multi-agent territory exploration task in which a group of agents is required to visit a set of checkpoints randomly placed in an environment in the shortest time possible. Performance tradeoffs between three dimensions (sensory range, group size, and prediction) are then used to illustrate the cost-benefit analyses performed to determine the best agent configurations for different practical settings.

We describe the tight integration of incremental natural language understanding, goal management, and action processing in a complex robotic architecture, which is required for natural interactions between robots and humans. Specifically, the natural language components need to process utterances while they are still spoken to be able to initiate feedback actions in a timely fashion, while the action manager might need information at various points during action execution that must be obtained from humans. We argue that a finer- grained integration provides much more natural human-robot interactions and much more reasonable multitasking.

Complex, autonomous robots integrate a large set of sometimes very diverse algorithms across at least three levels of system organization: the agent architecture, the implementation environment, and the hardware devices. Insofar as a distinction is maintained between them, the levels serve different purposes and thus exhibit different characteristic strengths and weaknesses. Exchanging information among organizational levels can be used to mitigate the shortcomings of one level by making use of the strengths of another. In this paper, we highlight the roles, characteristics, and relations between the infrastructure and the architecture of complex robots, describing a novel form of integration that results from enabling the exchange of information between these two levels, which otherwise is maintained internally. The information from the infrastructure is especially amenable for use by the architecture to achieve a higher level of robustness and system awareness. We demonstrate the functionality and utility of the proposed mechanisms in a set of experiments in which failures of architectural components are induced on an actual robot engaged in a joint human-robot team task.

In this paper, we introduce a hierarchical extension to the standard particle swarm optimization algorithm that allows swarms to cope better with dynamically changing fitness evaluations for a given parameter space. We present the formal framework and demonstrate the utility of the extension in an application system for dynamic face detection. Specifically, the feature detector/tracker uses the proposed "hierarchical real-time swarms" for a continuous concurrent dynamic search of the best locations in a two-dimensional parameter space and the image space to improve upon feature detection and tracking in changing environments. We show in several experimental evaluations on a robot interacting with people in real-time that the proposed method is robust to lighting changes and does not require any calibration. Moreover, the method is not limited to face detection, but can be applied to any n-dimensional search space

Recognizing and responding to human affect is important in collaborative tasks in joint human-robot teams. In this paper we present an integrated affect and cognition architecture for HRI and report results from an experiment with this architecture that shows that expressing affect and responding to human affect with affect expressions can significantly improve team performance in a joint human-robot task.

Robots that can interact naturally with humans require the integration and coordination of many different components with heavy computational demands. We argue that an architecture framework with facilities for dynamic, reliable, fault-recovering, remotely accessible, distributed computing is needed for the development and operation of applications that support and enhance human activities and capabilities. We describe a robotic architecture development system, called ADE, that is built on top of a multi-agent system in order to provide all of the above features. Specifically, we discuss support for autonomic computing in ADE, briefly comparing it to related features of other commonly used robotic systems. We also report our experiences with ADE in the development of an architecture for an intelligent robot assistant and provide experimental results demonstrating the system's utility

Evolutionary investigations are often very expensive in terms of the required computational resources and many general questions regarding the utility of a feature F of an agent (e.g., in competitive environments) or the likelihood of F evolving (or not evolving) are therefore typically difficult, if not practically impossible to answer. We propose and demonstrate in extensive simulations a methodology that allows us to answer such questions in setups where good predictors of performance in a task T are available. These predictors evaluate the performance of an agent kind A in a task T*, which can then transformed by including costs and additional factors to make predictions about the performance of A in T.

This paper is an examination of communication in hive-based swarms in the biological setting, focusing on the effect environmental factors have on the utility of communication. Our investigation utilizes a generational survival task to measure the benefit of communication in a biological setting. Swarm members forage for food, consuming energy in the process, and returning to a central "hive" to contribute any surplus. The resources of the hive determine when reproduction is possible, so it is in the best interest of the population to maximize the efficiency of foraging. The measure of performance is the size of the swarm surviving at the end of a simulation run. Each simulation begins with a swarm of fixed size (5 agents), making contributions to the hive (and subsequent procreation) necessary for good performance.

We present a physically realizable UAV model for locating and tracking chemical clouds. Simulation results are presented for implementations of this model with two configurations, one that is faster and requires more space to avoid collisions, and one that is slower and can cover an area more densely. Heterogeneous swarms of agents are shown to have better performance than homogeneous swarms of similar size because they take advantage of the strengths of each configuration.

We present a real-time system for a mobile robot that can reliably detect and track people in uncontrolled indoor environments. The system uses a combination of leg detection based on distance information from a laser range sensor and visual face detection based on an analogical algorithm implemented on specialized hardware (the CNN universal machine). Results from tests in a variety of environments with different lighting conditions, a different number of appearing and disappearing people, and different obstacles are reported to demonstrate that the system can find and subsequently track several, possibly people simultaneously in indoor environments. Applications of the system include in particular service robots for social events.

Evidence from recent psycholinguistic experiments suggests that humans resolve reference incrementally in the presence of constraining visual context. In this paper, we present and evaluate a computational model of human reference resolution that directly builds a semantic interpretation of an utterance without the need for a separate syntactic analysis phase, which typically involves the construction of parse trees. The model is implemented on a robot using real audio and video inputs, (thus it operates in real time), and is distributed over several computers, which run in parallel. Results from experiments with the model confirm the viability of the algorithm to process semantic interpretations, in particular reference incrementally, as demonstrated to be employed by humans.

Behavior selection is typically a "built-in" feature of behavior-based architectures and hence, not amenable to change. There are, however, circumstances where changing behavior selection strategies is useful and can lead to better performance. In this paper, we demonstrate that such dynamic changes of behavior selection mechanisms are beneficial in several circumstances. We first categorize existing behavior selection mechanisms along three dimensions and then discuss seven possible circumstances where dynamically switching among them can be beneficial. Using the agent architecture framework activation, priority, observer, and component (APOC), we show how instances of all (nonempty) categories can be captured and how additional architectural mechanisms can be added to allow for dynamic switching among them. In particular, we propose a generic architecture for dynamic behavior selection, which can integrate existing behavior selection mechanisms in a unified way. Based on this generic architecture, we then verify that dynamic behavior selection is beneficial in the seven cases by defining architectures for simulated and robotic agents and performing experiments with them. The quantitative and qualitative analyzes of the results obtained from extensive simulation studies and experimental runs with robots verify the utility of the proposed mechanisms.