This article reports on the development of capabilities for (on-screen) virtual agents and robots to support isolated older adults in their homes. A real-time architecture was developed to use a virtual agent or a robot interchangeably to interact via dialog and gesture with a human user. Users could interact with either agent on 12 different activities, some of which included on-screen games, and forms to complete. The article reports on a pre-study that guided the choice of interaction activities. A month-long study with 44 adults between the ages of 55 and 91 assessed differences in the use of the robot and virtual agent.

The assumption that we can preprogram robots with all the information necessary for their function becomes impractical as the range of robotics applications grows. One widely proposed solution is the specification of reusable task plans, or recipes, consisting of a sequence of primitive actions. However, the primitive actions, such as unscrewing a nut in a car maintenance task, cannot always be predefined and thus must be learned for a particular robot platform and workspace. In this work, we present a novel algorithm that enables a robot to identify a reusable motion trajectory associated with each primitive action of a plan. Our approach segments the motion data captured during the demonstration of a human performing the given task, while also leveraging the human's verbal cues. We evaluated our algorithms in a pilot study with 6 users executing 90 primitive actions.

A collaborative robot needs to be able to regulate and manage shared goals during collaboration. Emotion has a crucial influence on this goal management process. In this paper, we provide a cost function that we use to choose the goal in the shared plan with the lowest cost value out of a set of alternative goals. This cost function is a) based on the goal attributes, b) with respect to the reverse appraisal of the perceived emotion, and c) the appraisal of the collaborative environment.

We present a novel algorithm to identify reusable motion trajectories corresponding to the primitive actions in a human demonstration of a symbolic plan with accompanying narration. Our approach involves a multi-step process starting with time-series pattern mining applied to raw motion-capture data. We evaluated our algorithm on recordings of human motions and showed that it identifies reusable trajectories with 86% of the accuracy of human experts.

We have developed learning and interaction algorithms to support a human teaching hierarchical task models to a robot using a single demonstration in the context of a mixed-initiative interaction with bi-directional communication. In particular, we have identified and implemented two important heuristics for suggesting task groupings based on the physical structure of the manipulated artifact and on the data flow between tasks. We have evaluated our algorithms with users in a simulated environment and shown both that the overall approach is usable and that the grouping suggestions significantly improve the learning and interaction.

Current approaches to learning partial ordering constraints by demonstration require demonstrating all (or almost all) possible completion orders. We have developed an algorithm that, for plans involving relative placement of objects, learns the partial ordering constraints from a single demonstration by letting the user specify naturally conceived reference frame information. This work is an example of a broader research agenda that involves applying principles of human collaboration to robot learning from demonstration.

This paper describes a Behavior Markup Language (BML) realizer that we developed for use in our research on human-robot interaction. Existing BML realizers used with virtual agents are based on fixed-timing algorithms and because of that are not suitable for robotic applications. Our realizer uses an event-driven architecture, based on Petri nets, to guarantee the specified synchronization constraints in the presence of unpredictable variability in robot control systems. Our implementation is robot independent, open source and uses the Robot Operating System (ROS).

We have developed a systematic methodology for designing emotional facial expressions for humanoid robots, especially those with limited degrees of freedom. The methodology is firmly grounded in the psychological literature on human static and dynamic emotional facial expressions. We demonstrate the methodology by applying it to a recent humanoid robot and evaluate the results, confirming that the observed confusion matrix agrees qualitatively with the predictions of the methodology. We also investigate how robot facial emotion recognition compares for dynamic versus static expressions.

We report on an experiment in which a human collaborates with a small, autonomous, humanoid robotic toy. The experiment demonstrates that the robot's use of two simple gestures, namely orienting its head toward the addressee when it speaks and raising its arm in the direction of objects it refers to, significantly improve the human's perception of the robot's interaction skills and quality as a collaborator.

We have developed and tested a reusable Robot Operating System (ROS) module that supports engagement between a human and a humanoid robot by generating appropriate directed gaze, mutual facial gaze, adjacency pair and backchannel connection events. The module implements policies for adding gaze and pointing gestures to referring phrases (including deictic and anaphoric references), performing end-of-turn gazes, responding to human-initiated connection events and maintaining engagement. The module also provides an abstract interface for receiving information from a collaboration manager using the Behavior Markup Language (BML) and exchanges information with our previously developed engagement recognition module.

The future of robotics engineering is in the hands of engineers and must be handled to ensure the safety of all people and the reputation of the field. We are in the process of developing a code of ethics for professional robotics engineers to serve as a guideline for the ethical development of the field. This document contains the current version of this code and describes the methodology used in developing it.

Based on a study of the engagement process between humans, we have developed and implemented an initial computational model for recognizing engagement between a human and a humanoid robot. Our model contains recognizers for four types of connection events involving gesture and speech: directed gaze, mutual facial gaze, conversational adjacency pairs and backchannels. To facilitate integrating and experimenting with our model in a broad range of robot architectures, we have packaged it as a node in the open-source Robot Operating System (ROS) framework. We have conducted a preliminary validation of our computational model and implementation in a simple human-robot pointing game.

Ordinary people already have great difficulty using the advanced features of digitally-operated household devices, such personal video recorders, DVD burners, etc., and "white goods," such as washing machines, microwave ovens, programmable thermostats, etc. And the problem is getting worse as more customization and programming features are continually being added. This is challenging and practical application for intelligent user interface research, and one in which new ideas are badly needed. This panel brings together industrial and academic researchers as well as business people to report on their activities and stimulate others to join.

This work unifies two important threads of research in intelligent user interfaces which share the common element of explicit task modeling. On the one hand, longstanding research on task-centered GUI design (sometimes called model-based design) has explored the benefits of explicitly modeling the task to be performed by an interface and using this task model as an integral part of the interface design process. More recently, research on collaborative interface agents has shown how an explicit task model can be used to control the behavior of a software agent that helps a user perform tasks using a GUI. This paper describes a collection of tools we have implemented which generate both a GUI and a collaborative interface agent from the same task model. Our task-centered GUI design tool incorporates a number of novel features which help the designer to integrate the task model into the design process without being unduly distracted. Our implementation of collaborative interface agents is built on top of the COLLAGEN middleware for collaborative interface agents.

Tutoring agents can provide a dynamic and engaging way to help users understand an application. However, integrating tutoring agents into applications is difficult. It requires the expertise to create the tutoring agent, and also an understanding of the inner workings of the application itself. This demo presents a task-based GUI editor that produces a software agent tutor for free. The designer need only create a task model, and then use the editor to produce the GUI. A tutoring agent will automatically be included in the new application.

This paper addresses the issue of designing embodied conversational agents that exhibit appropriate posture shifts during dialogues with human users. Previous research has noted the importance of hand gestures, eye gaze and head nods in conversations between embodied agents and humans. We present an analysis of human monologues and dialogues that suggests that postural shifts can be predicted as a function of discourse state in monologues, and discourse and conversation state in dialogues. On the basis of these findings, we have implemented an embodied conversational agent that uses Collagen in such a way as to generate postural shifts.

Computer tutors and intelligent software assistants have traditionally been thought of as different kinds of systems. However tutors and assistants share many properties. We have incorporated tutorial strategies into an intelligent assistant based on the COLLAGEN architecture. We are working on an agent, named Triton, which teaches and helps users with the graphical user interface of an air travel planning system. We found that the collaborative model underlying COLLAGEN is an excellent foundation for both an assistant and a tutor, and that both modes of interaction can be implemented in the same system with different parameter settings.

Task models are used in many areas of computer science including planning, intelligent tutoring, plan recognition, interface design, and decision theory. However, developing task models is a significant practical challenge. We present a task model development environment centered around a machine learning engine that infers task models from examples. A novel aspect of the environment is support for a domain expert to refine past examples as he or she develops a clearer understanding of how to model the domain. Collectively, these examples constitute a "test suite" that the development environment manages in order to verify that changes to the evolving task model do not have unintended consequences.

We have developed an application-independent toolkit, called Collagen, based on the SharedPlan theory of collaborative discourse, in which interaction histories are hierarchically structured according to a user´s goals and intentions. We have used Collagen to implement an example collaborative interface agent with discourse processing, but not natural language understanding. In this paper, we concentrate on how a segmented interaction history supports user orientation, intelligent assistance, and transformations, such as returning to earlier points in the problem solving process and replaying segments in a new context. Superseded by TR97-21.

Article Adding a collaborative agent to graphical user interfaces Share on Authors: Charles Rich Mitsubishi Electric MERL, 201 Broadway, Cambridge, MA Mitsubishi Electric MERL, 201 Broadway, Cambridge, MAView Profile , Candace L. Sidner Lotus Development Corporation, One Rogers Street, Cambridge, MA Lotus Development Corporation, One Rogers Street, Cambridge, MAView Profile Authors Info & Claims UIST '96: Proceedings of the 9th annual ACM symposium on User interface software and technologyNovember 1996 Pages 21–30https://doi.org/10.1145/237091.237094Online:01 November 1996Publication History 14citation548DownloadsMetricsTotal Citations14Total Downloads548Last 12 Months16Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Most software engineering tools use a shallow representation of software objects and manipulate this representation using procedural methods. This approach allows one to get off to a fast start and quickly provides a tool that delivers benefits. However, a point will be reached where more knowledge-intensive approaches will be needed to achieve significantly higher levels of capability. The authors suggest that the software engineering tools of the future will have to rely on: deep representation to capture a sufficiently large part of knowledge about programming in general and particular programs; inspection methods to deal with complexity; and intelligent assistance.>

The authors' experience in the Programmer's Apprentice project in applying knowledge representation and automated reasoning to support software development is summarized. A system, called Cake, is described that comprises seven layers of knowledge representation and reasoning facilities: truth maintenance, Boolean constraint propagation, equality, types, algebra, frames, and Plan Calculus. Sessions with two experimental software development tools implemented using Cake, the Requirements Apprentice and the Debugging Assistant, are also included.>

This paper reports on the initial design and partial implementation of an interactive programming environment to be used by expert programmers. The system is based on three forms of program description: 1) definition of structured data objects, their parts, properties, and relations between them, 2) input–output specification of the behavior of program segments, and 3) a hierarchical representation of the internal structure of programs (plans). The plan representation is of major theoretical interest because it includes not only data flow and control flow relationships between subsegments of a program, but also goal-subgoal, prerequisite, and other logical dependencies between the specifications of the subsegments. Plans are utilized both for describing particular programs and in the compilation of a knowledge base of more abstract knowledge about programming, such as the concept of a loop and various specializations, such as enumeration loops and search loops. We also describe a deductive system which can verify the correctness of plans involving side effects on complex data with structure sharing.