The aim of our research is to create a system whereby human members of a team can collaborate in a natural way with robots. In this paper we describe a Wizard of Oz (WOZ) study conducted to find the natural speech and gestures people would use when interacting with a mobile robot as a team member. Results of the study show that in the beginning participants used simple speech, but once the users learned that the system understood more complicated speech, they began to use more spatially descriptive language. User responses indicate that gestures aided in spatial communication. The input mode that combined the use of speech and gestures was found to be best. We first discuss previous work and detail how our study contributes to this body of knowledge. Then we describe the design of our WOZ study and discuss the results and issues encountered during the completion of the experiment.

Future space operations will increasingly demand cooperation between humans and autonomous space systems such as robots, observer satellites, and distributed components. Human team members use a combination of gestures, gaze, posture, deictic references and speech to communicate effectively. When a human team collaborates on a given task, they discuss the task, create a plan and then review this plan prior to execution to ensure success. This is exactly the process we envision for effective human-autonomous agent collaborative teamwork. Visual spatial information is a common reference for increased shared situation awareness between humans and autonomous systems. We use a spatial dialog approach to enable multiple humans to naturally and effectively communicate and work with multiple autonomous space systems. With this in mind, we have created a prototype spatial dialog system to support teamwork between humans and autonomous robotic agents. We combine augmented reality gesture interaction and visualization with speech to realize a spatial dialog capability. This paper describes our prototype, general approach, and related issues for team-oriented spatial dialog interaction with autonomous space systems.

This paper describes an augmented reality (AR) multimodal interface that uses speech and paddle gestures for interaction. The application allows users to intuitively arrange virtual furniture in a virtual room using a combination of speech and gestures from a real paddle. Unlike other multimodal AR applications, the multimodal fusion is based on the combination of time-based and semantic techniques to disambiguate a users speech and gesture input. We describe our AR multimodal interface architecture and discuss how the multimodal inputs are semantically integrated into a single interpretation by considering the input time stamps, the object properties, and the user context.