Autonomous Systems like self-driving cars and collaborative robots must occasionally ask people around them for help in anomalous situations. A new generation of multiadaptive interaction platforms provides a comprehensive multimodal presentation of the current situation in real-time, so that a smooth transfer of control back and forth between human agents and AI systems is guaranteed. We present the anatomy of our multiadaptive human-environment interaction platform which includes explicit models of the attentional and cognitive state of the human agents as well as a dynamic model of the cyber-physical environment, and supports massive multimodality, multiscale and multiparty interaction. It is based on the principles of symmetric multimodality and bidirectional representations: all input modes are also available as output modes and vice versa, so that the system not only understands and represents the user’s multimodal input, but also its own multimodal output. We illustrate our approach with examples from advanced automotive and manufacturing applications.

Networked cyber-physical systems are the basis for intelli-gent environments in a variety of settings such as smart factories, smart transportation systems, smart shops, and smart buildings. However, one of the remaining grand challenges for the Internet of Things is to transform the way how humans interact with and control such cyber-physical environments (CPE). A major goal of our research is the creation of a multiadaptive dialogue management system that is adaptive in multiple ways: adaptive to various CPE, adaptive to diverse modality combinations, adaptive to a variety of interaction metaphors, and adaptive to diverse task domains and user models.

SmartWeb provides a context-aware user interface to web services, so that it can support the mobile user in different roles, e.g. as a car driver, a motorbiker, or a pedestrian. It provides a symmetric multimodal dialogue system [2] combining speech, gesture, haptic and video input with speech, haptic, video and acoustic output. It goes beyond traditional keyword search engines like Google by delivering higher quality results that are adapted to the mobile user's current task and situation. In mobile situations, users don't want to deal with hypertext lists of retrieved webpages, but simply want an answer to their query. If a desperate driver with a crying and acutely ill child on the backseat asks SmartWeb "Who is the closest paediatrician?" he needs just the name and address of the doctor. Based on SmartWeb's ability to combine various web services, the driver can then ask SmartWeb a follow-up question about route guidance to the doctor's practice. One of the innovative features of SmartWeb is that the user can specify whether he wants a textual or pictorial answer, a video clip or a sound file as a query result.

The design of mobile navigation systems adapting to limited resources will be an important future challenge. Since typically several different means of transportation have to be combined in order to reach a destination, the user interface of such a system has to adapt to the user's changing situation. This applies especially to the alternating use of different technologies to detect the user's position, which should be as seamless as possible. This article presents a hybrid navigation system that relies on different technologies to determine the user's location and that adapts the presentation of route directions to the limited technical resources of the output device and the limited cognitive resources of the user.

SmartKom is a multimodal dialog system that combines speech, gesture, and mimics input and output. Spontaneous speech understanding is combined with the video-based recognition of natural gestures. One of the major scientific goals of SmartKom is to design new computational methods for the seamless integration and mutual disambiguation of multimodal input and output on a semantic and pragmatic level. SmartKom is based on the situated delegation-oriented dialog paradigm, in which the user delegates a task to a virtual communication assistant, visualized as a life-like character on a graphical display. We describe the SmartKom architecture, the use of an XML-based mark-up language for multimodal content, and some of the distinguishing features of the first fully operational SmartKom demonstrator.

No abstract available.

Verbmobil is a speech-to-speech translation system for spontaneously spoken negotiation dialogs. The actual system translates 74.2% of spontaneously spoken German input. We give an overview of the Verbmobil system. After the introduction of the Verbmobil scenario and the unique constraints of the project, we describe the underlying system architecture and its realization. The progress that was achieved on the end-to-end translation rate owes much to the increase of the word recognition rate from 45% in 1993 to 87% in 1996. In order to achieve the envisaged coverage on the uncertain speech recognizer output, deep and shallow approaches to the analysis and transfer problem had to be combined.

In this talk, we will, show how techniques for planning text and discourse can be generalized to plan the structure and content of multimodal communications, that integrate natural language, pointing, graphics, and animations. The central claim of this talk is that the generation of multimodal discourse can be considered as an incremental planning process that aims to achieve a given communicative goal.

The paper investigates several approaches to user modeling in natural-language dialogue systems. First, reasons are pointed out why user modeling has become so important in the last few years, and definitions are proposed for the notions of "user model" and "user modeling component." Then, techniques for constructing user models in the course of a dialogue are presented and recent proposals for representing a wide range of assumptions about a user's beliefs and goals in a system's knowledge base are surveyed. Examples for the application of user models in systems developed to date are presented, and some social implications are discussed. Finally, unsolved problems like coping with collective beliefs or resource-limited processes are investigated, and prospects for application-oriented research are outlined.