We present a multimodal dialogue system that allows doctors to interact with a medical decision support system in virtual reality (VR).We integrate an interactive visualization of patient records and radiology image data, as well as therapy predictions.Therapy predictions are computed in realtime using a deep learning model.

In this paper, we describe a real-time knowledge acquisition and anomaly handling architecture in a maintenance scenario of an industrial cyber-physical production environment. We use the Business Process Model and Notation (BPMN) for modeling and controlling of the maintenance procedure. Automatic handwriting and pen gesture recognition is combined with a networked smart pen that is used on semantically structured paper forms. Here, we discuss our architecture with BPMN-modeled workflows for real-time data processing. All detected anomalies are automatically associated to the corresponding knowledge sources such as the process model, structure model, business process model and user models; linked Semantic Media-Wiki (SMW) pages are built up accordingly. Our architecture provides for a seamless integration of real-time knowledge sources by smart pen technology and real-time CPS processing by a BPMN engine handling the execution and coordination of the modeled interactions.

In this article we propose a generic framework for ontological intention recognition and action recommendation involving distributed and networked active digital object memories (ADOMe) for production factories and their components as well as individually manufactured products. For this purpose, a concrete and complex application scenario in the industrial environment is designed. In addition, the developed approach provides an important contribution to the utilization of ADOMes and their networking and intelligent representation of consumption, savings and savings potential. Furthermore, the processes and tasks of the proactive intention recognition component reflect the behavior and interaction of man and machine or man and object. Such a system can provide context-aware action recommendations without any explicit request by end users. Its prototypical realization is realized under integrated use of specially designed methods and technologies, such as the instrumentation of objects with ADOMes, process and role models, and rule recommendation procedures.

The Automatic Content Linking Device monitors a conversation and uses automatically recognized words to retrieve documents that are of potential use to the participants. The document set includes project related reports or emails, transcribed snippets of past meetings, and websites. Retrieval results are displayed at regular intervals.

LIDIAP

The SAMMIE system is an in-car multi-modal dialogue system for an MP3 application. It is used as a testing environment for our research in natural, intuitive mixed-initiative interaction, with particular emphasis on multimodal output planning and realization aimed to produce output adapted to the context, including the driver's attention state w.r.t. the primary driving task.

User acceptance of multimodal dialog systems in general depends not only on the quality of the system output but also significantly on the quality of the process feedback the user gets during interaction. We show an approach that permits the system to present simultaneously- and synchronously to the ongoing processing- adequate de-tailed and adaptive immediate feedback in various ways. For example, in multimodal dialog systems correspond-ing graphical presentations and/or appropriate behaviour patterns of an animated agent during user input process-ing significantly contribute to a more natural and expres-sively more powerful system. Additionally, in case of processing errors the approach provides sufficient infor-mation to be able to present the user detailed descriptive feedback on the stage of processing at which the error occurred. The approach is mainly based on a module working state model that is used to determine in real time general processing state information in terms of relatively fine grained global system working states. Evaluations of the overall system show that simultaneous processing feedback is highly appreciated by users. 1.

The development of large-scale dialog systems requires a flexible architecture model and adequate software support to cope with the challenge of system integration. This contributionOur current work in the context of the SmartKom project has been funded by the German Federal Ministry for Education and Research (BMBF) under grant 01 IL 905 K7. presents a general framework for building integrated natural-language and multimodal dialog systems. Our approach relies on a distributed component model that enables flexible re-use and extension of existing software modules and is able to deal with a heterogeneous software environment. A practical result of our research is the development of a sophisticated integration platform, called MULTIPLATFORM, which is based on the proposed framework. This MULTIPLATFORM testbed has been used in various large and mid-size research projects to develop integrated system prototypes.

The development of an intelligent user interface that supports multimodal access to multiple applications is a challenging task. In this paper we present a generic multimodal interface system where the user interacts with an anthropomorphic personalized interface agent using speech and natural gestures. The knowledge-based and uniform approach of SmartKom enables us to realize a comprehensive system that understands imprecise, ambiguous, or incomplete multimodal input and generates coordinated, cohesive, and coherent multimodal presentations for three scenarios, currently addressing more than 50 different functionalities of 14 applications. We demonstrate the main ideas in a walk through the main processing steps from modality fusion to modality fission.

This paper describes a novel functionality of the VERBMOBIL system, a large scale translation system designed for spontaneously spoken multilingual negotiation dialogues. The task is the on-demand generation of dialogue scripts and result summaries of dialogues. We focus on summary generation and show how the relevant data are selected from the dialogue memory and how they are packed into an appropriate abstract representation. Finally, we demonstrate how the existing generation module of VERBMOBIL was extended to produce multilingual and result summaries from these representations.

Based on our experiences in VERBMOBIL, a large scale speech-to-speech translation system, we identify two types of problems that a generation component must address in a realistic implementation and present relevant examples. As an extension to the architecture of a translation system, we present a module for robustness preprocessing on the interface between translation and generation.

A main purpose of natural language systems (NL‐systems) is the simulation of human speech behavior, which divides into generation and analysis, in all kinds of situations. Humans process natural language in an incremental style, which means that sentences are generated or analyzed in a piecemeal fashion. Observations show that there are bidirectional dependencies between these processes. During parsing, hypotheses for the continuation of an utterance may be generated. During generation, the output may be parsed in order to detect ambiguities. This paper presents a two‐level incremental parsing algorithm for LD/TLP‐TAGs on the basis of Earley's context‐free parser. LD/TLP‐TAG is a new extension for TAG that increases the descriptive power of the formalism without changing its generative power. Elementary rules are split into local dominance description (LD) and linear precedence rules (TLP) allowing for generalizations of ordering constraints. The presented algorithm is used within an anticipation feedback loop for the incremental syntactic generator TAG‐GEN.