Adaptation for social companions is a crucial requirement for future applications. Personalized interaction seems to be an important factor for long-term commitment to interact with a social robot. We present a study evaluating the feasibility of a dueling bandit learning approach for preference learning (PL) in Human-Robot Interaction (HRI). Furthermore, we explore whether the embodiment of the PL agent has an influence on the user's evaluation of the learner. We conducted a study (n=53) comparing a graphical user interface (GUI), a virtual robot and a real robot. We found no difference regarding the preference for the virtual or real robot. We used the obtained study data to compare the PL approach against a strategy that randomly selects preference rankings. The results show that that the dueling bandit PL approach can be used to learn a user's preference in HRI.

Nodding is an important factor in human communication, providing a physical cue for socially communicative acts such as turn taking, backchanneling, and confirmation. In this article, we describe a vision-based online head nodding detector that works with monocular camera images. Using SVM regression, our system estimates the head pose based on facial landmarks. Subsequence dynamic time-warping is then used to compare head pose features against nod templates. In contrast to many other previous implementations, our system was evaluated with study participants who were not instructed to reply by nodding, and shows good results while maintaining a low false positive rate.

Studying long-term human-robot interactions in the context of playing games can help answer many questions about how humans perceive robots. This paper presents the results of a study where the robot Flobi [11] plays a game of pairs against a human player and employs a memory with information about past interactions. The study focuses on long-term effects, namely the novelty effect, and how a memory with statistics about past game-plays can be used to cope with that effect. We also investigate how an autonomous interaction compares to a remotely controlled system that plays flawlessly. Results showed that providing information about how players performed throughout the interaction can help to keep them more interested and engaged. Nevertheless, results also showed that this information in combination with perfect playing skills tended to promote a more negative perception of the interaction and of the robot.

In this paper, we present our humanoid robot "Meka", participating in a multi party human robot dialogue scenario. Active arbitration of the robot's attention based on multi-modal stimuli is utilised to observe persons which are outside of the robots field of view. We investigate the impact of this attention management and addressee recognition on the robot's capability to distinguish utterances directed at it from communication between humans. Based on the results of a user study, we show that mutual gaze at the end of an utterance, as a means of yielding a turn, is a substantial cue for addressee recognition. Verification of a speaker through the detection of lip movements can be used to further increase precision. Furthermore, we show that even a rather simplistic fusion of gaze and lip movement cues allows a considerable enhancement in addressee estimation, and can be altered to adapt to the requirements of a particular scenario.

The detection of road area in the surroundings of the ego-vehicle is a key issue for modern ADAS. Camera-based direct detection systems are able to reliably accomplish this task only within a limited spatial range or in simple environments, due to hardware limitations and unfavorable situations, like shadows or occlusions. In complex environments, like inner city, traffic is a real issue, since the mere presence of other cars can significantly restrict the field of view of the ego-vehicle. In order to extend the spatial range of road detection, indirect detection systems are a viable resource. They can complement state-of-the-art direct detection systems and help motion control systems to plan smooth and stable trajectories. In this paper we propose a probabilistic grid-based approach based on the interpretation of the motion of other vehicles in the scene. The approach uses the position and velocity of those vehicles in order to infer the presence and location of occluded road area. We will show that this approach can complement an already established feature-based detection system, taking advantage of those situations that are the most challenging for the latter. Evaluations on real-world scenes show that the union between this approach and direct road detection significantly extends the spatial range of detection, thus is able to provide a motion control system a longer horizon for planning trajectories.

We present a concept for long-term feedback during robot assisted indoor cycling training. Our feedback model captures different aspects from sport motivation theory. Furthermore, we present our designed measurements to evaluate the robot's persuasiveness and user's compliance. We conducted an intensive 18-day isolation study in two campaigns (e.g. socially assistive robot vs. display instructed, n=16) in cooperation with the German Aerospace Center. The results show that users tend to comply to the robot's instructions and that there is a significant difference in compliance between the two conditions.

Detecting the road terrain ahead of the ego-vehicle is an important issue for modern driver assistance systems. In particular, vehicle motion planning in inner city environment requires the detection of road terrain up to 3 seconds in advance. State-of-the-art visual road terrain detection systems have a hard time fulfilling this task, due to their limited range and the presence of occlusions (other vehicles, buildings, etc.), which are expected to occur often in complex scenarios. However, those systems provide significant information where the conditions are favorable (proximity to the ego-vehicle, no occlusions). Therefore, a complementary approach is needed to enhance already existing and established detection systems. In this paper we propose a probabilistic grid-based approach based on the observation and interpretation of other vehicles' behavior in the scene. It exploits their movements in order to infer the presence and location of occluded road surface. We will show that this approach presents various advantages over current visual road terrain detection systems, especially in those situations that are the most challenging for them. We will illustrate how our approach is designed to work in concert also with other available resources, e.g. offline road maps. Qualitative results on real-world scenes taken from the KITTI benchmark[11] demonstrate that the fusion of this method with visual road terrain detection can potentially extend our time horizon well over the 3 seconds mentioned above. Finally, we will show how our approach is planned to develop into a semantically enriched representation of the road, including road properties such as availability, lanes and directions.

This paper presents results of a first study for a human-robot-interaction in which the robot Flobi autonomously plays the game 'pairs' against a human player. Integrated is a dialog system which can use different dynamic sentence structures for altering the verbal output during the interaction. The system is compared with a dialog system using static sentences with high information detail and a system using static short sentences. The results suggest that using a dynamic system results in a more satisfying gameplay.

The topic of motivation is a crucial issue for various human-robot interaction (HRI) scenarios. Interactional aspects of motivation can be studied in human-human interaction (HHI) and build the basis for modeling a robot's interactional conduct. Using an ethnographic approach we explored the factors relevant in the formation of motivation-relevant processes in an indoor-cycling activity. We propose an interactive, action-based motivation model for HRI that has been implemented in an autonomous robot system and tested during a long-term HRI study. The model is based on micro-analyses of human indoor cycling courses and resulted in an adaption of specific dialog patterns for HRI. A qualitative evaluation - accompanied by a quantitative analysis - demonstrated that the transfer of interaction patterns from HHI to HRI was successful with participants benefitting from the interaction experience (e.g., performance, subjective feeling of being motivated).

We present a method which is able to adapt from a generic facial representation to a person-specific model of a face. It is referred to as Adaptive Constrained Polynomial Trees (ACPT). Especially in vehicle driving scenarios, special assumptions can be made. A generic facial representation which is able to handle many different persons can be specialized to the current driver to cope with his/her individual face and his/her individual facial features. This leads to a more robust extraction of specified points in the face like nose tip or mouth corners. The proposed method is trained on the LFPW and tested on the FGnet “talking face” dataset. It can be shown, that the presented adaptive model is able to outperform the presented generic facial representation approach. These promising results can be used for further analysis of the driver.

Before taking a decision, a driver anticipates the future behavior of other traffic participants. However, if a driver is inattentive or overloaded, he may fail to consider relevant information. This can lead to bad decisions and potentially result in an accident. A computational system that is designed to anticipate other traffic participants' behaviors could assist the driver in his decision making by sending him an early warning when a risk of collision is predicted. Existing research in this area usually focuses on only one of two aspects, i.e., quality or scope. Quality refers to the ability to warn a driver early before a dangerous situation happens. Scope is the diversity of scenarios in which the approach can work. In general, we see methods targeting a broad scope but showing low quality, with others having a narrow scope but high quality. Our goal is to create a system with high quality and high scope. To achieve this, we propose an architecture that combines classifiers to predict behaviors for many scenarios. In this paper, we will first introduce the generic concept of such a system applicable to highway and inner-city scenarios. We will show that a combination of general and specific classifiers is a solution to improve quality and scope based on a concrete implementation for lane-change prediction in highway scenarios.

For future driver assistance systems and autonomous vehicles, the road course, i.e., the width and shape of the driving path, is an important source of information. In this paper, we introduce a new hierarchical two-stage approach for learning the spatial layout of road scenes. In the first stage, base classifiers analyze the local visual properties of patches extracted from monocular camera images and provide metric confidence maps. We use classifiers for road appearance, boundary appearance, and lane-marking appearance. The core of the proposed approach is the computation of SPatial RAY (SPRAY) features from each metric confidence map in the second stage. A boosting classifier selecting discriminative SPRAY features can be trained for different types of road terrain and allows capturing the local visual properties together with their spatial layout in the scene. In this paper, the extraction of road area and ego-lane on inner-city video streams is demonstrated. In particular, the detection of the ego-lane is a challenging semantic segmentation task showing the power of SPRAY features, because on a local appearance level, the ego-lane is not distinguishable from other lanes. We have evaluated our approach operating at 20 Hz on a graphics processing unit on a publicly available data set, demonstrating the performance on a variety of road types and weather conditions.

To drive safely, a good driver observes his surroundings, anticipates the actions of other traffic participants and then decides for a maneuver. But if a driver is inattentive or overloaded he may fail to include some relevant information. This can then lead to wrong decisions and potentially result in an accident. In order to assist a driver in his decision making, Advanced Driver Assistance Systems (ADAS) are becoming more and more popular in commercial cars. The quality of these existing systems compared to an experienced driver is weak, because they rely purely on physical observation and thus react shortly before an accident. For an earlier warning of the driver behavior prediction is used. We classify existing research in this area with respect to two aspects: quality and scope. Quality means the ability to warn a driver early before a dangerous situation. Scope means the diversity of scenes in which the approach can work. In general we see two tendencies, methods targeting for broad scope but having low quality and those targeting for narrow scope but high quality. Our goal is to have a system with high quality and wide scope. To achieve this, we propose a system that combines classifiers to predict behaviors for many scenarios. To show that a combination of general and specific classifiers is a solution to improve quality and scope, this paper will introduce the generic concept of our system followed by a concrete implementation for lane change prediction for highway scenarios.

Assigning the ego-vehicle to a lane is not only beneficial for navigation but will be an essential element in future Advanced Driver Assistance Systems. This paper describes an approach for ego-lane index estimation using only a monocular camera and no additional sensing equipment like, e.g., the typically employed GPS and Inertial Measurement Unit. Key aspect of the approach are SPatial RAY (SPRAY) features which represent the spatial layout of the road in the visual scene. The proposed method perceives a variety of local visual properties of the scene by means of base classifiers operating on patches extracted from camera images. The spatial arrangement of these local visual properties are captured using SPRAY features. With a boosting classifier trained on these features the ego-lane index is obtained. The system is evaluated on low traffic density and complementary to an object-based approach suitable for heavy traffic. In the conducted experiments, the proposed approach reaches recognition rates of 93% to 97% on individual highway images without applying any kind of temporal filtering.

Users draw on four sources to judge a robot's competence: (1) the robot's voice, (2) physical appearance of and (3) the interaction experience with the robot but also (4) the relationship between the robot's physical appearance and its conduct. Furthermore, most approaches in social robotics have an outcome-oriented focus and thus use questionnaires to measure a global evaluation of the robot after interaction took place. The present research takes a process-oriented approach to explore the factors relevant in the formation of users' attitudes toward the robot. To do so, an ethnographic approach (Conversation Analysis) was employed to analyze the micro-coordination between user and robot. We report initial findings from a study in which a robot took the role of a fitness instructor. Our results emphasize that the participant judges step-by-step the robot's capabilities and differentiates its competence on two levels regarding to the robot's role: a robot as a (1) social/interactional co-participant and as a (2) fitness instructor.

In this paper a novel approach for road detection with a monocular camera is introduced. We propose a two step approach, combining a patch-based segmentation with additional boundary detection. We use Slow Feature Analysis (SFA) which leads to improved appearance descriptors for road and non-road parts on patch level. From the slow features a low order feature set is formed which is used together with color and Walsh Hadamard texture features to train a patch-based GentleBoost classifier. This allows extracting areas from the image that correspond to the road with a certain confidence. Typically the border regions between road and non-road have the highest classification error rates, because the appearance is hard to distinguish on the patch level. Therefore we propose a post-processing step with a specialized classifier applied to the boundary region of the image to improve the segmentation results. In order to evaluate the quality of road segmentation we propose an application-based quality measurement applying an inverse perspective mapping on the image to obtain a Birds Eye View (BEV). The advantage of this approach is that the important distant parts and boundaries of the road in the real world, which are only a low fraction in the perspective image, can be assessed in this metric measure significantly better than on the pixel level. In addition, we estimate the driving corridor width and boundary error, because for Advanced Driver Assistant Systems (ADAS) metric information is needed. For all evaluations in different road and weather conditions, our system shows an improved performance of the two step approach compared to the basic segmentation.

We present a flexible and scalable architecture that can learn to predict the future behavior of a vehicle in inner-city traffic. While behavior prediction studies have mainly been focusing on lane change events on highways, we apply our approach to a simple inner-city scenario: approaching a traffic light. Our system employs dynamic information about the current ego-vehicle state as well as static information about the scene, in this case position and state of nearby traffic lights.

The recognition and prediction of intersection situations and an accompanying threat assessment are an indispensable skill of future driver assistance systems. This study focuses on the recognition of situations involving two vehicles at intersections. For each vehicle, a set of possible future motion trajectories is estimated and rated based on a motion database for a time interval of 2-4 s ahead. Possible situations involving two vehicles are generated by a pairwise combination of these individual motion trajectories. An interaction model based on the mutual visibility of the vehicles and the assumption that a driver will attempt to avoid a collision is used to rate possible situations. The correspondingly favoured situations are classified with a probabilistic framework. The proposed method is evaluated on a real-world differential GPS data set acquired during a test drive of about 10 km, including three road intersections. Our method is typically able to recognise the situation correctly about 1.5-3 s before the last vehicle has passed its minimum distance to the centre of the intersection.

The recognition and prediction of situations is an indispensable skill of future driver assistance systems. This study focuses on the recognition of situations involving two vehicles at intersections. For each vehicle, a set of possible future motion trajectories is estimated and rated based on a motion database for a time interval of 2-4 seconds ahead. Realistic situations are generated by a pairwise combination of these individual motion trajectories and classified according to nine categories with a polynomial classifier. In the proposed framework, situations are penalised for which the time to collision significantly exceeds the typical human reaction time. The correspondingly favoured situations are combined by a probabilistic framework, resulting in a more reliable situation recognition and collision detection than obtained based on independent motion hypotheses. The proposed method is evaluated on a real-world differential GPS data set acquired during a test drive of 10 km, including three road intersections. Our method is typically able to recognise the situation correctly about 1-2 seconds before the distance to the intersection centre becomes minimal.

In this paper we present a new system for a mobile robot to generate an articulated scene model by analyzing complex dynamic 3D scenes. The system extracts essential knowledge about the foreground, like moving persons, and the background, which consists of all visible static scene parts. In contrast to other 3D reconstruction approaches, we suggest to additionally distinguish between static parts, like walls, and movable objects like chairs or doors. The discrimination supports the reconstruction process and additionally, delivers important information about interaction objects. Here, the movable object detection is realized object independent by analyzing changes in the scenery. Furthermore, in the proposed system the background scene is feedbacked to the tracking part yielding a much better tracking and detection result which improves again the 3D reconstruction. We show in our experiments that we are able to provide a sound background model and to extract simultaneously persons and object regions representing chairs, doors, and even smaller movable objects.

The recognition of potentially hazardous situations on road intersections is an indispensable skill of future driver assistance systems. In this context, this study focuses on the task of vehicle tracking in combination with a long-term motion prediction (1-2 s into the future) in a dynamic scenario. A motion-attributed stereo point cloud obtained using computationally efficient feature-based methods represents the scene, relying on images of a stereo camera system mounted on a vehicle. A two-stage mean-shift algorithm is used for detection and tracking of the traffic participants. A hierarchical setup depending on the history of the tracked object is applied for prediction. This includes prediction by optical flow, a standard kinematic prediction, and a particle filter based motion pattern method relying on learned object trajectories. The evaluation shows that the proposed system is able to track the road users in a stable manner and predict their positions at least one order of magnitude more accurately than a standard kinematic prediction method.

In this paper we introduce a method to combine bottom-up saliency, texture descriptors and top-down attention to direct the focus on arbitrary objects in a human-robot interaction scenario. In a usual bottom-up attention process interesting features are extracted out of the image to focus on salient regions. We expand this approach with a top-down process, which determines in one single image the most discriminative features that separate fore- and background best for the observed context. In order to enable the robot also focussing on complex objects like humans, we propose to use a simple model consisting of multiple regions. Thus, the position of an object can be estimated using a probability distribution generated by the attention process. Finally, the system is unrestricted in terms of scene constraints or object appearance and can deal with the requirements of a mobile robot scenario. The attention system has been evaluated both in a human-robot interaction scenario, directing the attention towards interaction partners and in an object learning scenario, focusing on objects. Our results show that the system performed robustly in different applications without any changes to the algorithm.

BACKGROUND: The subcellular localisation of proteins in intact living cells is an important means for gaining information about protein functions. Even dynamic processes can be captured, which can barely be predicted based on amino acid sequences. Besides increasing our knowledge about intracellular processes, this information facilitates the development of innovative therapies and new diagnostic methods. In order to perform such a localisation, the proteins under analysis are usually fused with a fluorescent protein. So, they can be observed by means of a fluorescence microscope and analysed. In recent years, several automated methods have been proposed for performing such analyses. Here, two different types of approaches can be distinguished: techniques which enable the recognition of a fixed set of protein locations and methods that identify new ones. To our knowledge, a combination of both approaches--i.e. a technique, which enables supervised learning using a known set of protein locations and is able to identify and incorporate new protein locations afterwards--has not been presented yet. Furthermore, associated problems, e.g. the recognition of cells to be analysed, have usually been neglected. RESULTS: We introduce a novel approach to automated protein localisation in living cells. In contrast to well-known techniques, the protein localisation technique presented in this article aims at combining the two types of approaches described above: After an automatic identification of unknown protein locations, a potential user is enabled to incorporate them into the pre-trained system. An incremental neural network allows the classification of a fixed set of protein location as well as the detection, clustering and incorporation of additional patterns that occur during an experiment. Here, the proposed technique achieves promising results with respect to both tasks. In addition, the protein localisation procedure has been adapted to an existing cell recognition approach. Therefore, it is especially well-suited for high-throughput investigations where user interactions have to be avoided. CONCLUSION: We have shown that several aspects required for developing an automatic protein localisation technique--namely the recognition of cells, the classification of protein distribution patterns into a set of learnt protein locations, and the detection and learning of new locations--can be combined successfully. So, the proposed method constitutes a crucial step to render image-based protein localisation techniques amenable to large-scale experiments.

UNLABELLED: We present a method for automatic test case generation for protein-protein docking. A consensus-type approach is proposed processing the whole PDB and classifying protein structures into complexes and unbound proteins by combining information from three different approaches (current PDB-at-a-glance classification, search of complexes by sequence identical unbound structures and chain naming). Out of this classification test cases are generated automatically. All calculations were run on the database. The information stored is available via a web interface. The user can choose several criteria for generating his own subset out of our test cases, e.g. for testing docking algorithms. AVAILABILITY: http://bibiserv.techfak.uni-bielefeld.de/agt-sdp/ CONTACT: [email protected].

We describe a method to model pronunciation variation for ASR in a data-driven way, namely by use of automatically derived acoustic subword units. The inventory of units is designed so as to produce maximal separable pronunciation variants of words while at the same time only the most important variants for the particular application are trained. In doing so, the optimal number of variants per word is determined iteratively. All this is accomplished (almost) fully automatically by use of a state splitting algorithm and a variant distance measure. Compared to a baseline system using triphones as subword units and with minimal pronunciation variants, this method achieved a relative improvement of the word error rate by 10%.

In automatic speech recognition complex state spaces are searched during the recognition process. By limiting these search spaces the computation time can be reduced, but unfortunately the recognition rate mostly decreases, too. However, especially for time-critical recognition tasks a search-space pruning is necessary. Therefore, we developed a dynamic mechanism to optimize the pruning parameters for time-constrained recognition tasks, e.g. speech recognition for robotic systems, in respect to word accuracy and computation time. With this mechanism an automatic speech recognition system can process speech signals with an approximately constant processing rate. Compared to a system without such a dynamic mechanism and the same time available for computation, the variance of the processing rate is decreased greatly without a significant loss of word accuracy. Furthermore, the extended system can be sped up to real-time processing, if desired or necessary.

In automatic speech recognition MFCC or LPCC are features commonly used today. However, their calculation considers only a few features of the auditory system. On the assumption that the human representation of speech is an optimal representation, considering more features of the auditory system might lead to a better performance of automatic speech recognition systems. In this paper a model proposed by Strope and Alwan (see references), which relies on the human acoustic perception and allows to consider the effect of forward masking, is incorporated after some modifications into an automatic speech recognition system with a MFCC-based front-end. The extended system is evaluated on recognition tasks, that are closer to real recognition than (connected) digit recognition commonly used in the literature. The evaluations show an increased robustness of the speech recognition system with forward masking on all recognition tasks, but especially on data recorded in noisy environments.

We present an integrated approach combining region and contour-based techniques to enhance both segmentation and recognition processes. This cue integration operates on the level of contour-based groups and complete regions, which are matched to reflect a common cause in the image (and thus in the scene). Additionally, we realize a top-down scheme controlling the segmentation process on the basis of unresolvable image areas and the integration of processes on different time scales using a region memory. Results for real images are given showing distinct improvements of recognition results.

A major aspect of spontaneous dialogs is resembled by the collaborative process of communication where participants of a dialog cooperate by the production of utterances. Accordingly, not only independent utterances are produced but an utterance started by one agent may be continued by the other one based on the structural properties provided so far by the initial utterance. For establishing collaboration in dialogs the computer has to cooperate with the user in so far that the instructions of the user are resumed and carried on by the simulation model in that an appropriate continuation is generated by the system. 1.

In this paper, we present a hybrid speech recognizer combining Hidden Markov Models (HMMs) and a polynomial classifier. In our approach the emission probabilities are not modeled as a mixture of Gaussians but are calculated by the polynomial classifier. However, we do not apply the classifier directly to the feature vector but we make use of the density values of L Gaussians clustering the feature space. That means we model the emission probability as a polynomial of Gaussian distributions of n-th degree. As most of these density values are approximately zero for a single feature vector the calculation of a polynomial can be done very efficiently. The usefulness of this hybrid approach was successfully tested on a large conversational speech recognition task. 1.

Common speech recognition systems use phonetically motivated subword units. To utilize words in these systems, one has to translate the available graphemic word representation into a phonetic one. To reduce this manual effort we propose to build grapheme based recognition systems. They can be used as speech interfaces for devices that can provide a graphemic representation of words like city names of navigation systems. Results of experiments on a 10,000 word lexicon of German cities are presented. 1.

We present a hybrid approach attaching probabilistic formalisms, as artificial neural networks or hidden Markov models, to concepts of a semantic network for a robust and efficient detection of objects. Additionally, an efficient processing strategy for image sequences is outlined which propagates the structural results of the semantic network as an expectation for the next image. This method allows one to produce linked results over time supporting the recognition of events and actions.

This paper presents an HMM-based recognition system for perceptive relevant pitch movements of spontaneous German speech. The pitch movements are defined according to the perceptively and phonetically motivated IPO-approach to intonation. For recognition we use a hybrid approach combining polynomial classification with Hidden Markov Modelling. The recognition is based only on the speech signal, its fundamental frequency and eleven derived features. We evaluate the system on a speaker independant recognition task. 1 Introduction In current speech recognition systems, usually, no prosodic information is used. However, it is a wellknown assumption that prosody can contribute useful information to enhance speech recognition and understanding processes [6]. While for speech recognition, there is no doubt that recognition processes should be resulting in a sequence of word hypotheses for prosody recognition the chosen units depend on several competing linguistic theories. Although their ad...

The model presented here for parsing spoken German offers a way to proceed incrementally discontinuous constructions. Typical constructions in task oriented dialogs are Extrapositions to the right. These are defined as syntactical constructions where a constituent is extraposed in the Nachfeld. Basing on the assumption that the extraposed constituent is not part of the source sentence the model works by coordinating the syntactical information given by the source sentence and the extraposed constituent to complete the extaposed constituent to a whole sentence. Therefore, the standard LR(1)-parser is extended by two additional actions. The parsing strategy works by deriving the first part of the construction until the extraposed constituent is reached. The both new actions enable the parser to proceed further by taking the syntactical information of the source sentence to complete the extraposed constituent to a sentence of its own. The repeated use of these actions guarantees that every intended reading will be performed.

In this paper, we present an environment for labelling and testing of melodic aspects of spoken language. The environment has three modes of application: First, the environment provides labelling facilities for a model-based melodic description for German. Second, it supports a language independent pre-theoretical description of speech melody allowing the development of new melodic categories. Third, our test bed can be used to generate speech samples with controlled melodic parameters for further use in perception experiments. The melodic description facilities (model-based, pre-theoretical) are supported by visual and audible feedback allowing a step-bystep refinement of the melodic description in question. 1 Introduction Melodic aspects of speech are related with several linguistic and extralinguistic phenomenon. Linguistic aspects are for example the realization of accents or the marking of boundaries. Extralinguistic aspects concern e.g. speaker related qualities like emotions. T...

We present a hybrid system that integrates speech and image understanding. Given spoken references, it is able to identify objects of a 3D scene perceived via a stereo camera. Central to our approach is the extraction of qualitative object features and spatial scene properties from acoustic and visual data. The interaction of the understanding processes is performed using a procedural semantic network that interfaces with signal recognition and reconstruction modules, thus integrating semantic, neural and Bayesian networks and Hidden Markov Models.

A complete dialogue system within the task domain of making an appointment is presented.It is based on a semantic network representation of linguistic knowledge and a word recognition system that communicates with the interpretation component bidirectionally.System robustness is achieved using a special metascore that evaluates the advance of the linguistic interpretation.

Especially in recognition of spontaneous speech it is necessary to be able to cope with the occurance of unknown words. We present an approach to unknown word detection integrated with speech recognition. Though several schemes exist for the evaluation of detection algorithms all suffer from some deficiencies. Therefore, we also present a new evaluation measure called detection accuracy which is similar to the widely accepted word accuracy. We applied this measure to the evaluation of our approach on a large spontaneous speech recognition task from the German Verbmobil project.

We present a new approach towards using contextual information to enhance speech recognition and understanding. Dynamically inferred knowledge about the context is used in addition to the static linguistic and domain specific knowledge. Based on the results of image analysis of a given scene language models for constituents of possible utterances concerning that scene are generated.

The vast majority of speech understanding systems suffers from a bottleneck between the recognition and the interpretation components. Normally, only a relatively small set of word hypotheses is passed from the recognizer and no flow of information in the opposite direction is even possible. We propose an interaction scheme that tries to overcome many of the disadvantages of traditional systems. It makes use of the possibility to process abstract constituents in our word recognizer and pass them back as complex hypotheses. Predictions that define the complex analysis goal of the recognition can be derived dynamically during the interpretation of an utterance. A left-to-right processing in both recognition and interpretation makes an incremental analysis possible.

The analysis and interpretation of time constituents is a rather complex enterprise, since diverse time constituents are distributable in variable positions within an utterance. The first step in order to manage this complexity and variability is the syntactic analysis at phrase structure level. Within a utterance each time constituent is analyzed independently and tested for its syntactic coherence. A semantic interpretation of the time constituent has to follow. The second step consists of the analysis and interpretation at sentence structure level. The time interpretations need to be tested for consistency and merged into a single representation. Here it is usually possible to resolve ambiguities. As a last step, the interpretations of time constituents have to be merged at dialogue level. Although the system asks the user for verification of the time computed, users do seldom just reply ’ or ’. Mostly they add new information about time, and sometimes users correct the system’s interpretation without explicit negation. Thus, the merging of time constituents at dialogue level becomes a rather complex issue

This article presents the speech understanding and dialog system EVAR. All levels of linguistic knowledge are used both to control the analysis process and for the interpretation of an utterance. All kinds of knowledge are integrated in a homogeneous knowledge base. The control algorithm used for the analysis is defined within the representation scheme and does not depend on the application. One of the aims of EVAR is to develop a system structure where linguistic and nonlinguistic expectations could be used not only for the interpretation but also as predictions for the recognition process.>

In all fields of pattern recognition there may arise situations where severe distortions of the sensor data or errors of processing prevent a successful analysis. This is especially true in speech recognition. Therefore, a speech understanding system can not rely on being able to interpret the whole or even a given fixed percentage of the input data. Rather a dynamic criterion has to be applied to decide when the analysis has produced the best results obtainable from the input data. We propose an appropriate criterion and show how the requirements for robust interpretation of speech can be met in a dialog system. Keywords: Speech Understanding, Robustness, Search 1. INTRODUCTION In speech understanding systems a main problem is how to deal with poor segmentation results. How can it be decided that no or at least only a partial interpretation of the input data is possible? Let us consider the following two dialogs where the user's utterance (U), the word recognition result (R), and the...

Semantic Hidden Markov Networks (SHMNs) were first introduced in [2] as a new technique of interfacing between linguistic analysis and word recognition in speech understanding systems. The main difference between SHMNs and the use of traditional language models is that SHMNs always refer to a linguistic concept and impose the linguistic structure as closely as possible on its acoustic counterpart --- a hierarchically structured HMM. Normally the result of decoding a HMM is merely the sequence of best fitting elementary acoustic concepts, e.g. phonemes or words. Taking into account the structure of the recognition task a structured instance can be computed. This complex acoustic instance can easily be transformed into a linguistic instance by a recursive computation but without any searching. In this paper we present an algorithm for generating linguistic instances from word recognition results based on SHMNs. Additionally, we present recognition results obtained when evaluating a set o...

The analysis and interpretation of time constituents is important for most applications of speech understanding systems. Problems can be caused by the varying distribution of constituents. A basic set of time constituents were found in a corpus of domain specific (train schedule) utterances. A distributed representation of surface structure models and an incremental semantic analysis is used to manage the complexity. The knowledge base of the speech understanding system that provides the framework for the analysis and interpretation of time constituents uses the semantic network language ERNEST. Keywords: Speech Understanding, Time Constituents 1. INTRODUCTION In a speech understanding system there are several domains of analysis and interpretation. Firstly, the system has to recognize single words in a torrent of speech sounds. Secondly, the system combines words to constituents, i.e. it performs a syntactic analysis. It also has to reconstruct the meaning of the utterance in questio...

Presents a problem-independent control algorithm for image understanding providing both data-driven and model-driven control structures. By an easy combination of these structures any mixed strategy can be achieved. The basis is a framework for the representation of declarative and procedural knowledge using a semantic network.>

Although much effort has been put into speech understanding systems there still exists a rather wide gap between acoustic recognition and linguistic interpretation. We propose a formalism for an extremely close interaction of acoustic recognition and higher level analysis. Instead of a strict horizontal interface at the level of hypothesized word sequences or lattices, a vertical interface to the acoustic component is used that can be accessed from linguistic concepts of any degree of abstraction. As the linguistic knowledge is represented in the formalism of Semantic Networks and acoustic recognition is based on Hidden Markov Models the close interaction between the two components was termed Semantic Hidden Markov Networks. 1 INTRODUCTION Because of the high degree of uncertainty in the recognition of spoken language it is very important to exploit any possible predictions and restrictions to guide acoustic analysis as well as linguistic interpretation. Within traditional systems the...

This article describes the dialog module of EVAR. The dialog is seen as a sequence of dialog acts uttered by the system and the user. From a corpus of real humanhuman dialogs a model was extracted. The model covers all sequences of dialog acts observed in the corpus. Each dialog act is modelled as a set of pragmatic, semantic and syntactic concepts. The properties of the concepts and the current dialog state are used to identify the actual dialog act. Each user utterance is interpreted by the system and the system reacts with the appropriate dialog act. After the user has defined his request completly, the system starts a database query and finally generates a synthesized natural language answer. The answer generation is realised with sentence masks, where the information from the database enquiry is filled in at the appropriate slots. A dialog memory which allows the interpretation of elliptical sentences and proforms is updated after each utterance.

The authors give a detailed account of a system environment for treating general problems of image and speech understanding. A framework for representing declarative and procedural knowledge based on a suitable definition of a semantic network is provided. The syntax and semantics of the network are clearly defined. The pragmatics of the network in its use for pattern understanding are defined by several rules which are problem-independent. This allows one to formulate problem-independent control algorithms. Complete software environments are available to handle the described structures. The general applicability of the network system is demonstrated by short descriptions of three applications from different task domains.>

A flexible control strategy for a speech understanding and dialog system is presented. The system is organized around a homogeneous knowledge base. The used knowledge representation language is based on semantic networks. It does not only cover declarative but also procedural knowledge. The inference processes can be described by five problem-independent rules which form the procedural semantics of the network language. The control strategy itself is the well-known A*-algorithm, but for scoring purposes complex judgement vectors are used. It is shown that these vectors are admissible for the control algorithm and that they reflect the quality, the certainty, and the priority of both bottom-up and top-down hypotheses. So far, the system has been tested in a rough prototype version, and an efficient realization is being built.>