This is a large extension to a previous paper presented in LREC 2006 [6]. It describes the motivation, collection and format of the MIMUS corpus, as well as an in-depth and issue-focused analysis of the data. MIMUS [8] is the result of multimodal WoZ experiments conducted at the University of Seville as part of the TALK project. The main objective of the MIMUS corpus was to gather information about different users and their performance, preferences and usage of a multimodal multilingual natural dialogue system in the Smart Home scenario in Spanish. The focus group is composed by wheel-chair-bound users, because of their special motivation to use this kind of technology, along with their specific needs. Throughout this article, the WoZ platform, experiments, methodology, annotation schemes and tools, and all relevant data will be discussed, as well as the results of the in-depth analysis of these data. The corpus compresses a set of three related experiments. Due to the limited scope of this article, only some results related to the first two experiments (1A and 1B) will be discussed. This article will focus on subject's preferences, multimodal behavioural patterns and willingness to use this kind of technology.

One of the most widely pursued goals in dialogue system development is the improvement of usability, which is mainly achieved by providing users with both friendly and manageable interfaces. The MIMUS dialogue system supports multimodal interactions, allowing the user to interact not only verbally but also graphically. In addition to this, MIMUS (MultIModal, University of Seville) is a multilingual system that enables the user to communicate (dynamically) in Spanish and English. The present paper describes the MIMUS architecture, the components that entail direct relationship with multilinguality, and the way in which languages can be dynamically switched within a single dialogue. Index Terms : dialogue systems, multimodal, multilingual, dynamic language change

This is a new hybrid fusion strategy based primarily on the implementation of two former and differentiated approaches to multimodal fusion [11] in multimodal dialogue systems. Both approaches, their predecessors and their respective advantages and disadvantages will be described in order to illustrate how the new strategy merges them into a more solid and coherent solution. The first strategy was largely based on Johnston's approach [5] and implies the inclusion of multimodal grammar entries and temporal constraints. The second approach implied the fusion of information coming from different channels at dialogue level. The new hybrid strategy hereby described requires the inclusion of multimodal grammar entries and temporal constraints plus the additional information at dialogue level utilized in the second strategy. Within this new approach therefore, the fusion process will be initiated at grammar level and will be culminated at dialogue level.

This paper describes the architecture of MIMUS, a multimodal, multilingual interaction system which allows users to control the devices in their houses, interacting by voice and clicks. MIMUS design relies on Wizard of Oz experiments and is targeted at disabled users in the in-home scenario. MIMUS consists of a set of OAA collaborative agents with a clear distinction between knowledge resources, dialogue management and domain specific functionalities. The MIMUS system includes contributions to the scientific community in multimodal fusion, knowledge management and presentation strategies.