This paper describes a multimodal database which consists of image data of human gestures and corresponding speech data for the research on multimodal interaction systems.The purpose of this database is to provide an underlying foundation for research and development of multimodal interactive systems.Our primary concern in selecting utterances and gestures for inclusion in the database was to ascertain the kinds of expressions and gestures that artificial systems could produce and recognize.Total 25 kinds of gestures and speech were repeated four times for the recording of each subject.The speech and gestures for a total of 48 subjects were recorded, converted into files and in the first version, the files for 12 subjects were recorded on CD-ROMs.

In this paper, we describe a multimodal interface prototype system based on Dynamical Dialogue Model. This system not only integrates information of speech and gestures, but also controls the response timing in order to realize a smooth interaction between user and computer. Our approach consists of human-human dialogue analysis, and computational modeling of dialogue.

In this paper, we propose a response model for a multimodal human interface, by inserting listener responses particular times by detecting keywords from the user’s utterances and controlling the face direction of a humanlike Computer Graphics (CG) character according to the direction of the user’s attention which is determined by tracing user’s face. Then we integrated the response model into a prototype system, and evaluated its efficiency.