This paper evaluates the performance of a multimodal interface under exerted conditions in a natural field setting. The subjects in the present study engaged in a strenuous activity while multimodally performing map-based tasks using handheld computing devices. This activity made the users breathe heavily and become fatigued during the course of the study. We found that the performance of both speech and gesture recognizers degraded as a function of exertion, while the overall multimodal success rate was stable. This stabilization is accounted for by the mutual disambiguation of modalities, which increases significantly with exertion. The system performed better for subjects with a greater level of physical fitness, as measured by their running speed, with more stable multimodal performance and a later degradation of speech and gesture recognition as compared with subjects who were less fit. The findings presented in this paper have a significant impact on design decisions for multimodal interfaces targeted towards highly mobile and exerted users in field environments.

Conversation protocols are used to achieve certain goals or to bring about certain states in the world. Therefore, one may identify the landmarks or the states that must be brought about during the goal–directed execution of a protocol. Accordingly, the landmarks, characterized by propositions that are true in the state represented by that landmark, are the most important aspect of a protocol. Families of conversation protocols can be expressed formally as partially ordered landmarks after the landmarks necessary to achieve a goal have been identified. Concrete protocols represented as joint action expressions can, then, be derived from the partially ordered landmarks and executed directly by joint intention interpreters. This approach of applying Joint Intention theory to protocols also supports flexibility in the actions used to get to landmarks, shortcutting protocol execution, automatic exception handling, and correctness criterion for protocols and protocol compositions.