Eva is a multimodal conversational framework for building planning-based systems that help users accomplish their domain goals through collaborative dialogue. We argue that planning-based systems can do this by inferring users’ intentions, adopting goals to achieve them, developing plans to achieve those goals, detecting whether obstacles are present, finding plans to overcome them, and planning their actions, including speech acts. In the Eva framework, conversational agents can maintain and reason with their own beliefs, goals and intentions, and explicitly reason about those of their users. Belief reasoning is accomplished with a modal Horn-clause meta-interpreter. The planning and reasoning subsystems obey the principles of persistent goals and intentions, including the formation and decomposition of intentions to perform complex actions, as well as the conditions under which they can be given up. In virtue of its planning process, Eva treats its speech acts just like its other actions – physical acts affect physical states, digital acts affect digital states, and speech acts affect mental and social states. This general framework enables systems to plan a variety of speech acts including requests, informs, questions, confirmations, recommendations, offers, acceptances, greetings, and emotive expressions. Each of these has a formally specified semantics which is used during the planning and reasoning processes. Because Eva-based agents can keep track of different users’ mental states, they can engage in multi-party dialogues. Importantly, the framework supports systems’ explanations of their actions because they have created plans standing behind each of them. As a reaction to the near-universal focus on using large language models for every application, a trend has emerged recently towards integrated neuro-symbolic architectures. The Eva framework is an example of such an architecture in the area of collaborative dialogue systems.

Teachable robots in education have the ability to increase student engagement and learning through personalised interactions and the use of social behaviours, such as speech, gaze and emotional expressions. Adaptation of these behaviours is motivated by an interest in tailoring the learning experience to an individual user and improving user outcomes. This work proposes an adaptive response-selection algorithm for a teachable robot which aims to increase user task engagement. A Q-learning algorithm learns an individualised policy and is rewarded based on the user’s paraphrasing behaviour and response time when teaching the robot. A user study is conducted across two user groups, recruited from Australia and Japan. This study evaluates the performance of the adaptive approach for response selection against a non-adaptive method and explores the differences in response and perception of the teaching task between the two participant groups. The results show that measures of task engagement increase more when using the adaptive approach compared to a non-adaptive method of response selection, but that this difference is not consistent across both participant groups. The adaptive approach is also shown to have a positive effect on user perceptions of the interaction.

Multilingual semantic parsing aims to leverage the knowledge from the high-resource languages to improve low-resource semantic parsing, yet commonly suffers from the data imbalance problem.Prior works propose to utilize the translations by either humans or machines to alleviate such issues.However, human translations are expensive, while machine translations are cheap but prone to error and bias.In this work, we propose an active learning approach that exploits the strengths of both human and machine translations by iteratively adding small batches of human translations into the machine-translated training set.Besides, we propose novel aggregated acquisition criteria that help our active learning method select utterances to be manually translated.Our experiments demonstrate that an ideal utterance selection can significantly reduce the error and bias in the translated data, resulting in higher parser accuracies than the parsers merely trained on the machine-translated data.

Levon Haroutunian, Zhuang Li, Lucian Galescu, Philip Cohen, Raj Tumuluri, Gholamreza Haffari. Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Teachable robots can offer benefits to students through the use of social behaviours, such as speech, gaze, and gestures, to promote engagement and learning. Adapting these behaviours can deliver personalised interactions to better suit each individual. There is a growing body of research utilising reinforcement learning in social robotics, however there is limited research in the use of adaptive dialog behaviours for social robots. We propose an adaptive response-selection algorithm for a teachable robot which aims to improve user engagement in the teaching task. The proposed approach uses Q-learning to learn an individualised policy. The algorithm is rewarded according to the time taken per teaching input, and the amount paraphrasing in the user’s response. A user study has been conducted to evaluate the algorithm, compared to a method of random response-selection. The results indicate that an adaptive approach learns to select more rewarding actions over time, and personalise to the individual user.

This “blue sky” paper argues that future conversational systems that can engage in multiparty, collaborative dialogues will require a more fundamental approach than existing technology. This paper identifies significant limitations of the state of the art, and argues that our returning to the plan-based approach to dialogue will provide a stronger foundation. Finally, I suggest a research strategy that couples neural network-based semantic parsing with plan-based reasoning in order to build a collaborative dialogue manager.

In this paper, we examine the foundations of task-oriented dialogues, in which systems are requested to perform tasks for humans.We argue that the way this dialogue task has been framed has limited its applicability to processing simple requests with atomic "slot-fillers".However, such dialogues can contain more complex utterances.Furthermore, situations for which it would be desirable to build task-oriented dialogue systems, e.g., to engage in collaborative or multiparty dialogues, will require a more general approach.In order to provide such an approach, we give a logical analysis of the "intent+slot" dialogue setting that overcomes these limitations.

Extending semantic parsing systems to new domains and languages is a highly expensive, time-consuming process, so making effective use of existing resources is critical.In this paper, we describe a transfer learning method using crosslingual word embeddings in a sequence-tosequence model.On the NLmaps corpus, our approach achieves state-of-the-art accuracy of 85.7% for English.Most importantly, we observed a consistent improvement for German compared with several baseline domain adaptation techniques.As a by-product of this approach, our models that are trained on a combination of English and German utterances perform reasonably well on codeswitching utterances which contain a mixture of English and German, even though the training data does not contain any code-switching.As far as we know, this is the first study of code-switching in semantic parsing.We manually constructed the set of code-switching test utterances for the NLmaps corpus and achieve 78.3% accuracy on this dataset.

This talk will discuss progress in building collaborative multimodal systems, both systems that offer a collaborative interface that augments human performance, and autonomous systems with which one can collaborate. To begin, I discuss what we will mean by collaboration, which revolves around plan recognition skills learned as a child. Then, I present a collaborative multimodal operations planning system, Sketch-Thru-Plan, that enables users to interact multimodally with speech and pen as it attempts to infer their plans. The system offers suggested actions and allows the user to confirm/disconfirm those suggestions. I show how the collaborative multimodal interface enables more rapid task performance and higher user satisfaction than existing deployed GUIs built for the same task.

This paper demonstrates a multimodal system called Sketch-Thru-Plan (STP) that enables users to speak and draw doctrinal language and symbols in order to create courses of action. We argue that STP can meet many of the challenges inherent in building user interfaces for operations planning and command-and-control. The system is being transitioned to military organizations for use in planning courses of action.

Over the past 7 years, Adapx (formerly, Natural Interaction Systems) has been developing digital pen-based natural interfaces for field tasks. Examples include products for field note-taking, mapping and architecture/engineering/construction, which have been applied to such uses as: surveying, wild-fire fighting, land use planning and dispute resolution, and civil engineering. In this talk, I will describe the technology and some of these field-based use cases, discussing why natural interfaces are the preferred means for human-computer interaction for these applications.

The present paper reports on the design and performance of a novel dual-Wizard simulation infrastructure that has been used effectively to prototype next-generation adaptive and implicit multimodal interfaces for collaborative groupwork. This high-fidelity simulation infrastructure builds on past development of single-wizard simulation tools for multiparty multimodal interactions involving speech, pen, and visual input [1]. In the new infrastructure, a dual-wizard simulation environment was developed that supports (1) real-time tracking, analysis, and system adaptivity to a user's speech and pen paralinguistic signal features (e.g., speech amplitude, pen pressure), as well as the semantic content of their input. This simulation also supports (2) transparent user training to adapt their speech and pen signal features in a manner that enhances the reliability of system functioning, i.e., the design of mutually-adaptive interfaces. To accomplish these objectives, this new environment also is capable of handling (3) dynamic streaming digital pen input. We illustrate the performance of the simulation infrastructure during longitudinal empirical research in which a user-adaptive interface was designed for implicit system engagement based exclusively on users' speech amplitude and pen pressure [2]. While using this dual-wizard simulation method, the wizards responded successfully to over 3,000 user inputs with 95-98% accuracy and a joint wizard response time of less than 1.0 second during speech interactions and 1.65 seconds during pen interactions. Furthermore, the interactions they handled involved naturalistic multiparty meeting data in which high school students were engaged in peer tutoring, and all participants believed they were interacting with a fully functional system. This type of simulation capability enables a new level of flexibility and sophistication in multimodal interface design, including the development of implicit multimodal interfaces that place minimal cognitive load on users during mobile, educational, and other applications.

Lecturers, presenters and meeting participants often say what they publicly handwrite. In this paper, we report on three empirical explorations of such multimodal redundancy -- during whiteboard presentations, during a spontaneous brainstorming meeting, and during the informal annotation and discussion of photographs. We show that redundantly presented words, compared to other words used during a presentation or meeting, tend to be topic specific and thus are likely to be out-of-vocabulary. We also show that they have significantly higher tf-idf (term frequency-inverse document frequency) weights than other words, which we argue supports the hypothesis that they are dialogue-critical words. We frame the import of these empirical findings by describing SHACER, our recently introduced Speech and HAndwriting reCognizER, which can combine information from instances of redundant handwriting and speech to dynamically learn new vocabulary.

The availability of metadata annotations over media content such as photos is known to enhance retrieval and organization, particularly for large data sets. The greatest challenge for obtaining annotations remains getting users to perform the large amount of tedious manual work that is required.In this paper we introduce an approach for semi-automated labeling based on extraction of metadata from naturally occurring conversations of groups of people discussing pictures among themselves.As the burden for structuring and extracting metadata is shifted from users to the system, new recognition challenges arise. We explore how multimodal language can help in 1) detecting a concise set of meaningful labels to be associated with each photo, 2) achieving robust recognition of these key semantic terms, and 3) facilitating label propagation via multimodal shortcuts. Analysis of the data of a preliminary pilot collection suggests that handwritten labels may be highly indicative of the semantics of each photo, as indicated by the correlation of handwritten terms with high frequency spoken ones. We point to initial directions exploring a multimodal fusion technique to recover robust spelling and pronunciation of these high-value terms from redundant speech and handwriting.

The availability of metadata annotations over media content such as photos is known to enhance retrieval and organization, particularly for large data sets. The greatest challenge for obtaining annotations remains getting users to perform the large amount of tedious manual work that is required. In this demo we show a system for semi-automated labeling based on extraction of metadata from naturally occurring conversations of groups of people discussing pictures among themselves. The system supports a variety of collaborative label elicitation scenarios mixing co-located and distributed participants, operating primarily via speech, handwriting and sketching over tangible digital paper photo printouts. We demonstrate the real-time capabilities of the system by providing hands-on annotation experience for conference participants. Demo annotations are performed over public domain pictures portraying mainstream themes (e.g. from famous movies).

This paper evaluates the performance of a multimodal interface under exerted conditions in a natural field setting. The subjects in the present study engaged in a strenuous activity while multimodally performing map-based tasks using handheld computing devices. This activity made the users breathe heavily and become fatigued during the course of the study. We found that the performance of both speech and gesture recognizers degraded as a function of exertion, while the overall multimodal success rate was stable. This stabilization is accounted for by the mutual disambiguation of modalities, which increases significantly with exertion. The system performed better for subjects with a greater level of physical fitness, as measured by their running speed, with more stable multimodal performance and a later degradation of speech and gesture recognition as compared with subjects who were less fit. The findings presented in this paper have a significant impact on design decisions for multimodal interfaces targeted towards highly mobile and exerted users in field environments.

We describe an approach to 3D multimodal interaction in immersive augmented and virtual reality environments that accounts for the uncertain nature of the information sources. The resulting multimodal system fuses symbolic and statistical information from a set of 3D gesture, spoken language, and referential agents. The referential agents employ visible or invisible volumes that can be attached to 3D trackers in the environment, and which use a time-stamped history of the objects that intersect them to derive statistics for ranking potential referents. We discuss the means by which the system supports mutual disambiguation of these modalities and information sources, and show through a user study how mutual disambiguation accounts for over 45% of the successful 3D multimodal interpretations. An accompanying video demonstrates the system in action.

In command posts, officers maintain situational awareness using paper maps, Post-it notes, and hand-written annotations. They do so because paper is robust to failure, it is portable, it offers a flexible means of capturing information, it has ultra-high resolution, and it readily supports face-to-face collaboration. We report herein on an evaluation comparing maps and Post-its with a tangible multimodal system called Rasa. Rasa augments these paper tools with sensors, enabling it to recognize the multimodal language (both written and spoken) that naturally occurs on them. In this study, we found that not only do users prefer Rasa to paper alone, they find it as easy or easier to use than paper tools. Moreover, Rasa introduces no discernible overhead in its operation other than error repair, yet grants the benefits inherent in digital systems. Finally, subjects confirmed that by combining physical and computational tools, Rasa is resistant to computational failure

Groups of people involved in collaboration on a task often incorporate the objects in their mutual environment into their discussion. With this comes physical reference to these 3-D objects, including: gesture, gaze, haptics, and possibly other modalities, over and above the speech we commonly associate with human-human communication. From a technological perspective, this human style of communication not only poses the challenge for researchers to create multimodal systems capable of integrating input from various modalities, but also to do it well enough that it supports, but does not interfere with the primary goal of the collaborators, which is their own human-human interaction. This paper offers a first step towards building such multimodal systems for supporting face-to-face collaborative work by providing both qualitative and quantitative analyses of multiparty multimodal dialogues in a field setting.

We describe an augmentation of Quickset, a multimodal voice/pen system that allows users to create and control map-based, collaborative, interactive simulations. In this paper, we report on our extension of the graphical pen input mode front stylus/mouse to 3D hand movements. To do this, the map is projected onto a virtual plane in space, specified by the operator before the start of the interactive session. We then use our geometric model to compute the intersection of hand movements with the virtual plane, translating these into map coordinates on the appropriate system. The goal of this research is the creation of a body-centered, multimodal architecture employing both speech and 3D hand gestures, which seamlessly, and unobtrusively supports distributed interaction. The augmented system, built on top of an existing architecture, also provides an improved visualization, management and awareness of a shared understanding. Potential applications of this work include telemedicine, battlefield management and any kind of collaborative decision-making during which users may wish to be mobile.

The design and implementation of hybrid symbolic/statistical architectures is a major area of interest in current multimodal system development. Such an architecture attempts to improve multimodal recognition and disambiguation rates by using corpus-based statistics to weight the contributions from various input streams. This is in contrast to current architectures that assume independence between input streams, and combine un-weighted posterior probabilities simply by taking their cross product.

Conversation protocols are used to achieve certain goals or to bring about certain states in the world. Therefore, one may identify the landmarks or the states that must be brought about during the goal–directed execution of a protocol. Accordingly, the landmarks, characterized by propositions that are true in the state represented by that landmark, are the most important aspect of a protocol. Families of conversation protocols can be expressed formally as partially ordered landmarks after the landmarks necessary to achieve a goal have been identified. Concrete protocols represented as joint action expressions can, then, be derived from the partially ordered landmarks and executed directly by joint intention interpreters. This approach of applying Joint Intention theory to protocols also supports flexibility in the actions used to get to landmarks, shortcutting protocol execution, automatic exception handling, and correctness criterion for protocols and protocol compositions.

When building a complex pattern recognizer with high-dimensional input features, a number of selection uncertainties arise. Traditional approaches to resolving these uncertainties typically rely either on the researcher's intuition or performance evaluation on validation data, both of which result in poor generalization and robustness on test data. This paper describes a novel recognition technique called members to teams to committee (MTC), which is designed to reduce modeling uncertainty. In particular, the MTC posterior estimator is based on a coordinated set of divide-and-conquer estimators that derive from a three-tiered architectural structure corresponding to individual members, teams, and the overall committee. Basically, the MTC recognition decision is determined by the whole empirical posterior distribution, rather than a single estimate. This paper describes the application of the MTC technique to handwritten gesture recognition and multimodal system integration and presents a comprehensive analysis of the characteristics and advantages of the MTC approach.

Rasa is a tangible augmented reality environment that digitally enhances the existing paper-based command and control capability in a military command post. By observing and understanding the users' speech, pen, and touch-based multimodal language, Rasa computationally augments the physical objects on a command post map, linking these items to digital representations of the same-for example, linking a paper map to the world and Post-itâ notes to military units. Herein, we give a thorough account of Rasa's underlying multiagent framework, and its recognition, understanding, and multimodal integration components. Moreover, we examine five properties of language-generativity, comprehensibility, compositionality, referentiality, and, at times, persistence-that render it suitable as an augmentation approach, contrasting these properties to those of other augmentation methods. It is these properties of language that allow users of Rasa to augment physical objects, transforming them into tangible interfaces.

The influence that language has on contextual interpretations cannot be ignored by computer systems that strive to be context aware. Rather, once systems are designed to perceive language and other forms of human action, these interpretative processes will of necessity be context dependent. As an example, we illustrate how people simply and naturally create new contexts by naming and referring. We then describe Rasa, a mixed-reality system that observes and understands how users in a military command post create such contexts as part of the process of maintaining situational awareness. In such environments, commander's maps are covered with Post-it(r) notes. These paper artifacts are contextually transformed to represent units in the field by the application of multimodal language. Rasa understands this language, thereby allowing paper- based tools to become the basis for digital interaction. Finally, we argue that architectures for such context-aware systems will need to be built to process the inherent ambiguity and uncertainty of human input in order to be effective.

The growing interest in multimodal interface design is inspired in large part by the goals of supporting more transparent, flexible, efficient, and powerfully expressive means of human-computer interaction than in the past. Multimodal interfaces are expected to support a wider range of diverse applications, be usable by a broader spectrum of the average population, and function more reliably under realistic and challenging usage conditions. In this article, we summarize the emerging architectural approaches for interpreting speech and pen-based gestural input in a robust manner-including early and late fusion approaches, and the new hybrid symbolic-statistical approach. We also describe a diverse collection of state-of-the-art multimodal systems that process users' spoken and gestural input. These applications range from map-based and virtual reality systems for engaging in simulations and training, to field medic systems for mobile use in noisy environments, to web-based transactions and standard text-editing applications that will reshape daily computing and have a significant commercial impact. To realize successful multimodal systems of the future, many key research challenges remain to be addressed. Among these challenges are the development of cognitive theories to guide multimodal system design, and the development of effective natural language processing, dialogue processing, and error-handling techniques. In addition, new multimodal systems will be needed that can function more robustly and adaptively, and with support for collaborative multiperson use. Before this new class of systems can proliferate, toolkits also will be needed to promote software development for both simulated and functioning systems.

We present a statistical approach to developing multimodal recognition systems and, in particular, to integrating the posterior probabilities of parallel input signals involved in the multimodal system. We first identify the primary factors that influence multimodal recognition performance by evaluating the multimodal recognition probabilities. We then develop two techniques, an estimate approach and a learning approach, which are designed to optimize accurate recognition during the multimodal integration process. We evaluate these methods using Quickset, a speech/gesture multimodal system, and report evaluation results based on an empirical corpus collected with Quickset. From an architectural perspective, the integration technique presented offers enhanced robustness. It also is premised on more realistic assumptions than previous multimodal systems using semantic fusion. From a methodological standpoint, the evaluation techniques that we describe provide a valuable tool for evaluating multimodal systems.

This paper reports 1 on a case study comparison of a directmanipulation-based graphical user interface (GUI) with the QuickSet pen/voice multimodal interface for supporting the task of military force “laydown. ” In this task, a user places military units and “control measures, ” such as various types of lines, obstacles, objectives, etc., on a map. A military expert designed his own scenario and entered it via both interfaces. Usage of QuickSet led to a speed improvement of 3.2 to 8.7fold, depending on the kind of object being created. These results suggest that there may be substantial efficiency advantages to using multimodal interaction over GUIs for mapbased tasks. 1.

Recent empirical research has shown conclusive advantages of multimodal interaction over speech-only interaction for map-based tasks. This paper describes a multimodal language processing architecture which supports interfaces allowing simultaneous input from speech and gesture recognition. Integration of spoken and gestural input is driven by unification of typed feature structures representing the semantic contributions of the different modes. This integration method allows the component modalities to mutually compensate for each others' errors. It is implemented in Quick-Set, a multimodal (pen/voice) system that enables users to set up and control distributed interactive simulations.

This paper presents an emerging application of multimodal interface research to distributed applications.We have developed the QuickSet prototype, a pen/voice system running on a hand-held PC, communicating via wireless LAN through an agent architecture to a number of systems, including NRaD's' LeatherNet system, a distributed interactive training simulator built for the US Marine Corps.The paper describes the overall system architecture, a novel multimodal integration strategy offering mutual compensation among modalities, and provides examples of multimodal simulation setup.Finally, we discuss our applications experience and evaluation.

This paper describes a new simulation technique designed to support a wide spectrum of empirical studies on the characteristics of spoken, handwritten, and combined pen/voice input to future interactive systems. The simulatidn tech,fique alms: (1) to provide a tool for investigating interactive handwriting and pen systems, on which no simulation research currently is available, (2) to devise a technique appropriate for comparing people's use of speech and writing, such that differences be- tween these communication modalities and their related technologies can be better understood, and (3) to support a very rapid exchange with simulated speech, pen. and pen/voice sys- tems, such th,-tt interactions can be subject-paced. This paper outli,tes the pecifica,ions, general environment. and capabil- ities of a new semi-automatic simulation technique developed to achieve these goals

Although graphics and direct manipulation are effective interface technologies for some classes of problems, they are limited in many ways. In particular, they provide little support for identifying objects not on the screen, for specifying temporal relations, for identifying and operating on large sets and subsets of entities, and for using the context of interaction. On the other hand, these are precisely strengths of natural language. This paper presents an interface that blends natural language processing and direct manipulation technologies, using each for their characteristic advantages. Specifically, the paper shows how to use natural language to describe objects and temporal relations, and how to use direct manipulation for overcoming hard natural language problems involving the establishment and use of context and pronominal reference. This work has been implemented in SRI's Shoptalk system, a prototype information and decision-support system for manufacturing.

No abstract available.

A crucially important adequacy test of any theory of speech acts is its ability to handle performatives. This paper provides a theory of performatives as a test case for our rationally based theory of illocutionary acts. We show why "I request you..." is a request, and "I lie to you that p" is self-defeating. The analysis supports and extends earlier work of theorists such as Bach and Harnish [1] and takes issue with recent claims by Searle [10] that such performative-as-declarative analyses are doomed to failure.

Near-term spoken language systems will likely be limited in their interactive capabilities. To design them, we shall need to model how the presence or absence of speaker interaction influences spoken discourse patterns in different types of tasks. In this research, a comprehensive examination is provided of the discourse structure and performance efficiency of both interactive and noninteractive spontaneous speech in a seriated assembly task. More specifically, telephone dialogues and audiotape monologues are compared, which represent opposites in terms of the opportunity for confirmation feedback and clarification subdialogues. Keyboard communication patterns, upon which most natural language heuristics and algorithms have been based, also are contrasted with patterns observed in the two speech modalities. Finally, implications are discussed for the design of near-term limited-interaction spoken language systems.

This paper shows how the integration of natural language with direct manipulation produces a multimodal interface that overcomes limitations of these techniques when used separately. Natural language helps direct manipulation in being able to specify objects and actions by description, while direct manipulation enables users to learn which objects and actions are available in the system. Furthermore, graphical rendering and manipulation of context provides a partial solution to difficult problems of natural language anaphora.

This paper derives the basis of a theory of communication from a formal theory of rational interaction. The major result is a demonstration that illocutionary acts need not be primitive, and need not be recognized. As a test case. we derive Searle's conditions on requesting from principles of rationality coupled with a Gricean theory of imperatives. The theory is shown to distinguish insincere or nonserious imperatives from true requests. Extensions to indirect speech acts, and ramifications for natural language systems are also briefly discussed.

This paper considers what kind of speech act referring might be. I propose a generalization of Searle's "propositional" act of referring that treats it as an illocutionary act, a request, and argue that the propositional act of referring is unnecessary

A desirable long-range goal in building future speech understanding systems would be to accept the kind of language people spontaneously produce.We show that people do not speak to one another in the same way they converse in typewritten language.Spoken language is finer-grained and more indirect.The differences are striking and pervasive.Current techniques for engaging in typewritten dialogue will need to be extended to accomodate the structure of spoken language.

This panel was asked to consider how various "problem contexts" (e.g., cooperatively assembling a pump, or