When individuals are talking while performing multiple tasks at the same time, it is sometimes easy to miss parts of a conversation and misinterpret subsequent statements or have difficulty following the conversation. In this work, we aim to identify statements that may lead to misinterpretation of the subsequent statement if missed and to prevent communication discrepancies. Although there have been several attempts to present images and text that provide topics to support conversation, there is currently no system that supports conversation by taking interpretability into account. We propose a conversation support system SCAINs Presenter that presents Statements Crucial for Awareness of Interpretive Nonsense (SCAINs), which are statements that are important for interpreting other sentences and are extracted by reproducing the interpretations of those who missed part of the conversation and those who did not. The unique point of the SCAINs Presenter is to display extracted sentences that influence the context of the subsequent dialogue by taking into account their interpretability. In particular, since SCAINs are sentences that may cause misinterpretation of the subsequent dialogue if they are absent, the SCAINs Presenter helps the users to be aware of the possibility of a conversation gap coming from the misinterpretation. Our experiments show that when SCAINs are omitted, the intention of the following statements often becomes unclear, and the meaning of the following statements changes. We also found that SCAINs can capture a unique aspect different from the merely important statements. Moreover, the results of case studies in a realistic setting suggest that looking at SCAINs encourages conversation participants to switch their focus from a subtask chat to an ongoing conversation that is a primary task. Our research clarifies the linguistic processing underlying the identification of high-context utterances and demonstrates the effectiveness of using them to support real person-to-person interactions.

During remote conversations, communication breakdowns often occur when a listener misses certain statements.Our objective is to prevent such breakdowns by identifying Statements Crucial for Awareness of Interpretive Nonsense (SCAINs).If a listener misses a SCAIN, s/he may interpret subsequent statements differently from the speaker's intended meaning.To identify SCAINs, we adopt a unique approach where we create a dialogue by omitting two consecutive statements from the original dialogue and then generate text to make the following statement more specific.The novelty of the proposed method lies in simulating missing information by processing text with omissions.We validate the effectiveness of SCAINs through evaluation using a dialogue dataset.Furthermore, we demonstrate that SCAINs cannot be identified as merely important statements, highlighting the uniqueness of our proposed method.

When people are talking together in front of digital signage, advertisements that are aware of the context of the dialogue will work the most effectively. However, it has been challenging for computer systems to retrieve the appropriate advertisement from among the many options presented in large databases. Our proposed system, the Conversational Context-sensitive Advertisement generator (CoCoA), is the first attempt to apply masked word prediction to web information retrieval that takes into account the dialogue context. The novelty of CoCoA is that advertisers simply need to prepare a few abstract phrases, called Core-Queries, and then CoCoA automatically generates a context-sensitive expression as a complete search query by utilizing a masked word prediction technique that adds a word related to the dialogue context to one of the prepared Core-Queries. This automatic generation frees the advertisers from having to come up with context-sensitive phrases to attract users’ attention. Another unique point is that the modified Core-Query offers users speaking in front of the CoCoA system a list of context-sensitive advertisements. CoCoA was evaluated by crowd workers regarding the context-sensitivity of the generated search queries against the dialogue text of multiple domains prepared in advance. The results indicated that CoCoA could present more contextual and practical advertisements than other web-retrieval systems. Moreover, CoCoA acquired a higher evaluation in a particular conversation that included many travel topics to which the Core-Queries were designated, implying that it succeeded in adapting the Core-Queries for the specific ongoing context better than the compared method without any effort on the part of the advertisers. In addition, case studies with users and advertisers revealed that the context-sensitive advertisements generated by CoCoA also had an effect on the content of the ongoing dialogue. Specifically, since pairs unfamiliar with each other more frequently referred to the advertisement CoCoA displayed, the advertisements had an effect on the topics about which the pairs spoke. Moreover, participants of an advertiser role recognized that some of the search queries generated by CoCoA fit the context of a conversation and that CoCoA improved the effect of the advertisement. In particular, they learned how to design of designing a good Core-Query at ease by observing the users’ response to the advertisements retrieved with the generated search queries.

In this work, we present a system that sequentially generates advertisements within the context of a dialogue. Advertisements tailored to the user have long been displayed on the digital signage in stores, on web pages, and on smartphone applications. Advertisements will work more effectively if they are aware of the context of the dialogue between the users. Creating an advertising sentence as a query and searching the web by using that query is one way to present a variety of advertisements, but there is currently no method to create an appropriate search query for the search in accordance with the dialogue context. Therefore, we developed a method called the Conversational Context-sensitive Advertisement generator (CoCoA). The novelty of CoCoA is that advertisers simply need to prepare a few abstract phrases, called Core-Queries, and then CoCoA dynamically transforms the Core-Queries into complete search queries in accordance with the dialogue context. Here, “transforms” means to add words related to the context in the dialogue to the prepared Core-Queries. The transformation is enabled by a masked word prediction technique that predicts a word that is hidden in a sentence. Our attempt is the first to apply masked word prediction to a web information retrieval framework that takes into account the dialogue context. We asked users to evaluate the search query presented by CoCoA against the dialogue text of multiple domains prepared in advance and found that CoCoA could present more contextual and effective advertisements than Google Suggest or a method without the query transformation. In addition, we found that CoCoA generated high-quality advertisements that advertisers had not expected when they created the Core-Queries.

To create dialogue systems that provide information a user needs to know at an opportune moment, it is important to infer the user’s mental states such as his/her beliefs and desires. There are two types of study on inferring beliefs and desires: one type infers them from actions and the other infers them from the content of utterances. However, a method to infer beliefs and desires from both kinds of inference in an integrated way has not yet been established. In this paper, we propose Multimodal Inference of Mind Simultaneous Contextualization and Interpreting (MIoM SCAIN), a system for sequentially inferring users’ beliefs and desires on the basis of their walking behaviors and the content of their utterances. In our evaluation, we compared inferences of MIoM SCAIN with those of baselines that use either walking behaviors or the content of utterances. MIoM SCAIN’s predictions showed more correlation with subjective judgements compared with the baselines, indicating that the inference of beliefs and desires from both walking behaviors and utterance content is possible.