Recent theoretical advancement of information density in natural language have raised the following question: To what degree does natural language exhibit periodicity pattern in its encoded information?We address this question by introducing a new method called AutoPeriod of Surprisal (APS).APS adopts a canonical periodicity detection algorithm and is able to identify any significant periods that exist in the surprisal sequence of a single document.By applying the algorithm to a set of corpora, we have obtained the following empirical results: Firstly, a considerable proportion of human language demonstrates a strong pattern of periodicity in information.Secondly, new periods that are outside the distributions of typical structural units in text (e.g., sentence boundaries, elementary discourse units, etc.) are found and further confirmed via harmonic regression modeling.We conclude that the periodicity of information in language is a joint outcome from both structured factors and other driving factors that take effect at longer distances.The advantages of our periodicity detection method and its potentials in LLM-generation detection are further discussed.Our code is available as public repositories.1

Backchannels and fillers are important linguistic expressions in dialogue, but often treated as 'noise' to be bypassed in modern transformerbased language models (LMs).Here, we study how they are represented in LMs using three fine-tuning strategies on three dialogue corpora in English and Japanese, in which backchannels and fillers are both preserved and annotated.This allows us to investigate how fine-tuning can help LMs learn these representations.We first apply clustering analysis to the learnt representation of backchannels and fillers, and find increased silhouette scores in representations from fine-tuned models, which suggests that fine-tuning enables LMs to distinguish the nuanced semantic variation in different backchannel and filler use.We also employ natural language generation metrics and qualitative analyses to verify that utterances produced by fine-tuned LMs resemble those produced by humans more closely.Our findings suggest the potential for transforming general LMs into conversational LMs that can produce human-like language more adequately.

A critical aspect in proactive human–robot interaction is deciding when and how a robot should initiate spoken interaction. The nature of this initiation can shape whether an encounter feels natural and supportive, or intrusive and unwelcome. Often, technological development prioritises system capabilities, paying limited attention to the subjective experience of communication, vital for designing sociable robots. This paper presents the findings from an empirical study investigating user preferences and the perceived desirability of spoken interaction initiation with a service robot in a household setting. In the video-based study with 239 participants, we compared reactive person-initiated, reactive robot-initiated, and proactive robot-initiated modes. The results of the study revealed clear patterns with more than half favouring reactive person-initiated interaction, followed by proactive robot-initiated interaction. By combining thematic and statistical analyses, we found that the perceived user preferences are shaped more by general propensity towards interaction itself than by demographics or specific personality traits. Further, it reflected the tension between the comfort of privacy and control, and the appeal of naturalness and intelligent support in user preferences. The results from the study provide evidence necessitating the design of initiation strategies that flexibly adapt to different user profiles and contexts.

We investigate how verbal and nonverbal linguistic features, exhibited by speakers and listeners in dialogue, can contribute to predicting the listener's state of understanding in explanatory interactions on a moment-by-moment basis. Specifically, we examine three linguistic cues related to cognitive load and hypothesised to correlate with listener understanding: the information value (operationalised with surprisal) and syntactic complexity of the speaker's utterances, and the variation in the listener's interactive gaze behaviour. Based on statistical analyses of the MUNDEX corpus of face-to-face dialogic board game explanations, we find that individual cues vary with the listener's level of understanding. Listener states ('Understanding', 'Partial Understanding', 'Non-Understanding' and 'Misunderstanding') were self-annotated by the listeners using a retrospective video-recall method. The results of a subsequent classification experiment, involving two off-the-shelf classifiers and a fine-tuned German BERT-based multimodal classifier, demonstrate that prediction of these four states of understanding is generally possible and improves when the three linguistic cues are considered alongside textual features.

Since the emergence of the field of eXplainable Artificial Intelligence (XAI), a growing number of researchers have argued that XAI should consider insights from the social sciences in order to adapt explanations to the expectations and needs of human users. This has led to the emergence of a field called Social XAI, which is concerned with understanding how explanations are actively shaped in the interaction between a human user and an AI system. Recognizing this turn in XAI toward making XAI systems more “social” by providing explanations that focus on human information needs and incorporating insights from human–human explanatory interactions, in this paper we provide a formal foundation for Social XAI. We do so by proposing novel ontological accounts of the key terms used in Social XAI based on Basic Formal Ontology (BFO). Specifically, we provide novel ontological accounts for explanandum, explanans, understanding, explanation, explainer, explainee, and context. In doing so, we discuss multifaceted entities in Social XAI (having both continuant and occurrent facets; e.g., explanation) and the relationship between understanding and explanation. Additionally, we propose solutions to seemingly paradoxical views on some terms (e.g., social constructivist vs. individual constructivist perspective on explanandum).

Judith Sieker, Simeon Junker, Ronja Utescher, Nazia Attari, Heiko Wersing, Hendrik Buschmeier, Sina Zarrieß. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

To produce explanations that are more likely to be accepted by humans, Explainable Artificial Intelligence (XAI) systems need to incorporate explanation models grounded in human communication patterns. So far, little is known about how an explainee, who lacks understanding of an issue, and an explainer, who has knowledge to fill the explainee’s knowledge gap, actively shape an explanation process, and how their involvement relates to explanatory success in terms of maximizing the explainee’s level of understanding. In this paper, we characterize explanations as dialogues in which explainee and explainer take turns to advance the explanation process. We build on an existing annotation scheme of ‘explanatory moves’ to characterize such turns, and manually annotate 362 dialogical explanations from the “Explain Like I’m Five” subreddit. Building on the annotated data, we compute correlations between explanatory moves and explanatory success, measured on a five-point Likert scale, in order to identify factors that are significantly correlated with explanatory success. Based on a qualitative analysis of these factors, we develop a conceptual model of the main factors that contribute to the success of explanatory dialogues.

The ICMI 2024 workshop on “Multimodal Co-Construction of Explanations with XAI” bridges the fields of Explainable Artificial Intelligence (XAI) and Multimodal Interaction, focusing on the recent perspective that effective AI explanations should be dynamically co-constructed through interactive, social processes involving both the explainer and the explainee. By framing XAI explanations as a multimodal, interactive co-construction challenge, the workshop seeks to explore how these two fields can collaboratively address the complexities of creating understandable and context-sensitive XAI systems.

In explanatory interactions, explainees are expected to continuously provide feedback to explainers by signaling whether they understand an ongoing explanation. The study presented in this paper is based on the hypothesis that explainees use a set of multimodal cues, including vocalizations, facial expressions, and movements of the torso, head, and hands, to do so. We test this hypothesis by building a random forest classifier based on a multimodal corpus of dyadic explanations (21 explainers and explainees), in which windows of understanding or non-understanding were identified by participants in a retrospective video recall task. Results show that sequences of understanding can indeed be differentiated from those of non-understanding, and that a diverse set of predictors covering a wide range of modalities contributes to this classification. Due to data sparsity and a high degree of individual variation, the generalizability of our results is currently limited, but they support our hypothesis of the relevance of multimodal display in explanatory interactions.

Scripted dialogues such as movie and TV subtitles constitute a widespread source of training data for conversational NLP models.However, there are notable linguistic differences between these dialogues and spontaneous interactions, especially regarding the occurrence of communicative feedback such as backchannels, acknowledgments, or clarification requests.This paper presents a quantitative analysis of such feedback phenomena in both subtitles and spontaneous conversations.Based on conversational data spanning eight languages and multiple genres, we extract lexical statistics, classifications from a dialogue act tagger, expert annotations and labels derived from a fine-tuned Large Language Model (LLM).Our main empirical findings are that (1) communicative feedback is markedly less frequent in subtitles than in spontaneous dialogues and (2) subtitles contain a higher proportion of negative feedback.We also show that dialogues generated by standard LLMs lie much closer to scripted dialogues than spontaneous interactions in terms of communicative feedback.

Politeness is a social and linguistic phenomenon that humans use in communication to build and maintain relationships and spare others’ feelings. Research on whether humans also apply politeness strategies when interacting with robots – artifacts that lack feelings – yields contradictory findings. This paper presents a human–robot interaction study (N=40) and compares participants’ use of face saving politeness strategies in their responses to disconfirmation eliciting and face-threatening questions asked either by a robot or a human. An analysis of the linguistic properties of participants’ answers (response type, use of politeness markers) shows a higher use of indirect politeness in disconfirming answers directed at humans than at robots. This contradicts previous theories on the automatic and ‘mindless’ application of social strategies towards artificial agents. Alternative explanations for the differences in politeness behavior are discussed.

The use and interpretation of social linguistic strategies such as politeness is influenced by multiple factors, e.g., the speaker-hearer relation. Such relations influence an inter-locutor's expectations regarding the interaction and thus also its perception. This makes speaker-hearer relations constituting a partner model highly relevant for the user experience in human-robot interaction as well. This paper presents a questionnaire-based study on the perception of human-robot relations in comparison to human-human relations across different roles (e.g., colleague, assistant) and spaces of interaction (home, work, public). It was found that participants perceive robots differently based on space, as they do for human-human relations in corresponding roles. Overall, humans were evaluated to have more power over and more distance to a robot interaction partner compared to another human. Our results provide insights into an intuitive interaction-initial partner model based on roles.

Recent work has adopted models of pragmatic reasoning for the generation of informative language in, e.g., image captioning.We propose a simple but highly effective relaxation of fully rational decoding, based on an existing incremental and character-level approach to pragmatically informative neural image captioning.We implement a mixed, 'fast' and 'slow', speaker that applies pragmatic reasoning occasionally (only word-initially), while unrolling the language model.In our evaluation, we find that increased informativeness through pragmatic decoding generally lowers quality and, somewhat counter-intuitively, increases repetitiveness in captions.Our mixed speaker, however, achieves a good balance between quality and informativeness.

Social robots are expected to increasingly appear in private households. The deployment of social robots in the private spheres of humans raises concerns regarding privacy protection. This paper analyses some of the legal implications of using social robots in private households on the basis of four practical use cases. It identifies the privacy concerns associated with each use case and proposes potential technical measures in the form of an initial concept for a companion privacy-app that could resolve or mitigate these concerns, and thereby enhance privacy compliance. The proposed app concept was evaluated in an exploratory study with ten participants. The preliminary results are encouraging and show that this concept has the potential to support the maintenance of privacy and provide control over the user's personal data and the robot's functions.

Rottschäfer S, Buschmeier H, van Welbergen H, Kopp S. Online Lombard-adaptation in incremental speech synthesis. In: Proceedings of INTERSPEECH 2015. Dresden, Germany; 2015: 80-84.

It is established that driver distraction is the result of sharing cognitive resources between the primary task (driving) and any other secondary task. In the case of holding conversations, a human passenger who is aware of the driving conditions can choose to interrupt his speech in situations potentially requiring more attention from the driver, but in-car information systems typically do not exhibit such sensitivity. We have designed and tested such a system in a driving simulation environment. Unlike other systems, our system delivers information via speech (calendar entries with scheduled meetings) but is able to react to signals from the environment to interrupt when the driver needs to be fully attentive to the driving task and subsequently resume its delivery. Distraction is measured by a secondary short-term memory task. In both tasks, drivers perform significantly worse when the system does not adapt its speech, while they perform equally well to control conditions (no concurrent task) when the system intelligently interrupts and resumes.

When a passenger speaks to a driver, he or she is co-located with the driver, is generally aware of the situation, and can stop speaking to allow the driver to focus on the driving task. In-car dialogue systems ignore these important aspects, making them more distracting than even cell-phone conversations. We developed and tested a "situationally-aware" dialogue system that can interrupt its speech when a situation which requires more attention from the driver is detected, and can resume when driving conditions return to normal. Furthermore, our system allows driver-controlled resumption of interrupted speech via verbal or visual cues (head nods). Over two experiments, we found that the situationally-aware spoken dialogue system improves driving performance and attention to the speech content, while driver-controlled speech resumption does not hinder performance in either of these two tasks

Buschmeier H, Malisz Z, Wlodarczak M, Kopp S, Wagner P. 'Are you sure you're paying attention?' – 'Uh-huh'. Communicating understanding as a marker of attentiveness. In: Proceedings of INTERSPEECH 2011. International Speech Communication Association; 2011: 2057-2060.