Abstract Immersive virtual reality offers a range of unique possibilities. One of these is the realistic exploration of virtual worlds using natural walking. This however becomes difficult when the size of the virtual world exceeds that of the available physical space. Redirected walking in virtual reality presents a novel solution to this problem by typically making its users think to be walking in a straight line while they are in fact walking in a curve, thus allowing them to physically walk long distances in confined physical spaces. Yet, few studies have examined the effects of redirected walking on variables such as spatial memory, navigation and user experience as compared to other immersive and non-immersive locomotion methods. In a maze task we examined 1) redirected- and 2) natural walking in immersive virtual reality conditions, and 3) artificial locomotion on a non-immersive desktop monitor. Walked path lengths became shorter and distance estimates, object location memory and user experience improved using natural walking compared to a monitor condition. However, redirected walking yielded similar performance to natural walking while requiring less physical space, opening up possibilities for more pervasive use of real locomotion in virtual environments.

Anthropomorphic agents are generally evaluated more positively and trustworthy by human users than agents that are not humanlike. However, subtle mismatches in an agent’s appearance and behavior can lead to perceived uncanniness resulting in a disrupted trust during human-agent interaction. This study investigated the impact of an agent’s appearance and voice mismatch on user perception of the agent and their level of trust during a collaborative decision-making task. In a 2×2 between-subjects experimental design, participants performed an emotion recognition task while receiving recommendations from a virtual agent that either had a humanlike or robotic appearance with either a humanlike or synthesized robotic voice (4 conditions). Trust was measured both subjectively using a questionnaire and behaviorally by evaluating participants’ conformity to the agent’s input in their final decision-making. Results indicated that while the agent’s voice-appearance mismatch affected participants’ perception of anthropomorphism, it was not an influential factor in people’s trusting behavior. We discuss these results in the context of task complexity and make recommendations for future research.

This study introduces LittleGenius, a VR tool integrated with GPT-4 for primary education. Combining GPT-4’s advanced natural language processing with VR technology, LittleGenius creates an engaging learning environment. Users interact with an astronaut floating outside the International Space Station. A specialized GPT prompt was developed to foster deeper engagement through knowledge construction. The system enables natural speech interaction using Microsoft Azure’s Speech Cognitive Services. Nine in-service teachers evaluated the design, giving positive feedback, especially on the interactive astronaut agent’s engaging questions that could deepen student understanding. Future development will expand content and customize the system for diverse communication preferences and learning styles.

Physiological data have shown to be useful in tracking and differentiating cognitive processes in a variety of experimental tasks, such as numerical skills and arithmetic tasks. Numerical skills are critical because they are strong predictors of levels of ability in cognitive domains such as literacy, attention, and understanding contexts of risk and uncertainty. In this work, we examined frontal and parietal electroencephalogram signals recorded from 36 healthy participants performing a mental arithmetic task. From each signal, six RQA-based features (Recurrence Rate, Determinism, Laminarity, Entropy, Maximum Diagonal Line Length and, Average Diagonal Line Length) were extracted and used for classification purposes to discriminate between participants performing proficiently and participants performing poorly. The results showed that the three classifiers implemented provided an accuracy above 0.85 on 5-fold cross-validation, suggesting that such features are effective in detecting performance independently from the specific classifiers used. Compared to other successful methods, RQA-based features have the potential to provide insights into the nature of the physiological dynamics and the patterns that differentiate levels of proficiency in cognitive tasks.

We introduce an interactive embodied conversational agent for deployment in the healthcare sector. The agent is operated by a software architecture that integrates speech recognition, dialog management, and speech synthesis, and is embodied by a virtual human face developed using photogrammetry techniques. These features together allow for real-time, face-to-face interactions with human users. Although the developed software architecture is domain-independent and highly customizable, the virtual agent will initially be applied to healtcare domain. Here we give an overview of the different components of the architecture.

Human interlocutors automatically adapt verbal and non-verbal signals so that different behaviors become synchronized over time. Multimodal communication comes naturally to humans, while this is not the case for Embodied Conversational Agents (ECAs). Knowing which behavioral channels synchronize within and across speakers and how they align seems critical in the development of ECAs. Yet, there exists little data-driven research that provides guidelines for the synchronization of different channels within an interlocutor. This study focuses on intrapersonal dependencies of multimodal behavior by using cross-recurrence analysis on a multimodal communication dataset to better understand the temporal relationships between language and gestural behavior channels. By shedding light on the intrapersonal synchronization of communicative channels in humans, we provide an initial manual for modality synchronisation in ECAs.

With forty-six Action Units (AUs) forming the building blocks in the Facial Action Coding System (FACS), millions of facial configurations can be formed. Most research has focused on a subset of combinations to determine the link between facial configurations and emotions. Despite the value of this research for psychological and computational reasons, it is not clear what the most common combinations of AUs are to form the most commonly expressed facial configurations. We used three diverse corpora with human coded facial action units for a computational analysis. The analysis demonstrated that the largest portion of facial behavior consists of the absence of AU activations, yielding only one specific facial configuration, that of the neutral face. These results are important for cognitive scientists, computer graphics designers and virtual human developers alike. They suggest that only a relatively small number of AU combinations are initially needed for the creation of natural facial behavior in Embodied Conversational Agents (ECAs).

Zipf's law is a mathematically relatively simple formula stating that the frequency of a word is inversely correlated with its rank. Zipf's law is well-known in computational linguistics and cognitive sciences alike. In the context of agent development, however, Zipf's law has hardly ever been mentioned. This is surprising as principles regarding language likely benefit the development of conversational agents. This paper serves as a starting point to explore the role of Zipf's law in agent development, showing that Zipf's law also applies to dialog. Moreover, it can shed light on human-machine dialog. In addition to word frequency distributions that demonstrate Zipf's law, we also included frequency distributions of words at specific positions in the sentence as well as turn lengths. Zipf's law was found in the far majority of analyses we conducted. In addition, we investigated whether Zipf's law can be used to detect differences between human and agent-generated speech through correlating the distributions and found that even though both the human and agent frequency distributions follow Zipf's law, these distributions are not necessarily similar, shedding light on where agent dialog may distinguish itself from human dialog. The findings in this paper can thus serve as a way to monitor to what extent ubiquitous patterns in human-human dialog are found in human-machine dialog.

It is crucial that naturally-looking Embodied Conversational Agents (ECAs) display various verbal and non-verbal behaviors, including facial expressions. The generation of credible facial expressions has been approached by means of different methods, yet remains difficult because of the availability of naturalistic data. To infuse more variability into the facial expressions of ECAs, we proposed a model that considered temporal dynamic of facial behaviors as a countable-state Markov process. Once trained, the model was able to output new sequences of facial expressions from an existing dataset containing facial videos with Action Unit (AU) encodings. The approach was validated by having computer software and humans identify facial emotion from video. Half of the videos employed newly generated sequences of facial expressions using the model while the other half contained sequences selected directly from the original dataset. We found no statistically significant evidence that the newly generated facial expression sequences could be differentiated from the original ones, demonstrating that the model was able to generate new facial expression data that were indistinguishable from the original data. Our proposed approach could be used to expand the amount of labelled facial expression data in order to create new training sets for machine learning methods.

This paper explores the relative importance of discourse features, prosodic features and their fusion in robust classification of speech acts. Five different feature selection algorithms were used to select set of features to improve the robustness of the classification. The results showed that the ensemble-based classifiers performed best in the classification of 12 speech acts using subsets of both prosodic and discourse features.