With the rise of the aging population, the demand of care increases yet the turnover rate of caregivers for geriatric patients has increased over the past few decades. While many factors contribute to the caregivers’ turnover rate, inadequate training and lack of communication skills and relationship-building skills are vital issues. Technology can help in providing training for soft skills. Embodied Virtual Human (VH) assistants and Augmented Reality (AR) can be a key method to train caregivers and to improve their experience in interacting with older adults, and their perception of the interaction. We designed and developed an immersive simulation in AR where caregivers interact with an embodied VH in two different conditions, i.e., first with an unaware VH and then with a VH having simulated awareness achieved through human-in-the-loop. We conducted a study with caregivers of older adults to evaluate the usability and technology acceptance of our system. The findings suggest that the majority of the participants found the system acceptable, and rated the system as user-friendly and easy to use.

The existing educational programs for caregivers often fall short in providing adequate training for essential interpersonal skills, particularly in communication and relationship development. To address this gap, researchers have highlighted the potential of Intelligent Virtual Agents (IVAs) as a tool for soft skills training across various sectors, including healthcare. In this article, we introduce an aware IVA (A-IVA) designed to represent a geriatric patient, that demonstrates limited and simulated awareness of the environment and previous conversations achieved by Wizard-of-Oz. We developed a novel Augmented Reality (AR) simulation that immerses users in a realistic caregiving scenario. We conducted a pilot study to evaluate how the A-IVA’s awareness capabilities influence geriatric caregivers’ social presence and empathy levels as compared to an unaware IVA (U-IVA).

Embodied virtual agents serving as patient simulators are widely used in medical training scenarios, ranging from physical patients to virtual patients presented via virtual and augmented reality technologies. Physical-virtual patients are a hybrid solution that combines the benefits of dynamic visuals integrated into a human-shaped physical form that can also present other cues, such as pulse, breathing sounds, and temperature. Sometimes in simulation the visuals and shape do not match. We carried out a human-participant study employing graduate nursing students in pediatric patient simulations comprising conditions associated with matching/non-matching of the visuals and shape.

Conventional Intelligent Virtual Agents (IVAs) focus primarily on the visual and auditory channels for both the agent and the interacting human: the agent displays a visual appearance and speech as output, while processing the human's verbal and non-verbal behavior as input. However, some interactions, particularly those between a patient and healthcare provider, inherently include tactile components. We introduce an Intelligent Physical-Virtual Agent (IPVA) head that occupies an appropriate physical volume; can be touched; and via human-in-the-loop control can change appearance, listen, speak, and react physiologically in response to human behavior. Compared to a traditional IVA, it provides a physical affordance, allowing for more realistic and compelling human-agent interactions. In a user study focusing on neurological assessment of a simulated patient showing stroke symptoms, we compared the IPVA head with a high-fidelity touch-aware mannequin that has a static appearance. Various measures of the human subjects indicated greater attention, affinity for, and presence with the IPVA patient, all factors that can improve healthcare training.

The field of intelligent virtual agents (IVAs) has evolved immensely over the past 15 years, introducing new application opportunities in areas such as training, health care, and virtual assistants. In this survey paper, we provide a systematic review of the most influential user studies published in the IVA conference from 2001 to 2015 focusing on IVA development, human perception, and interactions. A total of 247 papers with 276 user studies have been classified and reviewed based on their contributions and impact. We identify the different areas of research and provide a summary of the papers with the highest impact. With the trends of past user studies and the current state of technology, we provide insights into future trends and research challenges.

While research in the field of augmented reality (AR) has produced many innovative human-computer interaction techniques, some may produce physical and visual perceptions with unforeseen negative impacts on user performance. In a controlled human-subject study we investigated the effects of mismatched physical and visual perception on cognitive load and performance in an AR touching task by varying the physical fidelity (matching vs. non-matching physical shape) and visual mechanism (projector-based vs. HMD-based AR) of the representation. Participants touched visual targets on four corresponding physical-visual representations of a human head. We evaluated their performance in terms of touch accuracy, response time, and a cognitive load task requiring target size estimations during a concurrent (secondary) counting task. After each condition, participants completed questionnaires concerning mental, physical, and temporal demands; stress; frustration; and usability. Results indicated higher performance, lower cognitive load, and increased usability when participants touched a matching physical head-shaped surface and when visuals were provided by a projector from underneath.

Currently healthcare practitioners use standardized patients, physical mannequins, and virtual patients as surrogates for real patients to provide a safe learning environment for students. Each of these simulators has different limitation that could be mitigated with various degrees of fidelity to represent medical cues. As we are exploring different ways to simulate a human patient and their effects on learning, we would like to compare the dynamic visuals between spatial augmented reality and a optical see-through augmented reality where a patient is rendered using the HoloLens and how that affects depth perception, task completion, and social presence.

We explore whether a peripheral observation of apparent mutual social presence between a real human (RH) and a virtual human (VH) can in turn increase a subject's sense of social presence with the VH. In other words, we explore whether social presence can “transfer” from one RH-VH interaction to another. Specifically, we carried out an experiment where human subjects were asked to play a game with a VH. As they entered the game room, approximately half of the subjects were exposed to a brief but apparently engaging conversation between an RH and the VH. The subjects who were exposed to the brief RH-VH interaction had significantly higher measures of both emotional connection and the attentional allocation dimension of social presence for the VH, compared to those who were not. We describe the motivation, the experiment, and the results.

While performing everyday interactions, we often incidentally touch and move objects in subtle ways. These objects are not necessarily directly related to the task at hand, and the movement of an object might even be entirely unintentional. If another person is touching the object at the same time, the movement can transfer through the object and be experienced — however subtly — by the other person. For example, when one person hands a drink to another, at some point both individuals will be touching the glass, and consequently exerting small (often unnoticed) forces on the other person. Despite the frequency of such subtle incidental movements of shared objects in everyday interactions, few have examined how these movements affect human-virtual human (VH) interaction. We ran an experiment to assess how presence and social presence are affected when a person experiences subtle, incidental movement through a shared real-virtual object. We constructed a real-virtual room with a table that spanned the boundary between the real and virtual environments. The participant was seated on the real side of the table, which visually extended into the virtual world via a projection screen, and the VH was seated on the virtual side of the table. The two interacted by playing a game of “Twenty Questions,” where one player asked the other a series of 20 yes/no questions to deduce what object the other player was thinking about. During the game, the “wobbly” group of subjects experienced subtle incidental movements of the real-virtual table: the entire real-virtual table tilted slightly away/toward the subject when the virtual/real human leaned on it. The control group also played the same game, except the table did not wobble. Results indicate that the wobbly group had higher presence and social presence with the virtual human in general, with statistically significant increases in presence, co-presence, and attentional allocation. We present the experiment and results, and discuss some potential implications for virtual human systems and some potential future studies.

We demonstrate a generalizable method for unified multitouch detection and response on a human head-shaped surface with a rear-projection animated 3D face. The method helps achieve hands-on touch-sensitive training with dynamic physical-virtual patient behavior. The method, which is generalizable to other non-parametric rear-projection surfaces, requires one or more infrared (IR) cameras, one or more projectors, IR light sources, and a rear-projection surface. IR light reflected off of human fingers is captured by cameras with matched IR pass filters, allowing for the localization of multiple finger touch events. These events are tightly coupled with the rendering system to produce auditory and visual responses on the animated face displayed using the projector(s), resulting in a responsive, interactive experience. We illustrate the applicability of our physical prototype in a medical training scenario.