How do user-defined gestures for mixed reality change when users’ hands are engaged in tasks? To address this question, we conducted a gesture elicitation study to understand user preferences and the characteristics of gestures conceptualized in three scenarios with varying levels of hand constraints, namely: "both hands free", "one hand fixed", and "both hands busy". We analyzed these gestures across multiple dimensions and compared our findings with those from prior research. Our results indicate that when both hands are occupied, users tend to favor head gestures over those involving other body parts, such as the eyes or legs. Additionally, we found that most of the proposed gestures were metaphorical, with many influenced by legacy bias. These insights enhance our understanding of how hand constraints influence gesture choices in mixed reality scenarios.

Prior research has highlighted users’ preferences for embodiment when interacting with virtual agents in augmented reality headsets. However, open questions remain regarding users’ preferences towards agent placement and gaze direction. In our study, we asked 48 adults to wear the Microsoft HoloLens 2 and find objects in a hidden object game with the help of embodied agents. We examined four distinct agent configurations for both male and female agents: a human-size agent standing beside participants, a human-size agent sitting beside participants, a small desk agent facing the screen, and a small desk agent facing the participant. Overall, participants preferred male over female virtual agents when receiving assistance, and no consistent preference emerged regarding the agents’ position or gaze direction. From our results, we build upon existing guidelines for designing better virtual agents for AR with headsets.

Although companion virtual agents (CVAs) are increasingly adopted to support well-being, interaction patterns between adults and CVAs remain underexplored. This study examines how users engage with CVAs over time, exploring conversation patterns, interaction frequency, attitudes, and emotional responses over a seven-day period. Twenty-four adults engaged with a GPT-4-powered embodied CVA daily, discussing topics from personal interests to emotional reflections. Quantitative measures, including loneliness and affect scales, revealed no significant reduction in loneliness but noted decreases in positive affect and nervousness. Qualitative analysis highlighted evolving conversational dynamics, with participants shifting from exploratory questions to more reflective and personal discussions. Participants appreciated the agent’s ability to engage in fluid and meaningful conversations. However, participants also noted shortcomings, including limited recall and occasional conversational unnaturalness. These findings inform the design of CVAs, emphasizing the need for adaptive conversational strategies, enhanced emotional responsiveness, and improved memory systems to foster meaningful connections.

To design intuitive and effective context-aware task guidance systems, we must understand users’ thought processes and the obstacles they experience when they perform tasks. Though task guidance systems have proven beneficial in many domains for improving task performance and reducing user frustration, there is a lack of general guidelines and design principles for their development. Prior work has shown that recipe-based cooking is a strong medium for studying task planning and execution. In response, we conducted a contextual inquiry study in home kitchens, observing eight different participants’ cooking sessions. We used affinity diagramming of our notes and transcripts to identify common obstacles faced by participants and establish user needs in the areas of object interaction, safety, knowledge base, and task coordination. We discuss how these findings can inform the design of technology-driven solutions for task guidance systems beyond cooking.