An interactive vignette is a popular and immersive visual storytelling approach that invites viewers to role-play a character and influences the narrative in an interactive environment. However, it has not been widely used by everyday storytellers yet due to authoring complexity, which conflicts with the immediacy of everyday storytelling. We introduce DiaryPlay, an AI-assisted authoring system for interactive vignette creation in everyday storytelling. It takes a natural language story as input and extracts the three core elements of an interactive vignette (environment, characters, and events), enabling authors to focus on refining these elements instead of constructing them from scratch. Then, it automatically transforms the single-branch story input into a branch-and-bottleneck structure using an LLM-powered narrative planner, which enables flexible viewer interactions while freeing the author from multi-branching. A technical evaluation (N=16) shows that DiaryPlay-generated character activities are on par with human-authored ones regarding believability. A user study (N=16) shows that DiaryPlay effectively supports authors in creating interactive vignette elements, maintains authorial intent while reacting to viewer interactions, and provides engaging viewing experiences.

Location-based games (LBGs) merge digital play with physical environments, creating hybrid spaces that require players to navigate complex trust dynamics. Despite their global popularity, LBGs introduce unique challenges around fairness, safety, and privacy, spanning interactions among players, game systems, local communities, and non-players in shared public spaces. To examine how trust is perceived, built, and sustained in these environments, we conducted in-depth interviews with 26 players of four major LBGs: Pokémon GO, Monster Hunter Now, Ingress, and Pikmin Bloom. Using reflexive thematic analysis, we identified dynamics of trust across four trustor–trustee relationships: player–system, player–player, player–community, and player–non-player in five key aspects: fair play, location privacy, online vetting, hybrid interaction, and public play. Drawing on our findings, we propose a trust model for analyzing and designing trust in LBGs as hybrid spaces, and we outline design implications aimed at strengthening trust building and sustaining trustworthy interactions across the LBG ecology.

Emerging multimodal conversational search (MCS) tools (e.g., Gemini Live) allow users to search for spatiotemporal information through natural language dialogues as they move through urban space. Despite the growing popularity of these tools, there is limited understanding of how people engage with this technology. To address this gap, we developed UrbanSearch, an MCS technology probe designed to capture the user’s current geolocation, time, and visual surroundings. A contextual inquiry (N=23) revealed that MCS tools provide two core values: requiring low effort in forming queries while offering highly relevant responses, and functioning as a central information gateway. As a promising technology, MCS supports environmental learning, in-situ decision making, and personalized navigation. Participants also revealed unmet needs for spatial reasoning and transparent integration of multi-source information, along with concerns related to peripheral awareness, social context, and personal space. Drawing from the findings, we discuss design implications for future MCS tools in urban spaces.

The more-than-human field has contributed numerous opportunities for interacting with nature, animals, plants, and microorganisms. However, few studies have examined water ecosystems. Current water-related work primarily treats water as a medium for human-centered activities, rarely positioning water as an interactive subject. Building upon prior research, we explore how to better integrate water more playfully into digital-physical interactions as an interactive subject. We designed and developed Water’s Echo, an AI-powered public installation that enables human-water communication through a playful stone-throwing dialogue. We conducted a field study at a local pond, recruiting 15 residents to participate in Water’s Echo—a playful conversational interaction. Our findings indicate that this playful dialogue approach raises participants’ awareness and understanding of surrounding aquatic environments. This research provides insights for design researchers to establish engaging water ecology interactions across cultural communities, promoting a More-than-human perspective in re-examining human-nature relationships.

Tasks in augmented reality (AR), such as 3D interaction and instructional comprehension, are often designed for users with uniform sensory abilities. Such an approach, however, can overlook the more nuanced needs of Deaf and Hard of Hearing (DHH) users who might have reduced auditory perception. To better understand these challenges, our study utilized the single-player AR game Angry Birds AR as a probe to explore how 11 DHH participants and 15 hearing participants experienced AR interactions. Our findings highlight that DHH users prefer interaction based on context, effective haptic cues, audio cue substitutes, and clear instructional design. We, therefore, propose the following design recommendations to enhance the accessibility of AR for DHH users. This includes customizable UI options, modular feedback systems, and virtual avatars for sign language instructions.

Shared Augmented Reality (Shared AR) is an emerging technology that enables multiple users to interact synchronously within a collocated AR environment. Yet, there is limited research on the group interactions and dynamics in Shared AR, particularly in the context of gaming. To address this gap, we investigate Shared AR group interactions using a phone-based Shared AR mobile game called Urban Legends . Through in situ observations, focus groups, and one-on-one interviews with 22 participants, we examine how users collaborate and communicate within the game. Our findings reveal that while verbal communication predominates, non-verbal cues are often overlooked by collocated participants, and users initially struggle to recognize the expansive virtual space and the need for physical movement. Over time, users adapt to the hybrid environment, demonstrating increasing spatial awareness and more dynamic collaboration. Based on these insights, we present a suite of design recommendations for enhancing spatial awareness, supporting multi-modal communication, and fostering engaging group dynamics in future Shared AR applications.

Sensory-intensive and attention-demanding tasks like visual scanning, interacting with 3D objects, comprehending and following instructions, etc. are becoming more common in Augmented Reality (AR) environments as the technology expands through diverse fields. It is important to understand how these types of tasks are experienced by Deaf and Hard of Hearing (DHH) people, especially if those tasks involve any sound or compete with attention shifts (e.g., observing someone signing) in both real and virtual environments. Our current research specifically aims to identify the challenges that DHH users encounter when engaging in visual scanning in an AR environment. Using Angry Birds AR as a probe in our research, 11 DHH participants, with varying hearing abilities played seven rounds of the game, followed by a short structured interview and a long semi-structured interview. Our findings revealed that subtle audio cues and excessive visual indicators impacted participants’ performances negatively. Additionally, they positioned themselves strategically for maximum spatial awareness but faced challenges with AR visual cues due to the lighting conditions in the real environment. We further suggested design implications such as customizable, user-friendly haptic and textual feedback, and intelligent spatially aware mechanisms for AR.

Co-located collaborative shared augmented reality (CS-AR) environments have gained considerable research attention, mainly focusing on design, implementation, accuracy, and usability. Yet, a gap persists in our understanding regarding the accessibility and inclusivity of such environments for diverse user groups, such as deaf and Hard of Hearing (DHH) people. To investigate this domain, we used Urban Legends, a multiplayer game in a co-located CS-AR setting. We conducted a user study followed by one-on-one interviews with 17 DHH participants. Our findings revealed the usage of multimodal communication (verbal and non-verbal) before and during the game, impacting the amount of collaboration among participants and how their coordination with AR components, their surroundings, and other participants improved throughout the rounds. We utilize our data to propose design enhancements, including onscreen visuals and speech-to-text transcription, centered on participant perspectives and our analysis.

One of the most recent developments in augmented reality (AR) technology is co-located collaborative multiplayer AR environments, yet little research has been done to determine how communication and collaboration function in these settings. There is a gap in our understanding of whether these environments are currently accessible for Deaf and Hard of Hearing (DHH) people. Although technical aspects, such as design, implementation, accuracy, etc., of the technologies and users’ behavior with assistive AR technologies, have been covered in plenty of prior literature, very few studies have paid attention to how DHH people communicate and collaborate in co-located collaborative AR environments. As an initial step toward addressing the gap in current literature, our ongoing research focuses on how DHH people communicate and collaborate in a co-located collaborative multiplayer AR environment. As part of our study, we conducted gameplay experiments and one-on-one semi-structured interviews with 17 DHH participants. In this piece of work, among the prominent themes that stood out in our findings, we further discuss communication and collaboration, specifically multi-modal communication (verbal and non-verbal) among users and how multi-modal communication affected their collaboration in the environment.

The overlaying of physical spaces with digital information produces hybrid spaces, redefining people’s experience of social interactions. Location-based games (LBGs) with social components are a good case. Yet, the impact LBGs have on sociability remains under-researched. In April 2020, the new in-person/remote raiding format in the LBG Pokémon GO provided a lens to explore people’s social interactions in hybrid spaces. We interviewed 41 Pokémon GO players to understand how players coordinate and collaborate for in-person/remote raids and other social patterns. Our findings demonstrate that new social dynamics occurred: participants’ social interactions highly rely on external social media groups bridging cyberspace and the physical world. In such external social media groups, spontaneously formed leadership roles and mentor-mentee relationships demonstrate autonomy among players in the hybrid space. However, we observed that the interoperability issue challenges people’s experience. Overall, this work sheds light on the social interactions in LBGs as hybrid spaces.

The SEIR model is widely used in simulating the spread of infectious diseases. COVID-19 virus is a very severe infectious disease. Some studies leverage the SEIR or SEIRD model to simulate the spread and estimate the number of infected and recovered people as time goes on. However, these models suffer from two key deficiencies: (i) conventional SEIRD does not update its model parameters w.r.t. time; (ii) it focuses on predicting the trend, instead of the actual number of infections in the future. In this paper, we propose a cascade SEIRD model. The model learns and updates its parameters every day. Moreover, it is able to predict the number of infection cases, recovered cases and deaths. Specifically, we leverage a machine learning like approach to dynamically estimate the parameters of infection rate, incubation rate, recovery rate and death rate, which can be updated by gradient descent algorithm. Once the nature of the parameters w.r.t. time is determined, ARIMA model is adopted to characterize the dynamics of the parameters and predict their future changes. To validate the effectiveness of the proposed cascade SEIRD model, we conduct experiments on five data sets of different scales of regions (China, Hubei, Wuhan, Shenzhen, US). Experimental results show that the proposed cascade SEIRD achieves the most accurate prediction and outperforms state-of-the-art techniques.