Device orchestration in ubiquitous computing remains cumbersome and highly static, particularly given the growing number of personal devices. Routine tasks, such as preparing to read at home, often require multiple manual interactions (lighting, heating, music), which are difficult to automate with rule-based systems due to contextual sensitivity. We present GenX, a perception-driven approach that applies multimodal generative AI to create dynamic, adaptive user experiences. By leveraging vision-language models, GenX interprets a user’s perception of their environment and generates a multimodal UX tailored to available output modalities. The system produces and executes source code in real time, enabling devices to be orchestrated flexibly and contextually. We introduce the concept of Generative Experiences as a scalable solution for dynamic UX design, capable of handling unexpected situations. GenX is demonstrated in two in-the-wild scenarios and a user study, showing positive user perceptions and effectiveness in supporting everyday tasks.

Stories have long captivated the human imagination with narratives that enrich our lives. Traditional storytelling methods are often static and not designed to adapt to the listener’s environment, which is full of dynamic changes. For instance, people often listen to stories in the form of podcasts or audiobooks while traveling in a car. Yet, conventional in-car storytelling systems do not embrace the adaptive potential of this space. The advent of generative AI is the key to creating content that is not just personalized but also responsive to the changing parameters of the environment. We introduce a novel system for interactive, real-time story narration that leverages environment and user context in correspondence with estimated arrival times to adjust the generated story continuously. Through two comprehensive real-world studies with a total of 30 participants in a vehicle, we assess the user experience, level of immersion, and perception of the environment provided by the prototype. Participants’ feedback shows a significant improvement over traditional storytelling and highlights the importance of context information for generative storytelling systems.

Augmented Reality (AR) can alter environments and steer attention. While prior work dominantly focuses on exploring performances of augmentations, this work aims to understand the societal impact of AR in complex social situations. Focusing on prosocial helping behaviour, we created two scenarios and designed five augmentations aiming to motivate a user to help. We wanted to understand (1) the impact on situation perception and (2) the impact on the social structure. In an online video experiment (n=294), we found that augmenting can impact anxiety about the situation and significantly increase the perceived reason to help being directed towards the augmentation. Similarly, we found that the helped rated the "reason" and "thankfulness" significantly higher towards AR than the helper, creating a disagreement around agency and responsibility. We discuss the implications of AR in complex social structures and how responsibility and agency will become important when embedding AR in our social lives.

Autonomous vehicles (AVs) are expected to communicate to vulnerable road users as a substitution of, for example, driver-pedestrian communication, leading to increased safety and acceptance. This communication is currently one-directional, i.e., from the AV to the pedestrian. However, today’s communication between drivers and pedestrians in crossing scenarios is bidirectional. Pedestrians gesture “thank you’’ or wave drivers through in case they do not want to cross. Human drivers often acknowledge this, for example, with a nod. We present an experiment in Virtual Reality (N=20), in which the effect of such acknowledgment of the AVs via its external communication is investigated for the two described scenarios and concerning pedestrian presence. Results show that such feedback is perceived as highly necessary, depends on the scenario, and improves the perceived intelligence of the AV, confirming a Halo-Effect.

Traffic behavior and its perception is shaped by various factors such as vehicle color or size. Decals are used to express information about the owner’s beliefs or are intended to be funny. In the future, with external displays on (automated) vehicles, individualized customization could be even more pronounced. While some research looked at the messages these decals convey, it is unclear how these decals influence the perception of surrounding drivers on the operator of the vehicle. We gathered data on decals in 29 cities in 8 countries. A thematic analysis unveiled 17 dominant themes among decals. Subsequently, we investigated effects of decals of 9 supra-regional common themes in an online study (N=64) finding that participants attributed different characteristics to the driver of a vehicle with a decal based on the type of decal and the participants’ country of origin. Additionally, a Virtual Reality study (N=16) revealed diverse opinions on future usage of such personalization options.