News outlets are currently adopting AI to summarize news stories and experiment with conversational agents to convey news. In this paper, we explore this emerging practice, in which an AI-powered agent retells news in dialogue with the user rather than presenting them with a fixed narrative. In a co-speculation workshop with industry professionals and through a field trial of an LLM-based conversational news agent probe, we explore the design space of this conversational news format and examine the experiences, insights, and concerns reported by the participants. Our contribution is twofold. First, we identify five dimensions that shape how news can be retold by a conversational agent. Second, we provide a detailed empirical account of how users experience conversational news as clear and easy to follow, enabling them to probe and question stories in new ways, and address how these interactions are marked by tensions around trust, accuracy, and transparency.

Traditionally, sports commentators provide viewers with diverse information, encompassing in-game developments and player performances. Yet young adult football viewers increasingly use mobile devices for deeper insights during football matches. Such insights into players on the pitch and performance statistics support viewers’ understanding of game stakes, creating a more engaging viewing experience. Inspired by commentators’ traditional roles and to incorporate information into a single platform, we developed AiCommentator, a Multimodal Conversational Agent (MCA) for embedded visualization and conversational interactions in football broadcast video. AiCommentator integrates embedded visualization, either with an automated non-interactive or with a responsive interactive commentary mode. Our system builds upon multimodal techniques, integrating computer vision and large language models, to demonstrate ways for designing tailored, interactive sports-viewing content. AiCommentator’s event system infers game states based on a multi-object tracking algorithm and computer vision backend, facilitating automated responsive commentary. We address three key topics: evaluating young adults’ satisfaction and immersion across the two viewing modes, enhancing viewer understanding of in-game events and players on the pitch, and devising methods to present this information in a usable manner. In a mixed-method evaluation (n=16) of AiCommentator, we found that the participants appreciated aspects of both system modes but preferred the interactive mode, expressing a higher degree of engagement and satisfaction. Our paper reports on our development of AiCommentator and presents the results from our user study, demonstrating the promise of interactive MCA for a more engaging sports viewing experience. Systems like AiCommentator could be pivotal in transforming the interactivity and accessibility of sports content, revolutionizing how sports viewers engage with video content.

Advancements in Natural Language Processing (NLP) and Computer Vision (CV) are revolutionizing how we experience sports broadcasting. Traditionally, sports commentary has played a crucial role in enhancing viewer understanding and engagement with live games. Yet, the prospects of automated commentary, especially in light of these technological advancements and their impact on viewers’ experience, remain largely unexplored. This paper elaborates upon an innovative automated commentary system that integrates NLP and CV to provide a multimodal experience, combining auditory feedback through text-to-speech and visual cues, known as italicizing, for real-time in-game commentary. The system supports color commentary, which aims to inform the viewer of information surrounding the game by pulling additional content from a database. Moreover, it also supports play-by-play commentary covering in-game developments derived from an event system based on CV. As the system reinvents the role of commentary in sports video, we must consider the design and implications of multimodal artificial commentators. A focused user study with eight participants aimed at understanding the design implications of such multimodal artificial commentators reveals critical insights. Key findings emphasize the importance of language precision, content relevance, and delivery style in automated commentary, underscoring the necessity for personalization to meet diverse viewer preferences. Our results validate the potential value and effectiveness of multimodal feedback and derive design considerations, particularly in personalizing content to revolutionize the role of commentary in sports broadcasts.

AI-powered symptom checkers are automating the work of telephone triage nurses in assessing patient urgency. Yet, these systems exclude several vulnerable patient groups and overlook telenurses’ competent interaction with their patients. This study, conducted in collaboration with telenurses, examines how AI can support their clinical assessment and was carried out in four phases: 1) interviews that revealed telenurses’ challenge of juggling decision-support and documentation interfaces, 2) a co-design workshop that conceptualized continuous nurse-AI interaction, 3) development of a prototype that suggested questions for nurses to ask callers, and 4) a role-play workshop that demonstrated nurse-AI interaction in practice. The study addresses how we can design for control in human-AI collaboration in order to enhance, rather than replace, human decision-making processes.

The TV industry has long been under pressure to adapt its workflows to use advanced Internet technologies. It also must face competition from social media, video blogs, and livestreaming platforms, which are enabled by lightweight production tools and new distribution channels. The social-distancing regulations introduced due to the COVID-19 pandemic added to the list of challenging adaptations. One of the remaining bastions of legacy TV production is the live broadcast of sporting events and news. These production practices rely on tight collaboration in small spaces, such as control rooms and outside broadcast vans. This paper focuses on current socio-technical changes, especially those changes and adaptations in collaborative practices and workflows in TV production. Some changes necessary during the pandemic may be imposed, temporary adjustments to the ongoing situation, but some might induce permanent changes in key work practices in TV production. Further, these imposed changes are aligned with already ongoing changes in the industry, which are now being accelerated. We characterize the changes along two main dimensions: redistribution of work and automation.

Recovering from serious illness involves a bodily and psychosocial reorientation in everyday life. Survivors of gynecological cancer often experience bodily changes, fear of cancer recurrence, and changes in sexual health. This paper explores how we can use audiovisual narratives based on experiences of gynecological cancer survival in the design of an online intervention. From a typology of cancer survival, we designed three audiovisual narratives in an experience-centered design process involving gynecological cancer survivors. The narratives were evaluated by 10 participants formerly treated for gynecological cancer. In a thematic analysis, we explore how these narratives set the stage for identification and reflection by being experienced as relatable, provoking, and realistic. Finally, we discuss how the survivors’ experience of the narratives can be construed as meaningful, and how accounts of experiences can be included in a design process to create narrative content for online interventions.

Recent advances in artificial intelligence (AI) have led to an increased focus on automating media production. One relevant application area for AI is using speech recognition to create subtitles and closed captions for videos. The AI methods based on machine learning are still not sufficiently reliable in terms of producing perfect or acceptable subtitles. To compensate for this unreliability, AI can be used to build tools that support, rather than replace, human efforts and to create semi-automated workflows. In this paper, we present a prototype for including automated speech recognition for subtitling in an existing production-grade video editing tool. We devised an experiment with 25 participants and tested the efficiency and effectiveness of this tool compared to a fully manual process. The results show that there is a significant increase in both effectiveness and efficiency for novices in subtitling. Furthermore, the participants found the augmented process to be more demanding. We identify some usability issues and design choices that pertain to making augmented subtitling easier.

Video production is a collaborative process involving creative, artistic and technical elements that require a multitude of specialised skill sets. This open-ended work is often marked by uncertainty and interpretive flexibility in terms of what the product is and should be. At the same time, most current video production tools are designed for single users. There is a growing interest, both in industry and academia, to design features that support key collaborative processes in editing, such as commenting on videos. We add to current research by unpacking specific forms of collaboration, in particular the social mechanisms and strategies employed to reduce interpretive flexibility and uncertainty in achieving agreements between editors and other collaborators. The findings contribute to the emerging design interest by identifying general design paths for how to support collaboration in video editing through scaffolding, iconic referencing, and suggestive editing.

Virtual reality (VR) applications for exposure therapy predominantly use computer-generated imagery to create controlled environments in which users can be exposed to their fears. Creating 3D animations, however, is demanding and time-consuming. This paper presents a participatory approach for prototyping VR scenarios that are enabled by 360° video and grounded in lived experiences. We organized a participatory workshop with adolescents to prototype such scenarios, consisting of iterative phases of ideation, storyboarding, live-action plays recorded by a 360° camera, and group evaluation. Through an analysis of the participants' interactions, we outline how they worked to design prototypes that depict situations relevant to those with a fear of public speaking. Our analysis also explores how participants used their experiences and reflections as resources for design. Six clinical psychologists evaluated the prototypes from the workshop and concluded they were viable therapeutic tools, emphasizing the immersive, realistic experience they presented. We argue that our approach makes the design of VR scenarios more accessible.

This paper presents the design efforts involved in making a system for supporting haptic communication between dentist and patient during dental treatment. We describe Grasp Live, a haptic interaction technology consisting of a tangible stone-like object connected to a vibro-tactile feedback device.