The Wizard of Oz (WoZ) method is a widely adopted research approach where a human Wizard “role-plays” a not readily available technology and interacts with participants to elicit user behaviors and probe the design space. With the growing ability for modern large language models (LLMs) to role-play, one can apply LLMs as Wizards in WoZ experiments with better scalability and lower cost than the traditional approach. However, methodological guidance on responsibly applying LLMs in WoZ experiments and a systematic evaluation of LLMs’ role-playing ability are lacking. Through two LLM-powered WoZ studies, we take the first step towards identifying an experiment lifecycle for researchers to safely integrate LLMs into WoZ experiments and interpret data generated from settings that involve Wizards role-played by LLMs. We also contribute a heuristic-based evaluation framework that allows the estimation of LLMs’ role-playing ability in WoZ experiments and reveals LLMs’ behavior patterns at scale.

Educational live streaming has become a complement to in-person teaching. While synchronous instructor-learner communication is useful, the technology-mediated nature of live streaming can obscure many interaction cues (e.g., learners' facial expressions and body language), which dampens the instructors' ability to respond to remote learners' needs. We explore the opportunity of leveraging real-time transcripts generated from instructors' audio as a basis for re-creating interaction cues. Transcripts can be leveraged to reveal the content of live streams in a form that learners can trace back and annotate, and such annotations can be further aggregated and presented to instructors as signals to assist them in tracking learners' engagement. By designing and evaluating our proof-of-concept prototype system, EduLive, we show that instructors benefited from the summative information extracted from learners' annotations, and the context provided by the transcript enhanced their ability to answer learners' questions. Our system contributes to the design space of social annotations in CSCW by employing social annotations in educational live streaming scenarios.

Online platforms commonly collect and display user-generated information to support subsequent users’ decision-making. However, studies have noticed that presenting collective information can pose social influences on individuals’ opinions and alter their preferences accordingly. It is essential to deepen understanding of people’s preferences when exposed to others’ opinions and the underlying cognitive mechanisms to address potential biases. Hence, we conducted a laboratory study to investigate how products’ ratings and reviews influence participants’ stated preferences and cognitive responses assessed by their Electroencephalography (EEG) signals. The results showed that social ratings and reviews could alter participants’ preferences and affect their status of attention, working memory, and emotion. We further conducted predictive analyses to show that participants’ Electroencephalography-based measures can achieve higher power than behavioral measures to discriminate how collective information is displayed to users. We discuss the design implications informed by the results to shed light on the design of collective rating systems.

Video-based learning is widely adopted by online learners, yet, learning experience and quality may be negatively affected by asynchronous and remote natures of video-based learning. As note-taking is a common practice employed by video-based learners and is known to be an effective way to trigger active construction and processing of knowledge, yet as a meta-skill, it is challenging to most learners. In this study, we aim to approach the goal of providing cognitive and social scaffolds to video-based learners by structuring their note-taking process. We presented and evaluated structured note-taking systems designed for learners in two contexts, namely, individual learning context and social learning context. With an online controlled study involving 43 participants, we compared the structured note-taking systems with two baseline systems (for individual learning and social learning contexts respectively) and found that structured note-taking significantly improved certain aspects of video-based learning such as and higher cognitive engagement and lower distraction. We discussed our results to inform the design, iteration, and adoption of note-taking tools in video-based learning.