Abstract Voice-based interaction is experiencing a second wind through the advent of machine learning (ML) techniques, affordable consumer products and renewed work on natural language processing (NLP) and large language models (LLMs). A growing body of work is exploring how users perceive new forms of computer-generated voices from qualitative and quantitative angles. However, critical voices have called for greater rigour, especially in confirming the voice as a manipulated variable, i.e. manipulation checks. We present three case studies that highlight the value of investing in rigorous manipulation checks for HCI researchers and designers. We demonstrate the importance of testing assumptions, the need for care and reflection in the design of response options and measurement and the advantages of more exploratory approaches to understanding user perceptions of and user experiences (UX) with voice phenomena. Through these case studies, we raise awareness, empirically justify and critically assess the value of manipulation checks for voice UX research and beyond.

Closed Captioning (CC) is a service primarily designed for deaf and hard of hearing (D/HoH) viewers. The CC translates spoken speech into text for television or film screen display. The quality assessment methods for live captioning are limited to quantitative measures, while the viewers are still dissatisfied with the current quality. One method to improve the current quality assessment procedure is to include D/HoH viewers in the evaluation procedure for their subjective assessment input. However, it could be costly and impractical to perform evaluations for the entire broadcasted shows. Therefore, it would be helpful to model subjective assessments that could replicate and predict human decisions. In this article, we report on a model of probabilities of D/HoH viewer assessment decisions for CC quality factors based on actual user preferences. An online survey was designed and conducted to collect assessment data for 22 error variation samples from four quality factors: delay, speed, missing words, and paraphrasing of captions. The results are analyzed using the signal detection theory framework to create decision probability models for D/HoH viewers.