Brain-Computer Interfaces (BCIs) can decode imagined speech from neural activity. However, these systems typically require extensive training sessions where participants imaginedly repeat words, leading to mental fatigue and difficulties identifying the onset of words, especially when imagining sequences of words. This paper addresses these challenges by transferring a classifier trained in overt speech data to covert speech classification. We used electroencephalogram (EEG) features derived from the Hilbert envelope and temporal fine structure, and used them to train a bidirectional long-short-term memory (BiLSTM) model for classification. Our method reduces the burden of extensive training and achieves state-of-the-art classification accuracy: 86.44% for overt speech and 79.82% for covert speech using the overt speech classifier.

As voice-enabled technologies are becoming increasingly more prevalent, voice-enabled permission requests become a crucial topic of investigation. It is yet unclear how to appropriately inform users in voice user interfaces (VUIs) about data processing practices. To understand how modality (text vs. voice) and the speech rate of the voice can influence users’ perceptions and decisions to grant permission, we conducted two preregistered studies (N = 343 and N = 594) and one pre-study, including two listening tasks to design potentially deceptive voice patterns. We found that users can distinguish between different levels of intrusiveness in the voice modality. However, they are less likely to accept voice-based permissions, pointing to cognitive problems associated with them. Moreover, we found that speech rate manipulations of action verbs “Accept” and “Decline” shifted users’ decisions towards acceptance, making the effect less controllable than predicted. This work highlights implications and design considerations for future voice-enabled permission requests.

Manipulative design in user interfaces (conceptualized as dark patterns) has emerged as a significant impediment to the ethical design of technology and a threat to user agency and freedom of choice. While previous research focused on exploring these patterns in the context of graphical user interfaces, the impact of speech has largely been overlooked. We conducted a listening test (N = 50) to elicit participants’ preferences regarding different synthetic voices that varied in terms of synthesis method (concatenative vs. neural) and prosodic qualities (speech pace and pitch variance), and then evaluated their impact in an online decision-making study (N = 101). Our results indicate a significant effect of voice qualities on the participant’s choices, independently from the content of the available options. Our results also indicate that the voice’s perceived engagement, ease of understanding, and domain fit directly translate to its impact on participants’ behavior in decision-making tasks.

In the rapidly evolving field of artificial intelligence (AI) agents, designing the agent's characteristics is crucial for shaping user experience.This workshop aims to establish a research community focused on AI agent persona design for various contexts, such as in-car assistants, educational tools, and smart home environments.We will explore critical aspects of persona design, such as voice, embodiment, and demographics, and their impact on user satisfaction and engagement.Through discussions and hands-on activities, we aim to propose practices and standards that enhance the ecological validity of agent personas.Topics include the design of conversational interfaces, the influence of agent personas on user experience, and approaches for creating contextually appropriate AI agents.This workshop will provide a platform for building a community dedicated to developing AI agent personas that better fit diverse, everyday interactions.

Abstract Graphical user interface (GUI) browsing and retrieval tools are becoming essential to interaction design research and practice. These tools allow GUI designers to browse large amounts of data and recover inspiring or relevant designs for their task. Unfortunately, data-driven market analysis or business intelligence (BI) applied to GUIs have mostly been left aside. To address this research gap, we elicit designers’ needs and responses regarding the development of Enricommender, a high-fidelity prototype of a market analysis recommender and reporting system. We identify and discuss key design challenges, as reported by more than 200 real-world designers, as well as their workflows and overall expectations towards such a BI system. Ultimately, this article sets the foundation for developing future GUI-oriented BI applications. RESEARCH HIGHLIGHTS Insights about ideation and design of graphical user interfaces. Elicited preferences and current practices from 200 real-world designers. A business intelligence prototype for comparative analysis and design recommendations.

Conversational agents (CAs) that deliver proactive interventions can benefit users by reducing their cognitive workload and improving performance. However, little is known regarding how such interventions would impact users’ reflection on choices in voice-only decision-making tasks. We conducted a within-subjects experiment to evaluate the effect of CA’s feedback delivery strategy at three levels (no feedback, unsolicited and solicited feedback) and the impact on users’ likelihood of changing their choices in an interactive food ordering scenario. We discovered that in both feedback conditions the CA was perceived to be significantly more persuasive than in the baseline condition, while being perceived as significantly less confident. Interestingly, while unsolicited feedback was perceived as less appropriate than the baseline, both types of proactive feedback led participants to relisten and reconsider menu options significantly more often. Our results provide insights regarding the impact of proactive feedback on CA perception and user’s reflection in decision-making tasks, thereby paving a new way for designing proactive CAs.

We applied state-space modelling technique to estimate the cognitive workload of a speech-in-noise (SIN) recall task, based on participants’ oculo-motor responses to speech signals. We estimated common latent attention levels in 15 time bins and observed temporal changes between pupillary dilations and saccade frequencies, given that the both conditions were independent. We also compared two speech type factors (natural vs. synthetic) and three levels of signal-to-noise (-1dB, -3dB, and -5dB) using the estimated parameter distribution. The comparison of experimental factors provided us with insights into differences in participants’ processing of spoken information during a SIN recall task.

Trust is a prerequisite for effective human-agent collaboration. While past work has studied how trust relates to an agent's reliability, it has been mainly carried out in turn based scenarios, rather than during real-time ones. Previous research identified the performance of an agent as a key factor influencing trust. In this work, we posit that an agent's predictability also plays an important role in the trust relationship, which may be observed based on users' interactions. We designed a 2x2 within-groups experiment with two baseline conditions: (1) no agent (users' individual performance), and (2) near-flawless agent (upper bound). Participants took part in an interactive aiming task where they had to collaborate with different agents that varied in terms of their predictability, and were controlled in terms of their performance. Our results show that agents whose behaviours are easier to predict have a more positive impact on task performance, reliance and trust while reducing cognitive workload. In addition, we modelled the human-agent trust relationship and demonstrated that it is possible to reliably predict users' trust ratings using real-time interaction data. This work seeks to pave the way for the development of trust-aware agents capable of adapting and responding more appropriately to users.

Voice search is currently widely available on the majority of mobile devices via use of Virtual Personal Assistants. However, despite its general availability, the use of voice interaction remains sporadic and is limited to basic search tasks such as checking weather updates and looking up answers to factual queries. Present-day voice search systems struggle to use relevant contextual information to maintain conversational state, and lack conversational initiative needed to clarify user's intent, which hampers their usability and prevents users from engaging in more complex interaction activities. This research investigates the potential of a hypothesised interactive information retrieval system with human-like conversational abilities. To this end, we propose a series of usability studies that involve a working prototype of a conversational system that uses real time speech synthesis. The proposed experiments seek to provide empirical evidence that enabling a voice search system with human-like conversational abilities can lead to increased likelihood of its adoption.