Silent speech recognition has emerged as a promising approach for enabling hands-free and discreet interaction with head-worn devices. In this paper, we present QuietSync, a multimodal system that combines inertial measurement unit (IMU) and contact electrode (ExG) signals to achieve accurate silent speech recognition using off-the-shelf devices. QuietSync utilizes an IMU attached to the lower part of the headphones near the ear and strategically places ExG electrodes on the headphones, glasses (nose and behind the ear), and face (for VR applications) to capture subtle movements and muscle activity associated with silent speech production. We conducted a user study with 9 participants and successfully recognized 12 commands with an accuracy of 94.2%. Our system leverages the complementary nature of IMU and ExG signals to enhance the robustness and reliability of silent speech recognition. The IMU captures subtle movements of the jaw and facial muscles, while the ExG electrodes detect low-amplitude surface muscle activity associated with speech production. We show that our system is not affected by the length and speech mannerisms of the commands, and can be fine-tuned for users of varied native languages with only 5 samples. Our findings demonstrate the feasibility of using off-the-shelf head-worn devices to enable silent speech recognition, opening up new possibilities for seamless and discreet interaction with devices such as VR/AR headsets and earables. To the best of our knowledge, QuietSync is the first system to enable silent speech interaction for multiple form factors.

Facebook Reactions are a collection of animated icons that enable users to share and express their emotions when interacting with Facebook content. The current design of Facebook Reactions utilizes visual stimuli (animated graphics and text) to convey affective information, which presents usability and accessibility barriers for visually-impaired Facebook users. In this paper, we investigate the use of sonification as a universally-accessible modality to aid in the conveyance of affect for blind and sighted social media users. We discuss the design and evaluation of 48 sonifications, leveraging Facebook Reactions as a conceptual framework. We conducted an online sound-matching study with 75 participants (11 blind, 64 sighted) to evaluate the performance of these sonifications. We found that sonification is an effective tool for conveying emotion for blind and sighted participants, and we highlight sonification design strategies that contribute to improved efficacy. Finally, we contextualize these findings and discuss the implications of this research with respect to HCI and the accessibility of online communities and platforms.

Interfaces designed to elicit empathy provide an opportunity for HCI with important pro-social outcomes. Recent research has demonstrated that perceiving expressive biosignals can facilitate emotional understanding and connection with others, but this work has been largely limited to visual approaches. We propose that hearing these signals will also elicit empathy, and test this hypothesis with sounding heartbeats. In a lab-based within-subjects study, participants (N = 27) completed an emotion recognition task in different heartbeat conditions. We found that hearing heartbeats changed participants’ emotional perspective and increased their reported ability to “feel what the other was feeling.” From these results, we argue that auditory heartbeats are well-suited as an empathic intervention, and might be particularly useful for certain groups and use-contexts because of its musical and non-visual nature. This work establishes a baseline for empathic auditory interfaces, and offers a method to evaluate the effects of future designs.

Deep learning paradigms have revolutionized the field of brain-computer interfacing and enabled the use of complex, nuanced methods for recognizing mental states. Prior work has demonstrated that these models can recognize the mental state in a variety of tasks, but few have specifically explored their performance concerning human factors that come into play with the use of brain-computer interfaces in day-to-day applications. For this research, we explored the use of 1D-convolutional neural networks to recognize two mental states–mental arithmetic and rest–from electroencephalograph signals. We focused our analysis on three parameters that affect the design and usability of brain-computer interfaces: input data representation (i.e. domain), window size (i.e. latency), and electrode montage (i.e. form-factor). In line with prior work, we found a clear bias in performance towards the frequency domain representation. We also found that training our model with short windows of time (i.e. 0.25s) provided close to peak accuracy. Furthermore, high accuracy was maintained with sparse electrode subsets of the full 10-20 system. We discuss these findings and how they can contribute to ongoing work to bring deep learning enabled brain-computer interfaces into day-to-day applications.

Design thinking is an approach to educational curriculum that builds empathy, encourages ideation, and fosters active problem solving through hands-on design projects. Embedding participatory “co-design” into design thinking curriculum offers students agency in finding solutions to real-world design challenges, which may support personal empowerment. An opportunity to explore this prospect arose in the design of sounds for an accessible interactive science-education simulation in the PhET Project. Over the course of three weeks, PhET researchers engaged blind and visually-impaired high-school students in a design thinking curriculum that included the co-design of sounds and auditory interactions for the Balloons and Static Electricity (BASE) sim. By the end of the curriculum, students had iterated through all aspects of design thinking and performed a quantitative evaluation of multiple sound prototypes. Furthermore, the group’s mean self-efficacy rating had increased. We reflect on our curriculum and the choices we made that helped enable the students to become authentic partners in sound design.