The expanding feature set of modern headphones puts a challenge on the design of their control interface. Users may want to separately control each feature or quickly switch between modes that activate different features. Traditional approach of physical buttons may no longer be feasible when the feature set is large. Keyword spotting with voice commands is a promising solution to the issue. Most existing methods of keyword spotting only support commands spoken in a regular voice. However, regular voice may not be desirable in quiet places or public settings. In this paper, we investigate the problem of on-device keyword spotting in whisper voice and explore approaches to improve noise robustness. We leverage the inner microphone on noise-cancellation headphones as an additional source of voice input. We also design a curriculum learning strategy that gradually increases the proportion of whisper keywords during training. We demonstrate through experiments that the combination of multi-microphone processing and curriculum learning could improve F1 score of whisper keyword spotting by up to 15% in noisy conditions.

If our noise-canceling headphones can understand our audio environments, they can then inform us of important sound events, tune equalization based on the types of content we listen to, and dynamically adjust noise cancellation parameters based on audio scenes to further reduce distraction. However, running multiple audio understanding models on headphones with a limited energy budget and on-chip memory remains a challenging task. In this work, we identify a new class of neural network accelerators (e.g., NE16 on GAP9) that allows network weights to be quantized to different common (e.g., 8 bits) and uncommon bit-widths (e.g., 3 bits). We then applied a differentiable neural architecture search to search over the optimal bit-widths of a network on two different sound event detection tasks with potentially different requirements on quantization and prediction granularity (i.e., classification vs. embeddings for few-shot learning). We further evaluated our quantized models on actual hardware, showing that we reduce memory usage, inference latency, and energy consumption by an average of 62%, 46%, and 61% respectively compared to 8-bit models while maintaining floating point performance. Our work sheds light on the benefits of such accelerators on sound event detection tasks when combined with an appropriate search method.

Modern noise-cancelling headphones have significantly improved users’ auditory experiences by removing unwanted background noise, but they can also block out sounds that matter to users. Machine learning (ML) models for sound event detection (SED) and speaker identification (SID) can enable headphones to selectively pass through important sounds; however, implementing these models for a user-centric experience presents several unique challenges. First, most people spend limited time customizing their headphones, so the sound detection should work reasonably well out of the box. Second, the models should be able to learn over time the specific sounds that are important to users based on their implicit and explicit interactions. Finally, such models should have a small memory footprint to run on low-power headphones with limited on-chip memory. In this paper, we propose addressing these challenges using HiSSNet (Hierarchical SED and SID Network). HiSSNet is an SEID (SED and SID) model that uses a hierarchical prototypical network to detect both general and specific sounds of interest and characterize both alarm-like and speech sounds. We show that HiSSNet outperforms an SEID model trained using non-hierarchical prototypical networks by 6.9 – 8.6%. When compared to state-of-the-art (SOTA) models trained specifically for SED or SID alone, HiSSNet achieves similar or better performance while reducing the memory footprint required to support multiple capabilities on-device.

Smart thermostats have been proposed as a way to reduce energy consumption in the home. While occupancy-based thermostat control and scheduling has been shown to provide energy savings, more recent work in comfort-aware thermostats promises to provide even greater savings. Comfort awareness and adaptive thermal comfort models, combined with the mixed-initiative eco-coaching approach to thermostat control, offer a promising approach to optimizing savings by offering both schedule and setpoint recommendations and actionable plans. In this paper, we investigate the design space of comfort-aware eco-coaching thermostats. Through a user enactment study wherein 11 participants encountered fifteen design probes covering various design attributes and interaction scenarios, we uncover insights on how to design such thermostats in a way that respect people's values relating to comfort, sustainability, control, convenience, and allocation of agency while also encouraging more energy efficient behaviors.

For decades, researchers have investigated ways to infer human thermal comfort. Studies have usually required cumbersome sensors and human observers, making them inappropriate for use in naturalistic settings such as the home. Emerging wearable and smart home sensing devices offer the opportunity to develop new models of thermal comfort based on data collected in-situ. To explore this opportunity, we deployed a sensing system in seven homes and collected self-report data from 11 participants for four weeks. Our system captures many factors employed in previous thermal comfort research, as well as new factors (e.g., activity level, sweat level). Machine learning-based models derived from the collected data show improvement over previous techniques, however significant prediction errors remain. In analyzing these errors we identify six problems that pose challenges for inferring comfort in the wild. Based on our findings, we suggest techniques to improve future in-situ thermal comfort modeling efforts.

Learning storytelling is beneficial for children's development. Various tools have been proposed to expand the set of materials children can use to compose their stories. However, most previous research focuses on enhancing the visual aspect of storytelling and underexplores the acoustic elements needed for children's stories. In this paper, Olegoru1, a sound composition tool in the form of magic gloves and soul stones, is proposed to augment children's storytelling when using physical objects. Children can create contextual and regional sound effects as well as event-based acoustics through speech, non-verbal and gestural interaction, and could potentially enable children to create more immersive story-worlds. To investigate the technical feasibility of such tool, a preliminary prototype was built that accepts a limited number of vocalized sound effects and vocabulary.