Sound has long been used to enhance eating experiences, and increasingly, we know sound can shape how we perceive taste. Yet intentionally composing sounds to modulate taste remains underexplored in cross-sensory HCI design. While prior research has mapped individual auditory parameter (e.g., pitch, timbre, rhythm) to basic taste perceptions, less is known about how holistic sound compositions combining multiple parameters can be systematically designed and evaluated. We present a human-in-the-loop workflow that brings together professional sound designers and generative AI to create 160 sounds with taste intentions informed by established auditory-taste parameters. In a study with 149 participants on sound-taste correspondences, we identify sounds that strongly correspond to basic taste words. Our contributions are threefold: (1) a workflow for designing taste-matching sounds; (2) a curated repository of reusable sound stimuli for cross-sensory research; and (3) a designerly framework for creating compositional taste-matching sounds alongside insights into sound-taste word correspondences and association strategies, offering implications for cross-sensory HCI.

Surfing, like many water sports, offers a unique opportunity to experience natural environments, yet the design of interactive technologies for water activities– “WaterHCI” – so far prioritised athletic performance over experience. To explore this opportunity, we designed SurfSync, a novel wearable system in the form of an actuating vest and a hat that provides oceanic information through sound, vibration, and heat actuation, aimed at enriching the surfing experience as a playful encounter with the ocean. We studied SurfSync in an ocean-based field study with eight surfers. Through thematic analysis of interviews, we articulated the surfers’ experiences, indicating how they made sense of playful cues while in the ocean. By reflecting on our soma design process and the surfers’ experiences, we provide six design strategies to enrich the surfing experience. Our work contributes to the emerging field of WaterHCI by providing insights into how wearables can enrich human-water experiences outdoors.

Dining is a social activity, yet human–computer interaction (HCI) research has rarely examined how extended reality (XR) headsets shape shared restaurant experiences. Although XR can augment taste perception, headsets also obscure facial expressions – particularly pertinent when only some diners wear them. We identify an opportunity to make a wearer’s biodata socially visible via an outward-facing display on the headset to facilitate novel social dining experiences with XR. To explore this, we designed E.A.T. (Expressive Augmented Togetherness), a novel XR dining system that combines food augmentation with external displays depicting physiological arousal and pupil movement to facilitate social interactions. Through interviews following a restaurant field study with five groups of three diners, we examined how the system influenced taste perception and social interactions. We identify six design considerations for socially legible XR systems for dining, ultimately, aiming to inform how HCI can better support social dining experiences.

Facial expression interactions play a crucial role in fostering social bonds and expressing emotions. However, in the dynamic, fast-paced, and noisy environments of parties, various factors hinder blind and low-vision (BLV) individuals from engaging fully in facial expression interactions. While previous research has explored how BLV users can convey emotions through non-verbal visual cues, it has largely overlooked the challenges they face in engaging with facial expressions after perceiving these cues. To address this gap, we conducted a formative study with 10 BLV users to identify their challenges and expectations regarding facial expression interactions in party settings. Guided by these insights, we developed EmojiFan, an AI-powered smart fan designed to offer a personalized representation of facial expressions through dynamic, expressive emojis. Finally, we carried out an in-the-field study with 6 BLV participants and 8 sighted social partners to examine the effectiveness of EmojiFan in enhancing facial-expression interactions during parties. Overall, our goal is to empower BLV individuals’ autonomy to actively participate in social interactions through digital facial expression, thereby contributing new insights for the accessibility community on designing expressive, socially responsive assistive technologies.

Sounds play a crucial role in shaping dining experiences. Recently, designers have increasingly integrated them into interfaces that connect diners with food. Yet, little is known about how sounds can function as culinary materials to enrich chefs’ creativity, particularly in creating meaningful auditory interactions that resonate with their culinary creations. Hence, we present GastroConcerto, an auditory dining system that combines a magnetic contact microphone equipped under the plate with a companion mobile application. This system delves into the interactive space between diners and their food to introduce a novel interactive mechanism of dining–sound pairing, enabling chefs to design specific auditory interactions that respond to diners’ individual interactions with the food on dining containers. Through a within-subjects study and field deployment, we examined how GastroConcerto enriches chefs’ creative practices in crafting "sonic dish" experiences. Ultimately, our goal is to shift the ownership of auditory interaction design from interface designers to chefs, thereby supporting their culinary creativity.

Drinking is an inherently multisensory activity, yet the potential of immersive technology to dynamically shape flavor experiences remains underexplored in Human-Food Interaction (HFI) research. We introduce “XTea”, an adaptive beverage cup-based system that integrates large language models to translate natural language input into modifications of a parameterized immersive environment experienced through a headset when drinking bubble tea. Through a study with 12 bubble tea enthusiasts, we derived themes that demonstrate how “XTea” can enrich sensory engagement, support personalized and agentic experiences, and foster social qualities of drinking, pointing toward new explorations for multisensory HFI design. We also present four design strategies for multisensory beverage experiences. Ultimately, we aim to contribute to the advancement of HFI research on how multisensory interaction design can enrich flavor perception and engagement.

The more-than-human field has contributed numerous opportunities for interacting with nature, animals, plants, and microorganisms. However, few studies have examined water ecosystems. Current water-related work primarily treats water as a medium for human-centered activities, rarely positioning water as an interactive subject. Building upon prior research, we explore how to better integrate water more playfully into digital-physical interactions as an interactive subject. We designed and developed Water’s Echo, an AI-powered public installation that enables human-water communication through a playful stone-throwing dialogue. We conducted a field study at a local pond, recruiting 15 residents to participate in Water’s Echo—a playful conversational interaction. Our findings indicate that this playful dialogue approach raises participants’ awareness and understanding of surrounding aquatic environments. This research provides insights for design researchers to establish engaging water ecology interactions across cultural communities, promoting a More-than-human perspective in re-examining human-nature relationships.

While interest in blending sound with culinary experiences has grown in Human-Food Interaction (HFI), the significance of food's material properties in shaping sound-related interactions has largely been overlooked. This paper explores the opportunity to enrich the HFI experience by treating food not merely as passive nourishment but as an integral material in computational architecture with input/output capabilities. We introduce “Sonic Delights,” where food is a comestible auditory-gustatory interface to enable users to interact with and consume digital sound. This concept redefines food as a conduit for interactive auditory engagement, shedding light on the untapped multisensory possibilities of merging taste with digital sound. An associated study allowed us to articulate design insights for forthcoming HFI endeavors that seek to weave food into multisensory design, aiming to further the integration of digital interactivity with the culinary arts.

During the COVID-19 pandemic, online classes became the only option for many students. The main challenge for these classes was conducting risky and complex chemical or biological experiments in a domestic environment. To address this challenge, a smart experiment system called MRLab was developed. MRLab used wearables such as a smart glove and head-mounted device to record sensory data and a multimodal hybrid fusion model GVVS to interpret the user’s experimental intent, which essentially transforms the user’s abstract behavioral actions into a probabilistic set of experimental intent that can be computed. Different experiments in MRLab used different libraries of experimental intents. The SrNet model in GVVS was used to estimate the probability of the user’s gesture behavior generated from the smart glove, while the SIPA algorithm compared speech information entered during the experiment with the experimental intent library to estimate the probability of the user’s intent. At the same time, the scene visual channel monitored the information about the object the user intended to operate, with the SVF algorithm computing the probability of the intended object in real-time. The results from ANOVA and post-hoc comparative testing conducted on 21 volunteers revealed that MRLab outperformed other experiment modes, including WEB, AR, and VR, with a higher intention understanding rate, efficiency, and user satisfaction. Therefore, MRLab proved to be a useful alternative to traditional physics laboratory experiments during the pandemic, along with being an additional teaching tool for remote learning purposes.

There is an increasing interest in combining interactive technology with food, leading to a new research area called human-food interaction. While food experiences are increasingly benefiting from interactive technology, for example in the form of food tracking apps, 3D-printed food and projections on dining tables, a more systematic advancement of the field is hindered because, so far, there is no comprehensive articulation of the grand challenges the field is facing. To further and consolidate conversations around this topic, we invited 21 HFI experts to a 5-day seminar. The goal was to review our own and prior work to identify the grand challenges in human-food interaction. The result is an articulation of 10 grand challenges in human-food interaction across 4 categories (technology, users, design and ethics). By presenting these grand challenges, we aim to help researchers move the human-food interaction research field forward.