’How was your lunch?’ Even simple food talk reveals how people translate multisensory impressions into language, offering insights for HCI applications such as food recommendation and health management. While prior work has emphasised the multisensory nature of eating, less is known about the linguistic strategies through which people articulate such experiences. We conducted a mixed-methods study combining self-reports and interviews to examine how non-verbal sensory input is transformed into verbal expression. Our analysis shows that participants structured their accounts through temporal phases that integrated perception, cognition, affective reaction, and behaviour; employed associative strategies that shaped expectations; and expressed ambivalence, where positive and negative evaluations coexisted in descriptions such as guilty pleasure. Building on these findings, we propose a lens that highlights the role of time, association, and evaluative language in food talk, enabling designers to translate everyday expressions into actionable insights for food-related HCI design.

Ultrasonic mid-air haptics (UMH) offers a novel way to modulate affective responses through contactless, biosignal features inspired haptics (i.e., BioHaptics). Yet, the effects of different BioHaptics on affective responses of emotions experienced during video-watching context are unclear, limiting the flexible emotion modulation and broader use of UMH as a natural affective design channel in enhancing everyday media consumption. This paper explores how BioHaptics encoded from different biosignal features (e.g., heart rate (HR), heart rate variability (HRV), and respiration amplitude (RA)) impact emotions in video-watching contexts. In two experiments with 70 participants, we assessed affective responses while adding three BioHaptics during the video-watching processes. Results showed that HR and RA haptics promoted RA and HRV, with potential increments in emotional pleasantness and regulation; HRV haptics lowered HR, leading to calmer responses. This research offers implications and recommendations for emotion modulation through UMH and advances the design of emotionally engaging multisensory feedback.

Aligning large language models (LLMs) behaviour with human intent is critical for future AI. An important yet often overlooked aspect of this alignment is the perceptual alignment. Perceptual modalities like touch are more multifaceted and nuanced compared to other sensory modalities such as vision. This study investigates how well LLMs can understand and interpret human touch experiences by focusing on their capacity to perceive the tactile qualities of everyday objects. For instance, it assesses whether LLMs can recognize that silk satin is softer and smoother than cotton denim. We developed a “Guess What Textile“ interaction using a custom AI system that enables participants to narrate their touch experiences in the “textile hand” task. Participants were given two textile samples–a target and a reference–to handle. Without seeing them, participants described the differences between them to the LLM. Using these descriptions, the LLM attempted to identify the target textile by assessing similarity within its high-dimensional embedding space, where its perceptual representations are encoded. Our results suggest that a degree of perceptual alignment exists; however, it varies significantly among different textile samples. For example, LLM predictions are well aligned for silk satin, but not for cotton denim. Moreover, participants felt that their textile experiences were not closely matched by the LLM predictions. This study is the first exploration into perceptual alignment around touch using LLM encoders, exemplified through textile hand task. We discuss possible sources of this alignment variance, and how better human-AI perceptual alignment can benefit future everyday tasks. • We address the gap in understanding perception alignment between human touch and AI. • First study on alignment between human touch experiences and LLMs in embeddings. • A novel interactive task probes LLMs’ learned representations for human alignment. • LLMs show perceptual biases, aligning better with certain textiles than others.

Figure 1: The research involved: (i) a design futuring workshop on acoustophoresis, (ii) the development and feasibility assessment of two SONARIOS (i.e., scenarios), and (iii) the synthesis of three strong concepts, along with key tensions and responsibilities in shaping the future of acoustophoresis applications.

Recovering a lost ability is rarely easy, but unlike strengthening a muscle, progress in smell training, repeated exposure to specific scents to support recovery or maintain function, is often invisible. For those undergoing Digital Smell Training (DST), data visualisations may be the only markers of change. But how can graphs and numbers sustain motivation over months of slow, unpredictable recovery? This pictorial adopts a Research through Design approach to explore how data visualisations might better support motivation, adherence, and long-term engagement in DST. We draw on a six-month in-home study with 18 participants with varying olfactory abilities using a technology probe. Following initial feedback, we ran a co-design workshop to understand participants’ visualisation needs. These insights informed eight design directions and three visualisation concepts, later evaluated in a focus group. We explore how visualisations help ‘flip perspectives’, from tracking outcomes to nurturing perseverance across the uncertain journey of smell rehabilitation.

MMFood'25, the 1st International Workshop on Multi-modal Food Computing, brings together researchers and practitioners at the intersection of artificial intelligence, computer vision, natural language processing, and sensory modeling to advance the study of food. The workshop highlights how multimodal methods can be applied to food recognition, recommendation, analysis, and monitoring to address pressing challenges in health, nutrition, sustainability, and food culture. It features a rich program including a keynote, an invited talk, paper presentations, a poster session, and a panel discussion on the role of multimodal AI in preserving cultural heritage, fostering sustainable food futures, and enabling personal well-being. By convening experts from academia, industry, and healthcare, MMFood'25 provides a unique platform for fostering interdisciplinary collaboration and for shaping the emerging field of multimodal food computing. The workshop proceedings can be found at: https://dl.acm.org/doi/proceedings/10.1145/3746264.

Perceived Continuity CRE f) FE STM CRE FE STM a) 1 2 3 3 4 c) b) d) Figure 1: a) Different sensitivity areas across the hand overlaid with pacinian receptors (dots).We create tactile shapes at low (c) and high (b) sensitivity areas, as well as across areas (d and e), comparing their perceived intensity and continuity when using spatiotemporal modulation (STM) and tactile illusions (funneling effect (FE) and cutaneous rabbit effect (CRE) ) (f).

With perpetually stressful lives, people are prompted to consider slowing down through mindfulness meditation. We extend HCI research beyond its pre-existing focus on interaction-intensive meditation technologies, to focus simply on whether one’s sense of touch can facilitate meditation. We capitalise on the touchless nature of mid-air haptics by rendering subtle sensations on the palm during meditation, using one design derived from pilot testing. In a novel and exploratory between-subjects study, we compare audio versus audio-haptic guidance in adult novices’ mindfulness meditation, through a mixed methods approach which combines standardised questionnaires with micro-phenomenological interviews ( n =10 per group). Interestingly, mid-air haptics quantitatively increased both hindrance and relaxation, which appears conflicting but is actually complementary when considered together with qualitative analysis. Specifically, mid-air haptics initially distracted participants but eventually facilitated mindfulness through three processes (embodied metaphor, breath alignment, and meditative grounding), though this requires further validation. We reflect on the need for future work exploring an integrated, multimodal meditation experience.

Sensory perception and emotional evaluation of a stimulus are deeply interconnected with sensory awareness and attention. Mindfulness meditation is a well-established tool for enhancing attention and awareness, but can this heightened awareness influence how we perceive and emotionally evaluate tactile stimuli? In this work, we investigate how mindfulness meditation influences sensory perception and emotional evaluation of gentle stroking stimuli delivered to the forearm using ultrasound-based mid-air haptics, providing tactile sensations without direct skin contact. Two studies were conducted to assess the effects of mindfulness on (1) perceived intensity, confidence, valence, and arousal in response to simple linear strokes and (2) pattern recognition and emotional evaluation of complex mid-air haptic patterns. Our findings suggest that mindfulness enhances perceived intensity, confidence, and valence ratings, but does not influence arousal ratings. Pattern recognition was challenging for the participants on the forearm leading to poor accuracy but mindfulness still enhances the recognition. These findings highlight the potential of mindfulness-enhanced haptic interactions in digital wellbeing and virtual environments. Future research should explore long-term mindfulness training, multi-sensory integration, and adaptive haptic systems to further optimize affective haptic experiences.

Take a deep breath; what do you smell? While sight and hearing dominate our perceptions, the sense of smell is often overlooked, even undervalued. However, the importance of our sense of smell goes beyond detecting odours. It shapes our emotions, memories, behaviour, and quality of life. Recent advances in olfactory interfaces have sparked discussions about the future of smell in human-computer interaction. While efforts mainly focus on the realization of new olfactory interfaces, here we collectively explore alternative worlds centred around the sense of smell. First, we conducted a design futuring workshop involving individuals with varying smell capabilities and expertise to envision and discuss smell-centred futures. Then, through iterative reflection, we arrived at three smell worlds presented as narratives and visuals. From this conceptual work, we offer new perspectives and generative possibilities for design research that prioritizes our noses.

Multi-point STM offers a great range of parameters (i.e., drawing frequency, number of points) to produce different tactile sensations. However, existing studies offer limited insight on the effects of these parameters, and ignore their effect on the physical stimuli delivered, limiting effective haptic design. We propose a two-stage model to predict response to multi-point STM. The first stage predicts physical stimulus properties with 7.8% error, while the second stage predicts mean and spread of perceived intensity with 8.0 % and 8.8% error. We report 3 studies conducted to derive this model: one to characterize physical stimuli, another one measuring user perceptual thresholds, and a third one measuring user’s perceptual response to multi-point STM. Besides, we characterize 4 effects that influence device performance, confirm if previous effects reported are due to physical or perceptual effects (or both) and derive recommendations for manufacturers, haptic designers and HCI researchers.

The sound produced when touching fabrics, like a blanket, often provides information regarding the fabric’s texture properties (e.g., its roughness). Fabric roughness is one of the most important aspects of assessing fabric tactile properties. Prior research has demonstrated that touch-related sounds can alter the perception of textures. However, understanding touch-related sound of digital fabric textures, and how they could convey affective responses remain a challenge. In this study, we mapped digital fabric textures using mid-air haptics stimuli and examined how auditory manipulation influences people’s roughness perception. Through qualitative interviews, participants detailed that while rubbing sounds smoothen fabric texture perception, pure tone sounds of 450Hz and 900Hz accent roughness perception. The rubbing sound of fabric evoked associations with soft-materials and led to more calming experiences. In addition, we discussed how haptic interaction can be extended to multisensory modes, revealing a new perspective of mapping multisensory experiences for digital fabrics.

Human perception of touch is a complex interplay of sensory inputs, extending beyond tactile stimuli to include visual and auditory cues. Yet, the potential of olfactory cues (e.g., scents) to enrich our understanding and experiences of digital touch remains largely unexplored. This paper explores the integration of scent with digital haptic feedback, specifically focusing on the perception of textiles using an ultrasonic mid-air haptic device. In an initial validation study, we identified scents associated with the tactile properties of roughness, smoothness, coldness, and warmth based on participant ratings (n = 31). In a subsequent immersive Virtual Reality (VR) experiment, we explored how these scents might affect perceptions of digital textiles presented using ultrasonic mid-air haptics (n = 32). Our results suggested that while scents did not affect the tactile properties of digital textiles, participants reported the sensations feeling more realistic and more arousing according to the scent delivered. This multimodal integration of scent with digital haptic stimuli can open up the creation of more immersive experiences and more in-depth understanding of digital material properties.

Commensality, the act of eating together, offers a rich multisensory and social experience that technology can enhance. Dining involves interactions with food, where smells, colors, sounds, and textures contribute to a multisensory experience. The table becomes a focal point for social interaction, with nonverbal cues and conversations being the indispensable ingredients of the commensal experience. The CoFI2024 workshop aims to stimulate discussion about how interactive technology can enrich dining experiences. The other aim is to build an interdisciplinary community related to commensality and human-food interaction, focusing on the role of multimodal interaction among commensal partners sharing food, being humans or artificial dining companions.

Human-artificial intelligence (AI) alignment ensures that AI systems align with human goals and behaviors. This paper introduces perceptual alignment as a critical aspect of this alignment, focusing on the concurrence between human judgments and AI evaluations across sensory modalities. We particularly explore how Multimodal Large Language Models (MLLMs), which process both visual and textual data, interpret the tactile qualities of textiles—a significant challenge in online shopping environments. Our research analyzes six vision-based MLLMs to see how they describe the tactile experience of textiles and compares these AI-generated descriptions with human assessments. Through semantic similarity measures and in-person evaluations, we investigate the extent of alignment between human perceptions and AI descriptions. Our findings indicate significant variability in the AI’s ability to interpret different textiles, highlighting both the potential and limitations of current AI models in achieving perceptual alignment. This work contributes to understanding the complexities of aligning AI capabilities with human touch sensory experiences.

There is an increasing interest in combining interactive technology with food, leading to a new research area called human-food interaction. While food experiences are increasingly benefiting from interactive technology, for example in the form of food tracking apps, 3D-printed food and projections on dining tables, a more systematic advancement of the field is hindered because, so far, there is no comprehensive articulation of the grand challenges the field is facing. To further and consolidate conversations around this topic, we invited 21 HFI experts to a 5-day seminar. The goal was to review our own and prior work to identify the grand challenges in human-food interaction. The result is an articulation of 10 grand challenges in human-food interaction across 4 categories (technology, users, design and ethics). By presenting these grand challenges, we aim to help researchers move the human-food interaction research field forward.

The rapid development of touchless systems has introduced many innovations in social interaction scenarios in recent years. People now can interact with touchless systems in social applications that are aimed to be used in everyday situations in the future. This accelerated development makes us ask, what will the next generation of touchless systems be like? How can we responsibly develop new touchless technologies in the future? To answer the first question, we brought together 20 experts to ideate, speculate, and evaluate possible touchless applications for social interactions. A total of 48 ideas were generated from two consecutive workshops. Then, to answer the second question, we critically analyzed those ideas through a thematic analysis using a responsible innovation (RI) framework, and identified key ethical considerations to guide developers, practitioners when designing future touchless systems. We argue that the social scenarios described, and the RI framework proposed in this paper are a useful starting point for responsibly designing the next generation of touchless systems.

Spatiotemporal modulation (STM) is used to render tactile patterns with ultrasound arrays. Previous research only explored the effects of single-point STM parameters, such as drawing speed (Vd). Here we explore the effects of multi-point STM on both perceptual (intensity) and emotional (valence/arousal) responses. This introduces a new control parameter for STM - the number of focal points (Nfp) – on top of conventional STM parameter (Vd). Our results from a study with 30 participants showed a negative effect of Nfp on perceived intensity and arousal, but no significant effects on valence. We also found the effects of Vd still aligned with prior results for single-point, even when different Nfp were used, suggesting that effects observed from single-point also apply to multi-point STM. We finally derive recommendations, such as using single-point STM to produce stimuli with higher intensity and/or arousal, or using multi-point STM for milder and more relaxing (less arousing) experiences.

As we constantly seek to improve and expand upon the capabilities of technology, we frequently wonder whether we use technology to its fullest extent. Studies indicate that increasing our awareness and mindfulness of our senses may lead to a journey of unexplored experiences. In this paper, we focus on the perception of mid-air haptics stimuli and whether it can be improved through mindfulness meditation. We have conducted an experiment with 22 participants given the task to recognize digits 0 to 9 drawn on their palms using a mid-air haptic device under two conditions - with and without prior mindfulness meditation. Results show that for frequencies targeting both Meissner (40 Hz) and Pacinian (200 Hz) receptors, meditation significantly improves performance of the participants, as well as increases their confidence. This suggests that including a short meditation step in haptic user interfaces could lead to improved system performance and user satisfaction.

The tactile experience of fabric is not only a sensory experience but also an affective one. Our choice of fabric products, like clothing, is often based on how they feel. Effectively communicating such experiences is crucial for designing tactile fabric experiences. However, there remains a lack of comprehensive understanding of the fabric tactile and affective experiences, preventing the development of tools to facilitate the communication of these experiences. In this paper, we examine the fabric experiences of 27 participants towards nine cotton samples. We combine qualitative and quantitative methods to create FabTouch, a novel tool to facilitate a dialogue in the design of fabric experiences. We found six phases of fabric touch experiences including fabric touch responses, sensory associations, and emotional responses. Initial feedback from designers suggested that FabTouch could enrich design processes both in practice and in education and can create inspiration for physical and digital design explorations.

Mid-air haptic stimulation can enrich user experience during human-computer interaction. However, the design space of such stimuli is large due to the number and range of stimulation parameters. It therefore remains difficult for designers to select a stimulus to induce an intended experience. We derive a mapping for mid-air experiences based on two user studies. In the first study, participants rated 36 stimuli varied across three parameters (frequency, pattern, and repetitions). These ratings allowed us to determine a set of five experientially distinct stimuli. In the second study, participants vocalized their experiences with those five stimuli. This allows us to generate a mapping of 17 sensations and 23 experiences related to the stimuli. Finally, we discuss how the mapping can inform designers and researchers working with mid-air haptic technologies.

Previous research has shown the influence of smell on emotions, memories, and body image. However, most of this work has taken place in laboratory settings and little is known about the influence of smell in real-world environments. In this article, we present novel insights gained from a field study investigating the emotional effect of smell on memories and body image. Taking inspiration from the cultural design probes approach, we designed QuintEssence, a probe package that includes three scents and materials to complete three tasks over a period of four weeks. Here, we describe the design of QuintEssence and the main findings based on the outcomes of the three tasks and a final individual interview. The findings show similar results between participants based on the scent. For example, with cinnamon, participants experienced feelings of warmth, coziness, happiness, and relaxation; they recalled blurred memories of past moments about themselves and reported a general feeling of being calm and peaceful towards their bodies. Our findings open up new design spaces for multisensory experiences and inspire future qualitative explorations beyond laboratory boundaries.

Driving is a task that is often affected by emotions. The effect of emotions on driving has been extensively studied. Anger is an emotion that dominates in such investigations. Despite the knowledge on strong links between scents and emotions, few studies have explored the effect of olfactory stimulation in a context of driving. Such an outcome provides HCI practitioners very little knowledge on how to design for emotions using olfactory stimulation in the car. We carried out three studies to select scents of different valence and arousal levels (i.e. rose, peppermint, and civet) and anger eliciting stimuli (i.e. affective pictures and on-road events). We used this knowledge to conduct the fourth user study investigating how the selected scents change the emotional state, well-being, and driving behaviour of drivers in an induced angry state. Our findings enable better decisions on what scents to choose when designing interactions for angry drivers.

Personalization of eating such that everyone consumes only what they need allows improving our management of food waste. In this paper, we explore the use of food 3D printing to create perceptual illusions for controlling the level of perceived satiety given a defined amount of calories. We present FoodFab, a system that allows users to control their food intake through modifying a food's internal structure via two 3D printing parameters: infill pattern and infill density. In two experiments with a total of 30 participants, we studied the effect of these parameters on users' chewing time that is known to affect people's feeling of satiety. Our results show that we can indeed modify the chewing time by varying infill pattern and density, and thus control perceived satiety. Based on the results, we propose two computational models and integrate them into a user interface that simplifies the creation of personalized food structures.

Human-Computer Integration (HInt) is an emerging paradigm in which computational and human systems are closely interwoven. Integrating computers with the human body is not new. however, we believe that with rapid technological advancements, increasing real-world deployments, and growing ethical and societal implications, it is critical to identify an agenda for future research. We present a set of challenges for HInt research, formulated over the course of a five-day workshop consisting of 29 experts who have designed, deployed and studied HInt systems. This agenda aims to guide researchers in a structured way towards a more coordinated and conscientious future of human-computer integration.

The Sense of Agency (SoA) is crucial in interaction with technology, it refers to the feeling of 'I did that' as opposed to 'the system did that' supporting a feeling of being in control. Research in human-computer interaction has recently studied agency in visual, auditory and haptic interfaces, however the role of smell on agency remains unknown. Our sense of smell is quite powerful to elicit emotions, memories and awareness of the environment, which has been exploited to enhance user experiences (e.g., in VR and driving scenarios). In light of increased interest in designing multimodal interfaces including smell and its close link with emotions, we investigated, for the first time, the effect of smell-induced emotions on the SoA. We conducted a study using the Intentional Binding (IB) paradigm used to measure SoA while participants were exposed to three scents with different valence (pleasant, unpleasant, neutral). Our results show that participants? SoA increased with a pleasant scent compared to neutral and unpleasant scents. We discuss how our results can inform the design of multimodal and future olfactory interfaces.

Here, we present the outcome of the 4th workshop on Multisensory Approaches to Human-Food Interaction (MHFI), developed in collaboration with ICMI 2020 in Utrecht, The Netherlands. Capitalizing on the increasing interest on multisensory aspects of human-food interaction and the unique contribution that our community offers, we developed a space to discuss ideas ranging from mechanisms of multisensory food perception, through multisensory technologies, to new applications of systems in the context of MHFI. All in all, the workshop involved 11 contributions, which will hopefully further help shape the basis of a field of inquiry that grows as we see progress in our understanding of the senses and the development of new technologies in the context of food.

Eating is one of the most multisensory experiences in everyday life. All of our five senses (i.e. taste, smell, vision, hearing and touch) are involved, even if we are not aware of it. However, while multisensory integration has been well studied in psychology, there is not a single platform for testing systematically the effects of different stimuli. This lack of platform results in unresolved design challenges for the design of taste-based immersive experiences. Here, we present LeviSense: the first system designed for multisensory integration in gustatory experiences based on levitated food. Our system enables the systematic exploration of different sensory effects on eating experiences. It also opens up new opportunities for other professionals (e.g., molecular gastronomy chefs) looking for innovative taste-delivery platforms. We describe the design process behind LeviSense and conduct two experiments to test a subset of the crossmodal combinations (i.e., taste and vision, taste and smell). Our results show how different lighting and smell conditions affect the perceived taste intensity, pleasantness, and satisfaction. We discuss how LeviSense creates a new technical, creative, and expressive possibilities in a series of emerging design spaces within Human-Food Interaction.

The human sense of smell is powerful. However, the way we use smell as an interaction modality in human–computer interaction (HCI) is limited. We lack a common reference point to guide designers’ choices when using smell. Here, we map out an olfactory design space to provide designers with such guidance. We identified four key design features: (i) chemical, (ii) emotional, (iii) spatial, and (iv) temporal. Each feature defines a building block for smell-based interaction design and is grounded in a review of the relevant scientific literature. We then demonstrate the design opportunities in three application cases. Each application (i.e., one desktop, two virtual reality implementations) highlights the design choices alongside the implementation and evaluation possibilities in using smell. We conclude by discussing how identifying those design features facilitates a healthy growth of this research domain and contributes to an intermediate-level knowledge space. Finally, we discuss further challenges the HCI community needs to tackle.

Mid-air tactile stimulation using ultrasonics has been used in a variety of human computer interfaces in the form of prototypes as well as products. When generating these tactile patterns with mid-air tactile ultrasonic displays, the common approach has been to sample the patterns using the hardware update rate capabilities to their full extent. In the current study we show that the hardware update rate can impact perception, but unexpectedly we find that higher update rates do not improve pattern perception. In a first user study, we highlight the effect of update rate on the perceived strength of a pattern, especially for patterns rendered at slow rate of less than 10 Hz. In a second user study, we identify the evolution of the optimal update rate according to variations in pattern size. Our main results show that update rate should be designated as additional parameter for tactile patterns. We also discuss how the relationships we defined in the current study can be implemented into designer tools so that designers remain oblivious to this additional complexity.

There is increasing interest in multisensory experiences in HCI. However, little research considers how sensory modalities interact with each other and how this may impact interactive experiences. We investigate how children associate emotions with scents and 3D shapes. 14 participants (10-17yrs) completed crossmodal association tasks to attribute emotional characteristics to variants of the "Bouba/Kiki" stimuli, presented as 3D tangible models, in conjunction with lemon and vanilla scents. Our findings support pre-existing mappings between shapes and scents, and confirm the associations between the combination of angular shapes ("Kiki") and lemon scent with arousing emotion, and of round shapes ("Bouba") and vanilla scent with calming emotion. This extends prior work on crossmodal correspondences in terms of stimuli (3D as opposed to 2D shapes), sample (children), and conveyed content (emotions). We outline how these findings can contribute to designing more inclusive interactive multisensory technologies.

Major virtual reality (VR) companies are trying to enhance the sense of immersion in virtual environments by implementing haptic feedback in their systems (e.g., Oculus Touch). It is known that tactile stimulation adds realism to a virtual environment. In addition, when users are not limited by wearing any attachments (e.g., gloves), it is even possible to create more immersive experiences. Mid-air haptic technology provides contactless haptic feedback and offers the potential for creating such immersive VR experiences. However, one of the limitations of mid-air haptics resides in the need for freehand tracking systems (e.g., Leap Motion) to deliver tactile feedback to the user's hand. These tracking systems are not accurate, limiting designers capability of delivering spatially precise tactile stimulation. Here, we investigated an alternative way to convey incongruent visual-tactile stimulation that can be used to create the illusion of a congruent visual-tactile experience, while participants experience the phenomenon of the rubber hand illusion in VR.

Overreliance in technology is safety-critical and it is assumed that this could have been a main cause of severe accidents with automated vehicles. To ease the complex task of permanently monitoring vehicle behavior in the driving environment, researchers have proposed to implement reliability/uncertainty displays. Such displays allow to estimate whether or not an upcoming intervention is likely. However, presenting uncertainty just adds more visual workload on drivers, who might also be engaged in secondary tasks. We suggest to use olfactory displays as a potential solution to communicate system uncertainty and conducted a user study (N=25) in a high-fidelity driving simulator. Results of the experiment (conditions: no reliability display, purely visual reliability display, and visual-olfactory reliability display) comping both objective (task performance) and subjective (technology acceptance model, trust scales, semi-structured interviews) measures suggest that olfactory notifications could become a valuable extension for calibrating trust in automated vehicles.

The excitement around computing technology in all aspects of life requires that we tackle fundamental issues of healthcare, leisure, labor, education, and food to create the society we want. The aim of this satellite event was to bring together a variety of different stakeholders, ranging from local food producers, chefs, designers, engineers, data scientists, and sensory scientists, to discuss the interwoven future of computing technology and food. This event was co-located with the AVI 2018 conference and supported by the ACM Future of Computing Academy (ACM-FCA). The event followed a co-creation approach that encourages conjoined creative and critical thinking that feeds into the formulation of a manifesto on the future of computing and food. We hope this will inspire future discussions on the transformative role of computing technology on food.

The Sense of Agency (SoA) refers to our capability to control our own actions and influence the world around us. Recent research in HCI has been investigating SoA to provide users an instinctive sense of "I did that" as opposed to "the system did that". However, current agency measurements are limited. The Intentional Binding (IB) paradigm provides an implicit measure of the SoA, however, it is constrained by requiring high visual attention to a "Libet clock" on-screen. In this paper, we extended the timing stimuli through auditory and tactile cues. Our results demonstrate that audio timing through voice commands and haptic timing through tactile cues on the hand, are an effective alternative measure of the SoA using the IB paradigm. They both address current limitations of the traditional method such as visual attention overload and lack of engagement. We discuss how our results can be applied to measure SoA in tasks involving different interactive scenarios such as in Mixed/Virtual Reality.

Cars provide drivers with task-related information (e.g. "Fill gas") mainly using visual and auditory stimuli. However, those stimuli may distract or overwhelm the driver, causing unnecessary stress. Here, we propose olfactory stimulation as a novel feedback modality to support the perception of visual notifications, reducing the visual demand of the driver. Based on previous research, we explore the application of the scents of lavender, peppermint, and lemon to convey three driving-relevant messages (i.e. "Slow down", "Short inter-vehicle distance", "Lane departure"). Our paper is the first to demonstrate the application of olfactory conditioning in the context of driving and to explore how multiple olfactory notifications change the driving behaviour. Our findings demonstrate that olfactory notifications are perceived as less distracting, more comfortable, and more helpful than visual notifications. Drivers also make less driving mistakes when exposed to olfactory notifications. We discuss how these findings inform the design of future in-car user interfaces.

Smell is a powerful tool for conveying and recalling information without requiring visual attention. Previous work identified, however, some challenges caused by user's unfamiliarity with this modality and complexity in the scent delivery. We are now able to overcome these challenges, introducing a training approach to familiarise scent-meaning associations (urgency of a message, and sender identity) and using a controllable device for the scent-delivery. Here we re-validate the effectiveness of smell as notification modality and present findings on the performance of smell in conveying information. In a user study composed of two sessions, we compared the effectiveness of visual, olfactory, and combined visual-olfactory notifications in a messaging application. We demonstrated that olfactory notifications improve users' confidence and performance in identifying the urgency level of a message, with the same reaction time and disruption levels as for visual notifications. We discuss the design implications and opportunities for future work in the domain of multimodal interactions.

This is the introduction paper to the third version of the workshop on 'Multisensory Approaches to Human-Food Interaction' organized at the 20th ACM International Conference on Multimodal Interaction in Boulder, Colorado, on October 16th, 2018. This workshop is a space where the fast growing research on Multisensory Human-Food Interaction is presented. Here we summarize the workshop's key objectives and contributions.

While our understanding of the sensory modalities for human-computer interaction (HCI) is advancing, there is still a huge gap in our understanding of 'how' to best integrate different modalities into the interaction with technology. While we have established a rich ecosystem for visual and auditory design, our sense of touch, taste, and smell are not yet supported with dedicated software tools, interaction techniques, and design tools. Hence, we are missing out on the opportunity to exploit the power of those sensory modalities, their strong link to emotions and memory, and their ability to facilitate recall and recognition in information processing and decision making. In this keynote, Dr Obrist presents an overview of scientific and technological developments in multisensory research and design with an emphasis on an experience-centered design approach that bridges and integrates knowledge on human sensory perception and advances in computing technology.

The sense of smell is well known to provide very vivid experiences and to mediate a strong activation of crossmodal semantic representations. Despite a growing number of olfactory HCI prototypes, there have been only a few attempts to study the sense of smell as an interaction modality. Here, we focus on the exploration of olfaction for in-car interaction design by establishing a mapping between three different driving-related messages ("Slow down", "Fill gas", "Passing by a point of interest") and four scents (lemon, lavender, peppermint, rose). The results of our first study demonstrate strong associations between, for instance, the "Slow down" message and the scent of lemon, the "Fill gas" message and the scent of peppermint, the "Passing by a point of interest" message and the scent of rose. These findings have been confirmed in our second study, where participants expressed their mapping preferences while performing a simulated driving task.

Apparent tactile motion has been shown to occur across many contiguous part of the body, such as fingers, forearms, and back. A recent study demonstrated the possibility of eliciting the illusion of movement from one hand to the other when interconnected by a tablet. In this paper we explore intermanual apparent tactile motion without any object between them. In a series of psychophysical experiments we determine the control space for generating smooth and consistent motion, using two vibrating handles which we refer to as the Hand-to-Hand vibrotactile device. In a first experiment we investigated the occurrence of the phenomenon (i.e., movement illusion) and the generation of a perceptive model. In a second experiment, based on those results, we investigated the effect of hand postures on the illusion. Finally, in a third experiment we explored two visuo-tactile matching tasks in a multimodal VR setting. Our results can be applied in VR applications with intermanual tactile interactions.

We present an introduction paper to the second version of the workshop on ‘Multisensory Approaches to Human-Food Interaction’ to be held at the 19th ACM International Conference on Multimodal Interaction, which will take place on November 13th, 2017, in Glasgow, Scotland. Here, we describe the workshop’s objectives, the key contributions of the different position papers, and the relevance of their respective topic(s). Both multisensory research and new technologies are evolving fast which has opened up a number of possibilities for designing new ways of interacting with foods and drinks. This workshop highlights the rapidly growing interest in the field of Multisensory Human-Food Interaction, which can, for example, be observed in the variety of novel research and technology developments in the area.

While there is an increased appreciation for integrating haptic feedback with audio-visual content, there is still a lack of understanding of how to quantify the added value of touch for a user's experience (UX) of multimedia content. Here we focus on three main concepts to measure this added value: UX, emotions, and expectations. We present a case study measuring the added value of haptic feedback for a standardized set of audio-visual content (i.e., short video clips), comparing two haptic stimulation modalities (i.e., mid-air vs. vibro-tactile stimuli). Our findings demonstrate that UX of haptically-enhanced audio-visual content is perceived as a more pleasant, unpredictable, and creative experience. The users' overall liking increases together with a positive change of the users' expectations, independently from the haptic stimulation modality. We discuss how our approach provides the foundation for future work on developing a measurement model to predict the added value of haptic feedback for users' experiences within and beyond the multimedia context.

When designing olfactory interfaces, HCI researchers and practitioners have to carefully consider a number of issues related to the scent delivery, detection, and lingering. These are just a few of the problems to deal with. We present OSpace - an approach for designing, building, and exploring an olfactory interaction space. Our paper is the first to explore in detail not only the scent-delivery parameters but also the air extraction issues. We conducted a user study to demonstrate how the scent detection/lingering times can be acquired under different air extraction conditions, and how the impact of scent type, dilution, and intensity can be investigated. Results show that with our setup, the scents can be perceived by the user within ten seconds and it takes less than nine seconds for the scents to disappear, both when the extraction is on and off. We discuss the practical application of these results for HCI.

Designing multisensory experiences is fascinating academics and practitioners alike. Recent advances in interactive systems and sensory devices also increasingly captures the attention of artists, museum curators, and creative industries, who aim to explore the opportunities of new and emerging technologies to create novel and unexpected experiences. The main objective of this tutorial is to increase the ISS communities' awareness of design spaces beyond audio-visual and tactile interactions. In particular, we will provide an overview on olfactory design spaces and opportunities building on our expertise and experience working on multisensory design projects including the Tate Sensorium, Famous Deaths, Feelies Multisensory VR experiences. We provide an overview on research efforts across disciplines, arts, and industries.

We present two realizations of TastyFloats, a novel system that uses acoustic levitation to deliver food morsels to the users' tongue. To explore TastyFloats' associated design framework, we first address the technical challenges to successfully levitate and deliver different types of foods on the tongue. We then conduct a user study, assessing the effect of acoustic levitation on users' taste perception, comparing three basic taste stimuli (i.e., sweet, bitter and umami) and three volume sizes of droplets (5μL, 10μL and 20μL). Our results show that users perceive sweet and umami easily, even in minimal quantities, whereas bitter is the least detectable taste, despite its typical association with an unpleasant taste experience. Our results are a first step towards the creation of new culinary experiences and innovative gustatory interfaces.

Virtual Reality allow users to explore virtual environments naturally, by moving their head and body. However, the size of the environments they can explore is limited by real world constraints, such as the tracking technology or the physical space available. Existing techniques removing these limitations often break the metaphor of natural navigation in VR (e.g. steering techniques), involve control commands (e.g., teleporting) or hinder precise navigation (e.g., scaling user's displacements). This paper proposes NaviFields, which quantify the requirements for precise navigation of each point of the environment, allowing natural navigation within relevant areas, while scaling users' displacements when travelling across non-relevant spaces. This expands the size of the navigable space, retains the natural navigation metaphor and still allows for areas with precise control of the virtual head. We present a formal description of our NaviFields technique, which we compared against two alternative solutions (i.e., homogeneous scaling and natural navigation). Our results demonstrate our ability to cover larger spaces, introduce minimal disruption when travelling across bigger distances and improve very significantly the precise control of the viewpoint inside relevant areas.

The use of the senses of vision and audition as interactive means has dominated the field of Human-Computer Interaction (HCI) for decades, even though nature has provided us with many more senses for perceiving and interacting with the world around us. That said, it has become attractive for HCI researchers and designers to harness touch, taste, and smell in interactive tasks and experience design. In this paper, we present research and design insights gained throughout an interdisciplinary collaboration on a six-week multisensory display – Tate Sensorium – exhibited at the Tate Britain art gallery in London, UK. This is a unique and first time case study on how to design art experiences whilst considering all the senses (i.e., vision, sound, touch, smell, and taste), in particular touch, which we exploited by capitalizing on a novel haptic technology, namely, mid-air haptics. We first describe the overall set up of Tate Sensorium and then move on to describing in detail the design process of the mid-air haptic feedback and its integration with sound for the Full Stop painting by John Latham (1961). This was the first time that mid-air haptic technology was used in a museum context over a prolonged period of time and integrated with sound to enhance the experience of visual art. As part of an interdisciplinary team of curators, sensory designers, sound artists, we selected a total of three variations of the mid-air haptic experience (i.e., haptic patterns), which were alternated at dedicated times throughout the six-week exhibition. We collected questionnaire-based feedback from 2500 visitors and conducted 50 interviews to gain quantitative and qualitative insights on visitors’ experiences and emotional reactions. Whilst the questionnaire results are generally very positive with only a small variation of the visitors’ arousal ratings across the three tactile experiences designed for the Full Stop painting, the interview data shed light on the differences in the visitors’ subjective experiences. Our findings suggest multisensory designers and art curators can ensure a balance between surprising experiences versus the possibility of free exploration for visitors. In addition, participants expressed that experiencing art with the combination of mid-air haptic and sound was immersive and provided an up-lifting experience of touching without touch. We are convinced that the insights gained from this large-scale and real-world field exploration of multisensory experience design exploiting a new and emerging technology provide a solid starting point for the HCI community, creative industries, and art curators to think beyond conventional art experiences. Specifically, our work demonstrates how novel mid-air technology can make art more emotionally engaging and stimulating, especially abstract art that is often open to interpretation.

In the field of Human Computer Interaction (HCI), vision and audition have been the dominating modalities for interacting with users. This is despite the fact that humans are equipped with five basic senses. Because of this, there is a limited number of tools that harness the olfactory system as a communication channel. Recently, several promising scent-delivery devices have been developed, however, there is a lack of guidance on how to use them in a meaningful way for different interactive tasks. In this paper, we propose a three-dimensional framework to compare different scent-delivery devices based on the distance, volume, and speed of the scent-delivery. We discuss how this initial exploration can guide the design of in-car olfactory interfaces beyond previous work on drivers' physical and emotional state.

Sensory Substitution Devices (SSDs) convert visual information into another sensory channel (e.g. sound) to improve the everyday functioning of blind and visually impaired persons (BVIP). However, the range of possible functions and options for translating vision into sound is largely open-ended. To provide constraints on the design of this technology, we interviewed ten BVIPs who were briefly trained in the use of three novel devices that, collectively, showcase a large range of design permutations. The SSDs include the 'Depth-vOICe,' 'Synaestheatre' and 'Creole' that offer high spatial, temporal, and colour resolutions respectively via a variety of sound outputs (electronic tones, instruments, vocals). The participants identified a range of practical concerns in relation to the devices (e.g. curb detection, recognition, mental effort) but also highlighted experiential aspects. This included both curiosity about the visual world (e.g. understanding shades of colour, the shape of cars, seeing the night sky) and the desire for the substituting sound to be responsive to movement of the device and aesthetically engaging.

This workshop aims at discussing the rich possibilities that the body offers to experience the external world and the prospects that arise for interaction designers when these often-neglected abilities are taken into account. In particular, the workshop will focus on the rediscovery of the five senses, either alone or in a multimodal combination, and of the perceptual-motor abilities of our body. The one-day workshop is divided in two steps: first, participants will have the possibility to briefly present and discuss with the other attendees their research. Workshop candidates are requested to send a position paper, including a short biography and detailing their research interests and background. Second, participants will have the possibilities to explore and rediscover their sensorimotor abilities through several exercises and games abilities using a critical design approach. Interdisciplinary groups will be challenged to design and develop new interaction experience concepts using our natural 'tools' as prototyping tools.

This paper presents insights on driver's User Experience (UX) in terms of systematically investigating predicted experience, actual experience, and recalled experience. By conducting a three-week field study with car commuters in two countries, we studied how frustration and anger differentiate in prediction, actual experience, and recall. Our results show that commuters accurately predict their upcoming anger or frustration in a traffic congestion, however, lower their experienced frustration when being recalled. Moreover, unexpected traffic congestions (in contrast to expected ones) are prone to higher levels of anger. We further found that time of day is related to the prediction of anger, and mood is related to the prediction of frustration. With our study we provide a holistic view on commuters' everyday emotions and experiences -- not only when being on the road, but also before and after the trip.

Touch is a powerful vehicle for communication between humans. The way we touch (how) embraces and mediates certain emotions such as anger, joy, fear, or love. While this phenomenon is well explored for human interaction, HCI research is only starting to uncover the fine granularity of sensory stimulation and responses in relation to certain emotions. Within this paper we present the findings from a study exploring the communication of emotions through a haptic system that uses tactile stimulation in mid-air. Here, haptic descriptions for specific emotions (e.g., happy, sad, excited, afraid) were created by one group of users to then be reviewed and validated by two other groups of users. We demonstrate the non-arbitrary mapping between emotions and haptic descriptions across three groups. This points to the huge potential for mediating emotions through mid-air haptics. We discuss specific design implications based on the spatial, directional, and haptic parameters of the created haptic descriptions and illustrate their design potential for HCI based on two design ideas.

We present rich descriptions of taste experience through an analysis of the diachronic and synchronic experiences of each of the five basic taste qualities: sweet, sour, salt, bitter, and umami. Our findings, based on a combination of user experience evaluation techniques highlight three main themes: temporality, affective reactions, and embodiment. We present the taste characteristics as a framework for design and discuss each taste in order to elucidate the design qualities of individual taste experiences. These findings add a semantic understanding of taste experiences, their temporality enhanced through descriptions of the affective reactions and embodiment that the five basic tastes elicit. These findings are discussed on the basis of established psychological and behavioral phenomena, highlighting the potential for taste-enhanced design.

Technologies for capturing and generating smell are emerging, and our ability to engineer such technologies and use them in HCI is rapidly developing. Our understanding of how these technologies match the experiences with smell that people have or want to have is surprisingly limited. We therefore investigated the experience of smell and the emotions that accompany it. We collected stories from 439 participants who described personally memorable smell experiences in an online questionnaire. Based on the stories we developed 10 categories of smell experience. We explored the implications of the categories for smell-enhanced technology design by (a) probing participants to envision technologies that match their smell story and (b) having HCI researchers brainstorm technologies using the categories as design stimuli. We discuss how our findings can benefit research on personal memories, momentary and first time experiences, and wellbeing.

A common problem with designing and developing applications with tactile interfaces is the lack of a vocabulary that allows one to describe or communicate about haptics. Here we present the findings from a study exploring participants' verbalizations of their tactile experiences across two modulated tactile stimuli (16Hz and 250Hz) related to two important mechanoreceptors in the human hand. The study, with 14 participants, applied the explicitation interview technique to capture detailed descriptions of the diachronic and synchronic structure of tactile experiences. We propose 14 categories for a human-experiential vocabulary based on the categorization of the findings and tie them back to neurophysiological and psychophysical data on the human hand. We finally discuss design opportunities created through this experiential understanding in relation to the two mechanoreceptors.

This workshop aims to identify and improve current strategies and practices for transferring knowledge from qualitative user studies to become inspiration for an experience-centered design process. The message (e.g., qualitative user study results) can be skewed when passed on from one phase to another in the design and development process. Moreover, the way insights are communicated is often not inspiring enough. We intend to get practitioners and academics engaged in a discussion about how to best transfer research findings of especially qualitative user studies and prevent loss of valuable information. Best practices, failure and success stories are especially welcomed as contributions. At the end of this workshop we will have identified strengths of a few methods, tools, and techniques selected from examples of best practices. Also, we will have summarized limitations and challenges of knowledge transfer and communication practices within experience-centered design.

The school context provides a fruitful and at the same challenging environment to get children engaged in creative development activities. Within this paper we highlight the opportunities and challenges when involving children in the design and development process of interactive technologies with a special focus on this educational context. A summary of the contributions to this IDC 2011 workshop is provided pointing out the main highlights from the position papers. We conclude by discussing future research challenges and opportunities when working with children in the school context.

Different methods, techniques and tools exist for supporting design activities in a user-centered design process. Due to the increasing relevance of experience-centered design, the need to advance existing design methods and tools becomes evident. Within this paper we present "experience characters" as a design tool for communicating a richer understanding of mobile phone experiences towards designers. We conducted a qualitative text analysis study of written experience reports from an online forum in the tradition of grounded theory. Major experience types were grouped into categories and further transformed into five fictive characters that are easy to remember and likely to invoke empathy in design. Thus, designers' communication on and understanding of mobile phone experiences are supported. We describe in detail the steps taken for analysing experience reports and developing the experience characters, including relevant properties for informing and supporting an experience-centered design process.

The task of this article is to focus on aspects of design from a theoretical perspective that works out a general framework of techno-social design that is grounded in the notion of the participatory, cooperative, sustainable information society. The approach is normative in that it is based on the judgment that not just any information society is needed, but one that has specific qualities that this article tries to identify. Methodologically, the article works out a conceptual framework that synthesizes general social theory and human–computer interaction. Design is seen as a social process that shapes society and techno-social systems. First, a model of society that is based on the dialectical interaction of economic, political, and cultural subsystems is introduced. Then, the notion of the the participatory, cooperative, sustainable information society is introduced. This theoretical definition is then used for classifying design principles of techno-social information systems from a social theory perspective.

The MobiMundi Workshop aims to provide a forum for researchers and developers from different backgrounds to gather together to explore and discuss the impact of existing and emerging mobile information and communications technologies and services on society. The theme for this workshop will examine the real and potential impacts of user-generated mobile content on individuals and society as a whole. The individual and societal impacts discussed will primarily focus on user centered and social perspectives, but will also explore how user behavior is changing and which problems are emerging, bearing in mind that individual actions are creating privacy and security issues which could lead towards a new quality of a hyper-surveillance society by providing the framework for the participatory panopticon. Moreover, mobile human-computer interaction and design related issues will be explored and considered in order to support users requirement to share and co-create mobile user-generated audiovisual content.

Interactive TV (iTV) will only succeed if user-centered solutions are provided. Despite an increasing focus on the home environment within the human–computer interaction field, there is still a lack of studies of the context home and empirical data on users' domestic activities (Blythe, Overbeeke, Monk, & Wright, 2004 Blythe, M. A., Overbeeke, K., Monk, A. F. and Wright, P. C. 2004. Funology. From usability to enjoyment, 277–281. Dordrecht, the Netherlands: Kluwer Academic. [Google Scholar]; Mateas, Salvador, Scholtz, & Sorensen, Mateas, M., Salvador, T., Scholtz, J. and Sorensen, D. . Engineering ethnography in the home. Proceedings of the Human Factors in Computing Systems Conference (CHI'96). pp.283–284. [Google Scholar] O'Brien, Rodden, Rouncefield, & Hughes, 1999 O'Brien, J., Rodden, T., Rouncefield, M. and Hughes, J. 1999. At home with the technology: An ethnographic study of a set-top-box trial. ACM Transactions on Computer-Human Interaction, 6(3): 282–308. [Crossref] , [Google Scholar]). In this article, the results of an ethnographic study conducted during the summer 2005 as part of a field trial on iTV in the country of Salzburg, Austria, are presented. The research aim was to gain a deeper understanding of selected household types and household members, their daily habits and social background, their TV viewing behavior and experiences, and expectations on iTV. Within the research, patterns of what people do, how existing technologies facilitate their daily activities, what meanings people connect with the TV, and how iTV services can support people at home could be identified. The findings of this study and the lessons learned from the usability study during the first field trial (Bernhaupt, Ploderer, & Tscheligi, 2005 Bernhaupt, R., Ploderer, B. and Tscheligi, M. . Relevance of prior experience in MHP-based ITV services. Human-computer interaction–INTERACT 2005, IFIP TC13 International Conference. Edited by: Costabile, M. F. and Paterno, F. pp.149–152. Rome, Italy: Springer. [Google Scholar]) guided the design and development of the iTV services for the second field trial in spring 2006.