Eating is one of the most multisensory experiences in everyday life. All of our five senses (i.e. taste, smell, vision, hearing and touch) are involved, even if we are not aware of it. However, while multisensory integration has been well studied in psychology, there is not a single platform for testing systematically the effects of different stimuli. This lack of platform results in unresolved design challenges for the design of taste-based immersive experiences. Here, we present LeviSense: the first system designed for multisensory integration in gustatory experiences based on levitated food. Our system enables the systematic exploration of different sensory effects on eating experiences. It also opens up new opportunities for other professionals (e.g., molecular gastronomy chefs) looking for innovative taste-delivery platforms. We describe the design process behind LeviSense and conduct two experiments to test a subset of the crossmodal combinations (i.e., taste and vision, taste and smell). Our results show how different lighting and smell conditions affect the perceived taste intensity, pleasantness, and satisfaction. We discuss how LeviSense creates a new technical, creative, and expressive possibilities in a series of emerging design spaces within Human-Food Interaction.

The sense of smell is well known to provide very vivid experiences and to mediate a strong activation of crossmodal semantic representations. Despite a growing number of olfactory HCI prototypes, there have been only a few attempts to study the sense of smell as an interaction modality. Here, we focus on the exploration of olfaction for in-car interaction design by establishing a mapping between three different driving-related messages ("Slow down", "Fill gas", "Passing by a point of interest") and four scents (lemon, lavender, peppermint, rose). The results of our first study demonstrate strong associations between, for instance, the "Slow down" message and the scent of lemon, the "Fill gas" message and the scent of peppermint, the "Passing by a point of interest" message and the scent of rose. These findings have been confirmed in our second study, where participants expressed their mapping preferences while performing a simulated driving task.

Touchless interfaces allow users to view, control and manipulate digital content without physically touching an interface. They are being explored in a wide range of application scenarios from medical surgery to car dashboard controllers. One aspect of touchless interaction that has not been explored to date is the Sense of Agency (SoA). The SoA refers to the subjective experience of voluntary control over actions in the external world. In this paper, we investigated the SoA in touchless systems using the intentional binding paradigm. We first compare touchless systems with physical interactions and then augmented different types of haptic feedback to explore how different outcome modalities influence intentional binding. From our experiments, we demonstrated that an intentional binding effect is observed in both physical and touchless interactions with no statistical difference. Additionally, we found that haptic and auditory feedback help to increase SoA compared with visual feedback in touchless interfaces. We discuss these findings and identify design opportunities that take agency into consideration.

We present two realizations of TastyFloats, a novel system that uses acoustic levitation to deliver food morsels to the users' tongue. To explore TastyFloats' associated design framework, we first address the technical challenges to successfully levitate and deliver different types of foods on the tongue. We then conduct a user study, assessing the effect of acoustic levitation on users' taste perception, comparing three basic taste stimuli (i.e., sweet, bitter and umami) and three volume sizes of droplets (5μL, 10μL and 20μL). Our results show that users perceive sweet and umami easily, even in minimal quantities, whereas bitter is the least detectable taste, despite its typical association with an unpleasant taste experience. Our results are a first step towards the creation of new culinary experiences and innovative gustatory interfaces.

Usability has a distinct subjective component, yet surprisingly little is known about its neural basis and relation to the neuroanatomy of aesthetics. To begin closing this gap, we conducted two functional magnetic resonance imaging studies in which participants were shown static webpages (in the first study) and videos of interaction with webpages (in the second study). The webpages were controlled so as to exhibit high and low levels of perceived usability and perceived aesthetics. Our results show unique links between perceived usability and brain areas involved in functions such as emotional processing (left fusiform gyrus, superior frontal gyrus), anticipation of physical interaction (precentral gyrus), task intention (anterior cingulate cortex), and linguistic processing (medial and bilateral superior frontal gyri). We use these findings to discuss the brain correlates of perceived usability and the use of fMRI for usability evaluation and for generating new user experiences.

The use of the senses of vision and audition as interactive means has dominated the field of Human-Computer Interaction (HCI) for decades, even though nature has provided us with many more senses for perceiving and interacting with the world around us. That said, it has become attractive for HCI researchers and designers to harness touch, taste, and smell in interactive tasks and experience design. In this paper, we present research and design insights gained throughout an interdisciplinary collaboration on a six-week multisensory display – Tate Sensorium – exhibited at the Tate Britain art gallery in London, UK. This is a unique and first time case study on how to design art experiences whilst considering all the senses (i.e., vision, sound, touch, smell, and taste), in particular touch, which we exploited by capitalizing on a novel haptic technology, namely, mid-air haptics. We first describe the overall set up of Tate Sensorium and then move on to describing in detail the design process of the mid-air haptic feedback and its integration with sound for the Full Stop painting by John Latham (1961). This was the first time that mid-air haptic technology was used in a museum context over a prolonged period of time and integrated with sound to enhance the experience of visual art. As part of an interdisciplinary team of curators, sensory designers, sound artists, we selected a total of three variations of the mid-air haptic experience (i.e., haptic patterns), which were alternated at dedicated times throughout the six-week exhibition. We collected questionnaire-based feedback from 2500 visitors and conducted 50 interviews to gain quantitative and qualitative insights on visitors’ experiences and emotional reactions. Whilst the questionnaire results are generally very positive with only a small variation of the visitors’ arousal ratings across the three tactile experiences designed for the Full Stop painting, the interview data shed light on the differences in the visitors’ subjective experiences. Our findings suggest multisensory designers and art curators can ensure a balance between surprising experiences versus the possibility of free exploration for visitors. In addition, participants expressed that experiencing art with the combination of mid-air haptic and sound was immersive and provided an up-lifting experience of touching without touch. We are convinced that the insights gained from this large-scale and real-world field exploration of multisensory experience design exploiting a new and emerging technology provide a solid starting point for the HCI community, creative industries, and art curators to think beyond conventional art experiences. Specifically, our work demonstrates how novel mid-air technology can make art more emotionally engaging and stimulating, especially abstract art that is often open to interpretation.

In the field of Human Computer Interaction (HCI), vision and audition have been the dominating modalities for interacting with users. This is despite the fact that humans are equipped with five basic senses. Because of this, there is a limited number of tools that harness the olfactory system as a communication channel. Recently, several promising scent-delivery devices have been developed, however, there is a lack of guidance on how to use them in a meaningful way for different interactive tasks. In this paper, we propose a three-dimensional framework to compare different scent-delivery devices based on the distance, volume, and speed of the scent-delivery. We discuss how this initial exploration can guide the design of in-car olfactory interfaces beyond previous work on drivers' physical and emotional state.

Error Related Negativity is triggered when a user either makes a mistake or the application behaves differently from their expectation. It can also appear while observing another user making a mistake. This paper investigates ERN in collaborative settings where observing another user (the executer) perform a task is typical and then explores its applicability to HCI. We first show that ERN can be detected on signals captured by commodity EEG headsets like an Emotiv headset when observing another person perform a typical multiple-choice reaction time task. We then investigate the anticipation effects by detecting ERN in the time interval when an executer is reaching towards an answer. We show that we can detect this signal with both a clinical EEG device and with an Emotiv headset. Our results show that online single trial detection is possible using both headsets during tasks that are typical of collaborative interactive applications. However there is a trade-off between the detection speed and the quality/prices of the headsets. Based on the results, we discuss and present several HCI scenarios for use of ERN in observing tasks and collaborative settings.

This paper examines the ability to detect a characteristic brain potential called the Error-Related Negativity (ERN) using off-the-shelf headsets and explores its applicability to HCI. ERN is triggered when a user either makes a mistake or the application behaves differently from their expectation. We first show that ERN can be seen on signals captured by EEG headsets like Emotiv™ when doing a typical multiple choice reaction time (RT) task -- Flanker task. We then present a single-trial online ERN algorithm that works by pre-computing the coefficient matrix of a logistic regression classifier using some data from a multiple choice reaction time task and uses it to classify incoming signals of that task on a single trial of data. We apply it to an interactive selection task that involved users selecting an object under time pressure. Furthermore the study was conducted in a typical office environment with ambient noise. Our results show that online single trial ERN detection is possible using off-the-shelf headsets during tasks that are typical of interactive applications. We then design a Superflick experiment with an integrated module mimicking an ERN detector to evaluate the accuracy of detecting ERN in the context of assisting users in interactive tasks. Based on these results we discuss and present several HCI scenarios for use of ERN.