Acoustophoresis uses sound waves to manipulate small objects in mid-air and has broad potential in various applications. However, stable multi-particle levitation remains challenging due to complex acoustic dynamics and limitations of existing models. We introduce AcoustoReinforce, a reinforcement learning-based path planner that autonomously controls the motion of multiple levitated particles. Leveraging a decentralized architecture, it learns local neural policies that generate particle trajectories independently, enabling scalable, communication-free control even in densely populated acoustic fields. To ensure physical feasibility, acoustic trapping strength is incorporated as a constraint during both training and inference, producing trajectories that are collision-free, acoustically stable, and physically realizable within real-world system constraints. Experiments on a real-world levitation platform show that AcoustoReinforce outperforms state-of-the-art planners, improving task success rates by up to 130% across diverse configurations. These results demonstrate the effectiveness of learning-based decentralized control for complex multi-object acoustophoresis in real environments.

’How was your lunch?’ Even simple food talk reveals how people translate multisensory impressions into language, offering insights for HCI applications such as food recommendation and health management. While prior work has emphasised the multisensory nature of eating, less is known about the linguistic strategies through which people articulate such experiences. We conducted a mixed-methods study combining self-reports and interviews to examine how non-verbal sensory input is transformed into verbal expression. Our analysis shows that participants structured their accounts through temporal phases that integrated perception, cognition, affective reaction, and behaviour; employed associative strategies that shaped expectations; and expressed ambivalence, where positive and negative evaluations coexisted in descriptions such as guilty pleasure. Building on these findings, we propose a lens that highlights the role of time, association, and evaluative language in food talk, enabling designers to translate everyday expressions into actionable insights for food-related HCI design.

Figure 1: The research involved: (i) a design futuring workshop on acoustophoresis, (ii) the development and feasibility assessment of two SONARIOS (i.e., scenarios), and (iii) the synthesis of three strong concepts, along with key tensions and responsibilities in shaping the future of acoustophoresis applications.

Cooperative transport, the simultaneous movement of an object by multiple agents, has been widely observed in biological systems such as ant colonies, which improve efficiency and adaptability in dynamic environments. Inspired by these natural phenomena, we present a novel acoustic robotic system for the transport of contactless objects in mid-air. Our system utilizes onboard phased ultrasonic transducers and a robotic control system to generate localized acoustic pressure fields, enabling the precise manipulation of airborne particles and robots. We categorize contactless object-transport strategies into independent transport (uncoordinated) and forward-facing cooperative transport (coordinated), drawing parallels with biological systems to optimize efficiency and robustness. The proposed system is experimentally validated by evaluating levitation stability using a microphone in the measurements lab, transport efficiency through a phase-space motion capture system, and clock synchronization accuracy using an oscilloscope. The results demonstrate the feasibility of both independent and cooperative airborne object transport. This research contributes to the field of acoustophoretic robotics, with potential applications in contactless material handling, microassembly, and biomedical applications.

Acoustic levitation is an emerging technique that has found application in contactless assembly and dynamic displays. It uses precise phase control in an ultrasound transducer array to manage the positions and movements of multiple particles. Yet, maintaining stable mid-air particles is challenging, with unexpected drops disrupting the intended motion and position. Here, we present StableLev, a data-driven pipeline for the detection and amendment of instabilities in multi-particle levitation. We first curate a hybrid levitation dataset, blending optimized simulations with labels based on actual trajectory outcomes. We then design an AutoEncoder to detect anomalies in the simulated data, correlating closely with observed particle drops. Finally, we reconstruct the acoustic field at anomaly regions to improve particle stability and experimentally demonstrate successful dynamic levitation for trajectories within our dataset. Our work provides new insights into multi-particle levitation and enhances its robustness, which will be valuable in a wide range of applications.

Data physicalisation is a technique that encodes data through the geometric and material properties of an artefact, allowing users to engage with data in a more immersive and multi-sensory way. However, current methods of data physicalisation are limited in terms of their reconfigurability and the types of materials that can be used. Acoustophoresis—a method of suspending and manipulating materials using sound waves—offers a promising solution to these challenges. In this paper, we present DataLev, a design space and platform for creating reconfigurable, multimodal data physicalisations with enriched materiality using acoustophoresis. We demonstrate the capabilities of DataLev through eight examples and evaluate its performance in terms of reconfigurability and materiality. Our work offers a new approach to data physicalisation, enabling designers to create more dynamic, engaging, and expressive artefacts.

In 2013, AmazonWeb Services revolutionized the data warehousing industry by launching Amazon Redshift, the first fully-managed, petabyte-scale, enterprise-grade cloud data warehouse. Amazon Redshift made it simple and cost-effective to efficiently analyze large volumes of data using existing business intelligence tools. This cloud service was a significant leap from the traditional on-premise data warehousing solutions, which were expensive, not elastic, and required significant expertise to tune and operate. Customers embraced Amazon Redshift and it became the fastest growing service in AWS. Today, tens of thousands of customers use Redshift in AWS's global infrastructure to process exabytes of data daily.

Acoustic levitation is gaining popularity as an approach to create physicalized mid-air content by levitating different types of levitation primitives. Such primitives can be independent particles or particles that are physically connected via threads or pieces of cloth to form shapes in mid-air. However, initialization (i.e., placement of such primitives in their mid-air target locations) currently relies on either manual placement or specialized ad-hoc implementations, which limits their practical usage. We present ArticuLev, an integrated pipeline that deals with the identification, assembly and mid-air placement of levitated shape primitives. We designed ArticuLev with the physical properties of commonly used levitation primitives in mind. It enables experiences that seamlessly combine different primitives into meaningful structures (including fully articulated animated shapes) and supports various levitation display approaches (e.g., particles moving at high speed). In this paper, we describe our pipeline and demonstrate it with heterogeneous combinations of levitation primitives.

We present Flowcuits, a DIY fabrication method to prototype tangible, interactive and functional electrical components by manipulating liquid metals. The prototypes afford both physical and visual interactions to demonstrate the inner working mechanics of fundamental electronic elements, which enables tangible and playful learning. The fabrication process follows simple imprinting and sealing of fluidic circuits with a 3D-printed stamp on an accessible moldable-substrates such as ‘Blu Tack’. Utilizing conductive gallium indium liquid metal, we demonstrated interactive and re-configurable electronic components such as switches, variable resistors, variable capacitors, logic gates and pressure sensors. In this paper, we present the design analogy of Flowcuits, DIY fabrication approach including a parametric 3D stamp design toolkit and results from a technical evaluation. The stamps are printed with a low-cost 3D printer and all the materials are inexpensive and reusable, enabling Flowcuits to be easily used without any advanced lab facilities.

Understanding user characteristics such as demographic information is useful for the personalization of online content promoted to users. However, it is difficult to obtain such data for each user visiting the website. Since demographic data for some users can be collected, their behavior can be used to predict the attributes of unknown users. Through online news consumption, we can infer the attributes of users from the articles they view. Most existing models take a supervised learning approach to this modeling task. However, by representing the user-URL interactions with a network, we can convert it to a semi-supervised learning problem and learn embeddings for users. Graph embeddings have become popular in recent years, with research mainly focusing on algorithmic developments. However, while we have an intuitive understanding of the problems they may overcome, such as data sparsity, this problem remains unexplored in the domain of demographic prediction using behavior. In this paper, we first investigate the effectiveness of using user embeddings generated from network representation learning for prediction by comparing its performance with other traditional feature sets, including content and item-based features. We find that the embeddings can represent a user generally on two prediction tasks, (1) gender prediction (classification) and (2) age prediction (regression). Second, we explore the advantages of using these embeddings over the other methods in two cases of data sparsity, where (1) the training and testing sets of users are temporally split and (2) the user labels are imbalanced. In both these cases, the embeddings outperform the baseline.

We present G-ID, a method that utilizes the subtle patterns left by the 3D printing process to distinguish and identify objects that otherwise look similar to the human eye. The key idea is to mark different instances of a 3D model by varying slicing parameters that do not change the model geometry but can be detected as machine-readable differences in the print. As a result, G-ID does not add anything to the object but exploits the patterns appearing as a by-product of slicing, an essential step of the 3D printing pipeline.

On-skin haptic interfaces using soft elastomers which are thin and flexible have significantly improved in recent years. Many are focused on vibrotactile feedback that requires complicated parameter tuning. Another approach is based on mechanical forces created via piezoelectric devices and other methods for non-vibratory haptic sensations like stretching, twisting. These are often bulky with electronic components and associated drivers are complicated with limited control of timing and precision. This paper proposes HapBead, a new on-skin haptic interface that is capable of rendering vibration like tactile feedback using microfluidics. HapBead leverages a microfluidic channel to precisely and agilely oscillate a small bead via liquid flow, which then generates various motion patterns in channel that creates highly tunable haptic sensations on skin. We developed a proof-of-concept design to implement thin, flexible and easily affordable HapBead platform, and verified its haptic rendering capabilities via attaching it to users' fingertips. A study was carried out and confirmed that participants could accurately tell six different haptic patterns rendered by HapBead. HapBead enables new wearable display applications with multiple integrated functionalities such as on-skin haptic doodles, visuo-haptic displays and haptic illusions.

The Sense of Agency (SoA) is crucial in interaction with technology, it refers to the feeling of 'I did that' as opposed to 'the system did that' supporting a feeling of being in control. Research in human-computer interaction has recently studied agency in visual, auditory and haptic interfaces, however the role of smell on agency remains unknown. Our sense of smell is quite powerful to elicit emotions, memories and awareness of the environment, which has been exploited to enhance user experiences (e.g., in VR and driving scenarios). In light of increased interest in designing multimodal interfaces including smell and its close link with emotions, we investigated, for the first time, the effect of smell-induced emotions on the SoA. We conducted a study using the Intentional Binding (IB) paradigm used to measure SoA while participants were exposed to three scents with different valence (pleasant, unpleasant, neutral). Our results show that participants? SoA increased with a pleasant scent compared to neutral and unpleasant scents. We discuss how our results can inform the design of multimodal and future olfactory interfaces.

This paper presents a thin digital full-face mask display that can reflect an entire facial expression of a user onto an avatar to support augmented face-to-face communication in real environments. Although camera-based facial expression recognition technology has enabled people to augment their faces with avatars, application was limited to face-to-face communication in virtual environments. To enable digital facial augmentation with an avatar in a real space, we propose a digital face mask display system that integrates a lightweight flexible display with a thin facial expression recognition system. The thin wearable facial expression recognition system was implemented with photo reflective sensor arrays which can measure facial expressions at 40 feature points distributed across an entire face. We investigated a ten-class facial expression identification model based on an SVM training algorithm. The trained model achieved an average accuracy of 79% when identifying the facial expressions of multiple users. User experiments indicated that the proposed thin digital full-face mask display allows the wearer to control the facial expression of the avatar with a fast response rate and create a positive sense of self-agency and self-ownership toward the augmented avatar face.

User behavior modeling is important for industrial applications such as demographic attribute prediction, content recommendation, and target advertising. Existing methods represent behavior log as a sequence of adopted items and find sequential patterns; however, concrete location and time information in the behavior log, reflecting dynamic and periodic patterns, joint with the spatial dimension, can be useful for modeling users and predicting their characteristics. In this work, we propose a novel model based on graph neural networks for learning user representations from spatiotemporal behavior data. Our model's architecture incorporates two networked structures. One is a tripartite network of items, sessions, and locations. The other is a hierarchical calendar network of hour, week, and weekday nodes. It first aggregates embeddings of location and items into session embeddings via the tripartite network, and then generates user embeddings from the session embeddings via the calendar structure. The user embeddings preserve spatial patterns and temporal patterns of a variety of periodicity (e.g., hourly, weekly, and weekday patterns). It adopts the attention mechanism to model complex interactions among the multiple patterns in user behaviors. Experiments on real datasets (i.e., clicks on news articles in a mobile app) show our approach outperforms strong baselines for predicting missing demographic attributes.

3D selection in dense VR environments (e.g., point clouds) is extremely challenging due to occlusion and imprecise mid-air input modalities (e.g., 3D controllers and hand gestures). In this paper, we propose "Slicing-Volume", a hybrid selection technique that enables simultaneous 3D interaction in mid-air, and a 2D pen-and-tablet metaphor in VR. Inspired by well-known slicing plane techniques in data visualization, our technique consists of a 3D volume that encloses target objects in mid-air, which are then projected to a 2D tablet view for precise selection on a tangible physical surface. While slicing techniques and tablets-in-VR have been previously explored, in this paper, we evaluated the potential of this hybrid approach to improve accuracy in highly occluded selection tasks, comparing different multimodal interactions (e.g., Mid-air, Virtual Tablet and Real Tablet). Our results showed that our hybrid technique significantly improved overall accuracy of selection compared to Mid-air selection only, thanks to the added haptic feedback given by the physical tablet surface, rather than the added visualization given by the tablet view.

Phased Arrays of Transducers (PATs) allow accurate control of ultrasound fields, with applications in haptics, levitation (i.e. displays) and parametric audio. However, algorithms for multi-point levitation or tactile feedback are usually limited to computing solutions in the order of hundreds of sound-fields per second, preventing the use of multiple high-speed points, a feature that can broaden the scope of applications of PATs. We present GS-PAT , a GPU multi-point phase retrieval algorithm, capable of computing 17K solutions per second for up to 32 simultaneous points in a mid-end consumer grade GPU (NVidia GTX 1660). We describe the algorithm and compare it to state of the art multi-point algorithms used for ultrasound haptics and levitation, showing similar quality of the generated sound-fields, and much higher computation rates. We then illustrate how the shift in paradigm enabled by GS-PAT (i.e. real-time control of several high-speed points) opens new applications for PAT technologies, such as in volumetric fully coloured displays, multi-point spatio-temporal tactile feedback, parametric audio and simultaneous combinations of these modalities.

Mid-air tactile stimulation using ultrasonics has been used in a variety of human computer interfaces in the form of prototypes as well as products. When generating these tactile patterns with mid-air tactile ultrasonic displays, the common approach has been to sample the patterns using the hardware update rate capabilities to their full extent. In the current study we show that the hardware update rate can impact perception, but unexpectedly we find that higher update rates do not improve pattern perception. In a first user study, we highlight the effect of update rate on the perceived strength of a pattern, especially for patterns rendered at slow rate of less than 10 Hz. In a second user study, we identify the evolution of the optimal update rate according to variations in pattern size. Our main results show that update rate should be designated as additional parameter for tactile patterns. We also discuss how the relationships we defined in the current study can be implemented into designer tools so that designers remain oblivious to this additional complexity.

Touchscreens are the predominant medium for interactions with digital services; however, their current fixed form factor narrows the scope for rich physical interactions by limiting interaction possibilities to a single, planar surface. In this paper we introduce the concept of PickCells, a fully re-configurable device concept composed of cells, that breaks the mould of rigid screens and explores a modular system that affords rich sets of tangible interactions and novel across-device relationships. Through a series of co-design activities -- involving HCI experts and potential end-users of such systems -- we synthesised a design space aimed at inspiring future research, giving researchers and designers a framework in which to explore modular screen interactions. The design space we propose unifies existing works on modular touch surfaces under a general framework and broadens horizons by opening up unexplored spaces providing new interaction possibilities. In this paper, we present the PickCells concept, a design space of modular touch surfaces, and propose a toolkit for quick scenario prototyping.

Centuries of development in optics have given us passive devices (i.e. lenses, mirrors and filters) to enrich audience immersivity with light effects, but there is nothing similar for sound. Beam-forming in concert halls and outdoor gigs still requires a large number of speakers, while headphones are still the state-of-the-art for personalized audio immersivity in VR. In this work, we show how 3D printed acoustic meta-surfaces, assembled into the equivalent of optical systems, may offer a different solution. We demonstrate how to build them and how to use simple design tools, like the thin-lens equation, also for sound. We present some key acoustic devices, like a "collimator", to transform a standard computer speaker into an acoustic "spotlight"; and a "magnifying glass", to create sound sources coming from distinct locations than the speaker itself. Finally, we demonstrate an acoustic varifocal lens, discussing applications equivalent to auto-focus cameras and VR headsets and the limitations of the technology.

Permeable, mid-air displays, such as those using fog or water mist are limited by our ability to shape and control the aerosol and deal with two major issues: (1) the size and complexity of the system, and (2) the creation of laminar flow, to retain display quality. Here we present SonicSpray, a technique using ultrasonic Bessel beams to create reconfigurable mid-air displays. We build a prototype from low-cost, off-the-shelf parts. We explore the potential and limitations of SonicSpray to create and redirect laminar flows of fog. We demonstrate a working prototype that precisely controls laminar aerosols through only 6x6 ultrasound transducers array. We describe the implementation steps to build the device, verify the control and projection algorithm for the display, and evaluate its performance. We finally report our exploration of several useful applications, in learning, entertainment and arts.

LeviProps are tangible structures used to create interactive mid-air experiences. They are composed of an acoustically- transparent lightweight piece of fabric and attached beads that act as levitated anchors. This combination enables real- time 6 Degrees-of-Freedom control of levitated structures which are larger and more diverse than those possible with previous acoustic manipulation techniques. LeviProps can be used as free-form interactive elements and as projection surfaces. We developed an authoring tool to support the creation of LeviProps. Our tool considers the outline of the prop and the user constraints to compute the optimum locations for the anchors (i.e. maximizing trapping forces), increasing prop stability and maximum size. The tool produces a final LeviProp design which can be fabricated following a simple procedure. This paper explains and evaluates our approach and showcases example applications, such as interactive storytelling, games and mid-air displays.

Scale adaptive techniques for VR navigation enable users to navigate spaces larger than the real space available, while allowing precise interaction when required. However, due to these techniques gradually scaling displacements as the user moves (changing user's speed), they introduce a Drift effect. That is, a user returning to the same point in VR will not return to the same point in the real space. This mismatch between the real/virtual spaces can grow over time, and turn the techniques unusable (i.e., users cannot reach their target locations). In this paper, we characterise and analyse the effects of Drift, highlighting its potential detrimental effects. We then propose two techniques to correct Drift effects and use a data driven approach (using navigation data from real users with a specific scale adaptive technique) to tune them, compare their performance and chose an optimum correction technique and configuration. Our user study, applying our technique in a different environment and with two different scale adaptive navigation techniques, shows that our correction technique can significantly reduce Drift effects and extend the life-span of the navigation techniques (i.e., time that they can be used before Drift draws targets unreachable), while not hindering users' experience.

Acoustic levitation enables a radical new type of human-computer interface composed of small levitating objects. For the first time, we investigate the selection of such objects, an important part of interaction with a levitating object display. We present Point-and-Shake, a mid-air pointing interaction for selecting levitating objects, with feedback given through object movement. We describe the implementation of this technique and present two user studies that evaluate it. The first study found that users could accurately (96%) and quickly (4.1s) select objects by pointing at them. The second study found that users were able to accurately (95%) and quickly (3s) select occluded objects. These results show that Point-and-Shake is an effective way of initiating interaction with levitating object displays.

The Sense of Agency (SoA) refers to our capability to control our own actions and influence the world around us. Recent research in HCI has been investigating SoA to provide users an instinctive sense of "I did that" as opposed to "the system did that". However, current agency measurements are limited. The Intentional Binding (IB) paradigm provides an implicit measure of the SoA, however, it is constrained by requiring high visual attention to a "Libet clock" on-screen. In this paper, we extended the timing stimuli through auditory and tactile cues. Our results demonstrate that audio timing through voice commands and haptic timing through tactile cues on the hand, are an effective alternative measure of the SoA using the IB paradigm. They both address current limitations of the traditional method such as visual attention overload and lack of engagement. We discuss how our results can be applied to measure SoA in tasks involving different interactive scenarios such as in Mixed/Virtual Reality.

We present Tangible Drops, a visio-tactile display that for the first time provides physical visualization and tactile feedback using a planar liquid interface. It presents digital information interactively by tracing dynamic patterns on horizontal flat surfaces using liquid metal drops on a programmable electrode array. It provides tactile feedback with directional information in the 2D vector plane using linear locomotion and/or vibration of the liquid metal drops. We demonstrate move, oscillate, merge, split and dispense-from-reservoir functions of the liquid metal drops by consuming low power (450 mW per electrode) and low voltage (8--15 V). We report on results of our empirical study with 12 participants on tactile feedback using 8 mm diameter drops, which indicate that Tangible Drops can convey tactile sensations such as changing speed, varying direction and controlled oscillation with no visual feedback. We present the design space and demonstrate the applications of Tangible Drops, and conclude by suggesting potential future applications for the technique.

Explainability/Interpretability in machine learning applications is becoming critical, with legal and industry requirements demanding human understandable machine learning results. We describe the additional complexities that occur when a known interpretability technique (canary models) is applied to a real production scenario. We furthermore argue that reproducibility is a key feature in practical usages of such interpretability techniques in production scenarios. With this motivation, we present a production ML reproducibility solution, namely a comprehensive time ordered event sequence for machine learning applications. We demonstrate how our approach can bring this known common interpretability technique into production viability. We further present the system design and early performance characteristics of our reproducibility solution.

Ultrasound manipulation is growing in popularity in the HCI community with applications in haptics, on-body interaction, and levitation-based displays. Most of these applications share two key limitations: a) the complexity of the sound fields that can be produced is limited by the physical size of the transducers, and b) no obstacles can be present between the transducers and the control point. We present SoundBender, a hybrid system that overcomes these limitations by combining the versatility of phased arrays of Transducers (PATs) with the precision of acoustic metamaterials. In this paper, we explain our approach to design and implement such hybrid modulators (i.e. to create complex sound fields) and methods to manipulate the field dynamically (i.e. stretch, steer). We demonstrate our concept using self-bending beams enabling both levitation and tactile feedback around an obstacle and present example applications enabled by SoundBender.

Many users value the ability to have quick and frequent sight of their mobiles when in public settings. However, in doing so, they expose themselves to potential risks, ranging from being targets of robbery to the more subtle social losses through being seen to be rude or inattentive to those around them. In nature, some animals can blend into their environments to avoid being eaten or to reduce their impact on the ecosystem around them. Taking inspiration from these evolved systems we investigate the notion of chameleon approaches for mobile interaction design. Our probes were motivated, inspired and refined through extended interactions with people drawn from contexts with differing ranges of security and privacy concerns. Through deployments on users' own devices, our prototypes show the value of the concept. The encouraging results motivate further research in materials and form factors that can provide more effective automatic plain-sight hiding.

Touchless interfaces allow users to view, control and manipulate digital content without physically touching an interface. They are being explored in a wide range of application scenarios from medical surgery to car dashboard controllers. One aspect of touchless interaction that has not been explored to date is the Sense of Agency (SoA). The SoA refers to the subjective experience of voluntary control over actions in the external world. In this paper, we investigated the SoA in touchless systems using the intentional binding paradigm. We first compare touchless systems with physical interactions and then augmented different types of haptic feedback to explore how different outcome modalities influence intentional binding. From our experiments, we demonstrated that an intentional binding effect is observed in both physical and touchless interactions with no statistical difference. Additionally, we found that haptic and auditory feedback help to increase SoA compared with visual feedback in touchless interfaces. We discuss these findings and identify design opportunities that take agency into consideration.

Many finger sensing input devices now support proximity input, enabling users to perform in-air gestures. While near-surface interactions increase the input vocabulary, they lack tactile feedback, making it hard for users to perform gestures or to know when the interaction takes place. Sparkle stimulates the fingertip with touchable electric arcs above a hover sensing device to give users in-air tactile or thermal feedback, sharper and more feelable than acoustic mid-air haptic devices. We present the design of a high voltage resonant transformer with a low-loss soft ferrite core and self-tuning driver circuit, with which we create electric arcs 6 mm in length, and combine this technology with infrared proximity sensing in two proof-of-concept devices with form factor and functionality similar to a button and a touchpad. We provide design guidelines for Sparkle devices and examples of stimuli in application scenarios, and report the results of a user study on the perceived sensations. Sparkle is the first step towards providing a new type of hover feedback, and it does not require users to wear tactile stimulators.

We present MistForm, a shape changing fog display that can support one or two users interacting with either 2D or 3D content. Mistform combines affordances from both shape changing interfaces and mid-air displays. For example, a concave display can maintain content in comfortable reach for a single user, while a convex shape can support several users engaged on individual tasks. MistForm also enables unique interaction possibilities by exploiting the synergies between shape changing interfaces and mid-air fog displays. For instance, moving the screen will affect the brightness and blurriness of the screen at specific locations around the display, creating spaces with similar (collaboration) or different visibility (personalized content). We describe the design of MistForm and analyse its inherent challenges, such as image distortion and uneven brightness on dynamic curved surfaces. We provide a machine learning approach to characterize the shape of the screen and a rendering algorithm to remove aberrations. We finally explore novel interactive possibilities and reflect on their potential and limitations.

The ability to control acoustic fields offers many possible applications in loudspeaker design, ultrasound imaging, medical therapy, and acoustic levitation. Sound waves are currently shaped using phased array systems, even though the complex electronics required are expensive and hinder widespread use. Here we show how to control, direct, and manipulate sound using 2-dimensional, planar, acoustic metasurfaces that require only one driving signal. This offers the advantages of ease of use and versatility over currently available phased arrays. We demonstrate the creation of a haptic sensation and steering of a beam produced by a parametric speaker. This simple, yet highly effective, method of creating single-beam manipulators could be introduced in medical or manufacturing applications.

The screen in our laptops or the white surface of a projector screen are common examples of displays that we use in our daily life to visualize information. Here, we propose to use levitated particles above these displays to enrich the presented information. By means of acoustic or magnetic levitation we show that it is possible to move a particle in mid-air in the space above these displays. We present different configurations that are feasible with current technology, as well as use cases. In this demo, the visitors will be able to explore and interact with various levitators integrated with traditional displays.

In this paper, we demonstrate a method for the dynamic 2D transformation of liquid matter and present unique organic animations based on spatio-temporally controlled electric fields. In particular, we deploy a droplet of liquid metal (Gallium indium eutectic alloy) in a 7x7 electrode array prototype system, featuring an integrated image tracking system and a simple GUI. Exploiting the strong dependance of EGaIn's surface tension on external electric voltages, we control multiple electrodes dynamically to manipulate the liquid metal into a fine-grained desired shape. Taking advantage of the high conductivity of liquid metals, we introduce a shape changing, reconfigurable smart circuit as an example of unique applications. We discuss system constraints and the overarching challenge of controlling liquid metals in the presence of phenomena such as splitting, self-electrode interference and finger instabilities. Finally, we reflect on the broader vision of this project and discuss our work in the context of the wider scope of programmable materials.

We present a programmable liquid matter which can dynamically transform its 2D shape into a variety of forms and present unique organic animations based on spatio-temporally controlled electric fields. We deployed a EGaIn (Gallium indium eutectic alloy) liquid metal as our smart liquid material since it features a superior electric conductivity in spite of a liquid state and presents a high dynamic range of surface tension and 2D area controlled by applied voltage strength and polarity. Our proposed liquid metal shape and motion control algorithms with dynamically patterned electric fields realize path tracing organic animation. We demonstrate an interactive 7x7 electrode array control system with a computer vision based GUI system to enable novice users to physically draw alphabet letters and 2D shapes by unique animatronics of liquid metals.

We present two realizations of TastyFloats, a novel system that uses acoustic levitation to deliver food morsels to the users' tongue. To explore TastyFloats' associated design framework, we first address the technical challenges to successfully levitate and deliver different types of foods on the tongue. We then conduct a user study, assessing the effect of acoustic levitation on users' taste perception, comparing three basic taste stimuli (i.e., sweet, bitter and umami) and three volume sizes of droplets (5μL, 10μL and 20μL). Our results show that users perceive sweet and umami easily, even in minimal quantities, whereas bitter is the least detectable taste, despite its typical association with an unpleasant taste experience. Our results are a first step towards the creation of new culinary experiences and innovative gustatory interfaces.

Interaction in VR involves large body movements, easily inducing fatigue and discomfort. We propose Erg-O, a manipulation technique that leverages visual dominance to maintain the visual location of the elements in VR, while making them accessible from more comfortable locations. Our solution works in an open-ended fashion (no prior knowledge of the object the user wants to touch), can be used with multiple objects, and still allows interaction with any other point within user's reach. We use optimization approaches to compute the best physical location to interact with each visual element, and space partitioning techniques to distort the visual and physical spaces based on those mappings and allow multi-object retargeting. In this paper we describe the Erg-O technique, propose two retargeting strategies and report the results from a user study on 3D selection under different conditions, elaborating on their potential and application to specific usage scenarios.

Usability has a distinct subjective component, yet surprisingly little is known about its neural basis and relation to the neuroanatomy of aesthetics. To begin closing this gap, we conducted two functional magnetic resonance imaging studies in which participants were shown static webpages (in the first study) and videos of interaction with webpages (in the second study). The webpages were controlled so as to exhibit high and low levels of perceived usability and perceived aesthetics. Our results show unique links between perceived usability and brain areas involved in functions such as emotional processing (left fusiform gyrus, superior frontal gyrus), anticipation of physical interaction (precentral gyrus), task intention (anterior cingulate cortex), and linguistic processing (medial and bilateral superior frontal gyri). We use these findings to discuss the brain correlates of perceived usability and the use of fMRI for usability evaluation and for generating new user experiences.

We present a study of children collaborating around interactive tabletops in three different countries: India, the United Kingdom and Finland. Our data highlights the key distinctive physical strategies used by children when performing collaborative tasks during this study. Children in India employ dynamic positioning with frequent physical contact and simultaneous object movement. Children in the UK tend to prefer static positioning with minimal physical contact and simultaneous object movement. Children in Finland use a mixture of dynamic and static positioning with minimal physical contact and object movement. Our findings indicate the importance of understanding collaboration strategies and behaviours when designing and deploying interactive tabletops in heterogeneous educational environments. We conclude with a discussion on how designers of tabletops for schools can provide opportunities for children in different countries to define and shape their own collaboration strategies for small group learning that take into account their different classroom practices.

We present TableHop, a tabletop display that provides controlled self-actuated deformation and vibro-tactile feedback to an elastic fabric surface while retaining the ability for high-resolution visual projection. The surface is made of a highly stretchable pure spandex fabric that is electrostatically actuated using electrodes mounted on its top or underside. It uses transparent indium tin oxide electrodes and high-voltage modulation to create controlled surface deformations. Our setup actuates pixels and creates deformations in the fabric up to +/- 5 mm. Since the electrodes are transparent, the fabric surface functions as a diffuser for rear-projected visual images, and avoid occlusion by users or actuators. Users can touch and interact with the fabric to experience expressive interactions as with any fabric based shape-changing interface. By using frequency modulation in the high-voltage circuit, it can also create localized tactile sensations on the user's fingertip when touching the surface. We provide simulation and experimental results for the shape of the deformation and frequency of the vibration of the surface. These results can be used to build prototypes of different sizes and form-factors. We present a working prototype of TableHop that has 30x40 cm2 surface area and uses a grid of 3x3 transparent electrodes. It uses a maximum of 9.46 mW and can create tactile vibrations of up to 20 Hz. TableHop can be scaled to make large interactive surfaces and integrated with other objects and devices. TableHop will improve user interaction experience on 2.5D deformable displays.

We introduce Cubimorph, a modular interactive device that accommodates touchscreens on each of the six module faces, and that uses a hinge-mounted turntable mechanism to self-reconfigure in the user's hand. Cubimorph contributes toward the vision of programmable matter where interactive devices reconfigure in any shape that can be made out of a chain of cubes in order to fit a myriad of functionalities, e.g. a mobile phone shifting into a console when a user launches a game. We present a design rationale that exposes user requirements to consider when designing homogeneous modular interactive devices. We present our Cubimorph mechanical design, three prototypes demonstrating key aspects (turntable hinges, embedded touchscreens and miniaturization), and an adaptation of the probabilistic roadmap algorithm for the reconfiguration.

The infrastructure beneath a worldwide social network has to continually serve billions of variable-sized media objects such as photos, videos, and audio clips. These objects must be stored and served with low latency and high throughput by a system that is geo-distributed, highly scalable, and load-balanced. Existing file systems and object stores face several challenges when serving such large objects. We present Ambry, a production-quality system for storing large immutable data (called blobs). Ambry is designed in a decentralized way and leverages techniques such as logical blob grouping, asynchronous replication, rebalancing mechanisms, zero-cost failure detection, and OS caching. Ambry has been running in LinkedIn's production environment for the past 2 years, serving up to 10K requests per second across more than 400 million users. Our experimental evaluation reveals that Ambry offers high efficiency (utilizing up to 88% of the network bandwidth), low latency (less than 50 ms latency for a 1 MB object), and load balancing (improving imbalance of request rate among disks by 8x-10x).

We present JOLED, a mid-air display for interactive physical visualization using Janus objects as physical voxels. The Janus objects have special surfaces that have two or more asymmetric physical properties at different areas. In JOLED, they are levitated in mid-air and controllably rotated to reveal their different physical properties. We made voxels by coating the hemispheres of expanded polystyrene beads with different materials, and applied a thin patch of titanium dioxide to induce electrostatic charge on them. Transparent indium tin oxide electrodes are used around the levitation volume to create a tailored electric field to control the orientation of the voxels. We propose a novel method to control the angular position of individual voxels in a grid using electrostatic rotation and their 3D position using acoustic levitation. We present a display in which voxels can be flipped independently, and two mid-air physical games with a voxel as the playable character that moves in 3D across other physical structures and rotates to reflect its status in the games. We demonstrate a voxel update speed of 37.8 ms/flip, which is video-rate.

Physical representations of data have existed for thousands of years. Yet it is now that advances in digital fabrication, actuated tangible interfaces, and shape-changing displays are spurring an emerging area of research that we call Data Physicalization. It aims to help people explore, understand, and communicate data using computer-supported physical data representations. We call these representations physicalizations, analogously to visualizations -- their purely visual counterpart. In this article, we go beyond the focused research questions addressed so far by delineating the research area, synthesizing its open challenges and laying out a research agenda.

Touch is a powerful vehicle for communication between humans. The way we touch (how) embraces and mediates certain emotions such as anger, joy, fear, or love. While this phenomenon is well explored for human interaction, HCI research is only starting to uncover the fine granularity of sensory stimulation and responses in relation to certain emotions. Within this paper we present the findings from a study exploring the communication of emotions through a haptic system that uses tactile stimulation in mid-air. Here, haptic descriptions for specific emotions (e.g., happy, sad, excited, afraid) were created by one group of users to then be reviewed and validated by two other groups of users. We demonstrate the non-arbitrary mapping between emotions and haptic descriptions across three groups. This points to the huge potential for mediating emotions through mid-air haptics. We discuss specific design implications based on the spatial, directional, and haptic parameters of the created haptic descriptions and illustrate their design potential for HCI based on two design ideas.

LeviPath is a modular system to levitate objects across 3D paths. It consists of two opposed arrays of transducers that create a standing wave capable of suspending objects in mid-air. To control the standing wave, the system employs a novel algorithm based on combining basic patterns of movement. Our approach allows the control of multiple beads simultaneously along different 3D paths. Due to the patterns and the use of only two opposed arrays, the system is modular and can scale its interaction space by joining several LeviPaths. In this paper, we describe the hardware architecture, the basic patterns of movement and how to combine them to produce 3D path visualisations.

The form factors of displays using fog or water surface are limited by our ability to control the non-solid substances used as the diffuser. We propose a charging technique for polar aerosols (e.g., mist or fog) that allows control of the shape and trajectory of such non-solid diffusers using electric fields. We report experiments that allowed us to design a charging mechanism that produces charged fog aerosols with homogeneous electrical mobility. We illustrate our idea by demonstrating how electric fields can be used to control the shape of a fog display and the trajectory of a bubble display.

There is a rapidly growing momentum driving the development of mobile payment systems for co-present interactions, using near-field communication on smartphones and contactless payment systems. The design (and marketing) imperative for this is to enable faster, simpler, effortless and secure transactions, yet our evidence shows that this focus on reducing transactional friction may ignore other important features around making payments. We draw from empirical data to consider user interactions around financial exchanges made on mobile phones. Our findings examine how the practices around making payments support people in making connections, to other people, to their communities, to the places they move through, to their environment, and to what they consume. While these social and community bonds shape the kinds of interactions that become possible, they also shape how users feel about, and act on, the values that they hold with their co-users. We draw implications for future payment systems that make use of community connections, build trust, leverage transactional latency, and generate opportunities for rich social interactions.

Apache Kafka is a scalable publish-subscribe messaging system with its core architecture as a distributed commit log. It was originally built at LinkedIn as its centralized event pipelining platform for online data integration tasks. Over the past years developing and operating Kafka, we extend its log-structured architecture as a replicated logging backbone for much wider application scopes in the distributed environment. In this abstract, we will talk about our design and engineering experience to replicate Kafka logs for various distributed data-driven systems at LinkedIn, including source-of-truth data storage and stream processing.

We present rich descriptions of taste experience through an analysis of the diachronic and synchronic experiences of each of the five basic taste qualities: sweet, sour, salt, bitter, and umami. Our findings, based on a combination of user experience evaluation techniques highlight three main themes: temporality, affective reactions, and embodiment. We present the taste characteristics as a framework for design and discuss each taste in order to elucidate the design qualities of individual taste experiences. These findings add a semantic understanding of taste experiences, their temporality enhanced through descriptions of the affective reactions and embodiment that the five basic tastes elicit. These findings are discussed on the basis of established psychological and behavioral phenomena, highlighting the potential for taste-enhanced design.

Shape-changing handheld devices are emerging as research prototypes, but it is unclear how users perceive them and which experiences they engender. The little data we have on user experience is from single prototypes, only covering a small part of the possibilities in shape change. We produce 51 videos of a shape-changing handheld device by systematically varying seven parameters of shape change. In a crowd-sourced study, 187 participants watched the videos and described their experiences using rating scales and free text. We find significant and large differences among parameters of shape change. Shapes that have previously been used for notifications were rated the least urgent; the degree of shape change was found to impact experience more than type of shape change. The experience of shape change was surprisingly complex: hedonic quality were inversely related to urgency, and some shapes were perceived as ugly, yet useful. We discuss how to advance models of shape change and improve research on the experience of shape change.

We present MisTable, a tabletop system that combines a conventional horizontal interactive surface with personal screens between the user and the tabletop surface. These personal screens, built using fog, are both see-through and reach-through. Being see-through provides direct line of sight of the personal screen and the elements behind it on the tabletop. Being reach-through allows the user to switch from interacting with the personal screen to reaching through it to interact with the tabletop or the space above it. The personal screen allows a range of customisations and novel interactions such as presenting 2D personal contents on the screen, 3D contents above the tabletop or augmenting and relighting tangible objects differently for each user. Besides, having a personal screen for each user allows us to customize the view of each of them according to their identity or preferences. Finally, the personal screens preserve all well-established tabletop interaction techniques like touch and tangible interactions. We explore the challenges in building such a reach-through system through a proof-of-concept implementation and discuss the possibilities afforded by the system.

Advances in shape changing assemblies have been made in reconfiguration algorithms, hardware designs and interaction techniques. However no tools exist for guiding designers in building those modular devices and especially for choosing the shape of the units. The task becomes even more complex when the units themselves can change their shapes to animate the entire assembly. In this paper, we contribute with the first analysis tool which helps the designer to both choose the right subset of forms for the units and to create an assembly with maximum accuracy from the set of given objects. We introduce the concept of Changibles that are interactive wireless units that can reshape themselves and be attached together to create an animated assembly. We present a use case to demonstrate the use of our tool, with an instantiation of six Changibles that are used to construct a pulsing heart assembly.

We present SensaBubble, a chrono-sensory mid-air display system that generates scented bubbles to deliver information to the user via a number of sensory modalities. The system reliably produces single bubbles of specific sizes along a directed path. Each bubble produced by SensaBubble is filled with fog containing a scent relevant to the notification. The chrono-sensory aspect of SensaBubble means that information is presented both temporally and multimodally. Temporal information is enabled through two forms of persistence: firstly, a visual display projected onto the bubble which only endures until it bursts; secondly, a scent released upon the bursting of the bubble slowly disperses and leaves a longer-lasting perceptible trace of the event. We report details of SensaBubble's design and implementation, as well as results of technical and user evaluations. We then discuss and demonstrate how SensaBubble can be adapted for use in a wide range of application contexts -- from an ambient peripheral display for persistent alerts, to an engaging display for gaming or education.

Error Related Negativity is triggered when a user either makes a mistake or the application behaves differently from their expectation. It can also appear while observing another user making a mistake. This paper investigates ERN in collaborative settings where observing another user (the executer) perform a task is typical and then explores its applicability to HCI. We first show that ERN can be detected on signals captured by commodity EEG headsets like an Emotiv headset when observing another person perform a typical multiple-choice reaction time task. We then investigate the anticipation effects by detecting ERN in the time interval when an executer is reaching towards an answer. We show that we can detect this signal with both a clinical EEG device and with an Emotiv headset. Our results show that online single trial detection is possible using both headsets during tasks that are typical of collaborative interactive applications. However there is a trade-off between the detection speed and the quality/prices of the headsets. Based on the results, we discuss and present several HCI scenarios for use of ERN in observing tasks and collaborative settings.

Ultrasonic haptic feedback is a promising means of providing tactile sensations in mid-air without encumbering the user with an actuator. However, controlled and rigorous HCI research is needed to understand the basic characteristics of perception of this new feedback medium, and so how best to utilise ultrasonic haptics in an interface. This paper describes two experiments conducted into two fundamental aspects of ultrasonic haptic perception: 1) localisation of a static point and 2) the perception of motion. Understanding these would provide insight into 1) the spatial resolution of an ultrasonic interface and 2) what forms of feedback give the most convincing illusion of movement. Results show an average localisation error of 8.5mm, with higher error along the longitudinal axis. Convincing sensations of motion were produced when travelling longer distances, using longer stimulus durations and stimulating multiple points along the trajectory. Guidelines for feedback design are given.

Snapshots are a common and heavily relied upon feature in storage systems. The high performance of flash-based storage systems brings new, more stringent, requirements for this classic capability. We present ioSnap, a flash optimized snapshot system. Through careful design exploiting common snapshot usage patterns and flash oriented optimizations, including leveraging native characteristics of Flash Translation Layers, ioSnap delivers low-overhead snapshots with minimal disruption to foreground traffic. Through our evaluation, we show that ioSnap incurs negligible performance overhead during normal operation, and that common-case operations such as snapshot creation and deletion incur little cost. We also demonstrate techniques to mitigate the performance impact on foreground I/O during intensive snapshot operations such as activation. Overall, ioSnap represents a case study of how to integrate snapshots into a modern, well-engineered flash-based storage system.

We present Portallax, a clip-on technology to retrofit mobile devices with 3D display capabilities. Available technologies (e.g. Nintendo 3DS or LG Optimus) and clip-on solutions (e.g. 3DeeSlide and Grilli3D) force users to have a fixed head and device positions. This is contradictory to the nature of a mobile scenario, and limits the usage of interaction techniques such as tilting the device to control a game. Portallax uses an actuated parallax barrier and face tracking to realign the barrier's position to the user's position. This allows us to provide stereo, motion parallax and perspective correction cues in 60 degrees in front of the device. Our optimized design of the barrier minimizes colour distortion, maximizes resolution and produces bigger view-zones, which support ~81% of adults' interpupillary distances and allow eye tracking implemented with the front camera. We present a reference implementation, evaluate its key features and provide example applications illustrating the potential of Portallax.

We present a two-part exploration on mobile true-3D displays, i.e. displaying volumetric 3D content in mid-air. We first identify and study the parameters of a mobile true-3D projection, in terms of the projection's distance to the phone, angle to the phone, display volume and position within the display. We identify suitable parameters and constraints, which we propose as requirements for developing mobile true-3D systems. We build on the first outcomes to explore methods for coordinating the display configurations of the mobile true-3D setup. We explore the resulting design space through two applications: 3D map navigation and 3D interior design. We discuss the implications of our results for the future design of mobile true-3D displays.

Reflective optical combiners like beam splitters and two way mirrors are used in AR to overlap digital contents on the users' hands or bodies. Augmentations are usually unidirectional, either reflecting virtual contents on the user's body (Situated Augmented Reality) or augmenting user's reflections with digital contents (AR mirrors). But many other novel possibilities remain unexplored. For example, users' hands, reflected inside a museum AR cabinet, can allow visitors to interact with the artifacts exhibited. Projecting on the user's hands as their reflection cuts through the objects can be used to reveal objects' internals. Augmentations from both sides are blended by the combiner, so they are consistently seen by any number of users, independently of their location or, even, the side of the combiner through which they are looking. This paper explores the potential of optical combiners to merge the space in front and behind them. We present this design space, identify novel augmentations/interaction opportunities and explore the design space using three prototypes.

We present a method for creating three-dimensional haptic shapes in mid-air using focused ultrasound. This approach applies the principles of acoustic radiation force , whereby the non-linear effects of sound produce forces on the skin which are strong enough to generate tactile sensations. This mid-air haptic feedback eliminates the need for any attachment of actuators or contact with physical devices. The user perceives a discernible haptic shape when the corresponding acoustic interference pattern is generated above a precisely controlled two-dimensional phased array of ultrasound transducers. In this paper, we outline our algorithm for controlling the volumetric distribution of the acoustic radiation force field in the form of a three-dimensional shape. We demonstrate how we create this acoustic radiation force field and how we interact with it. We then describe our implementation of the system and provide evidence from both visual and technical evaluations of its ability to render different shapes. We conclude with a subjective user evaluation to examine users' performance for different shapes.

A common problem with designing and developing applications with tactile interfaces is the lack of a vocabulary that allows one to describe or communicate about haptics. Here we present the findings from a study exploring participants' verbalizations of their tactile experiences across two modulated tactile stimuli (16Hz and 250Hz) related to two important mechanoreceptors in the human hand. The study, with 14 participants, applied the explicitation interview technique to capture detailed descriptions of the diachronic and synchronic structure of tactile experiences. We propose 14 categories for a human-experiential vocabulary based on the categorization of the findings and tie them back to neurophysiological and psychophysical data on the human hand. We finally discuss design opportunities created through this experiential understanding in relation to the two mechanoreceptors.

We introduce the term shape resolution, which adds to the existing definitions of screen and touch resolution. We propose a framework, based on a geometric model (Non-Uniform Rational B-splines), which defines a metric for shape resolution in ten features. We illustrate it by comparing the current related work of shape changing devices. We then propose the concept of Morphees that are self-actuated flexible mobile devices adapting their shapes on their own to the context of use in order to offer better affordances. For instance, when a game is launched, the mobile device morphs into a console-like shape by curling two opposite edges to be better grasped with two hands. We then create preliminary prototypes of Morphees in order to explore six different building strategies using advanced shape changing materials (dielectric electro active polymers and shape memory alloys). By comparing the shape resolution of our prototypes, we generate insights to help designers toward creating high shape resolution Morphees.

We introduce UltraHaptics, a system designed to provide multi-point haptic feedback above an interactive surface. UltraHaptics employs focused ultrasound to project discrete points of haptic feedback through the display and directly on to users' unadorned hands. We investigate the desirable properties of an acoustically transparent display and demonstrate that the system is capable of creating multiple localised points of feedback in mid-air. Through psychophysical experiments we show that feedback points with different tactile properties can be identified at smaller separations. We also show that users are able to distinguish between different vibration frequencies of non-contact points with training. Finally, we explore a number of exciting new interaction possibilities that UltraHaptics provides.

Interactive tables provide multi-touch capabilities that can enable children to collaborate face-to-face simultaneously. In this paper we extend existing understanding of children's use of interactive tabletops by examining their use by school children in a school in Delhi, India. In the study, we explore how the school children exhibit particular types of collaboration strategies and touch input techniques when dealing with digital objects. In particular, we highlight a number of behaviours of interest, such as how the children would move the same digital object on the table together. We also discuss how the children work in close proximity to each other and dynamically organize their spatial positions in order to work together, as well as establish territory and control. We go on to examine some of the finger-based interaction and manipulation strategies that arise in these contexts. Finally, the paper considers the implications of such behaviours for the deployment of tabletop applications in these particular educational contexts.

Mobile projectors are gaining momentum, with many pocket sized products reaching the market and projector phones being close to production. Although the usefulness of such devices for entertainment, collaboration as well as many other tasks is obvious, it is not yet clear how their limitations in terms of image quality, brightness and jitter due to hand motion might affect performance. To answer these questions, we conduct two experiments based on traditional psychophysical methods. Using a visual search and a text reading study we compare task performance on a mobile phone display and a mobile projector. In addition, we examine whether performance is affected when devices are handheld rather than placed on a stable surface. We find that the perceived task difficulty is worse on the mobile projector cases for both tasks, while only in the visual search task leads to quantitatively worse performance. In contrast, we find that the stability of the projection plays no role in task performance but tasks performed on a handheld device may be perceived as harder to complete.

Foot-based gestures have recently received attention as an alternative interaction mechanism in situations where the hands are pre-occupied or unavailable. This paper investigates suitable real-world mappings of foot gestures to invoke commands and interact with virtual workspaces. Our first study identified user preferences for mapping common mobile-device commands to gestures. We distinguish these gestures in terms of discrete and continuous command input. While discrete foot-based input has relatively few parameters to control, continuous input requires careful design considerations on how the user's input can be mapped to a control parameter (e.g. the volume knob of the media player). We investigate this issue further through three user-studies. Our results show that rate-based techniques are significantly faster, more accurate and result if far fewer target crossings compared to displacement-based interaction. We discuss these findings and identify design recommendations.

We present MUSTARD, a multi-user dynamic random hole see-through display, capable of delivering viewer dependent information for objects behind a glass cabinet. Multiple viewers are allowed to observe both the physical object(s) being augmented and their location dependent annotations at the same time. The system consists of two liquid-crystal (LC) panels within which physical objects can be placed. The back LC panel serves as a dynamic mask while the front panel serves as the data. We first describe the principle of MUSTARD and then examine various functions that can be used to minimize crosstalk between multiple viewer positions. We compare different conflict management strategies using PSNR and the quality mean opinion score of HDR-VDP2. Finally, through a user-study we show that users can clearly identify images and objects even when the images are shown with strong conflicting regions; demonstrating that our system works even in the most extreme of circumstances.

Tangible objects placed on interactive surfaces allow users to employ a physical object to manipulate digital content. However, creating the reverse effect - having digital content manipulate a tangible object placed on the surface - is a more challenging task. We present a new approach to this problem, using ultrasound-based air pressure waves to move multiple tangible objects, independently, around an interactive surface. We describe the technical background, design, implementation, and test cases for such a system. We conclude by discussing practical uses of our system, Ultra-Tangibles, in the creation of new tangible user interfaces.

This paper examines the ability to detect a characteristic brain potential called the Error-Related Negativity (ERN) using off-the-shelf headsets and explores its applicability to HCI. ERN is triggered when a user either makes a mistake or the application behaves differently from their expectation. We first show that ERN can be seen on signals captured by EEG headsets like Emotiv™ when doing a typical multiple choice reaction time (RT) task -- Flanker task. We then present a single-trial online ERN algorithm that works by pre-computing the coefficient matrix of a logistic regression classifier using some data from a multiple choice reaction time task and uses it to classify incoming signals of that task on a single trial of data. We apply it to an interactive selection task that involved users selecting an object under time pressure. Furthermore the study was conducted in a typical office environment with ambient noise. Our results show that online single trial ERN detection is possible using off-the-shelf headsets during tasks that are typical of interactive applications. We then design a Superflick experiment with an integrated module mimicking an ERN detector to evaluate the accuracy of detecting ERN in the context of assisting users in interactive tasks. Based on these results we discuss and present several HCI scenarios for use of ERN.

We present a new type of actuatable display, called Tilt Displays, that provide visual feedback combined with multi-axis tilting and vertical actuation. Their ability to physically mutate provides users with an additional information channel that facilitates a range of new applications including collaboration and tangible entertainment while enhancing familiar applications such as terrain modelling by allowing 3D scenes to be rendered in a physical-3D manner. Through a mobile 3x3 custom built prototype, we examine the design space around Tilt Displays, categorise output modalities and conduct two user studies. The first, an exploratory study examines users' initial impressions of Tilt Displays and probes potential interactions and uses. The second takes a quantitative approach to understand interaction possibilities with such displays, resulting in the production of two user-defined gesture sets: one for manipulating the surface of the Tilt Display, the second for conducting everyday interactions.

We introduce spatially-aware virtual workspaces for the mobile environment. The notion of virtual workspaces was initially conceived to alleviate mental workload in desktop environments with limited display real-estate. Using spatial properties of mobile devices, we translate this approach and illustrate that mobile virtual workspaces greatly improve task performance for mobile devices. In a first study, we compare our spatially-aware prototype (mSpaces) to existing context switching methods for navigating amongst multiple tasks in the mobile environment. We show that users are faster, make more accurate decisions and require less mental and physical effort when using spatially-aware prototypes. We furthermore prototype pSpaces and m+pSpaces, two spatially-aware systems equipped with pico-projectors as auxiliary displays to provide dual-display capability to the handheld device. A final study reveals advantages of each of the different configurations and functionalities when comparing all three prototypes. Drawing on these findings, we identify design considerations to create, manipulate and manage spatially-aware virtual workspaces in the mobile environment.

We present PiVOT, a tabletop system aimed at supporting mixed-focus collaborative tasks. Through two view-zones, PiVOT provides personalized views to individual users while presenting an unaffected and unobstructed shared view to all users. The system supports multiple personalized views which can be present at the same spatial location and yet be only visible to the users it belongs to. The system also allows the creation of personal views that can be either 2D or (auto-stereoscopic) 3D images. We first discuss the motivation and the different implementation principles required for realizing such a system, before exploring different designs able to address the seemingly opposing challenges of shared and personalized views. We then implement and evaluate a sample prototype to validate our design ideas and present a set of sample applications to demonstrate the utility of the system.

We present anticipatory memory allocation (AMA), a new method to build kernel code that is robust to memory-allocation failures. AMA avoids the usual difficulties in handling allocation failures through a novel combination of static and dynamic techniques. Specifically, a developer, with assistance from AMA static analysis tools, determines how much memory a particular call into a kernel subsystem will need, and then preallocates said amount immediately upon entry to the kernel; subsequent allocation requests are serviced from the preallocated pool and thus guaranteed never to fail. We describe the static and runtime components of AMA, and then present a thorough evaluation of Linux ext2-mfr, a case study in which we transform the Linux ext2 file system into a memory-failure robust version of itself. Experiments reveal that ext2-mfr avoids memory-allocation failures successfully while incurring little space or time overhead.

This paper presents the findings of a user study investigating conversational patterns across three conditions of table-based interaction (direct touch interactive table, pantograph interactive table and non-digital table) for different types of educational activities. Findings demonstrate that communication style is significantly affected by interaction techniques. The direct touch technique stimulated conversations based around the topic and pedagogical method. The pantograph technique promoted playfulness and had a higher number of directive utterances between participants, with fewer task-based, group-oriented utterances. The non-digital table promoted reflective forms of task-orientated utterance, encouraged group communication and fostered more equitable participation between members. The findings provide insights into the design of interactive tables to support particular forms of social interaction.

In this paper we describe the use of kick gestures for interaction with mobile devices. Kicking is a well-st udied leg action that can be harnessed in mobile contexts where the hands are busy or too dirty to interact with the phone. In this paper we examine the design space of kicki ng as an interaction technique through two user studies. The first study investigated how well users were able to control the direction of their kicks. Users were able to aim their kicks best when the movement range is divided into segments of at least 24°. In the second study we looked at the velocity of a kick. We found that the users are able to kick with at least two varying velocities. However, they also often undershoot the target velocity. Finally, we propose some specific applications in which kicks can prove beneficial.

Projector phones, handheld game consoles and many other mobile devices increasingly include more than one display, and therefore present a new breed of mobile Multi-Display Environments (MDEs) to users. Existing studies illustrate the effects of visual separation between displays in MDEs and suggest interaction techniques that mitigate these effects. Currently, mobile devices with heterogeneous displays such as projector phones are often designed without reference to visual separation issues; therefore it is critical to establish whether concerns and opportunities raised in the existing MDE literature apply to the emerging category of Mobile MDEs (MMDEs). This paper investigates the effects of visual separation in the context of MMDEs and contrasts these with fixed MDE results, and explores design factors for Mobile MDEs. Our study uses a novel eye-tracking methodology for measuring switches in visual context between displays and identifies that MMDEs offer increased design flexibility over traditional MDEs in terms of visual separation. We discuss these results and identify several design implications.

Interactive displays are increasingly distributed in a broad spectrum of everyday life environments: They have very diverse form factors and portability characteristics, support a variety of interaction techniques, and can be used by a variable number of people. The coupling of multiple displays can thus create interactive "ecosystems" which mingle in the social context, and generate novel settings of communication, performance and ownership. The objective of this workshop is to focus on the range of research challenges and opportunities afforded by applications that rely on visual interfaces that can spread across multiple displays. Such displays are physically decoupled (i.e. connected to multiple computers) yet are visually coupled due to the interfaces and interactions they support. This can range from visual interfaces spread across multiple small private input displays (e.g. information exchange or game play) to small private displays coupled with larger public displays (e.g. public photo sharing).

Accelerometers are common on many devices, including those required for text-entry. We investigate how to enter text with devices that are solely enabled with accelerometers. The challenge of text-entry with such devices can be overcome by the careful investigation of the human limitations in gestural movements with accelerometers. Preliminary studies provide insight into two potential text-entry designs that purely use accelerometers for gesture recognition. In two experiments, we evaluate the effectiveness of each of the text-entry designs. The first experiment involves novice users over a 45 minute period while the second investigates the possible performance increases over a four day period. Our results reveal that a matrix-based text-entry system with a small set of simple gestures is the most efficient (5.4wpm) and subjectively preferred by participants.

Despite the best intentions of disk and RAID manufacturers, on-disk data can still become corrupted. In this paper, we examine the effects of corruption on database management systems. Through injecting faults into the MySQL DBMS, we find that in certain cases, corruption can greatly harm the system, leading to untimely crashes, data loss, or even incorrect results. Overall, of 145 injected faults, 110 lead to serious problems. More detailed observations point us to three deficiencies: MySQL does not have the capability to detect some corruptions due to lack of redundant information, does not isolate corrupted data from valid data, and has inconsistent reactions to similar corruption scenarios. To detect and repair corruption, a DBMS is typically equipped with an offline checker. Unfortunately, the MySQL offline checker is not comprehensive in the checks it performs, misdiagnosing many corruption scenarios and missing others. Sometimes the checker itself crashes; more ominously, its incorrect checking can lead to incorrect repairs. Overall, we find that the checker does not behave correctly in 18 of 145 injected corruptions, and thus can leave the DBMS vulnerable to the problems described above.

With gesture-based interactions in mobile settings becoming more popular, there is a growing concern regarding the social acceptance of these interaction techniques. In this paper we begin by examining the various definitions of social acceptance that have been proposed in the literature to synthesize a definition that is based on how the user feels about performing a particular interaction as well as how the bystanders perceive the user during this interaction. We then present the main factors that influence gestures' social acceptance including culture, time, interaction type and the user's position on the innovation adoption curve. Through a user study we show that an important factor in determining social acceptance of gesture-based interaction techniques is the user's perception of others ability to interpret the potential effect of a manipulation.

We introduce Membrane, a set of changes to the operating system to support restartable file systems. Membrane allows an operating system to tolerate a broad class of file system failures, and does so while remaining transparent to running applications; upon failure, the file system restarts, its state is restored, and pending application requests are serviced as if no failure had occurred. Membrane provides transparent recovery through a lightweight logging and checkpoint infrastructure, and includes novel techniques to improve performance and correctness of its fault-anticipation and recovery machinery. We tested Membrane with ext2, ext3, and VFAT. Through experimentation, we show that Membrane induces little performance overhead and can tolerate a wide range of file system crashes. More critically, Membrane does so with little or no change to existing file systems, thus improving robustness to crashes without mandating intrusive changes to existing file-system code.

Most studies on tilt based interaction can be classified as point-designs that demonstrate the utility of wrist-tilt as an input medium; tilt parameters are tailored to suit the specific interaction at hand. In this paper, we systematically analyze the design space of wrist-based interactions and focus on the level of control possible with the wrist. In a first study, we investigate the various factors that can influence tilt control, separately along the three axes of wrist movement: flexion/extension, pronation/supination, and ulnar/radial deviation. Results show that users can control comfortably at least 16 levels on the pronation/supination axis and that using a quadratic mapping function for discretization of tilt space significantly improves user performance across all tilt axes. We discuss the findings of our results in the context of several interaction techniques and identify several general design recommendations.

In order for anthropomorphic robots to communicate with people in a human-like way they need to produce gestures along with speech. Beat gestures are a key category of gestures, accompanying emphasis and timing of talk. The standard approach for autonomously adding gestures to speech for artificial agents to perform, is to identify when gestures should occur, and then select appropriate movements from a prewritten library. However, selecting naively from a library is unlikely to result in particularly human-like gesture sequences. In order to obtain examples of engaging human beat gestures, we studied videos of chat show hosts. Our analysis reveals rules for the creation of human-like beat gestures. We have combined these rules with hand scripted non-beat gestures, to produce a monologue with a complete set of accompanying gestures; it is designed for performance by our humanoid robot BERTI. In order to test the gestures produced we carried out a pilot study at the London Science Museum. The results of the study indicate that having correctly designed gesture sequences improves observer engagement.

The human sciences have demonstrated that gesture is a critical element of human communication. While existing graphical solutions are appropriate for virtual agents, solving arm trajectories for physically embodied robots requires that we consider the challenges of robot dynamics within a realtime gesture framework. We explore and evaluate a low computational-cost gesture production algorithm that can generate adequate gesture trajectories in a humanoid torso, as judged by participants in Human-Robot gesturing studies presented in this paper. Our approach produces a constrained inverse-kinematic solution for the start and end points, and generates appropriate wrist angles. Gesture time is used to calculate the joint accelerations to give a smooth, direct hand movement. Selecting open hand gestures as an example gesture sub-domain, we implement our controller on BERTI, a bespoke upper-torso humanoid robot (Fig. 2). A qualitative pilot study highlights gesture features salient to users: gesture shape, timing, naturalness and smoothness. A controlled experimental study then demonstrates that, by these metrics, our algorithm performs well; despite some dissimilarities with users' own gestures. We establish some salient points of robot gestures based on these studies.

Multi-display environments (MDEs) are now becoming common, and are becoming more complex, with more displays and more types of display in the environment. One crucial requirement specific to MDEs is that users must be able to move objects from one display to another; this cross-display movement is a frequent and fundamental part of interaction in any application that spans two or more display surfaces. Although many cross-display movement techniques exist, the differences between MDEs—the number, location, and mixed orientation of displays, and the characteristics of the task they are being designed for—require that interaction techniques be chosen carefully to match the constraints of the particular environment. As a way to facilitate interaction design in MDEs, we present a taxonomy that classifies cross-display object movement techniques according to three dimensions: the referential domain that determines how displays are selected, the relationship of the input space to the display configuration, and the control paradigm for executing the movement. These dimensions are based on a descriptive model of the task of cross-display object movement. The taxonomy also provides an analysis of current research that designers and researchers can use to understand the differences between categories of interaction techniques.

Multi-flick, which consists of repeated flick actions, has received popular media attention as an intuitive and natural document-scrolling technique for stylus based systems. In this paper we put multi-flick to test, by designing several flick-based scrolling techniques. We first map out the de-sign space of multi-flick and identify mapping functions that make multi-flick a natural and intuitive technique for document navigation. In the first experiment we compare several multi-flick variations for navigating lists on three different devices -- a PDA, a tabletPC, and a large table. Our study shows that compound-multi-flick (CMF) is the most preferred technique and it is at least as fast, if not faster than the traditional scrollbar. In a follow-up experiment, we evaluate multi-flick for scrolling text-based documents. Results show that all implementations of multi-flick are as good as the scrollbar for short distances while CMF is the most preferred. We discuss the implications of our findings and present several design guidelines.

Studies investigating user control of pressure input have reported time-accuracy trade-offs of, on average, over 30%, when interacting with a large number of pressure levels. To increase the level of control with pressure input, we designed and evaluated four different discretization functions: linear, fisheye, visual fisheye, and clustered. The fisheye discretization dynamically modifies the range of pressure values based on the position of the pressure cursor. Our results show that a fisheye function results in significantly lower error rates and a lower number of crossings than have been reported in the literature. Furthermore, the fisheye function improves control without compromising speed. We discuss the findings of our study and identify several design recommendations for integrating pressure control into common interface tasks.

In this paper we investigate the use of a uni-pressure and dual-pressure augmented mouse. With a pressure augmented mouse users can simultaneously control cursor positions as well as multiple levels of discrete selection modes for common desktop application tasks. Two or more independent pressure sensors can be mounted onto several locations on the body of the mouse. To highlight the design potential of a pressure augmented mouse we conducted a multi-part study. In the first part we identified the number of maximum discrete levels controllable with a uni-pressure augmented mouse, the most appropriate locations for installing pressure sensors on the mouse, and the design of new interaction techniques to support selection with pressure-based input. In a follow-up design we introduced an additional sensor and two different types of selection techniques to control a larger number of discrete levels with two pressure sensors. Our results show that users can comfortably control up to 64 modes with a dual-pressure augmented mouse. We discuss the findings of our results in the context of several desktop interaction techniques and identify several design recommendations.

Interaction techniques that utilize the space above the display surface to extend the functionalities of digitized surfaces continue to emerge. In such techniques, movements are constrained by the bounds of a layer. In addition, constraints imposed on the direction of movement within the layer may be present. Despite the presence of such techniques, there is limited understanding of human capabilities for performing the required steering task. In this paper we study and model user performance when steering through constrained and unconstrained paths in above-the-surface layers. Through a series of experiments we validate the derivation and applicability of our proposed models.

The windows and doorways that connect offices to public spaces are a site for people to gather awareness information and initiate interaction. However, these portals often reveal more information to the public area than the office occupant would like. As a result, people often keep doors and window blinds closed, which means that nobody can gather awareness information, even those with whom the occupant would be willing to share. One solution to this problem is a co-present media space - a computer-mediated video connection at the boundary between an office and a public area. These systems can provide both greater privacy control to the occupant and greater overall awareness information to observers. To see how co-present media spaces would work in real world settings, we built what we believe are the first ever co-present media spaces, and deployed them in two offices. From observations gathered over fifteen months, it is clear that the systems can do a better job of balancing the occupant's need for privacy and the observers' need for awareness better, than a standard window. However, we also identified a number of issues that affected the use and the success of the systems: the existence of alternate information sources, confusion with existing social norms, disparities between effort and need, and reduced interactional subtlety for observers in the public area. Our work contributes both a novel arrangement of a media space for co-present collaborators, and the first investigation into the design factors that affect the use and acceptance of these systems.

Multi-display environments compose displays that can be at different locations from and different angles to the user; as a result, it can become very difficult to manage windows, read text, and manipulate objects. We investigate the idea of perspective as a way to solve these problems in multi-display environments. We first identify basic display and control factors that are affected by perspective, such as visibility, fracture, and sharing. We then present the design and implementation of E-conic, a multi-display multi-user environment that uses location data about displays and users to dynamically correct perspective. We carried out a controlled experiment to test the benefits of perspective correction in basic interaction tasks like targeting, steering, aligning, pattern-matching and reading. Our results show that perspective correction significantly and substantially improves user performance in all these tasks.

Several researchers have recently proposed technology for crowd-and-DJ interactions in nightclub environments. However, these attempts have not always met with success. In order to design better technologies and systems in this area, it is important to start with an understanding of how nightclub interaction currently happens. To build this understanding, we carried out an interview study focusing on DJ-audience interactions. We interviewed eleven DJs from several different cities, and asked them to discuss the ways that they interact with the audience, and the ways that they maintain and use awareness of the audience. We found that DJs gather a wide variety of information about their audiences, and that this information is important to them as they plan and shape the evening's musical experience. DJs are adept at gathering visual information about the audience, despite poor lighting conditions and a heavy workload of selecting and mixing music. Despite the difficulties, DJs took a dim view of technology designed to let crowds exert more control over the music. This study is one of the first to look closely at the interactive relationship between the DJ and the nightclub audience through the lens of HCI, and our findings provide a number of guidelines for the design of new DJ-focused nightclub technologies.

The rapid increase in display sizes and resolutions has led to the re-emergence of many pen-based interaction systems like tabletop and wall display environments. Pointing in these environments is an important task, but techniques have not exploited the manipulation of control and display parameters to the extent seen in desktop environments. We have overcome these in the design of a new pen-based interaction technique -- Bubble Radar. Bubble Radar allows users to reach both specific targets and empty space, and supports dynamic switching between selecting and placing. The technique is based on combining the benefits of a successful pen-based pointing technique, the Radar View, with a successful desktop object pointing technique -- the Bubble Cursor. We tested the new technique in a user study and found that it was significantly faster than existing techniques, both for overall pointing and for targeting specific objects.

Designers and architects regularly use piles to organise visual artifacts. Recent efforts have now made it possible for users to create piles in digital systems as well. However, there is still little understanding of how users shouldinteract with digital piles. In this paper we investigate this issue. We first identify three tasks that must be supported by a digitalpile—navigation, reorganisation and repositioning. We then present three interaction techniques—called HoverDeck, DragDeck and ExpandPile that meet these requirements. The techniques allow users to easily browse the piles, and also allow them to move elements between and within piles in an ad-hoc manner. In a user study that compared the different interaction techniques, we found that ExpandPile was significantly faster than the other techniques over all tasks. There were differences, however, in individual tasks. We discuss the benefits and limitations of the different techniques and identify several situations where each of them could prove useful.

CIRRIN [3] is a stylus based text input technique for mobile devices with a touch sensitive display. In this paper we explore the benefit of expanding the letters of CIRRIN to reduce the overall difficulty of selecting a letter. We adapted the existing CIRRIN to expand the characters as the stylus approached it to create a new text entry technique called expanding CIRRIN. In a small user study we compared the standard CIRRIN and expanding CIRRIN for different sentences. Our results indicate that expanding CIRRIN increases error rates and text input times. We observed that expanding the letters often made the stylus enter the CIRRIN ring adjacent to the intended letter, thereby increasing error rates. We discuss the implications of these results, and possible applications of expanding targets with other text input techniques such as the Metropolis [7] soft keyboard.

Multi-display environments and smart meeting rooms are now becoming more common. These environments build a shared display space from variety of devices: tablets, projected surfaces, tabletops, and traditional monitors. Since the different display surfaces are usually not organized in a single plane, traditional schemes for stitching the displays together can cause problems for interaction. However, there is a more natural way to compose display space -- using perspective. In this paper, we develop interaction techniques for multi-display environments that are based on the user's perspective on the room. We designed the Perspective Cursor, a mapping of cursor to display space that appears natural and logical from wherever the user is located. We conducted an experiment to compare two perspective-based techniques, the Perspective Cursor and a beam-based technique, with traditional stitched displays. We found that both perspective techniques were significantly faster for targeting tasks than the traditional technique, and that Perspective Cursor was the most preferred method. Our results show that integrating perspective into the design of multi-display environments can substantially improve performance.

Interaction on digital tables has been restricted to a single layer on the table's active work-surface. We extend the design space of digital tables to include multiple layers of interaction. We leverage 3D position information of a pointing device to support interaction in the space above the active work-surface by creating multiple layers with drift-correction in which the user can interact with an application. We also illustrate through a point-design that designers can use multiple-layers to create a rich and clutter free application. A subjective evaluation showed that users liked the interaction techniques and found that, because of the drift correction we use, they could control the pointer when working in any layer.

Recent advances in multi-user collaboration have seen a proliferation of interaction techniques for moving digital objects from one device to another. However, little is known about how these techniques work in realistic situations, or how they compare to one another. We conducted a study to compare the efficiency of six techniques for moving objects from a tablet to a tabletop display. We compared the techniques in four different distance ranges and with three movement directions. We found that techniques like the Radar View and Pick-and-Drop, that have a control-to-display ratio of 1, are significantly faster for object movement than techniques that have smaller control-to-display ratios. We also found that using spatial manipulation of objects was faster than pressure-based manipulation.

Although telepointers are valuable for supporting real-time collaboration, they are rarely seen in commercial groupware applications that run on the Internet. One reason for their absence is that current telepointer implementations perform poorly on real-world networks with varying traffic, congestion, and loss. In this paper, we report on a new implementation of telepointers (HPT) that is designed to provide smooth, timely, and accurate telepointers in real-world groupware: on busy networks, on cable and dialup connections, and on wireless channels. HPT maintains performance at usable levels with a combination of techniques from multimedia and distributed systems research, including UDP transport, message compression, motion prediction, adaptive rate control, and adaptive forward error correction. Although these techniques have been seen before, they have never been combined and tailored to the specific requirements of telepointers. Tests of the new implementation show that HPT provides good performance in a number of network situations where other implementations do not work at all - we can provide usable telepointers even over a lossy 28K modem connection. HPT sets a new standard for telepointers, and allows designers to greatly improve the support that groupware provides for real-time interaction over distance.