Visualization dashboards are regularly used for data exploration and analysis, but their complex interactions and interlinked views often require time-consuming onboarding sessions from dashboard authors. Preparing these onboarding materials is labor-intensive and requires manual updates when dashboards change. Recent advances in multimodal interaction powered by large language models (LLMs) provide ways to support self-guided onboarding. We present Diana (Dashboard Interactive Assistant for Navigation and Analysis), a multimodal dashboard assistant that helps users for navigation and guided analysis through chat, audio, and mouse-based interactions. Users can choose any interaction modality or a combination of them to onboard themselves on the dashboard. Each modality highlights relevant dashboard features to support user orientation. Unlike typical LLM systems that rely solely on text-based chat, Diana combines multiple modalities to provide explanations directly in the dashboard interface. We conducted a comparative qualitative user study to understand the use of different modalities for different types of onboarding tasks and their complexities.

Hybrid meetings have become common practice in collaborative work environments. However, they are constrained by the fixed spatial configurations of videoconferencing technology. This limits opportunities for mobile and spontaneous interactions; qualities that are critical to successful collaboration. In this paper, we explore the concept of radically mobile hybrid meetings. Our work investigates the design space of multimodal devices as mobile alternatives to traditional videoconferencing. We conducted three group co-design sessions, where participants prototyped mobile hybrid meeting technologies to explore how such meetings could be supported. From these workshops, we derive design fictions envisioning future uses of these technologies, which we evaluate with a questionnaire to spark reflections on future mobile hybrid collaboration tools and practices. We contribute an initial exploration of the design space for radically mobile hybrid meetings, laying the groundwork for developing tools that enable spontaneous, effective, and inclusive collaboration in hybrid mobile settings.

Backchannel communication, like whispering or instant messaging, is common in meetings and holds significant value. However, it remains largely unexplored in the context of hybrid meetings with co-located and remote participants. To address this gap, we derived unique challenges of backchannel communication in hybrid meetings through an interview study. These challenges inspired a new voice-based backchannel communication system, WhisperChannel, aimed to be inclusive and low effort. WhisperChannel enables users to whisper remotely to anyone in the meeting — similar to whispering to co-located seat neighbors. In a user study with three groups, each having two sessions of hybrid meetings, we investigated the inclusiveness and effort of WhisperChannel for both co-located and remote participants. We provide insights into the benefits and limitations of using remote whispering. We show that WhisperChannel helped remote and co-located participants feel more included while requiring low effort to use, however, also introduced new challenges in backchannel communication.

This paper introduces MouthIO, the first customizable intraoral user interface that can be equipped with various sensors and output components. MouthIO consists of an SLA-printed brace that houses a flexible PCB within a bite-proof enclosure positioned between the molar teeth and inner cheeks. Our MouthIO design and fabrication technique enables makers to customize the oral user interfaces in both form and function at low cost. All parts in contact with the oral cavity are made of bio-compatible materials to ensure safety, while the design takes into account both comfort and portability. We demonstrate MouthIO through three application examples ranging from beverage consumption monitoring, health monitoring, to assistive technology. Results from our full-day user study indicate high wearability and social acceptance levels, while our technical evaluation demonstrates the device’s ability to withstand adult bite forces.

The material qualities of technological mediators are vital to collaborative activity, but current paradigms for collaboration support leave the potential of material mediation largely untapped. The HCI literature addresses both material mediation for individuals and coordination between collaborators - but rarely does it discuss the more direct role of the material as standing between people: its communicative role. This paper unfolds a material/linguistic analysis of four empirical examples from previous work on collaborative writing to showcase how material qualities of both tools and the text-in-progress, used in both planned and improvised manners, help co-authors shift between levels of collaboration, from independent co-ordinated activity to highly collaborative co-constructive activity. On one hand, we see co-authors successfully collaborating through material means; on the other, we see frustrations resulting from limited material expressivity in current tools. This duality between the significance of material mediation and obstacles to drawing on its potential makes clear that our conception of materiality needs elaboration. We point to multimodality as one opportunity for this.

Designing new technologies to support synchronous interaction across distances has for many years focused on creating symmetry for participation between geographically distributed actors. Symmetry in synchronous interaction has, to some extent, been achieved technologically (while multiple social, historical, political, and hierarchical concerns continue to exist) and proven empirically in the increased use of remote-work technologies that were used during the pandemic. However, synchronous interaction in hybrid work is achieved differently, since the asymmetry produced by some participants being collocated while others geographically distributed introduces increased complexities for such interactions. Focusing on this challenge, we ask: To what extent can symmetry in cooperative work engagements be achieved in hybrid work contexts? We explore this question by interrogating multiple different empirical examples of synchronous hybrid interaction collected across different organizations, activities, and events. We found that the effort required to accomplish hybrid work includes additional articulation work necessary for bounding multiple intertwined artefacts across sites, devices, and applications. Further, the multiple artefacts setup across sites, combined with asymmetric collocation across participants, produce incongruence in technological frames of reference for each participant. All participants in hybrid work have only partial access to the hybrid setup, and no single person has access to the complete setup. The incongruence in technological frames produces insurmountable gaps in collaboration, causing all hybrid work situations to be characterized fundamentally by asymmetric relationships. We argue that symmetry in hybrid synchronous interaction is impossible to attain in attempts to solve this problem through design. Instead, we propose that designers of cooperative technologies for hybrid work shift towards developing artefact-ecologies supporting hybrid work, focusing on asymmetry as a necessary feature. Fundamentally, the design strategy should explore novel ways of taking advantage of the multiple different artefact-ecologies which serve as the foundation for the hybrid collaboration. Instead of striving for symmetry, we propose to feature asymmetric conditions in future technology designs for hybrid interaction.

Force feedback and shape change provide unique interaction qualities that can be favourable for the design of more intelligent physical user interfaces, as communication revolves to a large extent around body language and gestures. Currently, there is a dearth of design tools which are needed to accelerate exploration of the design space of haptic and shape change in user interfaces, given the challenges associated with the haptic modality and the expertise that is required for the development of such interfaces. Design tools have the potential to improve the accessibility of these modalities as design material within HCI and facilitate exploration of the design space. In this studio, we will use the design tool Feelix to explore possibilities and opportunities for force feedback and shape change in actuated interfaces.

How can we let users adapt video-based meetings as easily as they rearrange furniture in a physical meeting room? We describe a design space for video conferencing systems that includes a five-step “ladder of tailorability,” from minor adjustments to live reprogramming of the interface. We then present Mirrorverse and show how it applies the principles of computational media to support live tailoring of video conferencing interfaces to accommodate highly diverse meeting situations. We present multiple use scenarios, including a virtual workshop, an online yoga class, and a stand-up team meeting to evaluate the approach and demonstrate its potential for new, remote meetings with fluid transitions across activities.

This paper presents an evaluation of DynaKnob: a dynamic rotary knob that uses the relationship between physical shape and haptic force feedback to provide feedback about its functionality. It has the ability to morph into four distinct shapes whilst providing dynamic haptic force feedback. We explore how shape change and haptic force feedback can improve interactions by evaluating the DynaKnob in terms of usability, user experience and performance. Additionally, we look at how interaction techniques are influenced by these modalities. Results indicate that multimodal cues provided by haptic force feedback and different shapes did not improve usability, since the visual feedback provided by the GUI dominated the evaluated context. However, both modalities had a positive effect on the user experience, and showed potential to improve the performance in terms of accuracy for non-visual interaction.

In the last decade, haptic actuators have improved in quality and efficiency, enabling easier implementation in user interfaces. One of the next steps towards a mature haptics field is a larger and more diverse toolset that enables designers and novices to explore with the design and implementation of haptic feedback in their projects. In this paper, we look at several design projects that utilize haptic force feedback to aid interaction between the user and product. We analysed the process interaction designers went through when developing their haptic user interfaces. Based on our insights, we identified requirements for a haptic force feedback authoring tool. We discuss how these requirements are addressed by 'Feelix', a tool that supports sketching and refinement of haptic force feedback effects.

Despite the advantages of tangible interaction, physical controls like knobs seem to be disappearing from a wide range of products in our everyday life. The work presented in this paper explores how physical controls can become dynamic, in terms of both shape, and haptic force feedback. The paper contains two strands of work: First, we present a study that explores the relationship between haptic force feedback and different knob shapes, evaluating twelve distinct haptic stimuli in relation to six widely used knob shapes. Second, based on the insights collected in the study, we present the design of DynaKnob, a shape changing knob that can change between four different knob shapes. DynaKnob illustrates how dynamic content control, can be combined with dynamic shape and force feedback. Both the study and the design of DynaKnob contribute to understanding how adaptive physical interface controls could be designed in the future.

In this paper, we compare the use of physical and digital sticky notes in collaborative ideation. Inspired by a case study in a design company, we focus on a collaborative ideation task, which is often part of pair-wise brainstorming in design. For comparison and to focus on the different materiality, we developed a digital sticky notes setup designed to be as close to the physical setup as possible, not adding any advanced digital features, even though technology has reached a stage where more sophisticated use of digital sticky notes on digital boards is possible. In this paper, we present a study of ideation among pairs of experienced sticky note users. The ideation sessions were video recorded and analyzed to focus on how collaboration is supported across the two setups. Based on quantitative analyses of the participants’ interactions with the artefacts, talking patterns, position and attention during the sessions, we qualify how the differences and similarities between the 2 setups have an impact on note handling, ideation techniques, group dynamics and socio-spatial configuration, e.g. the use of the room, the boards and tables. We conclude that, while the physical setup seems more appropriate for creating notes and posting notes, the digital setup invites more note interaction. Nevertheless, we did not find significant differences in the ideation outcome (e.g., number of notes created) or how participants collaborated between the 2 setups. Hence, we argue that collaborative ideation can successfully be supported in a digital setup as well. Consequently, we believe that the next step in a technological setup is not an either or, but should bring the best of the two worlds together.

Drawing on 168 hours of video recordings of smartwatch use, this paper studies how context influences smartwatch use. We explore the effects of the presence of others, activity, location and time of day on 1,009 instances of use. Watch interaction is significantly shorter when in conversation than when alone. Activity also influences watch use with significantly longer use while eating than when socialising or performing domestic tasks. One surprising finding is that length of use is similar at home and work. We note that usage peaks around lunchtime, with an average of 5.3 watch uses per hour throughout a day. We supplement these findings with qualitative analysis of the videos, focusing on how use is modified by the presence of others, and the lack of impact of watch glances on conversation. Watch use is clearly a context-sensitive activity and in discussion we explore how smartwatches could be designed taking this into consideration.

Virtual collision techniques are interaction techniques for invoking discrete events in a virtual scene, e.g. throwing, pushing, or pulling an object with a pointer. The conventional approach involves detecting collisions as soon as the pointer makes contact with the object. Furthermore, in general, motor patterns can only be adjusted based on visual feedback. The paper presents a multimodal technique based on the principle that collisions should be aligned with the most salient sensory feedback. Boxer (1) triggers a collision at the moment where the pointer's speed reaches a minimum after first contact and (2) is synchronized with vibrotactile stimuli presented to the hand controlling the pointer. Boxer was compared with the conventional technique in two user studies (with temporal pointing and virtual batting). Boxer improved spatial precision in collisions by 26.7 % while accuracy was compromised under some task conditions. No difference was found in temporal precision. Possibilities for improving virtual collision techniques are discussed.

This paper describes the development and evaluation of Undulating Covers (UnCovers), mobile interfaces that can change their surface texture to transmit information. The Pin Array UnCover incorporates sinusoidal ridges controlled by servomotors, which can change their amplitude and granularity. The Mylar UnCover is a more organic interface that exploits the buckling properties of Mylar, using muscle wires, to change the texture granularity. The prototype development process and evaluations show that very low frequency texture changes, using amplitude or granularity, can be distinguished with high levels of accuracy. Since small changes are perceptible, it is possible to incorporate such interfaces into mobile devices without drastically increasing their size or actuation requirements. Finally, ratings from participants indicate that UnCovers would be appropriate for attention-grabbing, or caring and supportive interpersonal messages.

Virtual reality presents an extraordinary platform for multimodal communication. Haptic technologies have been shown to provide an important contribution to this by facilitating co-presence and allowing affective communication. However, the findings of the affective influences rely on studies that have used myriad different types of haptic technology, making it likely that some forms of tactile feedback are more efficient in communicating emotions than others. To find out whether this is true and which haptic technologies are most effective, we measured user experience during a communication scenario featuring an affective agent and interpersonal touch in virtual reality. Interpersonal touch was simulated using two types of vibrotactile actuators and two types of force feedback mechanisms. Self-reports of subjective experience of the agent’s touch and emotions were obtained. The results revealed that, regardless of the agent’s expression, force feedback actuators were rated as more natural and resulted in greater emotional interdependence and a stronger sense of co-presence than vibrotactile touch.

Rotations performed with the index finger and thumb involve some of the most complex motor action among common multi-touch gestures, yet little is known about the factors affecting performance and ergonomics. This note presents results from a study where the angle, direction, diameter, and position of rotations were systematically manipulated. Subjects were asked to perform the rotations as quickly as possible without losing contact with the display, and were allowed to skip rotations that were too uncomfortable. The data show surprising interaction effects among the variables, and help us identify whole categories of rotations that are slow and cumbersome for users.

This paper reports the results of an exploratory study into inadvertent grip pressure changes on mobile devices with a focus on the differences between static lab-based and mobile walking environments. The aim of this research is to inform the design of more robust pressure input techniques that can accommodate dynamic mobile usage. The results of the experiment show that there are significant differences in grip pressure in static and walking conditions with high levels of pressure variation in both. By combining the pressure data with accelerometer data, we show that grip pressure is closely related to user movement.

ForcePhone is a mobile synchronous haptic communication system. During phone calls, users can squeeze the side of the device and the pressure level is mapped to vibrations on the recipient's device. The pressure/vibrotactile messages supported by ForcePhone are called pressages. Using a lab-based study and a small field study, this paper addresses the following questions: how can haptic interpersonal communication be integrated into a standard mobile device? What is the most appropriate feedback design for pressages? What types of non-verbal cues can be represented by pressages? Do users make use of pressages during their conversations? The results of this research indicate that such a system has value as a communication channel in real-world settings with users expressing greetings, presence and emotions through pressages.

Touchscreens are becoming more and more popular, especially in mobile devices. There is also clear evidence of the benefits of tactile feedback in touchscreen interaction. However, the effect of the evident latency in interaction has been completely neglected in earlier investigations of touchscreen interaction. In this study we examined the effect of tactile feedback latencies on the usability of a touchscreen keypad. We used a realistic use case for number and character keypads; users entered three-number sequences and short sentences using the virtual buttons on the touch display. The experiments differed from each other in terms of the tactile feedback type (press-only or for press and release) and the keypad layout (number or QWERTY). The results were unexpected, but consistent in all three experiments: The performance did not drop significantly within the latency values used. However, the users evaluated the keypad with the shortest feedback latency more pleasant to use compared to others. We can conclude that latency makes the user experience worse, even though performance does not decrease significantly.

We report the results of an exploratory 8-day field study of CrossTrainer: a mobile game with crossmodal audio and tactile feedback. Our research focuses on the longitudinal effects on performance with audio and tactile feedback, the impact of context such as location and situation on performance and personal modality preference. The results of this study indicate that crossmodal feedback can aid users in entering answers quickly and accurately using a variety of different widgets. Our study shows that there are times when audio is more appropriate than tactile and vice versa and for this reason devices should support both tactile and audio feedback to cover the widest range of environments, user preference, locations and tasks.

When designing interfaces for mobile devices it is import-ant to take into account the variety of contexts of use. We present a study that examines how changing noise and dis-turbance in the environment affects user performance in a touchscreen typing task with the interface being presented through visual only, visual and tactile, or visual and audio feedback. The aim of the study is to show at what exact environmental levels audio or tactile feedback become inef-fective. The results show significant decreases in perform-ance for audio feedback at levels of 94dB and above as well as decreases in performance for tactile feedback at vibration levels of 9.18g/s. These results suggest that at these levels, feedback should be presented by a different modality. These findings will allow designers to take advantage of sensor enabled mobile devices to adapt the provided feed-back to the user's current context.

We report the results of a study focusing on the meanings that can be conveyed by audio and tactile icons. Our research considers the following question: how can audio and tactile icons be designed to optimise congruence between crossmodal feedback and the type of information this feedback is intended to convey? For example, if we have a set of system warnings, confirmations, progress up-dates and errors: what audio and tactile representations best match the information or type of message? Is one modality more appropriate at presenting certain types of information than the other modality? The results of this study indicate that certain parameters of the audio and tactile modalities such as rhythm, texture and tempo play an important role in the creation of congruent sets of feedback when given a specific type of information to transmit. We argue that a combination of audio or tactile parameters derived from our results allows the same type of information to be derived through touch and sound with an intuitive match to the content of the message.

This paper presents a study of finger-based text entry for mobile devices with touchscreens. Many devices are now coming to market that have no physical keyboards (the Apple iPhone being a very popular example). Touchscreen keyboards lack any tactile feedback and this may cause problems for entering text and phone numbers. We ran an experiment to compare devices with a physical keyboard, a standard touchscreen and a touchscreen with tactile feedback added. We tested this in both static and mobile environments. The results showed that the addition of tactile feedback to the touchscreen significantly improved finger-based text entry, bringing it close to the performance of a real physical keyboard. A second experiment showed that higher specification tactile actuators could improve performance even further. The results suggest that manufacturers should use tactile feedback in their touchscreen devices to regain some of the feeling lost when interacting on a touchscreen with a finger.

Our research considers the following question: how can visual, audio and tactile feedback be combined in a congruent manner for use with touchscreen graphical widgets? For example, if a touchscreen display presents different styles of visual buttons, what should each of those buttons feel and sound like? This paper presents the results of an experiment conducted to investigate methods of congruently combining visual and combined audio/tactile feedback by manipulating the different parameters of each modality. The results indicate trends with individual visual parameters such as shape, size and height being combined congruently with audio/tactile parameters such as texture, duration and different actuator technologies. We draw further on the experiment results using individual quality ratings to evaluate the perceived quality of our touchscreen buttons then reveal a correlation between perceived quality and crossmodal congruence. The results of this research will enable mobile touchscreen UI designers to create realistic, congruent buttons by selecting the most appropriate audio and tactile counterparts of visual button styles.

Mobile device user interface elements tend to be based on desktop widgets that were not originally intended for small screen finger-based interfaces. Mobile usage scenarios afford many completely different interactions, so should be designed accordingly. This paper presents a new type of widget, the T-Bar, designed specifically for finger-based touchscreen interfaces using tactile feedback. Using the tactile feedback for orientation, the user's fingertip is guided along the T-Bar until an item is successfully selected. This paper offers observations on our finger-based touchscreen widget and two applications of the T-Bar widget. Both, File-o-Feel and Touch 'n' Twist are multi-touch information browsing applications that deviate from traditional desktop GUI paradigms and are tailored for fingertip input where all interaction takes place through the use of T-Bars; eliminating the need for any other widgets.

This paper reports an experiment into the design of crossmodal icons which can provide an alternative form of output for mobile devices using audio and tactile modalities to communicate information. A complete set of crossmodal icons was created by encoding three dimensions of information in three crossmodal auditory/tactile parameters. Earcons were used for the audio and Tactons for the tactile crossmodal icons. The experiment investigated absolute identification of audio and tactile crossmodal icons when a user is trained in one modality and tested in the other (and given no training in the other modality) to see if knowledge could be transferred between modalities. We also compared performance when users were static and mobile to see any effects that mobility might have on recognition of the cues. The results showed that if participants were trained in sound with Earcons and then tested with the same messages presented via Tactons they could recognize 85% of messages when stationary and 76% when mobile. When trained with Tactons and tested with Earcons participants could accurately recognize 76.5% of messages when stationary and 71% of messages when mobile. These results suggest that participants can recognize and understand a message in a different modality very effectively. These results will aid designers of mobile displays in creating effective crossmodal cues which require minimal training for users and can provide alternative presentation modalities through which information may be presented if the context requires.

This paper describes an alternative form of interaction for mobile devices using crossmodal output. These crossmodal displays allow alternative senses such as hearing and touch to be used to perceive information normally presented to the visual modality. Initial experiments show that roughness and spatial location can be perceived as equivalent in both the auditory and tactile domain. This paper discusses how crossmodal displays can be constructed using the results from these experiments and the benefits they bring to mobile human computer interfaces