Computing interfaces are becoming increasingly sophisticated, with systems that engage multiple sensory channels simultaneously. Deformable and shape-changing interfaces offer rich tactile experiences, but there is limited understanding of how they can be combined with other modes of sensory feedback. We systematically explored the audio, visual and tactile cross-modal correspondences of deformable shapes with a particular focus on auditory feedback. 50 participants were asked to associate deformable tactile stimuli, varying in stiffness and shape, with the sound qualities pitch, brightness, fade-in time and fade-out time, under visuo-tactile and tactile-only conditions. Our findings provide the first insights on how (1) shape, both its form and visibility, play a significant role in associations for pitch and brightness; (2) stiffness plays a dominant role in associations over a sound's fade-in and fade-out times. These findings are distilled into the first design guidelines for integrating auditory feedback into physical interfaces.

Surfaces with deformable and shape-changing properties seek to enhance and diversify tangible interactions with computing systems. However, we currently lack fundamental knowledge and user interface design principles that connect the inherent properties of deformable shapes with our human senses and cognitive associations. To address this knowledge gap, we systematically explored deformable shapes’ cross-modal correspondences (CC) with colours and emotions. In our CC study, 52 participants were presented with deformable shape stimuli that varied in stiffness and angularity. They were asked to associate these stimuli with colours and emotions under (i) visuo-tactile and; (ii) tactile-only conditions. For the first time, our findings reveal (1) how stiffness level primarily influences the CC associations and; (2) that stiffness and angularity play a significant role in CC associations over the visibility of the shapes. The results were distilled into design guidelines for future deformable, shape-changing interfaces that engage specific human senses and responses.

Deformable interfaces provide unique interaction potential for force input, for example, when users physically push into a soft display surface. However, there remains limited understanding of which visual-tactile design elements signify the presence and stiffness of such deformable force-input components. In this paper, we explore how people correspond surface stiffness to colours, graphical shapes, and physical shapes. We conducted a cross-modal correspondence (CC) study, where 30 participants associated different surface stiffnesses with colours and shapes. Our findings evidence the CCs between stiffness levels for a subset of the 2D/3D shapes and colours used in the study. We distil our findings in three design recommendations: (1) lighter colours should be used to indicate soft surfaces, and darker colours should indicate stiff surfaces; (2) rounded shapes should be used to indicate soft surfaces, while less-curved shapes should be used to indicate stiffer surfaces, and; (3) longer 2D drop-shadows should be used to indicate softer surfaces, while shorter drop-shadows should be used to indicate stiffer surfaces.

What do pedestrian crossings, ATMs, elevators and ticket machines have in common? These are just a few of the ubiquitous yet essential elements of public-space infrastructure that rely on physical buttons or touchscreens; common interactions that, until recently, were considered perfectly safe to perform. This work investigates how we might integrate touchless technologies into public-space infrastructure in order to minimise physical interaction with shared devices in light of the ongoing COVID-19 pandemic. Drawing on an ethnographic exploration into how public utilities are being used, adapted or avoided, we developed and evaluated a suite of technology probes that can be either retrofitted into, or replace, these services. In-situ community deployments of our probes demonstrate strong uptake and provide insight into how hands-free technologies can be adapted and utilised for the public domain; and, in turn, used to inform the future of walk-up-and use public technologies.

Many of us daily encounter shadow and reflected light patterns alongside macro-level changes in ambient light levels. These are caused by elements—opaque objects, glass, mirrors, even clouds—in our environment interfacing with sunlight or artificial indoor lighting. Inspired by these phenomena, we explored ways of creating digitally-supported displays that use light, shade and reflection for output and harness the energy they need to operate from the sun or indoor ambient light. Through a set of design workshops we developed exemplar devices: SolarPix, ShadMo and GlowBoard. We detail their function and implementation, as well as evidencing their technical viability. The designs were informed by material understandings from the Global North and Global South and demonstrated in a cross-cultural workshop run in parallel in India and South Africa where community co-designers reflected on their uses and value given lived experience of their communication practices and unreliable energy networks.

Touchscreens are the predominant medium for interactions with digital services; however, their current fixed form factor narrows the scope for rich physical interactions by limiting interaction possibilities to a single, planar surface. In this paper we introduce the concept of PickCells, a fully re-configurable device concept composed of cells, that breaks the mould of rigid screens and explores a modular system that affords rich sets of tangible interactions and novel across-device relationships. Through a series of co-design activities -- involving HCI experts and potential end-users of such systems -- we synthesised a design space aimed at inspiring future research, giving researchers and designers a framework in which to explore modular screen interactions. The design space we propose unifies existing works on modular touch surfaces under a general framework and broadens horizons by opening up unexplored spaces providing new interaction possibilities. In this paper, we present the PickCells concept, a design space of modular touch surfaces, and propose a toolkit for quick scenario prototyping.

Digital painting is an increasingly popular medium of expression for many artists, yet when compared to its traditional equivalents of physical brushes and viscous paint it lacks a dimension of tangibility. We conducted observations and interviews with physical and digital artists, which gave us a strong understanding of the types of interactions used to create both physical and digital art, and the important role tangibility plays within these experiences. From this, we developed a unique liquid-like tangible display for mobile, digital colour mixing. Using a chemical hydrogel that changes its viscosity depending on temperature, we are able to create some resemblances to the feeling of mixing paint with a finger. This paper documents the information gathered from working with artists, how this process informed the development of a mobile painting attachment, and an exploration of its capabilities. After returning with our prototype, we found that it provided artists with sensations of oil and acrylic paint mixing and also successfully mimicked how paints are laid out on a paint palette.

My research explores the design and development of deformable controls and shape displays for mobile devices. My significant work so far has been set in the context of tools for creating digital art. I have been studying how we might design and develop deformable interfaces that support interaction with digital art applications. Along with how to bring elements of realistic painting to the digital experience through physical mobile controls. Part of this is designing and prototyping physical hardware interfaces and then evaluating them using HCI research methods in user studies and interviews. This extended abstract outlines my research aims, progress so far, and my future work and direction.