The way users hold a smartphone depends on the interaction task, yet little is known about the fingers’ engagement with the device’s surfaces beyond the touchscreen. Such an understanding not only opens up opportunities for novel on- and off-screen interactions, but also the device’s possible physical affordances. We present a study (N=23) that examines the hands’ physical engagement with the smartphone beyond the touchscreen across nine mobile interactions. Grasps were annotated from photographs, and contact regions were captured using residual heat traces from grasping the device. Our findings show that fingers and palms adopt a variety of support roles and postures when engaging with the smartphone’s back and side edges. The hand-contact maps reveal distinct patterns, differing in contact frequency and placement. This work contributes an empirical characterisation of hands’ back and edge engagement, highlighting design opportunities for future smartphone usage extending beyond the touchscreen.

Haptic sensations in pen-based surface interactions within virtual reality (VR) rely on specialised haptic devices like pens or tablets. However, as VR becomes more mobile, reducing the need for external hardware is crucial, shifting the focus to head-mounted displays (HMDs) for haptic feedback. In this work, we explore head-mounted relocation of multisensory feedback, combining vibrations and audio, to simulate varying levels of surface roughness during pen-based surface interaction in VR. In a within-subject user study (N=18), we evaluated the influence of the combined technique compared to the vibration-only and audio-only techniques on the perceived surface roughness. Our results indicate that all relocation techniques allowed participants to distinguish at least three out of five roughness levels through their haptic and/or auditory senses. Moreover, while the combination of vibration and audio feedback produced the highest perceived levels of surface roughness for rougher simulated surfaces, the vibration-only approach enabled the perception of more distinguishable roughness levels in the lower range of simulated surfaces. Our findings provide early evidence that haptic surface sensations can be relocated via head-mounted simulation during pen-based interactions, offering initial insights for future HMDs that integrate multisensory feedback.

As the gaming industry advances, it seeks increased immersion. Concurrently, recent advancements in speech recognition substantially enhanced its accuracy, creating opportunities for speech-based interactions in gameplay. Integrating speech into single-player games with non-player characters (NPCs) introduces a new social dimension. Yet, like toxicity in social media and multiplayer games, anti-social behavior can infiltrate NPC interactions, generating a toxic gaming atmosphere. We designed a speech-based game called “A Day at the Office” and conducted a within-subject study with 26 participants to explore player interactions with NPCs possessing different human qualities. Furthermore, we introduced courtesy-based game mechanics, where NPCs react to players’ choice of words and loudness, investigating player experiences. Results revealed that participants spoke differently with NPCs based on their human qualities, such as gender and hierarchical roles. Moreover, the courtesy-based interactions fostered greater cognitive engagement, suggesting that tactful interactions contribute to enhanced immersion and presence in the game.