Foundation models are rapidly improving the capability of robots in performing everyday tasks autonomously such as meal preparation, yet robots will still need to be instructed by humans due to model performance, the difficulty of capturing user preferences, and the need for user agency. Robots can be instructed using various methods-natural language conveys immediate instructions but can be abstract or ambiguous, whereas end-user programming supports longer-horizon tasks but interfaces face difficulties in capturing user intent. In this work, we propose using direct manipulation of images as an alternative paradigm to instruct robots, and introduce a specific instantiation called ImageInThat which allows users to perform direct manipulation on images in a timeline-style interface to generate robot instructions. Through a user study, we demonstrate the efficacy of ImageInThat to instruct robots in kitchen manipulation tasks, comparing it to a text-based natural language instruction method. The results show that participants were faster with ImageInThat and preferred to use it over the text-based method. Supplementary material including code can be found at: https://image-in-that.github.io/.

Voice assistants have afforded users rich interaction opportunities to access information and issue commands in a variety of contexts. However, some users feel uneasy or creeped out by voice assistants, leading to a decreased desire to use them. As there has yet to be a comprehensive understanding of the factors that cause users to perceive voice assistants as being creepy, this research developed an empirical scale to measure the creepiness inherent in various voice assistants. Utilizing prior scale creation methodologies, a 7-item Perceived Creepiness of Voice Assistants Scale (PCAS) was created and validated. The scale measures how creepy a new voice assistant would be for users of voice assistants. The scale was developed to ensure that researchers and designers can evaluate the next generation of voice assistants before such voice assistants are released to the wider public.

Studies in psychology have shown that framing effects, where the positive or negative attributes of logically equivalent choices are emphasised, influence people's decisions. When outcomes are uncertain, framing effects also induce patterns of choice reversal, where decisions tend to be risk averse when gains are emphasised and risk seeking when losses are emphasised. Studies of these effects typically use potent framing stimuli, such as the mortality of people suffering from diseases or personal financial standing. We examine whether these effects arise in users' decisions about interface features, which typically have less visceral consequences, using a crowd-sourced study based on snap-to-grid drag-and-drop tasks (n = 842). The study examined several framing conditions: those similar to prior psychological research, and those similar to typical interaction choices (enabling/disabling features). Results indicate that attribute framing strongly influences users' decisions, that these decisions conform to patterns of risk seeking for losses, and that patterns of choice reversal occur.

We introduce a new shortcut interface called KeyMap that is designed to leverage Norman's principle of natural mapping. Rather than displaying shortcut command labels in linear menus, KeyMap displays a virtual keyboard with command labels displayed directly on its keys. A crowdsourced experiment compares KeyMap to Malacria et al.'s ExposeHK using an extension of their protocol to also test recall. Results show KeyMap users remembered 1 more shortcut than ExposeHK immediately after training, and this advantage increased to 4.5 more shortcuts when tested again after 24 hours. KeyMap users also incidentally learned more shortcuts that they had never practised. We demonstrate how KeyMap can be added to existing web-based applications using a Chrome extension.

Rehearsal-based interfaces are designed to encourage a transition from novice to expert, but many users fail to make this transition. Most of these interfaces activate novice mode after a short delay, between 150 and 500 ms. We investigate the impact of delay time on expert usage and learning in three crowdsourced experiments. The first experiment examines an 8-item marking menu with delay times from 200 ms to 2 s. Results show longer delays increase successful expert selections. The second and third experiments generalise this result to a different rehearsal-based menu, a desktop clone of FastTap with 8 items and 15 items. Together, our results show that expert use correlates positively with increased delay time, but can increase errors since users are less risk averse. We also find imperceptible delays of 200 ms can harm long-term retention of menu items. Designers should consider longer delays in rehearsal-based interfaces to encourage a transition to expert usage.

Expert interaction techniques like hotkeys are efficient, but poorly adopted because they are hard to learn. HotStrokes removes the need for learning arbitrary mappings of commands to hotkeys. A user enters a HotStroke by holding a modifier key, then gesture typing a command name on a laptop trackpad as if on an imaginary virtual keyboard. The gestures are recognized using an adaptation of the SHARK2 algorithm with a new spatial model and a refined method for dynamic suggestions. A controlled experiment shows HotStrokes effectively augments the existing "menu and hotkey" command activation paradigm. Results show the method is efficient by reducing command activation time by 43% compared to linear menus. The method is also easy to learn with a high adoption rate, replacing 91% of linear menu usage. Finally, combining linear menus, hotkeys, and HotStrokes leads to 24% faster command activation overall.

Keyboard shortcuts can be more efficient than graphical input, but they are underused by most users. To alleviate this, we present "Guided Finger-Aware Shortcuts" to reduce the gulf between graphical input and shortcut activation. The interaction technique works by recognising when a special hand posture is used to press a key, then allowing secondary finger movements to select among related shortcuts if desired. Novice users can learn the mappings through dynamic visual guidance revealed by holding a key down, but experts can trigger shortcuts directly without pausing. Two variations are described: FingerArc uses the angle of the thumb, and FingerChord uses a second key press. The techniques are motivated by an interview study identifying factors hindering the learning, use, and exploration of keyboard shortcuts. A controlled comparison with conventional keyboard shortcuts shows the techniques encourage overall shortcut usage, make interaction faster, less error-prone, and provide advantages over simply adding visual guidance to standard shortcuts.