Systems’ latency – the time between user input and system response – slows down the human-computer interaction loop. Several studies revealed negative objective and subjective effects of high latency, typically treating latency as a constant delay. Because latency varies significantly in practice, recent work also assessed the effects of large and sudden latency changes. In practice, however, latency variations are small but frequent. As the effects of such variations are unclear, we investigate how small latency variations (± 50 ms) affect users’ performance and perceived task load for 2D target selection tasks with static and moving targets. For static targets, we found that latency variation causes significantly higher completion times and less efficient trajectories, however with small effect sizes. In contrast, we found no significant effects on any performance measure for moving targets. Our findings indicate that the effect of latency variation is generally very small and quickly disappears for non-trivial tasks.

Virtual reality enables immersion in any environment. According to the hue-heat hypothesis, an environment’s color temperature influences thermal perception and regulation, factors important for productivity and energy consumption. Previous work suggests that visual thermal cues, e.g., snowy or desert environments, affect thermal perception and regulation. As color temperature was not controlled, it is unknown if hue or thermal cues caused the effects. In our first study, we derived two virtual environments with cold or warm thermal cues. In the second study, participants experienced these environments with varying hues for 5 minutes. We found that hue influences thermal sensation and thermal cues influence sensation, comfort, and skin temperature. As these effects seem to increase beyond 5 minutes, a third study with longer exposure and more extreme hue differences confirmed and extended these findings. Overall, our results consistently show that hue and visual thermal cues independently affect thermal perception and regulation. This highlights the importance of considering both when designing virtual environments.

Virtual reality (VR) allows to embody avatars. Coined the Proteus effect, an avatar’s visual appearance can influence users’ behavior and perception. Recent work suggests that athletic avatars decrease perceptual and physiological responses during VR exercise. However, such effects can fail to occur when users do not experience avatar ownership and identification. While customized avatars increase body ownership and identification, it is unclear whether they improve the Proteus effect. We conducted a study with 24 participants to determine the effects of athletic and non-athletic avatars that were either customized or randomly assigned. We developed a customization editor to allow creating customized avatars. We found that customized avatars reduced perceived exertion. We also found that athletic avatars decreased heart rate while holding weights, however, only when being customized. Results indicate that customized avatars can positively influence users during physical exertion. We discuss the utilization of avatar customization in VR exercise systems.

Touch-sensitive surfaces offer an intuitive and flexible form of interaction, for example through gesture input. Despite being the primary input modality for mobile devices, they hardly find application in desktop settings. At the same time, the computer mouse is still the most efficient and accurate input device for pointing. Consequently, keeping the unmatched functionality of a mouse but extending it with new input options via a touch-sensitive surface, is a promising approach. While research prototypes and niche products for multi-touch mice exist, the concept has not yet become established. In this work, we follow a user-centered approach towards touch interaction on computer mice. In a user study (n=12), we identified which areas on the mouse are suitable for touch input. Further, we explored potential usage scenarios in a diary study (n=11) and intuitive gestures in an elicitation study (n=10). We compile our findings into a gesture set which future research can build upon to implement touch interaction on computer mice.

Perceived temperature and thermoregulation are not only influenced by actual temperature but also by visual perception. For instance, the hue-heat hypothesis posits that warm-colored lighting can increase perceived temperature. Recent work indicates that embodying avatars with visual cues suggesting extreme temperatures influences thermal perception and thermoregulation. However, recent work is limited by short exposure times, inconsistent temperature effects, and the absence of a baseline comparison. Thus, we conducted a study where participants embodied ice, neutral, and fire hands in virtual reality for 15 minutes. We show that thermal sensation is significantly higher with fire hands compared to ice hands, but found no consistent effects on skin temperature. As the effects on thermal sensation remain consistent at least over 15 minutes, we conclude that avatars’ appearance can be used to systematically influence users’ thermal perception.

Designers of virtual characters use facial expressions to display avatar emotions. Previous work revealed connections between emotions and thermoregulatory, e.g., happiness increases and sadness decreases skin temperature. As virtual reality (VR) enables embodying avatars with any facial expression, it is unclear whether embodying avatars with different emotions also affect users’ thermophysiological and emotional responses. We conducted a study with 24 participants who embodied customized avatars displaying happy, neutral, and sad facial expressions in VR. We found that participants’ skin temperature was higher with sad avatars than with happy ones while resting. Despite non-significant pairwise comparisons, descriptive statistics suggest an opposite trend for skin temperature responses during grabbing interactions. We also show that participants felt happier while embodying happy avatars compared to sad avatars. Results indicate that avatars’ emotions impact users’ thermoregulation and affective states. We discuss underlying mechanisms and potential applications.

Simulating accurate weight forces in Virtual Reality (VR) is an unsolved challenge. Therefore, providing real weight sensations by transferring liquid mass has emerged as a promising approach. However, key objectives conceptually interfere with each other. In particular, previous designs that support a high range of weight or high flow rate lack mobility. In this work, we present MobileGravity, a system, that decouples the weight-changing object from the liquid supply and the pump. It enables weight changes of up to 1 kg at a rate of 235 g/s and allows the user to walk around freely. Through a study with 30 participants, we show that the system enables users to perceive the weight of different virtual objects and enhances realism, as well as enjoyment.

Weight interfaces enable users of Virtual Reality (VR) to perceive the weight of virtual objects, significantly enhancing realism and enjoyment. While research on these systems primarily focused on their implementation, little attention has been given to determining the weight to be rendered by them: As the perceived weight of objects is influenced not only by their absolute mass, but also by their weight distribution and prior expectations, it is currently unknown which simulated mass provides the most realistic representation of a given object. We conducted a study, in which 30 participants chose the best fitting weight for a virtual object in 54 experimental trials. Across these trials, we systematically varied the virtual objects’ visual mass (three levels), their weight distribution (six levels), and the position of the physical mass on the grip (three levels). Our Bayesian analysis suggests that the visual weight distribution of objects does not affect which absolute physical mass best represents them, whereas the position of the provided physical mass does. Additionally, participants overweighted virtual objects with lower visual mass while underweighting objects with higher visual mass. We discuss how these findings can be leveraged by designers of weight interfaces and VR experiences to optimize realism.

Perceiving objects’ and avatars’ weight in Virtual Reality (VR) is important to understand their properties and naturally interact with them. However, commercial VR controllers cannot render weight. Controllers presented by previous work are single-handed, slow, or only render a small mass. In this paper, we present PumpVR that renders weight by varying the controllers’ mass according to the properties of virtual objects or bodies. Using a bi-directional pump and solenoid valves, the system changes the controllers’ absolute weight by transferring water in or out with an average error of less than 5%. We implemented VR use cases with objects and avatars of different weights to compare the system with standard controllers. A study with 24 participants revealed significantly higher realism and enjoyment when using PumpVR to interact with virtual objects. Using the system to render body weight had significant effects on virtual embodiment, perceived exertion, and self-perceived fitness.

Humans’ thermal regulation and subjective perception of temperature is highly plastic and depends on the visual appearance of the surrounding environment. Previous work shows that an environment’s color temperature affects the experienced temperature. As virtual reality (VR) enables visual immersion, recent work suggests that a VR scene’s color temperature also affects experienced temperature. It is, however, unclear if an avatar’s appearance also affects users’ thermal perception and if a change in thermal perception even influences the body temperature. Therefore, we conducted a study with 32 participants performing a task in an ice or fire world while having ice or fire hands. We show that being in a fire world or having fire hands increases the perceived temperature. We even show that having fire hands decreases the hand temperature compared to having ice hands. We discuss the implications for the design of VR systems and future research directions.

Temporal regularities and the timing of events and actions such as anticipating enemy movements or planning one’s next move are essential components of almost every video game. Thus, to succeed in video games, it is advantageous to anticipate events and prepare relevant actions before they occur. This work explores whether elapsed time can be used as a predictive cue for implicitly anticipating events in video games. Inspired by findings from psychology, we implemented multiple time-event correlations in a custom video game by pairing specific delays with specific game events. Participants had to shoot targets that appeared at different locations. After a certain delay (e.g., 0.8 s), the targets appeared more frequently (80 % of all appearances) at a specific location (e.g., left up). Our analysis of 25 participants provides evidence that players implicitly learned the implemented time-event correlations and used them to anticipate the location of upcoming targets. This led to improved game performance. Although no participant realised the implemented temporal regularities, targets were shot faster when preceded by the frequently paired delay. Our findings pave the way for game developers and researchers alike to more creatively combine human temporal processing with temporal aspects of video games.

Virtual reality (VR) allows to embody avatars—the digital self-representation of the user. An avatar’s appearance can change users’ perception and behavior—a phenomenon known as the Proteus effect. Previous work found that athletic avatars can reduce heart rate and perceived exertion during physical effort. Although these findings are promising to create more effective VR exercises, they have not been replicated yet. Hence, the reliability and consistency of such effects is unknown. Therefore, we conducted a study with 32 participants to investigate physiological and perceptual effects of athletic avatars while cycling in VR following a standardized exercise protocol. We could not find effects of the avatars’ athletic appearance on heart rate, perceived exertion, and imagined velocity. Our findings indicate that athletic avatars do not necessarily affect users during physical exertion. We discuss potential factors that can cause the Proteus effect fail to occur.

In interactive systems high latency affects user performance and experience. This is especially problematic in video games. A large number of studies on this topic investigated the effects of constant, high latency. However, in practice, latency is never constant but varies by up to 100 ms due to variations in processing time and delays added by polling between system components. In a large majority of studies, these variations in latency are neither controlled for nor reported. Thus, it is unclear to which degree small, continuous variations in latency affect user performance. If these unreported variations had a significant impact, this might cast into doubt the findings of some studies. To investigate how latency variation affects player performance and experience in games, we conducted an experiment with 28 participants playing a first-person shooter. Participants played with two levels of base latency (50 ms vs. 150 ms) and variation (0 ms vs. 50 ms). As expected, high base latency significantly reduces player performance and experience. However, we found strong evidence that small variations in latency in the order of 50 ms, do not affect player performance significantly. Thus, our findings mitigate concerns that previous latency studies might have systematically ignored a confounding effect.

Previous work shows that high latency, a prolonged delay between player in- and system output, negatively affects player experience and performance. However, previous work also comes to contrary conclusions about how the in-game perspective alters the latency sensitivity of video games. Currently, it is unclear if the in-game perspective independently modulates latency's effects. To investigate how a game's in-game perspective interacts with latency, we developed a shooting game incorporating three perspectives (First-Person-, Third-Person-, and Bird's-Eye-View). In a study, participants (N = 36) played with two levels of latency (low and high) and the three perspectives. We show that latency reduces performance and experience, independent of the perspective. Moreover, Bayesian analysis suggests that the in-game perspective does not interact with latency and does not affects performance or experience. We conclude that more robust means to categorize latency sensitivity of video games than the in-game perspective are required.

Users avoid engaging with privacy policies because they are lengthy and complex, making it challenging to retrieve relevant information. In response, research proposed contextual privacy policies (CPPs) that embed relevant privacy information directly into their affiliated contexts. To date, CPPs are limited to concept showcases. This work evolves CPPs into a production tool that automatically extracts and displays concise policy information. We first evaluated the technical functionality on the US’s 500 most visited websites with 59 participants. Based on our results, we further revised the tool to deploy it in the wild with 11 participants over ten days. We found that our tool is effective at embedding CPP information on websites. Moreover, we found that the tool’s usage led to more reflective privacy behavior, making CPPs powerful in helping users understand the consequences of their online activities. We contribute design implications around CPP presentation to inform future systems design.

Privacy policies (PPs) are currently the only way to inform users about their rights and choices during web browsing and searching. However, users avoid engaging with them, because of their length and abstract legal language, which makes them hard to read and understand. We propose to support the understanding of PPs by using modular icons. Icons have already proven to be helpful in visualizing concepts with high information density. However, the value of using icons to supplement PPs lacks a scientific foundation. Thus, we conducted two studies to evaluate existing icon sets for their understandability and to teach participants their meaning in situ. We show that modular privacy icons can be taught using our process, which has the potential to aid quicker and easier comprehension of PPs. We contribute a set of tested modular privacy icons and a verified process on how to teach them to users incidentally during web browsing.

Users’ expectations about systems alter how they interact with them, thereby influencing their experience. Latency is also known to alter experience and performance in interactive systems, particularly in video games. Currently, it is unclear if users’ expectations influence latency-based effects. We report the results of an experiment (N = 24) in which participants played a video game with four levels of phantom latency (30 ms, 60 ms, 90 ms, and 120 ms). Crucially, all rounds were played with 75 ms of actual latency, we merely changed its presentation in the game. We show that a high phantom latency reduces game experience and performance. Participants were least accurate and effective when playing with 120 ms while feeling the least competent and tensest. We concluded that the effects of latency are partly expectation-based. Latency researchers, developers, and gamers must be aware of the effects induced by the expectation of latency.

Avatars are used to represent users in virtual reality (VR) and create embodied experiences. Previous work showed that avatars’ stereotypical appearance can affect users’ physical performance and perceived exertion while exercising in VR. Although sweating is a natural human response to physical effort, surprisingly little is known about the effects of sweating avatars on users. Therefore, we conducted a study with 24 participants to explore the effects of sweating avatars while cycling in VR. We found that visualizing sweat decreases the perceived exertion and increases perceived endurance. Thus, users feel less exerted while embodying sweating avatars. We conclude that sweating avatars contribute to more effective exergames and fitness applications.

Feeling ownership of a virtual body is crucial for immersive experiences in VR. Knowledge about body ownership is mainly based on rubber hand illusion (RHI) experiments in the real world. Watching a rubber hand being stroked while one’s own hidden hand is synchronously stroked, humans experience the rubber hand as their own hand and underestimate the distance between the rubber hand and the real hand (proprioceptive drift). There is also evidence for a decrease in hand temperature. Although the RHI has been induced in VR, it is unknown whether effects in VR and the real world differ. We conducted a RHI experiment with 24 participants in the real world and in VR and found comparable effects in both environments. However, irrespective of the RHI, proprioceptive drift and temperature differences varied between settings. Our findings validate the utilization of the RHI in VR to increase our understanding of embodying virtual avatars.

Latency is inherently part of every interactive computing system and particularly important for video games. Previous work shows that constant latency above 25 ms reduces game experience and player performance. However, latency in the wild varies and is never constant due to multiple factors, such as updates in routing tables, users changing their location, or the system's workload. It is unclear if switching latency impairs the gaming experience stronger than a constant high latency. To elucidate, we conducted an experiment with 264 participants playing a shooting video game induced with 0 ms, 33 ms, and 66 ms controlled latency. While playing, the game switched between different latency levels based on three frequencies. Our analysis shows that switching latency significantly impaired the participants' flow. Additionally, we found effects on the perceived tension, the experienced challenge, and the players' performance. We conclude that games should prioritize constant latency, even if that entails artificially adding latency.

Avatars in virtual reality (VR) enable embodied experiences and induce the Proteus effect—a shift in behavior and attitude to mimic one’s digital representation. Previous work found that avatars associated with physical strength can decrease users’ perceived exertion when performing physical tasks. However, it is unknown if an avatar’s appearance can also influence the user’s physiological response to exercises. Therefore, we conducted an experiment with 24 participants to investigate the effect of avatars’ athleticism on heart rate and perceived exertion while cycling in VR following a standardized protocol. We found that the avatars’ athleticism has a significant and systematic effect on users’ heart rate and perceived exertion. We discuss potential moderators such as body ownership and users’ level of fitness. Our work contributes to the emerging area of VR exercise systems.

Reading is a fundamental activity to obtain information both in the real and the digital world. Virtual reality (VR) allows novel approaches for users to view, read, and interact with a text. However, for efficient reading, it is necessary to understand how a text should be displayed in VR without impairing the VR experience. Therefore, we conducted a study with 18 participants to investigate text presentation type and location in VR. We compared world-fixed, edge-fixed, and head-fixed text locations. Texts were displayed using Rapid Serial Visual Presentation (RSVP) or as a paragraph. We found that RSVP is a promising presentation type for reading short texts displayed in edge-fixed or head-fixed location in VR. The paragraph presentation type using world-fixed or edge-fixed location is promising for reading long text if movement in the virtual environment is not required. Insights from our study inform the design of reading interfaces for VR applications.

During everyday web browsing and search users reveal many pieces of private information to third parties. Even though people report being concerned about their privacy online, they often do not take steps to protect it. This is known as the 'privacy paradox' in the literature. In this work we study two well-known strategies based on theories from the behavioral sciences, nudging and boosting, which encourage users to browse in a way that their private data are less exposed. First, an online survey (N=127) tested the comprehensibility and efficacy of various facts (boosts), before the most effective of these were evaluated against 'nudge' interventions previously shown to be efficacious in lab-studies. A three week naturalistic study (N=68) using a browser extension revealed that both nudges and boosts improve browsing privacy, as approximated by different measures. Boosts are also shown to improve user knowledge about privacy in the short term, but the benefit weakens over time.

Cloud gaming services and remote play offer a wide range of advantages but can inherent a considerable delay between input and action also known as latency. Previous work indicates that deep learning algorithms such as artificial neural networks (ANN) are able to compensate for latency. As high latency in video games significantly reduces player performance and game experience, this work investigates if latency can be compensated using ANNs within a live first-person action game. We developed a 3D video game and coupled it with the prediction of an ANN. We trained our network on data of 24 participants who played the game in a first study. We evaluated our system in a second user study with 96 participants. To simulate latency in cloud game streaming services, we added 180 ms latency to the game by buffering user inputs. In the study we predicted latency values of 60 ms, 120 ms and 180 ms. Our results show that players achieve significantly higher scores, substantially more hits per shot and associate the game significantly stronger with a positive affect when supported by our ANN. This work illustrates that high latency systems, such as game streaming services, benefit from utilizing a predictive system.

Online services collect an increasing amount of data about their users. Privacy policies are currently the only common way to inform users about the kinds of data collected, stored and processed by online services. Previous work showed that users do not read and understand privacy policies, due to their length, difficult language, and often non-prominent location. Embedding privacy-relevant information directly in the context of use could help users understand the privacy implications of using online services. We implemented Contextual Privacy Policies (CPPs) as a browser extension and provide it to the community to make privacy information accessible for end-users. We evaluated CPPs through a one-week deployment and in situ questionnaires as well as pre- and post-study interviews. We found that CPPs were well received by participants. The analysis revealed that provided information should be as compact as possible, be adjusted to user groups and enable users to take action.

Collaborative Virtual Environments (CVEs) offer unique opportunities for human communication. Humans can interact with each other over a distance in any environment and visual embodiment they want. Although deictic gestures are especially important as they can guide other humans' attention, humans make systematic errors when using and interpreting them. Recent work suggests that the interpretation of vertical deictic gestures can be significantly improved by warping the pointing arm. In this paper, we extend previous work by showing that models enable to also improve the interpretation of deictic gestures at targets all around the user. Through a study with 28 participants in a CVE, we analyzed the errors users make when interpreting deictic gestures. We derived a model that rotates the arm of a pointing user's avatar to improve the observing users' accuracy. A second study with 24 participants shows that we can improve observers' accuracy by 22.9%. As our approach is not noticeable for users, it improves their accuracy without requiring them to learn a new interaction technique or distracting from the experience.

Cognitive performance fluctuates throughout the day partly due to our “inner clock”. Systems that are aware of these fluctuations can adapt to their users’ momentary capacity and adjust task difficulty accordingly. Here, we evaluate the use of performance measures obtained from mobile game data to estimate players’ varying alertness. We developed a smartphone game that emulates three validated tasks for cognitive performance and conducted an in-the-wild study over two weeks with 30 participants to collect performance measures and subjective ground truth on alertness levels. Our results show that players’ performance can be explained by a generative model which is based on two established alertness-modulating processes, namely the homeostatic process and circadian alertness fluctuations. Our method can be used to implicitly model players’ alertness levels as a basis for cognition-aware applications.

Touchscreens translate touches of all kinds into 2D coordinates. This limits the input vocabulary and constrains effective interaction to touches by the fingertip. Previous tabletop research extended the input vocabulary with a myriad of promising input techniques using the shape of fingers and hands. However, these techniques are not applicable to mobile devices due to differences in size, ergonomics, and technology. We conducted ideation sessions (N=17) to explore novel input techniques and use cases for imprint-based touch sensing on mobile devices. As a case study, we present FlexionTouch, a novel input technique that recognizes the finger flexion on a touchscreen. Using the finger flexion as an additional input dimension, FlexionTouch provides an always-available shortcut and can be used for value inputs, document previews, and gestures. We propose five example use cases for FlexionTouch input which we evaluated in a second user study (N=20). While the low resolution of the capacitive images leads to a less accurate input compared to tabletops, participants still find the presented use cases helpful. As our input technique is purely software-based, it can be readily deployed to every mobile device with a capacitive touchscreen.

With the rise of the Internet of Things, home appliances become connected and they can proactively provide status information to users. Facing a steadily increasing number of notification sources, it is unclear how information from smart home devices should be provided without overloading the users’ attention. In this paper, we investigate the design of non-urgent smart home notifications using a smart plant system. Based on feedback from focus groups, we designed four notification types and compared them in an eight-week in-situ study. We show that notifications displayed on smart home devices are preferred to those received on smartphones. Event-based notifications are unobtrusive, actionable and are preferred to persistent notifications. We derive guidelines that address the need of being in control, opportune locations for notification delivery at opportune moments, notification blindness, the importance of discretizing continuous information, and combining related notifications.

Virtual reality (VR) allows embodying any possible avatar. Known as the Proteus effect, avatars can change users’ behavior and attitudes. Previous work found that embodying Albert Einstein can increase cognitive task performance. The behavioral confirmation paradigm, however, predicts that our behavior is also affected by others’ perception of us. Therefore, we investigated the cognitive performance in collaborative VR when self-perception and external perception of the own avatar differ. 32 male participants performed a Tower of London task in pairs. One participant embodied Einstein or a young adult while the other perceived the participant as Einstein or a young adult. We show that the perception by others affects cognitive performance. The Einstein avatar also decreased the perceived workload. Results imply that avatars’ appearance to both, the user and the others must be considered when designing for cognitively demanding tasks.

Motion tracking technologies and avatars in virtual reality (VR) showing the movements of the own body enable high levels of presence and a strong illusion of body ownership (IBO) – key features of immersive systems and gaming experiences in virtual environments. Previous work suggests using software-based algorithms that can not only compensate system latency but also predict future movements of the user to increase input performance. However, the effects of movement prediction in VR on input performance are largely unknown. In this paper, we investigate neural network-based predictions of full-body avatar movements in two scenarios: In the first study, we used a standardized 2D Fitts’ Law task to examine the information throughput in VR. In the second study, we utilized a full-body VR game to determine the users’ performance. We found that both performance and subjective measures in a standardized 2D Fitts’ law task could not benefit from the predicted avatar movements. In an immersive gaming scenario, however, the perceived accuracy of the own body location improved. Presence and body assessments remained more stable and were higher than during the Fitts’ task. We conclude that machine-learning-based predictions could be used to compensate system-related latency but participants only subjectively benefit under certain conditions.

While advances in mobile text entry enable smartphone users to type almost as fast as on hardware keyboards, text-heavy activities are still not widely adopted. One reason is the lack of shortcut mechanisms. In this article, we determine shortcuts for text-heavy activities, elicit shortcut gestures, implement them for a fully touch-sensitive smartphone, and conduct an evaluation with potential users. We found that experts perform around 800 keyboard shortcuts per day, which are not available on smartphones. Interviews revealed the lack of shortcuts as a major limitation that prevents mobile text editing. Therefore, we elicited gestures for the 22 most important shortcuts for smartphones that are touch-sensitive on the whole device surface. We implemented the gestures for a fully touch-sensitive smartphone using deep learning and evaluated them in realistic scenarios to gather feedback. We show that the developed prototype is perceived as intuitive and faster than recent commercial approaches.

Sliders are one of the most fundamental components used in touchscreen user interfaces (UIs). When entering data using a slider, errors occur due e.g. to visual perception, resulting in inputs not matching what is intended by the user. However, it is unclear if the errors occur uniformly across the full range of the slider or if there are systematic offsets. We conducted a study to assess the errors occurring when entering values with horizontal and vertical sliders as well as two common visual styles. Our results reveal significant effects of slider orientation and style on the precision of the entered values. Furthermore, we identify systematic offsets that depend on the visual style and the target value. As the errors are partially systematic, they can be compensated to improve users' precision. Our findings provide UI designers with data to optimize user experiences in the wide variety of application areas where slider based touchscreen input is used.

As touchscreens are the most successful input method of current mobile devices, touch gestures became a widely used input technique. While gestures provide users with advantages to express themselves, they also introduce challenges regarding accuracy and memorability. In this paper, we investigate the effect of a gesture's orientation on how well the gesture can be performed. We conducted a study in which participants performed systematically rotated unistroke gestures. For straight lines as well as for compound lines, we found that users tend to align gestures with the primary axes. We show that the error can be described by a Clausen function with R² = .93. Based on our findings, we suggest design implications and highlight the potential for recognizing flick gestures, visualizing gestures and improving recognition of compound gestures.

Understanding social perception is important for designing mobile devices that are socially acceptable. Previous work not only investigated the social acceptability of mobile devices and interaction techniques but also provided tools to measure social acceptance. However, we lack a robust model that explains the underlying factors that make devices socially acceptable. In this paper, we consider mobile devices as social objects and investigate if the stereotype content model (SCM) can be applied to those devices. Through a study that assesses combinations of mobile devices and group stereotypes, we show that mobile devices have a systematic effect on the stereotypes' warmth and competence. Supported by a second study, which combined mobile devices without a specific stereotypical user, our result suggests that mobile devices are perceived stereotypically by themselves. Our combined results highlight mobile devices as social objects and the importance of considering stereotypes when assessing social acceptance of mobile devices.

Virtual Reality (VR) is gaining increasing importance in science, education, and entertainment. A fundamental characteristic of VR is creating presence, the experience of 'being' or 'acting', when physically situated in another place. Measuring presence is vital for VR research and development. It is typically repeatedly assessed through questionnaires completed after leaving a VR scene. Requiring participants to leave and re-enter the VR costs time and can cause disorientation. In this paper, we investigate the effect of completing presence questionnaires directly in VR. Thirty-six participants experienced two immersion levels and filled three standardized presence questionnaires in the real world or VR. We found no effect on the questionnaires' mean scores; however, we found that the variance of those measures significantly depends on the realism of the virtual scene and if the subjects had left the VR. The results indicate that, besides reducing a study's duration and reducing disorientation, completing questionnaires in VR does not change the measured presence but can increase the consistency of the variance.

Empirical studies are a cornerstone of HCI research. Technical progress constantly enables new study methods. Online surveys, for example, make it possible to collect feedback from remote users. Progress in augmented and virtual reality enables to collect feedback with early designs. In-situ studies enable researchers to gather feedback in natural environments. While these methods have unique advantages and disadvantages, it is unclear if and how using a specific method affects the results. Therefore, we conducted a study with 60 participants comparing five different methods (online, virtual reality, augmented reality, lab setup, and in-situ) to evaluate early prototypes of smart artifacts. We asked participants to assess four different smart artifacts using standardized questionnaires. We show that the method significantly affects the study result and discuss implications for HCI research. Finally, we highlight further directions to overcome the effect of the used methods.

Touchscreens enable intuitive mobile interaction. However, touch input is limited to 2D touch locations which makes it challenging to provide shortcuts and secondary actions similar to hardware keyboards and mice. Previous work presented a wide range of approaches to provide secondary actions by identifying which finger touched the display. While these approaches are based on external sensors which are inconvenient, we use capacitive images from mobile touchscreens to investigate the feasibility of finger identification. We collected a dataset of low-resolution fingerprints and trained convolutional neural networks that classify touches from eight combinations of fingers. We focused on combinations that involve the thumb and index finger as these are mainly used for interaction. As a result, we achieved an accuracy of over 92% for a position-invariant differentiation between left and right thumbs. We evaluated the model and two use cases that users find useful and intuitive. We publicly share our data set (CapFingerld) comprising 455,709 capacitive images of touches from each finger on a representative mutual capacitive touchscreen and our models to enable future work using and improving them.

Additional input controls such as fingerprint scanners, physical buttons, and Back-of-Device (BoD) touch panels improve the input capabilities on smartphones. While previous work showed the benefits of input beyond the touchscreen, unfavorably designed input controls force detrimental grip changes and increase the likelihood of unintended inputs. Researchers investigated all fingers' comfortable areas to avoid grip changes. However, there is no understanding of unintended BoD inputs which frustrate users and lead to embarrassing mistakes. In this paper, we study the BoD areas in which unintended inputs occur during interaction with the touchscreen. Participants performed common tasks on four smartphones which they held in the prevalent single-handed grip while sitting and walking. We recorded finger movements with a motion capture system and analyzed the unintended inputs. We identified comfortable areas on the back in which no unintended inputs occur and found that the least unintended inputs occurred on 5" devices. We derive three design implications for BoD input to help designers considering reachability and unintended input.

Museums are essential to make culture accessible to the mass audience. Human museum guides are important to explain the presented artifacts to the visitors. Recently, museums started to experiment with enhancing exhibitions through mixed reality. It enables cultural exhibitors to provide each visitor with an individualized virtual guide that adapts to the visitor's interests. The effect of the presence and appearance of a virtual museum guide is, however, unclear. In this paper, we compare a real-world guide with a realistic, an abstract, and an audio-only representation of the virtual guide. Participants followed four multimodal presentations while we investigated the effect on comprehension and perceived co-presence. We found that a realistic representation of a virtual guide increases the perceived co-presence and does not adversely affect the comprehension of learning content in mixed reality exhibitions. Insights from our study inform the design of virtual guides for real-world exhibitions.

Recent work demonstrated the opportunities of thermal imaging in the development of novel interactive systems. However, the exploration is limited to controlled lab setups. Hence, little we know about how thermal imaging could be useful for a broader range of daily applications by novice users. To investigate the potential of domestication of thermal imaging, we conducted an exploration with a technology-cultural probe. Ten households (26 individuals) used a mobile thermal camera in their daily life. We collected thermal photos taken by the participants and conducted interviews after using the camera. We found that the users were excited about using thermal cameras in their everyday lives and found many practical uses for them. Our study provides insights into how novice users wish to use thermal imaging technology to augment their vision in daily setups, as well as identifying and classifying common thermal imaging use cases. Our work contributes implications for designing thermal imaging devices targeted towards novice users.

The ubiquitous availability of smart speakers allows hands- and eyes-free interaction through Voice User Interfaces (VUIs). Control-ling music playback is the most commonly used feature of VUIs. Previous work investigated how users naturally interact with smart speakers and suggested that users' gender could affect the devices' usability. The usability of commercial devices compared to other interactive systems and the effects of users' gender is, however, unclear. Therefore, we conducted a study with 20 participants using an Amazon Echo Dot and a laptop device. Participants searched for artists and titles using a Graphical User Interface (GUI) and a VUI. In addition, they performed different tasks such as saving a song in a playlist or adding songs into a queue. The analysis revealed that the VUI provides significantly lower usability because it lacks features, requires higher mental effort, and provides confusing answers. In contrast to previous concerns, the analysis did not reveal significant device×gender effects.

Smart home appliances and smart homes, in general, are on the verge of ubiquity. Research and industry proposed a range of modalities, including speech, mid-air gestures, and touch displays, to control smart homes. While previous work designed for the individual modalities, it is unclear how they compare from a user-centered perspective. Therefore, we conducted an elicitation study that asked participants to propose commands using speech, mid-air gestures, and a touch display. Also, we asked participants to rate their suggestions and the modalities. The results show that using voice commands or a touch display is clearly preferred compared to the use of mid-air gestures. As we found high agreement scores for voice commands, our results also highlight the potential of elicitation studies for voice interfaces.

Navigating on large high-resolution displays (LHRDs) using devices built for traditional desktop computers can be strenuous and negatively impact user experience. As LHRDs transition to everyday use, new user-friendly interaction techniques need to be designed to capitalise on the potential offered by the abundant screen space on LHRDs. We conducted a study which compared mouse pointing and eye-tracker assisted pointing (MAGIC pointing) on LHRDs. In a controlled experiment with 35 participants, we investigated user performance in a one-dimensional pointing task and a map-based search task. We determined that MAGIC pointing had a lower throughput, but participants had the perception of higher performance. Our work contributes insights for the design of pointing techniques for LHRDs. The results indicate that the choice of technique is scenario-dependent which contrasts with desktop computers.

Disruptive colleagues, procrastinative web-browsing or low-priority e-mail are just a few types of distractions in the modern workplace. They reduce efficiency and increase perceived workload. Previous work shows that digital and social distractions can be reduced by tangible artifacts that signal phases of high concentration to colleagues or block websites. In this paper, we present the knob a holistic approach to distraction blocking. It simultaneously serves as a controller for blocking websites, managing smartphones' state, and signaling availability to colleagues. We evaluated the system through an in-situ deployment to understand how the artifact can reduce distractions. We show that the knob has the potential to improve users' self-discipline and provide suggestions for future distraction blocking solutions.

Notifications are an essential feature of smartphones. While they support users in staying up-to-date, they are also a prominent source of interruptions. A deeper understanding of mobile notifications is required to avoid adverse effects. However, assessing mobile notifications is challenging as user studies on mobile notifications are typically conducted in-situ. Surveying users may lead to additional interruptions, and the content of notifications is inherently private. In this paper, we introduce a privacy-aware system for annotating mobile notifications in user studies. In an in-situ case study, participants annotated their notifications for one week. Participants perceived 38.91% of their notifications as not important and over half (51.75%) as non-urgent. Only 6.33% of the notifications were rated as both very important and very urgent. We discuss influencing factors, including a breakdown of messaging notifications, and implications for future smart notification systems that continue to fulfill users' information need while respecting their digital well-being.

With smartphones being a prime example, touchscreens became one of the most widely used interface to interact with computing systems. Compared to other touchscreen devices, smartphones pose additional challenges as the hand that interacts with the device is commonly used to also hold the device. Consequently, determining how fingers of the hand holding the device can interact with the screen is a non-trivial challenge. A body of recent work investigated the comfortable area in controlled lab studies. This poses limitations as it is based on the assumption that the grips used in the studies are representative for normal smartphone use. In this paper, we extend previous work by providing insights from in-the-wild studies using two different apps that were deployed in the Android App Store. Comparing our results with previous work we confirm that our data fits previously proposed models. Further analyzing the data, we highlight the sweet spot, the position that is touched if the input can be performed on the whole screen.

As architects usually decide on the shape and look of windows during the design of buildings, opportunities for interactive windows have not been systematically explored yet. In this work, we extend the vision of sustainable and comfortable adaptive buildings using interactive smart windows. We systematically explore the design space of interactive windows to chart requirements, constraints, and challenges. To that end, we built proof-of-concept prototypes of smart windows with fine-grained control of transparency. In two studies, we explored user attitudes towards interactive windows and elicited control methods. We found that users understand and see potential for interactive windows at home. We provide specific usage contexts and specify interactions that may facilitate domestic applications. Our work illustrates the concept of interactive smart windows and provides insights regarding their design, development, and user controls for adaptive walls. We identify design dimensions and challenges to stimulate further development in the domain of adaptive buildings.

Touchscreens are a common fixture in current vehicles. With autonomous driving, we can expect touch interaction with such in-vehicle media systems to exponentially increase. In spite of vehicle suspension systems, road perturbations will continue to exert forces that can render in-vehicle touch interaction challenging. Using a motion simulator, we investigate how different vehicle speeds interact with road features (i.e., speed bumps) to influence touch interaction. We determine their effect on pointing accuracy and task completion time. We show that road bumps have a significant effect on touch input and can decrease accuracy by 19%. In light of this, we developed a Random Forest (RF) model that improves touch accuracy by 32.0% on our test set and by 22.5% on our validation set. As the lightweight model uses only features that can easily be determined through inertial measurement units, this model could be easily deployed in current automobiles.

In the era of ubiquitous computing, people expect applications to work across different devices. To provide a seamless user experience it is therefore crucial that interfaces and interactions are consistent across different device types. In this paper, we present a method to create gesture sets that are consistent and easily transferable. Our proposed method entails 1) the gesture elicitation on each device type, 2) the consolidation of a unified gesture set, and 3) a final validation by calculating a transferability score. We tested our approach by eliciting a set of user-defined gestures for reading with Rapid Serial Visual Presentation (RSVP) of text for three device types: phone, watch, and glasses. We present the resulting, unified gesture set for RSVP reading and show the feasibility of our method to elicit gesture sets that are consistent across device types with different form factors.

Commodity mobile devices are now equipped with high-resolution front-facing cameras, allowing applications in biometrics (e.g., FaceID in the iPhone X), facial expression analysis, or gaze interaction. However, it is unknown how often users hold devices in a way that allows capturing their face or eyes, and how this impacts detection accuracy. We collected 25,726 in-the-wild photos, taken from the front-facing camera of smartphones as well as associated application usage logs. We found that the full face is visible about 29% of the time, and that in most cases the face is only partially visible. Furthermore, we identified an influence of users' current activity; for example, when watching videos, the eyes but not the entire face are visible 75% of the time in our dataset. We found that a state-of-the-art face detection algorithm performs poorly against photos taken from front-facing cameras. We discuss how these findings impact mobile applications that leverage face and eye detection, and derive practical implications to address state-of-the art's limitations.

Entering text is one of the most common tasks when interacting with computing systems. Virtual Reality (VR) presents a challenge as neither the user's hands nor the physical input devices are directly visible. Hence, conventional desktop peripherals are very slow, imprecise, and cumbersome. We developed a apparatus that tracks the user's hands, and a physical keyboard, and visualize them in VR. In a text input study with 32 participants, we investigated the achievable text entry speed and the effect of hand representations and transparency on typing performance, workload, and presence. With our apparatus, experienced typists benefited from seeing their hands, and reach almost outside-VR performance. Inexperienced typists profited from semi-transparent hands, which enabled them to type just 5.6 WPM slower than with a regular desktop setup. We conclude that optimizing the visualization of hands in VR is important, especially for inexperienced typists, to enable a high typing performance.

Touchscreens are the most successful input method for smartphones. Despite their flexibility, touch input is limited to the location of taps and gestures. We present PalmTouch, an additional input modality that differentiates between touches of fingers and the palm. Touching the display with the palm can be a natural gesture since moving the thumb towards the device's top edge implicitly places the palm on the touchscreen. We present different use cases for PalmTouch, including the use as a shortcut and for improving reachability. To evaluate these use cases, we have developed a model that differentiates between finger and palm touch with an accuracy of 99.53% in realistic scenarios. Results of the evaluation show that participants perceive the input modality as intuitive and natural to perform. Moreover, they appreciate PalmTouch as an easy and fast solution to address the reachability issue during one-handed smartphone interaction compared to thumb stretching or grip changes.

Previous research and recent smartphone development presented a wide range of input controls beyond the touchscreen. Fingerprint scanners, silent switches, and Back-of-Device (BoD) touch panels offer additional ways to perform input. However, with the increasing amount of input controls on the device, unintentional input or limited reachability can hinder interaction. In a one-handed scenario, we conducted a study to investigate the areas that can be reached without losing grip stability (comfortable area), and with stretched fingers (maximum range) using four different phone sizes. We describe the characteristics of the comfortable area and maximum range for different phone sizes and derive four design implications for the placement of input controls to support one-handed BoD and edge interaction. Amongst others, we show that the index and middle finger are the most suited fingers for BoD interaction and that the grip shifts towards the top edge with increasing phone sizes.

Previous work has shown that large high resolution displays (LHRDs) can enhance collaboration between users. As LHRDs allow free movement in front of the screen, an understanding of movement behavior is required to build successful interfaces for these devices. This paper presents Pac-Many; a multiplayer version of the classical computer game Pac-Man to study group dynamics when using LHRDs. We utilized smartphones as game controllers to enable free movement while playing the game. In a lab study, using a 4m × 1m LHRD, 24 participants (12 pairs) played Pac-Many in collaborative and competitive conditions. The results show that players in the collaborative condition divided screen space evenly. In contrast, competing players stood closer together to avoid benefits for the other player. We discuss how the nature of the task is important when designing and analyzing collaborative interfaces for LHRDs. Our work shows how to account for the spatial aspects of interaction with LHRDs to build immersive experiences.

While the proliferation of mobile devices has rendered mobile notifications ubiquitous, researchers are only slowly beginning to understand how these technologies affect everyday social interactions. In particular, the negative social influence of mobile interruptions remains unexplored from a methodological perspective. This paper contributes a mixed-method evaluation procedure for assessing the disruptive impact of mobile interruptions in conversation. The approach combines quantitative eye tracking, qualitative analysis, and a simulated conversation environment to enable fast assessment of disruptiveness. It is intended to be used as a part of an iterative interaction design process. We describe our approach in detail, present an example of its use to study a new call declining technique, and reflect upon the pros and cons of our approach.

Pointing at remote objects to direct others' attention is a fundamental human ability. Previous work explored methods for remote pointing to select targets. Absolute pointing techniques that cast a ray from the user to a target are affected by humans' limited pointing accuracy. Recent work suggests that accuracy can be improved by compensating systematic offsets between targets a user aims at and rays cast from the user to the target. In this paper, we investigate mid-air pointing in the real world and virtual reality. Through a pointing study, we model the offsets to improve pointing accuracy and show that being in a virtual environment affects how users point at targets. In the second study, we validate the developed model and analyze the effect of compensating systematic offsets. We show that the provided model can significantly improve pointing accuracy when no cursor is provided. We further show that a cursor improves pointing accuracy but also increases the selection time.

Smart glasses are increasingly being used in professional contexts. Having key applications such as short messaging and newsreader, they enable continuous access to textual information. In particular, smart glasses allow reading while performing other activities as they do not occlude the user's world view. For efficient reading, it is necessary to understand how a text should be presented on them. We, therefore, conducted a study with 24 participants using a Microsoft HoloLens to investigate how to display text on smart glasses while walking and sitting. We compared text presentation in the top-right, center, and bottom-center positions with Rapid Serial Visual Presentation (RSVP) and line-by-line scrolling. We found that text displayed in the top-right of smart glasses increases subjective workload and reduces comprehension. RSVP yields higher comprehension while sitting. Conversely, reading with scrolling yields higher comprehension while walking. Insights from our study inform the design of reading interfaces for smart glasses.

A large number of today's systems use interactive touch surfaces as the main input channel. Current devices reduce the richness of touch input to two-dimensional positions on the screen. A growing body of work develops methods that enrich touch input to provide additional degrees of freedom for touch interaction. In particular, previous work proposed to use the finger's orientation as additional input. To efficiently implement new input techniques which make use of the new input dimensions, we need to understand the limitations of the input. Therefore, we conducted a study to derive the ergonomic constraints for using finger orientation as additional input in a two-handed smartphone scenario. We show that for both hands, the comfort and the non-comfort zone depend on how the user interacts with a touch surface. For two-handed smart-phone scenarios, the range is 33.3% larger than for tabletop scenarios. We further show that the phone orientation correlates with the finger orientation. Finger orientations which are harder to perform result in phone orientations where the screen does not directly face the user.

Notifications on mobile devices are a prominent source of interruptions. Previous work suggests using opportune moments to deliver notifications to reduce negative effects. In this paper, we instead explore the manual deferral of notifications. We developed an Android app that allows users to "snooze" mobile notifications for a user-defined amount of time or to a user-defined point in time. Using this app, we conducted a year-long in-the-wild study with 295 active users. To complement the findings, we recruited 16 further participants who used the app for one week and subsequently interviewed them. In both studies, snoozing was mainly used to defer notifications related to people and events. The reasons for deferral were manifold, from not being able to attend notifications immediately to not wanting to. Daily routines played an important role in the deferral of notifications. Most notifications were deferred to the same day or next morning, and a deferral of more than two days was an exception. Based on our findings, we derive design implications that can inform the design of future smart notification systems.

Human vision is limited to a small band of the electromagnetic spectrum. It has been shown that extending visual perception can be beneficial, but it is unclear if this is useful for a broader range of applications. In this paper, we explore user reactions to extended visual perception. We describe the design and implementation of TriSight, a head-mounted display that allows the user to perceive the environment in the visual spectrum, the thermal spectrum, and through depth maps. In a study, we asked participants to perform everyday tasks in a created home, kitchen, office and basement with TriSight. Through analyzing videos, interviews, questionnaires and logs we chart the users' feedback towards the augmented visual perception. Our findings, imply the positive impact of extended vision, as well as the potential of thermal vision as alternative perception mode in performing daily tasks. Finally, we discuss the implications for designing devices that extend human's visual perception beyond the visual spectrum.

We present VRHapticDrones, a system utilizing quadcopters as levitating haptic feedback proxy. A touchable surface is attached to the side of the quadcopters to provide unintrusive, flexible, and programmable haptic feedback in virtual reality. Since the users' sense of presence in virtual reality is a crucial factor for the overall user experience, our system simulates haptic feedback of virtual objects. Quadcopters are dynamically positioned to provide haptic feedback relative to the physical interaction space of the user. In a first user study, we demonstrate that haptic feedback provided by VRHapticDrones significantly increases users' sense of presence compared to vibrotactile controllers and interactions without additional haptic feedback. In a second user study, we explored the quality of induced feedback regarding the expected feeling of different objects. Results show that VRHapticDrones is best suited to simulate objects that are expected to feel either light-weight or have yielding surfaces. With VRHapticDrones we contribute a solution to provide unintrusive and flexible feedback as well as insights for future VR haptic feedback systems.

Reflecting on previous activities can improve episodic memory and well-being. While manually recording activities and experiences is not always feasible, there is a range of mobile sensors that allow implicit recording of users' lives. In this work, we investigate how reflection on daily summaries created from implicitly collected data improves episodic memory. Therefore, we built ReflectiveDiary, an Android application that collects personal data streams to create daily summaries automatically. Over the course of 16 days, we collected data from 11 participants using information, such as calls, messages, and calendar data to help people recollect their activities. By comparing reflected with non-reflected days, we show that reflecting on implicitly collected data improves remembering of events and their surrounding details. We further present an analysis of different types of memory cues and their usefulness to inform the design of reflective tools that help people to improve their episodic memory.

Calendars are an essential tool for users to manage their daily schedules. With the expanding availability of Internet of Things (IoT) devices and smart home appliances, more and more connected devices appear in the homes of users. Some of these devices have the potential to complement, extend or even replace existing physical and digital calendars. However, little is known about how these devices should display calendar information without overwhelming users and negatively influencing their digital well-being. In this paper, we report the results of a lab study with 18 participants in which we compared calendars on seven different types of devices in the smart home context, from existing smartphone apps to novel e-paper displays and smart mirrors. We developed a smart calendar platform as a research probe and provide first insights into how different devices should convey information in future smart homes.

Smartphones are the most successful mobile devices and offer intuitive interaction through touchscreens. Current devices treat all fingers equally and only sense touch contacts on the front of the device. In this paper, we present InfiniTouch, the first system that enables touch input on the whole device surface and identifies the fingers touching the device without external sensors while keeping the form factor of a standard smartphone. We first developed a prototype with capacitive sensors on the front, the back and on three sides. We then conducted a study to train a convolutional neural network that identifies fingers with an accuracy of 95.78% while estimating their position with a mean absolute error of 0.74cm. We demonstrate the usefulness of multiple use cases made possible with InfiniTouch, including finger-aware gestures and finger flexion state as an action modifier.

Although a large number of navigation support systems for visually impaired people have been proposed in the past, navigating through unknown environments is still a major challenge for visually impaired travelers. Existing systems provide navigation information through headphones, speakers or tactile actuators. In this paper, we propose to use small lightweight quadcopters instead to provide navigation information for people with visual impairments. Using a leashed or free-floating quadcopter, the user is navigated by the distinct sound that the quadcopter emits and a haptic stimulus provided by the leash. In a user with 14 visually impaired participants, we compared leashed quadcopter navigation, free-floating quadcopter navigation, and traditional audio navigation. The results show that compared to audio navigation, participants navigate significantly faster with a free-floating quadcopter and make fewer navigation errors using the quadcopter navigation methods.

Obtaining information about audience engagement in presentations is a valuable asset for presenters in many domains. Prior literature mostly utilized explicit methods of collecting feedback which induce distractions, add workload on audience and do not provide objective information to presenters. We present EngageMeter - a system that allows fine-grained information on audience engagement to be obtained implicitly from multiple brain-computer interfaces (BCI) and to be fed back to presenters for real time and post-hoc access. Through evaluation during an HCI conference (Naudience=11, Npresenters=3) we found that EngageMeter provides value to presenters (a) in real-time, since it allows reacting to current engagement scores by changing tone or adding pauses, and (b) in post-hoc, since presenters can adjust their slides and embed extra elements. We discuss how EngageMeter can be used in collocated and distributed audience sensing as well as how it can aid presenters in long term use.

Rendering the user's body in virtual reality increases immersion and presence the illusion of "being there". Recent technology enables determining the pose and position of the hands to render them accordingly while interacting within the virtual environment. Virtual reality applications often use realistic male or female hands, mimic robotic hands, or cartoon hands. However, it is unclear how users perceive different hand styles. We conducted a study with 14 male and 14 female participants in virtual reality to investigate the effect of gender on the perception of six different hands. Quantitative and qualitative results show that women perceive lower levels of presence while using male avatar hands and male perceive lower levels of presence using non-human avatar hands. While women dislike male hands, men accept and feel presence with avatar hands of both genders. Our results highlight the importance of considering the users' diversity when designing virtual reality experiences.

Learning a foreign language is a daunting and time-consuming task. People often lack the time or motivation to sit down and engage with learning content on a regular basis. We present an investigation of microlearning sessions on mobile phones, in which we focus on session triggers, presentation methods, and user context. Therefore, we built an Android app that prompts users to review foreign language vocabulary directly through notifications or through app usage across the day. We present results from a controlled and an in-the-wild study, in which we explore engagement and user context. In-app sessions lasted longer, but notifications added a significant number of "quick" learning sessions. 37.6% of sessions were completed in transit, hence learning-on-the-go was well received. Neither the use of boredom as trigger nor the presentation (flashcard and multiple-choice) had a significant effect. We conclude with implications for the design of mobile microlearning applications with context-awareness.

The latency of current mobile devices' touchscreens is around 100ms and has widely been explored. Latency down to 2ms is noticeable, and latency as low as 25ms reduces users' performance. Previous work reduced touch latency by extrapolating a finger's movement using an ensemble of shallow neural networks and showed that predicting 33ms into the future increases users' performance. Unfortunately, this prediction has a high error. Predicting beyond 33ms did not increase participants' performance, and the error affected the subjective assessment. We use more recent machine learning techniques to reduce the prediction error. We train LSTM networks and multilayer perceptrons using a large data set and regularization. We show that linear extrapolation causes an 116.7% higher error and the previously proposed ensembles of shallow networks cause a 26.7% higher error compared to the LSTM networks. The trained models, the data used for testing, and the source code is available on GitHub.

Previous research proposed a wide range of interaction methods and use cases based on the previously unused back side and edge of a smartphone. Common approaches to implementing Back-of-Device (BoD) interaction include attaching two smartphones back to back and building a prototype completely from scratch. Changes in the device's form factor can influence hand grip and input performance as shown in previous work. Further, the lack of an established operating system and SDK requires more effort to implement novel interaction methods. In this work, we present a smartphone prototype that runs Android and has a form factor nearly identical to an off-the-shelf smartphone. It further provides capacitive images of the hand holding the device for use cases such as grip-pattern recognition. We describe technical details and share source files so that others can re-build our prototype. We evaluated the prototype with 8 participants to demonstrate the data that can be retrieved for an exemplary grip classification.

While most current interactive surfaces use only the position of the finger on the surface as the input source, previous work suggests using the finger orientation for enriching the input space. Thus, an understanding of the physiological restrictions of the hand is required to build effective interactive techniques that use finger orientation. We conducted a study to derive the ergonomic constraints for using finger orientation as an effective input source. In a controlled experiment, we systematically manipulated finger pitch and yaw while performing a touch action. Participants were asked to rate the feasibility of the touch action. We found that finger pitch and yaw do significantly affect perceived feasibility and 21.1% of the touch actions were perceived as impossible to perform. Our results show that the finger yaw input space can be divided into the comfort and non-comfort zones. We further present design considerations for future interfaces using finger orientation.

One key feature of TensorFlow includes the possibility to compile the trained model to run efficiently on mobile phones. This enables a wide range of opportunities for researchers and developers. In this tutorial, we teach attendees two basic steps to run neural networks on a mobile phone: Firstly, we will teach how to develop neural network architectures and train them in TensorFlow. Secondly, we show the process to run the trained models on a mobile phone.

Over the last decade, a body of research investigated enriching touch actions by using finger orientation as an additional input. Beyond new interaction techniques, we envision new user interface elements to make use of the additional input information. We define the fingers orientation by the pitch, roll, and yaw on the touch surface. Determining the finger orientation is not possible using current state-of-the-art devices. As a first step, we built a system that can determine the finger orientation. We developed a working prototype with a depth camera mounted on a tablet. We conducted a study with 12 participants to record ground truth data for the index, middle, ring and little finger to evaluate the accuracy of our prototype using the PointPose [3] algorithm to estimate the pitch and yaw of the finger. By applying 2D linear correction models, we further show a reduction of RMSE by 45.4% for pitch and 21.83% for yaw.

Interactive systems, such as online search interfaces, require appropriate input if they are to produce accurate information. Without this, they can be inaccurate if the user is uncertain about the keywords. Current systems do not have the means to detect uncertainty, which may lead to a negative user experience. We explore physiological and behavioral measurements as tools to implicitly detect users' uncertainty, and provide a method to integrate input variability in interactive systems. We conducted a laboratory study where participants answered questions of varying difficulty, recording physiological data via a key logger, an eye tracker, and a heart rate sensor. Our results show that participants spent significantly more time on difficult questions and looked longer at their answers before submitting them. Based on our results, we provide initial insights on how data from physiological sensors and logged user behavior can be utilized to enrich interactive systems and evaluate a user's uncertainty level.

Visual exploration of large image data sets is a common and widely performed task. With the ubiquity of cameras, personal image collections are growing. Even when working with large high-resolution displays, collections are typically too large to display all images. In this work, we present two full-body, hands-free zooming interaction techniques for image exploration on large high-resolution displays that allow users to freely move in front of the display. We compare these techniques with state-of-the-art image browsing, where a large number of images is presented in a static grid. The first technique selects and enlarges whole columns by full-body-tracking of the user. The second technique additionally uses the height of the users head to select a single picture as a focus point. The results show that novel interaction techniques and visualizations for image sets on large high-resolution displays are required and can enhance the advantages of such displays.

Although interaction with computing systems has become remarkably diverse in recent years, the computer mouse remained the primary pointing device for daily computer use. Being solely an input device, the classical mouse decouples input from output. In this paper, we propose to extend the mouse to a device that can be actuated by the user and the computer. We developed a mouse that allows its position and button state to be actuated. In a technical evaluation, we test the spatial resolution of our system and how effectively feedback is communicated to the user. In a subjective assessment, we explore users' reactions to four use cases including games and office applications, highlighting the potential of the device. Through a quantitative assessment, we investigate whether perceiving the movement of the mouse helps to learn gestures. Finally, we discuss how a mouse providing feedback can be used to build novel interaction techniques.

Many countries are confronted with aging societies. With the increasing need for elder care, it is necessary to investigate how technology can support aging in place. In this paper, we propose an ambient smart calendar system that supports the self-sufficiency and activeness of older adults. We report the results of a survey of older adults on their use of physical and digital calendars. Based on the results, we developed Caloo (Calendar of opportunities) - a prototypical smart wall calendar which supports older users by generating awareness about their daily schedules as well as supporting them in remaining active in their lives through event suggestions. We evaluated Caloo in a study with retired older adults. Overall, we received positive feedback and identified data sources for events, which the smart calendar can suggest to the user. Our results indicate the potential of deploying an ambient and pervasive smart calendar system that supports aging in place.

Touchscreens are the dominant input mechanism for a variety of devices. One of the main limitations of touchscreens is the latency to receive input, refresh, and respond. This latency is easily perceivable and reduces users' performance. Previous work proposed to reduce latency by extrapolating finger movements to identify future movements - albeit with limited success. In this paper, we propose PredicTouch, a system that improves this extrapolation using inertial measurement units (IMUs). We combine IMU data with users' touch trajectories to train a multi-layer feedforward neural network that predicts future trajectories. We found that this hybrid approach (software: prediction, and hardware: IMU) can significantly reduce the prediction error, reducing latency effects. We show that using a wrist-worn IMU increases the throughput by 15% for finger input and 17% for a stylus.

In the last years, touchscreens became the most common input device for a wide range of computers. While touchscreens are truly pervasive, commercial devices reduce the richness of touch input to two-dimensional positions on the screen. Recent work proposed interaction techniques to extend the richness of the input vocabulary using the finger orientation. Approaches for determining a finger's orientation using off-the-shelf capacitive touchscreens proposed in previous work already enable compelling use cases. However, the low estimation accuracy limits the usability and restricts the usage of finger orientation to non-precise input. With this paper, we provide a ground truth data set for capacitive touch screens recorded with a high-precision motion capture system. Using this data set, we show that a Convolutional Neural Network can outperform approaches proposed in previous work. Instead of relying on hand-crafted features, we trained the model based on the raw capacitive images. Thereby we reduce the pitch error by 9.8% and the yaw error by 45.7%.

Although more and more data is collected automatically, many interfaces still require manual input. When we, for example, enter our daily calorie intake or calculate our ecological footprint, we often have to guess the weight of the food or what distance we have covered with our car. In this paper, we propose a solution to overcome the problem of forcing users to enter a single value when they are unsure about the actual input. On the basis of a slider, we designed four input controls which allow the input of uncertain data in the form of probability distribution functions. To evaluate our input controls, we conducted two studies collecting subjective and objective feedback. Based on the evaluation, we derived implications for their usage. We additionally provide an open-source toolkit with the evaluated input controls that can be included in web applications and customized for different contexts and tasks.

Our eyes use multiple cues to perceive depth. Current 3D displays do not support all depth cues humans can perceive. While they support binocular disparity and convergence, no commercially available 3D display supports focus cues. To use them requires accommodation, i.e. stretching the eye lens when focusing on an individual distance. Previous work proposed multilayer and light field displays that require the eye to accommodate. Such displays enable the user to focus on different depths and blur out content that is out of focus. Thereby, they might ease the separation of content displayed on different depth layers. In this paper we investigate the effect of focus cues by comparing 3D shutter glasses with a multilayer display. We show that recognizing content displayed on a multilayer display takes less time and results in fewer errors compared to shutter glasses. We further show that separating overlapping content on multilayer displays again takes less time, results in fewer errors, and is less demanding. Hence, we argue that multilayer displays are superior to standard 3D displays if layered 3D content is displayed, and they have the potential to extend the design space of standard GUI.

Reviewing lifelogging data has been proposed as a useful tool to support human memory. However, the sheer volume of data (particularly images) that can be captured by modern lifelogging systems makes the selection and presentation of material for review a challenging task. We present the results of a five-week user study involving 16 participants and over 69,000 images that explores both individual requirements for video summaries and the differences in cognitive load, user experience, memory experience, and recall experience between review using video summarisations and non-summary review techniques. Our results can be used to inform the design of future lifelogging data summarisation systems for memory augmentation.

Smart devices have arrived in our everyday lives. Being able to notify the user about events is a core feature of these devices. Related work investigated interruptions caused by notifications on single devices. In this paper, we investigate notifications in multi-device environments by analyzing the results of a week-long in-situ study with 16 participants. We used the Experience Sampling Method (ESM) and recorded the participants' interaction with smartphones, smartwatches, tablets and PCs. Disregarding the type or content of notifications, we found that the smartphone is the preferred device on which to be notified. Further, we found that the proximity to the device, whether it is currently being used and the user's current location can be used to predict if the user wants to receive notifications on a device. The findings can be used to design future multi-device aware smart notification systems.

Devices with touchscreens have an inherent latency. When a user's finger drags an object across the screen the object follows with a latency of around 100ms for current devices. Previous work showed that latencies down to 25ms reduce users' performance and that even 10ms latency is noticeable. In this paper we demonstrate an approach that reduces latency using a predictive model. Extrapolating the finger's movement we predict where the finger will be in the next moment. Comparing different prediction approaches we show for three different tasks that prediction using neural networks is more precise than linear and polynomial extrapolation. Furthermore, we show through a Fitts' Law dragging experiment that reducing touch latency can significantly increases users' performance. As the approach is software-based it can easily be integrated into existing mobile applications and systems.

Even after decades of research about navigation support for visually impaired people, moving independently still remains a major challenge. Previous work in HCI explored a large number of navigation aids, including auditory and tactile guidance systems. In this poster we propose a novel approach to guide visually impaired people. We use small lightweight drones that can be perceived through the distinct sound and the airflow they naturally produce. We describe the interaction concept we envision, first insights from proof-of-concept tests with a visually impaired participant, and provide an overview of potential application scenarios.

Recent work demonstrated the exciting opportunities that thermal imaging offers for the development of interactive systems. It was shown that a thermal camera can sense when a user touches a surface, performs gestures in the camera's direct field of view and, in addition, performs gestures outside the camera's direct field of view through thermal reflection. In this paper, we investigate the material properties that should be considered for detecting interaction using thermal imaging considering both in- and outdoor settings. We conducted a study to analyze the recognition performance for different gestures and different surfaces. Using the results, we derive guidelines on material properties of surfaces for detecting on-surface as well as mid-air interaction using a thermal camera. We discuss the constrains that should be taken into account using thermal imaging as the sensing technology. Finally, we present a material space based on our findings. The space depicts surfaces and the required properties that enable the different interaction techniques.

The advent of wearables (e.g., smartwatches, smartglasses, and digital jewelry) anticipates the need for text entry methods on very small devices. We conduct fundamental research on this topic using 3 qwerty-based soft keyboards for 3 different screen sizes, motivated by the extensive training that users have with qwerty keyboards. In addition to ZoomBoard (a soft keyboard for diminutive screens), we propose a callout-based soft keyboard and ZShift, a novel extension of the Shift pointing technique. We conducted a comprehensive user study followed by extensive analyses on performance, usability, and short-term learning. Our results show that different small screen sizes demand different types of assistance. In general, manufacturers can benefit from these findings by selecting an appropriate qwerty soft keyboard for their devices. Ultimately, this work provides designers, researchers, and practitioners with new understanding of qwerty soft keyboard design space and its scalability for tiny touchscreens.

Pixel densities are increasing rapidly. We can observe this trend in particular for mobile devices like smartphones and tablets. Previous work revealed an effect of pixel density on subjective feedback and objective performance only for low resolution cathode ray tube screens. It is unclear if this effect persists for the four times higher pixel densities of current mobile devices. Therefore, we conducted a study to compare four pixel densities with 359, 180, 120, and 90 pixels per inch. While participants performed three tasks involving images, text and videos on a tablet, we measured perceived effort, perceived visual quality, task completion time, error rate, and body pose. Our results show that the effect of the pixel density highly depends on the content. We found that only for text, the four pixel densities have clearly different perceived media qualities. Pixel density seems to have a smaller effect on perceived media quality for images and videos and we found no effect on objective measures. Results show that text should be displayed in high resolution, while this is less important for images and videos.

Distant pointing at objects and persons is a highly expressive gesture that is widely used in human communication. Pointing is also used to control a range of interactive systems. For determining where a user is pointing at, different ray casting methods have been proposed. In this paper we assess how accurately humans point over distance and how to improve it. Participants pointed at projected targets from 2m and 3m while standing and sitting. Testing three common ray casting methods, we found that even with the most accurate one the average error is 61.3cm. We found that all tested ray casting methods are affected by systematic displacements. Therefore, we trained a polynomial to compensate this displacement. We show that using a user-, pose-, and distant-independent quartic polynomial can reduce the average error by 37.3%

Eating is not only one of the most fundamental human needs but also among the most regular activities. Acquiring food, preparing meals, and socializing around food are deeply rooted in all human cultures. In this paper we show how food can not only serve to satisfy hunger but also become a new display technology. Through food augmentation, a dinner could communicate its ingredients, convey messages, or provide instructions such as the recipe of a meal. We show how to augment a large range of food with laser. We conducted a series of focus groups to gather people's first impressions and derive a broad range of meaningful augmentation scenarios. We discuss the perceived benefits, opportunities, and concerns. Additionally, we evaluated a number of scenarios through an online survey. The most readily accepted augmentation scenarios include adding practical information, increasing awareness about the food, and augmenting food items with a natural skin.

Public displays have advanced from isolated and non interactive "ad" displays which show images and videos to displays that are networked, interactive, and open to a wide variety of content and applications. Prior work has shown large potential of user-generated content on public displays. However, one of the problems with user-generated content on public displays is moderation as content may be explicit or troublesome for a particular location. In this work we explore the expectations of users with regard to content moderation on public displays. An online survey revealed that people not only think that display content should be moderated but also that a delay of up to 10 minutes is acceptable if display content is moderated. In a subsequent in the wild deployment we compared different moderation delays. We found that a moderation delay significantly decreases the number of user-generated posts while at the same time there is no significant effect on users' decision to repeatedly post on the display.

Thermal cameras have recently drawn the attention of HCI researchers as a new sensory system enabling novel interactive systems. They are robust to illumination changes and make it easy to separate human bodies from the image background. Far-infrared radiation, however, has another characteristic that distinguishes thermal cameras from their RGB or depth counterparts, namely thermal reflection. Common surfaces reflect thermal radiation differently than visual light and can be perfect thermal mirrors. In this paper, we show that through thermal reflection, thermal cameras can sense the space beyond their direct field-of-view. A thermal camera can sense areas besides and even behind its field-of-view through thermal reflection. We investigate how thermal reflection can increase the interaction space of projected surfaces using camera-projection systems. We moreover discuss the reflection characteristics of common surfaces in our vicinity in both the visual and thermal radiation bands. Using a proof-of-concept prototype, we demonstrate the increased interaction space for hand-held camera-projection system. Furthermore, we depict a number of promising application examples that can benefit from the thermal reflection characteristics of surfaces.

Notifications are a core feature of mobile phones. They inform users about a variety of events. Users may take immediate action or ignore them depending on the importance of a notification as well as their current context. The nature of notifications is manifold, applications use them both sparsely and frequently. In this paper we present the first large-scale analysis of mobile notifications with a focus on users' subjective perceptions. We derive a holistic picture of notifications on mobile phones by collecting close to 200 million notifications from more than 40,000 users. Using a data-driven approach, we break down what users like and dislike about notifications. Our results reveal differences in importance of notifications and how users value notifications from messaging apps as well as notifications that include information about people and events. Based on these results we derive a number of findings about the nature of notifications and guidelines to effectively use them.

Computing technology is currently adopted in emerging countries. Especially mobile phones and smart phones become widely used - with a much higher penetration than traditional computers. In our work we investigate how computing technologies and particularly mobile devices can support education. While previous work focused on controlled experiments, in this paper we present the results of a 20 weeks long study of mobile phone usage in an emerging region. Our aim was not only to investigate how the phones are used for education but also to learn how they are adopted by children in daily life. By logging screenshots, we used an unsupervised approach that allowed to unobtrusively observe usage patterns without the presence of researchers. Instead of offering tailored teaching applications, we used general-purpose applications to support teaching and found that the phone itself was an empowering technology similar to pen and paper. Based on a detailed analysis of actual use in a natural setting, we derived a set of typical use cases for mobile phones in education and describe how they change learning. From in-depth interviews with a teacher, selected guardians and pupils we show that introducing mobiles phones has great potential for supporting education in emerging regions.

Touch gestures become steadily more important with the ongoing success of touch screen devices. Compared to traditional user interfaces, gestures have the potential to lower cognitive load and the need for visual attention. However, nowadays gestures are defined by designers and developers and it is questionable if these meet all user requirements. In this paper, we present two exploratory studies that investigate how users would use unistroke touch gestures for shortcut access to a mobile phone's key functionalities. We study the functions that users want to access, the preferred activators for gesture execution, and the shapes of the user-invented gestures. We found that most gestures trigger applications, letter-shaped gestures are preferred, and the gestures should be accessible from the lock screen, the wallpaper, and the notification bar. We conclude with a coherent, unambiguous set of gestures for the 20 most frequently accessed functions, which can inform the design of future gesture-controlled applications.

With the recent success of tablet devices a new device type became available for mobile interaction. Just as for mobile phones, touch is the dominant way people interact with tablets. In contrast to the much smaller phones a firm grip with both hands is needed to securely hold tablet devices. While a large body of work has investigated touch interaction on smaller devices, is little empirical research has been carried out on touch-based pointing while holding the device with both hands. To understand touch-based interactions using tablet devices, we conducted an experiment to compare four pointing techniques on both the front and back of the devices while it was held in landscape format. We compare direct touch with the following alternatives for selecting targets, indirect pointing on a virtual touchpad, an inverse cursor, and a miniature interaction area. While direct touch is 35% faster than the fastest alternative, only 74% of the touchscreen and 64% of a back-of-device can be reached by each hand. We show that among the indirect pointing techniques, the miniaturized interaction area is significantly faster and received the best subjective ratings. We conclude that a miniaturized interaction area is a viable alternative to direct touch especially on the backside of tablet devices.

Drawing and handwriting play a central role in primary schools. So far handwriting is practiced mainly on paper and blackboards. Providing tasks on paper can be challenging in developing countries. With the potential availability of mobile phones in classrooms, there is a new medium that can be used. We determined the effect of different touch technologies on children's handwriting for 18 third grade and 20 sixth grade participants. Children drew and wrote using different input techniques. We measured their performance and asked teachers to assess the legibility. We show that writing on touchscreens is less legible and slower than on paper. Further, the comparison of touchscreen technologies indicates that capacitive screens operated with a stylus yield the highest readability and are faster to use for writing than resistive screens. In contrast to these quantitative findings participants from third grade indicated that they prefer resistive screens with a thin stylus compared to using capacitive screens with a stylus or fingers.

In this paper, we investigate how smartphone applications, in particular web browsers, are used on mobile phones. Using a publicly available widget for smart phones, we recorded app usage and the phones' acceleration and orientation from 1,330 devices. Combining app usage and sensor data we derive the device's typical posture while different apps are used. Analyzing motion data shows that devices are moved more while messaging and navigation apps are used as opposed to browser and other common applications. The time distribution between landscape and portrait depicts that most of the landscape mode time is used for burst interaction (e.g., text entry), except for Media apps, which are mostly used in landscape mode. Additionally, we found that over 31% of our users use more than one web browser. Our analysis reveals that the duration of mobile browser sessions is longer by a factor of 1.5 when browsers are explicitly started through the system's launcher in comparison to being launched from within another app. Further, users switch back and forth between apps and web browsers, which suggest that a tight and smooth integration of web browsers with native apps can improve the overall usability. From our findings we derive design guidelines for app developers.

With the rise of current smartphones, virtual keyboards for touchscreens became the dominant mobile text entry technique. We developed a typing game that records how users touch on the standard Android keyboard to investigate users' typing behaviour. 47,770,625 keystrokes from 72,945 installations have been collected by publishing the game. By visualizing the touch distribution we identified a systematic skew and derived a function that compensates this skew by shifting touch events. By updating the game we conduct an experiment that investigates the effect of shifting touch events, changing the keys' labels, and visualizing the touched position. Results based on 6,603,659 keystrokes and 13,013 installations show that visualizing the touched positions using a simple dot decreases the error rate of the Android keyboard by 18.3% but also decreases the speed by 5.2% with no positive effect on learnability. The Android keyboard outperforms the control condition but the constructed shift function further improves the performance by 2.2% and decreases the error rate by 9.1%. We argue that the shift function can improve existing keyboards at no costs.

Mobile application stores such as Apple's App Store and Google's Android Market enable researchers to use App Stores as a platform for user studies. Because successful studies can require a large number of users researchers might need to attract a large audience. The right timing when releasing or updating apps can considerably increase the number of installations. Using a game that is published in the Android Market we analyze when people install games. Furthermore, we determine when developers deploy games in the Android Market. We combine data from 157,438 installations of the game and the observation of 24,647 published apps. Our results suggest that the best time to deploy a game in the Android Market is on Sunday evening GMT.

Touchscreens became the dominant input device for smartphones. Users' touch behaviour has been widely studied in lab studies with a relative low number of participants. In contrast, we published a game in the Android Market that records the touch behaviour when executing a controlled task to collect large amounts of touch events. Players' task is to simply touch circles appearing on the screen. Data from 91,731 installations has been collected and players produced 120,626,225 touch events. We determined the error rates for different target sizes and screen locations. The amount of data enabled us to show that touch positions are systematically skewed. A compensation function that shifts the users' touches to reduce the amount of errors is derived from the data and evaluated by publishing an update of the game. The independent-measures experiment with data from 12,201 installations and 15,326,444 touch events shows that the function reduces the error rate by 7.79%. We argue that such a compensation function could improve the touch performance of virtually every smartphone user.

Map navigation is often limited due to the inherent size re-strictions of mobile devices ’ displays. Using a magic lens to interact with physical objects has been proposed as a way to reduce this limitation. The dynamic peephole interface is an alternative approach where a device is moved across a virtual surface. In this paper we study the effect of an additional visualization of objects beyond the screen on magic lens and dynamic peephole interfaces. In the conducted experiment the participants had to select points of interest shown on a map. We show that an additional visualization of off-screen objects decreases the task completion time and reduces the perceived task load. The advantage of an off-screen visual-ization is much higher than the difference between using a magic lens instead of a dynamic peephole interface.

The introduction of publicly available application stores for mobile devices enables to publish research prototypes to a wide audience. This distribution channel can be used to conduct studies with participants from all over the world and diverse backgrounds. We report from a study that compares three visualization techniques for off-screen objects on digital maps. Usage data from 362 persons was collected and 105 persons completed an interactive tutorial. Significant differences between the three conditions were found. The results support previous findings but we conjecture that the results are affected by unintended influences.

An emerging technology for tourism information systems is mobile Augmented Reality using the position and orientation sensors of recent smartphones. State-of-the-art mobile Augmented Reality application accompanies the Augmented Reality visualization with a small mini-map to provide an overview of nearby points of interest (POIs). In this paper we develop an alternative visualization for nearby POIs based on off-screen visualization techniques for digital maps. The off-screen visualization uses arrows directly embedded into the Augmented Reality scene which point at the POIs. In the conducted study 26 participants explored nearby POIs and had to interpret their position. We show that participants are faster and can interpret the position of POIs more precisely with the developed visualization technique.

Music is a fundamental part of most cultures. Controlling music playback has commonly been used to demonstrate new interaction techniques and algorithm. In particular, controlling music playback has been used to demonstrate and evaluate gesture recognition algorithms. Previous work, however, used gestures that have been defined based on intuition, the developers' preferences, and the respective algorithm's capabilities. In this paper we propose a refined process for deriving gestures from constant user feedback. Along this process a set of free-hand gestures for controlling music playback is developed. The situational context is analyzed to shape the usage scenario and derive an initial set of necessary functions. In a successive user study the set of functions is validated. Furthermore, proposals for gestures are collected from the participants for each function. Two gesture sets containing static and dynamic gestures are derived and analyzed in a comparative evaluation. The evaluation shows that we developed an appropriate set of free-hand gestures for music playback. Our results indicate that the proposed process, that includes validation of each design decision, improves the final results.

Paper maps are a proven means for navigating in unfamiliar environments, however, they do not prevent people from getting lost or taking unwanted detours. A well-known issue is interpreting the map's geocentric content, which is known to become prone to errors when the map is not aligned to the environment. In this paper we report our investigation of providing a cue about the destination's location from an egocentric perspective in order to improve the interpretation of the map. We used a vibrotactile belt to continuously indicate a destination's direction relative to the user's orientation. In an outdoor field study we compared the performance of map-based navigation with and without the added tactile cue. We found evidence that people take shorter routes, consult the map less often, and were less often disoriented with the tactile cue. Furthermore, females found the tactile cue more useful and used it more often.

The workshop on Mobile Interaction with the Real World (MIRW 2009) will invite papers which focus on new mobile and wearable input and output interfaces which allow simpler and straightforward interactions with mobile services and applications. An inherit problem of current mobile devices are their limited output and input capabilities. This workshop continues a successful series of workshops (2006--2008) that focus on new approaches to overcome these issues. Examples are the usage of external visual interfaces (e.g. projector phones, public displays, interactive surfaces) and additional input capabilities (e.g. gestures, on-body interfaces, pointing) and innovative feedback mechanisms (e.g. tactile feedback). The workshop combines technical presentations with the presentation of prototypes and focused discussions to drive interaction between participants.

The Mobile HCI community is moving beyond the interaction between a single user and her mobile device taking the users environment into account. Mobile interaction with the real world concentrates on using mobile devices as tools to interact with real world objects. This workshop continues the successful mobile interaction with the real world workshops 2006 and 2007. Relevant topics include (but are not limited to) mobile interaction with the real world; mobile devices as user interfaces for terminals; and Frameworks, middleware and APIs for the development of applications that take mobile interactions with the real world into account. The workshop combines technical presentations with the presentation of prototypes and focused discussions to drive interaction between participants.

In our everyday life we often see objects or persons and are aware that there are related digital services such as an online ticket service when seeing a poster advertising a concert. Currently it is a rather time consuming activity to find the related information. Using our Contextual Bookmark system a user can define a snapshot with her mobile phone consisting of a picture, time stamp and location. Such a bookmark can then be stored on the mobile phone, exchanged with friends and in particular be used to access related videos, web pages, and other services. This helps the user to bridge the gap between the virtual and the real world in order to use related services. Combining content and context analysis objects are recognized without any visual markers or electronic tags. We would like to demonstrate our system based on a nomadic usage scenario in which a person defines a Contextual Bookmark of a movie trailer, buys the corresponding movie, plays the movie on a TV, and exchanges the bookmark with a friend.

The sharing of photos with others, friends and family, has become even more popular with digital photography and Internet applications such as email or Web albums. At the same time, the physical touch of printed photos is still appreciated and customers use different services to print their photos on post cards, calendars or photobooks, often to give them as a present or to create a physical souvenir. Once printed, however, the sharing of photos with others becomes difficult as there is no link back from the now physical item to its digital counterpart. With Bookmarkr, we developed a system that employs a mobile camera phone to bridge the gap between the printed photo and its digital counterpart. The user takes an image of a photo in a photobook with the mobile phone's camera. The image is transmitted to a photobook server, which employs image analysis techniques to retrieve the corresponding photo. The photo is sent back to the user and establishes the digital-physical match. Bookmarkr allows the user to interact with printed photos in a similar way as interacting with digital photos by using a point-and-shoot metaphor. Our performance evaluation shows that the system is able to return the correct photo in up to 99% of all cases. A conducted user study revealed that the designed interaction is suitable for the participants and their printed photobooks.

In many applications today user interaction is moving away from mouse and pens and is becoming pervasive and much more physical and tangible. New emerging interaction technologies allow developing and experimenting with new interaction methods on the long way to providing intuitive human computer interaction. In this paper, we aim at recognizing gestures to interact with an application and present the design and evaluation of our sensor-based gesture recognition. As input device we employ the Wii-controller (Wiimote) which recently gained much attention world wide. We use the Wiimote's acceleration sensor independent of the gaming console for gesture recognition. The system allows the training of arbitrary gestures by users which can then be recalled for interacting with systems like photo browsing on a home TV. The developed library exploits Wii-sensor data and employs a hidden Markov model for training and recognizing user-chosen gestures. Our evaluation shows that we can already recognize gestures with a small number of training samples. In addition to the gesture recognition we also present our experiences with the Wii-controller and the implementation of the gesture recognition. The system forms the basis for our ongoing work on multimodal intuitive media browsing and are available to other researchers in the field.