We explore the metaphorical "daily memory pill"concept - a brief pictorial lifelog recap aimed at reviving and preserving memories. Leveraging psychological strategies, we explore the potential of such summaries to boost autobiographical memory. We developed an automated lifelogging memory prosthesis and a research protocol (Automated Memory Validation "AMV") for conducting privacy-aware, in-situ evaluations. We conducted a real-world lifelogging experiment for a month (n=11). We also designed a browser "Pixel Memories"for browsing one-week worth of lifelogs. The results suggest that daily timelapse summaries, while not yielding significant memory augmentation effects, also do not lead to memory degradation. Participants' confidence in recalled content remains unaltered, but the study highlights the challenge of users' overestimation of memory accuracy. Our core contributions, the AMV protocol and "Pixel Memories"browser, advance our understanding of memory augmentations and offer a privacy-preserving method for evaluating future ubicomp systems.

Learning to read Braille is crucial to academic success for people with blindness or severe visual impairment. In our work, we investigate how we can support early learning of Braille with tangible computing. In a human-centered inclusive design process with interviews, six design iterations with prototypes, and feedback from experts, students, and teachers, we created BrailleBuddy. BrailleBuddy is a tangible user interface supporting children with visual impairments in learning Braille. We evaluated BrailleBuddy in a user study with children with blindness. Our results show that BrailleBuddy provides intrinsic motivation for learning Braille and can be used by children without supervision. BrailleBuddy complements the educational program as it allows children to play with and explore Braille characters at their own pace, thus lowering the challenge of learning to read Braille. In addition, an open-source toolkit is provided to enable educators and researchers to support individual requirements.

Technologies that help users overcome their limitations and integrate with the human body are often termed “human augmentations”. Such technologies are now available on the consumer market, potentially supporting people in their everyday activities. To date, there is no systematic understanding of the perception of human augmentations yet. To address this gap and build an understanding of how to design positive experiences with human augmentations, we conducted a mixed-method study of the perception of augmented humans (AHs). We conducted two scenario-based studies: interviews (n = 16) and an online study (n = 506) with participants from four countries. The scenarios include one out of three augmentation categories (sensory, motor, and cognitive) and specify if the augmented person has a disability or not. Overall, results show that the type of augmentation and disability impacted user attitudes towards AHs. We derive design dimensions for creating technological augmentations for a diverse and global audience.

Many modern household appliances are challenging to operate for people with visual impairment. Low-contrast designs and insufficient tactile feedback make it difficult to distinguish interface elements and to recognize their function. Augmented reality (AR) can be used to visually highlight such elements and provide assistance to people with residual vision. To realize this goal, we (1) created a dataset consisting of 13,702 images of interfaces from household appliances and manually labeled control elements; (2) trained a neural network to recognize control elements and to distinguish between PushButton, TouchButton, Knob, Slider, and Toggle; and (3) designed various contrast-rich and visually simple AR augmentations for these elements. The results were implemented as a screen-based assistive AR application, which we tested in a user study with six individuals with visual impairment. Participants were able to recognize control elements that were imperceptible without the assistive application. The approach was well received, especially for the potential of familiarizing oneself with novel devices. The automatic parsing and augmentation of interfaces provide an important step toward the independent interaction of people with visual impairments with their everyday environment.

Eye tracking is increasingly becoming prevalent for health-related interactive systems. Eye tracking can automatically reveal the presence of Central Field Loss (CFL), a dysfunctional visual behavior requiring time-intensive medical assessments. Since CFL typically results in poor fixation stability and more frequent saccades, this work investigates the use of machine learning to estimate the likelihood of CFL based on eye-movement data. We compared random forests, support vector machines, and long-short-term memory (LSTM) neural networks for their ability to discriminate between the presence or absence of an experimentally-induced CFL. We found that the estimation accuracy increases with larger samples of eye-tracking data. However, the computational costs outweigh any increase in accuracy after classifying window sizes of 1600 msec. Here, traditional machine learning approaches outperform the LSTM neural network. We discuss implications for continuous end-user CFL monitoring and processing power to provide an outlook for gaze-based wearable health devices in human-computer interaction.

Modern games make creative use of First- and Third-person perspectives (FPP and TPP) to allow the player to explore virtual worlds. Traditionally, FPP and TPP perspectives are seen as distinct concepts. Yet, Virtual Reality (VR) allows for flexibility in choosing perspectives. We introduce the notion of a perspective continuum in VR, which is technically related to the camera position and conceptually to how users perceive their environment in VR. A perspective continuum enables adapting and manipulating the sense of agency and involvement in the virtual world. This flexibility of perspectives broadens the design space of VR experiences through deliberately manipulating perception. In a study, we explore users’ attitudes, experiences and perceptions while controlling a virtual character from the two known perspectives. Statistical analysis of the empirical results shows the existence of a perspective continuum in VR. Our findings can be used to design experiences based on shifts of perception.

As machine learning methods become more powerful and capture more nuances of human behavior, biases in the dataset can shape what the model learns and is evaluated on. This paper explores and attempts to quantify the uncertainties and biases due to annotator demographics when creating sentiment analysis datasets. We ask >1000 crowdworkers to provide their demographic information and annotations for multimodal sentiment data and its component modalities. We show that demographic differences among annotators impute a significant effect on their ratings, and that these effects also occur in each component modality. We compare predictions of different state-of-the-art multimodal machine learning algorithms against annotations provided by different demographic groups, and find that changing annotator demographics can cause >4.5 in accuracy difference when determining positive versus negative sentiment. Our findings underscore the importance of accounting for crowdworker attributes, such as demographics, when building datasets, evaluating algorithms, and interpreting results for sentiment analysis.

Mid-air haptics allow bare-hand tactile stimulation; however, it has a constrained workspace, making it unsuitable for room-scale haptics. We present a novel approach to rendering mid-air haptic sensations in a large rendering volume by turning a static array into a dynamic array following the user's hand. We used a 6DOF robot to drive a haptic ultrasound array over a large 3D space. Our system enables rendering room-scale mid-air experiences while preserving bare-hand interaction, thus, providing tangibility for virtual environments. To evaluate our approach, we performed three evaluations. First, we performed a technical system evaluation, showcasing the feasibility of such a system. Next, we conducted three psychophysical experiments, showing that the motion does not affect the user's perception with high likelihood. Lastly, we explored seven use cases that showcase our system's potential using a user study. We discuss challenges and opportunities in how large-scale mid-air haptics can contribute toward room-scale haptic feedback. Thus, with our system, we contribute to general haptic mid-air feedback on a large scale.

Partial vision loss occurs in several medical conditions and affects persons of all ages. It compromises many daily activities, such as reading, cutting vegetables, or identifying and accurately pressing buttons, e.g., on ticket machines or ATMs. Touchscreen interfaces pose a particular challenge because they lack haptic feedback from interface elements and often require people with impaired vision to rely on others for help. We propose a smartglasses-based solution to utilize the user’s residual vision. Together with visually-impaired individuals, we designed assistive augmentations for touchscreen interfaces and evaluated their suitability to guide attention towards interface elements and to increase the accuracy of manual inputs. We show that augmentations improve interaction performance and decrease cognitive load, particularly for unfamiliar interface layouts.

Virtual Reality experiences and games present believable virtual environments based on graphical quality, spatial audio, and interactivity. The interaction with in-game characters, controlled by computers (agents) or humans (avatars), is an important part of VR experiences. Pre-captured motion sequences increase the visual humanoid resemblance. However, this still precludes realistic social interactions (eye contact, imitation of body language), particularly for agents. We aim to make social interaction more realistic via social touch. Social touch is non-verbal, conveys feelings and signals (coexistence, closure, intimacy). In our research, we created an artificial hand to apply social touch in a repeatable and controlled fashion to investigate its effect on the perceived human-likeness of avatars and agents. Our results show that social touch is effective to further blur the boundary between computer- and human-controlled virtual characters and contributes to experiences that closely resemble human-to-human interactions.

Abstract Many individuals with visual impairments have residual vision that often remains underused by assistive technologies. Head-mounted augmented reality (AR) devices can provide assistance, by recoding difficult-to-perceive information into a visual format that is more accessible. Here, we evaluate symbolic and alphanumeric information representations for their efficiency and usability in two prototypical AR applications: namely, recognizing facial expressions of conversational partners and reading the time. We find that while AR provides a general benefit, the complexity of the visual representations has to be matched to the user’s visual acuity.

Cognitive performance fluctuates throughout the day partly due to our “inner clock”. Systems that are aware of these fluctuations can adapt to their users’ momentary capacity and adjust task difficulty accordingly. Here, we evaluate the use of performance measures obtained from mobile game data to estimate players’ varying alertness. We developed a smartphone game that emulates three validated tasks for cognitive performance and conducted an in-the-wild study over two weeks with 30 participants to collect performance measures and subjective ground truth on alertness levels. Our results show that players’ performance can be explained by a generative model which is based on two established alertness-modulating processes, namely the homeostatic process and circadian alertness fluctuations. Our method can be used to implicitly model players’ alertness levels as a basis for cognition-aware applications.

While Virtual Reality continues to increase in fidelity, it remains an open question how to effectively reflect the user's movements and provide congruent feedback in virtual environments. We present VRsneaky, a system for producing auditory movement feedback, which helps participants orient themselves in a virtual environment by providing footstep sounds. The system reacts to the user's specific gait features and adjusts the audio accordingly. In a user study with 28 participants, we found that VRsneaky increases users' sense of presence as well as awareness of their own posture and gait. Additionally, we find that increasing auditory realism significantly influences certain characteristics of participants' gait. Our work shows that gait-aware audio feedback is a means to increase presence in virtual environments. We discuss opportunities and design requirements for future scenarios where users walk through immersive virtual worlds.

Since the release of the first activity tracker, there has been a steady increase in the number of sensors embedded in wearable devices and with it in the amount and diversity of information that can be derived from these sensors. This development leads to novel privacy threats for users. In a web survey with 248 participants, we explored whether users' willingness to share private data is dependent on how the data is requested by an application. Specifically, requests can be formulated as access to sensor data or as access to information derived from the sensor data (e.g., accelerometer vs. sleep quality). We show that non-expert users lack an understanding of how the two representation levels relate to each other. The results suggest that the willingness to share sensor data over derived information is governed by whether the derived information has positive or negative connotations (e.g., training intensity vs. life expectancy). Using the results of the survey, we derive implications for supporting users in protecting their private data collected via wearable sensors.

Tactile overlays for touch-screen devices are an opportunity to display content for users with visual impairments. However, when users switch tactile overlays, the displayed content on the touch-screen devices still correspond to the previous overlay. Currently, users have to change the displayed content on the touch-screen devices manually which hinders a fluid user interaction. In this paper, we introduce self-identifying overlays - an automated method for touch-screen devices to identify tactile overlays placed on the screen and to adapt the displayed content based on the applied tactile overlay. We report on a pilot study with two participants with visual impairments to evaluate this approach with a functional content exploration application based on an adapted textbook.

The number of sensors in our surroundings that provide the same information steadily increases. Since sensing is prone to errors, sensors may disagree. For example, a GPS-based tracker on the phone and a sensor on the bike wheel may provide discrepant estimates on traveled distance. This poses a user dilemma, namely how to reconcile the conflicting information into one estimate. We investigated whether visualizing the uncertainty associated with sensor measurements improves the quality of users' inference. We tested four visualizations with increasingly detailed representation of uncertainty. Our study repeatedly presented two sensor measurements with varying degrees of inconsistency to participants who indicated their best guess of the "true" value. We found that uncertainty information improves users' estimates, especially if sensors differ largely in their associated variability. Improvements were larger for information-rich visualizations. Based on our findings, we provide an interactive tool to select the optimal visualization for displaying conflicting information.

We present VRHapticDrones, a system utilizing quadcopters as levitating haptic feedback proxy. A touchable surface is attached to the side of the quadcopters to provide unintrusive, flexible, and programmable haptic feedback in virtual reality. Since the users' sense of presence in virtual reality is a crucial factor for the overall user experience, our system simulates haptic feedback of virtual objects. Quadcopters are dynamically positioned to provide haptic feedback relative to the physical interaction space of the user. In a first user study, we demonstrate that haptic feedback provided by VRHapticDrones significantly increases users' sense of presence compared to vibrotactile controllers and interactions without additional haptic feedback. In a second user study, we explored the quality of induced feedback regarding the expected feeling of different objects. Results show that VRHapticDrones is best suited to simulate objects that are expected to feel either light-weight or have yielding surfaces. With VRHapticDrones we contribute a solution to provide unintrusive and flexible feedback as well as insights for future VR haptic feedback systems.

We often base our decisions on uncertain data - for instance, when consulting the weather forecast before deciding what to wear. Due to their uncertainty, such forecasts can differ by provider. To make an informed decision, many people compare several forecasts, which is a time-consuming and cumbersome task. To facilitate comparison, we identified three aggregation mechanisms for forecasts: manual comparison and two mechanisms of computational aggregation. In a survey, we compared the mechanisms using different representations. We then developed a weather application to evaluate the most promising candidates in a real-world study. Our results show that aggregation increases users' confidence in uncertain data, independent of the type of representation. Further, we find that for daily events, users prefer to use computationally aggregated forecasts. However, for high-stakes events, they prefer manual comparison. We discuss how our findings inform the design of improved interfaces for comparison of uncertain data, including non-weather purposes.

Perception is fundamentally underconstrained because different combinations of object properties can generate the same sensory information. To disambiguate sensory information into estimates of scene properties, our brains incorporate prior knowledge and additional "auxiliary" (i.e., not directly relevant to desired scene property) sensory information to constrain perceptual interpretations. For example, knowing the distance to an object helps in perceiving its size. The literature contains few demonstrations of the use of prior knowledge and auxiliary information in combined visual and haptic disambiguation and almost no examination of haptic disambiguation of vision beyond "bistable" stimuli. Previous studies have reported humans integrate multiple unambiguous sensations to perceive single, continuous object properties, like size or position. Here we test whether humans use visual and haptic information, individually and jointly, to disambiguate size from distance. We presented participants with a ball moving in depth with a changing diameter. Because no unambiguous distance information is available under monocular viewing, participants rely on prior assumptions about the ball's distance to disambiguate their -size percept. Presenting auxiliary binocular and/or haptic distance information augments participants' prior distance assumptions and improves their size judgment accuracy-though binocular cues were trusted more than haptic. Our results suggest both visual and haptic distance information disambiguate size perception, and we interpret these results in the context of probabilistic perceptual reasoning.