Locating a target based on auditory and visual cues—such as finding a car in a crowded parking lot or identifying a speaker in a virtual meeting—requires balancing effort, time, and accuracy under uncertainty. Existing models of audiovisual search often treat perception and action in isolation, overlooking how people adaptively coordinate movement and sensory strategies. We present Sensonaut, a computational model of embodied audiovisual search. The core assumption is that people deploy their body and sensory systems in ways they believe will most efficiently improve their chances of locating a target, trading off time and effort under perceptual constraints. Our model formulates this as a resource-rational decision-making problem under partial observability. We validate the model against newly collected human data, showing that it reproduces both adaptive scaling of search time and effort under task complexity, occlusion, and distraction, and characteristic human errors. Our simulation of human-like resource-rational search informs the design of audiovisual interfaces that minimize search cost and cognitive load.

The spatial flexibility of Extended Reality (XR) allows for personalized, context-aware organization of applications aligned with a user’s tasks and priorities. Notifications play a crucial role here, e.g., informing users of received messages they might otherwise miss. However, questions remain around how attention-grabbing they should be, how much information they should present, and how the presentation should adapt to the message’s context and content. While prior studies examined facets of message notification design, the impact of multimodal notifications and how they could be used holistically to support message awareness has not yet been explored. We address this by evaluating nine audio-visual notifications, investigating usability, interruptibility, preferences, and their use to inform of received messages. Our results show differing effects of multimodal notification designs and that individuals want notification modality and design to vary based on delivered message content. These results offer new insights into developing context-aware multimodal interaction strategies for spatial notifica¬tions and XR messaging.

Augmented Reality (AR) is increasingly positioned as a tool for knowledge work, providing beneficial affordances such as a virtually limitless display space that integrates digital information with the user's physical surroundings. However, for AR to supplant traditional screen-based devices in knowledge work, it must support prolonged usage across diverse contexts. Until now, few studies have explored the effects, opportunities, and challenges of working in AR outside a controlled laboratory setting and for an extended duration. This gap in research limits our understanding of how users may adapt its affordances to their daily workflows and what barriers hinder its adoption. In this article, we present findings from a longitudinal diary study examining how participants incorporated an AR laptop - Sightful's Spacetop EA - into their daily work routines. 14 participants used the device for 40-minute daily sessions over two weeks, collectively completing 103 hours of AR-based work. Through survey responses, workspace photographs, and post-study interviews, we analyzed usage patterns, workspace configurations, and evolving user perceptions. Our findings reveal key factors influencing participants' usage of AR, including task demands, environmental constraints, social dynamics, and ergonomic considerations. We highlight how participants leveraged and configured AR's virtual display space, along with emergent hybrid workflows that involved physical screens and tasks. Based on our results, we discuss both overlaps with current literature and new considerations and challenges for the future design of AR systems for pervasive and productive use.

Extended Reality (XR) interfaces offer engaging user experiences, but their effective design requires a nuanced understanding of user behavior and preferences. This knowledge is challenging to obtain without the widespread adoption of XR devices. We introduce MineXR, a design mining workflow and data analysis platform for collecting and analyzing personalized XR user interaction and experience data. MineXR enables elicitation of personalized interfaces from participants of a data collection: for any particular context, participants create interface elements using application screenshots from their own smartphone, place them in the environment, and simultaneously preview the resulting XR layout on a headset. Using MineXR, we contribute a dataset of personalized XR interfaces collected from 31 participants, consisting of 695 XR widgets created from 178 unique applications. We provide insights for XR widget functionalities, categories, clusters, UI element types, and placement. Our open-source tools and data support researchers and designers in developing future XR interfaces.

Smartphones provide convenient access to vast data collections (e.g., online shops, social media) within a compact, portable form factor. While the prevalent infinite scroll lists address the inherently restricted screen space, they also introduce navigation and orientation challenges. Users often lose track of their position within these lists and find it difficult to efficiently access, compare, and filter items of interest. To address this challenge, we introduce Push2AR, a novel interaction concept that extends the phone’s high-resolution display and familiar touch interaction with the virtual display space offered by Augmented Reality (AR) headsets. Push2AR enables users to transfer individual list items from their phone to its surrounding AR space, facilitating bookmarking, filtering, and side-by-side comparisons while maintaining orientation through visual links to the original scroll position. Our evaluation shows that our approach enhances user experience and reduces subjective workload involved in locating and comparing list items in contrast to conventional phone-only lists.

We introduce SonoHaptics, an audio-haptic cursor for gaze-based 3D object selection. SonoHaptics addresses challenges around providing accurate visual feedback during gaze-based selection in Extended Reality (XR), e. g., lack of world-locked displays in no- or limited-display smart glasses and visual inconsistencies. To enable users to distinguish objects without visual feedback, SonoHaptics employs the concept of cross-modal correspondence in human perception to map visual features of objects (color, size, position, material) to audio-haptic properties (pitch, amplitude, direction, timbre). We contribute data-driven models for determining cross-modal mappings of visual features to audio and haptic features, and a computational approach to automatically generate audio-haptic feedback for objects in the user’s environment. SonoHaptics provides global feedback that is unique to each object in the scene, and local feedback to amplify differences between nearby objects. Our comparative evaluation shows that SonoHaptics enables accurate object identification and selection in a cluttered scene without visual feedback.

Spatial audio in Extended Reality (XR) provides users with better awareness of where virtual elements are placed, and efficiently guides them to events such as notifications, system alerts from different windows, or approaching avatars. Humans, however, are inaccurate in localizing sound cues, especially with multiple sources due to limitations in human auditory perception such as angular discrimination error and front-back confusion. This decreases the efficiency of XR interfaces because users misidentify from which XR element a sound is coming from. To address this, we propose Auptimize, a novel computational approach for placing XR sound sources, which mitigates such localization errors by utilizing the ventriloquist effect. Auptimize disentangles the sound source locations from the visual elements and relocates the sound sources to optimal positions for unambiguous identification of sound cues, avoiding errors due to inter-source proximity and front-back confusion. Our evaluation shows that Auptimize decreases spatial audio-based source identification errors compared to playing sound cues at the paired visual-sound locations. We demonstrate the applicability of Auptimize for diverse spatial audio-based interactive XR scenarios.

Mixed Reality (MR) systems display content freely in space, and present nearly arbitrary amounts of information, enabling ubiquitous access to digital information. This approach, however, introduces clutter and distraction if too much virtual content is shown. We present BlendMR, an optimization-based MR system that blends virtual content onto the physical objects in users’ environments to serve as ambient information displays. Our approach takes existing 2D applications and meshes of physical objects as input. It analyses the geometry of the physical objects and identifies regions that are suitable hosts for virtual elements. Using a novel integer programming formulation, our approach then optimally maps selected contents of the 2D applications onto the object, optimizing for factors such as importance and hierarchy of information, viewing angle, and geometric distortion. We evaluate BlendMR by comparing it to a 2D window baseline. Study results show that BlendMR decreases clutter and distraction, and is preferred by users. We demonstrate the applicability of BlendMR in a series of results and usage scenarios.

Many digital wellbeing tools help users monitor and control social media use on their smartphones by tracking and setting limits on their usage time. Tracking is typically done at the granularity of phone- or app-level; however, recent social media apps provide various features such as direct messaging, comment reading/posting, and content uploading/viewing. While it is possible to track and analyze within-app feature usage, little is known about the effect of granularity on smartphone interventions. We designed and developed FinerMe to explore how the granularity of interventions (app-level vs. feature-level) affects the usage of popular social media such as Instagram and YouTube on smartphones. We conducted a field study with 56 participants over 16 days that consisted of three phases: baseline collection, self-reflection, and self-reflection with restrictive interventions. The results showed that while both app-level and feature-level interventions similarly reduced social media use, feature-level interventions enabled users to spend less time on passive app features related to content consumption (e.g., following feed on Instagram, and viewing comments on YouTube) than app-level interventions. Moreover, when self-reflection is combined with restrictive interventions at the feature-level, users were more reflective on their usage behavior than when done at the app-level.

The recent pandemic, war, and oil crises have caused many to reconsider their need to travel for education, training, and meetings. Providing assistance and training remotely has thus gained importance for many applications, from industrial maintenance to surgical telemonitoring. Current solutions such as video conferencing platforms lack essential communication cues such as spatial referencing, which negatively impacts both time completion and task performance. Mixed Reality (MR) offers opportunities to improve remote assistance and training, as it opens the way to increased spatial clarity and large interaction space. We contribute a survey of remote assistance and training in MR environments through a systematic literature review to provide a deeper understanding of current approaches, benefits and challenges. We analyze 62 articles and contextualize our findings along a taxonomy based on degree of collaboration, perspective sharing, MR space symmetry, time, input and output modality, visual display, and application domain. We identify the main gaps and opportunities in this research area, such as exploring collaboration scenarios beyond one-expert-to-one-trainee, enabling users to move across the reality-virtuality spectrum during a task, or exploring advanced interaction techniques that resort to hand or eye tracking. Our survey informs and helps researchers in different domains, including maintenance, medicine, engineering, or education, build and evaluate novel MR approaches to remote training and assistance. All supplemental materials are available at https://augmented-perception.org/publications/2023-training-survey.html.

Reflecting on stress-related data is critical in addressing one’s mental health. Personal Informatics (PI) systems augmented by algorithms and sensors have become popular ways to help users collect and reflect on data about stress. While prediction algorithms in the PI systems are mainly for diagnostic purposes, few studies examine how the explainability of algorithmic prediction can support user-driven self-insight. To this end, we developed MindScope, an algorithm-assisted stress management system that determines user stress levels and explains how the stress level was computed based on the user’s everyday activities captured by a smartphone. In a 25-day field study conducted with 36 college students, the prediction and explanation supported self-reflection, a process to re-establish preconceptions about stress by identifying stress patterns and recalling past stress levels and patterns that led to coping planning. We discuss the implications of exploiting prediction algorithms that facilitate user-driven retrospection in PI systems.

We demonstrate StressTrendmeter, a mobile app that targets college students for anonymously sharing the source of stress via the form of hashtags, viewing stress topics based on trends, and providing social support through the empathy button and hashtag-based chat.

People experience various stressful events in their daily lives. Receiving social support, especially from peers who went through a similar experience, helps individuals cope with such stress. We propose StressTrendmeter, a mobile application that targets college students for anonymously sharing the source of stress via the form of hashtags, viewing stress topics based on trends, and providing social support through the empathy button and hashtag-based chat. We deployed StressTrendmeter to 222 students from two universities for five weeks. With hashtags and trending features, students found StressTrendmeter (i)helpful to spontaneously yet concisely articulate their stress topics and (ii) easy to browse through and become aware of issues around the campus. Our study reveals that social sharing with StressTrendmeter brought awareness, resonance, and accountability as students empathized and expressed support. Based on our study, we share design implications for social support systems with community awareness.

We are witnessing a trend of users owning multiple data-generating wearable and IoT devices that continuously capture sensor data pertaining to a user's activities and context. Federated Learning is a potential technique to derive meaningful insights from this sensor data in a privacy-preserving way without revealing the raw sensor data to a central server. In this paper, we introduce a new problem setting in this multi-device context called Federated Learning in Multi-Device Local Networks (FL-MDLN). We identify core challenges for FL-MDLN in relation to its federation architecture, and statistical and systems heterogeneity across multiple users and multiple devices. Then, we introduce a new user-as-client (UAC) federation architecture, and propose various device selection strategies to counter statistical and systems heterogeneity in FL-MDLN. Early empirical findings show that our proposed techniques improve model test accuracy as well as battery power efficiency in FL. Based on these findings, we elucidate open research questions and future work in FL-MDLN.

Digital intervention tools against problematic smartphone usage help users control their consumption on smartphones, for example, by setting a time limit on an app. However, today's social media apps offer a mix of quasiessential and addictive features in an app (e.g., Instagram has following feeds, recommended feeds, stories, and direct messaging features), which makes it hard to apply a uniform logic for all uses of an app without a nuanced understanding of feature-level usage behaviors. We study when and why people regret using different features of social media apps on smartphones. We examine regretful feature uses in four smartphone social media apps (Facebook, Instagram, YouTube, and KakaoTalk) by utilizing feature usage logs, ESM surveys on regretful use collected for a week, and retrospective interviews from 29 Android users. In determining whether a feature use is regretful, users considered different types of rewards they obtained from using a certain feature (i.e., social, informational, personal interests, and entertainment) as well as alternative rewards they could have gained had they not used the smartphone (e.g., productivity). Depending on the types of rewards and the way rewards are presented to users, probabilities to regret vary across features of the same app. We highlight three patterns of features with different characteristics that lead to regretful use. First, "following"-based features (e.g., Facebook's News Feed and Instagram's Following Posts and Stories) induce habitual checking and quickly deplete rewards from app use. Second, recommendation-based features situated close to actively used features (e.g., Instagram's Suggested Posts adjacent to Search) cause habitual feature tour and sidetracking from the original intention of app use. Third, recommendation-based features with bite-sized contents (e.g., Facebook's Watch Videos) induce using "just a bit more," making people fall into prolonged use. We discuss implications of our findings for how social media apps and intervention tools can be designed to reduce regretful use and how feature-level usage information can strengthen self-reflection and behavior changes.

While mobile instant messaging (MIM) facilitates ubiquitous interpersonal communication, its constant connectivity could build the expectation of an immediate response to messages, and its notifications flood could cause interruptions at inopportune moments. We examine two design concepts for MIM-private status sharing and sender-controlled notifications-that aim to lower the pressure for an immediate reply and reduce unnecessary interruptions by untimely notifications. Private status sharing reactively reveals a customized status with a selected partner(s) only when the partner has sent a message. Sender-controlled notifications give senders the control of choosing whether to send a notification for their own messages. We built MyButler, an Android app prototype that instantiates these two concepts and integrated it with KakaoTalk, a commercial MIM app. During a two-week field study with 11 pairs (5 couples and 6 friend pairs), participants expressed themselves through a total of 210 different statuses, 64.3% of which indicated the current activity or task of the user. Participants reported that private status sharing enabled them to explain their unavailability and relieved the pressure and expectations for timely attendance. We reveal more findings on the types of privately shared statuses and their roles in MIM communication; the in-situ behaviors and patterns of using sender-controlled notifications; and the motivations of MIM users in choosing whether to alert their messages. In terms of message notifications, senders chose to send 25.4% of the messages without any notification. We found that senders' decisions to alert are affected by the receiver's status, their own status to chat, and the possibility of message content exposure to others through notifications. Based on our findings, we draw insights into how the concepts of private status sharing and sender-controlled notifications can be applied in future designs and explorations.

We propose the design of MyButler, a sender-controlled notification management system that mitigates disruption caused by mobile instant messaging through sharing the receiver's activity information with the sender.

We propose Knocker, a real-time object identification technique with smartphones. Knocker leverages unique impulse signals that are generated by knocking on an object with a smartphone. Knocker does not require any special augmentation for both smartphones and objects.