We address in this work the topic of broadcasting one’s visual reality, captured by the video cameras embedded in mobile and wearable devices, to a remote audience. We discuss several designs of such life broadcasting systems, which we position at the intersection of lifelogging, Alternate Reality, and Cross-Reality technology. To this end, we introduce the “Alternate Reality Broadcast-Time” matrix for the broadcasting and consumption of alternate realities, in which designs of systems that implement sharing and consumption of personal visual realities can be positioned, characterized, and compared. These design options range from simple video streaming over the web using conventional video protocols to mediated and augmented reality, to audio narration and vibrotactile rendering of concepts automatically detected from video captured by wearable cameras. To demonstrate the usefulness of our broadcast-time matrix, we describe three prototypes implementing lifelogging, concept recognition from video, augmented and mediated vision, and vibrotactile feedback. Our contributions open the way toward new applications that blend lifelogging, consumption of multimedia alternate realities, and XR technology to empower users with richer opportunities for self-expression and new means to connect with the followers of events in their lives.

Augmented Reality Television (ARTV) can take many forms, from AR content displayed outside the TV frame to video-projected TV screens to social TV watching in VR to immersive holograms in the living room. While the user experience (UX) of individual forms of ARTV has been documented before, “journeys” as transitions between such forms have not. In this work, we examine the UX of watching TV when switching between various levels of augmentation. Our findings from an experiment with fourteen participants reveal an UX characterized by high perceived usability, captivation, and involvement with a low to medium workload and a moderate feeling of dissociation from the physical world. We interpret our results in the context of Garrett’s established five-plane model of UX—strategy, scope, structure, skeleton, and surface—and propose a sixth plane, “switch,” which separates conceptually the design of user journeys in ARTV from the specifics of the other UX planes.

We address gesture input for TV control, for which we examine mid-air free-hand interactions that can be detected via radar sensing. We adopt a scenario-based design approach to explore possible locations from the living room where to integrate radar sensors, e.g., in the TV set, the couch armrest, or the user’s smartphone, and we contribute a four-level taxonomy of locations relative to the TV set, the user, personal robot assistants, and the living room environment, respectively. We also present preliminary results about an interactive system using a 15-antenna ultra-wideband 3D radar, for which we implemented a dictionary of six directional swipe gestures for the control of dichotomous TV system functions.

We present XR4ISL, an XR system designed to support psychology experiments examining Implicit Social Learning, a fundamental phenomenon that guides human behavior, cognition, and emotion. We discuss XR4ISL with reference to MR4ISL, a previous system designed for Mixed Reality only, and reflect on differences between Mixed and Virtual Reality for psychology experiments.

Smart eyewear and Augmented Reality technology have been examined closely in the scientific community to provide vision rehabilitation to people with visual impairments as well as augmented vision to people with and without visual impairments for various application scenarios and contexts of use. However, current systems lack flexibility in the configuration and customization of the features and functionalities they present to their users, as we show in this paper by means of a thorough literature review and categorization of prior work on augmented and mediated vision for smart eyewear devices. To address the flexibility aspect that has been missing in prior work, we introduce FlexiSee, an application for smart eyewear devices, such as see-through Augmented Reality glasses and Head-Mounted Mixed Reality Displays, specifically designed to enable flexible configuration, customization, and control of both augmented and mediated vision. FlexiSee achieves this desiderata by implementing visual filters (e.g., color correction, edge highlighting, contrast adjustment, and others) that are coupled with a web-based interface, readily accessible from smartphones, tablets, smartwatches, and other devices with web browsers, where authorized users can specify and apply custom parameters for the visual filters implemented by FlexiSee. We also introduce FlexiSee-DS, a three-dimensional design space for FlexiSee-like applications, that includes mediation & augmentation, user categories, and control design dimensions to specify a variety of FlexiSee-like systems. We show how the dimensions of FlexiSee-DS were applied to inform the design of our FlexiSee system, and we highlight and focus on the distinction between primary users and vision monitors and assistants, where the latter two categories represent new types of users for augmented and mediated vision that have various degrees of control, from their remote locations, over the visual reality delivered to and perceived by the primary users of smart eyewear devices. We conduct a user study to understand the perception of vision monitors and assistants regarding our new FlexiSee concept and system, and we report empirical results about usability aspects (e.g., we found an average SUS score of 75.3 and high ratings for the perceived usefulness of FlexiSee) as well as user feedback and suggestions to inform further developments of FlexiSee-like systems and applications.

We discuss engineering aspects for shifting from “do you see what I see?” applications that stream the user’s field of view to remote viewers toward “do you control what I see?” features in which remote viewers are given the opportunity and tool to control the primary user’s field of view. To this end, we present two applications for (1) smartglasses with embedded video camera for live video streaming and (2) the HoloLens HMD that presents users with mediated versions of the visual world controlled by remote viewers.

With this position paper, we want to draw the attention of the community toward theoretical work and practical opportunities that have been overlooked so far regarding concepts, principles, and design knowledge for the user experience of Augmented and Mixed Reality content, I/O devices, interactions, applications, and systems. Despite considerable innovations in commercial products and research prototypes enabling Augmented and Mixed Reality worlds, how to design great user experiences in such worlds has been overall neglected at core, while the information and knowledge currently available to practitioners cover usability aspects mostly. Therefore, we advocate for theoretical foundations of the user experience in Augmented and Mixed Reality and propose several directions for more scientific research in this regard.