Real-time pose estimation is crucial for autonomous agents for motion control and navigation, and it is essential for extended reality applications as well. As the current global positioning systems are not reliable indoors or are not precise enough, visual simultaneous localization and mapping is becoming prevalent in autonomous agents and mobile devices. The visual feature maps can be shared between agents and might be merged into a common map on a server and redistributed to all clients to enable re-localization and co-localization. However, merged maps can grow continuously in density and size that constrained agents are not able to handle anymore.In this paper, we investigate how the map density effects localization and visual odometry performance on low-cost mobile clients. We show that a sparse representation of the original VFM is necessary for real-time visual odometry but at the cost of a degraded re-localization performance.

We present a method to transform artificial notification sounds into various musical timbres. To tackle the issues of ambiguous timbre definition, the lack of paired notification-music sample sets, and the lack of sufficient training data of notifications, we adapt the problem for a cycle-consistent generative adversarial network and train it with unpaired samples from the source and the target domains. In addition, instead of training the network with notification sound samples, we train it with video game music samples that share similar timbral features. Through a number of experiments, we discuss the efficacy of the model in transferring the timbre of monophonic and even homophonic notifications while preserving their original melody envelopes. We envision notification timbre transfer as a way of less distracting information delivery, and we demonstrate example music pieces augmented with notifications after timbre transfer.

We present a microphone array structure for spherical sound incidence angle tracking that can be attached to headphones or directly integrated into earphones. We show that this microphone array together with an ultrasonic sound source, e.g., a home assistant speaker in the room, allows to estimate the direction and distance of sound reflections on wall surfaces in the room. With our presented method, we achieved sound incidence angle estimation errors of around 14° in the horizontal plane and around 5° in the vertical plane, and reflection position estimation errors of around 5 cm within 2 m. Having the reflection points in 3D allows us to sparsely capture the closest room geometry at interactive rate, which is a necessary step towards indoor audio augmented reality applications.

We present GestEar, a gesture recognition method for sound-emitting gestures, such as snapping, knocking, or clapping, using only a simple smartwatch. Besides the motion information from the built-in accelerometer and gyroscope, we exploit audio data recorded by the smartwatch microphone as input. We propose a lightweight convolutional neural network architecture for gesture recognition, specifically designed to run locally on resource-constrained devices, which achieves a user-independent recognition accuracy of 97.2% for nine distinct gestures. We further show how to incorporate gesture detection and gesture classification in the same network, compare different network designs, and showcase a number of different applications built with our method. We find that the audio input drastically reduces the false positive rate in continuous recognition compared to using only motion.

Coherent rendering in augmented reality deals with synthesizing virtual content that seamlessly blends in with the real content. Unfortunately, capturing or modeling every real aspect in the virtual rendering process is often unfeasible or too expensive. We present a post-processing method that improves the look of rendered overlays in a dental virtual try-on application. We combine the original frame and the default rendered frame in an autoencoder neural network in order to obtain a more natural output, inspired by artistic style transfer research. Specifically, we apply the original frame as style on the rendered frame as content, repeating the process with each new pair of frames. Our method requires only a single forward pass, our shallow architecture ensures fast execution, and our internal feedback loop inherently enforces temporal consistency.

The auditory sense is an intuitive and immersive channel to experience our surroundings, which motivates us to augment our perception of the real world with digital auditory content. We present a wearable audio augmented reality prototype that tracks the user with six degrees of freedom in a known environment, synthesizes 3D sounds, and plays spatialized audio from arbitrary objects to the user. Our prototype is built using head-mounted visual-inertial odometry, a sound simulation engine on a laptop, and off-the-shelf headphones. We demonstrate an application in a gallery scenario in which visitors can hear objects and scenes drawn in the paintings, feeling audio-visually engaged in the depicted surroundings. In a user study involving 26 participants, we observed that the audio-enhanced exhibition improved people's experience, as well as helped them remember more lively details of the artworks.

User interfaces rarely adapt to the specific user preferences or the task at hand. We present a method that allows to quickly and inexpensively create personalized interfaces from plain paper. Users can cut out shapes and assign control functions to these paper snippets via a simple configuration interface. After configuration, control takes place entirely through the manipulation of the paper shapes, providing the experience of a tailored tangible user interface. The shapes and assignments can be dynamically changed during use. Our system is based on markerless tracking of the user's fingers and the paper shapes on a surface using an RGBD camera mounted above the interaction space, which is the only hardware sensor required. Our approach and system are backed up by two studies where we determined what shapes and interaction abstractions users prefer, and verified that users can indeed employ our system to build real applications with paper snippet interfaces.

Common calibration techniques for head-mounted eye trackers rely on markers or an additional person to assist with the procedure. This is a tedious process and may even hinder some practical applications. We propose a novel calibration technique which simplifies the initial calibration step for mobile scenarios. To collect the calibration samples, users only have to point with a finger to various locations in the scene. Our vision-based algorithm detects the users' hand and fingertips which indicate the users' point of interest. This eliminates the need for additional assistance or specialized markers. Our approach achieves comparable accuracy to similar marker-based calibration techniques and is the preferred method by users from our study. The implementation is openly available as a plugin for the open-source Pupil eye tracking platform.

Identifying the finger used for touching and measuring the force of the touch provides valuable information on manual interactions. This information can be inferred from electromyography (EMG) of the forearm, measuring the activation of the muscles controlling the hand and fingers. We present Touch-Sense, which classifies the finger touches using a novel neural network architecture and estimates their force on a smartphone in real time based on data recorded from the sensors of an inexpensive and wireless EMG armband. Using data collected from 18 participants with force ground truth, we evaluate our system's performance and limitations. Our system could allow for new interaction paradigms with appliances and objects, which we exemplarily showcase in four applications.

While spatial audio has been an essential component in Virtual Reality, it has been rarely applied to Augmented Reality. We propose a concept and a prototype to enhance human-object interactions in daily life with 3D audio. We augment real objects in a small workspace around the user with spatial audio notifications.

Visual codes are used to embed digital data in physical objects, or they are shown in video sequences to transfer data over screen/camera links. Existing codes either carry limited data to make them robust against a range of channel conditions (e.g., low camera quality or long distances), or they support a high data capacity but only work over a narrow range of channel conditions. We present FOCUS, a new code design that does not require this explicit trade-off between code capacity and the reader's channel quality. Instead, FOCUS builds on concepts from OFDM to encode data at different levels of spatial detail. This enables each reader to decode as much data from a code as its channel quality allows. We build a prototype of FOCUS devices and evaluate it experimentally. Our results show that FOCUS gracefully adapts to the reader's channel, and that it provides a significant performance improvement over recently proposed designs, including Strata and PixNet.

We present a machine learning technique to recognize gestures and estimate metric depth of hands for 3D interaction, relying only on monocular RGB video input. We aim to enable spatial interaction with small, body-worn devices where rich 3D input is desired but the usage of conventional depth sensors is prohibitive due to their power consumption and size. We propose a hybrid classification-regression approach to learn and predict a mapping of RGB colors to absolute, metric depth in real time. We also classify distinct hand gestures, allowing for a variety of 3D interactions. We demonstrate our technique with three mobile interaction scenarios and evaluate the method quantitatively and qualitatively.

The built-in cameras and powerful processors have turned smartphones into ubiquitous barcode scanners. In smartphone-based barcode scanning, barcode localization is an important preprocessing step that quickly scans the entire camera image and passes barcode candidates to the actual decoder. This paper presents the implementation steps of a robust joint 1D and 2D barcode localization algorithm on the mobile GPU. The barcode probability maps are derived from the structure matrix and the color of the individual pixels. The different steps of the localization algorithm are formulated as OpenGL ES 2.0 fragment shaders and both 1D and 2D barcode saliency maps are computed directly on the graphics hardware. The presented method can detect barcodes at various scales and orientations at 6 frames per second in HD resolution images on current generation smartphones.

We present a novel machine learning based algorithm extending the interaction space around mobile devices. The technique uses only the RGB camera now commonplace on off-the-shelf mobile devices. Our algorithm robustly recognizes a wide range of in-air gestures, supporting user variation, and varying lighting conditions. We demonstrate that our algorithm runs in real-time on unmodified mobile devices, including resource-constrained smartphones and smartwatches. Our goal is not to replace the touchscreen as primary input device, but rather to augment and enrich the existing interaction vocabulary using gestures. While touch input works well for many scenarios, we demonstrate numerous interaction tasks such as mode switches, application and task management, menu selection and certain types of navigation, where such input can be either complemented or better served by in-air gestures. This removes screen real-estate issues on small touchscreens, and allows input to be expanded to the 3D space around the device. We present results for recognition accuracy (93% test and 98% train), impact of memory footprint and other model parameters. Finally, we report results from preliminary user evaluations, discuss advantages and limitations and conclude with directions for future work.

With the proliferation of built-in cameras barcode scanning on smartphones has become widespread in both consumer and enterprise domains. To avoid making the user precisely align the barcode at a dedicated position and angle in the camera image, barcode localization algorithms are necessary that quickly scan the image for possible barcode locations and pass those to the actual barcode decoder. In this paper, we present a barcode localization approach that is orientation, scale, and symbology (1D and 2D) invariant and shows better blur invariance than existing approaches while it operates in real time on a smartphone. Previous approaches focused on selected aspects such as orientation invariance and speed for 1D codes or scale invariance for 2D codes. Our combined method relies on the structure matrix and the saturation from the HSV color system. The comparison with three other real-time barcode localization algorithms shows that our approach outperforms the state of the art with respect to symbology and blur invariance at the expense of a reduced speed.

As the Internet of Things finds its way into private households, more and more everyday objects communicate with services that are running inside the home and on the Internet. For individuals to trust their smart homes, they should be aware of possibly privacy-sensitive data flows and control commands. In this demo paper, we present a system that combines a real time network analysis tool with an augmented reality user interface to visualize data streams within the home network and to remote services. Our system requires no modifications to a typical home network infrastructure, as it operates by merely observing packets sent over the network.

Smartphones and tablets are increasingly used to scan visual codes that act as physical hyperlinks to digital information. Compared with the outstanding performance of enterprise laser scanners, smartphone cameras suffer from defocus and motion blur. In this project, we propose to turn every smartphone into an enterprise-grade barcode scanner by adapting the latest research results in photograph restoration to the very specific properties of barcode images.

Visualization systems often require large monitors or projection screens to display complex information. Even very sophisticated systems that exhibit complex user interfaces do usually not exploit advanced input and output devices. One of the reasons for that is the high cost of special hardware. This paper introduces Augmented Visualization, an interaction method for projection walls as well as monitors using affordable and widely available hardware such as mobile phones or tablets. The main technical challenge is the tracking of the users' devices without any special equipment or fiducial markers in the working area. We propose to track natural features of the display content with the built-in camera of mobile devices. Tracking the visualized scene allows pose estimation of the mobile devices with six degrees of freedom. The position and orientation information is then used for advanced interaction metaphors like magic lenses. For a group of experts who are analyzing the data in front of the same screen, a personal augmented view of the visualized scene is presented, for each user on his/her personal device. The prototype Augmented Visualization System achieves interactive frame rates and may lead to a greatly enhanced user experience. The paper discusses the design and implementation questions and illustrates potential application scenarios.