We propose an inertial sensor-based approach to activity recognition for pedestrian indoor navigation. In the considered scenario a mobile device is held in a hand in front of the user. The recognized activities are the ones relevant to positioning in multi-floor buildings: walking and going up or down the stairs. To model the time dependency between consecutive activities we employ a Hidden Markov Model (HMM). For efficient quantization of continuous features, we apply a random forest classifier. For verification of the proposed algorithm, we conducted experiments with 12 participants and 4 different mobile devices. In our comparison to state-of-the-art approaches, we implement and evaluate major classification algorithms, such as nearest neighbour, decision tree and dynamic Bayesian Network. In the experiments we show the trade-off between computational complexity and classification performance. Furthermore, we demonstrate that the complexity of the HMM can be significantly reduced by replacing it with a dynamic Bayesian network with negligible impact on classification performance. The best of our proposed classifier achieves a classification accuracy of 91% for new users, which offers a 30% improvement compared to state-of-the-art approaches.

We propose a graph-based, low-complexity sensor fusion approach for ubiquitous pedestrian indoor positioning using mobile devices. We employ our fusion technique to combine relative motion information based on step detection with WiFi signal strength measurements. The method is based on the well-known particle filter methodology. In contrast to previous work, we provide a probabilistic model for location estimation that is formulated directly on a fully discretized, graph-based representation of the indoor environment. We generate this graph by adaptive quantization of the indoor space, removing irrelevant degrees of freedom from the estimation problem. We evaluate the proposed method in two realistic indoor environments using real data collected from smartphones. In total, our dataset spans about 20 kilometers in distance walked and includes 13 users and four different mobile device types. Our results demonstrate that the filter requires an order of magnitude less particles than state-of-the-art approaches while maintaining an accuracy of a few meters. The proposed low-complexity solution not only enables indoor positioning on less powerful mobile devices, but also saves much-needed resources for location-based end-user applications which run on top of a localization service.

Recent progress in the field of content-based image retrieval has enabled camera-based indoor positioning. The matching of smart-phone recordings with a database of geo-referenced images allows for meter accurate infrastructure-free localization. In mobile scenarios, however, three major constraints have to be considered: limited computational resources of mobile devices, limited network capacity and the need for scalability in large buildings. To address these issues, we modify the state-of-the-art Vector of Locally Aggregated Descriptors (VLAD) image signature to work with recently emerging binary feature descriptors. We show that this results in a substantial reduction in the overall computational complexity, which enables the matching of image signatures directly on the mobile device. The specific properties of this signature form the basis of our proposed scalable streaming approach that preemptively loads image signatures of reference images in the vicinity of the user onto the mobile device to mitigate the effect of network latency. In order to provide efficient streaming, we compress the signatures by exploiting the similarities of spatially neighboring reference images. In combination, the contributions of this paper lead to an indoor localization system, which allows instantaneous camera-based indoor positioning with very low requirements on the available network connection.

We present a fast and highly performant multiscale feature detector which is based on the established SURF algorithm. The additional speed-up is achieved by linearizing the SURF detector in a way that its detection characteristics are preserved. Our evaluations show that the proposed suSURF detector is roughly 30% faster without significant sacrifices in feature quality. The key points detected by suSURF are compatible with standard SURF features and those found by other determinant of Hessian (DoH) based detectors.

State-of-the-art visual odometry algorithms achieve remarkable efficiency and accuracy. Under realistic conditions, however, tracking failures are inevitable and to continue tracking, a recovery strategy is required. In this paper, we propose a relocalization system that enables realtime, 6D pose recovery for wide baselines. Our approach targets specifically resource-constrained hardware such as mobile phones. By exploiting the properties of low-complexity binary feature descriptors, nearest-neighbor search is performed efficiently using Locality Sensitive Hashing. Our method does not require time-consuming offline training of hash tables and it can be applied to any visual odometry system. We provide a thorough evaluation of effectiveness, robustness and runtime on an indoor test sequence with available ground truth poses. We investigate the system parameterization and compare the relocalization performance for the three binary descriptors BRIEF, unscaled BRIEF and ORB. In contrast to previous work on mobile visual odometry, we are able to quickly recover from tracking failures within maps with thousands of 3D feature points.

We study the impact of JPEG compression on the performance of an image retrieval system for different feature detector-descriptor combinations. The VLBenchmarks retrieval framework is used to compare a total of 60 detector-descriptor combinations for a dataset with JPEG-encoded query images. Our results show that among all tested detectors, the Hessian-Affine detector leads to the most robust performance in the presence of strong JPEG compression. Additionally, we compare the retrieval gains of the different detector-descriptor pairs after processing the JPEG-encoded query images with different deblocking filters. The results illustrate that for the MSER, MFD and WαSH detectors, the retrieval results benefit from two of the deblocking approaches at low bit rate irrespective of what descriptor the detectors are combined with. The same two deblocking filters are found to increase the retrieval performance for the MROGH descriptor when combined with most of the tested detectors.

Video synchronization is an essential processing step in many multimedia applications, and various methods have been proposed in the literature each of which addresses the problem from a different point of vantage. In this article, we present an information theoretic approach to video synchronization, based on the state-of-the-art in hybrid video coding. Time series derived from the videos' instantaneous bitrate demand are correlated in a robust manner employing the recently published ConCor algorithm. We enhance ConCor with integrated normalization capabilities in order to improve its shape-oriented matching performance. Furthermore, we present a mathematical framework to derive the most suitable ConCor parameters given a specific class of input videos. In an extensive experimental analysis, we give an insight into the representation of synchronization-relevant scene changes with bitrate data, and examine the influence of encoding parameters on the synchronization performance. Experiments on diverse video input substantiate the reliable performance of our easy to implement, yet effective video synchronization algorithm which distinguishes itself in that it operates largely without manual intervention.

Mobile visual location recognition needs to be performed in real-time for location based services to be perceived as useful. We describe and validate an approach that eliminates the network delay by preloading partial visual vocabularies to the mobile device. Retrieval performance is significantly increased by composing partial vocabularies based on the uncertainty about the location of the client. This way, prior knowledge is efficiently integrated into the matching process. Based on compressed feature sets, infrequently uploaded from the mobile device, the server estimates the client location and its uncertainty by fusing consecutive query results using a particle filter.

Recent advances in the field of content-based image retrieval (CBIR) have made it possible to quickly search large image databases using photographs or video sequences as a query. With appropriately tagged images of places, this technique can be applied to the problem of visual location recognition. While this task has attracted large interest in the community, most existing approaches focus on outdoor environments only. This is mainly due to the fact that the generation of an indoor dataset is elaborate and complex. In order to allow researchers to advance their approaches towards the challenging field of CBIR-based indoor localization and to facilitate an objective comparison of different algorithms, we provide an extensive, high resolution indoor dataset. The free for use dataset includes realistic query sequences with ground truth as well as point cloud data, enabling a localization system to perform 6-DOF pose estimation.

We present a visual odometry system for indoor navigation with a focus on long-term robustness and consistency. As our work is targeting mobile phones, we employ monocular SLAM to jointly estimate a local map and the device's trajectory. We specifically address the problem of estimating the scale factor of both, the map and the trajectory. State-of-the-art solutions approach this problem with an Extended Kalman Filter (EKF), which estimates the scale by fusing inertial and visual data, but strongly relies on good initialization and takes time to converge. Each visual tracking failure introduces a new arbitrary scale factor, forcing the filter to re-converge. We propose a fast and robust method for scale initialization that exploits basic geometric properties of the learned local map. Using random projections, we efficiently compute geometric properties from the feature point cloud produced by the visual SLAM system. From these properties (e.g., corridor width or height) we estimate scale changes caused by tracking failures and update the EKF accordingly. As a result, previously achieved convergence is preserved despite re-initializations of the map. To minimize the time required to continue tracking after failure, we perform recovery and re-initialization in parallel. This increases the time available for recovery and hence the likelihood for success, thus allowing almost seamless tracking. Moreover, fewer re-initializations are necessary. We evaluate our approach using extensive and diverse indoor datasets. Results demonstrate that errors and convergence times for scale estimation are considerably reduced, thus ensuring consistent and accurate scale estimation. This enables long-term odometry despite of tracking failures which are inevitable in realistic scenarios.

Determining the pose of a mobile device based on visual information is a promising approach to solve the indoor localization problem. We present an approach that transforms localized images along a mapping trajectory into virtual viewpoints that cover a set of densely sampled camera positions and orientations in a confined environment. The viewpoints are represented by their respective bag-of-features vectors and image retrieval techniques are applied to determine the most likely pose of query images at very low computational complexity. As virtual image locations and orientations are decoupled from actual image locations, the system is able to work with sparse reference imagery and copes well with perspective distortion. Experiments confirm that pose retrieval performance is significantly improved.

Recognizing the location and orientation of a mobile device from captured images is a promising application of image retrieval algorithms. Matching the query images to an existing georeferenced database like Google Street View enables mobile search for location related media, products, and services. Due to the rapidly changing field of view of the mobile device caused by constantly changing user attention, very low retrieval times are essential. These can be significantly reduced by performing the feature quantization on the handheld and transferring compressed Bag-of-Feature vectors to the server. To cope with the limited processing capabilities of handhelds, the quantization of high dimensional feature descriptors has to be performed at very low complexity. To this end, we introduce in this paper the novel Multiple Hypothesis Vocabulary Tree (MHVT) as a step towards real-time mobile location recognition. The MHVT increases the probability of assigning matching feature descriptors to the same visual word by introducing an overlapping buffer around the separating hyperplanes to allow for a soft quantization and an adaptive clustering approach. Further, a novel framework is introduced that allows us to integrate the probability of correct quantization in the distance calculation using an inverted file scheme. Our experiments demonstrate that our approach achieves query times reduced by up to a factor of 10 when compared to the state-of-the-art.

Detecting text in natural images is an important prerequisite. In this paper, we propose a novel text detection algorithm, which employs edge-enhanced Maximally Stable Extremal Regions as basic letter candidates. These candidates are then filtered using geometric and stroke width information to exclude non-text objects. Letters are paired to identify text lines, which are subsequently separated into words. We evaluate our system using the ICDAR competition dataset and our mobile document database. The experimental results demonstrate the excellent performance of the proposed method.

The temporal synchronization of presentation slides and lecture videos enables us to enhance the user experience of online lecture viewing systems. In this work, we present a novel approach to robustly and reliably detect and recognize slide changes, which is based on bit rate sequences. By exploiting readily available information, the computational complexity of the approach is very low. In contrast to prior work, no requirements on the amount or type of texture, motion of foreground objects, or text size on the slide are imposed. Experimental results show the ability to detect even minor slide changes and demonstrate the robustness against occlusions, foreground motion, and camera motion.

We present a novel mobile printed document retrieval system that utilizes both text and low bit-rate features. On the client phone, text are detected using an algorithm based on edge-enhanced Maximally Stable Extremal Regions. The title text image patch is rectified using a gradient based algorithm and recognized using Optical Character Recognition. Low bit-rate image features are extracted from the query image. Both text and compressed features are sent to a server. On the server, the title text is used for on-line search and the features are used for image-based comparison. The proposed system is capable of web-scale document retrieval using title text without the need of constructing a document image database. Using features for image-based comparison, we can reliably match retrieved documents to the query document. Last, by using text and low bit-rate features, we can reduce the transmitted query size significantly.

Distinctive visual cues are of central importance for image retrieval applications, in particular, in the context of visual location recognition. While in indoor environments typically only few distinctive features can be found, outdoors dynamic objects and clutter significantly impair the retrieval performance. We present an approach which exploits text, a major source of information for humans during orientation and navigation, without the need for error-prone optical character recognition. To this end, characters are detected and described using robust feature descriptors like SURF. By quantizing them into several hundred visual words we consider the distinctive appearance of the characters rather than reducing the set of possible features to an alphabet. Writings in images are transformed to strings of visual words termed visual phrases, which provide significantly improved distinctiveness when compared to individual features. An approximate string matching is performed using N-grams, which can be efficiently combined with an inverted file structure to cope with large datasets. An experimental evaluation on three different datasets shows significant improvement of the retrieval performance while reducing the size of the database by two orders of magnitude compared to state-of-the-art. Its low computational complexity makes the approach particularly suited for mobile image retrieval applications.

Cross-correlation is a classical similarity measure with broad applications in multimedia signal processing. While it is robust against uncorrelated noise in the input signals, it is severely affected by systematic disturbances which lead to biased results. To overcome this limitation, we propose in this paper consensus-based cross-correlation (ConCor) to deal with heavily corrupted signal parts that derail regular cross-correlation. ConCor builds upon the widely adopted RANSAC algorithm to reliably identify and eliminate corrupt signal parts at limited additional complexity. Our approach is universal in that it can be combined with existing cross-correlation variants. We apply ConCor in two example applications, namely video synchronization and template matching. Our experimental results demonstrate the improved robustness and accuracy when compared to classical cross-correlation.

We present a novel approach for the temporal synchronization of multiple videos which is based on cross-correlating bit rate profiles. The proposed scheme determines the temporal offset without major restrictions on viewing angles, camera properties and camera motion. We propose two extensions of the basic algorithm which reduce the influence of camera motion and distracting background objects. Additionally, we describe how to optimally combine different bit rate components in order to further improve the reliability of the synchronization scheme. The proposed approach, when combined with the three extensions, leads to a reliable, robust, and frame accurate temporal alignment of videos at remarkably low complexity.

Currently, there is a high demand for autonomous industrial production systems. This paper outlines the development of a cognitive system for autonomous robotic welding. This system is based on dimensionality reduction techniques and Support Vector Machines, allowing the system to learn to separate between acceptable and unacceptable welding results within one batch, and to transfer this ability to a batch with different workpiece properties. It does not aim at a complete and general relationship between all process variables and result quantities, since it has been demonstrated that this is not necessary to reduce significantly the costs of calibrating the welding system. The main objective is to examine a cognitive system that stabilizes robotic welding processes by learning how to improve at least one process steering variable. In order to evaluate and improve the cognitive system, an extensive experimental setup is realized and described. The ability to learn and autonomously adapt to changes in workpiece properties allows the system to reduce the time an expert needs, and relaxes the requirements with respect to workpiece tolerances.