360-degree filming and head-mounted displays (HMDs) give recorded media a new sense of space. Theatre practitioners' expertise in manipulating spatial interactions has much to contribute to immersive recorded content. Four theatre directors led teams of three actors to stage the same scene for both immersive theatre and for 360-degree filming. Each team was recorded performing the scene at least six times, three in each condition, to extract actors' coordinates. This study establishes how to quantify theatre practitioners' use of spatial interactions and examines the spatial adaptations made when transferring these relationships to 360-degree filming.

We present a fast and highly performant multiscale feature detector which is based on the established SURF algorithm. The additional speed-up is achieved by linearizing the SURF detector in a way that its detection characteristics are preserved. Our evaluations show that the proposed suSURF detector is roughly 30% faster without significant sacrifices in feature quality. The key points detected by suSURF are compatible with standard SURF features and those found by other determinant of Hessian (DoH) based detectors.

Video synchronization is an essential processing step in many multimedia applications, and various methods have been proposed in the literature each of which addresses the problem from a different point of vantage. In this article, we present an information theoretic approach to video synchronization, based on the state-of-the-art in hybrid video coding. Time series derived from the videos' instantaneous bitrate demand are correlated in a robust manner employing the recently published ConCor algorithm. We enhance ConCor with integrated normalization capabilities in order to improve its shape-oriented matching performance. Furthermore, we present a mathematical framework to derive the most suitable ConCor parameters given a specific class of input videos. In an extensive experimental analysis, we give an insight into the representation of synchronization-relevant scene changes with bitrate data, and examine the influence of encoding parameters on the synchronization performance. Experiments on diverse video input substantiate the reliable performance of our easy to implement, yet effective video synchronization algorithm which distinguishes itself in that it operates largely without manual intervention.

Mobile visual location recognition needs to be performed in real-time for location based services to be perceived as useful. We describe and validate an approach that eliminates the network delay by preloading partial visual vocabularies to the mobile device. Retrieval performance is significantly increased by composing partial vocabularies based on the uncertainty about the location of the client. This way, prior knowledge is efficiently integrated into the matching process. Based on compressed feature sets, infrequently uploaded from the mobile device, the server estimates the client location and its uncertainty by fusing consecutive query results using a particle filter.

Recent advances in the field of content-based image retrieval (CBIR) have made it possible to quickly search large image databases using photographs or video sequences as a query. With appropriately tagged images of places, this technique can be applied to the problem of visual location recognition. While this task has attracted large interest in the community, most existing approaches focus on outdoor environments only. This is mainly due to the fact that the generation of an indoor dataset is elaborate and complex. In order to allow researchers to advance their approaches towards the challenging field of CBIR-based indoor localization and to facilitate an objective comparison of different algorithms, we provide an extensive, high resolution indoor dataset. The free for use dataset includes realistic query sequences with ground truth as well as point cloud data, enabling a localization system to perform 6-DOF pose estimation.

Determining the pose of a mobile device based on visual information is a promising approach to solve the indoor localization problem. We present an approach that transforms localized images along a mapping trajectory into virtual viewpoints that cover a set of densely sampled camera positions and orientations in a confined environment. The viewpoints are represented by their respective bag-of-features vectors and image retrieval techniques are applied to determine the most likely pose of query images at very low computational complexity. As virtual image locations and orientations are decoupled from actual image locations, the system is able to work with sparse reference imagery and copes well with perspective distortion. Experiments confirm that pose retrieval performance is significantly improved.

Consensus-based Cross-correlation (ConCor) is a recently presented algorithm for robust synchronization of noisy and corrupted signals. ConCor has a number of interdependent parameters that need to be set correctly to guarantee good performance. In this paper we analyse the effects of the individual parameters on ConCor's behaviour and performance. As a second contribution, we show that a parameter sweep with subsequent majority voting can be used to boost ConCor's performance and produce a trustworthy confidence measure. As a final contribution we show how the proposed extension also allows performing multi-modal (joint audio-video) synchronization of casual multi-perspective video recordings enabling superior matching performance.

Recognizing the location and orientation of a mobile device from captured images is a promising application of image retrieval algorithms. Matching the query images to an existing georeferenced database like Google Street View enables mobile search for location related media, products, and services. Due to the rapidly changing field of view of the mobile device caused by constantly changing user attention, very low retrieval times are essential. These can be significantly reduced by performing the feature quantization on the handheld and transferring compressed Bag-of-Feature vectors to the server. To cope with the limited processing capabilities of handhelds, the quantization of high dimensional feature descriptors has to be performed at very low complexity. To this end, we introduce in this paper the novel Multiple Hypothesis Vocabulary Tree (MHVT) as a step towards real-time mobile location recognition. The MHVT increases the probability of assigning matching feature descriptors to the same visual word by introducing an overlapping buffer around the separating hyperplanes to allow for a soft quantization and an adaptive clustering approach. Further, a novel framework is introduced that allows us to integrate the probability of correct quantization in the distance calculation using an inverted file scheme. Our experiments demonstrate that our approach achieves query times reduced by up to a factor of 10 when compared to the state-of-the-art.

Distinctive visual cues are of central importance for image retrieval applications, in particular, in the context of visual location recognition. While in indoor environments typically only few distinctive features can be found, outdoors dynamic objects and clutter significantly impair the retrieval performance. We present an approach which exploits text, a major source of information for humans during orientation and navigation, without the need for error-prone optical character recognition. To this end, characters are detected and described using robust feature descriptors like SURF. By quantizing them into several hundred visual words we consider the distinctive appearance of the characters rather than reducing the set of possible features to an alphabet. Writings in images are transformed to strings of visual words termed visual phrases, which provide significantly improved distinctiveness when compared to individual features. An approximate string matching is performed using N-grams, which can be efficiently combined with an inverted file structure to cope with large datasets. An experimental evaluation on three different datasets shows significant improvement of the retrieval performance while reducing the size of the database by two orders of magnitude compared to state-of-the-art. Its low computational complexity makes the approach particularly suited for mobile image retrieval applications.

Cross-correlation is a classical similarity measure with broad applications in multimedia signal processing. While it is robust against uncorrelated noise in the input signals, it is severely affected by systematic disturbances which lead to biased results. To overcome this limitation, we propose in this paper consensus-based cross-correlation (ConCor) to deal with heavily corrupted signal parts that derail regular cross-correlation. ConCor builds upon the widely adopted RANSAC algorithm to reliably identify and eliminate corrupt signal parts at limited additional complexity. Our approach is universal in that it can be combined with existing cross-correlation variants. We apply ConCor in two example applications, namely video synchronization and template matching. Our experimental results demonstrate the improved robustness and accuracy when compared to classical cross-correlation.

We present a novel approach for the temporal synchronization of multiple videos which is based on cross-correlating bit rate profiles. The proposed scheme determines the temporal offset without major restrictions on viewing angles, camera properties and camera motion. We propose two extensions of the basic algorithm which reduce the influence of camera motion and distracting background objects. Additionally, we describe how to optimally combine different bit rate components in order to further improve the reliability of the synchronization scheme. The proposed approach, when combined with the three extensions, leads to a reliable, robust, and frame accurate temporal alignment of videos at remarkably low complexity.

Image feature points are the basis for numerous computer vision tasks, such as pose estimation or object detection. State of the art algorithms detect features that are invariant to scale and orientation changes. While feature detectors and descriptors have been widely studied in terms of stability and repeatability, their localisation error has often been assumed to be uniform and insignificant. We argue that this assumption does not hold for scale-invariant feature detectors and demonstrate that the detection of features at different image scales actually has an influence on the localisation accuracy. A general framework to determine the uncertainty of multi-scale image features is introduced. This uncertainty is represented via anisotropic covariances with varying orientation and magnitude. We apply our framework to the well-known SIFT and SURF algorithms, detail its implementation and make it available. Finally the usefulness of such covariance estimates for bundle adjustment and homography computation is illustrated.

We propose a novel framework for cooperation among mobile prosumers (CAMP) that aims at establishing semantic relations between multimedia content acquired by mobile users, focussing especially on audio-visual media. We discuss the basic properties of such a system and describe a number of possible applications. We also address technical requirements and possible solutions, and show a preliminary implementation of a cooperative video sharing platform.

This paper studies the calibration of a multi-sensor capture device for the acquisition of image-based scene representations. It is composed of cameras and a laser rangefinder. We use a plane based approach to calibrate the compound system, determining both intrinsic and extrinsic parameters. The novelty of our approach is that the cameras and the scanner are treated jointly. We have tested our method both in simulations and on real data and observe very good calibration results.