In this paper, we present a large-scale mobile augmented reality system that recognizes the buildings in the mobile device's live video and registers this live view with the 3-dimensional models of the buildings. Having the camera pose estimated and tracked, the system adds relevant information about the buildings to the video in the correct perspective. We demonstrate the system on a large database of geo-tagged panoramic images of an urban environment with associated 3-dimensional planar models. The system uses the capabilities of emerging mobile platforms such as location and orientation sensors, and computational power to detect, track, and augment buildings in urban scenes.

We describe an augmented reality prototype for exploring a 3D urban environment on mobile devices. Our system utilizes the location and orientation sensors on the mobile platform as well as computer vision techniques to register the live view of the device with the 3D urban data. In particular, the system recognizes the buildings in the live video, tracks the camera pose, and augments the video with relevant information about the buildings in the correct perspective. The 3D urban data consist of 3D point clouds and corresponding geo-tagged RGB images of the urban environment. We also discuss the processing steps to make such 3D data scalable and usable by our system.

This poster will present findings from a study of a shopping assistant prototype with simulated augmented reality information. The goal of the study was to find out the acceptable level of delay in presentation of augmented information and the acceptable rate of error of the information presented. Twelve participants interacted with a Samsung Omnia™ smartphone that presented a wine shopping scenario under several levels of delay in showing product information. Participants indicated their willingness to wait for each delay they experienced. Participants also answered a survey about which types of products they would want a shopping assistant application to assist with.

Estimating the 3D orientation of the camera in a video sequence within a global frame of reference is useful for video stabilization when displaying the video in a virtual 3D environment, as well as for accurate navigation and other applications. This task requires the input of orientation sensors attached to the camera to provide absolute 3D orientation in a geographical frame of reference. However, high-frequency noise in the sensor readings makes it impossible to achieve accurate orientation estimates required for visually stable presentation of video sequences that were acquired with a camera subject to jitter, such as a handheld camera or a vehicle mounted camera. On the other hand, image alignment has proven successful for image stabilization, providing accurate frame-to-frame orientation estimates but drifting over time due to error and bias accumulation and lacking absolute orientation. In this paper we propose a practical method for generating high accuracy estimates of the 3D orientation of the camera within a global frame of reference by fusing orientation estimates from an efficient image-based alignment method, and the estimates from an orientation sensor, overcoming the limitations of the component methods.

We present a system for making community driven websites easily accessible from the latest mobile devices. Many of these new devices contain an ensemble of sensors such as cameras, GPS and inertial sensors. We demonstrate how these new sensors can be used to bring the information contained in sites like Wikipedia to users in a much more immersive manner than text or maps. We have collected a large database of images and articles from Wikipedia and show how a user can query this database by simply snapping a photo. Our system uses the location sensors to assist with image matching and the inertial sensors to provide a unique and intuitive user interface for browsing results.

Accurate and fast system modeling is central to the rapid design space exploration needed for embedded-system design. With fast, complex SoCs playing a central role in such systems, system designers have come to require MIPS-range simulation speeds and near-cycle accuracy. The sophisticated simulation frameworks that have been developed for high-speed system performance modeling do not address power consumption, although it is a key design constraint. In this paper, we define a simulation-based methodology for extending system performance modeling frameworks to also include power modeling. We demonstrate the use of this methodology with a case study of a real, complex embedded system, comprising the Intel XScale embedded microprocessor, its WMMX SIMD co processor, L1 caches, SDRAM, and the on-board address and data buses. We describe detailed power models for each of these components and validate them against physical measurements from hardware, demonstrating that such frameworks enable designers to model both power and performance at high speeds without sacrificing accuracy. Our results indicate that the power estimates obtained are accurate within 5% of physical measurements from hardware, while simulation speeds consistently exceed a million instructions per second (MIPS).

While image clustering has many important applications ranging from personal to Web image management, its use is often limited by the difficulty of extracting reliable semantics from low level image features. The image clusters can be improved by using features extracted from image regions rather than the whole image. Region segmentation can be improved in turn, by considering all images within the same cluster rather than segmenting each image independently. This observation leads to the unified Bayesian framework for image clustering and segmentation presented in this paper. The experimental results, reported using several types of visual feature extractors on a database of Web documents containing over 6000 images, illustrates a significant improvement over existing techniques.

With the proliferation of digital cameras, the size of personal media data such as digital photos, videos, etc. has grown extremely large. The personal nature of the data has heightened the demands for a media management system on personal desktops. Existing solutions for media management target mostly server-based Web databases and rely on extensive metadata (i.e., labels) generation to aid retrieval. Personal media databases, on the other hand, have very limited labels generated by the end users themselves. This paper introduces a method for learning concept templates from web images to query personal image databases. The proposed method has the advantage of leveraging Web resources to ease personal photo retrieval in order to avoid costly annotation of personal image databases.

Accurate and fast system modeling is central to the rapid design space exploration needed for embedded-system design. With fast, complex SoCs playing a central role in such systems, system designers have come to require MIPS-range simulation speeds and near-cycle accuracy. The sophisticated simulation frameworks that have been developed for high-speed system performance modeling do not address power consumption, although it is a key design constraint. In this paper, we define a simulation-based methodology for extending system performance-modeling frameworks to also include power modeling. We demonstrate the use of this methodology with a case study of a real, complex embedded system, comprising the Intel XScale®g embedded microprocessor, its WMMX™ SIMD coprocessor, L1 caches, SDRAM and the on-board address and data buses. We describe detailed power models for each of these components and validate them against physical measurements from hardware, demonstrating that such frameworks enable designers to model both power and performance at high speeds without sacrificing accuracy. Our results indicate that the power estimates obtained are accurate within 5% of physical measurements from hardware, while simulation speeds consistently exceed a million instructions per second (MIPS).

With the proliferation of pervasive computing in digital home and office environment there is an increasing demand for hands-free audio and video interfaces. Microphone arrays are already present on desktops and mobile computers, and camera arrays are in transition from being highly sophisticated prototypes to become affordable consumer devices soon. Thanks to increasing CPU power of general-purpose computers and higher bandwidth in wired and wireless networks, it now becomes possible to perform array signal processing of audio using networked sensors, actuators and computers. This paper presents components of a prototype system developed to support hands-free audio capture for audio recording and voice recognition based on multiple wirelessly networked laptops with onboard microphone arrays. Several key technologies emonstrated in this system include: 1) time synchronization scheme for distributed audio input devices that uses wireless network; 2) localization algorithms to reconstruct the geometry of audio sensors and speakers in a room; 3) cascaded beamforming algorithms for signals captured by distributed microphone arrays. This paper discusses both theoretical and practical aspects of mapping array signal processing algorithms on a distributed network of general-purpose computers with integrated audio sensors. Our experimental results demonstrate great potential of distributed audio arrays for hands-free command-and-control when compared to an array located on a single platform.

With the advent and proliferation of digital cameras and computers, the number of digital photos created and stored by consumers has grown extremely large. This created increasing demand for image retrieval systems to ease interaction between consumers and personal media content. Active learning is a widely used user interaction model for retrieval systems, which learns the query concept by asking users to label a number of images at each iteration. In this paper, we study sampling strategies for active learning in personal photo retrieval. In order to reduce human annotation efforts in a content-based image retrieval setting, we propose using multiple sampling criteria for active learning: informativeness, diversity and representativeness. Our experimental results show that by combining multiple sampling criteria in active learning, the performance of personal photo retrieval system can be significantly improved

We present a novel algorithm to automatically determine the relative three-dimensional (3-D) positions of audio sensors and actuators in an ad-hoc distributed network of heterogeneous general purpose computing platforms such as laptops, PDAs, and tablets. A closed form approximate solution is derived, which is further refined by minimizing a nonlinear error function. Our formulation and solution accounts for the lack of temporal synchronization among different platforms. We compare two different estimators, one based on the time of flight and the other based on time difference of flight. We also derive an approximate expression for the mean and covariance of the implicitly defined estimator using the implicit function theorem and approximate Taylors' series expansion. The theoretical performance limits for estimating the sensor 3-D positions are derived via the Crame/spl acute/r-Rao bound (CRB) and analyzed, with respect to the number of sensors and actuators, as well as their geometry. We report extensive simulation results and discuss the practical details of implementing our algorithms in a real-life system.

We propose a novel synchronization scheme for distributed audio-video input and output on heterogeneous general purpose computers (GPC) such as laptops, tablets, PDA, smart telephones, audio recorders, and camcorders. These devices typically possess sensors such as microphones and possibly cameras, and actuators such as loudspeakers and displays. In order to combine them wirelessly into a distributed array signal processing system, it is necessary to provide relative time synchronization to sensors and actuators. In this work we propose a setup and an algorithm to synchronize input and output for a network of distributed multichannel audio sensors and actuators connected to GPC. An IEEE 802.11 wireless network is used to deliver the global clock to distributed GPC, while the interrupt mechanism is employed to distribute the clock between I/O devices. Experimental results demonstrate a precision in A/D D/A synchronization precision better than 50 /spl mu/s (a couple of samples at 48 kHz).

Most of the existing research work in the area of media adaptation is concentrated on content adaptation, transcoding and delivery mechanisms without addressing the actual input and output of multimedia data. However, it is the I/O stage of media processing that humans are concerned about. Up until now, most multimedia applications have relied on standalone I/O devices (microphone, headphones, monitor, camera) to capture or render multimedia data. This situation is about to change. Nowadays we are surrounded by a vast number of audio/video (AV) sensors and actuators. They are built into our cellular phones, PDAs, tablets, laptops, and surveillance systems. A natural idea that comes out of this fact is to combine multiple I/O devices into a distributed array of sensors and actuators. The paper shows the feasibility of this idea and shifts media adaptation research away from a single device/stream paradigm towards array multimedia processing. We demonstrate how to transform a network of off-the-shelf devices into a distributed I/O array by providing common time (with tens of microseconds precision) and 3D space coordinates (with a few centimetres precision). We also discuss the implications and potentials of self-calibrating distributed AV-sensor/actuator networks for improved media adaptation.

We propose a new paradigm for implementations of audio array processing algorithms on a network of distributed general-purpose computers. In contrast to currently existing DSP processor-based solutions, our approach offers new possibilities for advanced array signal processing by enabling the usage of general-purpose computing platforms with their superior computational and storage resources. We demonstrate that synchronization of sensors is essential for acoustic blind source separation (BSS) algorithms, and we propose a synchronization scheme that enables BSS on distributed, wirelessly networked computers and can easily be implemented on existing hardware.

We propose a universal synchronization scheme for distributed audio-video capture on heterogeneous computing devices such as laptops, tablets, PDAs, cellular phones, audio recorders, and camcorders. These devices typically possess sensors such as microphones and possibly cameras. In order to combine them wirelessly into a distributed sensing and computing system, it is necessary to provide relative time synchronization among the distributed sensors. In this work we propose a setup and an algorithm that provide synchronization between sampling times for a network of distributed multi-channel audio sensors connected to general purpose computing (GPC) platforms. Extensive experimental results on distributed acoustic Blind Source Separation (BSS) algorithms validate the performance of our synchronization scheme.

In this paper, we present a novel approach to automatically determine the positions of sensors and actuators in an ad-hoc distributed network of general purpose computing platforms. The formulation and solution accounts for the limited precision in temporal synchronization among multiple platforms. The theoretical performance limit for the sensor positions is derived via the Cramer-Rao bound. We analyze the sensitivity of localization accuracy with respect to the number of sensors and actuators as well as their geometry. Extensive Monte Carlo simulation results are reported together with a discussion of the real-time system. In a test platform consisting of 4 speakers and 4 microphones, the sensors' and actuators' three dimensional locations could be estimated with an average bias of 0.08 cm and average standard deviation of 3.8 cm.

We address the problem of real-time video streaming over wireless LANs for both unicast and multicast transmission. The wireless channel is modeled as a packet-erasure channel at the IP level. For the unicast scenario, we describe a novel hybrid Automatic Repeat reQuest (ARQ) algorithm that efficiently combines forward error control (FEC) coding with the ARQ protocol. For the multiple-users scenario, we formulate the problem of real-time video multicast as an optimization of a maximum regret cost function across the multicast user space. The proposed solution efficiently combines progressive source coding with FEC coding. We present a theoretical analysis of the unicast and multicast cases, as well as experimental results that demonstrate the performance advantages of the proposed algorithms over existing methods.

Video query by semantic keywords is one of the most challenging research issues in video data management. To go beyond low-level similarity and access video data content by semantics, we need to bridge the gap between the low-level representation and high-level semantics. This is a difficult multimedia understanding problem. We formulate this problem as a probabilistic pattern-recognition problem for modeling semantics in terms of concepts and context. To map low-level features to high-level semantics, we propose probabilistic multimedia objects (multijects). Examples of multijects in movies include explosion, mountain, beach, outdoor, music, etc. Semantic concepts in videos interact and appear in context. To model this interaction explicitly, we propose a network of multijects (multinet). To model the multinet computationally, we propose a factor graph framework which can enforce spatio-temporal constraints. Using probabilistic models for multijects, rocks, sky, snow, water-body, and forestry/greenery, and using a factor graph as the multinet, we demonstrate the application of this framework to semantic video indexing. We demonstrate how detection performance can be significantly improved using the multinet to take inter-conceptual relationships into account. Our experiments using a large video database consisting of clips from several movies and based on a set of five semantic concepts reveal a significant improvement in detection performance by over 22%. We also show how the multinet is extended to take temporal correlation into account. By constructing a dynamic multinet, we show that the detection performance is further enhanced by as much as 12%. With this framework, we show how keyword-based query and semantic filtering is possible for a predetermined set of concepts.

Block cyclic redundancy check (CRC) codes represent a popular and powerful class of error detection techniques used almost exclusively in modern data communication systems. Though efficient, CRCs can detect errors only after an entire block of data has been received and processed. In this work, we exploit the "continuous" nature of error detection that results from using arithmetic codes for error detection, which provides a novel tradeoff between the amount of added redundancy and the amount of time needed to detect an error once it occurs. We demonstrate how this continuous error detection framework improves the overall performance of communication systems, and show how considerable performance gains can be attained. We focus on several important scenarios: 1) automatic repeat request (ARQ) based transmission; 2) forward error correction (FEC frameworks based on (serially) concatenated coding systems involving an inner error-correction code and an outer error-detection code; and 3) reduced state sequence estimation (RSSE) for channels with memory. We demonstrate that the proposed CED framework improves the throughput of ARQ systems by up to 15% and reduces the computational/storage complexity of FEC and RSSE by a factor of two in the comparisons that we make against state-of-the-art systems.

In this work, we introduce an efficient hidden Markov field model for wavelet image coefficients and apply it to the image denoising problem. Specifically, we propose to model wavelet image coefficients within subbands as Gaussian random variables with parameters determined by underlying hidden Markov-type process. Our model is inspired by the recent estimation-quantization image coder. To reduce the computational complexity we apply the novel factor graph framework to combine two 1-D chain models to approximate a hidden Markov field (HMF) model. We then apply the proposed models for wavelet image coefficients to perform an approximate minimum mean square error (MMSE) estimation procedure to restore an image corrupted by additive white Gaussian noise. Our results are among the state-of-the-art in the field and they indicate the promise of the proposed modeling techniques.

We address the problem of transmission of a useful class of hidden Markov sources with Gaussian observations over a discrete-time power-constrained AWGN channel. Our source model approximates the wavelet representation of natural images, and is motivated by the successful modeling approaches in the state-of-the-art image compression and restoration methods. We derive information-theoretic performance bounds, and propose a simple yet efficient joint source-channel coding solution which combines analog and digital modes of transmission. Our results indicate the promise of the proposed approach to the problem of robust image communication.

We propose a novel probabilistic framework for semantic video in(cid:173) dexing. We define probabilistic multimedia objects (multijects) to map low-level media features to high-level semantic labels. A graphical network of such multijects (multinet) captures scene con(cid:173) text by discovering intra-frame as well as inter-frame dependency relations between the concepts. The main contribution is a novel application of a factor graph framework to model this network. We model relations between semantic concepts in terms of their co-occurrence as well as the temporal dependencies between these concepts within video shots. Using the sum-product algorithm [1] for approximate or exact inference in these factor graph multinets, we attempt to correct errors made during isolated concept detec(cid:173) tion by forcing high-level constraints. This results in a significant improvement in the overall detection performance.

This paper deals with the application to denoising of a very simple but effective "local" spatially adaptive statistical model for the wavelet image representation that was previously introduced successfully in a compression context. Motivated by the intimate connection between compression and denoising, this paper explores the significant role of the underlying statistical wavelet image model. The model used here, a simplified version of the one proposed by LoPresto, Ramchandran and Orchard (see Proc. IEEE Data Compression Conf., 1997), is that of a mixture process of independent component fields having a zero-mean Gaussian distribution with unknown variances /spl sigma//sub s//sup 2/ that are slowly spatially-varying with the wavelet coefficient location s. We propose to use this model for image denoising by initially estimating the underlying variance field using a maximum likelihood (ML) rule and then applying the minimum mean squared error (MMSE) estimation procedure. In the process of variance estimation, we assume that the variance field is "locally" smooth to allow its reliable estimation, and use an adaptive window-based estimation procedure to capture the effect of edges. Despite the simplicity of our method, our denoising results compare favorably with the best reported results in the denoising literature.

We introduce a simple spatially adaptive statistical model for wavelet image coefficients and apply it to image denoising. Our model is inspired by a recent wavelet image compression algorithm, the estimation-quantization (EQ) coder. We model wavelet image coefficients as zero-mean Gaussian random variables with high local correlation. We assume a marginal prior distribution on wavelet coefficients variances and estimate them using an approximate maximum a posteriori probability rule. Then we apply an approximate minimum mean squared error estimation procedure to restore the noisy wavelet image coefficients. Despite the simplicity of our method, both in its concept and implementation, our denoising results are among the best reported in the literature.

Summary form only given. With the rapid growth of wireless communications systems there is an increasing demand for efficient image and video transmission. Significant performance gains can be obtained from joint source channel matching where system resources are assigned based on the tradeoff between data and redundancy. We develop a more general approach for joint source channel matching based on a parametric distortion model that incorporates the flexibility and constraints of both the source and the channel. We use parametric models describing source and channel. We use parametric models describing source and channel characteristics that can be accurately applied to most classes of source and channel coders. To show the generality of our approach, we applied it to the familiar Said-Pearlman progressive image coder with two types of channel coders, a coder with orthogonal symbols of different power, and fixed-rate BPSK modulation overlaid with Reed Solomon codes.

Block cyclic redundancy check (CRC) codes represent a popular and powerful class of error detection techniques in modern data communication systems. Though efficient, CRCs can detect errors only after an entire block of data has been received and processed. We propose a new "continuous" error detection scheme using arithmetic coding that provides a novel tradeoff between the amount of added redundancy and the amount of time needed to detect an error once it occurs. We demonstrate how the new error detection framework improves the overall performance of transmission systems, and show how sizeable performance gains can be attained. We focus on two popular scenarios: (i) automatic repeat request (ARQ) based transmission; and (ii) forward error correction frameworks based on (serially) concatenated coding systems involving an inner error-correction code and an outer error-detection code.

We consider the problem of efficient image transmission over noisy time-varying channels subject to a low transmission energy constraint (and fixed bandwidth/delay constraints). We examine the limits of desirability of a highly compressed representation using a joint source-channel coding (JSCC) framework. Specifically, invoking as a platform a state-of-the-art wavelet image coder, we demonstrate how the resulting highly compressed digital stream, appropriately protected against channel noise, is not always the best solution. We show how a hybrid scheme based on simple partitioning of the wavelet image representation into "compressed" and "uncompressed" components can lead to significantly improved performance (of the order of 3 dB in PSNR for Rayleigh channels) over popular JSCC schemes which are based on compressed, entropy-coded, and appropriately unequal error-protected (UEP) source representations.

We explore joint source-channel coding (JSCC) for time-varying (slow-fading Rayleigh) channels, using a multiresolution (MR) framework for both source coding and transmission (via a novel MR modulation constellation). We tackle the important case of the informed receiver but uninformed transmitter, i.e. where the receiver has access to the channel state information (CSI), but the transmitter does not. We describe an algorithm which jointly optimizes the design of the MR source codebook, the MR constellation, and the decoding strategy of optimally matching the source and signal constellation resolution trees according to the time-varying channel, and show how this leads to improved performance over separately designed source and channel coders.