In this paper we propose a novel application, color Video Code (V-Code) and analyze its data transmission capacity through camera-based mobile data channels. Users can use the camera on a mobile device (PDA or camera phone) as a passive and pervasive data channel to download data encoded as a sequence of color visual patterns. The color V-Code is animated on a display, acquired by the camera and decoded by the pre-embedded software in the mobile device. One interesting question is what is the data transmission capacity it can achieve, theoretically and practically. To answer this question we build a camera channel model to measure color degradation using information theory and show that the capacity of the camera channel can be improved with the optimized color selection through color calibration. After initialization color models are learned automatically as downloading proceeds. We address the problem of precise registration, and implemented a fast perspective correction method to accelerate the decoder in real-time on a resource constrained device. With the optimized color set and efficient implementation we achieve a transmission bit rate of 15.4kbps on a common iMate Jamin phone (200MHz CPU). This speed is faster than the average GPRS bit rate (12kbps).

In this paper, we describe a novel data transfer scheme that uses the camera in a smart phone as an alternative data channel. The data is encoded as a sequence of 2-D barcode images, displayed on a flat panel display, acquired by the camera, and decoded in real time by the software embedded in device. The decoded data is written to a file. Compared with existing data channels, such as CDMA/GPRS, cables, Bluetooth, and Infrared, our method relies on visual communication and does not require special hardware or data plans. Users only need to point the camera at a monitor displaying the VCode to download. Technical challenges to overcome include correction of perspective distortion, compensation for contrast variation, and efficient implementation of small footprint software into a mobile device. We address these challenges and present our solution in detail. We have implemented a prototype which allows users to download various types of files successfully, including pictures, ring tones and Java games onto camera phones running Symbian and Windows Mobile platforms. We discuss the limitations of our solution and outline future work to overcome these limitations.

In this paper, we demonstrate a solution to use cameras to down-load data to cell phones as an alternative to existing wireless (CDMA/GPRS, BlueTooth), infrared or cable connections. In our method the data is encoded as a sequence of images, which can be displayed on any flat display, captured by users with their camera phones, and decoded by pre-embedded software. To solve this problem we need to be able to (1) encode arbitrary data as a sequence of images, (2) process captured images under various lighting variations and perspective distortions while maintaining realtime performance, and (3) decode the processed images robustly even when partial data is lost. In the paper we address these challenges in detail and present our solution. We have implemented a prototype which allows users to successfully download various types of files, including pictures, ring tones and Java programs to the camera phones. We discuss the limitations of our solution, and future works to overcome these limitations.

In this paper we present an efficient motion estimation algorithm for camera-enabled handheld devices, such as mobile phones and PDAs. Compared to general camera motion estimation, the estimation of ego-motion of handheld devices presents unique challenges because most devices are limited in resources (processing power, memory and battery). The algorithm must be lightweight so it can be efficiently embedded and run fast enough to produce smooth seamless translation. Our solution includes a multi-resolution scheme which searches the match from coarse to fine; and optimization of the search space. As a demonstration, we implemented the algorithm on the Symbian based Nokia 3650/6600 camera phone, and explored several interesting applications, including using camera motion for browsing documents and as a pointing device. A cross-platform implementation is also discussed.

The detection of groups of parallel lines is important in applications such as form processing and text (handwriting) extraction from rule lined paper. These tasks can be very challenging in degraded documents where the lines are severely broken. In this paper, we propose a novel model-based method which incorporates high-level context to detect these lines. After preprocessing (such as skew correction and text filtering), we use trained Hidden Markov Models (HMM) to locate the optimal positions of all lines simultaneously on the horizontal or vertical projection profiles, based on the Viterbi decoding. The algorithm is trainable so it can be easily adapted to different application scenarios. The experiments conducted on known form processing and rule line detection show our method is robust, and achieves better results than other widely used line detection methods.

In this paper, we address the problem of the identification of text in noisy document images. We are especially focused on segmenting and identifying between handwriting and machine printed text because: 1) Handwriting in a document often indicates corrections, additions, or other supplemental information that should be treated differently from the main content and 2) the segmentation and recognition techniques requested for machine printed and handwritten text are significantly different. A novel aspect of our approach is that we treat noise as a separate class and model noise based on selected features. Trained Fisher classifiers are used to identify machine printed text and handwriting from noise and we further exploit context to refine the classification. A Markov Random Field-based (MRF) approach is used to model the geometrical structure of the printed text, handwriting, and noise to rectify misclassifications. Experimental results show that our approach is robust and can significantly improve page segmentation in noisy document collections.

The increasing availability of high performance, low priced, portable digital imaging devices has created a tremendous opportunity for supplementing traditional scanning for document image acquisition. Digital cameras attached to cellular phones, PDAs, or as standalone still or video devices are highly mobile and easy to use; they can capture images of any kind of document including very thick books, historical pages too fragile to touch, and text in scenes; and they are much more versatile than desktop scanners. Should robust solutions to the analysis of documents captured with such devices become available, there is clearly a demand from many domains. Traditional scanner-based document analysis techniques provide us with a good reference and starting point, but they cannot be used directly on camera-captured images. Camera captured images can suffer from low resolution, blur, and perspective distortion, as well as complex layout and interaction of the content and background. In this paper we present a survey of application domains, technical challenges and solutions for recognizing documents captured by digital cameras. We begin by describing typical imaging devices and the imaging process. We discuss document analysis from a single camera-captured image as well as multiple frames and highlight some sample applications under development and feasible ideas for future development.

In this paper we present a novel model based approach to detect severely broken parallel lines in noisy textual documents. It is important to detect and remove these lines so the text can be segmented and recognized. We use directional single-connected chain, a vectorization based algorithm, to extract the line segments. We then instantiate a parallel line model with three parameters: the skew angle, the vertical line gap, and the vertical translation. A coarse-to-fine approach is used to improve the estimation accuracy. From the model we can incorporate the high level contextual information to enhance detection results even when lines are severely broken. Our experimental results show our method can detect 94% of the lines in our database with 168 noisy Arabic document images.

In this paper we address the problem of the identification of text from noisy documents. We segment and identify handwriting from machine printed text because 1) handwriting in a document often indicates corrections, additions or other supplemental information that should be treated differently from the main body or body content, and 2) the segmentation and recognition techniques for machine printed text and handwriting are significantly different. Our novelty is that we treat noise as a separate class and model noise based on selected features. Trained Fisher classifiers are used to identify machine printed text and handwriting from noise. We further exploit context to refine the classification. A Markov random field (MRF) based approach is used to model the geometrical structure of the printed text, handwriting and noise to rectify the mis-classification. Experimental results show our approach is promising and robust, and can significantly improve the page segmentation results in noise documents.

In this paper we address the problem of sports video classification using hidden Markov models (HMMs). For each sports genre, we construct two HMMs representing motion and color features respectively. The observation sequences generated from the principal motion direction and the principal color of each frame are fed to a motion and a color HMM respectively. The outputs are integrated to make a final decision. We tested our scheme on 220 minutes of sports video with four genre types: ice hockey, basketball, football, and soccer, and achieved an overall classification accuracy of 93%.

We present a superresolution-based text enhancement scheme to improve optical character recognition (OCR) and readability of text in digital video. The quality of video text is degraded by factors such as low resolution, antialiasing, clutter and blurring. We use the fact that the same text string often exists in consecutive frames to explore the temporal information and enhance the text image. For graphic text, we register text blocks to subpixel accuracy and fuse them to a new block with high resolution and a cleaner background. We use a projection onto convex sets based method to deblur scene text to improve readability. Experimental results show our scheme can improve OCR for graphic text and readability for scene text significantly.

In this paper we present a video text detection system based on automated neural network training. Compared with previous work which detects only graphical text with fixed parameters, our system (1) provides a training mechanism so the parameters of the system can be adapted to changing environments, (2) can detect both graphical text and scene text located in complex backgrounds, (3) can detect text in any orientation and (4) can perform multilingual text detection. Experiments show the effectiveness of our system in various text detection tasks.

Text that appears in a scene or is graphically added to video can provide an important supplemental source of index information as well as clues for decoding the video's structure and for classification. In this work, we present algorithms for detecting and tracking text in digital video. Our system implements a scale-space feature extractor that feeds an artificial neural processor to detect text blocks. Our text tracking scheme consists of two modules: a sum of squared difference (SSD)-based module to find the initial position and a contour-based module to refine the position. Experiments conducted with a variety of video sources show that our scheme can detect and track text robustly.

In this paper a multiple frame based technique to enhance text in digital video is presented. After extracting a reference text block, we use an image matching technique to find the corresponding text blocks in consecutive frames. We register these text blocks to subpixel levels by using image interpolation techniques to improve both correspondence and text resolution. The registered text blocks are averaged to obtain a new text block with a clean background and a higher resolution. Experiments conducted on several video sequences show that our enhancement scheme can improve the accuracy of commercial off-the-shelf OCR considerably.

Scene and graphic text can provide important supplemental index information in video sequences. In this paper we address the problem automatically identifying text regions in digital video key frames. The text contained in video frames is typically very noisy because it is aliased and/or digitized at a much lower resolution than typical document images, making identification, extraction and recognition difficult. The proposed method is based on the use of a hybrid wavelet/neural network segmenter on a series of overlapping small windows to classify regions which contain text. To detect text over a wide range of font sizes, the method is applied to a pyramid of images and the regions identified at each level are integrated.

We address the problem of automatically tracking moving text in digital videos. Our scheme consists of two separate processes: monitoring which detects the new text line entering a frame, and tracking which uses prediction techniques to reconcile the text from frame to frame. Temporal consistency allows one to monitor periodically and reduce the computation complexity. The tracking process uses a rapid prediction/search scheme to update the position of the text blocks between frames. We provide details of the implementation and results for text moving in the scene and text which moves as a result of camera motion.

We propose and implement a method for detecting duplicate documents in very large image databases. The method is based on a robust "signature" extracted from each document image which is used to index into a table of previously processed documents. The approach has a number of advantages over OCR or other recognition based methods, including speed and robustness to imaging distortions. To justify the approach and test the scalability, we have developed a simulator which allows us to change parameters of the system and examine performance for millions of document signatures. A complete system is implemented and tested on a test collection of technical articles and memos.