A popular approach for storyline grounding in videos is centered on the consistent classification of main characters in videos. This work investigates learning identity agnostic characters for video grounding. The system starts without any knowledge about the character identities and their facial images, and learns to build individual character models as the video frames stream in. It is challenging to establish character models, especially in movies where characters can go through a lifetime in a clip, and their facial appearances change dramatically between scenes. We designed an active learning algorithm on top of metric learning, with an interactive interface to query users. Different from conventional active learning algorithms, our query proposal function depends not only on appearance, but also on story context of these characters.

We explore how to optimally categorize regions for faster and more reliable image matching and registration. We propose using the entropy of histogram of oriented gradients(HOG) features to characterize image regions, and propose a region-sensitive feature selection algorithm for image registration. We apply the region categorization algorithms to several mobile applications, including mobile visual search and image registration for panorama. We demonstrate the effectiveness of our approach by experimental results on a large dataset.

Mobile devices are becoming ubiquitous multimedia computing platforms. However, due to limited computational power on these devices, a good mobile application requires far more considerations in algorithm design and optimization than for desktop systems. This demo shows two multimedia applications on a mobile phone: mobile visual search and panorama. The innovative components of the system include hybrid tracking and visual matching, optimal region classification and feature selection for faster and more reliable image-related applications, and an optimized image processing pipeline for multimedia applications on a mobile phone.

Panorama has an extensive research history in computer vision and image processing, with several commercial offerings on the market today. Photo registration is the key component for generating panoramas. Textured regions in photos bring challenges to the registration algorithms. However, so far there has been no specific research focusing on this problem. To address this issue, we proposed texture detection algorithms for robust photo registration in panorama applications. Moreover, we present a system that run texture detection and photo registration efficiently, so that panorama generation can be done in real-time on a mobile camera phone. Experimental results are provided for our proposed solution.

Camera motion filtering is important in many applications, such as camera-based visual control interface on a mobile device. In this paper, we first present a robust global motion estimation algorithm based on optical flow and RANSAC, and a strategy that decompose camera motions into individual components frame-by-frame. A motion filtering strategy is then proposed to detect the most dominant motion within each individual component. Acceleration strategies are proposed to make it run real-time. Applications in video analysis and a mobile visual control interface are discussed.

Robust local features such as SIFT and SURF have been applied to many interesting image matching applications. These features are by nature very computational intensive even for modern desktop PCs. We have developed a framework for efficient feature extraction and matching for still images on a mobile device. In this paper we extend the still-image framework to video sequences. It is inefficient to perform feature extraction and matching for each frame in the video sequence. By tracking the content of the frames, feature extraction and image matching need only be performed when there is new content. We show promising experimental results using this approach.

Recent convergence of imaging sensors and general purpose processors on mobile phones creates an opportunity for a new class of augmented reality applications. Robust image feature extraction is a crucial enabler of this type of systems. In this article, we discuss an efficient mobile phone implementation of a state-of-the-art algorithm for computing robust image features called SURF. We implement several improvements to the basic algorithm that significantly improve its performance and reduce its memory footprint making the use of this algorithm on the mobile phone practical. Our prototype implementation has been applied to several practical applications such as image search, object recognition and augmented reality applications.

We present a new adaptive algorithm for automatic detection of text from a natural scene. The initial cues of text regions are first detected from the captured image/video. An adaptive color modeling and searching algorithm is then utilized near the initial text cues, to discriminate text/non-text regions. EM optimization algorithm is used for color modeling, under the constraint of text layout relations for a specific language. The proposed algorithm combines the advantages of several previous approaches for text detection, and utilizes a focus-of-attention approach for text finding. The whole algorithm is applied in a prototype system that can automatically detect and recognize sign input from a video camera, and translate the signs into English text or voice streams. We present evaluation results of our algorithm on this system.

Image processing and shape analysis are important and useful in many document processing applications, such as segmentation of connected characters, recognition of confusing character shapes, etc. A skeleton segmentation algorithm and a shape analysis method based on the segmentation result is presented. The motivation of the work presented in this paper is the idea that the basic elements in character shapes are line segments with uniform directional attributes. We define these line segments as image cells. An algorithm is carefully designed to segment the character skeletons into image cells robustly. Based on the image cells, a new selective attention approach for character shape extraction is proposed. To guarantee the robustness of the proposed algorithm, the shape extraction problem is decomposed into two steps: attention focus area detection and shape growing. The effectiveness of the proposed algorithms is verified by experimental results.

Two new feature extraction strategies-modified multiple discriminant analysis (MMDA) and difference principal component analysis (DPCA)-are presented and derived. The proposed algorithms are especially useful in automatic feature extraction from patterns in a small category set. Experimental results for recognition of Chinese character fonts and handwritten numerals using MMDA and DPCA are presented. Compared with the traditional algorithms, MMDA and DPCA provide more effective feature metrics for pattern discrimination in some settings.

A new algorithm for segmenting handwritten Chinese character strings is presented. This approach is based on a split-and-merge strategy by locating possible ligatures between Chinese characters and merging the over-split components. The strategy is proposed by considering the structural properties of Chinese character strings. To guarantee the accuracy of the above segmentation algorithm, a recognizer is involved in order to aid the segmentation process. A maximum a posteriori probability index is derived for joint optimization of the segmentation and recognition results, and a dynamic programming algorithm-a modified level-building algorithm-is used to optimize this index. The whole algorithm is applied to a Chinese bank check amount recognition task, and some promising experimental results are obtained.