The recent breakthroughs in computer vision have benefited from the availability of large representative datasets (e.g. ImageNet and COCO) for training. Yet, robotic vision poses unique challenges for applying visual algorithms developed from these standard computer vision datasets due to their implicit assumption over non-varying distributions for a fixed set of tasks. Fully retraining models each time a new task becomes available is infeasible due to computational, storage and sometimes privacy issues, while naïve incremental strategies have been shown to suffer from catastrophic forgetting. It is crucial for the robots to operate continuously under open-set and detrimental conditions with adaptive visual perceptual systems, where lifelong learning is a fundamental capability. However, very few datasets and benchmarks are available to evaluate and compare emerging techniques. To fill this gap, we provide a new lifelong robotic vision dataset ("OpenLORIS-Object") collected via RGB-D cameras. The dataset embeds the challenges faced by a robot in the real-life application and provides new benchmarks for validating lifelong object recognition algorithms. Moreover, we have provided a testbed of 9 state-of-the-art lifelong learning algorithms. Each of them involves 48 tasks with 4 evaluation metrics over the OpenLORIS-Object dataset. The results demonstrate that the object recognition task in the ever-changing difficulty environments is far from being solved and the bottlenecks are at the forward/backward transfer designs. Our dataset and benchmark are publicly available at https://lifelong-robotic-vision.github.io/dataset/object.

Service robots should be able to operate autonomously in dynamic and daily changing environments over an extended period of time. While Simultaneous Localization And Mapping (SLAM) is one of the most fundamental problems for robotic autonomy, most existing SLAM works are evaluated with data sequences that are recorded in a short period of time. In real-world deployment, there can be out-of-sight scene changes caused by both natural factors and human activities. For example, in home scenarios, most objects may be movable, replaceable or deformable, and the visual features of the same place may be significantly different in some successive days. Such out-of-sight dynamics pose great challenges to the robustness of pose estimation, and hence a robot’s long-term deployment and operation. To differentiate the forementioned problem from the conventional works which are usually evaluated in a static setting in a single run, the term lifelong SLAM is used here to address SLAM problems in an ever-changing environment over a long period of time. To accelerate lifelong SLAM research, we release the OpenLORIS-Scene datasets. The data are collected in real-world indoor scenes, for multiple times in each place to include scene changes in real life. We also design benchmarking metrics for lifelong SLAM, with which the robustness and accuracy of pose estimation are evaluated separately. The datasets and benchmark are available online at lifelong-robotic-vision.github.io/dataset/scene.

There are two difficulties to utilize state-of-the-art object recognition/detection/segmentation methods to robotic applications. First, most of the deep learning models heavily depend on large amounts of labeled training data, which are expensive to obtain for each individual application. Second, the object categories must be pre-defined in the dataset, thus not practical to scenarios with varying object categories. To alleviate the reliance on pre-defined big data, this paper proposes a customized object recognition and segmentation method. It aims to recognize and segment any object defined by the user, given only one annotation. There are three steps in the proposed method. First, the user takes an exemplar video of the target object with the robot, defines its name, and mask its boundary on only one frame. Then the robot automatically propagates the annotation through the exemplar video based on a proposed data generation method. In the meantime, a segmentation model continuously updates itself on the generated data. Finally, only a lightweight segmentation net is required at testing stage, to recognize and segment the user-defined object in any scenes.

Point cloud registration is a popular research topic and has been widely used in many tasks, such as robot mapping and localization. It is a challenging problem when the overlap is small, or the rotation is large. The problem has not been well solved by existing methods such as the iterative closest point (ICP) and its variants. In this paper, a novel method named principal coordinate alignment with overlap tuning (PCAOT) is proposed based on the Manhattan world assumption. It solves two key problems together, the transformation estimation and the overlap estimation. The overlap is represented by a 3D cuboid and the transformation is computed only within the overlap region. Instead of finding point correspondence as in traditional methods, we estimate the rotation by principal coordinates alignment, which is faster and less sensitive than ICP and its variants to small overlaps and large rotations. Evaluations demonstrate that our method achieves much better results than the ICP and its variants when the overlap ratio is smaller than 50%, or the rotation angle is larger than 60°. Especially, it is effective when the overlap ratio is less than 30%, or the rotation angle is larger than 90°.

Human detection in RGB-D images is an important yet very challenging task in computer vision. In this paper, we propose a novel human detection approach in RGB-D images, which integrates ROI (region-of-interest) generation, depth-size relationship estimation and a human detector. Our approach has the following advantages: 1) ROI generation and depth-size relationship estimation take full advantage of color and depth information to fast reject about 70% negative samples while maintaining a high recall rate; 2) the cascade-structured human detector can seamlessly concatenate features extracted from both color and depth images; and 3) our method can detect human at a speed of more than 30 fps on 640 χ 480 images on a single laptop CPU without any GPU acceleration. Experiments on challenging public datasets demonstrate the effectiveness of our method.

Human pose estimation and tracking is the task of determining the states (location, orientation, and scale) of each body part over time. It is important for many vision understanding applications, such as visual interactive gaming, immersive virtual reality, visual surveillance, and content-based image retrieval. However, it remains a challenging task due to unknown image background, presence of clutter and especially the high dimensional state space (usually 30+ dimensions). In this paper, we contribute to human pose estimation and tracking in two aspects. First, we design two efficient Markov Chain dynamics under the data-driven Markov Chain Monte Carlo framework to effectively explore the high dimensional state space. Second, we parse the tree structure state space into a lexicographic order according to the image observations and body topology, and the optimization process is conducted in this order. This realizes a much more efficient exploration of the state space than the sampling based search or exhaustive search, and thus achieves a tremendous speed-up. Experimental results demonstrate the efficiency and effectiveness of the proposed method in estimating and tracking various kinds of human poses, even against cluttered backgrounds, in poor illumination or under partial self-occlusion.

This paper presents a novel learning framework for training boosting cascade based object detector from large scale dataset. The framework is derived from the well-known Viola-Jones (VJ) framework but distinguished by three key differences. First, the proposed framework adopts multi-dimensional SURF features instead of single dimensional Haar features to describe local patches. In this way, the number of used local patches can be reduced from hundreds of thousands to several hundreds. Second, it adopts logistic regression as weak classifier for each local patch instead of decision trees in the VJ framework. Third, we adopt AUC as a single criterion for the convergence test during cascade training rather than the two trade-off criteria (false-positive-rate and hit-rate) in the VJ framework. The benefit is that the false-positive-rate can be adaptive among different cascade stages, and thus yields much faster convergence speed of SURF cascade. Combining these points together, the proposed approach has three good properties. First, the boosting cascade can be trained very efficiently. Experiments show that the proposed approach can train object detectors from billions of negative samples within one hour even on personal computers. Second, the built detector is comparable to the state-of-the-art algorithm not only on the accuracy but also on the processing speed. Third, the built detector is small in model-size due to short cascade stages.

Cosmetic makeup is a general event in our daily life, which improves women's beauties and attractions. But, it is difficult for ordinary users to make a wonderful makeup as the cover girls. Moreover, when you are in nude look and want to share better look with your friends, the fastest and easiest way is virtual makeup. However, current existing makeup software needs many user inputs to adjust face landmarks, which influence the user experience. And, it cannot remove the flaws on skin as good as the real cosmetic makeup. In this paper, we describe an automatic framework to apply a cosmetic makeup and skin beautification to your face, which can be selected from many example make-up face images. Our method detects the face landmarks with existing algorithm and adjusts the landmark with skin color Gaussian Mixture model based segmentation. Then, the skin color area is separated into three layers, and makeup is transferred with different method to different layers. The results look pretty good for some natural input face images.

Along with the inclusion of GPU cores within the same CPU die, the performance of Intel's processor-graphics has been significantly improved over earlier generation of integrated graphics. The need to efficiently harness the computational power of the GPU in the same CPU die is more than ever. This paper presents a highly optimized Haar-based face detector which efficiently exploits both CPU and GPU computing power on the latest Sandy Bridge processor. The classification procedure of Haar-based cascade detector is partitioned to two phases in order to leverage both thread level and data level parallelism in the GPU. The image downscaling and integral image calculation running in the CPU core can work with the GPU in parallel. Compared to CPU-alone implementation, the experiments show that our proposed GPU accelerated implementation achieves a 3.07x speedup with more than 50% power reduction on the latest Sandy Bridge processor. On the other hand, our implementation is also more efficient than the CUDA implementation on the NVidia GT430 card in terms of performance as well as power. In addition, our proposed method presents a general approach for task partitioning between CPU and GPU, thus being beneficial not only for face detection but also for other multimedia and computer vision techniques.

Facial landmark detection is an essential module in many face related applications and it often appears as the most time consuming part in face processing pipeline. This paper proposes a fast and effective method for facial landmark detection using Haar cascade classifiers and a simple 3D head model, which not only detects the position of landmark points but also gives an estimation of head pose such as yaw and pitch angles. To reduce the amount of computation, only 7 landmark points are detected (4 eye corners, 2 mouth corners, 1 nose tip) that generally meets the requirement of face alignment and face recognition. Experiment on multiple datasets shows our algorithm can provide sufficient accuracy of facial landmark localization while being able to run in super real-time at Intel Atom 1.3 GHz embedded processors.

Avatar is the virtual representation of user's facial, body, and motion characteristics in computer game, social network, and augmented reality. Facial modeling needs enormous efforts to achieve immersive experience in applications like avatar chatting or online makeover. Great challenge exists in robust detection of 2D facial prominent points and mapping them to 3D models in a parameterized manner. Another challenge is how to characterize semantic components of eyes, mouth, nose, and cheek rather than low level mesh geometries. In this paper, we proposed an augmented makeover framework to deal with aforementioned challenges. Aiming to provide amateurs with flexible customizations, morphable model is constructed from a set of scanned 3D face data set. Appearance personalization is carried out in the offline phase where single image and multiple views are discussed respectively to generate deformative shape in a progressive manner. Augmentation is implemented in the online phase where a fast and robust 3D tracking is used to balance the tradeoff between accuracy and real-time requirements. By this means, immersive Human Computer Interaction such as virtual makeover and photo-realistic avatar chatting could be achieved.

In this article, we present a method to perform automatic player trajectories mapping based on player detection, unsupervised labeling, efficient multi-object tracking, and playfield registration in broadcast soccer videos. Player detector determines the players' positions and scales by combining the ability of dominant color based background subtraction and a boosting detector with Haar features. We first learn the dominant color with accumulate color histogram at the beginning of processing, then use the player detector to collect hundreds of player samples, and learn player appearance codebook by unsupervised clustering. In a soccer game, a player can be labeled as one of four categories: two teams, referee or outlier. The learning capability enables the method to be generalized well to different videos without any manual initialization. With the dominant color and player appearance model, we can locate and label each player. After that, we perform multi-object tracking by using Markov Chain Monte Carlo (MCMC) data association to generate player trajectories. Some data driven dynamics are proposed to improve the Markov chain's efficiency, such as label consistency, motion consistency, and track length, etc. Finally, we extract key-points and find the mapping from an image plane to the standard field model, and then map players' position and trajectories to the field. A large quantity of experimental results on FIFA World Cup 2006 videos demonstrate that this method can reach high detection and labeling precision, reliably tracking in scenes of player occlusion, moderate camera motion and pose variation, and yield promising field registration results.

Depth-map merging is one typical technique category for multi-view stereo (MVS) reconstruction. To guarantee accuracy, existing algorithms usually require either sub-pixel level stereo matching precision or continuous depth-map estimation. The merging of inaccurate depth-maps remains a challenging problem. This paper introduces a bundle optimization method for robust and accurate depth-map merging. In the method, depth-maps are generated using DAISY feature, followed by two stages of bundle optimization. The first stage optimizes the track of connected stereo matches to generate initial 3D points. The second stage optimizes the position and normals of 3D points. High quality point cloud is then meshed as geometric models. The proposed method can be easily parallelizable on multi-core processors. Middlebury evaluation shows that it is one of the most efficient methods among non-GPU algorithms, yet still keeps very high accuracy. We also demonstrate the effectiveness of the proposed algorithm on various real-world, high-resolution, self-calibrated data sets including objects with complex details, objects with large area of highlight, and objects with non-Lambertian surface.

This paper presents a general texture mapping framework for image-based 3D modeling. It aims to generating seamless texture map for 3D model created by real-world photos under uncontrolled environment. Our proposed method addresses two challenging problems: 1) texture discontinuity due to system error in 3D modeling from self-calibration; 2) color/lighting difference among images due to real-world uncontrolled environments. The general framework contains two stages to resolve these problems. The first stage globally optimizes the registration of texture patches and triangle faces with Markov Random Field (MRF) to optimize texture mosaic. The second stage does local radiometric correction to adjust color difference between texture patches and then blend texture boundaries to improve color continuity. The proposed method is evaluated on several 3D models by image-based 3D modeling, and demonstrates promising results.

Mobile Augmented Reality, as an emerging application for handheld devices, explores more natural interactions in real and virtual environments. For the purpose of accurate system response and manipulating objects in real-time, extensive efforts have been made to estimate six Degree-of-Freedom and extract robust feature to track. However there are still quite a lot challenges today in achieving rich user experience. To allow for a seamless transition from outdoor to indoor service, we investigated and integrated various sensing techniques of GPS, wireless, Inertial Measurement Units, and optical. A parallel tracking and matching scheme is presented to address the speed-accuracy tradeoff issue. Two prototypes, fine-scale mirror world navigation and context-aware trouble shooting, have been developed to demonstrate the suitability of our approach.

A novel method is proposed to achieve robust and real-time ball tracking in broadcast soccer videos. In sports video, the soccer ball is small, often occluded, and with high motion speed. Thus, it is difficult to detect the sole ball in a single frame. To solve this problem, rather than locate the ball in one of several frames through detection or tracking, we find the ball through optimizing its motion trajectory in successive frames. The proposed method includes three level processes: object level, intra-trajectory level, and inter-trajectory level processing. In object level, multiple objects instead of a single ball are detected and all of them are taken as ball candidates through shape and color features identification. Then at intra-trajectory level, each ball candidate is tracked by a Kalman filter and verified by detection in successive frames, which results in lots of initial short trajectories in a video shot. These trajectories are thereafter scored and filtered according to their length and spatial-temporal relationship in a time-line model. With these trajectories, we construct a distance graph, in which a node represents a trajectory, and an edge means distance between two trajectories. We then get the optimal path using the Dijkstra algorithm in the graph at the inter-trajectory level. The optimal path is composed by a sequence of initial trajectories which make the whole route smooth and long in duration. To get a complete and reasonable path, we finally apply cubic spline interpolation to bridge the gap between adjacent trajectories (the duration corresponding to when the ball is occluded). We select three representative real FIFA2006 soccer video clips (containing a total of 16,500 frames) and manually elaborately labeling each frame in it, and take it as ground-truth to evaluate the algorithm. The average F-score is 80.59%. The algorithm was used in our soccer analysis system and tested on a wide range of real soccer videos, and all the results are satisfied. The algorithm is effective and its whole speed far exceeds real-time, 35.6 fps on mpeg2 data on the Intel Conroe platform.

This paper proposes a novel patch-wise image inpainting algorithm using the image signal sparse representation over a redundant dictionary, which merits in both capabilities to deal with large holes and to preserve image details while taking less risk. Different from all existing works, we consider the problem of image inpainting from the view point of sequential incomplete signal recovery under the assumption that the every image patch admits a sparse representation over a redundant dictionary. To ensure the visually plausibility and consistency constraints between the filled hole and the surroundings, we propose to construct a redundant signal dictionary by directly sampling from the intact source region of current image. Then we sequentially compute the sparse representation for each incomplete patch at the boundary of the hole and recover it until the whole hole is filled. Experimental results show that this approach can efficiently fill in the hole with visually plausible information, and take less risk to introduce unwanted objects or artifacts.

Human pose estimation is the task of determining the states (location, orientation and scale) of each body part. It is important for many vision understanding applications, e.g. visual interactive gaming, immersive virtual reality, content-based image retrieval, etc. However, it remains a challenging task because of unknown image background, presence of clutter, partial occlusion and especially the high dimensional state space (usually 30+ dimensions). In this paper, we contribute to human pose estimation in two aspects. First, we design two efficient Markov Chain dynamics under the data-driven Markov Chain Monte Carlo (DDMCMC) framework to effectively explore the complex solution space. Second, we parse the tree structure state space into a lexicographic order according to the image observations and body topology, and the optimization process is conducted in this order. This realizes a much more efficient exploration than the sampling based search and exhaustive search, and thus achieves a tremendous speed-up. Experimental results demonstrate the efficiency and effectiveness of the proposed method in estimating various kinds of human poses, even with cluttered background , poor illumination or partial self-occlusion.

Technique advances have made image capture and storage very convenient, which results in an explosion of the amount of visual information. It becomes difficult to find useful information from these tremendous data. Content-based Visual Information Retrieval (CBVIR) is emerging as one of the best solutions to this problem. Unfortunately, CBVIR is a very compute-intensive task. Nowadays, with the boom of multi-core processors, CBVIR can be accelerated by exploiting multi-core processing capability. In this paper, we propose a parallelization implementation of a CBVIR system facing to server application and use some serial and parallel optimization techniques to improve its performance on an 8-core and on a 16-core systems. Experimental results show that optimized implementation can achieve very fast retrieval on the two multi-core systems.We also compare the performance of the application on the two multi-core systems and give an explanation of the performance difference between the two systems. Furthermore, we conduct detailed scalability and memory performance analysis to identify possible bottlenecks in the application. Based on these experimental results and performance analysis, we gain many insights into developing efficient applications on future multi-core architectures.

Finding a destination in unfamiliar environments of complex office building or shopping mall has been always bothering us in daily life. Fine-scale directional guidance, a combination of location based service and context aware service, is emerging along with the technology developments of mobile computing, wireless communication, and augmented reality. This paper addresses the core problem of how to align what we see with where we are. In order to achieve robust and accurate navigation by 2D/3D alignment, we developed a hybrid solution by fusing GPS, inertial measurement units and computer vision methods together. We have implemented a prototype of Mobile Augmented Reality and 3D navigation on the manually created virtual environment of Intel China Research Center office and surroundings.

This paper proposes a general framework called Markov stationary features (MSF) to extend histogram based features. The MSF characterizes the spatial co-occurrence of histogram patterns by Markov chain models, and finally yields a compact feature representation through Markov stationary analysis. Therefore, the MSF goes one step beyond histograms since it now involves spatial structure information of both within histogram bins and between histogram bins. Moreover, it still keeps simplicity, compactness, efficiency, and robustness. We demonstrate how the MSF is used to extend histogram based features like color histogram, edge histogram, local binary pattern histogram and histogram of oriented gradients. We evaluate the MSF extended histogram features on the task of TRECVID video concept detection. Results show that the proposed MSF extensions can achieve significant performance improvement over corresponding histogram features.

This paper presents a novel audio-visual fusion method for speech detection, which is an important front-end for content-based video processing. This approach aims to extract homogeneous speech segments from the accompanying audio stream in real-world movie/TV videos with the help of video captions. Note that captions are mainly created to help viewers to follow the dialog, rather than to accurately locate the speech regions. We propose a caption-aided speech detection approach, which makes use of both caption information and audio information. The inaccurate positions of the captions are refined through using audio features (pitch and MFCCs) and BIC-based acoustic change detection. Comparison experiments against several other traditional speech detection approaches are conducted, showing that the proposed approach improves the speech detection performance greatly.

Detection of salient moving object has great potentials in activity recognition, scene understanding, etc. However techniques to characterizing the object in fine granularity have not been well developed in real applications due to the computational intensity. The emerging multi-core technology in hardware design provides an opportunity for the compute intensive algorithms to boost speed in parallel. This paper proposed a scalable approach to detecting salient moving object which is designed inherently for parallelization. To characterize the object in fine granularity, we extract color-texture homogenous regions as the basic processing unit by image segmentation. To identify salient object, we generate probabilistic template by learning the space-time context. The parallel algorithm is implemented using OpenMP. Evaluations have been carried out on sports, news, and home video data. For the CIF size image, we get processing speed of 51.1 frames per second and near linear speed up on an eight-core machine. It indicates that the algorithm parallelization is a promising solution for practical applications in the multimedia field.

Local binary pattern histogram (LBPH) is one of the popular and excellent image texture descriptor. However, conventional LBPH lacks of the description of spatial structure information. This paper proposes an extension of LBPH called Markov chain local binary patterns (MCLBP) to alleviate this limitation. We apply MCLBP to the task of TRECVID video concept detection. Experimental results demonstrate that MCLBP achieves significant performance improvement over conventional LBPH features and other widely used visual features such as Gabor texture.

In this paper, we propose a boosted interactively distributed particle filter (BIDPF) to address the problem of automatic multi-object tracking in the application of player tracking in broadcast soccer video. The interactively distributed particle filter technique (IDPF) is adopted to handle the mutual occlusions among targets. The proposal distribution using a mixture model that incorporates information from the dynamic model and the boosting detection is introduced into the IDPF framework. The boosting proposal distribution quickly detects targets, while the IDPF process keeps the identity of targets during mutual occlusions. Moreover, the foreground observation is extracted by using the color model of the playfield to speed up the boosting detection and reduce false alarms. The foreground is also used to develop a data-driven potential model to improve the IDPF performance. We test the proposed approach on several video sequences and the results demonstrate that our system is able to track a variable number of objects in a dynamic scene and correctly maintain their identities regardless of camera motion and frequent mutual occlusions.

Probabilistic Latent Semantic Analysis (PLSA) is one of the most popular statistical techniques for the analysis of two-model and co-occurrence data. It has applications in information retrieval and filtering, nature language processing, machine learning from text, and other related areas. However, PLSA is rarely applied to large datasets due to its high computational complexity.This paper presents an optimized and parallelized implementation of PLSA which is capable of processing datasets with 10000 documents in seconds. Compared to the baseline program, our parallelized program can achieve speedup of more than six on an eight-processor machine. The characterization of the parallel program is also presented. The performance analysis of the parallel program indicates that this program is memory intensive and the limited memory bandwidth is the bottleneck for better speedup.

Scale invariant feature transform (SIFT) is an approach for extracting distinctive invariant features from images, and it has been successfully applied to many computer vision problems (e.g. face recognition and object detection). However, the SIFT feature extraction is compute-intensive, and a real-time or even super-real-time processing capability is required in many emerging scenarios. Nowadays, with the multi- core processor becoming mainstream, SIFT can be accelerated by fully utilizing the computing power of available multi-core processors. In this paper, we propose two parallel SIFT algorithms and present some optimization techniques to improve the implementation 's performance on multi-core systems. The result shows our improved parallel SIFT implementation can process general video images in super-real-time on a dual-socket, quad-core system, and the speed is much faster than the implementation on GPUs. We also conduct a detailed scalability and memory performance analysison the 8-core system and on a 32-core chip multiprocessor (CMP) simulator. The analysis helps us identify possible causes of bottlenecks, and we suggest avenues for scalability improvement to make this application more powerful on future large-scale multi- core systems.

Consumers are making increasing use of digital camera, phone camera, and camcorder every minute. Normal users have desires to produce high-quality photos although they are less of photography techniques. Consumers hunger for online instructing on how to capture a good photograph in terms of camera pose (location, orientation, field of view, etc.) Making it ubiquitously accessible is essential which requires excessive computation and communication capabilities over existing portable devices. The emerging Mobile Internet Devices (MIDs) and Ultra Mobile PCs (UMPCs) have been equipped with enabling features and turn into an alternative. This paper presents a new Location Based Service for photography assistance. It consists of three major components, focus of attention modeling, image quality assessment, and camera pose matching, to deal with the geo-referenced photos retrieved from Internet. Those techniques have been widely studied in the research domains of computer vision and pattern recognition. Having the MID / UMPC platforms at hand, richer experiences on Internet, such as instant travel blogging and photo sharing are more convenient.

Automatic player detection, labeling and tracking in broadcast soccer video are significant while quite challenging tasks. In this paper, we present a solution to perform automatic multiple player detection, unsupervised labeling and efficient tracking. Players ’ position and scale are determined by a boosting based detector. Players ’ appearance models are unsupervised learned from hundreds of samples automatically collected by detection. Thereafter, these models can be utilized for player labeling (Team A, Team B and Referee). Player tracking is achieved by Markov Chain Monte Carlo (MCMC) data association. Some data driven dynamics are proposed to improve the Markov chain’s efficiency. The testing results on FIFA World Cup 2006 video demonstrate that our method can reach high detection and labeling precision, and reliably tracking in cases of scenes such as multiple player occlusion, moderate camera motion and pose variation. 1

For video summarization and retrieval, one of the important modules is to group temporal-spatial coherent shots into high-level semantic video clips namely scene segmentation. In this paper, we propose a novel scene segmentation and categorization approach using normalized graph cuts(NCuts). Starting from a set of shots, we first calculate shot similarity from shot key frames. Then by modeling scene segmentation as a graph partition problem where each node is a shot and the weight of edge represents the similarity between two shots, we employ NCuts to find the optimal scene segmentation and automatically decide the optimum scene number by Q function. To discover more useful information from scenes, we analyze the temporal layout patterns of shots, and automatically categorize scenes into two different types, i.e. parallel event scenes and serial event scenes. Extensive experiments are tested on movie, and TV series. The promising results demonstrate that the proposed NCuts based scene segmentation and categorization methods are effective in practice.

This paper addresses the problem of audio segmentation in practical media (e.g. TV series, movies and etc.) which usually consists of segments in various lengths with quite a portion of short ones. An unsupervised audio segmentation approach is presented, including a segmentation-stage to detect potential acoustic changes, and a refinement-stage to refine these candidate changes by a tri-model Bayesian information criterion. Experiments show that the proposed approach has good detectability of short segments and the novel tri-model BIC effectively improves the overall segmentation performance.

Gabor filters are widely known as one of the best representation for face recognition. Since raw Gabor representation is of very high dimensionality, feature reduction is usually required in practice. This paper proposes the feature of integral Gabor-Haar transformation (FIGHT), which is a compact Gabor feature representation while still keeps high recognition performance. This paper also studies fusion strategies for groups of FIGHT feature, and present a discriminative learning scheme to combine group-wise results. Experiments show that FIGHT feature is effective, and the discriminative fusion over FIGHT feature group achieves the state-of-the-art performance on FERET database.

In automated home video editing, selecting out the most informative contents from the redundant footage is challenging. This paper proposes an information-theoretic approach to content selection by exploring the dependence relations between who (characters) and where (scenes) in the video. First the footage is segmented into basic units about the same characters at the same scene. To compactly represent the dependence relations between scenes and characters, contingency table is used to model their co-occurrence statistics. Suppose the contents about which characters at which scene are dominating by two random variables, an optimal selection criterion is proposed based on joint entropy. To improve the computation efficiency, a pruned N-Best heuristic algorithm is presented to search the most informative video units. Experimental results demonstrated the proposed approach is flexible and effective for automated content selection.

In this paper, a fast and robust multi-view lip localization algorithm in video is proposed. We consider lip localization as a binary classification problem, where a classifier is learned to distinguish between the lip and the region surrounding it. The classifier we use here is a histogram-based one which exploits the anthropometrical properties of the human face with the help of face scale normalization. Due to the perceptual uniformity and robustness for lip/skin color variations across different people, we adopt CIELUV color model to represent the color of lip. After classification, we propose a novel projection-cut algorithm by spatial deviation analysis (SDA) to locate the lip, which is effective to deal with the background clutters. Experimental results on teleplay videos demonstrate that the proposed approach is efficient and robust for lip localization.

Replay is one of the key cues indicating highlights in sports videos. A replay is usually sandwiched by two identical logos which prompt the start and end of a replay. A logo transition usually contains 10-30 frames, describes a flying or varying object(s). In this paper, a reliable logo and replay detecting approach is proposed. It contains two main stages: first, a logo transition template is unsupervised learned, a key frame (K-frame) and a set of pixels that describes logo object (logo pixels, L-pixels) accurately are also extracted; second, the learned information are used jointly to detect logos and replays in the video. In addition to traditional color analysis, optical flow feature is employed to depict the movement of the logo object(s). Extensive experiments show that the proposed approach can reliably detect logos and replays regardless of the types of sports videos.

As digital video data becomes more pervasive, mining information from multimedia data becomes increasingly important. However, the huge computational requirement prohibits its wide use in practice. This paper presents some optimization techniques and parallel approaches to accelerate a real video mining application -goalmouth detection in soccer video. By tuning the application performance with efficient optimization techniques (fast lookup tables, SIMD technology, and cache blocking) and exploiting the thread level parallelism, we enable the real time processing for this application on Intel latest dual core desktop machine -Core 2 Dual. Our study shows that with proper optimization and parallelization, multimedia mining can be used widely in our daily life soon.

In this paper, we present a novel approach which employs the temporal redundancy of video texts to improve the performance of text segmentation and stroke extraction from complex background. We first demonstrate how to obtain the required aligned text image sequences from video frames via a robust corners-set matching scheme. Then the changing background pixels can be identified and removed by exploiting the statistics of the temporal redundancy of video. The text stroke pixels can be clearly separated from the complex background. Experiments on TV programs and movies videos show the proposed approach can generate clean text strokes image which can efficiently improve the OCR's performance compare to some traditional approaches.

In this paper, a real time and robust multi-view lip localization in video frames is proposed. The objective of this research is to exactly localize lip surrounding box with sub-pixel accuracy in low quality video frames. This is achieved by multi-view face detection in video frames, lip ROI extraction, color space transformation, vertical edge feature detection and lip localization by a feature-spatial profile analysis on a joint color-edge plane. One possible application of talking face detection in video is used to evaluate our work. Experimental results on TV programs and movies demonstrate that the proposed approach is efficient and robust in lip localization.

In this paper, we propose an approach for body detection and tracking with a hierarchical and salient-half-body-prior scheme and apply it in real dynamic scenes. The human region is firstly located by an SVM classifier with histogram of oriented gradients (HOG). Two image observations, motion foreground and edge distance map, are then applied for body detection. In detail, the torso part is first roughly initialized, and then parts are localized in a salient-prior half-body sampling scheme by maximizing body configuration probability with both parts likelihood and body geometry constraints. For body tracking, the color-texture appearance templates of parts extracted in detection phase are further used in tracking process. The approach has been tested on soccer and skating videos and got good results.

Content-based video information retrieval (CBVIR) has becoming one of the best solutions for retrieving useful information from today's video information explosion. And with the rapid development of modern technologies, CBVIR is emerging as a mass market desktop application. There is evidence that visual feature extraction is the most time-consuming part in a CBVIR system. In this paper, we implement three video visual feature extractions in parallel by exploring different kinds of thread-level parallelism. We also conduct detailed scalability and memory performance analysis on two multi-core based systems, in order to gain more insights into video-analysis related applications on future multi-core systems. From our analysis we identify the likely causes of bottlenecks in these kinds of applications and suggest ways to improve scalability.

Audio search plays an important role in analyzing audio data and retrieving useful audio information. In this paper, a partially overlapping block-parallel active search method (POBPAS) is proposed to perform audio quick search on shared-memory multiprocessor systems (SMPs). This method uses a proper data segmentation to achieve parallelism and performs a high level of parallelism with little additional work. Several techniques including I/O optimization, proper data partition and dynamic scheduling are also introduced to maximize its scalability performance. In addition, we conduct a detailed performance characterization analysis of the parallel implementation of the POBPAS for three data sets on two Intel Xeon SMPs. Experimental results indicate that there are no obvious parallel limiting factors in the implementation except memory bandwidth. As a result, it can achieve 11.3X speedup for a larger data set (searching a 15 seconds' clip in a 27 hours' audio stream) on the 16-way processor system.

With the amount of data continuing to grow, extracting "data of interest" is becoming popular, pervasive, and more important than ever. Data mining, as this process is known as, seeks to draw meaningful conclusions, extract knowledge, and acquire models from vast amounts of data. These compute-intensive data-mining applications, where thread-level parallelism can be effectively exploited, are the design targets of future multi-core systems. As a result, future multi-core systems will be required to process terabyte-level workloads. To understand the memory system performance of data-mining applications, this paper presents the use of hardware-software co-simulation to explore the cache design space of several multi-threaded data mining applications. Our study reveals that the workloads are memory intensive, have large working-set sizes, and exhibit good data locality. We find that large DRAM caches can be useful to address their large working-set sizes

In order to help the user to grasp the long video content quickly, this paper proposes a novel video summarization approach based on redundancy removal and content ranking. By video parsing and cast indexing, the approach first constructs a story board to let user know about the main scenes and the main actors in the video. Then it generates a "story-constraint summary" by key frame clustering and repetitive segment detection. To shorten the video summary length to a target length, our approach constructs a "time-constraint summary" by important factor based content ranking. Extensive experiments are carried out on TV series, movies, and cartoons. Good results demonstrate the effectiveness of the proposed method.

In the field of automated home video editing, exploring the dependence relations between who (character) and where (scene) makes great sense to end-users for content selection. However, such techniques have not been well developed in real applications due to their computational intensity. The emerging multi-core architectures provide an opportunity to speed up those compute expensive algorithms if shift from serial thinking to parallelism. This demonstration presents a scalable parallel system for home video editing. In a realtime processing speed, the system analyzes how many characters and scenes are captured and provides end-users with flexible preference customization. Through kernel module optimization and data-level parallelization, evaluations on a real 8-core machine indicates a near linear speed up could be achieved along with the increasing number of cores.

As digital video data becomes more pervasive, mining information from multimedia data becomes increasingly important. Although researches in multimedia mining area have shown great potential in daily life, the huge computational requirement prohibits its wide use in practice. Since our personal computer is shifting from uniprocessors to multicore processors, exploiting thread level parallelism in multimedia mining applications is critical to utilize the hardware resources and accelerate the mining process. This paper presents three different parallel approaches (task level, data slicing and hybrid parallel) to parallelize one widely used application in video mining system. The hybrid scheme, with the exploration of data level and task level parallelism, delivers much better performance than other two schemes. We get 10x performance improvement on a 16-way multiprocessor system. Besides, we perform several efficient optimization techniques, such as subexpression optimization, SIMD, and data blocking, to improve the performance by more than 60%. Therefore, our parallelization and optimization of the application makes it 16x faster than it used to be. Our study shows that with proper parallelization and optimization, multimedia mining can be used widely in our daily life soon

Soccer highlight detection is an active research topic in recent years. One of the difficult problems is how to effectively fuse multi-modality cues, i.e. audio, visual and textual information, to improve the detection performance. This paper proposes a novel two-dependence Bayesian network (2d-BN) based fusion approach to soccer highlight detection. 2d-BN is a particular Bayesian network which assumes that each variable depends on two other variables at most. Through this assumption, 2d-BN can not only characterize the relationships among features but also be trained efficiently. Extensive experiments demonstrate the effectiveness of the proposed method

As digital video data becomes more pervasive, mining information from multimedia data becomes increasingly important, e.g., extraction of goal events in soccer game automatically from the video content. Though all of these advances in multimedia mining have shown great potential in daily life, the huge computational requirement prohibits its wide use in practice. As computer architecture evolves from uniprocessor to the era of multi-core processors, accelerating the multimedia application by exploiting thread level parallelism would be more promising to boost performance and provide more functionality. This paper presents three different parallel approaches, i.e., task level, data slicing and hybrid parallel scheme, to parallelize shot detection, a widely used application in the video mining system. The hybrid scheme, with the exploration of data level and task level parallelism, delivers much better performance than the other two schemes. Besides, we also employ several software optimization techniques, e.g. data blocking and thread affinity, to improve the performance by more than 50%. Experimental results indicate that there are no obvious parallel limiting factors in the hybrid parallel scheme. It scales well the increasing number of processors, and exhibits 13.6x speedup on 16-way processor system

Text mining is one of the best solutions for today and the future's information explosion. With the development of modern processor technologies, it will be a mass market desktop application in the many-core era. In text mining system, information extraction is a representative module and is the most compute intensive part. In this paper, we study the performance of parallel information extraction on shared memory multi-processor systems in order to gain some insights of such applications on the future's many-core architecture. In implementation, conditional random fields (CRFs) algorithm is selected as the core of module information extraction. Based on the newest CRFs toolkit FlexCRFs, we make several serial optimizations and then parallelize it with MPI and System V. IPC/shm. We also conduct a detailed performance analysis of this parallel application on the target system

In this paper, we present a tree-partition algorithm for parallel mining of frequent patterns. Our work is based on FP-Growth algorithm, which is constituted of tree-building stage and mining stage. The main idea is to build only one FP-Tree in the memory, partition it into several independent parts and distribute them to different threads. A heuristic algorithm is devised to balance the workload. Our algorithm can not only alleviate the impact of locks during the tree-building stage, but also avoid the overhead that do great harm to the mining stage. We present the experiments on different kinds of datasets and compare the results with other parallel approaches. The results suggest that our approach has great advantage in efficiency, especially on certain kinds of datasets. As the number of processors increases, our parallel algorithm shows good scalability.

As an extension of Bayesian network, module network is an appropriate model for inferring causal network of a mass of variables from insufficient evidences. However learning such a model is still a time-consuming process. In this paper, we propose a parallel implementation of module network learning algorithm using OpenMP. We propose a static task partitioning strategy which distributes sub-search-spaces over worker threads to get the tradeoff between load-balance and software-cache-contention. To overcome performance penalties derived from shared-memory contention, we adopt several optimization techniques such as memory pre-allocation, memory alignment and static function usage. These optimizations have different patterns of influence on the sequential performance and the parallel speedup. Experiments validate the effectiveness of these optimizations. For a 2,200 nodes dataset, they enhance the parallel speedup up to 88%, together with a 2X sequential performance improvement. With resource contentions reduced, workload imbalance becomes the main hurdle to parallel scalability and the program behaviors more stable in various platforms.

The prediction of protein 3D structure has become a hot research area in the post-genome era, through which people can understand a protein's function in health and disease, explore ways to control its actions and assist drug design. Many protein structure prediction approaches have been proposed in past decades. Among them, Rosetta is one of the best systems. However, the huge time complexity of Rosetta, e.g. a few days to predict a protein, limits its wide use in practice. To accelerate the prediction of protein 3D structure in Rosetta, this paper presents three different approaches, i.e., non-interactive, periodic interactive and asynchronous dynamic interactive scheme, to parallelize Rosetta. The asynchronous interactive scheme, with the adaptation of dynamic solution interaction, outperforms the other two, delivering much faster convergence speed and better solution quality. Detailed measurements and performance analysis also indicate that parallel Rosetta with asynchronous dynamic interactive scheme scales well

Single nucleotide polymorphisms (SNPs) are loci on the genome where different alleles are observed in the population. It has been observed that there might be some patterns or context dependences in the sequence segments adjacent to SNPs sites. Discovering such dependences is very important for understanding possible origins of SNPs in evolution. We collected 519,767 bi-allelic SNPs of human in gene regions from HGBASE and separated them in 6 groups according to the types of alleles at the SNP loci. Bayesian network structure learning technique is applied to discovery of possible dependences in sequence segments around these sites as well as in reference sequences collected as comparison. Noticeable probabilistic correlations among some loci were detected in all the 6 SNP groups and nothing significant was found in the reference sequences. The dependence relations found with different SNP groups are different. These putative context dependences around SNP sites provide important hints for further analyzing SNP-related sequences patterns. The work also illustrates the powerfulness of the Bayesian network method as a tool for biological sequence analysis.

We propose dynamic Bayesian network (DBN) based synchronous and asynchronous multi-stream models for noise-robust automatic speech recognition. In these models, multiple noise-robust features are combined into a single DBN to obtain better performance than any single feature system alone. Results on the Aurora 2.0 noisy speech task show significant improvements of our synchronous model over both single stream models and over a ROVER based fusion method.

DBNs (Dynamic Bayesian Networks) [1] are powerful tool in modeling time-series data, and have been used in speech recognition recently [2,3,4]. The “decoding” task in speech recognition means to find the viterbi path [5](in graphical model community, “viterbi path” has the same meaning as MPE “Most Probable Explanation”) for a given acoustic observations. In this paper we describe a new algorithm utilizes a new data structure “backpointer”, which is produced in the “marginalization” procedure in probability inference. With these backpointers, the viterbi path can be found in a simple backtracking. We first introduce the concept of backpointer and backtracking; then give the algorithm to compute the viterbi path for DBNs based on backpointer and backtracking. We prove that the new algorithm is correct, faster and more memory saving comparison with old algorithm. Several experiments are conducted to demonstrate the effectiveness of the algorithm on several well known DBNs. We also test the algorithm on a real world DBN model that can recognize continuous digit numbers.