The Internet of things (IoT) has been widely used for various applications including medical and transportation systems, among others. Smart medical systems have become the most effective and practical solutions to provide users with low-cost, noninvasive, and long-term continuous health monitoring. Recently, Jia et al. proposed an authentication and key agreement scheme for smart medical systems based on fog computing and indicated that it is safe and can withstand a variety of known attacks. Nevertheless, we found that it consists of several flaws, including known session-specific temporary information attacks and lack of per-verification. The opponent can readily recover the session key and user identity. In this paper, we propose a secure authentication and key agreement scheme, which compensates for the imperfections of the previously proposed. For a security evaluation of the proposed authentication scheme, informal security analysis and the Burrows–Abadi–Needham (BAN) logic analysis are implemented. In addition, the ProVerif tool is used to normalize the security verification of the scheme. Finally, the performance comparisons with the former schemes show that the proposed scheme is more applicable and secure.

Facial landmark detection is an essential module in many face related applications and it often appears as the most time consuming part in face processing pipeline. This paper proposes a fast and effective method for facial landmark detection using Haar cascade classifiers and a simple 3D head model, which not only detects the position of landmark points but also gives an estimation of head pose such as yaw and pitch angles. To reduce the amount of computation, only 7 landmark points are detected (4 eye corners, 2 mouth corners, 1 nose tip) that generally meets the requirement of face alignment and face recognition. Experiment on multiple datasets shows our algorithm can provide sufficient accuracy of facial landmark localization while being able to run in super real-time at Intel Atom 1.3 GHz embedded processors.

In this article, we present a method to perform automatic player trajectories mapping based on player detection, unsupervised labeling, efficient multi-object tracking, and playfield registration in broadcast soccer videos. Player detector determines the players' positions and scales by combining the ability of dominant color based background subtraction and a boosting detector with Haar features. We first learn the dominant color with accumulate color histogram at the beginning of processing, then use the player detector to collect hundreds of player samples, and learn player appearance codebook by unsupervised clustering. In a soccer game, a player can be labeled as one of four categories: two teams, referee or outlier. The learning capability enables the method to be generalized well to different videos without any manual initialization. With the dominant color and player appearance model, we can locate and label each player. After that, we perform multi-object tracking by using Markov Chain Monte Carlo (MCMC) data association to generate player trajectories. Some data driven dynamics are proposed to improve the Markov chain's efficiency, such as label consistency, motion consistency, and track length, etc. Finally, we extract key-points and find the mapping from an image plane to the standard field model, and then map players' position and trajectories to the field. A large quantity of experimental results on FIFA World Cup 2006 videos demonstrate that this method can reach high detection and labeling precision, reliably tracking in scenes of player occlusion, moderate camera motion and pose variation, and yield promising field registration results.

Mobile Augmented Reality, as an emerging application for handheld devices, explores more natural interactions in real and virtual environments. For the purpose of accurate system response and manipulating objects in real-time, extensive efforts have been made to estimate six Degree-of-Freedom and extract robust feature to track. However there are still quite a lot challenges today in achieving rich user experience. To allow for a seamless transition from outdoor to indoor service, we investigated and integrated various sensing techniques of GPS, wireless, Inertial Measurement Units, and optical. A parallel tracking and matching scheme is presented to address the speed-accuracy tradeoff issue. Two prototypes, fine-scale mirror world navigation and context-aware trouble shooting, have been developed to demonstrate the suitability of our approach.

A novel method is proposed to achieve robust and real-time ball tracking in broadcast soccer videos. In sports video, the soccer ball is small, often occluded, and with high motion speed. Thus, it is difficult to detect the sole ball in a single frame. To solve this problem, rather than locate the ball in one of several frames through detection or tracking, we find the ball through optimizing its motion trajectory in successive frames. The proposed method includes three level processes: object level, intra-trajectory level, and inter-trajectory level processing. In object level, multiple objects instead of a single ball are detected and all of them are taken as ball candidates through shape and color features identification. Then at intra-trajectory level, each ball candidate is tracked by a Kalman filter and verified by detection in successive frames, which results in lots of initial short trajectories in a video shot. These trajectories are thereafter scored and filtered according to their length and spatial-temporal relationship in a time-line model. With these trajectories, we construct a distance graph, in which a node represents a trajectory, and an edge means distance between two trajectories. We then get the optimal path using the Dijkstra algorithm in the graph at the inter-trajectory level. The optimal path is composed by a sequence of initial trajectories which make the whole route smooth and long in duration. To get a complete and reasonable path, we finally apply cubic spline interpolation to bridge the gap between adjacent trajectories (the duration corresponding to when the ball is occluded). We select three representative real FIFA2006 soccer video clips (containing a total of 16,500 frames) and manually elaborately labeling each frame in it, and take it as ground-truth to evaluate the algorithm. The average F-score is 80.59%. The algorithm was used in our soccer analysis system and tested on a wide range of real soccer videos, and all the results are satisfied. The algorithm is effective and its whole speed far exceeds real-time, 35.6 fps on mpeg2 data on the Intel Conroe platform.

Finding a destination in unfamiliar environments of complex office building or shopping mall has been always bothering us in daily life. Fine-scale directional guidance, a combination of location based service and context aware service, is emerging along with the technology developments of mobile computing, wireless communication, and augmented reality. This paper addresses the core problem of how to align what we see with where we are. In order to achieve robust and accurate navigation by 2D/3D alignment, we developed a hybrid solution by fusing GPS, inertial measurement units and computer vision methods together. We have implemented a prototype of Mobile Augmented Reality and 3D navigation on the manually created virtual environment of Intel China Research Center office and surroundings.

This paper proposes a general framework called Markov stationary features (MSF) to extend histogram based features. The MSF characterizes the spatial co-occurrence of histogram patterns by Markov chain models, and finally yields a compact feature representation through Markov stationary analysis. Therefore, the MSF goes one step beyond histograms since it now involves spatial structure information of both within histogram bins and between histogram bins. Moreover, it still keeps simplicity, compactness, efficiency, and robustness. We demonstrate how the MSF is used to extend histogram based features like color histogram, edge histogram, local binary pattern histogram and histogram of oriented gradients. We evaluate the MSF extended histogram features on the task of TRECVID video concept detection. Results show that the proposed MSF extensions can achieve significant performance improvement over corresponding histogram features.

After analyzing the performance bottlenecks of the Hough transform on multi-core processors, this paper proposes a new Hough transform implementation. The performance of microprocessors improves significantly because of the introduction of multiple cores. To harness the computation power of such multi-core processors, we must effectively execute many threads at the same time. This paper first studies a coarse-grain and a fine-grain parallelization of a straightforward Hough transform implementation on an 8-core machine. Due to parallelization overheads and memory requirements, these schemes do not fully utilize computation power. After that, we propose a new Hough transform implementation for parallelization. Experimental data shows that the new Hough transform exposes a significant amount of concurrency and pretty good data locality. On the 8-core machine, the new implementation has 25% better performance than the old ones.

This paper presents a novel audio-visual fusion method for speech detection, which is an important front-end for content-based video processing. This approach aims to extract homogeneous speech segments from the accompanying audio stream in real-world movie/TV videos with the help of video captions. Note that captions are mainly created to help viewers to follow the dialog, rather than to accurately locate the speech regions. We propose a caption-aided speech detection approach, which makes use of both caption information and audio information. The inaccurate positions of the captions are refined through using audio features (pitch and MFCCs) and BIC-based acoustic change detection. Comparison experiments against several other traditional speech detection approaches are conducted, showing that the proposed approach improves the speech detection performance greatly.

Local binary pattern histogram (LBPH) is one of the popular and excellent image texture descriptor. However, conventional LBPH lacks of the description of spatial structure information. This paper proposes an extension of LBPH called Markov chain local binary patterns (MCLBP) to alleviate this limitation. We apply MCLBP to the task of TRECVID video concept detection. Experimental results demonstrate that MCLBP achieves significant performance improvement over conventional LBPH features and other widely used visual features such as Gabor texture.

Automatic player detection, labeling and tracking in broadcast soccer video are significant while quite challenging tasks. In this paper, we present a solution to perform automatic multiple player detection, unsupervised labeling and efficient tracking. Players ’ position and scale are determined by a boosting based detector. Players ’ appearance models are unsupervised learned from hundreds of samples automatically collected by detection. Thereafter, these models can be utilized for player labeling (Team A, Team B and Referee). Player tracking is achieved by Markov Chain Monte Carlo (MCMC) data association. Some data driven dynamics are proposed to improve the Markov chain’s efficiency. The testing results on FIFA World Cup 2006 video demonstrate that our method can reach high detection and labeling precision, and reliably tracking in cases of scenes such as multiple player occlusion, moderate camera motion and pose variation. 1

For video summarization and retrieval, one of the important modules is to group temporal-spatial coherent shots into high-level semantic video clips namely scene segmentation. In this paper, we propose a novel scene segmentation and categorization approach using normalized graph cuts(NCuts). Starting from a set of shots, we first calculate shot similarity from shot key frames. Then by modeling scene segmentation as a graph partition problem where each node is a shot and the weight of edge represents the similarity between two shots, we employ NCuts to find the optimal scene segmentation and automatically decide the optimum scene number by Q function. To discover more useful information from scenes, we analyze the temporal layout patterns of shots, and automatically categorize scenes into two different types, i.e. parallel event scenes and serial event scenes. Extensive experiments are tested on movie, and TV series. The promising results demonstrate that the proposed NCuts based scene segmentation and categorization methods are effective in practice.

This paper addresses the problem of audio segmentation in practical media (e.g. TV series, movies and etc.) which usually consists of segments in various lengths with quite a portion of short ones. An unsupervised audio segmentation approach is presented, including a segmentation-stage to detect potential acoustic changes, and a refinement-stage to refine these candidate changes by a tri-model Bayesian information criterion. Experiments show that the proposed approach has good detectability of short segments and the novel tri-model BIC effectively improves the overall segmentation performance.

Gabor filters are widely known as one of the best representation for face recognition. Since raw Gabor representation is of very high dimensionality, feature reduction is usually required in practice. This paper proposes the feature of integral Gabor-Haar transformation (FIGHT), which is a compact Gabor feature representation while still keeps high recognition performance. This paper also studies fusion strategies for groups of FIGHT feature, and present a discriminative learning scheme to combine group-wise results. Experiments show that FIGHT feature is effective, and the discriminative fusion over FIGHT feature group achieves the state-of-the-art performance on FERET database.

In automated home video editing, selecting out the most informative contents from the redundant footage is challenging. This paper proposes an information-theoretic approach to content selection by exploring the dependence relations between who (characters) and where (scenes) in the video. First the footage is segmented into basic units about the same characters at the same scene. To compactly represent the dependence relations between scenes and characters, contingency table is used to model their co-occurrence statistics. Suppose the contents about which characters at which scene are dominating by two random variables, an optimal selection criterion is proposed based on joint entropy. To improve the computation efficiency, a pruned N-Best heuristic algorithm is presented to search the most informative video units. Experimental results demonstrated the proposed approach is flexible and effective for automated content selection.

In this paper, a fast and robust multi-view lip localization algorithm in video is proposed. We consider lip localization as a binary classification problem, where a classifier is learned to distinguish between the lip and the region surrounding it. The classifier we use here is a histogram-based one which exploits the anthropometrical properties of the human face with the help of face scale normalization. Due to the perceptual uniformity and robustness for lip/skin color variations across different people, we adopt CIELUV color model to represent the color of lip. After classification, we propose a novel projection-cut algorithm by spatial deviation analysis (SDA) to locate the lip, which is effective to deal with the background clutters. Experimental results on teleplay videos demonstrate that the proposed approach is efficient and robust for lip localization.

Replay is one of the key cues indicating highlights in sports videos. A replay is usually sandwiched by two identical logos which prompt the start and end of a replay. A logo transition usually contains 10-30 frames, describes a flying or varying object(s). In this paper, a reliable logo and replay detecting approach is proposed. It contains two main stages: first, a logo transition template is unsupervised learned, a key frame (K-frame) and a set of pixels that describes logo object (logo pixels, L-pixels) accurately are also extracted; second, the learned information are used jointly to detect logos and replays in the video. In addition to traditional color analysis, optical flow feature is employed to depict the movement of the logo object(s). Extensive experiments show that the proposed approach can reliably detect logos and replays regardless of the types of sports videos.

In this paper, we present a novel approach which employs the temporal redundancy of video texts to improve the performance of text segmentation and stroke extraction from complex background. We first demonstrate how to obtain the required aligned text image sequences from video frames via a robust corners-set matching scheme. Then the changing background pixels can be identified and removed by exploiting the statistics of the temporal redundancy of video. The text stroke pixels can be clearly separated from the complex background. Experiments on TV programs and movies videos show the proposed approach can generate clean text strokes image which can efficiently improve the OCR's performance compare to some traditional approaches.

In this paper, a real time and robust multi-view lip localization in video frames is proposed. The objective of this research is to exactly localize lip surrounding box with sub-pixel accuracy in low quality video frames. This is achieved by multi-view face detection in video frames, lip ROI extraction, color space transformation, vertical edge feature detection and lip localization by a feature-spatial profile analysis on a joint color-edge plane. One possible application of talking face detection in video is used to evaluate our work. Experimental results on TV programs and movies demonstrate that the proposed approach is efficient and robust in lip localization.

In order to help the user to grasp the long video content quickly, this paper proposes a novel video summarization approach based on redundancy removal and content ranking. By video parsing and cast indexing, the approach first constructs a story board to let user know about the main scenes and the main actors in the video. Then it generates a "story-constraint summary" by key frame clustering and repetitive segment detection. To shorten the video summary length to a target length, our approach constructs a "time-constraint summary" by important factor based content ranking. Extensive experiments are carried out on TV series, movies, and cartoons. Good results demonstrate the effectiveness of the proposed method.

In the field of automated home video editing, exploring the dependence relations between who (character) and where (scene) makes great sense to end-users for content selection. However, such techniques have not been well developed in real applications due to their computational intensity. The emerging multi-core architectures provide an opportunity to speed up those compute expensive algorithms if shift from serial thinking to parallelism. This demonstration presents a scalable parallel system for home video editing. In a realtime processing speed, the system analyzes how many characters and scenes are captured and provides end-users with flexible preference customization. Through kernel module optimization and data-level parallelization, evaluations on a real 8-core machine indicates a near linear speed up could be achieved along with the increasing number of cores.

As digital video data becomes more pervasive, mining information from multimedia data becomes increasingly important. Although researches in multimedia mining area have shown great potential in daily life, the huge computational requirement prohibits its wide use in practice. Since our personal computer is shifting from uniprocessors to multicore processors, exploiting thread level parallelism in multimedia mining applications is critical to utilize the hardware resources and accelerate the mining process. This paper presents three different parallel approaches (task level, data slicing and hybrid parallel) to parallelize one widely used application in video mining system. The hybrid scheme, with the exploration of data level and task level parallelism, delivers much better performance than other two schemes. We get 10x performance improvement on a 16-way multiprocessor system. Besides, we perform several efficient optimization techniques, such as subexpression optimization, SIMD, and data blocking, to improve the performance by more than 60%. Therefore, our parallelization and optimization of the application makes it 16x faster than it used to be. Our study shows that with proper parallelization and optimization, multimedia mining can be used widely in our daily life soon

Soccer highlight detection is an active research topic in recent years. One of the difficult problems is how to effectively fuse multi-modality cues, i.e. audio, visual and textual information, to improve the detection performance. This paper proposes a novel two-dependence Bayesian network (2d-BN) based fusion approach to soccer highlight detection. 2d-BN is a particular Bayesian network which assumes that each variable depends on two other variables at most. Through this assumption, 2d-BN can not only characterize the relationships among features but also be trained efficiently. Extensive experiments demonstrate the effectiveness of the proposed method

As digital video data becomes more pervasive, mining information from multimedia data becomes increasingly important, e.g., extraction of goal events in soccer game automatically from the video content. Though all of these advances in multimedia mining have shown great potential in daily life, the huge computational requirement prohibits its wide use in practice. As computer architecture evolves from uniprocessor to the era of multi-core processors, accelerating the multimedia application by exploiting thread level parallelism would be more promising to boost performance and provide more functionality. This paper presents three different parallel approaches, i.e., task level, data slicing and hybrid parallel scheme, to parallelize shot detection, a widely used application in the video mining system. The hybrid scheme, with the exploration of data level and task level parallelism, delivers much better performance than the other two schemes. Besides, we also employ several software optimization techniques, e.g. data blocking and thread affinity, to improve the performance by more than 50%. Experimental results indicate that there are no obvious parallel limiting factors in the hybrid parallel scheme. It scales well the increasing number of processors, and exhibits 13.6x speedup on 16-way processor system

As multi/many core processors become prevalent, programming language is important in constructing efficient parallel applications. In this work, we build a multithreaded video mining application with Java, examine the thread profiling information and micro-architecture metrics to identify the factors limiting the scalability, and employ a number of ways to improve performance. Besides, we conduct some thread scheduling experiments. According to the experiments and detailed analysis, we conclude that for this video mining application: (1) Java is a good parallel language candidate for many core processors in terms of performance, scalability, and ease of programming; (2) Thread affinity mechanism is effective in improving data locality, but brings little benefit to multithreaded Java application due to its conservative memory model in JVM

The prediction of protein 3D structure has become a hot research area in the post-genome era, through which people can understand a protein's function in health and disease, explore ways to control its actions and assist drug design. Many protein structure prediction approaches have been proposed in past decades. Among them, Rosetta is one of the best systems. However, the huge time complexity of Rosetta, e.g. a few days to predict a protein, limits its wide use in practice. To accelerate the prediction of protein 3D structure in Rosetta, this paper presents three different approaches, i.e., non-interactive, periodic interactive and asynchronous dynamic interactive scheme, to parallelize Rosetta. The asynchronous interactive scheme, with the adaptation of dynamic solution interaction, outperforms the other two, delivering much faster convergence speed and better solution quality. Detailed measurements and performance analysis also indicate that parallel Rosetta with asynchronous dynamic interactive scheme scales well