Traditional video annotation approaches focus on annotating keyframes/shots or whole videos with semantic keywords. However, the extraction processes of keyframes/shots might lack semantic meanings, and it is hard to use a few keywords to describe the content of a long video with multiple topics. In this work, near-scenes, which contain similar concepts, topics, or semantic meanings, are designed for better video content understanding and annotation. We propose a novel framework of hierarchical video-to-near-scene annotation not only to preserve but also to purify the semantic meanings of near-scenes. To detect near-scenes, a pattern-based prefix tree is first constructed to fast retrieve near-duplicate videos. Then, the videos containing similar near-duplicate segments and similar keywords are clustered with consideration of multimodal features including visual and textual features. To enhance the precision of near-scene detection, a pattern-to-intensity-mark (PIM) method is proposed to perform precise frame-level near-duplicate segment alignment. For each near-scene, a video-to-concept distribution model is designed to analyze the representativeness of keywords and discriminations of clusters by the proposed potential term frequency and inverse document frequency and entropy. Tags are ranked according to video-to-concept distribution scores, and the tags with the highest scores are propagated to near-scenes detected. Extensive experiments demonstrate that the proposed PIM outperforms state-of-the-art approaches compared in terms of quality segments and quality frames for near-scene detection. Furthermore, the proposed framework of hierarchical video-to-near-scene annotation can achieve high quality of near-scene annotation in terms of mean average precision.

With the rapid development and reduced cost of digital video capturing devices, driving video recorders (DVRs) begin to gain widespread popularity. What is seen and what happens along the way can thus be recorded in videos. However, searching for a specific scene or event among such massive video collections is laborious and tedious. In this paper, we develop a speed-adaptive street view image generation system using general front-mounted DVRs, requiring no additional devices deployed. Visual summaries of street scenes along the way can be provided, allowing users to retrieve a video clip corresponding to a specific road section quickly. An efficient algorithm for estimating the distance a pixel has moved between two consecutive frames is also proposed, so a street view image can be generated with an appropriate aspect ratio without demanding a constant driving speed. Experiments on an extensive data set show that our proposed system can efficiently generate street view images under different lighting and weather conditions, demonstrating its feasibility.

Since a proficient golf swing is a key element in success in golf, golfers always make much effort in improving their swing skills. However, if not under the supervision of a coach, a golfer cannot effectively perceive his/her faults in swings and comprehend how to change the swing mechanics. In this paper, we propose a vision-based golf training system, named "Improve My Golf Swing," which aims at improving the golfer's swing skills using intelligent glasses serving as a wearable displayer. The golfer's posture is analyzed through computation-efficient image processing techniques. Involving professional golf training knowledge, the proposed system automatically generates feature lines to evaluate whether the golfer's body is properly aligned. During a swing, the golfer can watch his/her own posture as well as the augmented feature lines displayed on the intelligent glasses. Instead of quantitative swing analysis, we focus on providing instructions of posture adjustment to the golfer in a more intuitive and unambiguous manner. Different audio alerts will be given to the golfer according to which part of a posture fault is detected. Experiments on different golf swing videos show encouraging results.

With the exponential growth of web multimedia contents, the Internet is rife with near-duplicate videos, the video copies applied with visual/temporal transformations and/or post productions. Two critical issues, copyright infringement and search result redundancy, arise accordingly. To resolve these problems, this paper proposes a spatiotemporal pattern-based approach under the hierarchical filter-and-refine framework for efficient and effective near-duplicate video retrieval and localization . Firstly, non-near-duplicate videos are fast filtered out through a computationally efficient data structure, termed pattern -based index tree (PI-tree). Then, an m- pattern -based dynamic programming (mPDP) algorithm is designed to localize near-duplicate segments and to re-rank the videos retrieved. The influence of time shift misalignment can be alleviated by time-shift m-pattern similarity (TPS) measurement. Comprehensive experiments on the five datasets are conducted to verify the effectiveness, efficiency, robustness, and scalability of the proposed approach. Convincing results demonstrate that our proposed approach outperforms the state-of-the-art approaches in terms of mean average precision (MAP) and normalized detection cost rate (NDCR) on the testing datasets. Furthermore, the proposed approach can achieve high quality of near-duplicate video localization in terms of quality frames (QF) and mean F1.

In several applications, sequence databases generally update incrementally with time. Obviously, it is impractical and inefficient to re-mine sequential patterns from scratch every time a number of new sequences are added into the database. Some recent studies have focused on mining sequential patterns in an incremental manner; however, most of them only considered patterns extracted from time point-based data. In this paper, we proposed an efficient algorithm, Inc_TPMiner, to incrementally mine sequential patterns from interval-based data. We also employ some optimization techniques to reduce the search space effectively. The experimental results indicate that Inc_TPMiner is efficient in execution time and possesses scalability. Finally, we show the practicability of incremental mining of interval-based sequential patterns on real datasets.

With the explosive growth of the social multimedia sharing, copyright protection and search result refinement are always the critical issues for the service operators. To resolve the problems, content-based near-duplicate video retrieval is developed in recent years. In this paper, we construct a condensed Pattern-based Prefix tree (PP-tree) to index the patterns of reference videos for fast retrieval. To calculate how likely a query video and a reference video are near-duplicates, a novel algorithm for discovering the temporal relations among patterns is proposed. Comprehensive experiments on public datasets are conducted to verify the effectiveness and efficiency of the proposed method. Experimental results show that the proposed near-duplicate video retrieval approach outperforms the state-of-the-art approaches in terms of precision, recall, and execution time.

Jump actions are typically accompanied by spiking and imply significant events in volleyball matches. In this paper, we propose an effective system capable of jump pattern recognition in player moving trajectories from long broadcast volleyball videos. First, the entire video is segmented into clips of rallies by shot segmentation and whistle detection. Then, camera calibration is adopted to find the correspondence between coordinates in the video frames and real-world coordinates. With the homographic transformation matrix computed, real-world player moving trajectories can be derived by a sequence of tracked player locations in video frames. Jump patterns are recognized from the player moving trajectory by using a sliding window scheme with physics-based validation and context constraint. Finally, the jump locations can be estimated and jump tracks can be separated from the planar moving tracks. The experiments conducted on broadcast volleyball videos show promising results.

With the exponential growth of the web multimedia contents, illegal video copies are widespread and easy to be obtained from search engines and video sharing websites. For video copyright protection, near-duplicate video retrieval becomes more and more important. In this paper, we proposed a Pattern Set based Dynamic Programming (PSDP) algorithm to retrieve near-duplicate videos efficiently and effectively. In addition, the precise positions of the near-duplicate segments in videos can also be located. To better deal with the slow/fast motion and the dropped frame problem, the Time-shift Pattern set Similarity (TPS) is then applied. Two datasets are used to evaluate the effectiveness and efficiency of the proposed method, and the experimental results show that the proposed method outperforms the compared approaches in terms of precision and execution time in both two datasets.

Serve receive-to-attack (SR2A) is the most principal way to gain points in volleyball games. In addition, the positions of players on the court reveal informative clues about both offensive and defensive formations. Hence, in this paper, we propose an effective system capable of extracting SR2A periods from long broadcast volleyball videos (taken from a pan-tilt-zoom camera), and then locating the players using a novel histogram-based approach. Enriched visual presentation can be provided to give the audience or professional players/coaches a further sight into the game. The proposed system consists of four major processing modules, including court detection, SR2A period extraction, camera calibration, and histogram-based player localization. The experiments conducted on broadcast volleyball videos show promising results.

In volleyball matches, spiking is the most effective way to gain points, while blocking is the action to prevent the opponents from getting scores by spiking. In this paper, we propose an intelligent system for automatic spiking events detection and blocking pattern classification in real volleyball videos. First, the entire videos are segmented into clips of rallies by whistle detection. Then, we find the court region based on proper camera calibration, and detect the location of the net for judging the positions of spiking and blocking. Via analyzing the changes of moving pixels along the net, we make a bounding box around the blocking location, so as to classify the blocking patterns into two main categories based on the width of bounding box. Finally, two important tactic patterns, delayed spiking and alternate position spiking, are recognized. With the information of spiking events and blocking locations, we can collect the statistical data and make tactics inference easily. To the best of our knowledge, no previous work is focused on spiking or blocking event detection. The experimental results on the videos recorded by a university volleyball team are promising and demonstrate the effectiveness of our proposed scheme.

In baseball games, different release points of pitchers form several kinds of pitching styles. Different pitching styles possess individual advantages. This paper presents a novel pitching style recognition approach for automatic generation of game information and video annotation. First, an effective object segmentation algorithm is designed to compute the body contour and extract the pitcher's body. Then, star skeleton is used as the representative descriptor of the pitcher posture for pitching style recognition. The proposed approach has been tested on broadcast baseball video and the promising experimental results validate the robustness and practicability.

Free viewpoint video presentation is a new challenge in multimedia analysis. This paper presents an innovative physics-based scheme to reconstruct the 3D ball trajectory from single-camera volleyball video sequences for free viewpoint virtual replay. The problem of 2D-to-3D inference is arduous due to the loss of 3D information in projection to 2D images. The proposed scheme incorporates the domain knowledge of court specification and the physical characteristics of ball motion to accomplish the 2D-to-3D inference. Motion equations with the parameters are set up to define the 3D trajectories based on physical characteristics. Utilizing the geometric transformation of camera calibration, the 2D ball coordinates extracted over frames are used to approximate the parameters of the 3D motion equations, and finally the 3D ball trajectory can be reconstructed from single-camera sequences. The experiments show promising results. The reconstructed 3D trajectory enables the free viewpoint virtual replay and enriched visual presentation, making game watching a whole new experience.

In basketball games, screen is a blocking move performed by an offensive player, who stands beside or behind a defender, in order to free a teammate to shoot, to receive a pass, or to drive in to score. Screen is the fundamental essence that most offensive tactics are executed with. In this paper, a screen- strategy analysis system is designed, and through combining the identified screens, what tactics are executed in basketball games can be speculated. The proposed system is capable of court region detection, camera calibration and player extraction. Player trajectories are computed by a Kalman filter-based tracking method and mapped to the real-world court coordinates. The player position/trajectory information greatly assists professional-oriented applications such as screen-strategy analysis and tactic inference. The experiments on broadcast basketball videos show encouraging results.

Image annotation based on visual features has been a difficult problem due to the diverse associations that exist between visual features and human concepts. In this paper, we propose a novel approach called Annotation by Image-to-Concept Distribution Model (AICDM) for image annotation by discovering the associations between visual features and human concepts from image-to-concept distribution. Through the proposed image-to-concept distribution model, visual features and concepts can be bridged to achieve high-quality image annotation. In this paper, we propose to use “visual features”, “models”, and “visual genes” which represent analogous functions to the biological chromosome, DNA, and gene. Based on the proposed models using entropy, tf-idf, rules, and SVM, the goal of high-quality image annotation can be achieved effectively. Our empirical evaluation results reveal that the AICDM method can effectively alleviate the problem of visual-to-concept diversity and achieve better annotation results than many existing state-of-the-art approaches in terms of precision and recall.

Up to the present, the contemporary studies are not really successful in image annotation due to some critical problems like diverse regularities between visual features and human concepts. Such diverse regularities make it hard to annotate the image semantics correctly. In this paper, we propose a novel approach called AICDM (Annotation by Image-Concept Distribution Model) for image annotation by discovering the associations between visual features and human concepts from image-concept distribution. Through the proposed image-concept distribution model, the uncertain regularities between visual features and human concepts can be clarified for achieving high-quality image annotation. The empirical evaluation results also reveal that our proposed AICDM method can effectively alleviate the uncertain regularity problem and bring out better annotation results than other existing approaches in terms of precision and recall.