Obstacle avoidance and environment sensing are crucial applications in autonomous driving and robotics. Among all types of sensors, RGB camera is widely used in these applications as it can offer rich visual contents with relatively low-cost, and using a single image to perform depth estimation has become one of the main focuses in resent research works. However, prior works usually rely on highly complicated computation and power-consuming GPU to achieve such task; therefore, we focus on developing a real-time light-weight system for depth prediction in this paper. Based on the well-known encoder-decoder architecture, we propose a supervised learning-based CNN with detachable decoders that produce depth predictions with different scales. We also formulate a novel log-depth loss function that computes the difference of predicted depth map and ground truth depth map in log space, so as to increase the prediction accuracy for nearby locations. To train our model efficiently, we generate depth map and semantic segmentation with complex teacher models. Via a series of ablation studies and experiments, it is validated that our model can efficiently performs real-time depth prediction with only 0.32M parameters, with the best trained model outperforms previous works on KITTI dataset for various evaluation matrices.

Drones are well-liked nowadays. However, deep learning models for object detection still cannot have high detection rates for pedestrians in aerial images even though they already show high precision on PASCAL VOC 2007. The main challenges of aerial image analysis include: (i) the size of an object in aerial images can be very small, and (ii) the objects in aerial images are tilted outward due to perspective projection deformation, which make the pedestrians hard to recognize in aerial images. In this paper, we utilize image partition and vanishing point transformation to overcome the above challenges. Experimental results demonstrate that such pre-processing methods can indeed increase the detection rates significantly for some deep learning models.

People locations bring rich information for a wide spectrum of applications in intelligent video surveillance systems. In addition to localization accuracy, computational efficiency is another significant issue to be highly concerned in people localization. As an essential early stage, people localization has to be accomplished in a very short time, enabling further semantic analysis. However, most state-of-the-art people localization methods pay little attention to computational efficiency. Hence, in this paper we propose an effective and efficient multi-view people localization scheme with several acceleration mechanisms. First, a torso-high reference plane is introduced since in general the torso part (after foreground segmentation) is more intact and stable than the other parts of a human body, and thus can predict potential people locations more reliably. Then, a novel and computationally efficient bitwise-operation scheme is proposed to predict people locations at the intersection regions of foreground line samples from multiple views. After rule-based validation, people locations can be accurately obtained and visualized on a real world plane. Experiments on multi-view surveillance videos not only validate the high accuracy of the proposed method in locating people under crowded scenes with serious occlusions, but also demonstrate an outstanding computational speed.

As an essential step in many computer vision tasks, camera calibration has been studied extensively. In this paper, we propose a novel calibration technique that, based on geometric analysis, camera parameters can be estimated effectively and accurately from just one view of only five corresponding points. Our core contribution is the geometric analysis for deriving the basic equations to realize camera calibration from four coplanar corresponding points and a fifth noncoplanar one. The position, orientation, and focal length of a zooming camera can be directly estimated with unique solution. The estimated parameters are further optimized by the bundle adjustment technique. The proposed calibration method is examined and evaluated on both computer simulated data and real images. The experimental results confirm the validity of the proposed method that camera parameters can be estimated with sufficient accuracy using just five-point correspondences from a single image, even in the presence of image noise.

In advanced video surveillance systems, people localization is usually a part of the complete system and should be accomplished in a short time so as to reserve sufficient processing time for subsequent high-level analysis, such as abnormal event/behavior detection and intruder detection. Hence, in addition to localization accuracy, computational efficiency is of critical importance as well. In this paper, we adopt a vanishing point-based line sampling scheme and propose a fast multicamera people localization approach capable of locating a crowd of dense people and estimating their heights in a fairly short time with high accuracy. For each camera view, sample lines, originated from a vanishing point, of foreground objects are projected onto the ground plane. Then, people locations are estimated by detecting the ground regions containing a high density of the projected lines. Enhanced from some previous works, the proposed approach does not require processing steps of high computation cost, such as projecting all foreground pixels of all views to multiple reference planes or computing pairwise intersections of projected sample lines at different heights. In addition, some novel acceleration modules, such as torso validation and physical rule-based filtering, are developed to further reduce the computation time of people localization. The experiments on real surveillance scenes validate that the proposed approach achieves significant speedup (up to 186%) over state-of-the-art techniques while still ensure a comparably high localization accuracy, even for crowded scenes with serious occlusions.

Traditional video annotation approaches focus on annotating keyframes/shots or whole videos with semantic keywords. However, the extraction processes of keyframes/shots might lack semantic meanings, and it is hard to use a few keywords to describe the content of a long video with multiple topics. In this work, near-scenes, which contain similar concepts, topics, or semantic meanings, are designed for better video content understanding and annotation. We propose a novel framework of hierarchical video-to-near-scene annotation not only to preserve but also to purify the semantic meanings of near-scenes. To detect near-scenes, a pattern-based prefix tree is first constructed to fast retrieve near-duplicate videos. Then, the videos containing similar near-duplicate segments and similar keywords are clustered with consideration of multimodal features including visual and textual features. To enhance the precision of near-scene detection, a pattern-to-intensity-mark (PIM) method is proposed to perform precise frame-level near-duplicate segment alignment. For each near-scene, a video-to-concept distribution model is designed to analyze the representativeness of keywords and discriminations of clusters by the proposed potential term frequency and inverse document frequency and entropy. Tags are ranked according to video-to-concept distribution scores, and the tags with the highest scores are propagated to near-scenes detected. Extensive experiments demonstrate that the proposed PIM outperforms state-of-the-art approaches compared in terms of quality segments and quality frames for near-scene detection. Furthermore, the proposed framework of hierarchical video-to-near-scene annotation can achieve high quality of near-scene annotation in terms of mean average precision.

With the rapid development and reduced cost of digital video capturing devices, driving video recorders (DVRs) begin to gain widespread popularity. What is seen and what happens along the way can thus be recorded in videos. However, searching for a specific scene or event among such massive video collections is laborious and tedious. In this paper, we develop a speed-adaptive street view image generation system using general front-mounted DVRs, requiring no additional devices deployed. Visual summaries of street scenes along the way can be provided, allowing users to retrieve a video clip corresponding to a specific road section quickly. An efficient algorithm for estimating the distance a pixel has moved between two consecutive frames is also proposed, so a street view image can be generated with an appropriate aspect ratio without demanding a constant driving speed. Experiments on an extensive data set show that our proposed system can efficiently generate street view images under different lighting and weather conditions, demonstrating its feasibility.

Vision-based video surveillance systems have gained increasing popularity. However, their functionality is substantially limited under nighttime conditions due to the poor visibility caused by improper illumination. Equipped on night vision cameras, ordinary infrared (IR) illuminators of fixed-intensity usually lead to the imaging problem of overexposure (or underexposure) when the object is too close to (or too far from) the camera. To overcome this limitation, we use a novel multi-intensity IR illuminator to extend the effective range of distance of camera surveillance, and establish in this paper the MI3 (Multi-Intensity Infrared Illumination) database based on such an illuminator. The database contains intensity varying video sequences of several indoor and outdoor scenes. Ground truths including people counting and foreground labelling are provided for different research usages. Performances of related algorithms are tested for demonstration and evaluation.

Since a proficient golf swing is a key element in success in golf, golfers always make much effort in improving their swing skills. However, if not under the supervision of a coach, a golfer cannot effectively perceive his/her faults in swings and comprehend how to change the swing mechanics. In this paper, we propose a vision-based golf training system, named "Improve My Golf Swing," which aims at improving the golfer's swing skills using intelligent glasses serving as a wearable displayer. The golfer's posture is analyzed through computation-efficient image processing techniques. Involving professional golf training knowledge, the proposed system automatically generates feature lines to evaluate whether the golfer's body is properly aligned. During a swing, the golfer can watch his/her own posture as well as the augmented feature lines displayed on the intelligent glasses. Instead of quantitative swing analysis, we focus on providing instructions of posture adjustment to the golfer in a more intuitive and unambiguous manner. Different audio alerts will be given to the golfer according to which part of a posture fault is detected. Experiments on different golf swing videos show encouraging results.

The widespread use of vision-based video surveillance systems has inspired many research efforts on people localization. One of the current main trends in this field is based on probabilistic occupancy map (POM) obtained from multiple camera views. Although the POM-based approaches are robust against noisy foregrounds and can achieve great localization accuracy, they require high computation complexity. In this paper, two enhancement schemes are proposed to improve the efficiency of the POM-based people localization: (i) quick screening of potential people locations, and (ii) timely termination of iterations for occupancy probability estimation. Experimental results show that the proposed approach achieves up to 7.25 times speed-up compared to the standard POM-based approach, while delivering comparable people localization accuracy.

With the exponential growth of web multimedia contents, the Internet is rife with near-duplicate videos, the video copies applied with visual/temporal transformations and/or post productions. Two critical issues, copyright infringement and search result redundancy, arise accordingly. To resolve these problems, this paper proposes a spatiotemporal pattern-based approach under the hierarchical filter-and-refine framework for efficient and effective near-duplicate video retrieval and localization . Firstly, non-near-duplicate videos are fast filtered out through a computationally efficient data structure, termed pattern -based index tree (PI-tree). Then, an m- pattern -based dynamic programming (mPDP) algorithm is designed to localize near-duplicate segments and to re-rank the videos retrieved. The influence of time shift misalignment can be alleviated by time-shift m-pattern similarity (TPS) measurement. Comprehensive experiments on the five datasets are conducted to verify the effectiveness, efficiency, robustness, and scalability of the proposed approach. Convincing results demonstrate that our proposed approach outperforms the state-of-the-art approaches in terms of mean average precision (MAP) and normalized detection cost rate (NDCR) on the testing datasets. Furthermore, the proposed approach can achieve high quality of near-duplicate video localization in terms of quality frames (QF) and mean F1.

With the explosive growth of the social multimedia sharing, copyright protection and search result refinement are always the critical issues for the service operators. To resolve the problems, content-based near-duplicate video retrieval is developed in recent years. In this paper, we construct a condensed Pattern-based Prefix tree (PP-tree) to index the patterns of reference videos for fast retrieval. To calculate how likely a query video and a reference video are near-duplicates, a novel algorithm for discovering the temporal relations among patterns is proposed. Comprehensive experiments on public datasets are conducted to verify the effectiveness and efficiency of the proposed method. Experimental results show that the proposed near-duplicate video retrieval approach outperforms the state-of-the-art approaches in terms of precision, recall, and execution time.

Advanced vehicle safety is a recently emerging issue, appealed from the rapidly explosive population of car owners. Increasing driver assistance systems have been designed for warning drivers of what should be noticed by analyzing surrounding environments with sensors and/or cameras. As one of the hazard road conditions, road bumps not only damage vehicles but also cause serious danger, especially at night or under poor lighting conditions. In this paper we propose a vision-based road bump detection system using a front-mounted car camcorder, which tends to be widespread deployed. First, the input video is transformed into a time-sliced image, which is a condensed video representation. Consequently, we estimate the vertical motion of the vehicle based on the time-sliced image and infer the existence of road bumps. Once a bump is detected, the location fix obtained from GPS is reported to a central server, so that the other vehicles can receive warnings when approaching the detected bumpy regions. Encouraging experimental results show that the proposed system can detect road bumps efficiently and effectively. It can be expected that traffic security will be significantly promoted through the mutually beneficial mechanism that a driver who is willing to report the bumps he/she meets can receive warnings issued from others as well.

Jump actions are typically accompanied by spiking and imply significant events in volleyball matches. In this paper, we propose an effective system capable of jump pattern recognition in player moving trajectories from long broadcast volleyball videos. First, the entire video is segmented into clips of rallies by shot segmentation and whistle detection. Then, camera calibration is adopted to find the correspondence between coordinates in the video frames and real-world coordinates. With the homographic transformation matrix computed, real-world player moving trajectories can be derived by a sequence of tracked player locations in video frames. Jump patterns are recognized from the player moving trajectory by using a sliding window scheme with physics-based validation and context constraint. Finally, the jump locations can be estimated and jump tracks can be separated from the planar moving tracks. The experiments conducted on broadcast volleyball videos show promising results.

The widespread use of vision-based surveillance systems has inspired many research efforts on people localization. In this paper, a series of novel image transforms based on the vanishing point of vertical lines is proposed for enhancement of the probabilistic occupancy map (POM)-based people localization scheme. Utilizing the characteristic that the extensions of vertical lines intersect at a vanishing point, the proposed transforms, based on image or ground plane coordinate system, aims at producing transformed images wherein each standing/walking person will have an upright appearance. Thus, the degradation in localization accuracy due to the deviation of camera configuration constraint specified can be alleviated, while the computation efficiency resulted from the applicability of integral image can be retained. Experimental results show that significant improvement in POM-based people localization for more general camera configurations can indeed be achieved with the proposed image transforms.

With the exponential growth of the web multimedia contents, illegal video copies are widespread and easy to be obtained from search engines and video sharing websites. For video copyright protection, near-duplicate video retrieval becomes more and more important. In this paper, we proposed a Pattern Set based Dynamic Programming (PSDP) algorithm to retrieve near-duplicate videos efficiently and effectively. In addition, the precise positions of the near-duplicate segments in videos can also be located. To better deal with the slow/fast motion and the dropped frame problem, the Time-shift Pattern set Similarity (TPS) is then applied. Two datasets are used to evaluate the effectiveness and efficiency of the proposed method, and the experimental results show that the proposed method outperforms the compared approaches in terms of precision and execution time in both two datasets.

The widespread use of vision-based surveillance systems has inspires many research efforts on people localization. In this paper, a novel nonlinear image transform based on the vanishing point of vertical lines, entitled VP-transform, is proposed for enhancement of the people localization scheme proposed in [1]. Utilizing the characteristic that the extensions of vertical lines intersect at a vanishing point, the proposed transformation aims at producing rectified images wherein each standing/walking person will have an upright appearance. Thus, the degradation in localization accuracy, due to deviation of the constraint of camera configuration specified in [1], can be alleviated. Experimental results validate the effectiveness of the proposed image transform that significant improvement in people localization can be achieved.

Serve receive-to-attack (SR2A) is the most principal way to gain points in volleyball games. In addition, the positions of players on the court reveal informative clues about both offensive and defensive formations. Hence, in this paper, we propose an effective system capable of extracting SR2A periods from long broadcast volleyball videos (taken from a pan-tilt-zoom camera), and then locating the players using a novel histogram-based approach. Enriched visual presentation can be provided to give the audience or professional players/coaches a further sight into the game. The proposed system consists of four major processing modules, including court detection, SR2A period extraction, camera calibration, and histogram-based player localization. The experiments conducted on broadcast volleyball videos show promising results.

In volleyball matches, spiking is the most effective way to gain points, while blocking is the action to prevent the opponents from getting scores by spiking. In this paper, we propose an intelligent system for automatic spiking events detection and blocking pattern classification in real volleyball videos. First, the entire videos are segmented into clips of rallies by whistle detection. Then, we find the court region based on proper camera calibration, and detect the location of the net for judging the positions of spiking and blocking. Via analyzing the changes of moving pixels along the net, we make a bounding box around the blocking location, so as to classify the blocking patterns into two main categories based on the width of bounding box. Finally, two important tactic patterns, delayed spiking and alternate position spiking, are recognized. With the information of spiking events and blocking locations, we can collect the statistical data and make tactics inference easily. To the best of our knowledge, no previous work is focused on spiking or blocking event detection. The experimental results on the videos recorded by a university volleyball team are promising and demonstrate the effectiveness of our proposed scheme.

In baseball games, different release points of pitchers form several kinds of pitching styles. Different pitching styles possess individual advantages. This paper presents a novel pitching style recognition approach for automatic generation of game information and video annotation. First, an effective object segmentation algorithm is designed to compute the body contour and extract the pitcher's body. Then, star skeleton is used as the representative descriptor of the pitcher posture for pitching style recognition. The proposed approach has been tested on broadcast baseball video and the promising experimental results validate the robustness and practicability.

Free viewpoint video presentation is a new challenge in multimedia analysis. This paper presents an innovative physics-based scheme to reconstruct the 3D ball trajectory from single-camera volleyball video sequences for free viewpoint virtual replay. The problem of 2D-to-3D inference is arduous due to the loss of 3D information in projection to 2D images. The proposed scheme incorporates the domain knowledge of court specification and the physical characteristics of ball motion to accomplish the 2D-to-3D inference. Motion equations with the parameters are set up to define the 3D trajectories based on physical characteristics. Utilizing the geometric transformation of camera calibration, the 2D ball coordinates extracted over frames are used to approximate the parameters of the 3D motion equations, and finally the 3D ball trajectory can be reconstructed from single-camera sequences. The experiments show promising results. The reconstructed 3D trajectory enables the free viewpoint virtual replay and enriched visual presentation, making game watching a whole new experience.

In basketball games, screen is a blocking move performed by an offensive player, who stands beside or behind a defender, in order to free a teammate to shoot, to receive a pass, or to drive in to score. Screen is the fundamental essence that most offensive tactics are executed with. In this paper, a screen- strategy analysis system is designed, and through combining the identified screens, what tactics are executed in basketball games can be speculated. The proposed system is capable of court region detection, camera calibration and player extraction. Player trajectories are computed by a Kalman filter-based tracking method and mapped to the real-world court coordinates. The player position/trajectory information greatly assists professional-oriented applications such as screen-strategy analysis and tactic inference. The experiments on broadcast basketball videos show encouraging results.

In baseball, the strike zone plays an important role in each pitch. Pitches which pass through the strike zone count as strikes, three of which strike out the batter. Thus, pitchers should acquire mastery of the strike zone. Moreover, the strike zone also provides the reference for positioning the pitch locations, about which sports fans and professionals have an intense interest in compile statistics. This paper presents a stance-based approach for strike zone shaping and visualization in broadcast baseball video. We design efficient and effective algorithms to detect the home plate, contour the batter, locate the features points on the batter contour, and finally outline the strike zone. The experiments show that the proposed framework is able to shape the strike zone fairly well in various broadcast baseball video sequences, needless of manual operation and additional camera setting.

Despite a lot of research efforts in sports video processing, little work was done in volleyball video analysis due to the high density of players on the court and the complicated overlapping of player-player or ball-player, which lead to great complexity of object tracking for advanced analysis. For ball detection and trajectory extraction in volleyball videos, this paper presents a physics-based scheme which utilizes the motion characteristics to extract ball trajectory from lots of moving objects. The experiments of ball tracking show encouraging results. Moreover, based on game-specific properties, the ball trajectory can be exploited to recognize set types for tactics inference and to detect basic actions in the volleyball game for close-up presentation.

In this paper, we propose a system that can automatically segment a basketball video into several clips on the basis of a GOP-based scene change detection method. The length of each clip and the number of dominant color pixels of each frame are used to classify shots into close-up view, medium view, and full court view. Full court view shots are chosen to do advanced analyses such as ball tracking and parameter extracting for the transformation from a 3D real-world court to a 2D image. After that, we map points in the 2D image to the corresponding coordinates in a real-world court by some physical properties of the 3D shooting trajectory, and compute the statistics of all shooting positions. Eventually we can obtain the information about the most possible shooting positions of a professional basketball team, which is useful for opponents to adopt appropriate defense tactics.

This paper presents a novel method for reducing a baseball video segment from one batter to next batter into a more compact pitch-by-pitch video by pitching ball trajectory detection. The pitch-by-pitch video shows the complete pitching and batting process and largely reduces the source video data, making pitching analysis of broadcast baseball sequences an easier task. The proposed method has been tested for several long sequences, and promising results are reported.

This paper proposes a music accompaniment system capable of catching the tempo of music. The original signal is first reduced to a detection function revealing the pulses of music. To induce the tempo and locate beats, a Fourier analysis-based algorithm with high practicality and generality is designed that it is robust to beat strength and not restricted to specific music genres. The regularity and periodicity of music beats are extracted and even the missing beats are recovered. The performance is validated using a comprehensive testing data set, and results in both formal objective experiments and subjective listening evaluations show convincible performance. Moreover, an interactive interface is designed that users are allowed to select an instrument to accompany the music based on the obtained tempo.