Lane lines play a crucial role in the traffic system. However, due to the diversity of lane categories, road conditions and weather environments, as well as the different aspect ratios of lane lines, lane detection algorithms face many challenges. This paper proposes a multi-dimensional feature refinement method for complex scene lane detection based on start point guidance. Due to the fact that lane lines often traverse the entire image, capturing sufficient context is crucial, and lane lines also have specific local patterns that require detailed low-level features for accurate localization. We propose global feature refinement (GFR) and lane aware gather (LAG) to refine the features from the following two dimensions: global enhancement and local refinement. To generate high-quality anchors, we predict the start point coordinate of lane instances through start point coordinate prediction (SPCP). To better fit 2D lane detection, we adopt the more general penalty LaneIoU (PLIoU) as the loss function to evaluate the predicted results. Experimental results demonstrate that the proposed method performs well in in lane detection tasks in complex scenes and has strong competitiveness among existing methods.

Multi-view subspace clustering aims to divide a set of multisource data into several groups according to their underlying subspace structure. Although the spectral clustering based methods achieve promotion in multi-view clustering, their utility is limited by the separate learning manner in which affinity matrix construction and cluster indicator estimation are isolated. In this paper, we propose to jointly learn the self-representation, continue and discrete cluster indicators in an unified model. Our model can explore the subspace structure of each view and fusion them to facilitate clustering simultaneously. Experimental results on two benchmark datasets demonstrate that our method outperforms other existing competitive multi-view clustering methods.

Abnormal activity detection is a challenging problem divided into global abnormal activity detection and local abnormal activity detection. First, the hybrid histogram of optical flow feature is extracted; then, the double sparse representation is proposed to tackle the issue of global abnormal activity detection; finally, for the issue of local abnormal activity detection, the foreground of region of interest within the current frame is first detected, and then the method of online weighted clustering is utilized to detect local abnormal activity. Experiments results conducted on UMN datasets and UCSD datasets validate the advantages of the proposed method.

Artificial features are adopted in most of the existing abnormal behavior detection. However, it is difficult to choose and design an effective behavior feature for the reason of highly computational complexity and complex scenarios. To solve this problem, temporal consistency based weak supervised abnormal behavior detection method is proposed in this paper. First, temporal gram matrices are constructed for a given video sequence. Then, a pair of behavior units (Candidate action fragment) are formed by exploiting the temporal consistency and the smoothness of human behavior, which helps to locate the start frame and end frame of the related class of abnormal behavior in the video sequence to train the corresponding classifier. Finally, sparse reconstruction is here utilized to detect abnormal behavior. Experiments conducted on common database including CAVIAR and Crossing demonstrate the effectiveness of the proposed method.

The overwhelming amounts of digital images on the Web and personal computers have triggered the requirement of an effective tool to retrieve images of interest using semantic concepts. Due to the semantic gap between low level content features and its high level semantic features of an image, however, the performances of many existing automatic image annotation algorithms are not so satisfactory. In this paper, a novel approach based on the cognitive science is proposed to improve the quality of annotations. The main idea is that the tags of an image are considered as nodes in a semantic network, and the relevance between the tags and image contents is regulated using the spreading activation theory. After the spreading activation process finishes, each tag will be assigned an appropriate value with respect to its relation to other tags. Experimental results conducted on the 50,000 Flickr images demonstrate that the proposed scheme can effectively improve the performance in automatic image annotation.

Recently learning to rank has become one of the popular means to create a ranking model for social image search. However, the results of existing approaches are not as satisfactory for the large gap between low-level visual features and high-level semantic concepts, and these sophisticated approaches require a significant amount of parameters tuning to be effective and efficient. In this paper, we propose a novel framework for social image re-ranking based on a non-parametric quantum technique, which reranks top retrieved images by considering the interrelationship between images through the quantum estimation and requires no explicit parameter tuning. The basic idea of the proposed framework is inspired by the photon polarization experiment supporting the theory of quantum measurement. Experimental results conducted on the Flickr dataset demonstrate the effectiveness and efficiency of the proposed framework.

An appropriate movie abstraction is helpful for movie producers to promote the progress of the storyline as well as for the audiences to capture the theme of the movie before watching the full-length movie. Most existing movie abstraction schemes rely heavily on video content only, which may not deliver ideal results because of the semantic gap between computer calculated low-level audiovisual features and human used high-level perceptual understanding. In this study, the authors incorporate script into movie content understanding and present a new movie abstraction approach via the progress of the storyline, which is the soul of a film that actually catches the audiences' attention. The authors first segment the movie scenes by analysis of the movie script. Then the authors conduct storyline analysis using the attention analysis and audiovisual features. Given the transition intensity values, the authors calculate the storyline progress score and adopt this as the criterion to generate movie abstraction. The promising experimental results demonstrate that the analysis of storyline evolution is an effective approach for the abstraction and understanding of movie content.

Graph-based semi-supervised learning approaches have been proven effective and efficient in solving the problem of the inefficiency of labeled data in many real-world application areas, such as video annotation. However, the pairwise similarity metric, a significant factor of existing approaches, has not been fully investigated. That is, these graph-based semi-supervised approaches estimate the pairwise similarity between samples mainly according to the spatial property of video data. On the other hand, temporal property, an essential characteristic of video data, is not embedded into the pairwise similarity measure. Accordingly, a novel framework for video annotation, called Joint Spatio-Temporal Correlation Learning (JSTCL), is proposed in this paper. This framework is characterized by simultaneously taking into account the spatial and temporal property of video data to achieve more accurate pairwise similarity values. We apply the proposed framework to video annotation and report superior performance compared to key existing approaches over the benchmark TRECVID data set.

We address the issue of categorizing scenes from feature films into semantic classifications based on the audio-visual cues. Specifically, we first exploit the grammar of film production to specify the semantic content of scenes. Then, each scene is classified into one of the following categories: conversation, action and suspense. Finally, to achieve more specific scene and consist with human perception, conversation scene is further categorizes into emotional conversation and common one, and action scene is further categorizes into gunfight, beating and chasing scene. This work is a step toward browsing and retrieval content of feature films in limited bandwidth, video repository, and rating of feature films of interest effectively and efficiently.

Graph-based semi-supervised learning approaches have been proven effective and efficient in solving the problem of the inefficiency of training samples in many real-world application areas, such as video annotation. As a significant factor of these algorithms, however, pairwise similarity metric has not been fully investigated. On the one hand, for general video annotation methods, the estimation of pairwise similarity between two samples relies on the spatial property of video data. On the other hand, temporal property, which is an essential character of video data, is not embedded into the pairwise similarity metric. Therefore, in this paper, a novel method, called Joint Spatio-Temporal Correlation Learning (JSTCL), is proposed to improve the accuracy of video annotation. This method is characterized by simultaneously taking into account both the spatial and temporal property of video data to well represent the pairwise similarity. Experiments conducted on the TRECVID demonstrate the efficiency of the proposed method.

Automatic image annotation is a promising way to achieve more effective image management and retrieval. However, system performances of the existing state-of-the-art keyword annotation schemes are often not so satisfactory. Therefore, image annotation refinement is crucial to improve the imprecise annotation results. In this paper, a novel approach is developed to automatically annotate image content by a semi-supervised learning model. With perceptual visual characteristics, the candidate annotations of unlabelled images are first obtained based on a progressive model. Then, a transducitive model, random walk with restart algorithm is used to refine these candidate annotations and the top ones are reserved as the final annotations. Experiments conducted on the typical Corel dataset show the effectiveness of the proposed approach.

Grouping video contents into semantic segments is the crucial pass to content-based video summarization and retrieval. In this paper, we present a novel scene segmentation and semantic representation scheme for various video types. We first detect video shot using a coarse-to-fine algorithm. The key frames without useful information are detected and removed using template matching. Spatio-temporal coherent shots are then grouped into the same scene based on the temporal constraint of video content and visual similarity of shot activity. With general editing technique used in the continuously recorded video, semantic representation of scene content is specified to satisfy human demand on video retrieval. The proposed algorithm has been performed on various types of videos containing movie and TV program. Promising experimental results shows that the proposed method makes sense to efficient retrieval of video contents of interest.

Automatic image annotation is a promising way to achieve more effective image management and retrieval by using keywords. However, system performances of the existing state-of-the-art keyword annotation schemes are often not so satisfactory. Therefore, image annotation refinement is crucial to improve the imprecise annotation results. In this paper, a novel approach is developed to automatically refine the initial annotation of images. First, for a query image, the candidate annotations are obtained by a step-up model-based algorithm using perceptual visual characteristic. Then, a refine algorithm, fast random walk with restart is used to re-rank the candidate annotations and the top ones are reserved as the final annotations. Experiments conducted on the typical Corel dataset shows that the proposed scheme can effectively improve the automatic annotation performance.

In this letter, a novel framework to segment video scene and represent scene content is proposed. Firstly, video shots are detected using a rough-to-fine algorithm. Secondly, key frames are selected adaptively, and redundant key frames are removed using template matching. Then, spatio-temporal coherent shots are clustered into the same scene. Finally, under the full analysis of typical characters on continuously recorded videos, video scene content is semantically represented to satisfy human demand on video retrieval. Experimental results show the proposed method makes sense to efficient retrieval of video content of interest.