Zero-shot learning aims to learn knowledge from existing information to classify new classes with no visual training data. In the current work on zero-shot video classification, only the category name information can be used for unseen classes. While, most of the category names cannot fully describe the entire video information, but are only precise labels assigned by humans to actions, in which the amount of information is very small. In order to make up for the semantic deficiencies of video databases and build relationships between categories, we propose a multi-modal generalized zero-shot video classification framework based on multi-grained semantic information with a proposed video description text database. Our model explores semantic knowledge from accurate but lacking informative category names and exhaustive but redundant description texts, and learns visual knowledge from semantic embeddings of varying granularity. Further, we use the learned semantic and visual knowledge to perform multi-grained classification on test video data with both seen and unseen classes. To describe actions in detail and provide complete semantic information, we propose a description text database. The textual descriptions, including category definitions and explanations, in our proposed textual database effectively help establish relationships between categories, thus providing a more reliable basis for visual feature synthesis. Furthermore, our framework generates synthesized features for unseen classes from both coarse-grained and fine-grained semantic information, which would effectively avoid the bias of generalized zero-shot learning on seen classes. Extensive experimental results on the database prove the validity of our method and the effectiveness of the description texts in generalized zero-shot video classification problems.

Generalized zero-shot video classification aims to train a classifier to classify videos including both seen and unseen classes. Since the unseen videos have no visual information during training, most existing methods rely on the generative adversarial networks to synthesize visual features for unseen classes through the class embedding of category names. However, most category names only describe the content of the video, ignoring other relational information. As a rich information carrier, videos include actions, performers, environments, etc., and the semantic description of the videos also express the events from different levels of actions. In order to use fully explore the video information, we propose a fine-grained feature generation model based on video category name and its corresponding description texts for generalized zero-shot video classification. To obtain comprehensive information, we first extract content information from coarse-grained semantic information (category names) and motion information from fine-grained semantic information (description texts) as the base for feature synthesis. Then, we subdivide motion into hierarchical constraints on the fine-grained correlation between event and action from the feature level. In addition, we propose a loss that can avoid the imbalance of positive and negative examples to constrain the consistency of features at each level. In order to prove the validity of our proposed framework, we perform extensive quantitative and qualitative evaluations on two challenging datasets: UCF101 and HMDB51, and obtain a positive gain for the task of generalized zero-shot video classification.

Zero-shot learning (ZSL) is to classify images according to detailed attribute annotations into new categories that are unseen during the training stage. Generalized zero-shot learning (GZSL) adds seen categories to the test samples. Since the learned classifier has inherent bias against seen categories, GZSL is more challenging than traditional ZSL. However, at present, there is no detailed attribute description dataset for video classification. Therefore, the current zero-shot video classification problem is based on the synthesis of generative adversarial networks trained on seen-class features into unseen-class features for ZSL classification. In order to solve this problem, we propose a description text dataset based on the UCF101 action recognition dataset. To the best of our knowledge, this is the first work to add description of the classes to zero-shot video classification. We propose a new loss function that combines visual features with textual features. We extract text features from the proposed text data set, and constrain the process of generating synthetic features based on the principle that videos with similar text types should be similar. Our method reapplies the traditional zero-shot learning idea to video classification. From the experimental point of view, our proposed dataset and method have a positive impact on the generalized zero-shot video classification.