In e-commerce, product related video is important content to introduce product characteristics and attract consumers. Especially in the recommendation system of e-commerce platform, video highlight detection methods are usually adopted to capture the most attractive clips for showing to consumers, so as to improve the click through rate of products. However, the effect of the current research methods applied to the actual scene is not satisfactory. Compared with other video understanding tasks, video highlight detection is relatively abstract and subjective, and it is difficult to make accurate judgment only by using visual information. Consequently, we put forward multi-modal video highlight detection task, which introduces video related linguistic information as supervised information. And we propose a graph-based commodity-aware model to solve multi-modal video highlight detection in e-commerce scene. Our model consists of multi-modal highlight detection stage and graph-based fine-tuning stage, in which we adopt graph aggregation method to fuse multi-source natural language information and introduce effective visual feature composition method for graph convolution network based highlight detection. Besides, we release the largest e-commerce video highlight detection dataset, TaoHighlight, in which the videos and related data are collected from Taobao e-commerce platform. Our model achieves state-of-art in all separate categories and overall dataset of TaoHighlight, which shows the superiority of our model.

With the rapidly increasing number of personal image collections on the web, it is of great importance to annotate these user-uploaded images in personalized manner. But personalized image annotation is largely ignored by the mainstream of image annotation research. In this paper, we focus on personalizing the automatic image annotation by proposing a general framework which jointly exploits the generic content-based image annotation, personal image tagging history and the content of personal history images. In our framework, two sets of candidate annotations are extracted for each image based on content-based annotation and personal image tagging history. Considering that the user's interest may not stay the same, when exploiting the personal image tagging history, we also take the content of personal history images into account to avoid the noise. To get the final annotations, we propose an unsupervised algorithm based on reinforcement learning to combine the above two candidate annotation sets. Encouraging results show that the proposed framework is effective and promising for personalizing automatic image annotation.