Image captioning is an interesting and challenging task. The previously established image captioning approach is based mainly on the encoder-decoder architecture, but it suffers from problems such as inaccurate captioning information, and the generated captioning sentences are not sufficiently rich. This paper proposes a novel image captioning model that is based on a self-attention network and a scene graph relationship network. First, an improved self-attention network is added to the extraction of visual features to evaluate the effectiveness of image global information for image generation. Then, we design a visual intensity parameter to coordinate the strategies of visual features and language model for word generation. Finally, a graph convolutional network is designed to extract the relationships from the scene information to render the generated caption more exciting and to increase the accuracy of the fine-grained captioning. We demonstrated the satisfactory performance of the model on the MS-COCO and Flickr 30K datasets. The experimental results demonstrate that the proposed model realizes state-of-the-art performance.

Automatically generating a human-like description for a given image is a potential research in artificial intelligence, which has attracted a great of attention recently. Most of the existing attention methods explore the mapping relationships between words in sentence and regions in image, such unpredictable matching manner sometimes causes inharmonious alignments that may reduce the quality of generated captions. In this paper, we make our efforts to reason about more accurate and meaningful captions. We first propose word attention to improve the correctness of visual attention when generating sequential descriptions word-by-word. The special word attention emphasizes on word importance when focusing on different regions of the input image, and makes full use of the internal annotation knowledge to assist the calculation of visual attention. Then, in order to reveal those incomprehensible intentions that cannot be expressed straightforwardly by machines, we inject external knowledge extracted from knowledge graph into the encoder-decoder framework to facilitate meaningful captioning. We validate our model on two freely available captioning benchmarks: Microsoft COCO dataset and Flickr30k dataset. The results demonstrate that our approach achieves state-of-the-art performance and outperforms many of the existing approaches.

Due to the deteriorated quality of feature in the propagation process of the neural network, it may be hard for traditional detector to identify a small object by just utilizing information within one region proposal. To overcome the limitation of the traditional object detector, we proposed a graph based relation mining network, to capture the relation information from labels and images. The semantic relation network is proposed to mine the global semantic relation in labels, and the spatial relation network is proposed to capture the local spatial relation in images. The feature representation is further improved by aggregating the outputs of the two networks. Instead of directly disseminating visual features in the network, the relation mining network explores more advanced feature information. Experiments on the PASCAL VOC and MS COCO datasets demonstrate that key relation information significantly improve the performance of object detection with better ability to detect small objects and reasonable bounding box. The results on COCO dataset demonstrate our method can detect objects robustly, increasing the detection performance of small objects from average precision and average recall by 4.7% and 7.6% respectively in performance relative to Faster R-CNN.

In this paper, we study the problem of image-text matching in order to make the image and text have better semantic matching. In the previous work, people just simply used the pre-training network to extract image and text features and project directly into a common subspace, or change various loss functions on this basis, or use the attention mechanism to directly match the image region proposals and the text phrases. This is not a good match for the semantics of the image and the text. In this study, we propose a method of cross-media retrieval based on global representation and local representation. We constructed a cross-media two-level network to explore better semantic matching between images and text, which contains subnets that handle both global and local features. Specifically, we not only use the self-attention network to obtain a macro representation of the global image but also use the local fine-grained patch with the attention mechanism. Then, we use a two-level alignment framework to promote each other to learn different representations of cross-media retrieval. The innovation of this study lies in the use of more comprehensive features of image and text to design the two kinds of similarity and add them up in some way. Experimental results show that this method is effective in image-text retrieval. Experimental results on the Flickr30K and MS-COCO datasets show that this model has a better recall rate than many of the current advanced cross-media retrieval models.

To learn a well-performed image annotation model, a large number of labeled samples are usually required. In this paper, we propose a novel semi-supervised approach based on adaptive weighted fusion for automatic image annotation, which can utilize the labeled data and unlabeled data simultaneously. Firstly, two different classifiers, namely the CNN (convolutional neural network) and the LDA-SVM, are constructed by all the labeled data. These two classifiers are independently represented as different feature views. Then, the most confident data with relevant pseudo-labels are chosen and amalgamated with the whole labeled dataset. After that, the two classifiers are retrained with the new labeled dataset until a stop condition is reached. In each iteration process, the unlabeled samples are labeled by high confidence pseudo-labels that are estimated by an adaptive weighted fusion strategy. Finally, we conduct experiments on two datasets, namely IAPR TC12 and NUS-WIDE, and measure the performance of the model with standard criteria, including precision, recall, F-measure, N+ and mAP. The experimental results show that our approach outperforms many state-of-the-art automatic image annotation approaches.

The main factors affecting the shooting rate are the angle of shooting, the speed of shot, the height of the shot and the trajectory of the basketball under the influence of direction and speed of wind. In this paper, we propose the method to analyze the influence factors of shooting rate based on trajectory prediction. It can accurately get the motion trajectory of basketball through the forecast. We use forecast analysis with mechanical theory and motion trajectory to get shot height, angle and the release speed of the shooting of the impact, providing theoretical basis for shooting training. The experimental results show that the shooting parameters can be greatly improved by shooting the relevant parameters of the shooting point.