Text-to-image synthesis aims to generate realistic images conditioned on text descriptions. To fuse text information into synthesized images, conditional affine transformations (CATs), such as conditional batch normalization (CBN) and conditional instance normalization (CIN), are usually used to predict batch statistics of different layers. However, ordinary CAT blocks control the batch statistics independently disregarding the consistency among neighboring layers. To address the above issue, we propose a new fusion approach names recurrent affine transformation (RAT) for synthesizing images conditioned on text information. RAT connects all the CAT blocks with recurrent connections for explicitly fitting the temporal consistency between CAT blocks. To verify the effectiveness of RAT, we propose a novel visualization method to show how generative adversarial network (GAN) fuses conditional information. Our microscopic and macroscopic visualizations not only demonstrate the effectiveness of RAT but also turn out to be a useful perspective to analyze how GAN fuses conditional information. In addition, we propose a more stable spatial attention mechanism for the discriminator, which helps the text description to supervise the generator to synthesize more relevant image contents. Extensive experiments on the CUB, Oxford-102, and COCO datasets demonstrate the proposed model's superiority in comparison to state-of-the-art models. Our code is available onhttps://github.com/senmaoy/RAT-GAN.

We propose a novel attention framework called attentive linear transformation (ALT). Instead of learning the spatial or channel-wise attention in existing models, ALT learns to attend to the high-dimensional transformation matrix from the image feature space to the context vector space. Thus ALT can learn various relevant feature abstractions, including spatial attention, channel-wise attention and visual dependence. Besides, we propose a soft threshold regression to predict the attention probabilities for local regions. Soft threshold regression preserves more useful visual information than popular softmax regression. Extensive experiments on the MS COCO and the Flickr30k datasets demonstrate the superiority of our model compared with other state-of-the-art methods.

Recent progresses in multimedia event detection have enabled us to find videos about a predefined event from a large-scale video collection. Research towards more intrinsic unsupervised video understanding is an interesting but understudied field. Specifically, given a collection of videos sharing a common event of interest, the goal is to discover the salient fragments, i.e., the curt video fragments that can concisely portray the underlying event of interest, from each video. To explore this novel direction, this paper proposes an unsupervised event saliency revealing framework. It first extracts features from multiple modalities to represent each shot in the given video collection. Then, these shots are clustered to build the cluster-level event saliency revealing framework, which explores useful information cues (i.e., the intra-cluster prior, inter-cluster discriminability, and inter-cluster smoothness) by a concise optimization model. Compared with the existing methods, our approach could highlight the intrinsic stimulus of the unseen event within a video in an unsupervised fashion. Thus, it could potentially benefit to a wide range of multimedia tasks like video browsing, understanding, and search. To quantitatively verify the proposed method, we systematically compare the method to a number of baseline methods on the TRECVID benchmarks. Experimental results have demonstrated its effectiveness and efficiency.