Abstract Few-shot learning (FSL) aims at distilling transferable knowledge on existing concepts to cope with novel concepts for which only a few labeled data are available. Most of the popular FSL methods acquire this knowledge by learning on large-scale supervised data from the existing concepts. Considering obtaining supervised data might sometimes be difficult and heavy-burden, we pursue a relatively mild prerequisite for FSL, that is, using unsupervised instead of supervised data to acquire the transferable knowledge. We propose a novel easy-to-implement FSL framework, U nsupervised T ractive M omentum (UTM), composed of modular dual encoders, a combinatorial loss mechanism, and a classifier that together form a reusable and extensible learning system, that only requires unsupervised data of existing concepts. UTM randomly samples unsupervised data and augments them to create many synthetic query-key matching tasks on-the-fly, and deploys two different encoders while possessing identical architecture, named traction encoder and momentum encoder , to learn a representation space by a combinatorial parameter updating manner. The representation space learned on unsupervised data is expected to be a good fit to few-shot recognition on novel concepts. UTM is composed of parallelizable dual encoders and optimized for scalable training in GPU-based high-performance computing environments. Theoretical convergence and bound analysis further support its deployment in distributed systems. Theoretical justifications of the parameter updating mechanism in UTM are given from the perspective of convergence, and a theoretical loss bound for UTM is proved, which mathematically quantifies the relationship between our self-supervised UTM and the vanilla supervised method. Extensive experimental evaluation on several benchmark datasets demonstrates that UTM yields significant improvement to state-of-the-art unsupervised methods even very close to supervised methods, which can also be well explained using our theory.

Generating videos with semantic meaning, such as gestures in sign language, is a challenging problem. The model should not only learn to generate videos with realistic appearance, but also take notice of crucial details in frames to convey precise information. In this paper, we focus on the problem of generating long-term gesture videos containing precise and complete semantic meanings. We develop a novel architecture to learn the temporal and spatial transforms in regions of interest, i.e., gesticulating hands or face in our case. We adopt a hierarchical approach for generating gesture videos, by first making predictions on future pose configurations, and then using the encoder-decoder architecture to synthesize future frames based on the predicted pose structures. We develop the scheme of action progress in our architecture to represent how far the action has been performed during its expected execution, and to instruct our model to synthesize actions with various paces. Our approach is evaluated on two challenging datasets for the task of gesture video generation. Experimental results show that our method can produce gesture videos with more realistic appearance and precise meaning than the state-of-the-art video generation approaches.

Human has the ability of drawing inferences about other things from only one instance. Few-shot learning is aimed at imitating this generalized learning behavior of human beings, where the learning machine is expected to recognize novel categories not seen in the training set, given only a few training data for each novel category. In this paper, we enhance the Prototypical Network for few-shot learning tasks by introducing a domain alignment module, which takes into account the domain shifts existing between different categories. Compared to original Prototypical Network (PN), the most excellent model for few-shot learning at present, our proposed Domain Alignment Prototypical Network (DA-PN) is able to abate the distribution differences among the data of training and test classes, further optimizing the embedding space of prototype feature for each category and then boosting few-shot recognition. Comprehensive empirical evidence demonstrates that the proposed DA-PN can yield state-of-the-art few-shot recognition performance on the public benchmark dataset mini-ImageNet as well as a novel proposed few-shot dataset MNIST&CIFAR10.