The dilemma between plasticity and stability arises as a common challenge for incremental learning. In contrast, the human memory system is able to remedy this dilemma owing to its multilevel memory structure, which motivates us to propose a Bilevel Memory system with Knowledge Projection (BMKP) for incremental learning. BMKP decouples the functions of learning and remembering via a bilevel-memory design: a working memory responsible for adaptively model learning, to ensure plasticity; a long-term memory in charge of enduringly storing the knowledge incorporated within the learned model, to guarantee stability. However, an emerging issue is how to extract the learned knowledge from the working memory and assimilate it into the long-term memory. To approach this issue, we reveal that the parameters learned by the working memory are actually residing in a redundant high-dimensional space, and the knowledge incorporated in the model can have a quite compact representation under a group of pattern basis shared by all incremental learning tasks. Therefore, we propose a knowledge projection process to adaptively maintain the shared basis, with which the loosely organized model knowledge of working memory is projected into the compact representation to be remembered in the long-term memory. We evaluate BMKP on CIFAR-10, CIFAR-100, and Tiny-ImageNet. The experimental results show that BMKP achieves state-of-the-art performance with lower memory usage11The code is available at https://github.com/SunWenJu123/BMKP.

Zero-shot learning (ZSL) aims to recognize unseen categories whose data is unavailable during the training stage. Most existing ZSL algorithms focus on learning an embedding space and determine the classes of test samples according to sample-prototype similarities in this space. However, we observe that, in contrast to the single sample-prototype relationship, an ensemble criterion usually benefits the final classification, just as the saying "more than one". Inspired by this, we introduce a novel cluster-prototype matching (CPM) strategy and propose a ZSL framework based on CPM. Firstly, we learn a mapping between the visual space and the semantic space utilizing a well-established ZSL algorithm. Via the learned mapping, all test samples are projected into the embedding space and clustered in this space. Secondly, two CPM methods, soft-CPM and hard-CPM, are proposed to match clusters and class prototypes, along with cluster-prototype similarities calculated. Finally, the label of each sample is determined by the combination of the sample-prototype similarity and the cluster-prototype similarity. We apply our framework to five basic ZSL methods and compare them with several advanced baselines of ZSL. The experimental results demonstrate that the proposed framework can significantly improve the performance of the basic ZSL models and help them achieve or beyond the state-of-the-art.

Lightning prediction is a complicated and challenging task requiring meteorologists to integrate information from multiple data sources to make decisions. Although some data-driven models have been proposed to make prediction automatically, most of them are based on a single data source or several basically-homogeneous data sources, making them hard to adapt to complex and diverse data in practice. In this work, we propose a heterogeneous spatiotemporal network (HSTN) for lightning prediction, aiming at mining knowledge from several heterogeneous spatiotemporal (ST) data sources. Specifically, HSTN comprises three modules: Gaussian diffusion module, ST encoder and ST decoder. Noting that most of meteorological data can be formatted into either a dense ST tensor or a sparse ST tensor, the ST encoder, with the help of the Gaussian diffusion module, is designed to extract information from both two types of tensors. On the other hand, ST decoder is responsible for merging all information from the other modules and generate the final prediction. By organically combining the three modules, HSTN can handle complex input with heterogeneity in both space and time domains. We conduct experimental evaluations on a real-world lightning dataset. The results demonstrate that HSTN achieves state-of-the-art performance compared with several established baselines.