Time series forecasting with additional spatial de-pendencies has attracted a tremendous amount of research interest in social sciences, due to its importance in modern real-world applications. The Graph Neural Networks (GNN) is one of the most exciting deep learning techniques among these spatio-temporal modeling approaches. Most existing spatio-temporal GNN frameworks are based on a two-step modeling process. In such scenario, spatial and temporal dependencies are modeled in separate steps, which lead to problems such as complex architecture design, hard to scale, etc. Targeting the shortcomings of existing studies, we take both spatial and temporal dependencies from another perspective, and consider them as two heterogeneous types of edges in the graph. We propose a unified spatio-temporal GNN framework that captures both dependencies in a single step. More specifically, for each node in the graph, a unified neural network component is designed to simultaneously extract information from its sur-rounding neighbors (spatial) and its past records (temporal), which enables much easier dependency aggregation with faster execution. Experiment results demonstrate the superiority of the proposed framework over state-of-the-art (SOTA) baselines on various applications, including modeling smart cities and data-driven political science research.

As 3D point clouds become the representation of choice for multiple vision and graphics applications, such as autonomous driving, robotics, etc., the generation of them by deep neural networks has attracted increasing attention in the research community. Despite the recent success of deep learning models in classification and segmentation, synthesizing point clouds remains challenging, especially from a single image. State-of-the-art (SOTA) approaches can generate a point cloud from a hidden vector, however, they treat 2D and 3D features equally and disregard the rich shape information within the 3D data. In this paper, we address this problem by integrating image features with 3D prototype features. Specifically, we propose to learn a set of 3D prototype features from a real point cloud dataset and dynamically adjust them through the training. These prototypes are then integrated with incoming image features to guide the point cloud generation process. Experimental results show that our proposed method outperforms SOTA methods on single image based 3D reconstruction tasks.

Reconstructing point clouds from images would extremely benefit many practical CV applications, such as robotics, automated vehicles, and Augmented Reality. Fueled by the advances of deep neural network, many deep learning frameworks are proposed to address this problem recently. However, these frameworks generally rely on a large amount of labeled training data (e.g., image and point cloud pairs). Although we usually have numerous 2D images, corresponding 3D shapes are insufficient in practice. In addition, most available 3D data covers only a limited amount of classes, which further restricts the models' generalization ability to novel classes. To mitigate these issues, we propose a novel few-shot single-view point cloud generation framework by considering both class-specific and class-agnostic 3D shape priors. Specifically, we abstract each class by a prototype vector that embeds class-specific shape priors. Class-agnostic shape priors are modeled by a set of learnable shape primitives that encode universal 3D shape information shared across classes. Later, we combine the input image with class-specific prototypes and class-agnostic shape primitives to guide the point cloud generation process. Experiments on the popular ModelNet and ShapeNet datasets demonstrate that our method outperforms state-of-the-art methods in the few-shot setting.

Multi-label learning algorithms have attracted more and more attention as of recent. This is mainly because real-world data is generally associated with multiple and non-exclusive labels, which could correspond to different objects, scenes, actions, and attributes. In this paper, we consider the following challenging multi-label stream scenario: the new labels emerge continuously in the changing environments, and are assigned to the previous data. In this setting, data mining solutions must be able to learn the new concepts and avoid catastrophic forgetting simultaneously. We propose a novel continual and interactive feature distillation-based learning framework (CIFDM), to effectively classify instances with novel labels. We utilize the knowledge from the previous tasks to learn new knowledge to solve the current task. Then, the system compresses historical and novel knowledge and preserves it while waiting for new emerging tasks. CIFDM consists of three components: 1) a knowledge bank that stores the existing feature-level compressed knowledge, and predicts the observed labels so far; 2) a pioneer module that aims to learn and predict new emerged labels based on knowledge bank.; 3) an interactive knowledge compression function which is used to compress and transfer the new knowledge to the bank, and then apply the current compressed knowledge to initialize the label embedding of the pioneer for the next task.

The aim of image-to-image translation algorithms is to tackle the challenges of learning a proper mapping function across different domains. Generative Adversarial Networks (GANs) have shown superior ability to handle this problem in both supervised and unsupervised ways. However, one critical problem of GAN in practice is that the discriminator is typically much stronger than the generator, which could lead to failures such as mode collapse, diminished gradient, etc. To address these shortcomings, we propose a novel framework, which incorporates a powerful spatial attention mechanism to guide the generator. Specifically, our designed discriminator estimates the probability of realness of a given image, and provides an attention map regarding this prediction. The generated attention map contains the informative regions to distinguish the real and fake images, from the perspective of the discriminator. Such information is particularly valuable for the translation because the generator is encouraged to focus on those areas and produce more realistic images. We conduct extensive experiments and evaluations, and show that our proposed method is both qualitatively and quantitatively better than other state-of-the-art image translation frameworks.

Non-stationary data stream mining aims to classify large scale online instances that emerge continuously. The most apparent challenge compared with the offline learning manner is the issue of consecutive emergence of new categories, when tackling non-static categorical distribution. Non-stationary stream settings often appear in real-world applications, e.g., online classification in E-commerce systems that involves the incoming productions, or the summary of news topics on social networks (Twitter). Ideally, a learning model should be able to learn novel concepts from labeled data (in new tasks) and reduce the abrupt degradation of model performance on the old concept (also named catastrophic forgetting problem). In this work, we focus on improving the performance of the stream mining approach under the constrained resources, where both the memory resource of old data and labeled new instances are limited/scarce. We propose a simple yet efficient resource-constrained framework CLEAR to facilitate previous challenges during the one-pass stream mining. Specifically, CLEAR focuses on creating and calibrating the class representation (the prototype) in the embedding space. We first apply the contrastive-prototype learning on large amount of unlabeled data, and generate the discriminative prototype for each class in the embedding space. Next, for updating on new tasks/categories, we propose a drift estimation strategy to calibrate/compensate for the drift of each class representation, which could reduce the knowledge forgetting without storing any previous data. We perform experiments on public datasets (e.g., CUB200, CIFAR100) under stream setting, our approach is consistently and clearly better than many state-of-the-art methods, along with both the memory and annotation restriction.

Continuously mining complexity data stream has recently been attracting an increasing amount of attention, due to the rapid growth of real-world vision/signal applications such as self-driving cars and online social media messages. In this paper, we aim to address two significant problems in the lifelong/incremental stream mining scenario: first, how to make the learning algorithms generalize to the unseen classes only from a few labeled samples; second, is it possible to avoid storing instances from previously seen classes to solve the catastrophic forgetting problem? We introduce a novelty stream mining framework to classify the infinite stream of data with different categories that occurred during different times. We apply a few-sample learning strategy to make the model recognize the novel class with limited samples; at the same time, we implement an incremental generative model to maintain old knowledge when learning new coming categories, and also avoid the violation of data privacy and memory restrictions simultaneously. We evaluate our approach in the continual class-incremental setup on the classification tasks and ensure the sufficient model capacity to accommodate for learning the new incoming categories.

Existing solutions to multi-task learning typically rely on manually enumerating multiple network architectures to find the optimal structure, which incurs a heavy design workload. In addition, extending these models to new tasks is difficult in many cases, since it often requires a network redesign for achieving the best performance. To overcome these limitations, in this paper, we propose a novel principle for multitask learning, which focuses on the learning process itself. For each task, we consider its feature as a state, and treat the learning problem as a state transformation process, which is driven by a task-specific gradient. Specifically, we introduce a Gradient Modification Unit (GMU), which consists of a Gradient Estimation Network (GEN) (shared among all tasks) and multiple tiny task-specific gradient correctors (one for each task). At each iteration, for any task T, the corresponding gradient corrector modifies the gradient estimated by the GEN to produce the desired task-specific gradient tensor that is applied to update the state of task T. Our solution has several benefits including automatic end-to-end learning of the optimal transformation process, simplification of network design and high expansibility to new tasks. We demonstrate the superiority of our approach over existing solutions on a variety of datasets, across both image classification and image retrieval tasks.

For many real-world classification problems, e.g., sentiment classification, most existing machine learning methods are biased towards the majority class when the Imbalance Ratio (IR) is high.To address this problem, we propose a set convolution (SetConv) operation and an episodic training strategy to extract a single representative for each class, so that classifiers can later be trained on a balanced class distribution.We prove that our proposed algorithm is permutation-invariant despite the order of inputs, and experiments on multiple large-scale benchmark text datasets show the superiority of our proposed framework when compared to other SOTA methods.

Deep Neural Networks (DNNs) have primarily been demonstrated to be useful for closed-world classification problems where the number of categories is fixed. However, DNNs notoriously fail when tasked with label prediction in a non-stationary data stream scenario, which has the continuous emergence of the unknown or novel class (categories not in the training set). For example, new topics continually emerge in social media or e-commerce. To solve this challenge, a DNN should not only be able to detect the novel class effectively but also incrementally learn new concepts from limited samples over time. Literature that addresses both problems simultaneously is limited. In this paper, we focus on improving the generalization of the model on the novel classes, and making the model continually learn from only a few samples from the novel categories. Different from existing approaches that rely on abundant labeled instances to re-train/update the model, we propose a new approach based on Few Sample and Adversarial Representation Learning (FSAR). The key novelty is that we introduce the adversarial confusion term into both the representation learning and few-sample learning process, which reduces the over-confidence of the model on the seen classes, further enhance the generalization of the model to detect and learn new categories with only a few samples. We train the FSAR operated in two stages: first, FSAR learns an intra-class compacted and inter-class separated feature embedding to detect the novel classes; next, we collect a few labeled samples belong to the new categories, utilize episode-training to exploit the intrinsic features for few-sample learning. We evaluated FSAR on different datasets, using extensive experimental results from various simulated stream benchmarks to show that FSAR effectively outperforms current state-of-the-art approaches.

One of the key challenges of performing label predictions over a data stream is concerned with the emergence of instances belonging to unobserved (or novel) classes over time. Although existing studies have proposed various solutions to address this challenge, they mostly focus on streams with lowdimensional data and strongly rely on the intrinsic cohesion and separation data property, i.e., instances belonging to the same class are closer to each other (cohesion) than those belonging to different classes (separation) in the observed feature space, to detect instances from unknown classes. Unfortunately, such a property is typically not inherent in high-dimensional data such as images and texts. Thus, to perform classification and novel class detection on high-dimensional data streams, we need to address two main problems: 1) Finding a feature space that exhibit cohesion and separation properties, and 2) Training with limited amount of labeled data. In this paper, we propose a multi-task metric learning mechanism useful for identifying a latent space in which the cohesion and separation data property is valid and have designed a semi-supervised stream classifier called SIM based on this mechanism. We empirically measure the performance of SIM over multiple real-world image and text datasets, and demonstrate its superiority by comparing the performance with existing state-of-the-art frameworks.

Deep Neural Network (DNN) has been largely demonstrated to be effective for real-world classification problems. However, such model requires a huge amount of training samples to get more accurate result. When limited samples allowed for the training step, the model may perform weak generalization ability on the test set, especially when the novel/unseen class may occur during the test period (we call it open-set classification). This severely limits its further utility in many real-world large scale applications, such as the open-set image and text classification scenarios. In this paper, we focus on addressing this key challenge by developing a DNN based co-representation learning approach RLCN. It utilizes limited samples for training a model then applies it to classify normal instances and detect the emergence of novel class over time. The key novelty is that we design a weighted pairwise-constraint loss (WPC) function to learn an enhanced generalization and robust feature embedding, where the intra-class (same class) compactness and inter-class (different class) separation are achieved. Moreover, we apply the temperature scaling scheme on the softmax function to replace traditional softmax output in our open-world classifier to achieve the classification and novel class detection simultaneously. Our extensive empirical evaluation on benchmark datasets demonstrate the effectiveness of our framework compared to other competing techniques.