Continual learning (CL) aims to learn a sequence of tasks without forgetting. Numerous efforts have been made to tackle CL including data-centric, model-centric, and algorithm-centric methods. The more information the algorithm can obtain from previous tasks, e.g., the training data, the easier the CL task will be. However, few studies focus on the most difficult setting, i.e., dark CL (DCL) where only the model of the last task can be obtained. DCL is a typical setting in real-world applications, e.g., Cl tasks based on the models trained on private or privileged data. For solving DCL, we propose a novel recursive generalization bound, which can also be applied to arbitrary Traditional CL (TCL). To minimize the bound proposed, we propose a novel method, i.e., conditional past experience generation (CPEG), which reconstructs the previous conditional training data in the DCL setting. In the experiment, we apply CPEG to a wide range of benchmarks. The experimental results show that CPEG significantly reduces forgetting. On the other hand, CPEG can be used as a regularization term for any CL baseline. We also conduct experiments on the TCL setting. The performance of almost all baselines is improved, especially for the most difficult class-incremental tasks.

The accuracy of deep neural networks is easily degraded by image corruption. Therefore, there is a need to develop adaptation techniques to ensure durable models and predictions against changes in data distribution. We focus on the task to fit a trained model with a different distribution from training data under the condition that the training data are not available for test time. In this paper, we propose a novel adaptation method in test time for online learning named multi-step layer adaptation (MuSLA). The proposed method achieves high adaptive accuracy by sequentially applying loss functions to specific layers only, especially considering the roles and inter-actions of the layers and employing domain adaptation and semi-supervised learning techniques. The proposed method can be widely applied to already existing trained models with-out additional networks. We show that our approach outperforms conventional methods in image corruption benchmark data experiments.

Unsupervised domain adaptation (UDA) has attracted much attention among those seeking to transfer a model from a labeled source domain to an unlabeled target domain. Many effective algorithms for single target domain adaptation (STDA) have been designed, however, STDA cannot satisfy the scenarios of transferring simultaneously to multiple target domains or transferring to a blending target domain. This paper proposes a novel discriminative mutual learning method for multi-target domain adaptation covering both blending target domain adaptation (BTDA) and multiple target domain adaptation (MTDA). Two key points are considered in the proposed method: one is to learn discriminative features for better prediction, and the other is to self-train the model with pseudo-labeled target data based on distance information. These two aspects are integrated through a mutual learning strategy via two different classifiers. According to extensive experiments on three domain adaptation benchmarks, the proposed method demonstrates the state-of-the-art performance in both BTDA and MTDA settings.

Existing unsupervised domain adaptation (UDA) studies focus on transferring knowledge in an offline manner. However, many tasks involve online requirements, especially in real-time systems. In this paper, we discuss Online UDA (OUDA) which assumes that the target samples are arriving sequentially as a small batch. OUDA tasks are challenging for prior UDA methods since online training suffers from catastrophic forgetting which leads to poor generalization. Intuitively, a good memory is a crucial factor in the success of OUDA. We formalize this intuition theoretically with a generalization bound where the OUDA target error can be bounded by the source error, the domain discrepancy distance, and a novel metric on forgetting in continuous online learning. Our theory illustrates the tradeoffs inherent in learning and remembering representations for OUDA. To minimize the proposed forgetting metric, we propose a novel source feature distillation (SFD) method which utilizes the source-only model as a teacher to guide the online training. In the experiment, we modify three UDA algorithms, i.e., DANN, CDAN, and MCC, and evaluate their performance on OUDA tasks with real-world datasets. By applying SFD, the performance of all baselines is significantly improved.

In this work, we focus on supervised domain adaptation for object detection in few-shot loose annotation setting, where the source images are sufficient and fully labeled but the target images are few-shot and loosely annotated. As annotated objects exist in the target domain, instance level alignment can be utilized to improve the performance. Traditional methods conduct the instance level alignment by semantically aligning the distributions of paired object features with domain adversarial training. Although it is demonstrated that point-wise surrogates of distribution alignment provide a more effective solution in few-shot classification tasks across domains, this point-wise alignment approach has not yet been extended to object detection. In this work, we propose a method that extends the point-wise alignment from classification to object detection. Moreover, in the few-shot loose annotation setting, the background ROIs of target domain suffer from severe label noise problem, which may make the point-wise alignment fail. To this end, we exploit moving average centroids to mitigate the label noise problem of background ROIs. Meanwhile, we exploit point-wise alignment over instances and centroids to tackle the problem of scarcity of labeled target instances. Hence this method is not only robust against label noises of background ROIs but also robust against the scarcity of labeled target objects. Experimental results show that the proposed instance level alignment method brings significant improvement compared with the baseline and is superior to state-of-the-art methods.

Unsupervised domain adaptation (UDA) methods aim to transfer knowledge from a labeled source domain to an unlabeled target domain. Most existing UDA methods try to learn domain-invariant features so that the classifier trained by the source labels can automatically be adapted to the target domain. However, recent works have shown the limitations of these methods when label distributions differ between the source and target domains. Especially, in partial domain adaptation (PDA) where the source domain holds plenty of individual labels (private labels) not appeared in the target domain, the domain-invariant features can cause catastrophic performance degradation. In this paper, based on the originally favorable underlying structures of the two domains, we learn two kinds of target features, i.e., the source-approximate features and target-approximate features instead of the domain-invariant features. The source-approximate features utilize the consistency of the two domains to estimate the distribution of the source private labels. The target-approximate features enhance the feature discrimination in the target domain while detecting the hard (outlier) target samples. A novel Coupled Approximation Neural Network (CANN) has been proposed to co-train the source-approximate and target-approximate features by two parallel sub-networks without sharing the parameters. We apply CANN to three prevalent transfer learning benchmark datasets, Office-Home, Office-31, and Visda2017 with both UDA and PDA settings. The results show that CANN outperforms all baselines by a large margin in PDA and also performs best in UDA.

Unsupervised domain adaptation (UDA) methods are generally proposed based on the assumption that the source domain and the target domain share an identical group of classes. However, in transfer learning tasks in reality, the target domain often has fewer data with missing classes. Partial domain adaptation (PDA) allows the source domain to have un-shared categories. Anchor points are used to describe the easily identified target samples. It is observed that the shared classes tend to have more anchor points compared with the unshared classes and we introduce a novel progressive optimization method named Ebb PDA tasks. Ebb could resist the negative transfer caused by the category gap and can be applied to any domain adaptation model. Ebb picks the anchor points by analyzing the features of the base model, and it uses the class-wise distribution of anchor points to estimate the category gap. Then Ebb minimizes the errors of shared classes and corrects the error samples caused by blind alignment. To verify the effectiveness of the method, we apply Ebb to three PDA image classification tasks based on three widely used data sets, i.e, Office-Home, Office-31 and ImageCLEF-DA while using three state-of-the-art methods as the base models. The results show that Ebb brings a significant improvement in all tasks and the models optimized by Ebb have stable performance under a wide range of category gaps.

Unsupervised domain adaptation (UDA) is a challenging task characterized by unlabeled target data with domain discrepancy to labeled source data. Many methods have been proposed to learn domain invariant features by marginal distribution alignment, but they ignore the intrinsic structure within target domain, which may lead to insufficient or false alignment. Class-level alignment has been demonstrated to align the features of the same class between source and target domains. These methods rely extensively on the accuracy of predicted pseudo-labels for target data. Here, we develop a novel self-training method that focuses more on accurate pseudo-labels via a dual-consistency strategy involving modelling the intrinsic structure of the target domain. The proposed dual-consistency strategy first improves the accuracy of pseudo-labels through voting consistency, and then reduces the negative effects of incorrect predictions through structure consistency with the relationship of intrinsic structures across domains. Our method has achieved comparable performance to the state-of-the-arts on three standard UDA benchmarks.

Domain shift is regarded as a key factor affecting the robust-ness of many models. Recently, unsupervised auxiliary learning (e.g., input reconstruction) has been proposed to improve the model’s domain transferability and alleviate cross-domain performance degradation; however, in the paradigm of existing approaches, the features extracted from various tasks are shared, which mixes the domain-invariant features from the main task and domain-specific feature from the auxiliary task, leading to an imperfect learning. To solve this problem, we propose a novel unsupervised domain adaptation method - the Disentangled Reconstruction Neural Network (DRNN) - for cross-domain retina vessel segmentation. DRNN leverages two tandem nets and disentangles the domain-invariant features and the domain-specific features in the multi-task learning process. We perform extensive experiments on public retina datasets and our proposed DRNN outperforms the competitors by a significant margin to achieve state-of-the-art results pertaining to retina vessel segmentation.

It is important to estimate the performance gap of a given deep learning model on the target data set, since discrepancy or bias between source and target domains is a common and fundamental problem in the practice of machine learning techniques. Without any assumptions on data bias, such as label shift or covariate shift and without target data labels, we propose a robustness estimation method based on prediction consistency evaluation between source and target data in the neighborhood of the source samples. Considering outliers and whether the user provided model is fully trained, a variety of variant methods are also tried, including setting neighborhood threshold to average intra-class distance for each category and relative robustness. Furthermore, the time complexity of this method is O(nlogn), which is applicable for large datasets. Experiments on the handwritten digit recognition and Japanese handwriting recognition show that the proposed methods are effective.