Open set domain adaptation (OSDA) is an important problem in machine learning and computer vision. In OSDA, one is given a labeled dataset from a source domain (source joint distribution) and an unlabeled dataset from a target domain (target joint distribution), where the target domain contains not only the known classes presented in the source domain but also the unknown class. The goal of OSDA is to train a neural network with minimal target classification risk. From the statistical learning perspective, there are two fundamental challenges in this problem: (1) the source-target joint distribution difference regarding the known classes and (2) the target classification risk estimation regarding the unknown class. Although prior works have proposed various sophisticated solutions to the problem and achieved inspiring experimental results, they do not fully resolve these two challenges. In this article, we introduce a principled approach named known joint distribution matching and unknown classification risk reformulation (KMUR). KMUR tackles the first challenge by matching the source joint distribution to the target known joint distribution such that the distribution difference can be reduced and addresses the second challenge by reformulating the target unknown classification risk such that the reformulated risk can be estimated on the unlabeled target and source data. To be specific, we exploit cross entropy as the classification loss and triangular discrimination (TD) distance as the joint distribution matching loss. Since the TD distance needs to be estimated from data, we develop an innovative technique named least squares TD estimation (LSTDE), which casts the estimation into least squares classification. To achieve the OSDA goal, we train the network to minimize the estimations of target classification risk and TD distance. Experiments on benchmark and real-world datasets confirm the effectiveness of our approach. The introductory video and PyTorch code are available on GitHub (https://github.com/sentaochen/Known-Joint-Distribution-Matching-and-Unknown-Classification-Risk-Reformulation). One can also visit https://github.com/sentaochen for more source code on domain adaptation (DA), multi-source DA, partial DA, and domain generalization approaches.

The objective of Multi-Source Domain Adaptation (MSDA) is to train a neural network on labeled data from multiple joint source distributions (source domains) and unlabeled data from a joint target distribution (target domain), and use the trained network to estimate the target data labels. The challenge in this MSDA problem is that the multiple joint source distributions are relevant but distinct from the joint target distribution. To address this challenge, we propose a Joint Distribution Weighted Alignment (JDWA) approach to align a weighted joint source distribution to the joint target distribution under the relative entropy. Specifically, the weighted joint source distribution is defined as the weighted sum of the multiple joint source distributions, and is parameterized by the relevance weights. Since the relative entropy is unknown in practice, we propose a Kernel Relative Entropy Estimation (KREE) method to estimate it from data. Our KREE method first reformulates relative entropy as the negative of the minimal value of a functional, then exploits a function from the Reproducing Kernel Hilbert Space (RKHS) as the functional's input, and finally solves the resultant convex problem with a global optimal solution. We also incorporate entropy regularization to enhance the network's performance. Together, we minimize cross entropy, relative entropy, and entropy to learn both the relevance weights and the neural network. Experimental results on benchmark image classification datasets demonstrate that our JDWA approach performs better than the comparison methods. Pytorch code of our approach will be released upon the paper's publication.

We tackle the problem of generalizing a predictor trained on a set of source domains to an unseen target domain, where the source and target domains are different but related to one another, i.e., the domain generalization problem. Prior adversarial methods rely on solving the minimax problems to align in the neural network embedding space the components of the domains (i.e., a set of marginal distributions, a set of marginal distributions and multiple sets of class-conditional distributions). However, these methods introduce additional parameters (for each set of distributions) to the network predictor and are difficult to train. In this work, we propose to directly align the domains themselves via solving a minimax problem that can be decomposed and converted into a min one. Particularly, we analytically solve the max problem with respect to (w.r.t.) the domain discriminators, and convert the minimax problem into a min one w.r.t. the embedding function. This is more advantageous since in the end our approach introduces no additional network parameters and simplifies the training procedure. We evaluate our approach on several multi-domain datasets and testify its superiority over the relevant methods. The source code is available at https://github.com/sentaochen/Decomposed-Adversarial-Domain-Generalization.

Abstract In this work, we study the problem of generalizing a prediction (classification or regression) model trained on a set of source domains to an unseen target domain, where the source and target domains are different but related,i.e, the domain generalization problem. The challenge in this problem lies in the domain difference, which could degrade the generalization ability of the prediction model. To tackle this challenge, we propose to learn a neural network representation function to align a joint distribution and a product distribution in the representation space, and show that such joint-product distribution alignment conveniently leads to the alignment of multiple domains. In particular, we align the joint distribution and the product distribution under the $$L^{2}$$ L2 -distance, and show that this distance can be analytically estimated by exploiting its variational characterization and a linear variational function. This allows us to comfortably align the two distributions by minimizing the estimated distance with respect to the network representation function. Our experiments on synthetic and real-world datasets for classification and regression demonstrate the effectiveness of the proposed solution. For example, it achieves the best average classification accuracy of 82.26% on the text dataset Amazon Reviews, and the best average regression error of 0.114 on the WiFi dataset UJIIndoorLoc.

Domain adaptation is concerned with the problem of generalizing a classification model to a target domain with little or no labeled data, by leveraging the abundant labeled data from a related source domain. The source and target domains possess different joint probability distributions, making it challenging for model generalization. In this article, we introduce domain neural adaptation (DNA): an approach that exploits nonlinear deep neural network to 1) match the source and target joint distributions in the network activation space and 2) learn the classifier in an end-to-end manner. Specifically, we employ the relative chi-square divergence to compare the two joint distributions, and show that the divergence can be estimated via seeking the maximal value of a quadratic functional over the reproducing kernel hilbert space. The analytic solution to this maximization problem enables us to explicitly express the divergence estimate as a function of the neural network mapping. We optimize the network parameters to minimize the estimated joint distribution divergence and the classification loss, yielding a classification model that generalizes well to the target domain. Empirical results on several visual datasets demonstrate that our solution is statistically better than its competitors.

An intrinsic problem in domain adaptation is the joint distribution mismatch between the source and target domains. Therefore, it is crucial to match the two joint distributions such that the source domain knowledge can be properly transferred to the target domain. Unfortunately, in semi-supervised domain adaptation (SSDA) this problem still remains unsolved. In this article, we therefore present an asymmetric joint distribution matching (AJDM) approach, which seeks a couple of asymmetric matrices to linearly match the source and target joint distributions under the relative chi-square divergence. Specifically, we introduce a least square method to estimate the divergence, which is free from estimating the two joint distributions. Furthermore, we show that our AJDM approach can be generalized to a kernel version, enabling it to handle nonlinearity in the data. From the perspective of Riemannian geometry, learning the linear and nonlinear mappings are both formulated as optimization problems defined on the product of Riemannian manifolds. Numerical experiments on synthetic and real-world data sets demonstrate the effectiveness of the proposed approach and testify its superiority over existing SSDA techniques.

Domain adaptation addresses the learning problem where the training data are sampled from a source joint distribution (source domain), while the test data are sampled from a different target joint distribution (target domain). Because of this joint distribution mismatch, a discriminative classifier naively trained on the source domain often generalizes poorly to the target domain. In this paper, we therefore present a Joint Distribution Invariant Projections (JDIP) approach to solve this problem. The proposed approach exploits linear projections to directly match the source and target joint distributions under the L2-distance. Since the traditional kernel density estimators for distribution estimation tend to be less reliable as the dimensionality increases, we propose a least square method to estimate the L2-distance without the need to estimate the two joint distributions, leading to a quadratic problem with analytic solution. Furthermore, we introduce a kernel version of JDIP to account for inherent nonlinearity in the data. We show that the proposed learning problems can be naturally cast as optimization problems defined on the product of Riemannian manifolds. To be comprehensive, we also establish an error bound, theoretically explaining how our method works and contributes to reducing the target domain generalization error. Extensive empirical evidence demonstrates the benefits of our approach over state-of-the-art domain adaptation methods on several visual data sets.

Domain adaptation tries to adapt a model trained from a source domain to a different but related target domain. Currently, prevailing methods for domain adaptation rely on either instance reweighting or feature transformation. Unfortunately, instance reweighting has difficulty in estimating the sample weights as the dimension increases, whereas feature transformation sometimes fails to make the transformed source and target distributions similar when the cross-domain discrepancy is large. In order to overcome the shortcomings of both methodologies, in this article, we model the unsupervised domain adaptation problem under the generalized covariate shift assumption and adapt the source distribution to the target distribution in a subspace by applying a distribution adaptation function. Accordingly, we propose two frameworks: Bregman-divergence-embedded structural risk minimization (BSRM) and joint structural risk minimization (JSRM). In the proposed frameworks, the subspace distribution adaptation function and the target prediction model are jointly learned. Under certain instantiations, convex optimization problems are derived from both frameworks. Experimental results on the synthetic and real-world text and image data sets show that the proposed methods outperform the state-of-the-art domain adaptation techniques with statistical significance.