Open set domain adaptation (OSDA) aims to transfer classification-oriented knowledge from a labeled source domain to an unlabeled target domain, which faces the challenges from unseen knowledge in open-set scenarios, i.e., unknown classes privileged to the target domain. Existing methods usually identify unknown classes from classifier prediction directly, which are sensitive to the intrinsic clustering structure and cluster numbers of the unknown class data. In this paper, inspired by the sample relation characterization ability of Optimal Transport (OT), we propose a new type of OT method for OSDA, namely, Target-relaxed Optimal Transport (TROT). Compared with existing OT with strict marginal constraints, TROT imposes a single-side relaxation to the mass requirement on the open-set target domain. Theoretically, we prove that such a relaxation can reduce mis-matches between known and unknown classes, which indicates the transport plan of TROT is promising to identify unknown classes. Methodologically, TROT can identify unknown classes adaptively and map the cross-domain shared data with a sparse plan assignment, which improves both the effectiveness and robustness of known class alignment; besides, a graph embedding with multi-cluster structure of unknown classes is designed to learn a discriminative metric space for open-set classification. Empirically, extensive evaluations are conducted on several image datasets, where TROT achieves significant performance improvements compared with existing techniques for visual recognition in open-set scenarios.