When robots move autonomously for long-term, varied appearance such as the transition from day to night and seasonal variation brings challenges to visual place recognition. Defining an appearance condition (e.g. a season, a kind of weather) as a domain, we consider that the desired representation for place recognition (i) should be domain-unrelated so that images from different time can be matched regardless of varied appearance, (ii) should be learned in a self-supervised manner without the need of massive manually labeled data, and (iii) should be able to train among multiple domains in one model to keep limited model complexity. This paper sets to find domain-unrelated features across extremely changing appearance, which can be used as image descriptors to match between images collected at different conditions. We propose to use the adversarial network to disentangle domain-unrelated and domain-related features, which are named place and appearance features respectively. During training, only domain information is needed without requiring manually aligned image sequences. Experiments demonstrated that our method can disentangle place and appearance features in both toy case and images from the real world, and the place feature is qualified in place recognition tasks under different appearance conditions. The proposed network is also adaptable to multiple domains without increasing model capacity and shows favorable generalization.