Transfer learning for collaborative filtering (TLCF) aims to solve the sparsity problem by transferring rating knowledge across multiple domains. Taking domain difference into ac- count, one of the issues in cross-domain collaborative filtering is to selectively transfer knowledge from source/auxiliary domains. In particular, this paper addresses the problem of inconstant users (users with changeable preferences across different domains) when transferring knowledge about users from another auxiliary domain. We first formulate the problem of inconstant users caused by domain difference and then propose a new model that performs constrained collective matrix factorization (CCMF). Our experiments on simulated and real data show that CCMF has superior performance than other methods.