Cross-domain few-shot classification problem is one of the most challenging problems in few-shot learning, as it requires transferring useful information from the source dataset, which has large distribution differences, to downstream few-shot tasks. Most previous methods focused on improving model representation from the perspective of model structure while ignoring the impact of data distribution. Our exploratory experiment indicates that different categories of data in the source dataset have different impacts on downstream tasks. This signifies that we should adaptively adjust the contribution of different categories of training data based on downstream tasks. Inspired by this, we design a novel training framework, which allows us to disentangle the information of different sample clusters from a well-trained network and adaptively aggregate the information to assist the downstream tasks. Our comprehensive experiments on four commonly used datasets verify that our method can improve classification accuracy while converging with fewer training epochs during fine-tuning.