Skin diseases are one of the most common diseases worldwide, and the mismatch between skin disease patients and dermatologists leads to a huge waste of healthcare resources. Accurately matching skin disease patients to appropriate dermatologists by an image-based method of skin disease recognition can reduce the waste of healthcare resources. However, existing image-based methods of skin disease recognition do not fully utilize multi-source and multi-scale features, which leaves us the chance to improve skin disease recognition further. In this paper, we propose a fusion method of multi-source image features and multi-scale image features to improve skin disease recognition. First, we design a fusion module of multi-source image features to integrate multi-source image information. By dual Convolutional Block Attention Module (CBAM) blocks, the fusion module of multi-source image features enhances the feature representation of key regions and then obtains a comprehensive representation of skin diseases. Second, we propose a fusion module of multi-scale image features. By two parallel backbone networks, the fusion module of multi-scale image features can extract deep feature representations from different scales and exploit their complementarity. To validate the effectiveness of our method, we conduct extensive experiments. The experiment results demonstrate that our method outperforms the state-of-the-art method, achieving improvements of 6.30%, 12.52%, 10.85%, 12.16%, and 5.06% in accuracy, precision, recall, F1-score, and AUC, respectively.