Due to the potential values in many areas such as e-commerce and inventory management, fabric image retrieval, which is a special case in Content Based Image Retrieval (CBIR), has recently become a research hotspot. It is also a challenging issue with serval obstacles: variety and complexity of fabric appearance, high requirements for retrieval accuracy. To address this issue, this paper proposes a novel approach for fabric image retrieval based on multi-task learning and deep hashing. According to the cognitive system of fabric, a multi-classification-task learning model with uncertainty loss and constraint is presented to learn fabric image representation. Then we adopt an unsupervised deep network to encode the extracted features into 128-bits hashing codes. Further, the hashing codes are regarded as the index of fabrics image for image retrieval. To evaluate the proposed approach, we expanded and upgraded the dataset WFID, which was built in our previous research specifically for fabric image retrieval. The experimental results show that the proposed approach outperforms the state-of-the-art.