This paper proposes a method of fabric structure defect detection based on tactile information. Different from traditional visual-based detection methods, the proposed method uses a tactile sensor to attain the information of fabric. The advantage of using tactile information is to avoid different irregular dyeing patterns and reduce the influence of ambient light. Therefore, the proposed method can be more concise and universal, which makes the defect detection system more robust. Experiments are conducted to verify the performance of the developed tactile method. The results demonstrate that the proposed method is much better than the visual method in the detection of structural defects. In addition, we propose a design of a tactile sensing device that can greatly improve the actual detection efficiency of the tactile method. It is showed that the proposed method is efficient enough to meet our requirements and it provides a possible solution for the application of the tactile method in actual fabric production.