As an important part of intelligent manufacturing, pixel-level surface defect detection (SDD) aims to locate defect areas through mask prediction. Previous methods adopt the image-independent static convolution to indiscriminately classify per-pixel features for mask prediction, which leads to suboptimal results for some challenging scenes such as weak defects and cluttered backgrounds. In this paper, inspired by query-based methods, we propose a Wavelet and Prototype Augmented Query-based Transformer (WP-Former) for surface defect detection. Specifically, a set of dynamic queries for mask prediction is updated through the dual-domain transformer decoder. Firstly, a Wavelet-enhanced Cross-Attention (WCA) is proposed, which aggregates meaningful high-and low-frequency information of image features in the wavelet domain to refine queries. WCA enhances the representation of high-frequency components by capturing multi-scale relationships between different frequency components, enabling queries to focus more on defect details. Secondly, a Prototype-guided Cross-Attention (PCA) is proposed to refine queries through meta-prototypes in the spatial domain. The prototypes aggregate semantically meaningful tokens from image features, facilitating queries to aggregate crucial defect information under the cluttered backgrounds. Extensive experiments on three defect detection datasets (i.e., ESDIs-SOD, CrackSeg9k, and ZJU-Leaper) demonstrate that the proposed method achieves state-of-the-art performance in defect detection. The code will be available at https://github.com/yfhdm/WPFormer.

Deep learning based defect inspection methods have achieved promising performance, which usually relies on a large number of well-labeled training samples. However, it requires much effort to obtain enough annotated samples especially pixel-level annotations in practical production. Generative adversarial networks (GANs) can be utilized to generate defect samples. However, training GANs typically requires a large amount of defect data and most of them cannot generate defect samples with pixel-level annotations. In this paper, we present a Mask-Guided Defect image generation method, called MGDefect, which can generate high-quality defect samples with pixel-level annotations and effectively improves the performance of downstream tasks. Specifically, MGDefect consists of a Mask-Guided Defect Generation GAN (MGDG-GAN) and a Defect Mask GAN (DM-GAN). MGDG-GAN generates images containing defects with specific locations, shapes, and sizes via mask guidance and the dual discrimination for defects at the region level and image level. DM-GAN aims to generate diverse and rational masks for MGDG-GAN. It also adopts region-level and image-level dual discrimination for masks to generate compatible masks with the target objects. MGDG-GAN mainly focuses on generating local defect regions and DM-GAN specializes in generating masks, which are both trained on limited defect samples and abundant normal samples. Experiments conducted on the MVTec AD, DAGM 2007, and KolektorSDD2 benchmark datasets demonstrate that our method achieves promising results compared with other state-of-the-art approaches. Meanwhile, the generated defect samples significantly improve the performance of defect inspection tasks including classification and segmentation. Specifically, our method achieves KID$\times 10^{3}$/IS scores of 48.35/2.27 on MVTec AD, 15.37/2.44 on DAGM 2007, and 19.70/2.01 on KolektorSDD2. Furthermore, our method improves mIoU by 10.59%, 2.20%, and 2.17% on these datasets, respectively, using U-Net as the segmentation model.

Segment anything model (SAM) has recently demonstrated powerful segmentation ability for natural scene images (NSIs). However, the SAM exhibits limited performance in defect detection owing to the weak appearance of defects and cluttered backgrounds in industrial images. In this article, we propose a hierarchically adapting SAM for pixel-wise surface defect detection, named DefectSAM, which effectively modulates and decodes multilevel features of the encoder to capture defect information. Specifically, we introduce a learnable feature adaptation component between the image encoder and the decoder to modulate each level of features via the dual-feature adaptation unit. The dual-feature adaptation unit mainly includes the correlation-gated feature adaptation (CGFA) module and the mask-guided feature adaptation (MGFA) module. The CGFA exploits cross correlation spatial gating maps to adaptively incorporate a convolutional feature pyramid and Transformer features during feature adaptation, which is beneficial for capturing defect details. Moreover, the MGFA utilizes the mask prediction of high-level features as semantic guidance to select top-confidence foreground and background tokens for feature adaptation, focusing more on defect details and suppressing background noise. Extensive experiments on three defect detection datasets (i.e., MVTec AD, CrackSeg9k, ZJU-Leaper, and Magnetic tile) demonstrate that the proposed method achieves state-of-the-art performance with few learnable parameters, which greatly improves the generalization of SAM in defect detection.