Large Visual Language Models (LVLMs) have achieved remarkable success in vision tasks. However, the significant differences between industrial and natural scenes make applying LVLMs challenging. Existing LVLMs rely on user-provided prompts to segment objects. This often leads to suboptimal performance due to the inclusion of irrelevant pixels. In addition, the scarcity of data also makes the application of LVLMs in industrial scenarios remain unexplored. To fill this gap, this paper proposes an open industrial dataset and a Refined Text-Visual Prompt (RTVP) for zero-shot industrial defect detection. First, this paper constructs the Multi-Modal Industrial Open Dataset (MMIO) containing 80K+ samples. MMIO contains diverse industrial categories, including 6 super categories and 18 subcategories. MMIO is the first large-scale multi-scenes pre-training dataset for industrial zero-shot learning, and provides valuable training data for open models in future industrial scenarios. Based on MMIO, this paper provides a RTVP specifically for industrial zero-shot tasks. RTVP has two significant advantages: First, this paper designs an expert-guided large model domain adaptation mechanism and designs an industrial zero-shot method based on Mobile-SAM, which enhances the generalization ability of large models in industrial scenarios. Second, RTVP automatically generates visual prompts directly from images and considers text-visual prompt interactions ignored by previous LVLM, improving visual and textual content understanding. RTVP achieves SOTA with 42.2% and 24.7% AP in zero-shot and closed scenes of MMIO.

In certain specific industrial scenarios, smoking and cellphone usage are strictly prohibited behaviors. In these scenarios, it is crucial to rapidly and accurately detect smoking and cell phone usage, and promptly issue warnings, to ensure industrial safety. The detection of prohibited behaviors using computer vision has gained attention from researchers with the development of artificial intelligence. However, detecting small objects like cigarettes and cell phones in complex backgrounds and varying angles poses a challenge due to their changing shapes. In this paper, we propose a detection model called CB-YOLO, which utilizes YOLOv7 as the baseline model, for detecting smoking and mobile phone usage behavior. Firstly, we propose a new channel space pyramid network called CSPPF that pools features at different scales and introduces scale focusing on improving the perceptual ability of the network to better handle targets at various scales and locations. Secondly, we propose an enhanced feature pyramid called AWBFPN. This introduces additional learnable weight parameters to improve the model's ability to fuse multi-scale features effectively. Finally, we have also proposed a new loss function called Size-IoU. The experimental results demonstrate that our algorithm outperforms the baseline model. Specifically, it achieves a 3.9% improvement in Precision, a 2.1% improvement in Recall, a 6.4% improvement in [email protected], and a 2% improvement in [email protected]:0.95 on the Phone_check dataset. Similarly, on the Smoking dataset, our algorithm achieves a 3.3% improvement in Precision, a 1.9% improvement in Recall, a 1.7% improvement in [email protected], and a 0.3% improvement in [email protected]:0.95.