Several deep learning techniques have been used to detect pavement cracks for the partial replacement of inefficient traditional inspections. However, the extensively varying real-world situations limit the detection accuracy. While many existing studies utilized the attention modules in pavement crack detection to improve model performance, few studies considered the impact of “how” and “where” to add attention modules on model performance, namely the module optimization. Combined with the attention mechanism, a new pavement crack detection method was proposed based on the You Only Look Once 5th version (YOLOv5) in this paper. Considering two adding ways and three adding positions, the spatial and channel squeeze and excitation (SCSE) module and convolutional block attention module (CBAM) were used to build a total of 12 different attention models for the cracking detection. Each model was trained on 3248 images, and the weight with the best performance on the validation set was saved for testing. The test results show that the [email protected]:0.95 of the best attention model is improved by nearly 6.7% compared to the original model without the attention mechanism. In addition, it can process images at 13.15ms/pic while maintaining 94.4% precision, fully meeting the needs of real-time detection. Compared with the existing pavement crack detection methods, the advantages of the proposed method include a great detection speed, high accuracy, and good robustness.