Summary The detection and classification of traffic signs is a major challenge for self‐driving vehicles. The task can be narrowed down to detecting and classifying Indian Cautionary Traffic Signs (ICTS). In this proposed work, the difficulty of detecting and identifying Indian Cautionary Traffic Signs is addressed, and sincere attempts have been made to attain a possible solution using a Generative Adversarial Network (GAN), Improved Mask R‐CNN, and GrabCut algorithms. Data augmentation is done using Cascade Pyramid Generative Adversarial Network (CP‐GAN) to upsize the data set. The Mask R‐CNN with certain adoptions termed Improved mask RCNN is used in conjunction with the Grab‐Cut method to handle ICTS detection and identification through automatic end‐to‐end learning. Initially, Improved Mask R‐CNN generates a pixel‐by‐pixel segmentation mask for each item in the input sample. Masks developed using Improved Mask R‐CNN are not always clean, that is, some background pixels are often seen in the foreground segmentation. Hence, the generated masks are refined using the Grab Cut algorithm to enhance image segmentation. This combined approach works well in isolating the traffic signs from the ground truth images. Improved Mask R‐CNN attained better performance in the overall performance of traffic sign detection in the Indian data‐set. This method recognizes 40 different types of cautionary traffic signs from the unique Indian data set. The results are provided for complicated traffic sign categories that have not been addressed before. The proposed technique is trained and evaluated on ICTS, GTSDB, STSD, LISA, and DITS data sets, and it is also compared against cutting‐edge object recognition methods like Mask RCNN, Faster RCNN, SSD, and YOLOv3.

Abstract Cautionary traffic signs are of immense significance to traffic safety. In this study, a robust and optimal real‐time approach to recognize the Indian cautionary traffic signs (ICTS) is proposed. ICTS are all triangles with a white backdrop, a red border, and a black pattern. A dataset of 34,000 real‐time images has been acquired under various environmental conditions and categorized into 40 distinct classes. Pre‐processing techniques are used to transform RGB images to gray‐scale images and enhance contrast in images for superior performance. To find the ICTS, an optimized adaptive boosting cascade classifier is used. To classify the specific category of signs found by the optimized adaptive boosting cascade classifier, an 11‐layer CNN model is built. Finally, using computer vision methods, this model is tested in real‐time. Evaluation metrics such as precision, recall, F1 measure, error rate, and mAP are expressed on the ICTS, GTSDB, LISA, STSD, and DITS‐based datasets to evaluate the proposed method and compare the results of predictions with other investigations. When compared to other state‐of‐the‐art objects detection models such as SSD, YOLOv3, and Faster RCNN, the proposed model outperformed them all, with a precision of 97.15%, recall rate of 96.74%, an error rate of 3.26%, f1‐score of 96.94%, and [email protected] of 95.6%. (Dataset available: 10.21227/yy4h‐rc98 )