Worldwide inflation in the count of road accidents has raised an alarming scenario wherein driver distraction is identified as one of the main causes. According to the National Highway Traffic Safety Administration survey, talking over the mobile or fellow passengers, eating or drinking while driving, and operating in-vehicle menus can lead to distract the driver. Therefore, this paper presents a new driver distraction detection method to cater such scenarios. The proposed method is a convolution-based capsule network with attention mechanism, termed as CAT-CapsNet, that combines the computation capabilities of capsule network with attention module and convolution filters. We evaluate the proposed CAT-CapsNet on two publicly available driver distraction datasets namely, American University in Cairo (AUC) distracted driver dataset and Statefarmŝ dataset. A comparative analysis of the proposed model is conducted against 16 (approx) state-of-the-art methods in terms of accuracy, number of parameters, and AUC curves. Experiments affirm that CAT-CapsNet outperforms with the accuracy of 99.88% on SFD dataset and 96.7% accuracy on the AUC dataset. Moreover, the proposed model has only$8.5M$parameters to train and no data augmentation is needed in-case training samples are less.