With the increasing number of motor vehicles per capita, the problem of road traffic congestion is becoming increasingly prominent. Intelligent transportation systems are playing an increasingly important role in traffic management. Establishing a reasonable and effective traffic congestion prediction model is crucial for the implementation of intelligent transportation systems. This article proposes an end-to-end congestion prediction framework for unsupervised learning. First, a traffic data deep embedding clustering model based on contrast learning (CL-DEC) was constructed to discretize and label the congestion status of spatiotemporal rasterized GPS data. Then, the Time Convolutional Network (TCN) is used to extract features from the labeled data, and the Dependency Matrix Attention Mechanism (DMAttention) is combined to highlight important features for predicting traffic congestion. In this study, we found that DEC can effectively alleviate the false positive problem of CL, and the combination of the two improves discriminability and comparability, which helps to improve the accuracy of data annotation. In addition, TCN-DMAttention can balance the short-term fluctuations and long-term trends of traffic time series data. We compared it with 13 baseline models and conducted robustness analysis using 20 random number seeds. The experimental results indicate that the TCN-DMAttention model has better predictive performance and good stability than other comparative models do.