Tor is one of the most widely adopted anonymity networks, yet its anonymity can be undermined by adversaries through flow correlation attacks. Current mainstream technologies focus on exploiting the sequence characteristics of packet lengths and timestamps to execute attacks. However, the padding mechanism of the Tor network and time delays caused by multi-hop relays obscure these single-modal features. Additionally, the diversity of network services and the randomness of user behavior result in sparse packet distributions, which impact model training and inference. In this paper, we propose SSRCorr, a novel self-supervised learning framework for flow correlation attacks, incorporating the Flow Feature Aggregation (FFA) module and Global-Local Fusion (GLoF) Encoder to address these challenges. Firstly, we construct a Byte-based Traffic Aggregation Matrix (BTAM) by integrating time and length sequences and applying two data augmentation methods tailored for Tor flow correlation, thereby reducing the impact of Tor network noise on attack effectiveness. Secondly, we employ GLoF to extract features from the output by FFA and fuse the global context information of the traffic, thus mitigating the impact of low-information traffic on model performance. Experiments show that SSRCorr achieves a TPR of 96%, surpassing other methods, and maintains robust performance under temporal drift and obfuscation, supporting future research on countering anonymity system defenses.