High-precision localization of the Mars rovers is fundamental for path planning and safe navigation toward exploration targets during Mars missions. In cross-site visual localization, image matching is the key step to obtain corresponding points connecting images from different sites. The cross-site visual localization method based on Affine SIFT (ASIFT) is used in Tianwen-1 mission but is constrained in regions of Mars with poor texture and large viewpoint invariance. In this article, we propose a cross-site visual localization methodology of Mars rover based on self-supervised keypoint extraction and robust matching. The self-supervised keypoint extraction network, which is called MRSS-Net, uses multiscale deformable structures (MSDSs) during the feature encoding stage to enhance the network’s ability of extracting invariant features in regions with large viewpoint variations and improve the rate of identical points for cross-site images with poor texture. In addition, we develop self-attention descriptor enhancement mechanism (SADEM) to distinguish local features in repetitive patterns. The robust matching, which is called adaptive 2-D–3-D matching, uses GNC dead-reckoning (3-D priori information) to construct the initial coarse matching domain and homography matrix (2-D information) to construct a progressively shrinking refined matching domain. We compared our method against ASIFT based cross-site visual localization model and advanced deep learning algorithms and evaluate the performance using NaTeCam images collected during the traversal of four long-distance traversals (a total of 44 Martian sol sites) by Zhurong rover. The experimental results show that our framework reduces the localization error by 12.5% and improves localization robustness by 50.8%, compared with ASIFT-based cross-site visual localization method used in Zhurong rover. In addition, our method outperforms state-of-the-art deep learning techniques and ensures the current accuracy of cross-site visual localization for Mars rover, while significantly increasing the level of automation.