Template matching of multi-modal image has been a challenge to image matching, and it is difficult to balance the speed and the accuracy, especially for images with large sizes. Based on this, we propose a stepwise image matching method to achieve a precise location from the coarse-to-fine image matching by utilizing cascaded networks. In the proposed method, a coarse-grained matching network is firstly constructed to locate a rough matching position based on cross-correlating features of optical and SAR images. Specially, to enhance the credible matching position, a suppression network is designed to evaluate for the obtained cross-correlation feature and added into the coarse-grained network as a feedback. Secondly, a fine-grained matching network is constructed based on the obtained rough matching result to gain a more precise matching. In this part, ternary groups are utilized to construct the training samples. Interestingly, we apply the region with a few pixels offset as the negative class, which effectively distinguishes similar neighbourhoods of the rough matching position. Moreover, a modified Siamese network is used to extract features of SAR and optical images, respectively. Finally, experimental results illustrate that the proposed method obtains more precise matching compared with the state-of-the-art methods.