Visual Place Recognition (VPR) is the task of estimating the location of a query image by searching for the most similar reference image. The ever-changing appearance and viewpoint pose a significant challenge to the VPR task. The mainstream methods estimate the distance between images by comparing their compact representations. However, the representations are usually generated by aggregating all deep local features, which lose detailed semantics and spatial relationships. As a result, it is hard to distinguish the location of images that are highly similar in global semantics, but differ in local semantics or viewpoints. To this end, we proposed a novel Multi-Scale Spatial Aggregation (MS2A) framework, which takes detailed and spatial information into account to generate representations. MS2A fuses multi-scale semantics, which calculates the feature map of an image only once and requires no fine-tuning, regardless of the number of scales and regions. In addition, we introduced an approach for quantifying the choice of multi-scale parameters and an efficient method to further reduce the dimension of MS2A representation. Experimental results on three large-scale benchmark datasets demonstrate that MS2A can achieve state-of-the-art performance.