Traffic prediction is one of the fundamental spatio-temporal prediction tasks in urban computing, which is of great significance to a wide range of applications, e.g., traffic controlling, vehicle scheduling, etc. Recently, with the expansion of the city and the development of public transportation, long-range and long-term spatio-temporal correlations play a more important role in traffic prediction. However, it is challenging to model long-range spatial dependencies and long-term temporal dependencies simultaneously in two aspects: 1) complex influential factors, including spatial, temporal and external factors. 2) multiple spatio-temporal correlations, including long-range and short-range spatial correlations, as well as long-term and short-term temporal correlations. To solve these issues, we propose a spatio-temporal memory augmented multi-level attention network for fine-grained traffic prediction, entitled ST-MAN. Specifically, we design a spatio-temporal memory network to encode and memorize fine-grained spatial information and representative temporal patterns. Then, we propose a multi-level attention network to explicitly model both short-term local spatio-temporal dependencies and long-term global spatio-temporal dependencies at different spatial scales (i.e., grid and region levels) and temporal scales (i.e., daily and weekly levels). In addition, we design an external component that takes external factors and spatial embeddings as inputs to generate location-aware influence of the external factors much more efficiently. Finally, we design an end-to-end framework optimized with the contrastive objective and supervised objective to boost model performance. Empirical experiments over coarse-grained and fine-grained real-world datasets demonstrate the superiority of the ST-MAN model compared to several state-of-the-art baselines.