Performing chemical mechanical polishing (CMP) modeling for physical verification on an integrated circuit (IC) chip is vital to minimize its manufacturing yield loss. Traditional CMP models calculate post-CMP topography height of the IC’s layout based on physical principles and empirical experiments, which is computationally costly and time-consuming. In this work, we propose a CmpCNN framework based on convolutional neural networks (CNNs) with a transfer learning method to accelerate the CMP modeling process. It utilizes a multi-input strategy by feeding the binary image of layout and its density into our CNN-based model to extract features more efficiently. The transfer learning method is adopted to different CMP process parameters and different categories of circuits to further improve its prediction accuracy and convergence speed. Experimental results show that our CmpCNN framework achieves a competitive root mean square error ( RMSE ) of 2.7733Å with 1.89× reduction compared to the prior work, and a 57× speedup compared to the commercial CMP simulation tool.