In this paper, we present a conceptually novel framework for vehicle pose estimation from given RGB images. Our approach extends Mask R-CNN by adding two branches for coarse viewpoint estimation and keypoint detection, in parallel with the existing branches for mask segmentation and 2D object detection in the training stage. Capitalizing on the estimated mask and the mask renderings from ShapeNet in the inference stage, we propose a mask optimization scheme to recover the vehicle poses from 2D-3D correspondences. Then, we enforce geometric constraint on these vehicle poses in a coarse-to-fine hybrid approach for robustness. Experimentally, our framework outperforms the state-of-the-art approaches on the very challenging PASCAL3D+ dataset.