Neural network models have been widely used in many fields and achieved many successes. Quantifying model uncertainty, which is able to show the reliability degree of predictions, has attracted more and more researchers' attention. Bayesian neural networks are well known in this category, as they could provide the distributions of predictions, but it takes a prohibitive computational cost to train them. In this paper, we develop a novel way to quantify the model uncertainty of the models trained with data augmentation, i.e., performing data transformation such as adding Gaussian noise to input data before every forward pass of model training and inference. We show that data augmentation is equivalent to performing a corresponding transformation on model weights for some data augmentation methods. We also show that training with Gaussian noise approximates Bayesian inference in Gaussian processes. The experiments on both regression and classification tasks demonstrate that the proposed data augmentation models achieve better predictive performance than baseline models. For all four datasets, the calculated predictive uncertainty can be used as an uncertainty function in selective prediction to reject high risk predictions effectively.