Deep learning has achieved state‐of‐the‐art performance in accuracy of many computer vision tasks. However, convolutional neural network is difficult to deploy on resource constrained devices due to their limited computation power and memory space. Thus, it is necessary to prune the redundant weights and filters rationally and effectively. Considering that the pruned model still exists, redundancy after weight pruning or filter pruning alone, a method of combining weight pruning and filter pruning is proposed. First, filter pruning is performed, which is to remove filters with least importance and using fine‐tuning to recover the model's accuracy. Then, all connection weights below a threshold are set to zero. Finally, the pruned model obtained by the first two steps is fine‐tuned to recover its predictive accuracy. Experiments on MNIST and CIFAR‐10 datasets demonstrate that the proposed approach is effective and feasible. Compared with only weight pruning or filter pruning, the mixed pruning can achieve higher compression ratio of the model parameters. For LeNet‐5, the proposed approach can achieve a compression rate of 13.01×, with 1% drop in accuracy. For VGG‐16, it can achieve a compression rate of 19.20×, incurring 1.56% accuracy loss.