Network pruning is an effective technique that reduces the computational costs of networks while maintaining accuracy. However, pruning requires expert knowledge and hyperparameter tuning, such as determining the pruning rate for each layer. Automatic pruning methods address this challenge by proposing an effective training-free metric to quickly evaluate the pruned network without fine-tuning. However, most existing automatic pruning methods only investigate a certain pruning granularity, and it remains unclear whether metrics benefit automatic pruning at different granularities. Neural architecture search also studies training-free metrics to accelerate network generation. Nevertheless, whether they apply to pruning needs further investigation. In this study, we first systematically analyze various advanced training-free metrics for various granularities in pruning, and then we investigate the correlation between the training-free metric score and the after-fine-tuned model accuracy. Based on the analysis, we proposed FreePrune score, a more general metric compatible with all pruning granularities. Aiming at generating high-quality pruned networks and unleashing the power of FreePrune score, we further propose FreePrune, an automatic framework that can rapidly generate and evaluate the candidate networks, leading to a final pruned network with both high accuracy and pruning rate. Experiments show that our method achieves high correlation on various pruning granularities and comprehensively improves the accuracy.

Pruning algorithms and sparse matrix-matrix multiplication (SpMM) kernels have been widely studied to accelerate DNN inference on GPUs. However, unstructured pruning spoils the regularity of data layout and incurs undesired speedup performance. Prior structured pruning methods remove weights coarsely and cause significant accuracy loss. In this work, we co-design Re-compact Pruning and SpMM kernel to achieve high acceleration while maintaining accuracy. Re-compact Pruning compacts both the sparse rows and columns with similar sparse patterns into dense blocks, on which the vector pruning is performed. An SpMM kernel is designed to fully leverage the regularity of the pruned matrix. It contains a Tile Padding operation to balance the workloads, a Tile Sorting operation to enable loop unrolling during compilation, and a Merge Write-back strategy to reduce the memory accesses. Experimental results show that our Re-compact Pruning and SpMM kernel outperform its counterparts by 4.53 ×, 4.44 ×, and 3.24 × for accelerating ResNet50, NMT, and Transformer, respectively.