Crowd gathering events deeply affect public safety. To enhance city management and avoid potential risks, many algorithms are designed for crowd analysis and deployed on video surveillance. Widely applied deep learning models also can be trained for crowd analysis. However, there are still few works focusing on crowd gathering behavior. Furthermore, as a result of the lack of interpretability of deep learning models, which also brings potential risk of being rejected by the users. In this paper, we categorize crowd behaviors into wandering, merging, walking gathering, standing gathering, and dispersing. Also, we propose an interpretable framework for crowd gathering understanding based on crowd density estimation model and proposed crowd descriptors, named Irregularity, Sparsity, Randomness, and Volatility. The experiments on the PETS2009 dataset demonstrate our method has outperformed the previous works on the crowd gathering understanding task. Moreover, we further analyze the framework performance with different crowd feature extraction models and the relations between our descriptors and crowd behavior. Besides, an ablation study is conducted to investigate the effectiveness of the descriptors and differences between density estimation models. The results demonstrate the effectiveness and the much better interpretability of our framework. Our descriptors also show significant contributions to the quantification of crowd gathering behaviors.