In this paper we propose to effectively consider the shape and size variations for object classification. Specifically, a novel image matching method is proposed to incorporate the image segmentation with Spatial Pyramid Matching (SPM), and test our method on flower classification. A Level Influence Factor (LIF) is introduced to represent weights of different pyramid levels based on the statistical information of each segmented image. Then the images are classified based on the LIF weighted spatial pyramid bag-of-visual-words feature, and some levels with weight values zeros are not needed to be compared further. Also, in SPM matching stage, the block in one image is compared with not only its corresponding block in another image, but also the spatially neighboring blocks of the corresponding blocks to find the best match. This fuzzy matching method can incorporate some translation of objects. Experiments are performed on a flower dataset containing 1360 images from 17 different categories. And experimental results demonstrate that our proposed method has better time efficiency than traditional SPM and outperforms the state-of-art flower classification methods.