This paper proposes a new approach for the similarity measure and ranking of audio clips by graph modeling and matching. Instead of using frame-based or salient-based features to measure the acoustical similarity of audio clips, segment-based similarity is proposed. The novelty of our approach lies in two aspects: segment-based representation, and the similarity measure and ranking based on four kinds of similarity factors. In segmentbased representation, segments not only capture the change property of audio clip, but also keep and present the change relation and temporal order of audio features. In the similarity measure and ranking, four kinds of similarity factors: acoustical, granularity, temporal order and interference are progressively and jointly measured by optimal matching and dynamic programming, which guarantee the comprehensive and sufficient similarity measure between two audio clips. The experimental result shows that the proposed approach is better than some existing methods in terms of retrieval and ranking capabilities.