Recommender systems (RSs) are designed to retrieve candidate items a user might be interested in from a large pool, with a typical approach being the use of graph neural networks (GNNs) to capture high-order interaction relationships. As large language models (LLMs) have demonstrated remarkable success across various domains, researchers are exploring ways to apply their capabilities for improving recommendation performance. However, existing work limits the use of LLMs to either re-ranking recommendation results of traditional RSs or pre-processing the datasets as data augmenters. Both lines of work failed to explore LLMs' capabilities during the filtering process of candidate items, which may lead to suboptimal performance. Instead, we propose to leverage LLMs' reasoning abilities during the candidate filtering process, and introduce Chain Of Retrieval ON grAphs (CORONA) to progressively narrow down the range of candidate items on interaction graphs with the help of LLMs: (1) First, LLM performs preference reasoning based on user profiles, with the response serving as a query to extract relevant users and items from the interaction graph as preference-assisted retrieval ; (2) Then, using the information retrieved in the previous step along with the purchase history of target user, LLM conducts intent reasoning to help refine an even smaller interaction subgraph as intent-assisted retrieval ; (3) Finally, we employ a GNN to capture high-order collaborative filtering information from the extracted subgraph, performing GNN-enhanced retrieval to generate the final recommendation results. The proposed framework leverages the reasoning capabilities of LLMs during the retrieval process, while seamlessly integrating GNNs to enhance overall recommendation performance. Extensive experiments on various datasets and settings demonstrate that our proposed CORONA achieves state-of-the-art (SOTA) performance with an 18.6% relative improvement in recall and an 18.4% relative improvement in NDCG on average. Our code is available on GitHub at https://github.com/BUPT-GAMMA/CORONA.