Large Language Models (LLMs) have demonstrated remarkable capabilities, yet they often suffer from hallucinations and lack up-to-date knowledge. Retrieval-Augmented Generation (RAG) addresses these limitations by grounding LLMs in external knowledge. While vector-based RAG is effective for simple queries, it struggles with complex queries that require multi-hop reasoning. Graph-based RAG frameworks have emerged to solve this by constructing knowledge graphs that capture global relationships and enable multi-hop reasoning. However, these graph-based approaches frequently underperform on simple fact-based queries compared to their vector-based counterparts, as they may lose detailed entity information. In this paper, we conduct dataset-level and framework-level analysis targeting graph-based RAG approaches. We find that existing QA benchmark datasets can be split to "Local" and "Global" queries that have different properties; and different RAG frameworks perform differently on these two kinds of queries. Concretely, existing graph-based RAG frameworks, including recent dual-branch ones, cannot consistently outperform vector-based RAG on "Local" queries. We attribute this phenomenon to the fact that graph-based RAG often employs a fixed retrieval strategy, leading to redundant information retrieval and unnecessary cost for simple queries. Based on the analysis, we propose QA-GraphRAG, a new query-adaptive plug-and-play retrieval integration for graph-based RAG frameworks. QA-GraphRAG incorporates a pre-trained router that predicts the optimal knowledge hierarchy from which to start retrieval based on the characteristics of the input query. Extensive experiments on conventional KGQA datasets and GraphRAG-Bench demonstrate that equipping existing graph-based RAG frameworks with our QA-GraphRAG leads to substantial performance improvements.