Temporal Knowledge Graph Question Answering (TKGQA) aims to answer natural language questions based on quadruple facts stored in Temporal Knowledge Graphs (TKGs). Recent studies have integrated Large Language Models (LLMs) to handle the complex semantic reasoning required by temporal questions. They typically linearize TKG quadruples into plain text, reducing TKGQA to a top-n text retrieval and context-based question answering problem. However, this textualization process destroys the original quadruple structure and entangles structural and temporal information with text semantics. Moreover, selecting top-n facts based on semantic similarity inevitably introduces a large amount of irrelevant noise into the LLM input, which degrades reasoning performance. To address these limitations, we propose SSR, a Structured Subgraph Retrieval framework for TKGQA with LLMs. In SSR, a Temporal Question Parser is first introduced to extract structured subgraph patterns and temporal constraints from the input questions, leveraging background context obtained via text retrieval. The Subgraph Retrieval module is then applied to directly filter a relevant subgraph from the TKG that contains the facts required to answer the question. The retrieved subgraph is further temporally compressed and subsequently incorporated into the LLM context for final question answering. Experimental results on two benchmark datasets demonstrate that SSR consistently outperforms strong baselines by a clear margin, achieving state-of-the-art performance. Our code is available at https://github.com/zhangli-coding/SSR.