Despite continuous advancements in the capabilities of large language models (LLMs), numerical reasoning remains a challenging area.Techniques like chain-of-thought prompting, tree-of-thought prompting, and program-ofthought prompting guide LLMs through intermediate reasoning steps.Although in-context learning with few-shot prompting has improved performance, LLMs still lag behind state-ofthe-art models on financial numerical reasoning datasets such as FinQA and ConvFinQA.In this work, we introduce FINDER, a novel twostep framework, to enhance LLM's capabilities in financial numerical reasoning.The first step utilizes a generative retriever to extract relevant facts from unstructured data, including both text and tables.This is followed by contextaware Program of Thought prompting with dynamic selection of in-context examples.Our model FINDER achieves a new state-of-theart performance on both the FinQA and Con-vFinQA datasets, surpassing previous benchmarks with execution accuracy improvements of 5.98% and 4.05%, respectively.