The real-time synthesis of 3D spatial audio has many applications, from virtual reality to navigation for the visually-impaired. Head-related transfer functions (HRTF) can be used to generate spatial audio based on a model of the user's head. Previous studies have focused on the creation and interpolation of these functions with little regard for real-time performance. In this paper, we present an FPGA-based platform for real-time synthesis of spatial audio using FIR filters created from head-related transfer functions. For performance reasons, we run filtering, crossfading, and audio output on FPGA fabric, while calculating audio source locations and storing audio files on the CPU. We use a head-mounted 9-axis IMU to track the user's head in real-time and adjust relative spatial audio locations to create the perception that audio sources are fixed in space. Our system, running on a Xilinx Zynq Z-7020, is able to support 4X more audio sources than a comparable GPU and 8X more sources than a CPU while maintaining sub-millisecond latency and comparable power consumption. Furthermore, we show how our system can be leveraged to communicate the location of landmarks and obstacles to a visually-impaired user during a sailing race or other navigation scenario. We test our system with multiple users and show that, as a result of our reduced latency, a user is able to locate a virtual audio source with an extremely high degree of accuracy and navigate toward it.