This paper outlines a video-based method for detecting intruder aircraft to assist with sense and avoid for small, unmanned aerial vehicles (UAVs). A key consideration is that the algorithm is suitable for real-time implementation on field-programmable gate arrays (FPGAs). The method begins by estimating the motion in the scene using regional phase correlation, and then fitting the positional predictions obtained using these regional motion vectors to an affine model representing the effect of camera motion on the background imagery. A combination of metrics, including phase correlation peak height (a confidence measure) and the error between the position predicted by the affine model and that obtained using the measured phase correlation vector, is used to indicate regions of interest where moving targets are present. The ability of the algorithm to detect approaching aircraft is analyzed using a number of aerial video sequences with different encounter geometries.