This paper presents a general approach for terrain following (including obstacle avoidance) of a vertical take-off and landing vehicle (VTOL) using multiple observation points. The VTOL vehicle is assumed to be a rigid body, equipped with a minimum sensor suite (camera, IMU and barometric altimeter), manoeuvering over a textured rough terrain made of a concatenation of planar surfaces. Assuming that the forward velocity is separately regulated to a desired non-zero value, the proposed control approach ensures terrain following and guarantees the vehicle does not collide with obstacles during the task. The proposed control acquires an optical flow from multiple spatially separate observation points, typically obtained via multiple cameras or non collinear directions in a unique camera. The proposed control algorithm has been tested extensively in simulation and then implemented on a quadrotor UAV to demonstrate the performance of the closed loop system.