Vision-based control of unmanned aerial vehicles (UAVs) has been adopted in a wide range of applications due to the availability of low-cost onboard sensors and computers. Tuning such systems to work properly requires extensive domain-specific experience, which limits the growth of emerging applications. Moreover, obtaining performance limits for UAVs performing visual servoing is difficult due to the complexity of the models used. In this article, we propose a novel noise-tolerant approach for real-time identification and tuning of visual servoing systems. This is based on the deep neural networks (DNNs) classification of the system response generated by the modified relay feedback test (MRFT). The proposed method, called DNN with noise-protected MRFT (DNN-NP-MRFT), can be used with a multitude of vision sensors and estimation algorithms despite high levels of sensor noise. The response of DNN-NP-MRFT to noise perturbations is investigated and its effect on identification and tuning performance is analyzed. The proposed DNN-NP-MRFT is able to detect performance changes induced by the use of high latency vision sensors or by integrating an inertial measurement unit sensor into the UAV states estimation pipeline. Experimental identification closely matches simulation results, which can be used to explain the system behavior and predict the closed-loop performance limits for a given hardware and software setup. We also demonstrate the ability of DNN-NP-MRFT-tuned UAVs to reject external disturbances like wind or human push and pull. Finally, we discuss the advantages of the proposed DNN-NP-MRFT visual servoing design approach compared with other approaches in the literature.