A visuo-motor coordination scheme is proposed for a robot manipulator in the paper. The motion determination problem is learned by this scheme using neural networks. The motion process consists of basic motion and adjusting motion. A basic network is employed to determine the gross configuration of the robot end-effector using the visual information from two fixed cameras. An adjusting network serves to adjust the configuration of the robot finely using the visual information from two end-effector mounted cameras, so that the robot can handle an object according to the task. The efficiency and the adaptability of the basic network is shown in the simulation. The proposed scheme essentially regards complex calibration and geometric calculations as a simple mapping of the neural networks. The efficiency of this scheme will lead to expanding the application of the robot in a flexible environment.