Integration of visual sensing and motion planning can play a critical role in autonomous robot operation. We present a framework for vision-based robot motion planning that uses learning to handle arbitrarily configured cameras and robots. The theoretical basis of this approach is the concept of the perceptual control manifold (PCM) that extends the notion of the robot configuration space to include sensor space. This allows the inclusion of visual constraints in the motion planning. However, the analytical derivation of PCM is difficult in most cases and also depends on calibration of the camera. To overcome this modeling uncertainly, we propose the use of a topology representing network (TRN) to learn a suitable representation of the PCM. By exploiting the topology preserving features of the neural network, path planning strategies defined on the TRN lead to flexible obstacle avoidance. The practical feasibility of the approach is demonstrated by the results of simulation with a PUMA robot and experiments with a Mitsubishi robot.