We address the problem of controlling a robot to track a randomly moved object using visual servoing techniques. A new visual servo mechanism, based on the image-based servoing principle, is proposed in this study. A fuzzy estimator is used to estimate the relationship between perspective projection and the kinematics and an adaptive robust controller is used to compensate the tracking error induced by the estimator and the tracking error caused by the robot's parameter uncertainties. Using this method, the extrinsic calibration between the robot system and the camera system can be eliminated. The proposed controller not only guarantees the global stability of the visual servoing system but also drives the target error to a neighborhood of zero. Simulation results are presented which show the performance of the proposed controller works well.