Text-based CAPTCHA remains a widely employed scheme for distinguishing between human users and machine attackers during logging-in on systems. In this paper, we propose a reinforced perturbation generation (RPG) framework to automatically construct effective perturbation factors with reinforcement learning, and achieve a perturbed CAPTCHA that is user-friendly but challenging for machine attackers. More specifically, RPG exploits a perturbation initialization (PI) component to provide a preliminary perturbation factor. Furthermore, a perturbation reinforcement (PR) component is devised to optimize the combinations of multiple perturbation factors by a number of perturbation generation methods, which is achieved by reducing the gap between estimated cumulative rewards and real cumulative rewards. In particular, an attack model is introduced to produce the reward based on whether it can correctly recognize the perturbation CAPTCHA. The multiple perturbation factors are fused to be combined with the original CAPTCHA to against machine attackers. Extensive experiments conducted on eight real-world CAPTCHA datasets show outstanding performance against the CAPTCHA attack models.