In this paper, we propose a formalised process for the generation of dynamically generated SQL queries that are vulnerable to SQL injection attacks for the training of reinforcement learning offensive security agents. These queries are presented as capture-the-flag style challenges and deployed as a scalable, realistic and virtualised infrastructure using the Docker framework. These challenges are deployed for integration with OpenAI Gym environments for the implementation of state of the art reinforcement learning models. We analyse the needs and requirements of recent research that utilises reinforcement learning for offensive security and propose a method of generating dynamic CTF environments with limited domain knowledge on CTF challenges. Our solution can generate vulnerable SQL queries with any provided SQL dataset and deliver a front-end interface for both agent and human interaction. We analyse the performance of the environment when deployed on an end user device to demonstrate the feasibility of such an environment for training reinforcement learning agents. Our work fits in the wider research on the development of training scenarios for reinforcement learning agents focused on conducting offensive security tasks. Our results aim to contribute to enhancing the challenges facing reinforcement learning in offensive security tasks by providing variable realistic and dynamic environments for training.