Windows domain is regarded as a primary target for intranet penetration since a large amount of sensitive information is stored in such domain with Windows OS. However, penetration testing is a intricate and time-consuming task, which is usually dedicated to experienced experts. To alleviate and partially solve this problem, we hereby propose an automated Windows domain penetration testing framework (AWDP). Firstly, we establish the test scenario as a Markov Decision Process (MDP) and then design a simulator for the Windows Domain penetration testing with OpenAI's Gymnasium. Secondly, we implement our automated Windows Domain penetration approach with four sequential steps, collecting domain and host information, modeling with acquired data, discovering optimal attack path through Deep Q-Learning Network (DQN), and performing penetration testing actions. Finally, to validate the effects of the proposed method, we conduct tests in real deployed domains. Experimental results demonstrate that, the proposed models and algorithms in the AWDP framework exhibit robust performance. Moreover, the framework adapts to different environments with rational and efficient estimated attack paths, which eventually enables end-to-end automation of Windows Domain penetration testing.