In the field of autonomous driving, safe and efficient decision-making through deep reinforcement learning remains a significant challenge. Existing methods often struggle to adapt to the dynamic and complex conditions of urban environments, while the lack of interpretability in reinforcement learning algorithms raises safety concerns. To address these issues, this paper proposes a reinforcement learning method guided by reward machine. Firstly, a reward machine tailored for autonomous driving scenarios is constructed to better guide the agent's behavior selection. Then, a time sensitivity factor is proposed to adjust the generation of counterfactual experiences, optimizing the effectiveness of policy learning. Furthermore, to improve the safety of autonomous driving decisions, formal verification of the decision-making process is conducted via the UPPAAL model checker, enabling the identification and handling of potential hazardous states. Finally, the effectiveness and safety of this method are validated through a case study of an autonomous driving system, demonstrating that the proposed reward machine-guided reinforcement learning algorithm performs well in complex road scenarios.