Wi-Fi device-free tracking enables the inference of user behaviors without physical contact, which is crucial for intelligent indoor location-based services. Nevertheless, the practical implementation of current tracking systems is constrained by several critical limitations: 1) The low-quality sensing signals in complex scenarios lead to increased tracking errors; 2) Existing methods inadequately adjust to dynamic environments, necessitating additional data collection or retraining processes. To address these challenges, this paper introduces EOC-Tracking, a device-free Wi-Fi tracking system that dynamically incorporates environmental information. Our key innovation involves leveraging obstacles to correct illogical users' trajectories and facilitate adjustment to varying environments. This significantly improves the accuracy of the follow-up in complex and changing environments. The EOC-Tracking system is built upon three fundamental design principles: 1) A lightweight dual-branch neural network architecture that effectively fuses environmental data with Wi-Fi signal characteristics; 2) An autonomous map updating mechanism that facilitates real-time adaptation to environmental layout modifications without human intervention; 3) A sophisticated data-driven, phased training paradigm that optimizes the model's ability to learn and apply obstacle constraints. We implement EOC-Tracking using commercial Wi-Fi devices and deploy it on low-power embedded systems such as the MCU. Experimental results demonstrate that EOC-Tracking can reduce tracking errors by at most 49.48% compared to datadriven methods and 62.21% compared to model-based methods in various complex scenarios.