Edge computing provides a promising solution to the resource-constrained problem of mobile devices running simultaneous localization and mapping (SLAM) systems. However, the existing traditional edge-assisted SLAM systems rely on ideal network conditions and static environment assumptions, which poses challenges for applications in bandwidth constrained networks and dynamic scenes. Aiming at the issue, this paper proposes fast and accurate edge-assisted semantic SLAM (FAES-SLAM) towards dynamic environments. First, an edge-assisted framework is presented by the integration of object detection and dense mapping, which accelerates the SLAM system and maintains accuracy. Then, an adaptive keyframe offloading strategy and an efficient dynamic feature point culling scheme are designed and incorporated into the above framework to improve the speed and accuracy further. The offloading strategy adaptively decreases the excessive latency in terms of various network conditions, and the culling scheme effectively increases the localization and mapping precision for dynamic scenarios. We perform extensive experiments on public datasets and in the actual coal mine laboratory. Compared with classical edge-assisted SLAM system, FAES-SLAM can achieve over 90.9% improvement in localization accuracy with a 13.2% increase in tracking time.