Automation of mobile machinery is critical in the construction industry to improve efficiency and ensure safety. Perception technologies, particularly for detecting and monitoring the actions of construction machinery, are essential for optimizing workflows and mitigating accident risks. However, the complex nature of construction environments, the variety of machines, and the dynamic interactions at construction sites pose significant challenges for reliable object detection and action recognition. This study introduces a deep learning approach using temporal vision information for object detection and action recognition of mobile machinery in construction environments. In particular, a novel strategy called Integrated YL-SF is proposed, which integrates the YOLOv8 framework with the SlowFast model enhanced by Transformers to achieve robust action recognition and motion analysis of construction machinery. The proposed method is evaluated on a custom dataset with a variety of machine types and real-world operating environments, and it is benchmarked against the standard YOLOv8 model. The results show that the Integrated YL-SF framework outperforms existing methods and effectively addresses challenges such as dynamic scenarios, object occlusion, and multi-machine interactions in complex environments.