Recognizing construction activities from images enhances decision-making by providing context-aware insights into project progress, resource allocation, and productivity. However, conventional approaches, such as supervised learning and knowledge-based approach, struggle to generalize to the dynamic nature of construction sites due to limited annotated data and rigid knowledge patterns. To address these limitations, we propose a novel method that integrates Large Language Models (LLMs) with structured domain knowledge via ontology-based prompting. In our approach, visual features such as entities, spatial arrangements, and actions are mapped to predefined concepts in a construction-specific ontology, resulting in symbolic scene representations. In-context learning is employed by constructing prompts that include multiple structured examples, each describing a scenario with its associated activities. By analyzing these ontology-grounded examples, the LLM learns patterns that connect symbolic representations to construction activity labels, enabling generalization to new, unseen scenes. We evaluated the method using GPT-based models on a dataset covering 29 construction activity types. The model achieved an activity recognition accuracy of 73.68 %, and 50.00 % when jointly identifying the activity and its associated entities. Ablation studies confirmed the positive effects of including Chain-of-Thought reasoning, diverse visual concepts, and richer context examples. These results demonstrate the potential of ontology-informed prompting to support scalable and adaptive visual understanding in construction domains.

Deep Learning (DL)-based visual analytic systems have demonstrated substantial potential in enhancing construction management. Yet, these systems should go beyond merely facilitating recognition-based tasks and strive to capture high-level semantics. While ontologies have emerged as foundational tools for semantic-enriched visual analytic systems, many existing application-oriented ontologies in the construction domain lack a comprehensive conceptualization and formalization of visual information, restricting their robustness and applicability. Furthermore, there is a notable gap in integrating low-level visual information and high-level semantics. To address these challenges, we introduce the Construction Semantic Enrichment ontology (ConSE). ConSE offers a formal representation of visual information relevant to construction site imagery. It is designed to support the development of semantic-enriched visual analytic systems in the construction domain. To evaluate the ontology, we implement an automated consistency-checking process internally. In addition, we execute three task-based use cases to validate the proposed ontology. The evaluation results demonstrate the efficacy and practicality of the ConSE, confirming its ability to bridge the identified research gaps and contribute to the advancement of semantic-enriched visual analytic systems in the construction domain.