This research presents MAVE, a framework that combines a multi-agent large language model (LLM) system with an augmented reality (AR) interface to support interactive design and robotic fabrication. While LLM agents are highly flexible, grounding user input in task-relevant digital and physical context remains a key challenge for effective human-machine collaboration. To address this, we introduce five grounding strategies – model, object, spatial, tool, and goal references – that structure how agents interpret and act on user input. We conducted ablation studies to evaluate these strategies, showing that they improve agent execution efficiency and accuracy while reducing the number of interaction turns. We further demonstrate the framework’s adaptability through a design workshop, in which student teams extended a truss design-and-fabrication scenario involving a human and a cobot to prototype diverse interactive workflows. The workshop outcomes highlight the relevance of augmented multi-agent LLM systems for design and fabrication and suggest the framework’s potential to support rapid prototyping of interactive multi-actor workflows.

Digital Twins (DTs) hold promise for Construction 4.0, yet current applications remain fragmented, especially at the fabrication and construction level, where heterogeneous machines and workflows must be coordinated in real time. This paper addresses this gap in two steps. First, by understanding what the requirements are for developing a DT architecture for fabrication and construction. Second, using the derived requirements to propose a DT architecture for digital fabrication and construction structured around three components: tasks with a semi-structured data schema, virtual–physical pairs that bridge machines through protocol-specific virtual actors, and modular services for monitoring, simulation, and adaptive control. An execution engine coordinates these components via an event-driven mechanism, ensuring real-time task management and robust traceability. The DT architecture is validated through three case studies: prefabricated timber assembly, collective robotic construction, and large-scale 3D printing, demonstrating its capacity to integrate diverse machines, manage sequential and adaptive processes, and support on-the-fly task injection. By combining flexibility, interoperability, and data integration, the proposed framework contributes a generalisable foundation for DT-enabled fabrication and construction workflows, advancing towards more adaptive and resilient Construction 4.0 practices. • Proposes a scalable digital twin architecture for fabrication and construction. • Introduces data cores and virtual–physical pairs for heterogeneous systems. • Integrates monitoring, simulation, and adaptive control as modular services. • Validates the approach in timber assembly, robotics, and large-scale 3D printing.

Human–robot collaboration (HRC) offers promising potential for more flexible and sustainable production practices in architecture and construction. This requires HRC setups to scale up from light-payload collaborative robots to conform with the scale of building construction while considering the safety and teamwork culture for workers. This research proposes a system for large-scale multi-user HRC using head-mounted augmented reality (AR) devices. To achieve this, we contribute three methods that work in conjunction: (1) an AR system that enables multiple users to share tasks and work together with robots; (2) a dynamic human task allocation engine that reacts to the changing production teams and task types; and (3) a safety zone generation and allocation method to configure human collaboration in shared space with large-scale robots. The system is evaluated using a case study of prefabricated timber cassettes combining discrete event simulations , a user study and a fabrication process demonstrator with an industry partner.

Recent efforts to connect builders to digital designs during construction have primarily focused on visual augmented reality, which requires accurate registration and specific lighting, and which could prevent a user from noticing safety hazards. Haptic interfaces, on the other hand, can convey physical design parameters through tangible local cues that don’t distract from the surroundings. We propose two edge-changing haptic devices that use small inertial measurement units (IMUs) and linear actuators to guide users to perform construction tasks in real time: Drangle gives feedback for angling a drill relative to gravity, and Brangle assists with orienting bricks in the plane. We conducted a study with 18 participants to evaluate user performance and gather qualitative feedback. All users understood the edge-changing cues from both devices with minimal training. Drilling holes with Drangle was somewhat less accurate but much faster and easier than with a mechanical guide; 89% of participants preferred Drangle over the mechanical guide. Users generally understood Brangle’s feedback but found its hand-size-specific grip, palmar contact, and attractive tactile cues less intuitive than Drangle’s generalized form factor, fingertip contact, and repulsive cues. After summarizing design considerations, we propose application scenarios and speculate how such devices could improve construction workflows.

The use of augmented reality technology to support humans with situated visualization in complex tasks such as navigation or assembly has gained increasing importance in research and industrial applications. One important line of research regards supporting and understanding collaborative tasks. Analyzing collaboration patterns is usually done by conducting observations and interviews. To expand these methods, we argue that eye tracking can be used to extract further insights and quantify behavior. To this end, we contribute a study that uses eye tracking to investigate participant strategies for solving collaborative sorting and assembly tasks. We compare participants’ visual attention during situated instructions in AR and traditional paper-based instructions as a baseline. By investigating the performance and gaze behavior of the participants, different strategies for solving the provided tasks are revealed. Our results show that with situated visualization, participants focus more on task-relevant areas and require less discussion between collaboration partners to solve the task at hand.

Abstract In architectural design optimization, fitness landscapes are used to visualize design space parameters in relation to one or more objective functions for which they are being optimized. In our design study with domain experts, we developed a visual analytics framework for exploring and analyzing fitness landscapes spanning data, projection, and visualization layers. Within the data layer, we employ two surrogate models and three sampling strategies to efficiently generate a wide array of landscapes. On the projection layer, we use star coordinates and UMAP as two alternative methods for obtaining a 2D embedding of the design space. Our interactive user interface can visualize fitness landscapes as a continuous density map or a discrete glyph-based map. We investigate the influence of surrogate models and sampling strategies on the resulting fitness landscapes in a parameter study. Additionally, we present findings from a user study (N= 12), revealing how experts’ preferences regarding projection methods and visual representations may be influenced by their level of expertise, characteristics of the techniques, and the specific task at hand. Furthermore, we demonstrate the usability and usefulness of our framework by a case study from the architecture domain, involving one domain expert.

Integrating robotic systems in architectural and construction processes is of core interest to increase the efficiency of the building industry. Automated planning for such systems enables design analysis tools and facilitates faster design iteration cycles for designers and engineers. However, generic task-and-motion planning (TAMP) for long-horizon construction processes is beyond the capabilities of current approaches. In this paper, we develop a multi-agent TAMP framework for long horizon problems such as constructing a full-scale building. To this end we extend the Logic-Geometric Programming framework by sampling-based motion planning, a limited horizon approach, and a task-specific structural stability optimization that allow an effective decomposition of the task. We show that our framework is capable of constructing a large pavilion built from several hundred geometrically unique building elements from start to end autonomously.

In recent years, extensive research in the HCI literature has explored interactive techniques for digital fabrication. However, little attention in this body of work has examined how to involve and guide human workers in fabricating larger-scale structures. We propose a novel model of crowdsourced fabrication, in which a large number of workers and volunteers are guided through the process of building a pre-designed structure. The process is facilitated by an intelligent construction space capable of guiding individual workers and coordinating the overall build process. More specifically, we explore the use of smartwatches, indoor location sensing, and instrumented construction materials to provide real-time guidance to workers, coordinated by a foreman engine that manages the overall build process. We report on a three day deployment of our system to construct a 12-tall bamboo pavilion with assistance from more than one hundred volunteer workers, and reflect on observations and feedback collected during the exhibit.