This research presents MAVE, a framework that combines a multi-agent large language model (LLM) system with an augmented reality (AR) interface to support interactive design and robotic fabrication. While LLM agents are highly flexible, grounding user input in task-relevant digital and physical context remains a key challenge for effective human-machine collaboration. To address this, we introduce five grounding strategies – model, object, spatial, tool, and goal references – that structure how agents interpret and act on user input. We conducted ablation studies to evaluate these strategies, showing that they improve agent execution efficiency and accuracy while reducing the number of interaction turns. We further demonstrate the framework’s adaptability through a design workshop, in which student teams extended a truss design-and-fabrication scenario involving a human and a cobot to prototype diverse interactive workflows. The workshop outcomes highlight the relevance of augmented multi-agent LLM systems for design and fabrication and suggest the framework’s potential to support rapid prototyping of interactive multi-actor workflows.

Human–robot collaboration (HRC) offers promising potential for more flexible and sustainable production practices in architecture and construction. This requires HRC setups to scale up from light-payload collaborative robots to conform with the scale of building construction while considering the safety and teamwork culture for workers. This research proposes a system for large-scale multi-user HRC using head-mounted augmented reality (AR) devices. To achieve this, we contribute three methods that work in conjunction: (1) an AR system that enables multiple users to share tasks and work together with robots; (2) a dynamic human task allocation engine that reacts to the changing production teams and task types; and (3) a safety zone generation and allocation method to configure human collaboration in shared space with large-scale robots. The system is evaluated using a case study of prefabricated timber cassettes combining discrete event simulations , a user study and a fabrication process demonstrator with an industry partner.

The use of augmented reality technology to support humans with situated visualization in complex tasks such as navigation or assembly has gained increasing importance in research and industrial applications. One important line of research regards supporting and understanding collaborative tasks. Analyzing collaboration patterns is usually done by conducting observations and interviews. To expand these methods, we argue that eye tracking can be used to extract further insights and quantify behavior. To this end, we contribute a study that uses eye tracking to investigate participant strategies for solving collaborative sorting and assembly tasks. We compare participants’ visual attention during situated instructions in AR and traditional paper-based instructions as a baseline. By investigating the performance and gaze behavior of the participants, different strategies for solving the provided tasks are revealed. Our results show that with situated visualization, participants focus more on task-relevant areas and require less discussion between collaboration partners to solve the task at hand.