With the increasing complexity in manual assembly and a demographic decline in skilled workforce, the importance of well-documented processes through assembly instructions has grown. Creating these instructions is a time-consuming and knowledge-intensive task that typically relies on experienced employees. Although various automation solutions have been proposed to assist in generating assembly instructions, they often fall short in providing detailed textual guidance. With the rise of generative artificial intelligence (AI), new potentials arise in this domain. Therefore, this paper explores these potentials by employing various large language models (LLMs), prompting techniques and input data in an experimental setup for generating detailed assembly instructions, including the planning of assembly sequences as well as textual guidance on tools, assembly activities, and quality assurance measures. The findings reveal promising opportunities in leveraging LLMs but also substantial challenges, particularly in assembly sequence planning. To improve the reliability of generating assembly instructions, we propose a multi-agent concept that decomposes the complex task into simpler subtasks, each managed by specialized agents.

The efficiency of manual assembly can be significantly improved by utilizing assistance systems that display as-sembly instructions. However, generating and maintaining these instructions require substantial effort, especially for low-volume or highly customized products. If neglected, this can lead to outdated instructions, frustrated operators, and the abandonment of the assistance platform. Recent advancements in Large Language Models (LLMs) have made it possible to generate high-quality assembly instructions given the right input. However, relying solely on LLMs to plan the assembly process risks producing unfeasible assembly sequences due to potential hallucinations by the models. To address this, Reinforcement Learning (RL) can be used in simulations to plan the assembly process, imposing restrictions on impractical movements. We propose a framework that integrates RL and LLMs to generate practical and accurate assembly instructions. In our framework, RL is used within a simulation to generate feasible assembly sequences. These sequences are then transformed into detailed assembly instructions by an LLM. We evaluate our framework on real-world product assemblies, generating comprehensive assembly instructions from corresponding CAD files. Our results demonstrate the potential of combining RL and LLMs to automate the generation of assembly instructions, thereby overcoming the limitations of current assistance systems and enhancing the efficiency of manual assembly processes.

Accurate 3D human pose estimation from single images is possible with sophisticated deep-net architectures that have been trained on very large datasets. However, this still leaves open the problem of capturing motions for which no such database exists. Manual annotation is tedious, slow, and error-prone. In this paper, we propose to replace most of the annotations by the use of multiple views, at training time only. Specifically, we train the system to predict the same pose in all views. Such a consistency constraint is necessary but not sufficient to predict accurate poses. We therefore complement it with a supervised loss aiming to predict the correct pose in a small set of labeled images, and with a regularization term that penalizes drift from initial predictions. Furthermore, we propose a method to estimate camera pose jointly with human pose, which lets us utilize multiview footage where calibration is difficult, e.g., for pan-tilt or moving handheld cameras. We demonstrate the effectiveness of our approach on established benchmarks, as well as on a new Ski dataset with rotating cameras and expert ski motion, for which annotations are truly hard to obtain.