We present a reinforcement learning framework for constructing assemblies composed of rigid parts, which are commonly seen in many historical masonry buildings and bridges. Traditional construction methods for such structures often depend on dense scaffolding to stabilize their intermediate assembly steps, making the process both labor-intensive and time-consuming. This work utilizes multiple robots to collaboratively assemble structures, offering temporary support by holding parts in place without additional scaffolding. Precomputing the robotic assembly process to ensure structural stability involves a time-consuming offline process due to the combinatorial nature of its search space. However, the precomputed assembly plans may get disrupted during real-world execution due to unforeseen changes, such as setup modifications or delays in part delivery. Recomputing these plans using traditional offline methods results in significant project delays. Therefore, we propose a reinforcement learning-based approach in which a neural network is trained to efficiently generate alternative assembly plans for a given structure online, enabling adaptation to external changes. To enable effective and efficient training, we introduce three key innovations: a GPU-based stability simulator for parallelizing simulations, a novel curriculum-based training scheme to address sparse rewards during training, and a new graph neural network architecture for efficiently encoding assembly geometry. We validate our approach by training reinforcement learning agents on various assemblies and evaluating their performance on unseen assembly tasks. Furthermore, we demonstrate the effectiveness of our framework in planning multi-robot assembly processes, effectively handling disruptions in both simulation and physical environments.