While modular structures offer great construction efficiency, scalability, safety, and reusability in engineering and architectural applications, their wide-spread adoption is hindered by the perceived material inefficiency and low design flexibility. Finding an optimal design within a modular system is a significant challenge, mostly because of associated computational complexity. Existing methods of accelerating combinatorial optimization with machine learning rely on heuristics and are often not transferrable between varying domain shapes, boundary conditions, and external loads. In this work, we present two key contributions to address this issue: (i) a deep neural network (DNN)-based surrogate model that accelerates the evaluation of mechanical responses by predicting reduced-order stiffness matrices , and (ii) a stochastic gradient optimization method that leverages the surrogate’s capability to compute sensitivities of the structure’s response to changes in module types. Our model combines convolutional layers with a physics-guided approach, ensuring that the output stiffness matrices are symmetric positive definite , consistent with the structure’s reduced-order representation via Schur’s complement. A distinguishing feature of our approach is its intrinsic independence from the specific domain shape, boundary conditions, and applied loads, allowing for broader applicability once the DNN-based surrogate is trained on a specific module set. We validate our method by optimizing multiple modular layout plans differing in size and loading conditions and demonstrate its efficacy by comparing its performance against the standard density-based topology optimization method. We achieve a computational speed-up of up to 1000x compared to the full-scale simulation, with a fast converging optimization for different domain sizes. This work lays the foundation for more flexible, efficient, and scalable modular design processes.