The ever-increasing demand for data rates by mobile users can be fulfilled by radio over fiber (RoF) combined with spectrally-efficient modulation techniques. Thus, RoF is critical to the design of the fronthaul for 5thgeneration (5G) mobile communication networks. We propose and experimentally demonstrate a low-cost directly modulated laser (DML)-based wavelength division multiplexing (WDM)-RoF transmission system for use in next-generation 5G networks. The proposed architecture provides flexible resource allocation as well, under low and high traffic loads at the remote radio heads (RRHs). We also present robust analytical models of the proposed DML-based WDM-RoF network with spectrally-efficient M-ary-quadrature amplitude modulation ( M-QAM), beneficial for system designers. In OptiSystem software, we evaluate the error vector magnitudes (EVM) as a function of the optical link lengths, input optical power, and the received optical power with 4/16/64-QAM over the optical link. The EVMs of the link for 4/16/64-QAM are below the 3GPP standards for 5G. We find good agreement between the simulations and the analytical results. We also experimentally demonstrate a small scale model of this network with resource allocation. Such a flexible architecture is suitable for meeting the demands of femtocells in 5G fronthaul networks.