Quantum networks (QNs) enable secure distributed quantum computing and sensing over next-generation optical communication networks by distributing entangled states over optical channels. However, quantum switches (QSs) in such QNs, which perform entanglement distribution, have limited resources, e.g., single-photon sources (SPSs) and quantum memories, which are sensitive to noise and losses. Efficient QS resource allocation is needed to minimize entanglement distribution delay. This paper proposes a QS resource allocation framework that jointly optimizes the average entanglement distribution delay and entanglement distillation operations to improve end-to-end (e2e) fidelity and meet user-specific rate and fidelity requirements. The proposed framework accounts for realistic QN noise and imperfections, deriving analytical expressions for quantum memory decoherence noise and resulting e2e fidelity after distillation. It also considers practical deployment factors, allowing QSs to control 1) nitrogen-vacancy (NV) center SPS types based on their isotopic decomposition, and 2) nuclear spin regions based on coupling strength and distance from NV center’s electron spin. The QS resource allocation optimization problem is solved using a simulated annealing algorithm. Simulation results show that the proposed framework manages to satisfy all users rate and fidelity requirements, unlike existing distillation-agnostic, minimal distillation, and physics-agnostic frameworks which do not perform distillation, perform minimal distillation, and do not control the physics-based NV center characteristics, respectively. Furthermore, the proposed framework results in significant reductions in the average e2e entanglement distribution delay, along with enhancements in the average e2e fidelity compared to the aforementioned existing frameworks.