Source localization has attracted increasing attention in recent years due to its vital role in governing the harmful propagation. However, existing localization methods do not fully consider the temporal characteristics in propagation and struggle to leverage the continuous-time information of real-world propagation scenarios. Moreover, the aggregation ability of GNN based localization models is limited by the structural noise commonly present in complicated real-world topologies. To address these challenges, a Spectral Neural Ordinary Differential Equation (SNODE) is proposed to infer the source in real-world continuous-time scenarios. First, the forward propagation is formulated as a flow based ODE system, and the source localization problem is transformed into an inverse ODE modeling task. Second, a neural process based on a graph variational autoencoder is introduced to encode global latent propagation patterns as a conditioning variable for the ODE system. Third, a spectral graph optimization is performed to suppress topological noise by filtering out high-frequency components that degrade the quality of graph aggregation in the neural process. Comprehensive experiments demonstrate that SNODE not only outperforms the optimal baseline in real-world continuous-time propagation scenarios with an average performance improvement of 43.8%, but also achieves consistently superior performance in synthetic discrete-time datasets with an improvement of 4.5%, highlighting its strong generalization ability in different propagation settings. Our code is available at https://github.com/cgao-comp/SNODE.