Random prime number generation is crucial for the implementation of several encryption and signature protocols in cryptographic applications. Recently, Quantum Random Number Generators (QRNGs) have emerged as a solution to obtain information-theoretically secure randomness, setting them apart from algorithmic generators. Nonetheless, their output is still prone to deterioration under a flawed implementation and thus requires continuous statistical validation. This process is uniquely challenging for prime number generators since they cannot be submitted to the traditional testing suites, which are primarily designed to test uniform pseudorandom sources. Here, we enable direct statistical testing of prime Random Number Generators (RNGs) by extending a validation framework based on an equiprobable binning of the prime distribution with a predictive machine learning model, which can independently learn correlations in the prime output. The validity of this framework is then verified through extensive cryptanalysis of a prime QRNG based on quadrature measurements of the vacuum state. The developed model successfully identifies flawed configurations of the prime QRNG for output lengths up to 128 bits. This assessment was additionally extended to a purely classical scheme derived from electronic noise measurements. Although no increase in the model’s predictive capacity is seen, our testing framework can nonetheless demonstrate that this classical source produces a conclusively biased prime distribution. In turn, the implemented QRNG remains resilient against this type of cryptanalysis.