The detection and localization of radiation sources using low-cost, mobile detectors is a challenging application, necessitating new research into sensing devices and detection algorithms. While new sensing that employs small detectors for detecting γ -rays has emerged, the decreased sensitivity of the sensor makes it challenging to maintain reliability compared to larger detectors. Machine learning could be a viable method for enhancing sensitivity by classifying background radiation spectra from anomalous spectra, but this approach can struggle to identify novel radioactive sources or identify sources in dynamic background environments. To address these challenges, we propose the use of adversarial auto encoders (AAEs) for anomaly detection in radiation sensing systems. With the use of our AAE architecture, we eliminate the need for obtaining examples of radiation anomalies for training data and increase the resilience of the sensing when the background radiation is dynamic. We evaluate the system in various contexts using a custom designed detector, showing the AAE model generalizes to various locations and radiation sources. We also show a real-time field test with the detection system in both handheld and drone mounted testing. • Our model improves detection of γ -ray anomalies on cesium iodide detectors. • These improvements extend to simulated sodium iodide detectors. • We demonstrate real-time detection capabilities in both lab and field environments. • These results are competitive with other unsupervised methods.