Biometric recognition systems typically use a fixed threshold to differentiate between legitimate users and imposters. Yet, this method can be problematic due to differences in individual user performance, whereas some users are more easily recognizable than others. Furthermore, fixed thresholds require extensive tuning on a large test set a priori to determine an optimal threshold value. Adaptive thresholds address these shortcomings by adjusting threshold values based on population characteristics. However, our research demonstrates that adaptive thresholds suffer from a significant weakness as they inadvertently increase the attack surface against face recognition systems. We do so by introducing a novel attack, the doppelgänger attack, where a malicious actor inserts adversarial examples that mimic legitimate users and increase the false rejection rate for these legitimate users by 70%.