Standardized benchmarks like RobustBench are crucial for evaluating adversarial robustness. However, the increasing dominance of models trained on massive synthetic datasets (orders of magnitude larger than original training sets) raises questions about reported performance gains. This work identifies and investigates a potential inflation factor: high feature-level similarity between large-scale synthetic training data and benchmark test sets. We argue this similarity is an inherent characteristic arising from the probabilistic generation process of these large datasets, which naturally produces examples highly similar to test instances in feature space. This creates what we term an “Implicit Information Advantage,” where models effectively train on near-duplicates of test instances. Through comprehensive empirical analysis, we demonstrate that: (1) Synthetic datasets exhibit significantly higher similarity to the test set compared to the original training data. (2) A direct correlation exists between this similarity and robustness outcomes, with test images benefiting most having the highest similarity scores. (3) Strikingly, ablation studies show that training on just a small fraction (e.g., 1%) of the most similar synthetic examples can yield robustness comparable to using the full massive dataset. These findings suggest current benchmarks may overestimate true robust generalization due to this similarity artifact. We call for revised evaluation protocols and greater transparency to ensure benchmarks accurately measure true generalization. Code and data can be found in https://github.com/fzjcdt/RethinkingRobustBench.

Designing advanced neural architectures to tackle specific tasks involves weeks or even months of intensive investigation by experts with rich domain knowledge. In recent years, neural architecture search (NAS) has attracted the interest of many researchers due to its ability to automatically design efficient neural architectures. Among different search strategies, evolutionary algorithms have achieved significant successes as derivative-free optimization algorithms. However, the tremendous computational resource consumption of the evolutionary neural architecture search dramatically restricts its application. In this paper, we explore how fitness approximation-based evolutionary algorithms can be applied to neural architecture search and propose NAS-EA-FA to accelerate the search process. We further exploit data augmentation and diversity of neural architectures to enhance the algorithm, and present NAS-EA-FA V2. Experiments show that NAS-EA-FA V2 is at least five times faster than other state-of-the-art neural architecture search algorithms like regularized evolution and iterative neural predictor on NASBench-101, and it is also the most effective and stable algorithm on NASBench-201. All the code used in this paper is available at https://github.com/fzjcdt/NAS-EA-FA.