For researchers studying cyber-physical system security, working with realistic datasets is essential. To produce the datasets, the existing methodology is to emulate the cyber network. A challenge is that the industrial control systems (ICS) network consists of not just computers and communication equipment, but also field devices that collect data and execute controls. These devices play a significant role in the operation and the security of the system. However, in comparison to the cyber network, the research reproducibility and realism of the cyber-physical system emulation and its data has received far less attention. This paper thus develops an approach to answer, ”How well can emulated devices replicate the behavior of physical intelligent electronics devices (IEDs) in a realistic cyber attack and defense environment?” To study this, we perform a comparison study based on an emulation experiment using theminimegatestbed environment that is entirely virtual and a hardware-in-the-loop experiment using the Resilient Energy Systems Lab (RESLab) cyber-physical testbed featuring real industrial controllers and communications devices. Results show that under different reconnaissance attack scenarios,RESLabgenerates realistic datasets that validate the emulation-based cybersecurity model inminimega. The approach is generalizable toward validating the realism of other types of ICS devices in security studies.

Short-term forecasting of residential and commercial building energy consumption is widely used in power systems and continues to grow in importance. Data-driven short-term load forecasting (STLF), although promising, has suffered from a lack of open, large-scale datasets with high building diversity. This has hindered exploring the pretrain-then-fine-tune paradigm for STLF. To help address this, we present BuildingsBench, which consists of: 1) Buildings-900K, a large-scale dataset of 900K simulated buildings representing the U.S. building stock; and 2) an evaluation platform with over 1,900 real residential and commercial buildings from 7 open datasets. BuildingsBench benchmarks two under-explored tasks: zero-shot STLF, where a pretrained model is evaluated on unseen buildings without fine-tuning, and transfer learning, where a pretrained model is fine-tuned on a target building. The main finding of our benchmark analysis is that synthetically pretrained models generalize surprisingly well to real commercial buildings. An exploration of the effect of increasing dataset size and diversity on zero-shot commercial building performance reveals a power-law with diminishing returns. We also show that fine-tuning pretrained models on real commercial and residential buildings improves performance for a majority of target buildings. We hope that BuildingsBench encourages and facilitates future research on generalizable STLF. All datasets and code can be accessed from https://github.com/NREL/BuildingsBench.