The accuracy of Machine Learning (ML) models is highly dependent on the hyperparameters that have to be chosen by the user before the training. However, finding the optimal set of hyperparameters is a complex process, as many different parameter combinations need to be evaluated, and obtaining the accuracy of each combination usually requires a full training run. It is therefore of great interest to reduce the computational runtime of this process. On High-Performance Computing (HPC) systems, several configurations can be evaluated in parallel to speed up this Hyperparameter Optimization (HPO). State-of-the-art HPO methods follow a bandit-based approach and build on top of successive halving, where the final performance of a combination is estimated based on a lower than fully trained fidelity performance metric and more promising combinations are assigned more resources over time. Frequently, the number of epochs is treated as a resource, letting more promising combinations train longer. Another option is to use the number of workers as a resource and directly allocate more workers to more promising configurations via data-parallel training. This article proposes a novel Resource-Adaptive Successive Doubling Algorithm (RASDA), which combines a resource-adaptive successive doubling scheme with the plain Asynchronous Successive Halving Algorithm (ASHA). Scalability of this approach is shown on up to 1,024 Graphics Processing Units (GPUs) on modern HPC systems. It is applied to different types of Neural Networks (NNs) and trained on large datasets from the Computer Vision (CV), Computational Fluid Dynamics (CFD), and Additive Manufacturing (AM) domains, where performing more than one full training run is usually infeasible. Empirical results show that RASDA outperforms ASHA by a factor of up to 1.9 with respect to the runtime. At the same time, the solution quality of final ASHA models is maintained or even surpassed by the implicit batch size scheduling of RASDA. With RASDA, systematic HPO is applied to a terabyte-scale scientific dataset for the first time in the literature, enabling efficient optimization of complex models on massive scientific data.

Data will be made available on request.

The application of Artificial Neural Networks (ANNs) has been extensively investigated for fluid dynamic problems. A specific form of ANNs are Physics-Informed Neural Networks (PINNs). They incorporate physical laws in the training and have increasingly been explored in the last few years. In this work, the prediction accuracy of PINNs is compared with that of conventional Deep Neural Networks (DNNs). The accuracy of a DNN depends on the amount of data provided for training. The change in prediction accuracy of PINNs and DNNs is assessed using a varying amount of training data. To ensure the correctness of the training data, they are obtained from analytical and numerical solutions of classical problems in fluid mechanics. The objective of this work is to quantify the fraction of training data relative to the maximum number of data points available in the computational domain, such that the accuracy gained with PINNs justifies the increased computational cost. Furthermore, the effects of the location of sampling points in the computational domain and noise in training data are analyzed. In the considered problems, it is found that PINNs outperform DNNs when the sampling points are positioned in the Regions of Interest. PINNs for predicting potential flow around a Rankine oval have shown a better robustness against noise in training data compared to DNNs. Both models show higher prediction accuracy when sampling points are randomly positioned in the flow domain as compared to a prescribed distribution of sampling points. The findings reveal new insights on the strategies to massively improve the prediction capabilities of PINNs with respect to DNNs.