This paper presents an evolvable conditional diffusion method such that black-box, non-differentiable multi-physics models, as are common in domains like computational fluid dynamics and electromagnetics, can be effectively used for guiding the generative process to facilitate autonomous scientific discovery. We formulate the guidance as an optimization problem where one optimizes for a desired fitness function through updates to the descriptive statistic for the denoising distribution, and derive an evolution-guided approach from first principles through the lens of probabilistic evolution. Interestingly, the final derived update algorithm is analogous to the update as per common gradient-based guided diffusion models, but without ever having to compute any derivatives. We validate our proposed evolvable diffusion algorithm in two AI for Science scenarios: the automated design of fluidic topology and meta-surface. Results demonstrate that this method effectively generates designs that better satisfy specific optimization objectives without reliance on differentiable proxies, providing an effective means of guidance-based diffusion that can capitalize on the wealth of black-box, non-differentiable multi-physics numerical models common across Science.

Recent rapid advances in artificial intelligence methods and computational resources have opened up the possibility of solving complex problems in physics with machine learning. While promising, the training of performant deep learning models remains challenging, especially as the choice of various neural architectures or model hyper-parameters can all greatly impact model performance. Hence, it is now fairly common for researchers to systematically conduct either a neural architecture search (NAS) or hyper-parameter tuning as part of the model training process. However, these usually involve repeated model training and evaluation (typically via a cross-validation framework) and can be very computationally expensive. While there have been training-free heuristics proposed for NAS in recent years, this remains an active area of work, and much remains unresolved. Concurrently, physics-informed neural networks (PINNs) have been proposed in recent years as an attractive means of incorporating physics-derived knowledge into a deep learning framework for the modelling of real-world physical systems. However, recent literature has begun to emerge on the additional complexity of training PINNs due to the addition of these physics-derived equations (typically in the form of partial differential equations, PDEs). The extended optimization trajectories observed further exacerbate the aforementioned computational cost of NAS or hyper-parameter search. Hence, in this work, we propose a first demonstration of the utility of training-free heuristics for accelerating the selection of model parameters. In particular, we assess the correlation between basic concepts espoused in Nyquist-Shannon sampling, and the appropriate selection of mini-batch size for PINN training with stochastic gradient descent (specifically, the state-of-the-art ADAM algorithm). As a proof-of-concept, this paper focuses on training PINNs to predict the solution to a canonical 2D Helmholtz equation with varying complexity (as measured by the frequency of the signal present), and we evaluate and report the influence of different mini-batch size parameters on the effective training of PINNs. The results show that PINN model performance exhibits a step-like phase transition with increased mini-batch size during training. Critically, we note that the minimum mini-batch size used increases proportionally with the presence of higher frequencies in the physical system being modelled, as is intuitively described by the Nyquist-Shannon sampling theorem. This both shows the utility of similar potential heuristics for the training-free selection of hyper-parameters when training PINNs and suggests the potential for insights towards accelerated design and training of PINNs for physical systems from prior theoretical concepts in signal processing and reconstruction.

We present a novel loss formulation for efficient learning of complex dynamics from governing physics, typically described by partial differential equations (PDEs), using physics-informed neural networks (PINNs). In our experiments, existing versions of PINNs are seen to learn poorly in many problems, especially for complex geometries, as it becomes increasingly difficult to establish appropriate sampling strategy at the near boundary region. Overly dense sampling can adversely impede training convergence if the local gradient behaviors are too complex to be adequately modelled by PINNs. On the other hand, if the samples are too sparse, existing PINNs tend to overfit the near boundary region, leading to incorrect solution. To prevent such issues, we propose a new Boundary Connectivity (BCXN) loss function which provides linear local structure approximation (LSA) to the gradient behaviors at the boundary for PINN. Our BCXN-loss implicitly imposes local structure during training, thus facilitating fast physics-informed learning across entire prob-lem domains with order of magnitude sparser training samples. This LSA-PINN method shows a few orders of magnitude smaller errors than existing methods in terms of the standard L2-norm metric, while using dramatically fewer training samples and iterations. Our proposed LSA-PINN does not pose any requirement on the differentiable property of the networks, and we demonstrate its benefits and ease of implementation on both multi-layer perceptron and convolutional neural network versions as commonly used in current PINN literature.