Mathematical equations have been unreasonably effective in describing complex natural phenomena across various scientific disciplines. However, discovering such insightful equations from data presents significant challenges due to the necessity of navigating extremely large combinatorial hypothesis spaces. Current methods of equation discovery, commonly known as symbolic regression techniques, largely focus on extracting equations from data alone, often neglecting the domain-specific prior knowledge that scientists typically depend on. They also employ limited representations such as expression trees, constraining the search space and expressiveness of equations. To bridge this gap, we introduce LLM-SR, a novel approach that leverages the extensive scientific knowledge and robust code generation capabilities of Large Language Models (LLMs) to discover scientific equations from data. Specifically, LLM-SR treats equations as programs with mathematical operators and combines LLMs' scientific priors with evolutionary search over equation programs. The LLM iteratively proposes new equation skeleton hypotheses, drawing from its domain knowledge, which are then optimized against data to estimate parameters. We evaluate LLM-SR on four benchmark problems across diverse scientific domains (e.g., physics, biology), which we carefully designed to simulate the discovery process and prevent LLM recitation. Our results demonstrate that LLM-SR discovers physically accurate equations that significantly outperform state-of-the-art symbolic regression baselines, particularly in out-of-domain test settings. We also show that LLM-SR's incorporation of scientific priors enables more efficient equation space exploration than the baselines.

Scientific equation discovery is a fundamental task in the history of scientific progress, enabling the derivation of laws governing natural phenomena. Recently, Large Language Models (LLMs) have gained interest for this task due to their potential to leverage embedded scientific knowledge for hypothesis generation. However, evaluating the true discovery capabilities of these methods remains challenging, as existing benchmarks often rely on common equations that are susceptible to memorization by LLMs, leading to inflated performance metrics that do not reflect actual discovery. In this paper, we introduce LLM-SRBench, a comprehensive benchmark with 239 challenging problems across four scientific domains specifically designed to evaluate LLM-based scientific equation discovery methods while preventing trivial memorization. Our benchmark comprises two main categories: LSR-Transform, which transforms common physical models into less common mathematical representations to test reasoning beyond memorization, and LSR-Synth, which introduces synthetic, discovery-driven problems requiring data-driven reasoning. Through extensive evaluation of several state-of-the-art methods on LLM-SRBench, using both open and closed LLMs, we find that the best-performing system so far achieves only 31.5% symbolic accuracy. These findings highlight the challenges of scientific equation discovery, positioning LLM-SRBench as a valuable resource for future research.

In an era where symbolic mathematical equations are indispensable for modeling complex natural phenomena, scientific inquiry often involves collecting observations and translating them into mathematical expressions. Recently, deep learning has emerged as a powerful tool for extracting insights from data. However, existing models typically specialize in either numeric or symbolic domains, and are usually trained in a supervised manner tailored to specific tasks. This approach neglects the substantial benefits that could arise from a task-agnostic multi-modal understanding between symbolic equations and their numeric counterparts. To bridge the gap, we introduce SNIP, a Symbolic-Numeric Integrated Pre-training model, which employs contrastive learning between symbolic and numeric domains, enhancing their mutual similarities in the embeddings. By performing latent space analysis, we observe that SNIP provides cross-domain insights into the representations, revealing that symbolic supervision enhances the embeddings of numeric data and vice versa. We evaluate SNIP across diverse tasks, including symbolic-to-numeric mathematical property prediction and numeric-to-symbolic equation discovery, commonly known as symbolic regression. Results show that SNIP effectively transfers to various tasks, consistently outperforming fully supervised baselines and competing strongly with established task-specific methods, especially in the low data regime scenarios where available data is limited.

Symbolic regression (SR) is a challenging task in machine learning that involves finding a mathematical expression for a function based on its values. Recent advancements in SR have demonstrated the effectiveness of pre-trained transformer models in generating equations as sequences, leveraging large-scale pre-training on synthetic datasets and offering notable advantages in terms of inference time over classical Genetic Programming (GP) methods. However, these models primarily rely on supervised pre-training objectives borrowed from text generation and overlook equation discovery goals like accuracy and complexity. To address this, we propose TPSR, a Transformer-based Planning strategy for Symbolic Regression that incorporates Monte Carlo Tree Search planning algorithm into the transformer decoding process. Unlike conventional decoding strategies, TPSR enables the integration of non-differentiable equation verification feedback, such as fitting accuracy and complexity, as external sources of knowledge into the transformer equation generation process. Extensive experiments on various datasets show that our approach outperforms state-of-the-art methods, enhancing the model's fitting-complexity trade-off, extrapolation abilities, and robustness to noise.

Identification of nonlinear dynamical systems using data-driven frameworks facilitates the prediction and control of systems in a range of applications. Identification of a single system from the measurements of the system’s states leads to the discovery of explicit or implicit models that cannot generalize beyond the system for which the data are provided. By learning the effect of parameters in the system, we propose a generalizable model for the Identification of Parametric forms of dynamical systems using Integer Programming (IP2). We first build general libraries of basis functions that take into account both states and parameters. Subsequently, leveraging dimension analysis and the assumption of having integer coefficients in the equations, we show that our framework can identify the exact forms of parametric mechanical dynamical systems like an ideal pendulum or an inverted pendulum on a cart. Moreover, by applying object tracking techniques and taking advantage of a sequential filtering scheme, we can identify the state and energy equations of these dynamical systems from videos of the systems, i.e. pixel space noisy data, rather than state-space measurements. The results show that using integer programming makes the proposed framework significantly (more than 40 times in case of inverted pendulum on a cart) more robust to noise compared to previous optimization models.

The performance of optimization algorithms significantly depends on the landscape of the problems. It is known that there is no single algorithm that outperforms others on problems with different fitness landscapes. One of the issues in metaheuristic algorithms is keeping the balance between exploration and exploitation. The features extracted from analysis of fitness landscapes can be used to select the suitable algorithm for the given problem. However, these features are usually expensive and extracted prior to the optimization process which leads to a single algorithm to be selected. In this work, we propose an intelligent switch mechanism that enjoys an efficient non-convex ratio (ENCR) feature extracted online during the optimization to switch between two choices of algorithms, each favoring a type of landscape in terms of modality. For this work, two case studies including a pair of Harris hawks optimizer (HHO) and differential evolution (DE) and another pair of multiverse optimizer (MVO) and moth-flame optimizer (MFO) are selected among several algorithms to evaluate the performance of this framework. The proposed one-way and two-way switch algorithms take advantage of the merits of the two base algorithms to reach better final solutions and higher convergence rates in the majority of case studies. The overall comparison and ranking of the algorithms, including a random switch baseline, demonstrates the superiority of the intelligent switch mechanisms over the baselines.