Obtaining human-readable symbolic formulas via genetic programming-based symbolic distillation of a deep neural network trained on the target dataset presents a promising yet underexplored path towards explainable artificial intelligence (XAI); however, the standard pipeline frequently yields symbolic models with poor predictive accuracy. We identify a fundamental misalignment in functional complexity as the primary barrier to achieving better accuracy: standard Artificial Neural Networks (ANNs) often learn accurate but highly irregular functions, while Symbolic Regression typically prioritizes parsimony, often resulting in a much simpler class of models that are unable to sufficiently distill or learn from the ANN teacher. To bridge this gap, we propose a framework that actively regularizes the teacher's functional smoothness using Jacobian and Lipschitz penalties, aiming to distill better student models than the standard pipeline. We characterize the trade-off between predictive accuracy and functional complexity through a robust study involving 20 datasets and 50 independent trials. Our results demonstrate that students distilled from smoothness-regularized teachers achieve statistically significant improvements in R^2 scores, compared to the standard pipeline. We also perform ablation studies on the student model algorithm. Our findings suggest that smoothness alignment between teacher and student models is a critical factor for symbolic distillation.

Symbolic regression (SR) algorithms learn, from a training set, a concise closed-form function that relates the predictors (features) to the outcomes (labels). Many successful SR approaches, including those based on genetic programming and other evolutionary techniques, rely on stochastic search over a large hypothesis space. While this stochasticity often produces strong performance, it can also lead to sensitivity to perturbations in the training data, resulting in variability across learned models. Understanding and characterizing this behavior is important for assessing reliability and generalization. This paper studies the theoretical relationship between algorithmic stability and generalization in SR. We present the first stability-based theoretical analysis of the generalization gap for SR algorithms. We introduce two stability metrics, the add-one point-wise gap and the add-one training gap, which can be estimated empirically. Using these metrics, we derive a bound on the generalization gap and show how it relates to generalization theory. We provide empirical evidence across multiple SR algorithms, hyperparameter settings, and datasets demonstrating that the proposed stability metrics track and explain generalization behavior. Algorithms and configurations that exhibit greater stability tend to achieve smaller generalization gaps. Finally, we conduct ablation studies to validate the robustness and explanatory power of the proposed metrics.

Genetic Programming Symbolic Regression (GPSR) generates mathematical expressions to model input-output relationships using an evolutionary process. A significant challenge in GPSR lies in the repeated evaluation of entire expressions or their sub-expression, which inflates computational runtime. To address this inefficiency, caching mechanisms have been employed to reduce redundant computations. However, prior studies predominantly employ a single caching strategy, offering limited insights into their comparative performance or memory-runtime trade-offs. In this paper, we present a comprehensive analysis of caching mechanisms for GPSR on synthetic and real-world datasets. We also include an empirical study of key-value usage frequencies under an infinitely large cache, offering insights into optimal cache sizing. Furthermore, we provide actionable guidelines for configuring caching strategies based on computational and memory constraints. Our findings indicate that complex caching mechanisms necessitate a minimum cache size to achieve computational time reductions. Conversely, lightweight caching strategies, such as Least Recently Used (LRU) and, notably, First-In-First-Out (FIFO), can significantly decrease computation time for fitness evaluations, which are a substantial component of the overall runtime.

Symbolic Regression (SR) algorithms select expressions based on prediction performance while also keeping the expression lengths short to produce explainable white box models. In this context, SR algorithms can be evaluated by measuring the extent to which the expressions discovered are Pareto-optimal, in the sense of having the best R-squared score for a given expression length. This evaluation is most commonly done based on relative performance, in the sense that an SR algorithm is judged on whether it Pareto-dominates other SR algorithms selected in the analysis, without any indication on efficiency or attainable limits. In this paper, we explore absolute Pareto-optimal (APO) solutions instead, which have the optimal tradeoff between the multiple SR objectives, for 34 datasets in the widely-used SR benchmark, SRBench, by performing exhaustive search. Additionally, we include comparisons between eight numerical optimization methods. We extract, for every dataset, an APO front of expressions that can serve as a universal baseline for SR algorithms that informs researchers of the best attainable performance for selected sizes. The APO fronts provided serves as an important benchmark and performance limit for SR algorithms and is made publicly available at: https://github.com/kentridgeai/SRParetoFronts

In this work, we propose a novel approach that combines the strengths of FEAT and TabNet through knowledge distillation (KD), which we term FEAT-KD. FEAT is an intrinsically interpretable machine learning (ML) algorithm that constructs a weighted linear combination of concisely-represented features discovered via genetic programming optimization, which can often be inefficient. FEAT-KD leverages TabNet’s deep-learning-based optimization and feature selection mechanisms instead. FEAT-KD finds a weighted linear combination of concisely-represented, symbolic features that are derived from piece-wise distillation of a trained TabNet model. We analyze FEAT-KD on regression tasks from two perspectives: (i) compared to TabNet, FEAT-KD significantly reduces model complexity while retaining competitive predictive performance, effectively converting a black-box deep learning model into a more interpretable white-box representation, (ii) compared to FEAT, our method consistently outperforms in prediction accuracy, produces more compact models, and reduces the complexity of learned symbolic expressions. In addition, we demonstrate that FEAT-KD easily supports multi-target regression, in which the shared features contribute to the interpretability of the system. Our results suggest that FEAT-KD is a promising direction for interpretable ML, bridging the gap between deep learning’s predictive power and the intrinsic transparency of symbolic models.

Symbolic Regression (SR) algorithms are powerful machine learning tools that discover mathematical expressions from data, but their evaluation is often hindered by the computational cost of optimizing numerical parameters (also called constants) within candidate expressions. To address this limitation, we introduce POVE (PreOptimized Vault of Expressions), a comprehensive repository of preoptimized symbolic expressions where numerical parameters have been efficiently computed using an established optimization technique in SR (i.e., BFGS) on widely used SR regression problems (i.e., SRBench). In addition to optimized constants, POVE stores the corresponding fitness values, quantifying the accuracy of each expression on its respective regression problem, allowing researchers to directly retrieve both optimized expressions and their precomputed performance metrics from memory (e.g., hash tables or dictionaries). By eliminating the need for computationally expensive numerical optimization steps, POVE enables significantly faster evaluation, large-scale benchmarking, and extensive hyperparameter analysis. It provides a structured and standardized foundation for comparing SR algorithms, assessing their consistency across large numbers of random seeds and investigating the impact of hyperparameters without the overhead of numerical parameter optimization. By offering a collection of precomputed expressions, parameters, and fitness values across diverse regression problems, POVE enhances reproducibility, accelerates experimental workflows, and enables scalable SR research. It serves as a valuable resource for both developing novel SR methodologies and improving existing algorithms by providing a reliable reference for optimal expression discovery. With POVE, SR research can shift away from costly optimization, enabling deeper algorithmic exploration and more comprehensive systematic evaluation. POVE also lowers the barrier to entry for SR research and makes large-scale evaluation significantly more accessible. POVE is publicly available at: https://github.com/kentridgeai/POVE

We introduce a conceptually simple yet effective method to create small, compact decision trees - by using splits found via Symbolic Regression (SR). Traditional decision tree (DT) algorithms partition a dataset on axis-parallel splits. When the true boundaries are not along the feature axes, DT is likely to have a complicated structure and a dense decision boundary. In this paper, we introduce SR-Enhanced DT (SREDT) - a method which utilizes SR to increase the richness of the class of possible DT splits. We evaluate SREDT on both synthetic and real-world datasets. Despite its simplicity, our method produces surprisingly small trees that outperform both DT and oblique DT (ODT) on supervised classification tasks in terms of accuracy and F-score. We show empirically that SREDTs decrease inference time (compared to DT and ODT) and argue that they allow us to obtain more explainable descriptions of the decision process. SREDT also performs competitively against state-of-the-art tabular classification methods, including tree ensembles and deep models. Finally, we introduce a local search mechanism to improve SREDT and evaluate it on 56 PMLB datasets. This mechanism shows improved performance on 77.2% of the datasets, outperforming DT and ODT. In terms of F-Score, local SREDT outperforms DT and ODT in 82.5% and 73.7% of the datasets respectively and in terms of inference time, local SREDT requires 25.8% and 26.6% less inference time than DT and ODT respectively.

Symbolic Regression (SR) is an approach which learns a closed-form function relating the predictors to the outcome in a dataset. Datasets are often multi-level (MuL), meaning that certain features can be used to split data into groups for analysis (we refer to these features as levels). The advantage of viewing datasets as MuL is that we can exploit the high similarity of data within a group. SR is well-suited for MuL datasets, in which the learnt function structure serves as ‘shared information’ between the groups while the learnt parameter values capture the unique relationships within each group. In this context, this paper makes three contributions: (i) We design an algorithm, Multi-level Symbolic Regression (MSR), which runs multiple parallel SR processes for each group and merges them to produce a single function structure. (ii) To tackle datasets that are not explicitly MuL, we develop a metric termed MLICC to select the best feature to serve as a level. (iii) We also release MSRBench, a database of MuL datasets (synthetic and real-world) which we developed and collated, that can be used to evaluate MSR. Our results and ablation studies demonstrate that MSR achieves a higher recovery rate and lower error on MSRBench compared to SOTA methods for SR and MuL datasets.

State-of-the-art Symbolic Regression (SR) algorithms employ evolutionary techniques to fulfill the task of generating a concise mathematical expression that fulfills an objective. A common objective is to fit to a dataset of input-output pairs, in which the faithfulness of a predicted output to the actual output is used as the fitness measure (e.g., R-squared). In many datasets, among the candidate expressions evaluated, there tends to be a large number of pseudo-expressions, referring to expressions that achieve high fitness but do not resemble the ground-truth equation. These pseudo-expressions decrease the equation recovery rate of SR algorithms. To formulate novel fitness measures that function as better discriminators of the ground-truth equation, we introduce a novel meta-learning approach to SR, MetaSR, in which we utilize SR itself to discover new fitness measures that can be complex combinations of existing base measures. In this paper, we focus on frequency-aware symbolic regression, where the fitness can depend on the frequency domain. We show that our new fitness measures better discriminate the ground-truth equation from other equations and demonstrate the improved performance of our method against existing algorithms.

Symbolic Regression (SR) is the task of finding a concise white-box mathematical expression that fulfills a given machine learning (ML) objective. In this work, we introduce the first-of-its-kind SR-based ensemble combiner (SyREC) for ML which utilizes the quasi-arithmetic mean (parameterized by a function$f$) as an ensemble combiner and discovers$f$via SR. Our Sy REC demonstrates advantages over existing ensemble combiners, i.e., averaging methods and meta-learners. Compared to averaging methods, Sy REC allows for increased combiner complexity via exploring a large function class and provides a learning process to tune the ensemble combiner to the specific dataset. Compared to meta-learners, SyREC preserves important ensemble properties, like idempotency, monotonicity and boundedness, which produce predictions more consistent with the base models. We note SyREC produces a white-box ensemble combiner that lies in the intersection of both categories of existing combiners, addressing the weaknesses while capturing the strengths of each category. To evaluate Sy REC, we present experiments on 16 open-source PMLB datasets and commonly used base models, including a large variety of experiments and analyses across different settings: (i) regression and classification tasks, (ii) equally weighted combiners and custom-weighted combiners, (iii) homogeneous and heterogeneous ensembles, (iv) sequential and parallel ensembles. We find that in all these settings, SyREC shows consistent improvement over existing ensembling approaches. This indicates that SyREC is a robust ensemble combiner, which produces explainable and consistent ensemble predictions.