The problem of symbolic regression (SR) arises in many different applications, such as identifying physical laws or deriving mathematical equations describing the behavior of financial markets from given data. Various methods exist to address the problem of SR, often based on genetic programming. However, these methods are usually complicated and involve various hyperparameters. In this paper, we present our new approach ParFam that utilizes parametric families of suitable symbolic functions to translate the discrete symbolic regression problem into a continuous one, resulting in a more straightforward setup compared to current state-of-the-art methods. In combination with a global optimizer, this approach results in a highly effective method to tackle the problem of SR. We theoretically analyze the expressivity of ParFam and demonstrate its performance with extensive numerical experiments based on the common SR benchmark suit SRBench, showing that we achieve state-of-the-art results. Moreover, we present an extension incorporating a pre-trained transformer network (DL-ParFam) to guide ParFam, accelerating the optimization process by up to two magnitudes. Our code and results can be found at https://github.com/Philipp238/parfam.

Abstract Regularization is used in many different areas of optimization when solutions are sought which not only minimize a given function, but also possess a certain degree of regularity. Popular applications are image denoising, sparse regression and machine learning. Since the choice of the regularization parameter is crucial but often difficult, path-following methods are used to approximate the entire regularization path, i.e., the set of all possible solutions for all regularization parameters. Due to their nature, the development of these methods requires structural results about the regularization path. The goal of this article is to derive these results for the case of a smooth objective function which is penalized by a piecewise differentiable regularization term. We do this by treating regularization as a multiobjective optimization problem. Our results suggest that even in this general case, the regularization path is piecewise smooth. Moreover, our theory allows for a classification of the nonsmooth features that occur in between smooth parts. This is demonstrated in two applications, namely support-vector machines and exact penalty methods.

We present a novel algorithm that allows us to gain detailed insight into the effects of sparsity in linear and nonlinear optimization. Sparsity is of great importance in many scientific areas such as image and signal processing, medical imaging, compressed sensing, and machine learning, as it ensures robustness against noisy data and yields models that are easier to interpret due to the small number of relevant terms. It is common practice to enforce sparsity by adding the$\ell _1$-norm as a penalty term. In order to gain a better understanding and to allow for an informed model selection, we directly solve the corresponding multiobjective optimization problem (MOP) that arises when minimizing the main objective and the$\ell _1$-norm simultaneously. As this MOP is in general non-convex for nonlinear objectives, the penalty method will fail to provide all optimal compromises. To avoid this issue, we present a continuation method specifically tailored to MOPs with two objective functions one of which is the$\ell _1$-norm. Our method can be seen as a generalization of homotopy methods for linear regression problems to the nonlinear case. Several numerical examples – including neural network training – demonstrate our theoretical findings and the additional insight gained by this multiobjective approach.