Interpretable machine learning is important for scientific, biomedical, and other high-stakes domains where transparency, trust, and human understanding of models are required. Recently, an evolutionary rule-based machine learning algorithm called HEROS was proposed, which separated rule from rule-set discovery each with distinct multi-objective optimization strategies yielding a two-phase, noise agnostic framework that found accurate, highly compact, and interpretable solutions on a diverse set of challenging benchmarks without requiring hyperparameter optimizations. However, HEROS is currently limited by its strictly sequential two-phase design which requires committing in advance to a fixed number of rule-discovery iterations. This work extends HEROS with a phase alternation scheme interleaving rule discovery and rule-set optimization. This enables refinement of both tasks throughout training in an 'anytime-algorithm' manner. We also introduce a tree-based rule initialization strategy to accelerate early-stage rule discovery. These extensions aim to preserve the empirical performance of the original HEROS framework under a similar computational budget while providing greater flexibility for algorithm application and problem scalability. Using the same diverse set of benchmark datasets, we evaluate and compare these HEROS extensions to the original HEROS algorithm and established rule-based learning systems including RIPPER and BioHEL. We demonstrate the advantages of this extended HEROS framework.

Survival analysis is widely utilized to model time-to-event data across biomedical, epidemiological, and engineering domains. However, traditional methods such as Cox regression impose strong assumptions, including proportional hazards, and are often inadequate for capturing complex, non-linear relationships in high-dimensional or heterogeneous data. Rule-based machine learning algorithms, such as "ExSTraCS," can interpretably model complex biomedical associations in classification tasks. This study extends ExSTraCS to the challenges of right-censored survival data to yield "Survival-LCS," the first rule-based survival analysis algorithm that additionally handles heterogeneous feature types and missing values and makes no assumptions about baseline hazard or survival time distributions. Survival-LCS is evaluated and compared across a variety of simulated genetic survival datasets covering distinct genetic architectures (additive, epistatic, heterogeneous, and univariate), censoring values, number of features, and survival distributions (random monotonic spline, Gamma, Gaussian, and Weibull) using Integrated Brier Scores and statistical significance testing. Results demonstrate that Survival-LCS reliably captures complex associations with survival outcomes, performing competitively with standard approaches, particularly in settings that challenge traditional models. This work highlights the capability of rule-based algorithms to be adapted as interpretable, assumption-free (i.e., data-driven) survival analysis methods, particularly in domains with higher-dimensional data or complex underlying associations.

Rule-based machine learning (RBML) algorithms, e.g. learning classifier systems (LCSs), can capture complex relationships while yielding more interpretable models than most other machine learning algorithms. Traditional LCSs rely on a single fitness function for both rule and/or rule-set optimization. However, ideal rule vs. rule-set discovery often requires distinct and multiple objectives. Recently, hybrid-LCSs were proposed that explicitly separated the task of rule vs. rule-set discovery but relied on distinct single-objective or weighted multi-objective fitness functions. This study introduces a newly developed Heuristic Evolutionary Rule Optimization System (HEROS) that combines previous LCS innovations aimed at tackling noisy, larger-scale, classification tasks, while adopting separation of rule vs. rule-set evolution. Uniquely, HEROS employs a custom Pareto-front-based multi-objective fitness function (for rule discovery) and NSGA-II-style multi-objective optimization (for rule-set discovery) to solve both clean and noisy-signal classification problems agnostically. Rule discovery is driven by rule-accuracy and instance coverage objectives, while rule-set discovery is driven by prediction accuracy and rule-set size objectives. Using diverse simulated benchmark datasets, i.e. noisy (GAMETES) and clean (MUX), we demonstrate proof-of-principle that HEROS can directly discover accurate, highly-compact, interpretable, and ideal solutions when compared to the established 'ExSTraCS' RBML algorithm, without objective weightings or adjusting hyperparameters.

We have recently presented SAFE-Solution And Fitness Evolution-a commensalistic coevolutionary algorithm that maintains two coevolving populations: a population of candidate solutions and a population of candidate objective functions. We showed that SAFE was successful at evolving solutions within a robotic maze domain. Herein we present an investigation of SAFE's adaptation and application to multiobjective problems, wherein candidate objective functions explore different weightings of each objective. Though preliminary, the results suggest that SAFE, and the concept of coevolving solutions and objective functions, can identify a similar set of optimal multiobjective solutions without explicitly employing a Pareto front for fitness calculation and parent selection. These findings support our hypothesis that the SAFE algorithm concept can not only solve complex problems, but can adapt to the challenge of problems with multiple objectives.

MOTIVATION: Relief is a family of machine learning algorithms that uses nearest-neighbors to select features whose association with an outcome may be due to epistasis or statistical interactions with other features in high-dimensional data. Relief-based estimators are non-parametric in the statistical sense that they do not have a parameterized model with an underlying probability distribution for the estimator, making it difficult to determine the statistical significance of Relief-based attribute estimates. Thus, a statistical inferential formalism is needed to avoid imposing arbitrary thresholds to select the most important features. We reconceptualize the Relief-based feature selection algorithm to create a new family of STatistical Inference Relief (STIR) estimators that retains the ability to identify interactions while incorporating sample variance of the nearest neighbor distances into the attribute importance estimation. This variance permits the calculation of statistical significance of features and adjustment for multiple testing of Relief-based scores. Specifically, we develop a pseudo t-test version of Relief-based algorithms for case-control data. RESULTS: We demonstrate the statistical power and control of type I error of the STIR family of feature selection methods on a panel of simulated data that exhibits properties reflected in real gene expression data, including main effects and network interaction effects. We compare the performance of STIR when the adaptive radius method is used as the nearest neighbor constructor with STIR when the fixed-k nearest neighbor constructor is used. We apply STIR to real RNA-Seq data from a study of major depressive disorder and discuss STIR's straightforward extension to genome-wide association studies. AVAILABILITY AND IMPLEMENTATION: Code and data available at http://insilico.utulsa.edu/software/STIR. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

The detection, modeling and characterization of complex patterns of association in bioinformatics has focused on feature interactions and, more recently, instance-subgroup specific associations (e.g. genetic heterogeneity). Previously, attribute tracking was proposed as an instance-linked memory approach, leveraging the incremental learning of learning classifier systems (LCSs) to track which features were most useful in making class predictions within individual instances. These 'attribute tracking' signatures could later be used to characterize patterns of association in the data. While effective, true underlying patterns remain difficult to characterize in noisy problems, and the original approach places equal weight on tracked feature scores obtained early as well as late in learning. In this work we investigate alternative strategies for attribute tracking scoring, including the adoption of a time recency update scheme taken from reinforcement learning, to gain insight into how to optimize this approach to improve modeling performance and downstream pattern interpretability. We report mixed results over a variety of performance metrics that point to promising future directions for building effective building blocks and improving model interpretability.

As the field of data science continues to grow, there will be an ever-increasing demand for tools that make machine learning accessible to non-experts. In this paper, we introduce the concept of tree-based pipeline optimization for automating one of the most tedious parts of machine learning--pipeline design. We implement an open source Tree-based Pipeline Optimization Tool (TPOT) in Python and demonstrate its effectiveness on a series of simulated and real-world benchmark data sets. In particular, we show that TPOT can design machine learning pipelines that provide a significant improvement over a basic machine learning analysis while requiring little to no input nor prior knowledge from the user. We also address the tendency for TPOT to design overly complex pipelines by integrating Pareto optimization, which produces compact pipelines without sacrificing classification accuracy. As such, this work represents an important step toward fully automating machine learning pipeline design.

An accuracy-based rule fitness is a hallmark of most modern Michigan-style learning classifier systems (LCS), a powerful, flexible, and largely interpretable class of machine learners. However, rule-fitness based solely on accuracy is not ideal for identifying 'optimal' rules in supervised learning. This is particularly true for noisy problem domains where perfect rule accuracy essentially guarantees over-fitting. Rule fitness based on accuracy alone is unreliable for reflecting the global 'value' of a given rule since rule accuracy is based on a subset of the training instances. While moderate over-fitting may not dramatically hinder LCS classification or prediction performance, the interpretability of the solution is likely to suffer. Additionally, over-fitting can impede algorithm learning efficiency and leads to a larger number of rules being required to capture relationships. The present study seeks to develop an intuitive multi-objective fitness function that will encourage the discovery, preservation, and identification of 'optimal' rules through accuracy, correct coverage of training data, and the prior probability of the specified attribute states and class expressed by a given rule. We demonstrate the advantages of our proposed fitness by implementing it into the ExSTraCS algorithm and performing evaluations over a large spectrum of complex, noisy, simulated datasets.

BACKGROUND AND OBJECTIVE: Detecting complex patterns of association between genetic or environmental risk factors and disease risk has become an important target for epidemiological research. In particular, strategies that provide multifactor interactions or heterogeneous patterns of association can offer new insights into association studies for which traditional analytic tools have had limited success. MATERIALS AND METHODS: To concurrently examine these phenomena, previous work has successfully considered the application of learning classifier systems (LCSs), a flexible class of evolutionary algorithms that distributes learned associations over a population of rules. Subsequent work dealt with the inherent problems of knowledge discovery and interpretation within these algorithms, allowing for the characterization of heterogeneous patterns of association. Whereas these previous advancements were evaluated using complex simulation studies, this study applied these collective works to a 'real-world' genetic epidemiology study of bladder cancer susceptibility. RESULTS AND DISCUSSION: We replicated the identification of previously characterized factors that modify bladder cancer risk--namely, single nucleotide polymorphisms from a DNA repair gene, and smoking. Furthermore, we identified potentially heterogeneous groups of subjects characterized by distinct patterns of association. Cox proportional hazard models comparing clinical outcome variables between the cases of the two largest groups yielded a significant, meaningful difference in survival time in years (survivorship). A marginally significant difference in recurrence time was also noted. These results support the hypothesis that an LCS approach can offer greater insight into complex patterns of association. CONCLUSIONS: This methodology appears to be well suited to the dissection of disease heterogeneity, a key component in the advancement of personalized medicine.

The application of learning classifier systems (LCSs) to classification and data mining in genetic association studies has been the target of previous work. Recent efforts have focused on: (1) correctly discriminating between predictive and non-predictive attributes, and (2) detecting and characterizing epistasis (attribute interaction) and heterogeneity. While the solutions evolved by Michigan-style LCSs (M-LCSs) are conceptually well suited to address these phenomena, the explicit characterization of heterogeneity remains a particular challenge. In this study we introduce attribute tracking, a mechanism akin to memory, for supervised learning in M-LCSs. Given a finite training set, a vector of accuracy scores is maintained for each instance in the data. Post-training, we apply these scores to characterize patterns of association in the dataset. Additionally we introduce attribute feedback to the mutation and crossover mechanisms, probabilistically directing rule generalization based on an instance's tracking scores. We find that attribute tracking combined with clustering and visualization facilitates the characterization of epistasis and heterogeneity while uniquely linking individual instances in the dataset to etiologically heterogeneous subgroups. Moreover, these analyses demonstrate that attribute feedback significantly improves test accuracy, efficient generalization, run time, and the power to discriminate between predictive and non-predictive attributes in the presence of heterogeneity.

Genetic epidemiologists, tasked with the disentanglement of genotype-to-phenotype mappings, continue to struggle with a variety of phenomena which obscure the underlying etiologies of common complex diseases. For genetic association studies, genetic heterogeneity (GH) and epistasis (gene-gene interactions) epitomize well recognized phenomenon which represent a difficult, but accessible challenge for computational biologists. While progress has been made addressing epistasis, methods for dealing with GH tend to "side-step" the problem, limited by a dependence on potentially arbitrary cutoffs/covariates, and a loss in power synonymous with data stratification. In the present study, we explore an alternative strategy (Learning Classifier Systems (LCSs)) as a direct approach for the characterization, and modeling of disease in the presence of both GH and epistasis. This evaluation involves (1) implementing standardized versions of existing Michigan-Style LCSs (XCS, MCS, and UCS), (2) examining major run parameters, and (3) performing quantitative and qualitative evaluations across a spectrum of simulated datasets. The results of this study highlight the strengths and weaknesses of the Michigan LCS architectures examined, providing proof of principle for the application of LCSs to the GH/epistasis problem, and laying the foundation for the development of an LCS algorithm specifically designed to address GH.

The study of common, complex multifactorial diseases in genetic epidemiology is complicated by nonlinearity in the genotype-to-phenotype mapping relationship that is due, in part, to epistasis or gene-gene interactions. Symobolic discriminant analysis (SDA) is a flexible modeling approach which uses genetic programming (GP) to evolve an optimal predictive model using a predefined collection of mathematical functions, constants, and attributes. This has been shown to be an effective strategy for modeling epistasis. In the present study, we introduce the genetic "mask" as a novel building block which exploits expert knowledge in the form of a pre-constructed relationship between two attributes. The goal of this study was to determine whether the availability of "mask" building blocks improves SDA performance. The results of this study support the idea that pre-processing data improves GP performance.