BACKGROUND: Identifying relevant biomarkers is critical in clinical research and precision medicine, particularly when analysing high-dimensional data. Random forests (RFs) are promising for such settings due to their flexibility, ease of use, and their ability to handle data sets with more variables than samples. RFs assess the importance of each variable in predicting the outcome using variable importance (VIMP) scores. However, since the distribution of VIMP scores is intricate, standard statistical testing and multiple testing adjustments for variable selection are challenging. METHODS: We propose shadowVIMP, a novel method for multiple testing-controlled variable selection, based on an approach similar to permutation testing. It generates permuted counterparts for each variable and compares their VIMPs with those of the original variables over multiple iterations to calculate p-values. Unlike conventional permutation testing, shadowVIMP preserves the correlation structure between variables, mitigating biases caused by the over-selection of correlated variables in RFs. We evaluated shadowVIMP against three competing RF variable selection approaches using simulation designs previously employed in studies considering VIMPs and variable selection for RFs. These designs included high- and low-dimensional data, as well as correlated and categorical variables. For illustration, we also applied the method to a real-world example on Alzheimer's disease. CONCLUSIONS: Our results showed that, compared to competing approaches, shadowVIMP offers advantages in high-dimensional settings, improving sensitivity while enabling multiple testing-adjusted results. Additionally, it demonstrated robustness against VIMP biases induced by correlated and categorical variables when using permutation-based VIMP. The method can be used to annotate standard VIMP plots, visually presenting selected variable sets based on different types of multiple testing adjustments and significance levels. Overall, shadowVIMP is a promising approach for providing multiple testing-adjusted variable selection while explicitly addressing known biases of RF's permutation-based VIMP measure. The shadowVIMP method is implemented in an R package shadowVIMP, which is available on CRAN.

BACKGROUND : Recent technological advances have enabled the simultaneous collection of multi-omics data, i.e., different types or modalities of molecular data. Integrative predictive modeling of such data is particularly challenging. Ideally, data from the different modalities are measured in the same individuals, allowing for early or intermediate integrative techniques. However, they are often not applicable when patient data only partially overlap, which requires either reducing the datasets or imputing missing values. Additionally, the diversity of data modalities may necessitate specific statistical methods rather than applying the same method across all modalities. Late integration modeling approaches analyze each data modality separately to obtain modality-specific predictions. These predictions are then aggregated into a meta-model by training a machine learning (ML) model, or by computing the weighted mean of modality-specific predictions. RESULTS : We introduce the R package fuseMLR for late integration prediction modeling. The package is user-friendly, enables variable selection and the application of different ML algorithms for each modality, and automatically performs aggregation once modality-specific training is completed. We illustrate the package’s functionality in a small simulation study and with two publicly available multi-omics datasets from The Cancer Genome Atlas. CONCLUSION : The package fuseMLR enables predictive modeling with late integration in a systematic, structured, and reproducible way.

In longitudinal studies variables are measured repeatedly over time, leading to clustered and correlated observations. If the goal of the study is to develop prediction models, machine learning approaches such as the powerful random forest (RF) are often promising alternatives to standard statistical methods, especially in the context of high-dimensional data. In this paper, we review extensions of the standard RF method for the purpose of longitudinal data analysis. Extension methods are categorized according to the data structures for which they are designed. We consider both univariate and multivariate response longitudinal data and further categorize the repeated measurements according to whether the time effect is relevant. Even though most extensions are proposed for low-dimensional data, some can be applied to high-dimensional data. Information of available software implementations of the reviewed extensions is also given. We conclude with discussions on the limitations of our review and some future research directions.

MOTIVATION: High-throughput technologies allow comprehensive characterization of individuals on many molecular levels. However, training computational models to predict disease status based on omics data is challenging. A promising solution is the integration of external knowledge about structural and functional relationships into the modeling process. We compared four published random forest-based approaches using two simulation studies and nine experimental datasets. RESULTS: The self-sufficient prediction error approach should be applied when large numbers of relevant pathways are expected. The competing methods hunting and learner of functional enrichment should be used when low numbers of relevant pathways are expected or the most strongly associated pathways are of interest. The hybrid approach synthetic features is not recommended because of its high false discovery rate. AVAILABILITY AND IMPLEMENTATION: An R package providing functions for data analysis and simulation is available at GitHub (https://github.com/szymczak-lab/PathwayGuidedRF). An accompanying R data package (https://github.com/szymczak-lab/DataPathwayGuidedRF) stores the processed and quality controlled experimental datasets downloaded from Gene Expression Omnibus (GEO). SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Machine learning methods and in particular random forests are promising approaches for prediction based on high dimensional omics data sets. They provide variable importance measures to rank predictors according to their predictive power. If building a prediction model is the main goal of a study, often a minimal set of variables with good prediction performance is selected. However, if the objective is the identification of involved variables to find active networks and pathways, approaches that aim to select all relevant variables should be preferred. We evaluated several variable selection procedures based on simulated data as well as publicly available experimental methylation and gene expression data. Our comparison included the Boruta algorithm, the Vita method, recurrent relative variable importance, a permutation approach and its parametric variant (Altmann) as well as recursive feature elimination (RFE). In our simulation studies, Boruta was the most powerful approach, followed closely by the Vita method. Both approaches demonstrated similar stability in variable selection, while Vita was the most robust approach under a pure null model without any predictor variables related to the outcome. In the analysis of the different experimental data sets, Vita demonstrated slightly better stability in variable selection and was less computationally intensive than Boruta. In conclusion, we recommend the Boruta and Vita approaches for the analysis of high-dimensional data sets. Vita is considerably faster than Boruta and thus more suitable for large data sets, but only Boruta can also be applied in low-dimensional settings.

MOTIVATION: It has been shown that the machine learning approach random forest can be successfully applied to omics data, such as gene expression data, for classification or regression and to select variables that are important for prediction. However, the complex relationships between predictor variables, in particular between causal predictor variables, make the interpretation of currently applied variable selection techniques difficult. RESULTS: Here we propose a new variable selection approach called surrogate minimal depth (SMD) that incorporates surrogate variables into the concept of minimal depth (MD) variable importance. Applying SMD, we show that simulated correlation patterns can be reconstructed and that the increased consideration of variable relationships improves variable selection. When compared with existing state-of-the-art methods and MD, SMD has higher empirical power to identify causal variables while the resulting variable lists are equally stable. In conclusion, SMD is a promising approach to get more insight into the complex interplay of predictor variables and outcome in a high-dimensional data setting. AVAILABILITY AND IMPLEMENTATION: https://github.com/StephanSeifert/SurrogateMinimalDepth. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.