Apache Spark's performance is critically dependent on efficient in-memory computation; however, insufficient executor memory frequently results in costly disk spills, severely degrading performance. Traditional spill mitigation techniques, including static configuration tuning, reactive Adaptive Query Execution (AQE), and internal spill handling mechanisms, exhibit limitations in addressing the dynamic and fine-grained nature of memory pressure in complex and large-scale workloads. This paper introduces Proactive Spill Mitigation (PSM), a novel methodology designed to anticipate and mitigate memory spills in PySpark applications before they occur. PSM leverages real-time task and executor metrics, accessible via the Spark Listener API, as input to a machine learning model trained to predict the likelihood and potential severity of impending spills. Upon detecting a high spill risk, a control mechanism triggers preemptive actions, such as checkpointing intermediate results or dynamically adjusting partition counts, to alleviate memory pressure. Experimental evaluation using the TPC-DS and HiBench Sort benchmarks demonstrates that PSM significantly reduces both execution time and the volume of data spilled to disk across various cluster sizes, outperforming default Spark configurations, statically tuned setups, and configurations with AQE enabled. This work establishes the efficacy of predictive control for enhancing Spark's memory management and improving application performance in dynamic environments.

Depression is a widespread mental health issue with profound global impact, often leading to diminished life quality and increased suicide risk. Despite available treatments, many depression cases go unnoticed and untreated. This underscores the necessity for a precise, personalized model to predict depression severity and individual risk factors, utilizing machine learning on comprehensive, multimodal datasets. While previous efforts employing machine learning (ML) to gauge depression severity exist, their effectiveness has been curtailed by small datasets and a lack of personalization. To address this gap, we propose an advanced ML-based approach for predicting depression severity and identifying personalized risk factors. ML enhances the precision of depression severity assessments, facilitates personalized treatment strategies, and improves the identification of individual risk factors. In our study, we implemented, assessed, and compared five supervised ML algorithms-Linear Regression (LR), Support Vector Machine (SVM), Extreme Gradient Boosting (XGBoost), Random Forest (RF), and Least Absolute Shrinkage and Selection Operator (LASSO)-known for their accuracy, interpretability, and computational efficiency. We utilized a multimodal dataset from the National Health and Nutrition Examination Survey (NHANES), encompassing demographic, dietary, socio-economic, lifestyle, medical, laboratory, and clinical data. The Random Forest algorithm proved to be the most effective, demonstrating an R-squared of 0.93, an explained variance score (EVS) of 0.93, a mean absolute error (MAE) of 0.51, a mean squared error (MSE) of 1.73, and a root mean squared error (RMSE) of 1.32. It effectively pinpointed both general and personalized risk factors for depression severity. Our model not only proves effective in predicting depression severity and identifying personalized risk factors but also shows promise for clinical application in assessment, diagnosis, treatment planning, and depression management.