Patient readmissions to Emergency Departments (EDs) pose significant challenges to healthcare systems, often indicating suboptimal care transitions, inadequate patient education, or insufficient post-discharge support. These unplanned readmissions not only compromise patient outcomes but also contribute to escalating healthcare costs. In response, healthcare providers are increasingly seeking predictive models to identify at-risk patients and implement preventive strategies. While advanced deep learning models have shown promise in predicting readmission risks, their "black-box" nature and substantial data requirements often limit clinical applicability due to a lack of interpretability. This study investigates the integration of Machine Learning (ML) and natural language processing (NLP) techniques to enhance the accuracy and explainability of patient readmission risk predictions. Utilizing data from the Sinai Urban Health Institute (SUHI), we analysed both structured data including demographics, interaction logs, social determinants of health (SDoH) survey answers and unstructured data, specifically notes capturing conversations between patients and Community Health Workers (CHWs). Our findings indicate that incorporating unstructured textual data improved model performance, with the area under the receiver operating characteristic curve (AUC) increasing from 0.68 to 0.74. This enhancement suggests that patient-CHW conversations capture critical, non-medical factors influencing readmissions, such as personal needs and social support deficits, which are not typically recorded in standard medical records. The study underscores the value of integrating patient-CHW contact notes into predictive modelling, helping to highlight the important role of community health in informing targeted interventions to reduce preventable readmissions.