BACKGROUND: Key challenges in leveraging unstructured clinician notes for predictive models include identifying and timing patient outcomes. To address these challenges we applied large language models (LLMs) to identify and temporally localize patient outcomes in clinician notes, and evaluated whether this contextual data enhances predictive modeling for conditions like sepsis. METHODOLOGY: We applied the Medical Concept Annotation Tool (MedCAT) and two LLMs, Meta-Llama-3.1-8B and BioMistral 7B, to clinician notes in the Medical Information Mart for Intensive Care version III to identify and time International Classification of Diseases-based patient outcomes. A physician manually validated the accuracy of the models' ability to identify and time outcomes including sepsis. Finally, downstream time series predictive modeling was used to assess the impact of unstructured clinician notes on sepsis prediction. RESULTS: Meta-Llama-3.1-8B demonstrated the best balance of coverage and precision, reliably identifying outcomes while minimizing false positives. Our manual physician validation evaluated model performance, which varied between outcomes. When comparing derived sepsis timestamps to clinical sepsis onset within a 5-h window, Meta-Llama-3.1-8B showed the smallest median time difference. Finally, incorporating temporally localized outcomes into downstream sepsis prediction improved model performance, area under the receiver operating characteristic (AUC) of 0.84 (95% CI: 0.83-0.86) from 0.77 (95% CI: 0.76-0.79). CONCLUSION: We demonstrate that LLM-driven identification and timing of patient outcomes from unstructured clinician notes is feasible. Contextual outcome identification unlocks the potential of unstructured clinician notes for predictive modeling. We provide TIMED-MIMIC, an automatically generated dataset encompassing 1697 temporally localized patient outcomes, as a publicly available resource.

Charles Nimo, Tobi Olatunji, Abraham Toluwase Owodunni, Tassallah Abdullahi, Emmanuel Ayodele, Mardhiyah Sanni, Ezinwanne C. Aka, Folafunmi Omofoye, Foutse Yuehgoh, Timothy Faniran, Bonaventure F. P. Dossou, Moshood O. Yekini, Jonas Kemp, Katherine A Heller, Jude Chidubem Omeke, Chidi Asuzu Md, Naome A Etori, Aïmérou Ndiaye, Ifeoma Okoh, Evans Doe Ocansey, Wendy Kinara, Michael L. Best, Irfan Essa, Stephen Edward Moore, Chris Fourie, Mercy Nyamewaa Asiedu. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Biomedical knowledge graphs (KGs) encode rich, structured information critical for drug discovery tasks, but extracting meaningful insights from large-scale KGs remains challenging due to their complex structure. Existing biomedical subgraph retrieval methods are tailored for graph neural networks (GNNs), limiting compatibility with other paradigms, including large language models (LLMs). We introduce K-Paths, a model-agnostic retrieval framework that extracts structured, diverse, and biologically meaningful multi-hop paths from dense biomedical KGs. These paths enable prediction of unobserved drug-drug and drug-disease interactions, including those involving entities not seen during training, thus supporting inductive reasoning. K-Paths is training-free and employs a diversity-aware adaptation of Yen's algorithm to extract the K shortest loopless paths between entities in a query, prioritizing biologically relevant and relationally diverse connections. These paths serve as concise, interpretable reasoning chains that can be directly integrated with LLMs or GNNs to improve generalization, accuracy, and enable explainable inference. Experiments on benchmark datasets show that K-Paths improves zero-shot reasoning across state-of-the-art LLMs. For instance, Tx-Gemma 27B improves by 19.8 and 4.0 F1 points on interaction severity prediction and drug repurposing tasks, respectively. Llama 70B achieves gains of 8.5 and 6.2 points on the same tasks. K-Paths also boosts the training efficiency of EmerGNN, a state-of-the-art GNN, by reducing the KG size by 90% while maintaining predictive performance. Beyond efficiency, K-Paths bridges the gap between KGs and LLMs, enabling scalable and explainable LLM-augmented scientific discovery. We release our code and the retrieved paths as a benchmark for inductive reasoning.

Mardhiyah Sanni, Tassallah Abdullahi, Devendra Deepak Kayande, Emmanuel Ayodele, Naome A Etori, Michael Samwel Mollel, Moshood O. Yekini, Chibuzor Okocha, Lukman Enegi Ismaila, Folafunmi Omofoye, Boluwatife A. Adewale, Tobi Olatunji. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

The meteoric rise of Artificial Intelligence (AI), with its rapidly expanding market capitalization, presents both transformative opportunities and critical challenges. Chief among these is the urgent need for a new, unified paradigm for trustworthy evaluation, as current benchmarks increasingly reveal critical vulnerabilities. Issues like data contamination and selective reporting by model developers fuel hype, while inadequate data quality control can lead to biased evaluations that, even if unintentionally, may favor specific approaches. As a flood of participants enters the AI space, this "Wild West" of assessment makes distinguishing genuine progress from exaggerated claims exceptionally difficult. Such ambiguity blurs scientific signals and erodes public confidence, much as unchecked claims would destabilize financial markets reliant on credible oversight from agencies like Moody's.In high-stakes human examinations (e.g., SAT, GRE), substantial effort is devoted to ensuring fairness and credibility; why settle for less in evaluating AI, especially given its profound societal impact? This position paper argues that a laissez-faire approach is untenable. For true and sustainable AI advancement, we call for a paradigm shift to a unified, live, and quality-controlled benchmarking framework—robust by construction rather than reliant on courtesy or goodwill. Accordingly, we dissect the systemic flaws undermining today’s evaluation ecosystem and distill the essential requirements for next-generation assessments. To concretize this position, we introduce the idea of PeerBench, a community-governed, proctored evaluation blueprint that seeks to improve security and credibility through sealed execution, item banking with rolling renewal, and delayed transparency. PeerBench is presented as a complementary, certificate-grade layer alongside open benchmarks, not a replacement. We discuss trade-offs and limits and call for further research on mechanism design, governance, and reliability guarantees. Our goal is to lay the groundwork for evaluations that restore integrity and deliver genuinely trustworthy measures of AI progress.

The incidence of most diseases varies greatly with seasons, and global climate change is expected to increase its risk. Predictive models that automatically capture trends between climate and diseases are likely to be beneficial in minimizing disease outbreaks. Machine learning (ML) predictive analytic tools have been popularized across many health-care applications, however the optimal task performance of such ML tools largely depends on manual parameter tuning and calibration. Such manual tuning significantly limits the full potential of ML methods, especially for high-dimensional and complex task domains, as typified by real-world health-care application data-sets. Additionally, the inaccessibility of many health-care data-sets compounds innate problems of method comparison, predictive accuracy and the overall advancement of ML based health-care applications. In this study we investigate the impact of Relevance Estimation and Value Calibration, an evolutionary parameter optimization method applied to automate parameter tuning for comparative ML methods (Deep learning and Support Vector Machines) applied to predict daily diarrhoea cases across various geographic regions. Data-augmentation is also used to complement real-world noisy, sparse and incomplete data-sets with synthetic data-sets for training, validation and testing. Results support the efficacy of evolutionary parameter optimization and data synthesis to boost predictive accuracy in the given task, indicating a significant prediction accuracy boost for the deep-learning models across all data-sets.