This paper introduces essential resources for Qur’anic studies: an annotated Tafsir ontology, a dataset of approximately 4,200 question-answer pairs, and a collection of 15 structured Tafsir books available in two formats. We present a comprehensive framework for handling sensitive Qur’anic Tafsir data that spans the entire pipeline from dataset construction through evaluation and error analysis. Our work establishes new benchmarks for retrieval and question-answering tasks on Qur’anic content, comparing performance across state-of-the-art embedding models and large language models (LLMs).We introduce OntologyRAG-Q, a novel retrieval-augmented generation approach featuring our custom Ayat-Ontology chunking method that segments Tafsir content at the verse level using ontology-driven structure. Benchmarking reveals strong performance across various LLMs, with GPT-4 achieving the highest results, followed closely by ALLaM. Expert evaluations show our system achieves 69.52% accuracy and 74.36% correctness overall, though multi-hop and context-dependent questions remain challenging. Our analysis demonstrates that answer position within documents significantly impacts retrieval performance, and among the evaluation metrics tested, BERT-recall and BERT-F1 correlate most strongly with expert assessments. The resources developed in this study are publicly available at https://github.com/sazani/OntologyRAG-Q.git.

The widespread interest in large language models (LLMs) is rooted in their remarkable capacity to generate human-like and contextually relevant responses. However, the precision of LLMs within specific domains or intricate tasks, such as Arabic dialect identification, remains largely unexplored. This task presents a substantial challenge in Arabic natural language processing, given its language-dependent nature. This paper provides a framework for evaluating LLMs for Arabic dialect identification and conducts a comprehensive evaluation of LLMs, employing both tuning-free and fine-tuning-based learning paradigms. The evaluation encompasses GPT-3.5, chatGPT, GPT-4, and Google BARD for Arabic dialect identification under the tuning-free learning paradigm. Furthermore, it assesses the performance of GPT-3.5 along with AraBERT and MARBERT using the fine-tuning learning paradigm. In the tuning-free approach, GPT-4 achieves the most favorable results, reporting an F1MACof 45.60%. Under the fine-tuning learning paradigm, both AraBERT and MARBERT exhibit comparable performance (around 50% F 1MAC) to GPT-3.5, without incurring any financial costs, in contrast to the expenses associated with GPT-3.5.