Retrieval-Augmented Generation (RAG) extends LLMs with external knowledge, typically from continuous text sources. In this work, we explore the application of RAG to ontologies to support ontology engineers and users in querying structured knowledge. To demonstrate the generalizability of our approach, we evaluate RAG on three ontologies from the domains of video games, IoT infrastructure, and rhetorical figures. Two of the ontologies are in English, while one is in German. Our study compares chunking strategies, chunk sizes, and the use of rerankers. We observe that advanced chunking techniques offer limited performance gains due to the inherently modular structure of ontologies. Additionally, we compare RAG on ontological document structure with the same information in a continuous format and reveal that the performance of RAG is not negatively affected by the ontological document structure. An advantage of using ontologies is that they can be evaluated against their competency questions that are typically developed alongside and define the intended use. With the Ragas evaluation framework, we achieve promising results across domains and languages. To address challenges regarding verbose LLM outputs that decrease performance, we propose a new metric Answer Average Match (AAM), which is specifically designed for the evaluation of RAG on ontologies. Our findings highlight the viability of ontology-aware RAG and offer practical insights for its optimization and evaluation.

We propose a lightweight and adaptable model tailored to German argument mining in two dissimilar domains. It offers improvements over the state-of-the-art, as well as over recent LLM approaches. Specifically, we achieve an 11.5 % absolute improvement across all classes of the SPWSLE dataset as well as 3.5% absolute detection performance boost across all datasets of the CIMT collection. We show that for both tasks, the hitherto disjoint steps of argument detection followed by argument classification can be merged with no loss in quality, resulting in a greater than$2 x$training and inference speed-up. For the naive baseline, this joint scheme also significantly improves results. For CIMT we propose realistic evaluation that incorporates actual detection results instead of simulating perfect detection. We also investigate the transferability between and to new datasets and find it to be higher than in previous work and in certain cases even better than within the dataset. A super model trained on all other datasets adapts to unseen data without re-training or performance loss.

Grades deliver important quantitative feedback on student learning, but grading can be time-consuming for large student cohorts. Recent advances in large language models (LLMs) offer promising opportunities for automating this process. Still, a systematic examination of different models and prompting paradigms remains lacking. To fill this gap, we compare the performance of 23 closed-source and open-source LLMs, testing the best-performing six across five different prompt paradigms including zero-shot, pre-written sample solutions with an integrated grading rubric, few-shot examples, and retrievalaugmented generation (RAG) based on textbook excerpts. We hence investigate the extent to which external information is necessary for reliable grading outcomes. As a benchmark, we include four fine-tuned encoder models. Our study focuses on German-language undergraduate economics courses, using an open macroeconomics (Macro) and a novel institutional economics (Insti) dataset graded by three human examiners. LLMhuman agreement reaches Quadratic Weighted Kappas of up to 0.811 for Macro and 0.687 for Insti, with the best LLM-prompt combinations nearing fine-tuned encoder model performance for Macro. Providing a sample solution with a grading rubric emerges as a crucial prompting feature, as it enables near-human performance even in zero-shot setups, particularly for easier-to-grade answers. Adding few-shot examples further improves grading for difficult answers. Few-shot examples without a sample solution, RAG, or relying solely on LLMs' inherent knowledge, on the other hand, perform significantly worse.

Supervised machine learning often encounters concept drift, where the data distribution changes over time, degrading model performance. Existing drift detection methods focus on identifying these shifts but often overlook the challenge of acquiring labeled data for model retraining after a shift occurs. We present the Strategy for Drift Sampling (SUDS), a novel method that selects homogeneous samples for retraining using existing drift detection algorithms, thereby enhancing model adaptability to evolving data. SUDS seamlessly integrates with current drift detection techniques. We also introduce the Harmonized Annotated Data Accuracy Metric (HADAM), a metric that evaluates classifier performance about the quantity of annotated data required to achieve the stated performance, thereby taking into account the difficulty of acquiring labeled data. Our contributions are twofold: SUDS combines drift detection with strategic sampling to improve the retraining process, and HADAM provides a metric that balances classifier performance with the amount of labeled data, ensuring efficient resource utilization. Empirical results demonstrate the efficacy of SUDS in optimizing labeled data use in dynamic environments, significantly improving the performance of machine learning applications in real-world scenarios. Our code is open source and available at https://github.com/cfellicious/SUDS/

In the 2015 migration crisis thousands of refugees and migrants crossed the border to Hungary, Austria and Germany. The movements of these people are reflected in social media, especially on Twitter. In this paper we present a dataset of 3275 Tweets from the months September and October 2015. These Tweets are annotated regarding their relevance of containing quantitative movement information of refugees/migrants into Hungary, Austria and Germany. We present this dataset for a posterior analysis of the 2015 migration crisis or as a basis for creating an automated extraction / prediction system.