Auditing complex Artificial Intelligence (AI) systems is gaining importance in light of new regulations and is particularly challenging in terms of system complexity, knowledge integration, and differing transparency needs. Current AI auditing tools however, lack semantic context, resulting in difficulties for auditors in effectively collecting and integrating, but also for analysing and querying audit data. In this position paper, we explore how Knowledge Graphs (KGs) can address these challenges by offering a structured and integrative approach to collecting and transforming audit traces. This work discusses the current limitations in both AI auditing processes and tools. Furthermore, we examine how KGs can play a transformative role in overcoming these obstacles to achieve improved auditability and transparency of AI systems.

In many countries worldwide, including Austria, the environmental impact of production facilities is strongly regulated leading to authorities issuing a large number of legal permits on this topic. The access of interested parties to these permits is typically supported by search systems that present a structured view of the permits along their key elements, such as issuing authority or their legal basis. In this paper, we present a real-life use case from Austria's Environment Agency, where the extraction of such key elements represents a non-trivial task for laypersons with limited legal knowledge: the heterogeneity of data, complex language, and implicit information hinder the manual data extraction process and can lead to poor quality in data management. Based on an analysis of the use case's main requirements, we propose an architecture for a system to support the extraction of key elements from legal permits by laypersons. The system combines methods and techniques based on Knowledge Graphs / Semantic Web and Machine Learning technologies and aims to be auditable in terms of its operation.