Conformance checking allows auditors to detect process deviations automatically, resulting in numerous deviations, with only a few being relevant. Identifying notable items amidst this large data set is challenging. Machine learning techniques offer potential solutions, but questions about the required number of labeled deviations and the impact of label quality remain. Our study investigates these factors’ effects on Decision Trees and Random Forests. Results demonstrate these models’ effectiveness in identifying notable items within imbalanced deviation populations. Achieving 90% precision and recall is feasible with about 400 to 600 labeled deviations, depending on the notable items’ population fraction. A higher fraction of notables reduces the required labeled deviations. Varying label quality produced similar results. Additionally, classifications identifying at least 90% notable items are linked to less complex processes.

Although conformance checking is great at detecting process deviations, it still poses challenges that hinders adoption in auditing practice. A major challenge is that in real life a large number of deviating cases is often detected of which only a small amount are true anomalies and thus of real interest to auditors. The number of deviations are often too large to inspect one by one, which explains why auditing requires a sample-based approach. This paper contributes to the research on the practical feasibility of continuous auditing and studies the potential of weak supervision to classify deviations into anomalies and exceptions, allowing auditors to do a full-population analysis of the identified deviations. The Snorkel framework is applied which uses a set of imperfect domain expert rules to classify the set of deviations into anomalies and exceptions. A controlled and artificial experiment has been set up to explore the relation between the performance of this approach and the number and quality of domain expert rules. The results demonstrate the potential of this approach as a limited number of medium to high quality domain expert rules succeeds to classify deviations with acceptable accuracy.