In Process Mining (PM), “high-level” conceptual models of business processes, in the form of directly-follows graphs, Petri nets, and finite-state automata, are discovered from “low-level” event data recorded by information systems. The quality of the discovered models is usually assessed by measures that depend on assumptions made by discovery algorithms; for example, they often assume that sequences of activities recorded in the event data do not interfere. Models produced by recent discovery algorithms consider domain knowledge and relax these assumptions, making traditional PM measures less suitable for evaluating their quality. This paper proposes an ontology-aware framework, called SOLID-M, for analyzing the quality of conceptual models discovered from event data generated by systems. SOLID-M relies on domain knowledge and provides guidelines for introducing quality measures for models constructed by process discovery algorithms that go beyond the traditional PM assumptions. In addition, the paper describes an instantiation of the framework for assessing the quality of Multi-Agent System models discovered using Agent System Mining techniques, hence addressing a growing demand for data-driven analysis of business processes emerging in interactions of human and artificial intelligence agents.

Conformance checking is a subarea of process mining that studies relations between designed processes, also called process models, and records of observed processes, also called event logs. In the last decade, research in conformance checking has proposed a plethora of techniques for characterizing the discrepancies between process models and event logs. Often, these techniques are also applied to measure the quality of process models automatically discovered from event logs. Recently, the process mining community has initiated a discussion on the desired properties of such measures. This discussion witnesses the lack of measures with the desired properties and the lack of properties intended for measures that support partially matching processes, i.e., processes that are not identical but differ in some steps. The paper at hand addresses these limitations. Firstly, it extends the recently introduced precision and recall conformance measures between process models and event logs that possess the desired property of monotonicity with the support of partially matching processes. Secondly, it introduces new intuitively desired properties of conformance measures that support partially matching processes and shows that our measures indeed possess them. The new measures have been implemented in a publicly available tool. The reported qualitative and quantitative evaluations based on our implementation demonstrate the feasibility of using the proposed measures in industrial settings.