Abstract Key Information Extraction (KIE) systems based on Deep Learning achieve strong token-level performance but offer no formal guarantees on prediction reliability, limiting their adoption in business-critical document workflows. In this work, we introduce a post hoc Uncertainty Quantification framework for KIE using Split Conformal Prediction (CP). After fine-tuning multimodal transformer models on a challenging receipt dataset, we reserve a held-out calibration set to derive nonconformity scores and construct entity-level prediction sets that satisfy a user-specified error rate. On unseen receipts, CP achieves tight marginal coverage (98.3% for $$\alpha =0.02$$ α = 0.02 ), with 70% of predictions being high-confidence singletons. A detailed analysis shows that highly structured fields such as dates and prices yield small, singleton sets with near–perfect reliability, whereas rare or semantically ambiguous fields such as tips or generic keywords produce larger sets and lower coverage. By exposing positional biases and common label confusions that standard F1-scores and document-accuracy metrics overlook, CP reveals critical risk areas for downstream automation. Finally, we demonstrate how calibrated prediction-set sizes can drive risk-aware workflows by automatically processing high-confidence extractions and flagging uncertain cases for human review, thereby enhancing the efficiency, trustworthiness and operational feasibility of real-world document-processing systems.

The creation of Business Process Model and Notation (BPMN) models is a complex and time-consuming task requiring both domain knowledge and proficiency in modeling conventions. Recent advances in large language models (LLMs) have significantly expanded the possibilities for generating BPMN models directly from natural language, building upon earlier text-to-process methods with enhanced capabilities in handling complex descriptions. However, there is a lack of systematic evaluations of LLM-generated process models. Current efforts either use LLM-as-a-judge approaches or do not consider established dimensions of model quality. To this end, we introduce BEF4LLM, a novel LLM evaluation framework comprising four perspectives: syntactic quality, pragmatic quality, semantic quality, and validity. Using BEF4LLM, we conduct a comprehensive analysis of open-source LLMs and benchmark their performance against human modeling experts. Results indicate that LLMs excel in syntactic and pragmatic quality, while humans outperform LLMs in semantic aspects; however, the differences in scores are relatively modest, highlighting LLMs’ competitive potential despite challenges in validity and semantic quality. The insights highlight current strengths and limitations of using LLMs for BPMN modeling and guide future model development and fine-tuning. Addressing these areas is essential for advancing the practical deployment of LLMs in business process modeling.

As artificial intelligence (AI) systems are increasingly deployed in high-stakes environments, the need for explanations that convey uncertain information has become evident. Conventional explainable AI (XAI) methods often overlook uncertainty, focusing solely on point predictions. To address this gap, we propose using permutation feature importance (PFI) combined with predictive uncertainty evaluation measures. This novel approach examines the significance of features by relating them to the model’s confidence in its predictions. By using split conformal prediction (SCP) to quantify predictive uncertainty and integrating the outcomes to PFI, we aim to enhance the robustness and interpretability of machine learning (ML) algorithms. More importantly, we examine three scenarios for conformal prediction-based PFI explanations: permuting feature values in the test data, the calibration data, and both. These scenarios assess the impact of feature permutations from different perspectives, revealing feature sensitivity and the importance of features in various settings. We also perform a series of sensitivity analyses, particularly exploring calibration data size and computational efficiency, to demonstrate the robustness and scalability of our approach for industrial applications. Our comprehensive evaluation offers insights into feature impact on predictions and their associated confidence levels. We validate our proposed approach through a real-world predictive process monitoring use case in manufacturing.

Abstract This paper proposes a multi‐stage approach consisting of deep learning‐based image classification, process trace clustering, and visual/statistical knowledge discovery of process data. The proposed decision augmentation solution aims to facilitate the production planners in estimating the process‐specific production parameters such as activity duration, idle time, or machine utilization. This study focuses on ‘one‐of‐a‐kind production’ (OKP). Planning in OKP is especially challenging due to the increasing individualization of customer requirements. Furthermore, the uniqueness of products adds complexity to data and information structuring. To tackle this issue, we first train deep convolutional neural networks (CNN) with image data of production parts obtained from computer‐aided design (CAD) systems to extract meaningful features. After cross‐validation, uncertainty, and robustness assessment of the adopted deep learning approach, we use the data representation from the penultimate layer as input for clustering production parts. The goodness of clustering results is evaluated using a series of internal clustering validation indices. Finally, process event log data provided by manufacturing execution systems (MES) is mapped to each production part, allowing us to conduct statistical and visual knowledge discovery of process parameters for each cluster. The relevance of our proposed approach has been validated by studying a real‐world use case in a small, medium‐sized enterprise (SME) operating in the fixture and jig manufacturing industry.

Recent advancements in sensor technology offer opportunities to manage business processes in a proactive manner. To enable an effective and real-time monitoring, sensor data have to be treated and processed in an event processing manner. Complex Event Processing is an efficient technology that detects useful complex events by matching primitive sensor events using event patterns. Event patterns can be represented as templates that combine primitive events by temporal, logical, spatial and sequential correlations to detect more complex events. Identifying event patterns out of streaming data with a high data volume and velocity is a challenging task. In this paper, we propose an Ensemble Model consisting of a crisp and fuzzy rule based classifiers in order to derive decision rules as event patterns. Before implementing the ensemble classifier directly to the streaming data, we select the most influential feature subset using a multi-objective evolutionary algorithm. The performance of the proposed model was evaluated using real data obtained from accelerometer sensors. Promising results with high accuracy and appropriate level of computational complexity were obtained and discussed.