Context: Laws and regulations increasingly influence software design, development, and quality assurance in regulated domains; however, the technology-neutral formulation of legal provisions complicates the derivation of concrete specifications, requirements, and acceptance criteria needed to verify software compliance. Producing these artifacts manually is labour-intensive and error-prone. Recent advances in generative AI, particularly large language models (LLMs), offer the potential for automated assistance in deriving software engineering artifacts from legal texts. Objective: Following a quasi-experimental design, we present the first systematic human-subject evaluation of LLMs’ ability to automatically derive Gherkin behavioural specifications from legal texts. Gherkin is a domain-specific language for specifying system behaviours through scenario-based descriptions written in the Given--When--Then format. Due to their structured and machine-readable nature, Gherkin specifications lend themselves more readily to automation within software-development processes. Methods: We recruited 10 participants to evaluate Gherkin specifications generated from food-safety regulations by two LLMs, Claude and Llama. Sixty specifications were generated. Each participant independently assessed 12 specifications across five quality criteria: relevance , clarity , completeness , singularity , and time savings . Each specification was evaluated by two participants, yielding 120 assessments with quantitative ratings and qualitative feedback. Results: Ratings were uniformly high (top-two categories): relevance 95%, clarity 100%, completeness 94.2%, singularity 93.4%, and time savings 91.7%. No statistically reliable differences were observed across participants or between LLMs. Qualitative feedback noted occasional omissions, hallucinations, and mixed intents; the first two, in particular, underscore the importance of human oversight, especially in safety-critical domains where non-compliance can have severe consequences. Conclusion: Our results suggest that, in the context of food safety, LLMs can assist in deriving Gherkin specifications from legal texts; however, observed omissions and hallucinations necessitate systematic human review.

This research explores the application of Large Language Models (LLMs) for automating the extraction of requirement-related legal content in the food safety domain and checking legal compliance of regulatory artifacts. With Industry 4.0 revolutionizing the food industry and with the General Data Protection Regulation (GDPR) reshaping privacy policies and data processing agreements, there is a growing gap between regulatory analysis and recent technological advancements. This study aims to bridge this gap by leveraging LLMs, namely BERT and GPT models, to accurately classify legal provisions and automate compliance checks. Our findings demonstrate promising results, indicating LLMs' significant potential to enhance legal compliance and regulatory analysis efficiency, notably by reducing manual workload and improving accuracy within reasonable time and financial constraints.

As software-intensive systems face growing pressure to comply with laws and regulations, providing automated support for compliance analysis has become paramount. Despite advances in the Requirements Engineering (RE) community on legal compliance analysis, important obstacles remain in developing accurate and generalizable compliance automation solutions. This paper highlights some observed limitations of current approaches and examines how adopting new automation strategies that leverage Large Language Models (LLMs) can help address these shortcomings and open up fresh opportunities. Specifically, we argue that the examination of (textual) legal artifacts should, first, employ a broader context than sentences, which have widely been used as the units of analysis in past research. Second, the mode of analysis with legal artifacts needs to shift from classification and information extraction to more end-to-end strategies that are not only accurate but also capable of providing explanation and justification. We present a compliance analysis approach designed to address these limitations. We further outline our evaluation plan for the approach and provide preliminary evaluation results based on data processing agreements (DPAs) that must comply with the General Data Protection Regulation (GDPR). Our initial findings suggest that our approach yields substantial accuracy improvements and, at the same time, provides justification for compliance decisions.