LLM-based dialogue assistants have become mainstream tools for software developers, yet current evaluation benchmarks focus exclusively on functional correctness. This leaves a critical gap in assessing the quality and accuracy of these conversations when handling Non-Functional Requirements (NFRs), which are inherently vague, context-dependent, and involve many parts of a program. Evaluating how well these systems support collaborative reasoning about NFRs requires methods that go beyond single-turn accuracy to capture both the correctness of the system’s outputs and the quality of the multi-turn interaction. In this paper, we investigate the accuracy and quality of multi-turn conversations between developers and an LLM-based agent in the domain of Health Insurance Portability and Accountability Act (HIPAA) regulatory compliance, a representative case of regulatory NFRs. We hired 49 programmers to interact with GitHub Copilot to assess 148 HIPAA-derived NFRs against the iTrust codebase, a system designed to comply with HIPAA regulations, across three dimensions: requirement satisfaction level, reasoning, and code localization. We find that developers tend to agree with LLM assessments, but accuracy against expert ground truth is low. We model user satisfaction and find that longer system responses and more information-providing turns negatively affect user satisfaction, whereas proactive interactions positively affect it. Our findings provide insights for designing LLM-based dialogue systems that support NFR assessment.

Software practitioners use online forums to navigate complex and often ambiguous legal privacy requirements, yet little is known about their professional backgrounds, what challenges they face, and how they use and assess the credibility of the advice received, or how they resolve ambiguities in posts. We report the findings of a survey of 223 Reddit users from regulatory-focused subreddits, complemented by a qualitative analysis of 2,248 posts and responses. Our results show that, despite holding privacy-related certifications, most participants frequently use forums to seek legal advice. Key challenges reported or identified include implementing a data protection impact assessment, reporting a data breach, and obtaining cookie consent. Reddit users often assess credibility by reviewing respondents' post history, verifying sources cited, trusting advice from recognized experts, and following up for clarity before responding. We highlight research and educational directions to bridge gaps in support needed for regulatory compliance guidance.

Research shows that analysts and developers consider privacy as a security concept or as an afterthought, which may lead to non-compliance and violation of users’ privacy. Most current approaches, however, focus on extracting legal requirements from the regulations and evaluating the compliance of software and processes with them. In this paper, we develop a novel approach based on chain-of-thought prompting (CoT), in-context-learning (ICL), and Large Language Models (LLMs) to extract privacy behaviors from various software documents prior to and during software development, and then generate privacy requirements in the format of user stories. Our results show that most commonly used LLMs, such as GPT-4o and Llama 3, can identify privacy behaviors and generate privacy user stories with F1 scores exceeding 0.8. We also show that the performance of these models could be improved through parameter-tuning. Our findings provide insight into using and optimizing LLMs for generating privacy requirements given software documents created prior to or throughout the software development lifecycle.

Language models that can learn a task at inference time, called in-context learning (ICL), show increasing promise in natural language inference tasks. In ICL, a model user constructs a prompt to describe a task with a natural language instruction and zero or more examples, called demonstrations. The prompt is then input to the language model to generate a completion. In this paper, we apply ICL to the design and evaluation of satisfaction arguments, which describe how a requirement is satisfied by a system specification and associated domain knowledge. The approach builds on three prompt design patterns, including augmented generation, prompt tuning, and chain-of-thought prompting, and is evaluated on a privacy problem to check whether a mobile app scenario and associated design description satisfies eight consent requirements from the EU General Data Protection Regulation (GDPR). The overall results show that GPT-4 can be used to verify requirements satisfaction with 96.7% accuracy and dissatisfaction with 93.2% accuracy. Inverting the requirement improves verification of dissatisfaction to 97.2%. Chain-of-thought prompting improves overall GPT-3.5 performance by 9.0% accuracy. We discuss the trade-offs among templates, models and prompt strategies and provide a detailed analysis of the generated specifications to inform how the approach can be applied in practice.

Privacy labels help developers communicate their application’s privacy behaviors (i.e., how and why an application uses personal information) to users. But, studies show that developers face several challenges in creating them and the resultant labels are often inconsistent with their application’s privacy behaviors. In this paper, we create a novel methodology called fine-grained localization of privacy behaviors to locate individual statements in source code which encode privacy behaviors and predict their privacy labels. We design and develop an attention-based multi-head encoder model which creates individual representations of multiple methods and uses attention to identify relevant statements that implement privacy behaviors. These statements are then used to predict privacy labels for the application’s source code and can help developers write privacy statements that can be used as notices. Our quantitative analysis shows that our approach can achieve high accuracy in identifying privacy labels, with the lowest accuracy of 91.41% and the highest of 98.45%. We also evaluate the efficacy of our approach with six software professionals from our university. The results demonstrate that our approach reduces the time and mental effort required by developers to create high-quality privacy statements and can finely localize statements in methods that implement privacy behaviors.

This tool demonstration presents a research toolkit for a language model of Java source code. The target audience includes researchers studying problems at the granularity level of subroutines, statements, or variables in Java. In contrast to many existing language models, we prioritize features for researchers including an open and easily-searchable training set, a held out test set with different levels of deduplication from the training set, infrastructure for deduplicating new examples, and an implementation platform suitable for execution on equipment accessible to a relatively modest budget. Our model is a GPT2-like architecture with 350m parameters. Our training set includes 52m Java methods (9b tokens) and 13m StackOverflow threads (10.5b tokens). To improve accessibility of research to more members of the community, we limit local resource requirements to GPUs with 16GB video memory. We provide a test set of held out Java methods that include descriptive comments, including the entire Java projects for those methods. We also provide deduplication tools using precomputed hash tables at various similarity thresholds to help researchers ensure that their own test examples are not in the training set. We make all our tools and data open source and available via Huggingface and Github.

Interpreting and describing mobile applications' privacy behaviors to ensure creating consistent and accurate privacy notices is a challenging task for developers. Traditional approaches to creating privacy notices are based on predefined templates or questionnaires and do not rely on any traceable behaviors in code which may result in inconsistent and inaccurate notices. In this paper, we present an automated approach to detect privacy behaviors in code of Android applications. We develop Privacy Action Taxonomy (PAcT), which includes labels for Practice (i.e. how applications use personal information) and Purpose (i.e. why). We annotate ~5,200 code segments based on the labels and create a multi-label multi-class dataset with ~14,000 labels. We develop and train deep learning models to classify code segments. We achieve the highest F-1 scores across all label types of 79.62% and 79.02% for Practice and Purpose.

Ensuring compliance with complex privacy related regulations is more challenging in the presence of changing operating environments, such as cloud based services. This motivates the needs for tools and techniques that support systematic modeling and analysis of compliance requirements checking while considering the impact of operating environment changes on them. This paper demonstrates how goaloriented requirements engineering tools can be used to model variability in compliance sources and operating environments. We report a case study to demonstrate that the proposed modeling approach enables preforming trade-off analysis between organizational goals and alternative compliance controls.

This paper is concerned with the goal of maintaining legal information and compliance systems: the ‘resource consumption bottleneck’ of creating semantic technologies manually. The use of automated information extraction techniques could significantly reduce this bottleneck. The research question of this paper is: How to address the resource bottleneck problem of creating specialist knowledge management systems? In particular, how to semi-automate the extraction of norms and their elements to populate legal ontologies? This paper shows that the acquisition paradox can be addressed by combining state-of-the-art general-purpose NLP modules with pre- and post-processing using rules based on domain knowledge. It describes a Semantic Role Labeling based information extraction system to extract norms from legislation and represent them as structured norms in legal ontologies. The output is intended to help make laws more accessible, understandable, and searchable in legal document management systems such as Eunomos (Boella et al., 2016).

Privacy-preserving data analysis has become essential in Machine Learning (ML), where access to vast amounts of data can provide large gains the in accuracies of tuned models. A large proportion of user-contributed data comes from natural language e.g., text transcriptions from voice assistants. It is therefore important for curated natural language datasets to preserve the privacy of the users whose data is collected and for the models trained on sensitive data to only retain non-identifying (i.e., generalizable) information. The workshop aims to bring together researchers and practitioners from academia and industry to discuss the challenges and approaches to designing, building, verifying, and testing privacy-preserving systems in the context of Natural Language Processing (NLP).

Contains fulltext : 161870.pdf (Publisher’s version ) (Open Access)

We apply an existing formal framework for practical reasoning with arguments and evidence to the Goal-oriented Requirements Language (GRL), which is part of the User Requirements Notation (URN). This formal framework serves as a rationalization for elements in a GRL model: using attack relations between arguments we can automatically compute the acceptability status of elements in a GRL model, based on the acceptability status of their underlying arguments and the evidence. We integrate the formal framework into the GRL metamodel and we set out a research to further develop this framework.

Every year, governments introduce new or revised regulations that are imposing new types of requirements on software development. Analyzing and modeling these legal requirements is time consuming, challenging and cumbersome for software and requirements engineers. Having regulation models can help understand regulations and converge toward better compliance levels for software and systems. This paper introduces a systematic method to extract legal requirements from regulations by mapping the latter to the Legal Profile for Goal-oriented Requirements Language (GRL) (Legal GRL). This profile provides a conceptual meta-model for the anatomy of regulations and maps its elements to standard GRL with specialized annotations and links, with analysis techniques that exploit this additional information. The paper also illustrates examples of Legal GRL models for The Privacy and Electronic Communications Regulations. Existing tool support (jUCMNav) is also extended to support Legal GRL modeling.

Most systems and business processes in organizations need to comply with more than one law or regulation. Different regulations can partially overlap (e.g., one can be more detailed than the other) or even conflict with each other. In addition, one regulation can permit an action whereas the same action in another regulation might be mandatory or forbidden. In each of these cases, an organization needs to take different strategies. This paper presents an approach to handle different situations when comparing and attempting to comply with multiple regulations as part of a goal-oriented modeling framework named LEGAL-URN. This framework helps organizations find suitable trade-offs and priorities when complying with multiple regulations while at the same time trying to meet their own business objectives. The approach is illustrated with a case study involving a Canadian health care organization that must comply with four laws related to privacy, quality of care, freedom of information, and care consent.

In this article, we introduce the application of rigorous analysis procedures to goal models to provide several benefits beyond the initial act of modeling. Such analysis can allow modelers to assess the satisfaction of goals, facilitate evaluation of high-level design alternatives, help analysts decide on the high-level requirements and design of the system, test the sanity of a model, and support communication and learning. The analysis of goal models can be done in very different ways depending on the nature of the model and the purpose of the analysis. In our work, we use the Goal-oriented Requirement Language (GRL), which is part of the User Requirements Notation (URN). URN, a new Recommendation of the International Telecommunications Union, provides the first standard goal-oriented language. Using GRL, we develop an approach to analysis that can be done by evaluating qualitative or quantitative satisfaction levels of the actors and intentional elements (e.g., goals and tasks) composing the model. Initial satisfaction levels for some of the intentional elements are provided in a strategy and then propagated to the other intentional elements of the model through the various links that connect them. The results allow for an assessment of the relative effectiveness of design alternatives at the requirements level. Although no specific propagation algorithm is imposed in the URN standard, different criteria for defining evaluation mechanisms are described. We provide three algorithms (quantitative, qualitative, and hybrid) as examples, which satisfy the constraints imposed by the standard. These algorithms have been implemented in the open-source jUCMNav tool, an Eclipse-based editor for URN models. The algorithms are presented and compared with the help of a telecommunication system example. © 2010 Wiley Periodicals, Inc.

In recent years, many governmental regulations have been introduced to protect the privacy of personal information. As a result, organizations must take a systematic approach to ensure that their business processes comply with these regulations. In the past, we introduced a requirements framework that mapped regulations documents and goals to goal and scenario models of organizational processes. The intent was to help organizations document and manage the compliance of their processes in the face of evolutionary changes. In this paper, we extend our framework by incorporating regulation scenario models and by adding the notion of contribution link level to the compliance link types. These extensions result in a frame-work that is more aligned to the needs of an organization when it must evaluate and ensure the legal compliance of its organizational processes.

In November 2008, the User Requirements Notation (URN) was approved as a standard by the International Telecommunication Union (ITU-T). jUCMNav is the most comprehensive tool available to date that supports the definition, analysis, transformation, and management of URN requirements engineering models. URN is the first standardized framework unifying modeling concepts and notations for goals and intentions (mainly for non-functional requirements, quality attributes, and reasoning about alternatives) and scenarios (mainly for operational/functional requirements and reasoning about scenario interactions, performance, and high-level architecture). jUCMNav has been and continues to be instrumental in validating key concepts for the current standard as well as prototyping new concepts.