Large Language Models (LLMs) are increasingly used for code generation, yet their support for programming languages is uneven, particularly for low-resource languages and infrequently used language constructs. Existing evaluation methodologies primarily assess functional correctness through test execution, which fails to reveal blind spots in a model's knowledge of a target language. In this paper, we introduce a systematic framework to identify and quantify the syntactic knowledge exhibited by an LLM with respect to a given programming language. We define syntactic coverage as a set of measurable indicators of how comprehensively language constructs are represented in datasets and generated code. Our approach analyzes code snippets against grammatical rules and structural features to compute interpretable coverage metrics, detect underrepresented constructs, and expose gaps in language support. The proposed framework supports comparative analysis across datasets, languages, and models through an automated pipeline that produces actionable diagnostic reports. Our experimental results demonstrate that syntactic coverage analysis reveals substantial differences in language support that are not captured by functional correctness alone, enabling more informed model selection, benchmark design, and dataset improvement strategies.

Software modelling is a creative yet challenging task. Modellers often find themselves lost in the process, from understanding the modelling problem to solving it with proper modelling strategies and modelling tools. Students learning modelling often get overwhelmed with the notations and tools. To teach students systematic modelling, we must investigate students’ practical modelling knowledge and the challenges they face while modelling. We aim to explore students’ modelling knowledge and modelling actions. Further, we want to investigate students’ challenges while solving a modelling task on specific modelling tools. We conducted an empirical study by observing 16 pairs of students from two universities and countries solving modelling tasks for one hour. We find distinct patterns of modelling of class and sequence diagrams based on individual modelling styles, the tools’ interface and modelling knowledge. We observed how modelling tools influence students’ modelling styles and how they can be used to foster students’ confidence and creativity. Based on these observations, we developed a set of guidelines aimed at enhancing modelling education and helping students acquire practical modelling skills. The guidance for modelling in education needs to be structured and systematic. Our findings reveal that different modelling styles exist, which should be properly studied. It is essential to nurture the creative aspect of a modeller, particularly while they are still students. Therefore, selecting the right tool is important, and students should understand how a tool can influence their modelling style.

Context: Having domain models derived from textual specifications has proven to be very useful in the early phases of software engineering. However, creating correct domain models and establishing clear links with the textual specification is a challenging task, especially for novice modelers. Objective: We propose an approach for determining the alignment between a partial domain model and a textual specification. Methods: To this aim, we use Natural Language Processing techniques to pre-process the text, generate an artificial natural language specification for each model element, and then use an LLM to compare the generated description with matched sentences from the original specification. Ultimately, our algorithm classifies each model element as either aligned (i.e., correct), misaligned (i.e., incorrect), or unclassified (i.e., insufficient evidence). Furthermore, it outputs the related sentences from the textual specification that provide the evidence for the determined class. Results: We have evaluated our approach on a set of examples from the literature containing diverse domains, each consisting of a textual specification and a reference domain model, as well as on models containing modeling errors that were systematically derived from the correct models through mutation. Our results show that we are able to identify alignments and misalignments with a precision close to 1 and a recall of approximately 78%, with execution times ranging from 18 s to 1 min per model element. Conclusion: Since our algorithm almost never classifies model elements incorrectly, and is able to classify over 3/4 of the model elements, it could be integrated into a modeling tool to provide positive feedback or generate warnings, or employed for offline validation and quality assessment.

Large language models (LLMs) can be used to support software development tasks, e.g. , through code completion or code generation. However, their effectiveness drops significantly when considering less popular programming languages such as domain-specific languages (DSLs). In this paper, we propose a generic framework for evaluating the capabilities of LLMs generating DSL code from textual specifications. The generated code is assessed from the perspectives of well-formedness and correctness. This framework is applied to a particular type of DSL, constraint languages, focusing our experiments on OCL and Alloy and comparing their results to those achieved for Python, a popular general-purpose programming language. Experimental results show that, in general, LLMs have better performance for Python than for OCL and Alloy. LLMs with smaller context windows such as open-source LLMs may be unable to generate constraint-related code, as this requires managing both the constraint and the domain model where it is defined. Moreover, some improvements to the code generation process such as code repair (asking an LLM to fix incorrect code) or multiple attempts (generating several candidates for each coding task) can improve the quality of the generated code. Meanwhile, other decisions like the choice of a prompt template have less impact. All these dimensions can be systematically analyzed using our evaluation framework, making it possible to decide the most effective way to set up code generation for a particular type of task.

Model-Driven Engineering (MDE) simplifies software development through abstraction, yet challenges such as time constraints, incomplete domain understanding, and adherence to syntactic constraints hinder the design process. This article presents a study to evaluate the usefulness of a novel approach utilizing Large Language Models (LLMs) and few-shot prompt learning to assist in domain modeling. The aim of this approach is to overcome the need for extensive training of traditional AI-based completion algorithms on domain-specific datasets and to offer versatile support for various modeling activities, providing valuable recommendations to software modelers. To support this approach, we developed MAGDA, a user-friendly tool, through which we conduct a user study and assess the real-world applicability of our approach in the context of domain modeling, offering valuable insights into its usability and effectiveness.

Software Product Lines (SPL) are not static software artifacts, but they evolve over time. The planning, realization, and release of a SPL requires many high-level decisions involving many different stakeholders with different expertise. Taking their opinions into account to make the right decisions is not trivial. Currently there are no mechanisms to assist stakeholders in the decision making process in an informed manner. In this paper, we propose the use of belief uncertainty in conjunction with feature models to assist in the evolution of SPLs by explicitly quantifying opinions and their associated uncertainty. We present a novel approach in which subjective logic is used to represent the opinions of stakeholders in three evolution scenarios, namely feature model evolution, next release problem and variability reduction. We apply our approach to the evolution of the Xiaomi MiBand SmartWatch SPL over the time period from July 2014 to October 2023. We present an implementation of our approach and evaluate its scalability. • Novel approach for decision making during SPL evolution. • Applicable to different evolution scenarios: feature planning, next release problem and variability reduction. • Use of subjective logic to quantify stakeholder beliefs (opinions). • Illustration on the evolution of a real-world SPL – the Xiaomi MiBand SPL – from July 2014 to October 2023. • Proof of concept implementation in Python.

Model-Driven Engineering (MDE) provides a huge body of knowledge of automation for many different engineering tasks, especially those involving transitioning from design to implementation. With the huge progress made in AI, questions arise about the future of MDE, such as how existing MDE techniques and technologies can be improved or how other activities that currently lack dedicated support can also be automated. However, at the same time, it has to be revisited where and how models should be used to keep the engineers in the loop for creating, operating, and maintaining complex systems. To trigger dedicated research on these open points, we discuss the history of automation in MDE and present perspectives on how automation in MDE can be further improved and which obstacles have to be overcome in both the medium and long-term.

To succeed with the development of modern and complex systems (e.g., aircrafts or production systems), organizations must have the agility to adapt faster to constantly evolving requirements in order to deliver more reliable and optimized solutions that can be adapted to the needs and environments of their stakeholders including users, customers, suppliers, and partners. However, stakeholders do not have sufficiently explicit and systematic support for global decision making, considering the vast decision space and complex inter-relationships. This decision space is characterized by increasing yet inadequately represented variability and the uncertainty of the impact of decisions on stakeholders and the solution space. This leads to an ad-hoc decision making process that is slow, error-prone, and often favors local knowledge over global, organization-wide objectives. As a result, one team's design decisions may impose too restrictive requirements on another team. In this paper, we evaluate our understanding of global decision making in the context of complex system development based on a conceptual model which explicitly represents and manages decision spaces including variability and impacts. We have conducted our evaluation by means of an exploratory case study where we interviewed domain experts with an average of 20 years of experience in complex system industries and report the key findings and remaining challenges. In the future, we aim at providing explicit and systematic tool-supported approaches for global decision making support for complex systems.

Abstract The integration of Large Language Models (LLMs) in software modeling tasks presents both opportunities and challenges. This Expert Voice addresses a significant gap in the evaluation of these models, advocating for the need for standardized benchmarking frameworks. Recognizing the potential variability in prompt strategies, LLM outputs, and solution space, we propose a conceptual framework to assess their quality in software model generation. This framework aims to pave the way for standardization of the benchmarking process, ensuring consistent and objective evaluation of LLMs in software modeling. Our conceptual framework is illustrated using UML class diagrams as a running example.

Abstract Most experts agree that large language models (LLMs), such as those used by Copilot and ChatGPT, are expected to revolutionize the way in which software is developed. Many papers are currently devoted to analyzing the potential advantages and limitations of these generative AI models for writing code. However, the analysis of the current state of LLMs with respect to software modeling has received little attention. In this paper, we investigate the current capabilities of ChatGPT to perform modeling tasks and to assist modelers, while also trying to identify its main shortcomings. Our findings show that, in contrast to code generation, the performance of the current version of ChatGPT for software modeling is limited, with various syntactic and semantic deficiencies, lack of consistency in responses and scalability issues. We also outline our views on how we perceive the role that LLMs can play in the software modeling discipline in the short term, and how the modeling community can help to improve the current capabilities of ChatGPT and the coming LLMs for software modeling.

Abstract The increasing essential complexity of software systems makes current software engineering methods and practices fall short in many occasions. Software assistants have the ability to help humans achieve a variety of tasks, including the development of software. Such assistants, which show human‐like competences such as autonomy and intelligence, help software engineers do their job by empowering them with new knowledge. This article investigates the research efforts that have been conducted on the creation of assistants for software design, construction and maintenance paying special attention to the user‐assistant interactions. To this end, we followed the standard systematic mapping study method to identify and classify relevant works in the state of the art. Out of the 7580 articles resulting from the automatic search, we identified 112 primary studies that present works which qualify as software assistants. We provide all the resources needed to reproduce our study. We report on the trends and goals of the assistants, the tasks they perform, how they interact with users, the technologies and mechanisms they exploit to embed intelligence and provide knowledge, and their level of automation. We propose a classification of software assistants based on interactions and present an analysis of the different automation patterns. As outcomes of our study, we provide a classification of software assistants dealing with the design, construction and maintenance phases of software development, we discuss the results, identify open lines of work and challenges and call for new innovative and rigorous research efforts in this field.

There are numerous domains in which information systems need to deal with uncertain information. These uncertainties may originate from different reasons such as vagueness, imprecision, incompleteness, or inconsistencies, and in many cases, they cannot be neglected. In this article, we are interested in representing and processing uncertain information in domain models, considering the stakeholders’ beliefs (opinions). We show how to associate beliefs to model elements and how to propagate and operate with their associated uncertainty so that domain experts can individually reason about their models enriched with their personal opinions. In addition, we address the challenge of combining the opinions of different domain experts on the same model elements, with the goal to come up with informed collective decisions. We provide different strategies and a methodology to optimally merge individual opinions.

To succeed with the development of modern software, organizations must have the agility to adapt faster to constantly evolving environments to deliver more reliable and optimized solutions that can be adapted to the needs and environments of their stakeholders including users, customers, business, development, and IT. However, stakeholders do not have sufficient automated support for global decision making, considering the increasing variability of the solution space, the frequent lack of explicit representation of its associated variability and decision points, and the uncertainty of the impact of decisions on stakeholders and the solution space. This leads to an ad-hoc decision making process that is slow, error-prone, and often favors local knowledge over global, organization-wide objectives. The Multi-Plane Models and Data (MP-MODA) framework explicitly represents and manages variability, impacts, and decision points. It enables automation and tool support in aid of a multi-criteria decision making process involving different stakeholders within a feedback-driven software development process where feedback cycles aim to reduce uncertainty. We present the conceptual structure of the framework, discuss its potential benefits, and enumerate key challenges related to tool supported automation and analysis within MP-MODA.

Although model transformations are considered to be the heart and soul of Model Driven Engineering (MDE), there are still several challenges that need to be addressed to unleash their full potential in industrial settings. Among other shortcomings, their performance and scalability remain unsatisfactory for dealing with large models, making their wide adoption difficult in practice. This paper presents A2L, a compiler for the parallel execution of ATL model transformations, which produces efficient code that can use existing multicore computer architectures, and applies effective optimizations at the transformation level using static analysis. We have evaluated its performance in both sequential and multi-threaded modes obtaining significant speedups with respect to current ATL implementations. In particular, we obtain speedups between 2.32x and 38.28x for the A2L sequential version, and between 2.40x and 245.83x when A2L is executed in parallel, with expected average speedups of 8.59x and 22.42x, respectively.

Context: Managing Non-Functional Requirements (NFRs) in software projects is challenging, and projects that adopt Model-Driven Development (MDD) are no exception. Although several methods and techniques have been proposed to face this challenge, there is still little evidence on how NFRs are handled in MDD by practitioners. Knowing more about the state of the practice may help researchers to steer their research and practitioners to improve their daily work. Objective: In this paper, we present our findings from an interview-based survey conducted with practitioners working in 18 different companies from 6 European countries. From a practitioner's point of view, the paper shows what barriers and benefits the management of NFRs as part of the MDD process can bring to companies, how NFRs are supported by MDD approaches, and which strategies are followed when (some) types of NFRs are not supported by MDD approaches. Results: Our study shows that practitioners perceive MDD adoption as a complex process with little to no tool support for NFRs, reporting productivity and maintainability as the types of NFRs expected to be supported when MDD is adopted. But in general, companies adapt MDD to deal with NFRs. When NFRs are not supported, the generated code is sometimes changed manually, thus compromising the maintainability of the software developed. However, the interviewed practitioners claim that the benefits of using MDD outweight the extra effort required by these manual adaptations. Conclusion: Overall, the results indicate that it is important for practitioners to handle `NFRs in MDD, but further research is necessary in order to lower the barrier for supporting a broad spectrum of NFRs with MDD. Still, much conceptual and tool implementation work seems to be necessary to lower the barrier of integrating the broad spectrum of NFRs in practice.

OCR engines aim to identify and extract text strings fromdocuments or images. While current efforts focus mostly inmainstream languages, there is little support for program-ming or domain-specific languages (DSLs). In this paper, wepresent our vision about the current state of OCR recognitionfor DSLs and its challenges. We discuss some strategies toimprove the OCR quality applied to DSL textual expressionsby leveraging DSL specifications and domain data. To bettersupport our ideas we present the preliminary results of anempirical study and outline a research roadmap.

This paper discusses the representation of Belief Uncertainty in software models. This kind of uncertainty refers to the situation in which the modeler, or any other belief agent, is uncertain about the behavior of the system, or the statements that the model expresses about it. In this work, we propose to assign a degree of belief to model statements (let they be constraints, or any other model expression), which is expressed by a probability (called credence, in statistical terms) that represents a quantification of such a subjective degree of belief. We discuss how it can be represented using current modeling notations, and how to operate with it in order to make informed decisions.

Model transformations are a key element in any model-driven engineering approach. But writing them is a time-consuming and error-prone activity that requires specific knowledge of the transformation language semantics. We propose to take advantage of the advances in Artificial Intelligence and, in particular Long Short-Term Memory Neural Networks (LSTM), to automatically infer model transformations from sets of input-output model pairs. Once the transformation mappings have been learned, the LSTM system is able to autonomously transform new input models into their corresponding output models without the need of writing any transformation-specific code. We evaluate the correctness and performance of our approach and discuss its advantages and limitations.

Although Model-Based Software Engineering (MBE) is a widely accepted Software Engineering (SE) discipline, no agreed-upon core set of concepts and practices (i.e., a Body of Knowledge) has been defined for it yet. With the goals of characterizing the contents of the MBE discipline, promoting a global consistent view of it, clarifying its scope with regard to other SE disciplines, and defining a foundation for the development of educational curricula on MBE, this paper proposes the contents for a Body of Knowledge for MBE. We also describe the methodology that we have used to come up with the proposed list of contents, as well as the results of a survey study that we conducted to sound out the opinion of the community on the importance of the proposed topics and their level of coverage in the existing SE curricula.

Complex Event Processing (CEP) is a powerful technology in realtime distributed environments for analyzing fast and distributed streams of data, and deriving conclusions from them. CEP permits defining complex events based on the events produced by the incoming sources in order to identify complex meaningful circumstances and to respond to them as quickly as possible. However, in many situations the information that needs to be analyzed is not structured as a mere sequence of events, but as graphs of interconnected data that evolve over time. This paper proposes an extension of CEP systems that permits dealing with graph-structured information. Two case studies are used to validate the proposal and to compare its performance with traditional CEP systems. We discuss the benefits and limitations of the CEP extensions presented.

The expression and management of uncertainty, both in the information and in the operations that manipulate it, is a critical issue in those systems that work with physical environments. Measurement uncertainty can be due to several factors, such as unreliable data sources, tolerance in the measurements, or the inability to determine if a certain event has actually happened or not. In particular, this contribution focuses on the expression of one kind of uncertainty, namely the confidence on the model elements, i.e., the degree of belief that we have on their occurrence, and on how such an uncertainty can be managed and propagated through model transformations, whose rules can also be subject to uncertainty.

This contribution proposes a new technique for developing test cases for UML and OCL models. The technique is based on an approach that automatically constructs object models for class models enriched by OCL constraints. By guiding the construction process through so-called classifying terms, the built test cases in form of object models are classified into equivalence classes. A classifying term can be an arbitrary OCL term on the class model that calculates for an object model a characteristic value. From each equivalence class of object models with identical characteristic values one representative is chosen. The constructed test cases behave significantly different with regard to the selected classifying term. By building few diverse object models, properties of the UML and OCL model can be explored effectively. The technique is applied for automatically constructing relevant source model test cases for model transformations between a source and target metamodel.

Model-Driven Development (MDD) is no longer a novel development paradigm. It has become mature from a research perspective and recent studies show its adoption in industry. Still, some issues remain a challenge. Among them, we are interested in the treatment of non-functional requirements (NFRs) in MDD processes. Very few MDD approaches have been reported to deal with NFRs (and they do it in a limited way). However, it is clear that NFRs need to be considered somehow in the final product of the MDD process. To better understand how NFRs are integrated into the existing MDD approaches, we have initiated the NFR4MDD project, a multi-national empirical study, based on interviews with companies working on MDD projects. Our project aims at surveying the state of the practice for this topic. In this paper, we summarize our research protocol and present the current status of our study. The discussion will focus on the peculiarities of our study's context and organization involving about 20 researchers from 8 European countries.

As the complexity of model transformations grows, there is an increasing need to count on methods, mechanisms, and tools for checking their correctness, i.e., the alignment between specifications and implementations. In this paper we present a light-weight and static approach for locating the faulty rules in model transformations, based on matching functions that automatically establish these alignments using the metamodel footprints, i.e., the metamodel elements used. The approach is implemented for the combination of Tracts and ATL, both residing in the Eclipse Modeling Framework, and is supported by the corresponding toolkit. An evaluation discussing the accuracy and the limitations of the approach is also provided. Furthermore, we identify the kinds of transformations which are most suitable for validation with the proposed approach and use mutation techniques to evaluate its effectiveness.