Large Language Models (LLMs) are increasingly used for code generation, yet their support for programming languages is uneven, particularly for low-resource languages and infrequently used language constructs. Existing evaluation methodologies primarily assess functional correctness through test execution, which fails to reveal blind spots in a model's knowledge of a target language. In this paper, we introduce a systematic framework to identify and quantify the syntactic knowledge exhibited by an LLM with respect to a given programming language. We define syntactic coverage as a set of measurable indicators of how comprehensively language constructs are represented in datasets and generated code. Our approach analyzes code snippets against grammatical rules and structural features to compute interpretable coverage metrics, detect underrepresented constructs, and expose gaps in language support. The proposed framework supports comparative analysis across datasets, languages, and models through an automated pipeline that produces actionable diagnostic reports. Our experimental results demonstrate that syntactic coverage analysis reveals substantial differences in language support that are not captured by functional correctness alone, enabling more informed model selection, benchmark design, and dataset improvement strategies.

Context: Interest in diversity in software development has significantly increased in recent years. Reporting on diversity in software projects can enhance user trust and assist regulators in evaluating adoption. Recent AI directives include clauses that mandate diversity information during development, highlighting the growing interest of public regulators. However, current documentation often neglects diversity in favor of technical features, partly due to a lack of tools for its description and annotation. Objectives: This work introduces the Software Diversity Card, a structured approach to document and share diversity-related aspects within software projects. It aims to profile the various teams involved in software development and governance, including user groups in testing and software adaptations for diverse social groups. Methods: We performed a literature review on diversity and inclusion in software development and an analysis of 1,000 top-starred Open Source Software (OSS) repositories in GitHub to identify diversity-related information. Moreover, we present a diversity modeling language, a toolkit for generating the cards using that language, and a study of its application in two real-world software projects. Results: Despite the growing awareness of diversity in the research community, our analysis found a notable lack of diversity reporting in OSS projects. Applying the card to real-world examples highlighted challenges like balancing anonymity with transparency, managing sensitive data, and ensuring authenticity. Conclusion: We believe that our proposal can enhance diversity practices in software development, support public administrations in software assessment, and help businesses promote diversity as a key asset.

Large language models (LLMs) can be used to support software development tasks, e.g. , through code completion or code generation. However, their effectiveness drops significantly when considering less popular programming languages such as domain-specific languages (DSLs). In this paper, we propose a generic framework for evaluating the capabilities of LLMs generating DSL code from textual specifications. The generated code is assessed from the perspectives of well-formedness and correctness. This framework is applied to a particular type of DSL, constraint languages, focusing our experiments on OCL and Alloy and comparing their results to those achieved for Python, a popular general-purpose programming language. Experimental results show that, in general, LLMs have better performance for Python than for OCL and Alloy. LLMs with smaller context windows such as open-source LLMs may be unable to generate constraint-related code, as this requires managing both the constraint and the domain model where it is defined. Moreover, some improvements to the code generation process such as code repair (asking an LLM to fix incorrect code) or multiple attempts (generating several candidates for each coding task) can improve the quality of the generated code. Meanwhile, other decisions like the choice of a prompt template have less impact. All these dimensions can be systematically analyzed using our evaluation framework, making it possible to decide the most effective way to set up code generation for a particular type of task.

Advances in online text-to-image (T2I) generator models allow users and organizations alike to generate millions of images from text prompts. Unfortunately, recent studies have demonstrated how elementary prompts result in displaying noticeable social biases in the models' output imagery. The potential representational harm of T2I models could lead to further marginalize minority groups. Thus, a systematic approach is required to comprehensively and continuously assess the absence of bias in T2I generators. In this paper, we present ImageBiTe, a framework to systematically and thoroughly evaluate representational discrimination in T2I-generated images, seamlessly integrable into AI engineering processes. ImageBiTe enables development teams to customize their test scenarios and automatically create and run test cases based on user-defined non-discrimination requirements. We have implemented an open-source tool, available on GitHub, to support the application of our approach.

Abstract Generative artificial intelligence (AI) systems are capable of synthesizing complex artifacts such as text, source code or images according to the instructions provided in a natural language prompt. The quality of the input prompt, in terms of both content and structure, has a large impact on the quality of the output. This has given rise to prompt engineering , the process of designing natural language prompts to best take advantage of the capabilities of generative AI systems. This paper describes , a model-driven engineering framework to support the creation, management and reuse of prompts for generative AI. offers a domain-specific language (DSL) to define multimodal prompts in a modular and tool-independent way. The language offers additional features such as versioning, prompt chaining and multi-language support. Moreover, it provides tool support to adapt prompts for specific generative AI systems, execute those prompts on a generative AI system and validate the quality of the response that is generated. is available as a Langium-based Visual Studio Code plugin.

Abstract At its core, OCL as currently defined is a first-order functional language: its expressions evaluate to single values, with collections accounting for multitudes of values, and special values and for partiality. By contrast, the data model providing the context of OCL expressions is inherently relational: the associations of UML class diagrams are essentially relations, with uniqueness and order designators extending expressiveness to ordered multirelations. As a result, OCL suffers from a functional/relational impedance mismatch, which is only superficially addressed by its navigation shorthands. At the same time, OCL is inherently unsafe: expressions containing subexpressions evaluating to may be invalid, translating to a runtime error in programming languages. We address this situation by turning OCL into a relational language that retains most of OCL’s original syntax and semantics, yet revises its fundamental design decisions that lead up to the noted problems. In particular, our version of OCL, which we call OCL $${^\sharp }$$ ♯ , is type-safe.

Large language models (LLMs) are increasingly integrated into software systems to enhance them with generative AI capabilities. But LLMs may reflect a biased behavior, resulting in systems that could discriminate against gender, age or ethnicity, among other ethical concerns. Society and upcoming regulations will force companies and development teams to ensure their AI-enhanced software is ethically fair. To facilitate such ethical assessment, we propose LangBiTe, a model-driven solution to specify ethical requirements, and customize and automate the testing of ethical biases in LLMs. The evaluation can raise awareness on the biases of the LLM-based components of the system and/or trigger a change in the LLM of choice based on the requirements of that particular application. The model-driven approach makes both the requirements specification and the test generation platform-independent, and provides end-to-end traceability between the requirements and their assessment. We have implemented an open-source tool set, available on GitHub, to support the application of our approach.

Large Language Models (LLMs) are being quickly integrated in a myriad of software applications. This may introduce a number of biases, such as gender, age or ethnicity, in the behavior of such applications. To face this challenge, we explore the automatic generation of tests suites to assess the potential biases of an LLM. Each test is defined as a prompt used as input to the LLM and a test oracle that analyses the LLM output to detect the presence of biases.

There are numerous domains in which information systems need to deal with uncertain information. These uncertainties may originate from different reasons such as vagueness, imprecision, incompleteness, or inconsistencies, and in many cases, they cannot be neglected. In this article, we are interested in representing and processing uncertain information in domain models, considering the stakeholders’ beliefs (opinions). We show how to associate beliefs to model elements and how to propagate and operate with their associated uncertainty so that domain experts can individually reason about their models enriched with their personal opinions. In addition, we address the challenge of combining the opinions of different domain experts on the same model elements, with the goal to come up with informed collective decisions. We provide different strategies and a methodology to optimally merge individual opinions.

Model finders can build instances of declarative specifications that satisfy a set of correctness constraints. Some model finders ensure some degree of diversity among the instances they compute. Nevertheless, each model finder uses its own definition of diversity, that may or may not match designer intent. In this paper, we propose a procedure that enables designers to capture the desired notion of diversity they are looking for. Using a simple domain-specific language, they can specify what elements in the specification are relevant when comparing the differences between two instances. This information can then be used to make any model finder diversity-aware while using it as a black box. As a proof of concept, this approach has been implemented on top of the Alloy Analyzer.

Participation inequality is a major challenge in any shared-resource system. This is known as the "volunteer's dilemma": everybody wants to benefit from a resource without contributing, expecting others will do the work. This paper explores whether this problem also arises in open source development. In particular, we analyze the behaviour of GitHub users to assess whether the 90-9-1 principle applies to open source. We study it both from a qualitative (ratio of activity types) and a quantitative (total number of activities) perspective and we show that the principle does not hold if we consider the GitHub platform as a whole. Surprisingly, results are reversed depending on the specific projects we look at. We believe these results are useful to project managers to better understand and optimize the behaviour of the community around their projects and, as a side effect, they show the importance of diversity in sample selection.

This paper discusses the representation of Belief Uncertainty in software models. This kind of uncertainty refers to the situation in which the modeler, or any other belief agent, is uncertain about the behavior of the system, or the statements that the model expresses about it. In this work, we propose to assign a degree of belief to model statements (let they be constraints, or any other model expression), which is expressed by a probability (called credence, in statistical terms) that represents a quantification of such a subjective degree of belief. We discuss how it can be represented using current modeling notations, and how to operate with it in order to make informed decisions.

The network of collaborations in an open source project can reveal relevant emergent properties that influence its prospects of success. In this work, we analyze open source projects to determine whether they exhibit a rich-club behavior, i.e., a phenomenon where contributors with a high number of collaborations (i.e., strongly connected within the collaboration network) are likely to cooperate with other well-connected individuals. The presence or absence of a rich-club has an impact on the sustainability and robustness of the project.

Correctness of UML class diagrams annotated with OCL constraints can be checked using bounded verification techniques, e.g., SAT or constraint programming (CP) solvers. Bounded verification detects faults efficiently but, on the other hand, the absence of faults does not guarantee a correct behavior outside the bounded domain. Hence, choosing suitable bounds is a non-trivial process as there is a trade-off between the verification time (faster for smaller domains) and the confidence in the result (better for larger domains). Unfortunately, bounded verification tools provide little support in the bound selection process. In this paper, we present a technique that can be used to (i) automatically infer verification bounds whenever possible, (ii) tighten a set of bounds proposed by the user and (iii) guide the user in the bound selection process. This approach may increase the usability of UML/OCL bounded verification tools and improve the efficiency of the verification process.

The specification of a family of software products may include documents written in natural language. Automatically extracting knowledge from these documents is a challenging problem that requires using Natural Language Processing (NLP) techniques. This knowledge can be formalized as a Feature Model (FM), a diagram capturing the key features and the relationships among them.

Model transformations are used to automate model manipulation in Model-Driven Engineering (MDE). In particular, model-to-model transformations produce target models (conformant to a target meta-model) from source ones (conformant to a source meta-model). While transformation correctness is crucial in MDE, developing transformations is error-prone due to the difficulty in testing them. This problem is further aggravated if the source and target meta-models contain OCL integrity constraints, as every transformed source model should satisfy the target integrity constraints.In order to attack this problem, we present a novel method that translates target OCL constraints to the source meta-model using the transformation definition. This way, if a source model satisfies the advanced constraint, the transformed model will satisfy the target constraint. The method has been implemented for the ATL transformation language and integrated with the anATLyzer tool. We show its benefits in combination with model finders, and the promising results of its validation using mutation techniques and transformations developed by third parties.

Domain experts typically have detailed knowledge of the concepts that are used in their domain; however they often lack the technical skills needed to translate that knowledge into model-driven engineering (MDE) idioms and technologies. Flexible or bottom-up modelling has been introduced to assist with the involvement of domain experts by promoting the use of simple drawing tools. In traditional MDE the engineering process starts with the definition of a metamodel which is used for the instantiation of models. In bottom-up MDE example models are defined at the beginning, letting the domain experts and language engineers focus on expressing the concepts rather than spending time on technical details of the metamodelling infrastructure. The metamodel is then created manually or inferred automatically. The flexibility that bottom-up MDE offers comes with the cost of having nodes in the example models left untyped. As a result, concepts that might be important for the definition of the domain will be ignored while the example models cannot be adequately re-used in future iterations of the language definition process. In this paper, we propose a novel approach that assists in the inference of the types of untyped model elements using Constraint Programming. We evaluate the proposed approach in a number of example models to identify the performance of the prediction mechanism and the benefits it offers. The reduction in the effort needed to complete the missing types reaches up to 91.45% compared to the scenario where the language engineers had to identify and complete the types without guidance.

Nowadays, many MOOC platforms have arisen to provide free knowledge. These platforms have a large catalog of courses for different specializations that progressively demand more specific learning resources and assessment methods to evaluate the progression of students. Current MOOC platforms are gradually giving support to these new requirements but with a limited assistance. This paper presents the state of art of the analytical system for three relevant MOOC platforms, one of the main pillars for analyzing the progression of courses. Other initiatives are also reviewed to show that current MOOC analytical systems are not ready to support custom MOOC-aware intelligent tutoring systems (ITSs). Thus, the design of a learning analytics system to assist these tools for MOOC platforms is presented.

This paper presents the analysis and design of an innovative web-based eLearning platform featuring Learning Analytics and Gamification called ICT-FLAG. A previous contribution presented the software development framework that will guide and support the effective construction of the platform. Following the selected framework, this paper defines the requirements of this platform and propose a solution to satisfy those requirements in terms of software artifacts. The design includes challenging non-functional requirements, such as scalability, security and performance. The research reported in this paper is currently undertaken within the project "Enhancing ICT education through Formative assessment, Learning Analytics and Gamification" (ICT-FLAG) funded by the Spanish Government.

Model defects are a significant concern in the Model-Driven Development (MDD) paradigm, as model transformations and code generation may propagate errors to other notations where they are harder to detect and trace. Formal verification techniques can check the correctness of a model, but their high computational complexity can limit their scalability. In this paper, we consider a specific static model (UML class diagrams annotated with unrestricted OCL constraints) and a specific property to verify (satisfiability, i.e., "is it possible to create objects without violating any constraint?"). Current approaches to this problem have an exponential worst-case runtime. We propose a technique to improve their scalability by partitioning the original model into submodels (slices) which can be verified independently and where irrelevant information has been abstracted. The definition of the slicing procedure ensures that the property under verification is preserved after partitioning.

We present UMLtoCSP, a tool for the formal verification of UML/OCL models. Given a UML class diagram annotated with OCL constraints, UMLtoCSP is able to automatically check several correctness properties, such as the strong and weak satisfiability of the model or the lack of redundant constraints. The tool uses Constraint Logic Programming as the underlying formalism and the constraint solver ECLiPSe as the verification engine.