Large Language Models (LLMs) are increasingly used for code generation, yet their support for programming languages is uneven, particularly for low-resource languages and infrequently used language constructs. Existing evaluation methodologies primarily assess functional correctness through test execution, which fails to reveal blind spots in a model's knowledge of a target language. In this paper, we introduce a systematic framework to identify and quantify the syntactic knowledge exhibited by an LLM with respect to a given programming language. We define syntactic coverage as a set of measurable indicators of how comprehensively language constructs are represented in datasets and generated code. Our approach analyzes code snippets against grammatical rules and structural features to compute interpretable coverage metrics, detect underrepresented constructs, and expose gaps in language support. The proposed framework supports comparative analysis across datasets, languages, and models through an automated pipeline that produces actionable diagnostic reports. Our experimental results demonstrate that syntactic coverage analysis reveals substantial differences in language support that are not captured by functional correctness alone, enabling more informed model selection, benchmark design, and dataset improvement strategies.

Large language models (LLMs) can be used to support software development tasks, e.g. , through code completion or code generation. However, their effectiveness drops significantly when considering less popular programming languages such as domain-specific languages (DSLs). In this paper, we propose a generic framework for evaluating the capabilities of LLMs generating DSL code from textual specifications. The generated code is assessed from the perspectives of well-formedness and correctness. This framework is applied to a particular type of DSL, constraint languages, focusing our experiments on OCL and Alloy and comparing their results to those achieved for Python, a popular general-purpose programming language. Experimental results show that, in general, LLMs have better performance for Python than for OCL and Alloy. LLMs with smaller context windows such as open-source LLMs may be unable to generate constraint-related code, as this requires managing both the constraint and the domain model where it is defined. Moreover, some improvements to the code generation process such as code repair (asking an LLM to fix incorrect code) or multiple attempts (generating several candidates for each coding task) can improve the quality of the generated code. Meanwhile, other decisions like the choice of a prompt template have less impact. All these dimensions can be systematically analyzed using our evaluation framework, making it possible to decide the most effective way to set up code generation for a particular type of task.