Large Language Models (LLMs) are extensively used nowadays for various software engineering tasks, primarily code generation. Previous research has shown how suitable prompt engineering can help developers improve their code-generation prompts. However, so far, there are no specific guidelines guiding developers in writing suitable prompts for code generation. In this work, we derive and evaluate development-specific prompt optimization guidelines. We use an iterative, test-driven approach to automatically refine code-generation prompts, and we analyze the results of this process to identify prompt improvement items that lead to test passes. We use such elements to elicit 10 guidelines for prompt improvement, related to better specifying I/O, pre/post conditions, providing examples, various types of details, or clarifying ambiguities. We conducted an assessment with 50 practitioners, who reported their use of the elicited prompt improvement patterns and their perceived usefulness. Our results have implications not only for practitioners and educators, but also for those creating LLM-aided software development tools.

Deep learning (DL)-based code completion tools have revolutionized software development by providing unprecedented code generation capabilities. The DL models behind these tools are usually trained on large amounts of code from thousands of software repositories. This makes them good in learning natural coding patterns observed across many training instances. However, little is known about the extent to which additional training effort (fine-tuning) aimed at specializing the models towards the code base of a given organization/developer further benefits their code completion capabilities. In this work, we fill this gap by presenting solid empirical evidence answering this question. More specifically, we consider 136 developers from two organizations (Apache and Spring), two model architectures (T5 and Code Llama), and three model sizes (60M, 750M, and 7B trainable parameters). For T5 models (60M and 750M), we pre-train and fine-tune them on over 2,000 open source projects, making sure that code from the two subject organizations is not part of their training sets. Then, we compare their completion capabilities against the same models further fine-tuned on organization- and developer-specific datasets. For the Code Llama model (7B), we compare the performance of the already pre-trained model publicly available online with the same model fine-tuned via parameter-efficient fine-tuning on organization- and developer-specific datasets. Our results show that there is a boost in prediction capabilities provided by both an organization-specific and a developer-specific additional fine-tuning, with the former being particularly performant. Such a finding generalizes across (i) the two subject organizations (i.e., Apache and Spring) and (ii) models of completely different magnitude (from 60M to 7B trainable parameters). Finally, we show that DL models fine-tuned on an organization-specific dataset achieve the same completion performance of pre-trained code models used out of the box and being \(\sim\) 10 \(\times\) larger, with consequent savings in terms of deployment and inference cost (e.g., smaller GPUs needed).

The advent of Large Language Models (LLMs) has significantly advanced the field of automated code generation. LLMs rely on large and diverse datasets to learn syntax, semantics, and usage patterns of programming languages. For low-resource languages (i.e., niche programming languages characterized by the scarcity of training data), the limited availability of such data hampers the models' ability to generalize effectively, resulting in poorer code generation performance as compared to high-resource languages. For this reason, there is a quest for techniques able to close this performance gap. We present an empirical study investigating the effectiveness of several approaches for boosting LLMs' performance on low-resource languages, namely: (i) a classic fine-tuning, which is however capped in size by the scarcity of training data; (ii) three variants of in-context learning, with prompts crafted to provide the LLM with additional information about the low-resource language (e.g., few-shot examples showcasing features of the targeted language); and (iii) a pre-training objective teaching the model how to translate between high- and low-resource languages. The context of our study are two low-resource languages (R and Racket) and six LLMs having different architectures and sizes. Our findings reveal that a fine-tuning is usually the best choice for smaller LLMs, possibly due to the fact that even a small dataset is sufficient to train their limited number of parameters. With the increase in size of the models, in-context learning becomes more and more effective, representing a safe and cheap bet (i.e., it always helps, but with different magnitudes). Differently, very large LLMs may deteriorate their performance on low-resource languages when fine-tuning is performed, possibly due to the lack of enough data needed to effectively update their weights.