Training Large Language Models (LLMs) with synthetic data is a prevalent practice in code generation. A key approach is self-training, where LLMs are iteratively trained on self-generated correct code snippets. In this case, the self-generated codes are drawn from a conditional distribution, conditioned on a specific seed description. However, the seed description is not the only valid representation that aligns with its intended meaning. With all valid descriptions and codes forming a joint space, codes drawn from the conditional distribution would lead to an underrepresentation of the full description-code space. As such, we propose Gibbs Fine-Tuning (GiFT), a novel self-training method inspired by Gibbs sampling. GiFT allows self-generated data to be drawn from the marginal distribution of the joint space, thereby mitigating the biases inherent in conditional sampling. We provide a theoretical analysis demonstrating the potential benefits of fine-tuning LLMs with code derived from the marginal distribution. Furthermore, we propose a perplexity-based code selection method to mitigate the imbalanced long-tail distribution of the self-generated codes. Empirical evaluation of two LLMs across four datasets demonstrates that GiFT achieves superior performance, particularly on more challenging benchmarks. Source code is available at https://github.com/Alex-HaochenLi/GiFT.

Generated Exemplar Code: def count_frequency(my_list): frequency = {} for element in my_list: if element not in frequency: frequency[element] = 0 frequency[element] += 1 return frequency True Code: import collections def freq_count(list1): freq_count = collections.Counter(list1) return freq_count Query: Write a function to get the frequency of the elements in a list.Query: How was the COVID-19 pandemic impacted mental health?True Passage: …two studies investigating COVID-19 patients … significantly higher level of depressive… Generated Reference: …depression and anxiety had increased by 20% since the start of the pandemic… Passage Retrieval Code Search

Recently, contrastive learning has become a key component in fine-tuning code search models for software development efficiency and effectiveness.It pulls together positive code snippets while pushing negative samples away given search queries.Among contrastive learning, InfoNCE is the most widely used loss function due to its better performance.However, the following problems in negative samples of In-foNCE may deteriorate its representation learning: 1) The existence of false negative samples in large code corpora due to duplications.2).The failure to explicitly differentiate between the potential relevance of negative samples.As an example, a bubble sorting algorithm example is less "negative" than a file saving function for the quick sorting algorithm query.In this paper, we tackle the above problems by proposing a simple yet effective Soft-InfoNCE loss that inserts weight terms into InfoNCE.In our proposed loss function, we apply three methods to estimate the weights of negative pairs and show that the vanilla InfoNCE loss is a special case of Soft-InfoNCE.Theoretically, we analyze the effects of Soft-InfoNCE on controlling the distribution of learnt code representations and on deducing a more precise mutual information estimation.We furthermore discuss the superiority of proposed loss functions with other design alternatives.Extensive experiments demonstrate the effectiveness of Soft-InfoNCE and weights estimation methods under state-of-the-art code search models on a large-scale public dataset consisting of six programming languages.

Code search, which aims at retrieving the most relevant code fragment for a given natural language query, is a common activity in software development practice.Recently, contrastive learning is widely used in code search research, where many data augmentation approaches for source code (e.g., semantic-preserving program transformation) are proposed to learn better representations.However, these augmentations are at the raw-data level, which requires additional code analysis in the preprocessing stage and additional training costs in the training stage.In this paper, we explore augmentation methods that augment data (both code and query) at representation level which does not require additional data processing and training, and based on this we propose a general format of representationlevel augmentation that unifies existing methods.Then, we propose three new augmentation methods (linear extrapolation, binary interpolation, and Gaussian scaling) based on the general format.Furthermore, we theoretically analyze the advantages of the proposed augmentation methods over traditional contrastive learning methods on code search.We experimentally evaluate the proposed representationlevel augmentation methods with state-of-theart code search models on a large-scale public dataset consisting of six programming languages.The experimental results show that our approach can consistently boost the performance of the studied code search models.