Large Language Models (LLMs) have demonstrated remarkable success in various natural language processing and software engineering tasks, such as code generation. The LLMs are mainly utilized in the prompt-based zero/few-shot paradigm to guide the model in accomplishing the task. GPT-based models are one of the popular ones studied for tasks such as code comment generation or test generation. These tasks are 'generative' tasks. However, there is limited research on the usage of LLMs for 'non-generative' tasks such as classification using the prompt-based paradigm. In this preliminary exploratory study, we investigated the applicability of LLMs for Code Clone Detection (CCD), a non-generative task. By building a mono-lingual and cross-lingual CCD dataset derived from CodeNet, we first investigated two different prompts using ChatGPT to detect Type-4 code clones in Java-Java and Java-Ruby pairs in a zero-shot setting. We then conducted an analysis to understand the strengths and weaknesses of ChatGPT in CCD. ChatGPT surpasses the baselines in cross-language CCD attaining an F1-score of 0.877 and achieves comparable performance to fully fine-tuned models for mono-lingual CCD, with an F1-score of 0.878. Also, the prompt and the difficulty level of the problems has an impact on the performance of ChatGPT. Finally, we provide insights and future directions based on our initial analysis1.

Code Clone Detection (CCD) is a software engineering task that is used for plagiarism detection, code search, and code comprehension. Recently, deep learning-based models have achieved an F1-Score (a metric used to assess classifiers) of $$\sim $$ 95% on the CodeXGLUE benchmark. These models require many training data, mainly fine-tuned on Java or C++ datasets. However, no previous study evaluates the generalizability of these models where a limited amount of annotated data is available. The main objective of this research is to assess the ability of the CCD models as well as few-shot learning algorithms for unseen programming problems and new languages (i.e., the model is not trained on these problems/languages). We assess the generalizability of the state-of-the-art models for CCD in few-shot settings (i.e., only a few samples are available for fine-tuning) by setting three scenarios: i) unseen problems, ii) unseen languages, iii) combination of new languages and new problems. We choose CodeNet and conduct our experiments on Java, C++, and Ruby languages. Then, we employ Model Agnostic Meta-learning (MAML), where the model learns a meta-learner capable of extracting transferable knowledge from the train set; so that the model can be fine-tuned using a few samples. Finally, we combine contrastive learning with MAML to further study whether it can improve the results of MAML. Our results show that the performance of the models drops $$\sim 50\%$$ for Java and $$\sim 20\%$$ for C++ and Ruby for unseen problems, which are then boosted by $$13\%$$ to $$24\%$$ F1 scores for Java and C++/Ruby, respectively when MAML is used. Similar observations are found for unseen languages and the third scenario. Though in case of third scenario (i.e., unseen problems and unseen languages) the scores are lower. Integrating contrastive learning with MAML did not help in boosting the performance more than what we could achieve with MAML. Our results open new avenues of research and the need to develop robust models for clone detection, in the settings we investigated here.