The recent development of single-cell genomics requires more powerful computational tools to differentiate between different phenotypes, and rare cell identification has been one of the most important problems in this area. Recent data-driven approaches usually adopt feature selection techniques to identify important genes for anomaly detection, which require extensive training data or domain knowledge from experts. In comparison, large language models (LLMs) have achieved certain progress in scientific research with strong generalization ability, which has shown potential in this area. In this paper, we make an attempt to comprehensively evaluate the performance of in-context learning with LLMs in rare cell identification. In particular, we carefully design a chain-of-thought prompt combining token probability analysis and cross-query comparison to generate scores to identify rare cells. From the experimental results on benchmark datasets, we find that LLMs are competitive compared to existing training-based methods, which demonstrates extensive potential for rare cell identification. Our source code and data are available at https://github.com/Hyan-Yao/InContextCells/.