Traditional fine-grained image classification relies extensively on large-scale training samples with annotated ground truth. However, in the real world, some fine-grained categories are represented by only a few available images, and existing few-shot models struggle to distinguish the subtle differences among them. Furthermore, the challenge is compounded by the fact that intra-class distances for some fine-grained categories can be significantly large, whereas inter-class distances might be minimal, leading to distinct task-specific distinguishing features for each category. To address these challenges, we propose a novel network, FicNet, utilizing Multi-Frequency Neighborhood (MFN) and Double-Cross Modulation (DCM). MFN captures a multi-frequency structure representation that is independent of the background by integrating spatial and frequency domain information, reducing intra-class distances. Concurrently, DCM modulates the representation through global context and inter-class relationships, enabling both support and query features to align with complete targets and respond to identical parts. This approach facilitates the accurate identification of subtle inter-class differences. Comprehensive experiments conducted on three fine-grained benchmark datasets for two few-shot tasks have verified that FicNet exhibits exceptional performance compared to state-of-the-art methods. Notably, it achieves classification accuracies of 93.17% and 95.36% on the “Caltech-UCSD Birds” and “Stanford Cars” datasets, respectively, surpassing the benchmarks set by general fine-grained image classification methods.