Voice conversion refers to transforming the speaker of a voice into a target speaker while keeping the content unchanged. Current solutions either rely on feature representations from large-scale pre-trained models or require complex model designs and intensive training, lacking exploration of intrinsic speech features. This restricts the exploration of lightweight and robust methods. In this study, we remove pre-trained models and depart from complex mutual information minimization for feature decoupling. Instead, we revisit decoupling methods based on instance normalization. To address it, we introduce a novel feature coupling module named cross-adaptive instance normalization (CAIN), which extends the adaptive instance normalization (AdaIN). Beyond offering style injection capabilities, CAIN is designed to maintain content consistency by reconstructing frame-level statistics in mel-spectrograms. The results indicate that CAIN, serving as a lightweight plugin, significantly improves conventional instance normalization-driven approaches. Building upon this, we propose RLVC, which achieves robust performance with a mere 5.29M parameters.