Smart eyeglasses are emerging as a new medium for human-computer interaction. Existing solutions typically rely on cameras or touchpads, raising privacy invasion concerns or requiring users to physically interact with the glass frames. Here, we present, a novel mid-air gesture interaction system for eyeglasses, based on radio frequency identification (RFID). The design of our system involves embedding a conformal RFID tag into the eyeglass frame, where the received signals change with user gestures. We optimize the radiation gain of the tag to avoid view blockage while preserving a long working range. To discriminate different gestures, we propose an adaptively weighted multi-channel fusion network to extract their respective distinctive features. To allow efficient adaptation of the pre-trained network to new users, we also introduce a novel diagonal dot-product attention in our contrastive learning framework to uncover the feature similarities of different users. The proposed system was evaluated through extensive experiments, demonstrating an average recognition accuracy of 98.86% across twenty users and a cross-user accuracy of 98.75%.