Medical contrastive Vision-Language Pre-training (VLP) has emerged as a promising approach, enabling models to learn joint representations from paired medical images and radiology reports. Despite existing methods exploring local visual representation learning techniques, they often fall short in local alignment and knowledge infusion, e.g., uniform token treatment and isolated knowledge assignment. To address these issues, we propose a novel Fine-grained Knowledge-Guided Alignment (FKGA) framework for medical VLP. Specifically, we propose a Fine-grained Disease Knowledge Integration (FDKI) module to inject detailed disease descriptions into corresponding disease tokens in reports. Based on these semantic-enriched tokens, we introduce global instance-wise and local token-wise contrastive learning to further align the semantically related visual and textual modalities. In contrast to previous local visual representation learning methods, our design of semantic-enriched token alignment and context-preserved knowledge infusion enhances the semantic understanding of diseases. Extensive experimental results on five downstream tasks demonstrate that our proposed method outperforms other state-of-the-art methods across seven datasets.

Existing medical vision-language contrastive pretraining methods aim to bring the paired image-report embeddings close together while pushing the unpaired ones apart. However, medical images often exhibit high inter-class visual similarity with only subtle differences, leading to the presence of hard negative samples that are semantically distinct from the anchor but incorrectly close to it in the embedding space, making it challenging to distinguish semantically dissimilar samples. Previous methods consider only the embedding similarity between samples to identify hard negatives, often wrongly treating false negatives as hard negatives. To address this issue, we design a simple yet effective approach called Semantic-Aware Hard Negative mining (SAHN), distinguishing hard negatives from false negatives and encouraging the model to pay greater attention to hard negatives. Specifically, hard negatives are identified as samples with high embedding similarity but low semantic similarity to the anchor and assigned greater importance weights. By integrating these importance weights into the InfoNCE loss, SAHN enhances the model's ability to separate semantically dissimilar samples while clustering semantically similar ones. We further conduct a gradient-based theoretical analysis to validate the effectiveness of SAHN. Extensive experimental results on four downstream medical tasks covering image classification, object detection, semantic segmentation, and cross-modal retrieval demonstrate the superiority of our approach.