Multispectral object detection, which combines RGB visible light and thermal infrared spectral information, has broad applications in complex environments and varying illumination conditions. However, existing methods face challenges in processing multispectral data, such as inconspicuous object features in spectral images and significant discrepancies between input modality spaces and output detection spaces. To address these issues, we propose an innovative multispectral object detection method that combines contrastive learning and a new cross-modal feature fusion module. We introduce a mask feature contrastive loss that maximizes the similarity between the box-level mask features and modal features while suppressing background responses, enabling effective representative alignment between the input and output spaces. Additionally, we propose a mask-guided attention fusion module that uses a predicted pseudo mask to guide the fusion of different modal features, enhancing object responses and reducing background noise interference. Our extensive experiments on several challenging multispectral datasets demonstrate that our proposed COFNet achieves state-of-the-art performance.