In recent years, object detection utilizing both visible (RGB) and thermal infrared (IR) imagery has garnered extensive attention and has been widely implemented across a diverse array of fields. By leveraging the complementary properties between RGB and IR images, the object detection task can achieve reliable and robust object localization across a variety of lighting conditions, from daytime to nighttime environments. While RGB-IR multi-modal data input generally enhances overall detection performance, most existing multi-modal object detection methods fail to fully exploit the complementary potential of these two modalities. We believe that this issue arises not only from the challenges associated with effectively integrating multi-modal information but also from the presence of redundant features in both the RGB and IR modalities. The redundant information of each modality will exacerbate the fusion imprecision problems during propagation. To address this issue, we draw inspiration from the human cognitive mechanisms for processing multi-modal information and propose a novel coarse-to-fine perspective to purify and fuse features from both modalities. Specifically, following this perspective, we design a Redundant Spectrum Removal module to remove interfering information within each modality coarsely and a Dynamic Feature Selection module to finely select the desired features for feature fusion. To verify the effectiveness of the coarse-to-fine fusion strategy, we construct a new object detector called the Removal then Selection Detector (RSDet). Extensive experiments on five RGB-IR object detection datasets verify the superior performance of our method. The source code and results are available athttps://github.com/Zhao-Tian-yi/RSDet.git