Weakly Supervised Object Localization (WSOL) relies only on image-level labels to realize object localization, significantly reducing the cost for fine-grained annotations. While traditional CAM-based methods excel at identifying the most prominent regions of objects, they frequently neglect other essential components, resulting in partial or incomplete object localization. The foreground prediction map (FPM) generates finer-grained activation maps using underlying features to address the shortcomings of CAM, but it may still have coverage blind spots. To this end, this paper proposes a collaborative optimization framework based on cross-modal semantic alignment that deeply integrates the saliency awareness of CAM with the refined representation capabilities of FPM. It introduces a multimodal pretrained model (CLIP) to construct a semantic-driven WSOL paradigm. By dynamically interacting CLIP's text embeddings with the semantic of image categories, a semantic-enhanced FPM based on similarity measurement is generated. Leveraging CLIP's cross-modal alignment capabilities, a targeted generation scheme is designed. On the one hand, the CLIP model is frozen and its features are refined through a decoder to obtain richer semantic representations; On the other hand, by using knowledge distillation, the CAM generated by CLIP is taken as a reference benchmark, guiding the network to learn more accurate target localization. Additionally, to enhance FPM's focus on foreground regions, the Exponential Decay Foreground Emphasis (EDFE) module is designed, which uses a differentiated excitation strategy to effectively suppress background interference and highlight target areas. Experimental results show that our method significantly improves the completeness and boundary accuracy of target localization under weak supervision, laying a solid foundation for subsequent downstream tasks.