Text-Based Person Search (TBPS) aims to retrieve target pedestrian images through language descriptions. However, the visual attributes and textual descriptions of different identities (pedestrians) tend to exhibit considerable similarity, leading to Similar Semantic Interference (SSI). To mitigate this issue, we propose the Adapting Cross-Modal Semantic Discrepancy (ACMSD) method, employing a cross-modal constraint approach to alleviate interference in model training. Specifically, we introduce the Consistent Constraint Alignment (CCA) strategy, which establishes both inter-modal alignment and intra-modal alignment, along with an Identity-Balanced Distribution (IBD) loss. This paradigm utilizes Cyclic Image-Text Contrastive to regularize the spatial distribution of the modalities, while the IBD loss implicitly clusters strong positive samples by using identity as a key index. Additionally, we incorporate an Attention-based Implicit Alignment (AIA) module to enforce modality-specific embeddings, thereby strengthening the interaction between cross-modal information. Extensive experiments are conducted on three public benchmark datasets to evaluate the performance of the ACMSD method.