The primary challenge in text-video retrieval lies in achieving cross-modal semantic alignment, particularly the discrepancy between the conciseness of textual descriptions, which often fail to fully encapsulate the breadth of video content, and the redundancy in video data, which introduces noise and masks important semantic features. Current methods align text and video by mapping them into a shared feature space. Despite notable advancements, the inherent differences in modality-specific representations create a bottleneck for fixed-point embedding techniques, making models highly sensitive to dataset distribution and hindering their generalization ability. In this paper, we present DomainDiff, a framework that enhances the embedding space through a two-stage process. In the first stage, stochastic domain modeling, we semantically expand text embeddings to explore potential regions aligned with video content. Simultaneously, we filter video segments to reduce redundancy and highlight key frames. In the second stage, the dynamic agent attention diffusion network, we leverage the generative properties of diffusion models to optimize the embedding space by viewing it from a joint probability distribution perspective. An agent attention mechanism dynamically integrates text and video features, ensuring accurate cross-modal alignment. Experimental results demonstrate that DomainDiff significantly improves retrieval performance across five benchmark datasets, with R@1 improvements ranging from 3% to 7.4%. Moreover, DomainDiff outperforms existing methods in handling long videos and complex textual descriptions, showcasing superior semantic robustness and generalization across varying distributions.