Due to the high coverage and low cost nature of trajectory data, an increasing number of works have utilized trajectory data to infer maps. Nevertheless, limited by the sparse trajectories in some areas and intermingled trajectories on parallel roads, the existing inferring methods still face a high missed detection rate of the roads. In view of that, we propose a Complementarity and Disparity-aware Map Inference Framework, called CDMap, consisting of grid dual feature extraction, contextual road difference-embedded grid representation, dual feature complementary network-based road topology prediction and parallel roads disparity-enhanced model optimization. To improve the prediction accuracy of the roads in areas with sparse trajectories, we extract point-wise features and segment-wise features separately for the grids, then design a dual feature complementary network to adaptively model the importance of both types of features in different road scenarios. Further, to proliferate the detection accuracy of parallel roads, we incorporate the contextual roads' differences between parallel roads into grid representations, then put forward a parallel roads disparity-enhanced model optimization strategy. Extensive comparative experiments conducted on three real-world datasets demonstrate the superiority of CDMap over the state-of-the-art methods, especially by achieving the most significant reduction in missed detection rate (30.23%) on the trajectory data collected from DidiChuxing platform.