Cuff-less methods, especially pulse wave analysis (PWA) techniques with PPG/mmWave sensing, have shown great potential for non-intrusive blood pressure (BP) monitoring. However, the state-of-the-art solutions are only validated on small-scale healthy subjects, neglecting patients with abnormal BP and thus a more urgent need for BP monitoring. To bridge the gap, we first build the largest mmWave-BP dataset to our knowledge, including 930 real patients with cardiovascular diseases, and perform extensive experiments, which reveals that all existing PWA methods exhibit far less satisfactory performance with standard deviation errors (STD) exceeding 16 mmHg for systolic BP (SBP) and 11mmHg for diastolic BP (DBP). An in-depth investigation shows that physiological factors have complex effect on vascular elasticity and structure, thus people with very different BP values may exhibit extremely similar pulse waveform, which leads to confusion in model learning. In this work, we propose BP3, which fuses physiological factors into sensing-data-driven deep-learning framework, so as to capture the intricate effect of physiological factors during the whole process of learning pulse waveforms. Evaluation results show that BP3 achieves the mean errors of-1.57 mmHg and -0.34 mmHg, STD of 9.77 mmHg and 7.93 mmHg for SBP and DBP, respectively. Moreover importantly, BP3 shows remarkable gain particularly for subjects with abnormal BP, achieving mean errors that are only 0.48% ~ 20.86% of the state-of-the-art solutions.