Unsupervised anomaly detection in multivariate time series (MTS) has always been a challenging problem, and the modeling based on reconstruction has garnered significant attention. The insensitivity of these methods towards normal patterns poses challenges in distinguishing between normal and abnormal points. Firstly, the general reconstruction strategies may exhibit limited sensitivity to spatio-temporal dependencies, and their performance remains largely unaffected by such dependencies. Secondly, most methods fail to model the heteroscedastic uncertainty in MTS, hindering their abilities to derive a distinguishable criterion. For instance, normal data with high noise levels may lead to detection failure due to excessively high reconstruction errors. In this work, we emphasize the necessity of sensitivity to normal patterns, which could improve the discrimination between normal and abnormal points remarkably. To this end, we propose SensitiveHUE, a probabilistic network by implementing both reconstruction and heteroscedastic uncertainty estimation. Its core includes a statistical feature removal strategy to ensure the dependency sensitive property, and a novel MTS-NLL loss for modeling the normal patterns in important regions. Experimental results demonstrate that SensitiveHUE exhibits nontrivial sensitivity to normal patterns and outperforms the existing state-of-the-art alternatives by a large margin. Code is publicly available at this URL\footnotehttp://github.com/yuesuoqingqiu/SensitiveHUE.