Robust rigid point cloud registration is effective for accurate positioning and measurement of complex components. The existing registration algorithms, however, fail to overcome the matching distortion caused by structural deviation, unknown abnormal allowance, and various measurement inherent defects. Although the recently proposed VMM and WPMAVM algorithms can inhibit the matching distortion to some extent, they still fail in the presence of numerous abnormal points. In this study, we present a progressive and adaptive variance minimization (PAVM) algorithm to address these issues. A progressive de-pseudo weight is established to ensure the involvement of all point pairs in optimization at the initial registration stage. Then, an approximately truncated weight function is employed to mitigate the influence of abnormal points on registration results. Furthermore, a novel adaptive coordination distance function is established by improving the symmetric point-to-plane distance metric and combining the first-order approximate point-to-point distance metric, which enhances the algorithm speed and stability. The analysis investigates the anti-abnormal interference ability and quadratic convergence, validating the feasibility of the PAVM algorithm. Experiments are undertaken to illustrate the notable benefits of our algorithm in convergence stability, matching speed, and universality. These attributes render the algorithm well-suited for registration tasks involving diverse complex components.