Nowadays, artificial intelligence (AI)-based voice conversion (VC) models generate high-quality and natural-sounding voices efficiently. While existing studies have focused on detecting voice spoofing attacks, this study explores a proactive approach to safeguard speaker identity from such threats. Attackers can exploit voice biometrics by using either advanced VC techniques or traditional methods. Although zero-shot VC has been applied for source speaker anonymization, it typically fails to protect the identity of the target speaker. To address this gap, we propose a speaker privacy preservation framework that combines zero-shot AutoVC with a multi-target voice frame fusion technique. This method anonymizes the source speaker while also preventing leakage of the target speaker’s identity by blending voice frames from multiple targets. The VCTK public dataset is employed to assess the performance of the proposed model. The average cosine similarity between the original and anonymized voices yielded a result closer to the inter-speaker voice cosine similarity, which is the most suitable benchmark for speaker anonymization. Additionally, speaker identification models trained on original and vocoded data yielded low identification accuracies of 27.4% and 41.2%, respectively. The mean opinion score (MOS) evaluation confirmed that the anonymized voice quality adeptly preserved linguistic content and naturalness. Overall, the proposed method effectively anonymizes both source and target speakers, offering robust protection against voiceprint spoofing.