Mobile devices have penetrated our daily life, and so have the fast chargers. However, while improving users' charging efficiency, fast chargers may pose severe security threats. In this work, we present a novel side-channel attack called ChargerWhisper, which exploits the acoustic signals generated by fast chargers to infer users' private information in an on-charging mobile device. In particular, when users perform different activities on the charging device, the fast charger will supply distinct amounts of power outputs to the mobile device. By conducting a deep investigation of the fast charger circuits, we find that the electronic components, i.e., inductors, capacitors, and transformers, vibrate at certain frequencies and generate high-frequency inaudible sounds. Meanwhile, the leaked sound frequencies are highly correlated with electric intensity flowing through these electronic components (charger's output power). To demonstrate the acoustic side-channel attack leveraging fast chargers, we present two specific exploitation scenarios: a website fingerprinting attack to identify users' on-browsing websites, and a PIN inference attack to infers the device's unlock PIN. Extensive evaluations are performed on off-the-shelf devices. The results show that both attacks are robust under diverse usage settings and achieve good performance in inferring users' privacy information. These findings highlight the essential need to address unnoticed privacy leakage risks associated with fast chargers.