Floating-point arithmetic is ubiquitous across computing, with its wide range of values, large and small, making it the preferred tool for storing, analysing, and manipulating numerical data. Its flexibility comes at the cost of additional risks in some security/privacy-aware settings. In this paper, we discuss the threat of information leakage caused by floating-point arithmetic when adding noise to sensitive values, which can allow the sensitive information to be recovered (e.g., in differential privacy). We present a solution, Mantissa Bit Manipulation (MBM), that is orders of magnitude faster than the current state-of-the-art, applicable to most continuous probability distributions and to all floating-point number formats.