Ultra-wideband (UWB) communication technology has gained attention in robotics due to its ability to provide range measurements possibly with centimeter-level accuracy. Nevertheless, practical UWB range measurements are susceptible to disturbances from multiple sources, including the anisotropic characteristics of antennas, non-line-of-sight (NLOS) conditions, and multipath propagation. In this paper, we introduce a UWB range measurement model that addresses these sources of error. To accommodate the effects of antenna anisotropy, we adopt real spherical harmonics to represent directional bias in the UWB range measurement model. To handle delayed measurements induced by NLOS conditions and multipath propagation, an asymmetric heavy-tailed distribution is utilized to model the measurement noise. We calibrate this measurement model based on the maximum likelihood estimation method and propose a UWB-based localization system based on that. Our localization system provides: 1) anchor self-calibration, which identifies anchor placement by fusing visual-inertial-ranging measurements based on continuous-time state representation; and 2) filtering-based state estimation, which applies our measurement model into Kalman filtering framework via an iterative update algorithm. Experimental validation is conducted to demonstrate the effectiveness of the measurement model for our localization system. We open source our implementation of the proposed UWB-based localization system at https://github.com/INRoL/inrol_uwb_localization.