Dense 3D reconstructions generate globally consistent data of the environment suitable for many robot applications. Current RGB-D based reconstructions, however, only maintain the color resolution equal to the depth resolution of the used sensor. This firmly limits the precision and realism of the generated reconstructions. In this paper we present a real-time approach for creating and maintaining a surface reconstruction in as high as possible geometrical fidelity with full sensor resolution for its colorization (or surface texture). A multi-scale memory management process and a Level of Detail scheme enable equally detailed reconstructions to be generated at small scales, such as objects, as well as large scales, such as rooms or buildings. We showcase the benefit of this novel pipeline with a PrimeSense RGB-D camera as well as combining the depth channel of this camera with a high resolution global shutter camera. Further experiments show that our memory management approach allows us to scale up to larger domains that are not achievable with current state-of-the-art methods.