A major challenge in inverse reflectometry is the acquisition of spatially varying materials. In this paper, we introduce a method to recover spatial reflectance from a sparse set of images under general illumination. Specifically, we first remove the high-frequency varying diffuse reflection term by using a low-order spherical harmonic approximation. This allows us to directly estimate the specular properties with a cluster fitting process, which simplifies the fitting processes and addresses the problem of data inadequacy for sparse images. As a result, we can reconstruct a truly spatially varying BRDF model of the surface from less than 10 images. Experimental results will be presented in order to demonstrate the effectiveness of the proposed algorithm.