We propose objective, image-based techniques for quantitative evaluation of facial skin gloss that is consistent with human judgments. We use polarization photography to obtain separate images of surface and subsurface reflections, and rely on psychophysical studies to uncover and separate the influence of the two components on skin gloss perception. We capture images of facial skin at two levels, macro-scale (whole face) and meso-scale (skin patch), before and after cleansing. To generate a broad range of skin appearances for each subject, we apply photometric image transformations to the surface and subsurface reflection images. We then use linear regression to link statistics of the surface and subsurface reflections to the perceived gloss obtained in our empirical studies. The focus of this paper is on within-subject gloss perception, that is, on visual differences among images of the same subject. Our analysis shows that the contrast of the surface reflection has a strong positive influence on skin gloss perception, while the darkness of the subsurface reflection (skin tone) has a weaker positive effect on perceived gloss. We show that a regression model based on the concatenation of statistics from the two reflection images can successfully predict relative gloss differences.