Traditional intensity imaging does not offer a general approach for the perception of textureless and specular reflecting surfaces. Intensity based methods for shape reconstruction of specular surfaces rely on virtual (i.e. mirrored) features moving over the surface under viewer motion. We present a novel method based on polarization imaging for shape recovery of specular surfaces. This method overcomes the limitations of the intensity based approach because no virtual features are required. It recovers whole surface patches and not only single curves on the surface. The presented solution is general as it is independent of the illumination. The polarization image encodes the projection of the surface normals onto the image and therefore provides constraints on the surface geometry. Taking polarization images from multiple views produces enough constraints to infer the complete surface shape. The reconstruction problem is solved by an optimization scheme where the surface geometry is modelled by a set of hierarchical basis functions. The optimization algorithm proves to be well converging, accurate and noise resistant. The work is substantiated by experiments on synthetic and real data.