Perception of garments is a challenging task for robots due to the large variety in shapes, fabric patterns, and materials. We investigate a multi-sensorial approach, making no assumptions about the garments' configuration or properties. We use a robot equipped with RGB-D, tactile, and photometric stereo sensors that interacts with the garment through a combination of different basic actions. By applying machine learning techniques on the autonomously acquired data of different modalities we recognize the manipulated garment's type, fabric pattern, and material. Despite the challenges imposed by the unconstrained environment, promising performances are achieved for the majority of the recognition tasks.