The inherent elasticity of soft materials can be used to create robotic grippers that deform and comply to a variety of irregular shapes. To date, several soft adaptive grasping strategies have been reported, however, most of them focus on adapting to the overall shape of the structure, while the adaptive grasping of small surface asperities is overlooked. In this paper, we propose a novel method to achieve adaptive grasping on surface asperities with a smart shape-memory silicone sponge. Heating above 60°C makes the sponge soft and deformable to allow it to penetrate within surface asperities via a pressure normal to the surface. Cooling down below 60°C makes the sponge "jam" to retain its deformed shape. The interlocking force between the jammed sponge and the asperities, and the increased area of contact, allows for adaptive grasping on asperities down to 0.4 mm with an adhesive force of up to 27.7 N in a 40 × 40 mm contacting area. We introduce the design, working principle, fabrication, and optimization of a robotic gripper based on this shape-memory silicone sponge. This sponge-jamming gripper shows great potential for developing next-generation robotic grippers for the manipulation of textured and discontinuous surfaces.