This paper describes rubble recognition using a depth image sensor and an automatic rubble crushing system using a construction machine for automatic rubble crushing at a building demolition site. Depth Distribution Split Labeling (DDSL) is proposed to recognize irregularly shaped rubble using depth images and to identify the largest rubble in the workspace. In DDSL, we focused on the fact that the depth of the contact area of adjacent rubble is lower, and labeling by dividing the depth makes it possible to identify the rubble. The automatic rubble crushing system enables to avoid excessive pushing and to grasp objects based on the cylinder driving force calculated from the hydraulic cylinder internal pressure of the construction machine. Through evaluation experiments simulating the crushing environment, it was confirmed that the proposed system can identify and automatically crush rubble.