3D perception of objects is critical for many real-world applications, such as autonomous cars and robots. Among them, most state-of-the-art (SOTA) 3D perception systems are based on deep learning models. Recently, the research community found that 3D object classifiers on point cloud based on deep learning are easily fooled by adversarial point cloud craft by attackers. To overcome this, adversarial defenses are considered the most effective ways to improve the robustness of deep learning models, and most adversarial defenses on point cloud are focused on input transformation. However, all previous defense methods decrease the natural accuracy, and the nature of the point cloud classifiers itself has been overlooked. To this end, in this paper, we propose a novel adversarial defense for 3D point cloud classifiers that makes full use of the nature of the point cloud classifiers. Due to the disorder of point cloud, all point cloud classifiers have the property of permutation invariant to the input point cloud. Based on this nature, we design invariant transformations defense (IT-Defense). We show that, even after accounting for obfuscated gradients, our IT-Defense is a resilient defense against SOTA 3D attacks. Moreover, IT-Defense does not hurt clean accuracy compared to previous SOTA 3D defenses. Our code will be available at: https://github.com/cuge1995/IT-Defense.