Pattern recognition from two-dimensional (2D) images is an indispensable tache in computer vision. However, one legacy hinders its progress: multifarious visual distortions (e.g., partially occluded signs, fisheye respective, affine or 3D projections) caused by spatiotemporal-varying perturbations (e.g., shrunk, sharpening, overexposure, jitter, and motion) significantly degrade the performance of neural networks in terms of high-level intelligent behaviors (e.g., target localization and recognition). Leveraging the growing availability of capsule network (CapsNet), we suppress the deformation effect to the final prediction by proposing a CapsNetORB framework to implement distorted target recognition. Two highlights, the customized Siamese CapsNet (S-CapsNet) and vector-based oriented fast and rotated brief (VB-ORB), can cast a mutual positive stimulation: the former encodes capsule feature vectors for the later, whilst the later detects space-scale invariant interval dimensions (instead of pixels) to bridge association between source standard images (high-quality training images) and distorted ones (testing images). Thus, the category of one source standard image owning the most correspondences is the final predicted category. We believe that capsule vectors own higher representability and stability compared with conventional pixels/feature maps, which can be well exploited in feature learning while resisting visual distortions. Experimentally, we show that employing our pipeline for distorted target categorization can outperform state-of-the-arts by delivering promising performance on CUB-200-2011, Stanford Dogs, Stanford Cars, and our hand-crafted data set.