Traditional fine-grained classification focuses on visible light domains, such as animals and cars. However, these methods often perform poorly when applied to radar images and images of satellites because of challenges such as distinguishing between noise and objects and the significant scale differences among object components. To address these unique scenarios, we propose the scale-shift attention in polarization domain (SAPD) method for fine-grained classification in satellite ISAR images. Specifically, radar emits different types of waves, each with distinct imaging effects. We utilize multipolarization inputs and introduce a polarization domain query module to integrate complementary features from various radar wave types captured from the same viewpoint. This multipolarization learning helps distinguish noise and leverages complementary features from different inputs. Moreover, to handle the substantial scale differences between centimeter-level payloads and the overall meter-level structure of satellites, we propose a scale-shift attention mechanism based on shift kernels. This mechanism extends attention in the direction specified by the shift kernel by incorporating adjacent pixels, allowing for the diffusion of attention. This is beneficial for capturing features of satellite components with varying scales and shapes. Extensive experiments on a novel satellite ISAR image dataset validate the effectiveness and superiority of the SAPD.