As a distributed machine learning paradigm, vertical federated learning enables multiple passive parties with distinct features and an active party with labels to train a model collaboratively. Although it has been widely applied for its ability to protect privacy to some extent, this paradigm still faces various threats, especially the label inference attack (LIA). In this paper, we present the first observation of the disparity in LIAs resulting from differences in feature distribution among passive parties. To substantiate this, we study four different types of LIAs across five benchmark datasets, investigating the potential influencing factors and their combined impact. The results show that attack performance disparities can vary up to 15 times among different passive parties. So, how to eliminate this disparity? We explore methods from both attack and defense perspectives, including learning rate adjustment and noise perturbation with differential privacy. Our findings indicate that a modest increase in the learning rate of the passive party effectively enhances the LIA performance. In light of these, we propose a novel defense strategy that identifies passive parties with important features and applies adaptive noise to their gradients. Experiments show that it effectively reduces both attack disparity among passive parties and overall attack accuracy, while maintaining low computational complexity and avoiding additional communication overhead. Our code is publicly accessible athttps://github.com/WWlnZSBMaXU/Attackers-Are-Not-the-Same.