The multi-view hash method is crucial in multimedia retrieval via transforming heterogeneous data from multiple views into binary hash codes. Existing methods primarily focus on leveraging complementary information across multiple views, while ignoring the issue of imbalanced optimization. That is, the features from some views in multi-view data are stronger than others, which leads to a less optimization of the networks handling those weaker features. To fully utilize the data from all the views, we propose a novel Adaptive Loss-aware Modulation (ALM) method to address this imbalance issue during the fusion of multi-view features. Specifically, in training, ALM automatically calculates the total loss for each view to reflect the performance of the respective view's backbone network. The modulation coefficient is then determined based on the total loss of the corresponding view. By multiplying the gradient of the network of each view with its corresponding modulation coefficient, we can suppress the gradient update rate of the view with stronger features, while maintaining the normal gradient update rate for the ones with weaker features. Based on ALM, we further introduce a new Balanced Multi-View Hashing (BMVH) method. Extensive experiments on three public datasets demonstrate that the proposed BMVH outperforms state-of-the-art methods, with a maximum increase of 3.22% in mAP.