MOTIVATION: Predicting protein-ligand binding affinity accurately and quickly is a major challenge in drug discovery. Recent advancements suggest that deep learning-based computational methods can effectively quantify binding affinity, making them a promising alternative. Environmental factors significantly influence the interactions between protein pockets and ligands, affecting the binding strength. However, many existing deep learning approaches tend to overlook these environmental effects, focusing instead on extracting features from proteins and ligands based solely on their sequences or structures. RESULTS: We propose a deep learning method, EM-PLA, which is based on an environment-aware heterogeneous graph neural network and utilizes multimodal data. This method improves protein-ligand binding affinity prediction by incorporating environmental information derived from the biochemical properties of proteins and ligands. Specifically, EM-PLA employs a heterogeneous graph neural network (HGT) with environmental information to improve the calculation of non-covalent interactions, while also considering the interaction calculations between protein sequences and ligand sequences. We evaluate the performance of the proposed EM-PLA through comprehensive benchmark experiments for binding affinity prediction, demonstrating its superior performance and generalization capability compared to state-of-the-art baseline methods. Furthermore, by analyzing the results of the ablation experiments and integrating visual analyses and case studies, we validate the rationale of the proposed method. These results indicate that EM-PLA is an effective method for binding affinity prediction and may provide valuable insights for future applications. AVAILABILITY AND IMPLEMENTATION: The source code is available at https://github.com/littlemou22/EM-PLA.