Multi-person motion prediction is an emerging and intricate task with broad real-world applications. Unlike single person motion prediction, it considers not just the skeleton structures or human trajectories but also the interactions between others. Previous methods achieve impressive predictions using various networks but often overlook the distinct representations of joint relations within individuals (intra-relations) and interactions among groups (inter-relations), inevitably leading to undesired dependencies. To address this issue, we introduce a new collaborative framework for multi-person motion prediction that explicitly modeling these relations: a GCN-based network for intra-relations and a novel reasoning network for inter-relations. Specifically, we propose a distance-aware cross-attention that incorporates physical distance constraints into inter-relation learning through a learnable distance weighting coefficient. Moreover, we propose a novel plug-and-play aggregation module called the Interaction Aggregation Module (IAM), which employs an aggregate-attention mechanism to seamlessly integrate these relations. Experiments indicate that the module can also be applied to other dual-path models. Extensive experiments on the 3DPW, 3DPW-RC, CMU-Mocap, MuPoTS-3D, as well as synthesized datasets Mix1 & Mix2 (9$\sim$15 persons), demonstrate that our method achieves state-of-the-art performance.