Human–robot collaborative assembly is required to comprehensively perceive the working scenarios for the most possible assembly collaborations. Nevertheless, existing works have paid much attention to physical entities (i.e., object detection, pose estimation), while ignores the weight of interactive relationships. This research gap makes it difficult to become aware of the cues for decision-making, especially in a complicated assembly task. Furthermore, inadequate relative position characteristics and indescribable object influence remain quite challenging for visual relationship representation. To overcome these abovementioned gaps, a multimodality scene graph generation approach is proposed to more robustly describe the abstract visual relationships. A novel heat modality is presented to better represent the relative spatial characteristic. Three strategies are developed for adapting different baselines in the multimodality feature encoder module. Experimental results show the generality and superb performance for multimodality scene graph generation tasks in human–robot collaborative assembly scenarios.