As transformer sequence lengths grow, existing pipeline parallelisms incur suboptimal performance due to the quadratic attention computation and the substantial memory overhead. To relieve these challenges, we propose HelixPipe, a novel pipeline parallelism for long sequence transformer training. First, HelixPipe introduces attention parallel partition, which schedules attention computations of different micro batches across different pipeline stages in parallel, reducing pipeline bubbles. Second, it employs a two-fold first-in-last-out micro batch schedule to balance memory usage and overlap communication with computation. Additionally, HelixPipe utilizes recomputation without attention and chunked MLP to mitigate fragmentation and enable longer sequences. Experiments demonstrate that HelixPipe gains increasing advantages with longer sequence lengths, and outperforms existing methods in throughput and scalability across varying pipeline sizes, model sizes, and cluster configurations. Notably, it achieves a 26% speedup over baseline methods when training a 7B model with 128k sequence length on 64 H20 GPUs. Code is available at https://github.com/zxgx/Megatron-LM.

Complex knowledge graph question answering (KGQA) aims at answering natural language questions by entities retrieving from a knowledge graph (KG). Recently, the relation path-based models have shown the unique advantage for complex KGQA. However, these existing models ignore the dependency between different relation paths, which leads to aimless reasoning over the KG. To resolve this issue, we propose the question-directed reasoning with relation-aware graph attention network (QRGAT) that encodes the reasoning process as a reasoning graph. The relation-aware GAT can recognize neighbor entities along with the corresponding relations for each entity. With the relation-aware GAT stacked in multiple layers, it can collaboratively capture the dependency of different relation paths for each entity. The question-directed reasoning utilizes the information learned by the relation-aware GAT to solve the aimless reasoning on the KG by constructing a reasoning graph. Extensive experiments demonstrate that our QRGAT outperforms the baseline models on both popular datasets WebQuestionsSP and ComplexWebQuestions. Compared with the strong GNN-based baseline NSM$_{+h}$, our QRGAT achieves the performance improvements of 2.3% on WebQuestionsSP and 3.6% on ComplexWebQuestions by the metric Hits@1.