Anshumann Anshumann, Mohd Abbas Zaidi, Akhil Kedia, Jinwoo Ahn, Taehwak Kwon, Kangwook Lee, Haejun Lee, Joohyung Lee. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

In spite of their huge success, transformer models remain difficult to scale in depth. In this work, we develop a unified signal propagation theory and provide formulae that govern the moments of the forward and backward signal through the transformer model. Our framework can be used to understand and mitigate vanishing/exploding gradients, rank collapse, and instability associated with high attention scores. We also propose DeepScaleLM, an initialization and scaling scheme that conserves unit output/gradient moments throughout the model, enabling the training of very deep models with 1000 layers. We find that transformer models could be much deeper - our deep models with fewer parameters outperform shallow models in Language Modeling, Speech Translation, and Image Classification, across encoder-only, decoder-only and encoder-decoder variants, for both Pre-LN and Post-LN transformers, for multiple datasets and model sizes. These improvements also translate into improved performance on downstream Question Answering tasks and improved robustness for Image Classification.

Generative models have recently started to outperform extractive models in Open Domain Question Answering, largely by leveraging their decoder to attend over multiple encoded passages and combining their information.However, generative models tend to be larger than extractive models due to the need for a decoder, run slower during inference due to auto-regressive decoder beam search, and their generated output often suffers from hallucinations.We propose to extend transformer encoders with the ability to fuse information from multiple passages, using global representation to provide cross-sample attention over all tokens across samples.Furthermore, we propose an alternative answer span probability calculation to better aggregate answer scores in the global space of all samples.Using our proposed method, we outperform the current stateof-the-art method by 2.5 Exact Match score on the Natural Question dataset while using only 25% of parameters and 35% of the latency during inference, and 4.4 Exact Match on We-bQuestions dataset.When coupled with synthetic data augmentation, we outperform larger models on the TriviaQA dataset as well.The latency and parameter savings of our method make it particularly attractive for open-domain question answering, as these models are often compute-intensive.Global Tokens Global Encoder Layer Question + Passage 1 Passage Encoder Layer Question + Passage 2 Passage Encoder Layer Span Classifier 𝑃𝑃 𝑠𝑠 (𝑠𝑠 1 ) Span Classifier 𝑃𝑃 𝑠𝑠 (𝑠𝑠 2 ) 𝑃𝑃 𝐴𝐴 ("𝑁𝑁𝑁𝑁𝑁𝑁 𝑌𝑌𝑌𝑌𝑌𝑌𝑌𝑌") Σ Multi-Head Attention (Global Encoder Layer)x N Encoder Layers Q KV

Recent approaches to Open-domain Question Answering refer to an external knowledge base using a retriever model, optionally rerank passages with a separate reranker model and generate an answer using another reader model.Despite performing related tasks, the models have separate parameters and are weaklycoupled during training.We propose casting the retriever and the reranker as internal passage-wise attention mechanisms applied sequentially within the transformer architecture and feeding computed representations to the reader, with the hidden representations progressively refined at each stage.This allows us to use a single question answering model trained end-to-end, which is a more efficient use of model capacity and also leads to better gradient flow.We present a pre-training method to effectively train this architecture and evaluate our model on the Natural Questions and TriviaQA open datasets.For a fixed parameter budget, our model outperforms the previous state-of-the-art model by 1.0 and 0.7 exact match scores.Reading Layer Reranking Layer Retrieval Layer Encoder Query Passage 1 Passage 2 Passage 3 Passage 4 Passage n Encoder Encoder Encoder Encoder Encoder Encoder … … Passage-wise Disjoint Attention Encoder Encoder Encoder Encoder Passage-wise Joint Attention Concat Query and Passage Answer Concat All Decoder Encoder Encoder Encoder … Decoder

A common approach in many machine learning algorithms involves self-supervised learning on large unlabeled data before fine-tuning on downstream tasks to further improve performance.A new approach for language modelling, called dynamic evaluation, further fine-tunes a trained model during inference using trivially-present ground-truth labels, giving a large improvement in performance.However, this approach does not easily extend to classification tasks, where ground-truth labels are absent during inference.We propose to solve this issue by utilizing self-training and back-propagating the loss from the model's own class-balanced predictions (pseudo-labels), adapting the Reptile algorithm from meta-learning, combined with an inductive bias towards pre-trained weights to improve generalization.Our method improves the performance of standard backbones such as BERT, Electra, and ResNet-50 on a wide variety of tasks, such as question answering on SQuAD and NewsQA, benchmark task SuperGLUE, conversation response selection on Ubuntu Dialog corpus v2.0, as well as image classification on MNIST and ImageNet without any changes to the underlying models.Our proposed method outperforms previous approaches, enables self-supervised finetuning during inference of any classifier model to better adapt to target domains, can be easily adapted to any model, and is also effective in online and transfer-learning settings.