In this paper, we investigate the role of attention heads in Context-aware Machine Translation models for pronoun disambiguation in the English-to-German and English-to-French language directions. We analyze their influence by both observing and modifying the attention scores corresponding to the plausible relations that could impact a pronoun prediction. Our findings reveal that while some heads do attend the relations of interest, not all of them influence the models’ ability to disambiguate pronouns. We show that certain heads are underutilized by the models, suggesting that model performance could be improved if only the heads would attend one of the relations more strongly. Furthermore, we fine-tune the most promising heads and observe the increase in pronoun disambiguation accuracy of up to 5 percentage points which demonstrates that the improvements in performance can be solidified into the models’ parameters.

Achieving human-level translations requires leveraging context to ensure coherence and handle complex phenomena like pronoun disambiguation.Sparsity of contextually rich examples in the standard training data has been hypothesized as the reason for the difficulty of context utilization.In this work, we systematically validate this claim in both single-and multilingual settings by constructing training datasets with a controlled proportions of contextually relevant examples.We demonstrate a strong association between training data sparsity and model performance confirming sparsity as a key bottleneck.Importantly, we reveal that improvements in one contextual phenomenon do no generalize to others.While we observe some cross-lingual transfer, it is not significantly higher between languages within the same sub-family.Finally, we propose and empirically evaluate two training strategies designed to leverage the available data.These strategies improve context utilization, resulting in accuracy gains of up to 6 and 8 percentage points on the ctxPro evaluation in single-and multilingual settings respectively.1