Transformers often struggle to generalize to longer sequences than those seen during training - a limitation known as length extrapolation. Most existing Relative Positional Encoding (RPE) methods attempt to address this by introducing either fixed linear biases or globally learned biases, which lack the capacity to adapt to different input contexts. In this work, we propose an additive RPE, Context-Aware Biases for Length Extrapolation (CABLE), a method that learns token-specific, context-aware biases for each attention head in transformers. By dynamically adjusting positional biases based on the input sequence, CABLE overcomes the rigidity of fixed RPEs. When evaluated on sequences longer than originally trained with, GPT-2 Medium (334M parameters) with CABLE achieves lower perplexity than counterparts using other widely adopted positional encoding methods. Additionally, by applying CABLE to the BERT base model we improved performance in long-context retrieval tasks. Our method significantly enhances the extrapolation performance of existing RPE methods tested on the FineWeb-Edu-10B and WikiText-103 datasets. Our code is available at: https://github.com/AlgonetLabs/Cable.

This paper presents a novel method for 3D angle of arrival (AOA) localization using a mobile receiver with limited number of antennas. We propose three convolutional neural networks to estimate elevation and azimuth AOAs, along with their pairing from recorded signals in sequential time windows. Then, a multi-source 3D-localization algorithm is proposed to estimate source positions across recorded time windows. Simulation results validate the effectiveness and robustness of the proposed method even in scenarios where uncertainties arise regarding the receiver’s position or direction during movement.

Despite its longstanding importance in wireless communications, multi-source localization remains a significant challenge, particularly in scenarios involving moving receivers. This study focuses on the 3D angle of arrival (AOA) localization problem involving multiple sparse signal sources, utilizing a moving receiver with limited antennas. At first, a simple energy detector algorithm is proposed for noise removal in sparse signals. Next, rough angle estimates are obtained by two-dimensional multiple signal classification (2D-MUSIC) and then, a new algorithm based on K-SVD is proposed for precise 2D AOA estimation using a novel sparse recovery method. Finally, a new multi-source 3D-localization algorithm is introduced to estimate the positions of sources using the estimated AOAs. Extensive simulations validate the framework's efficacy, especially in challenging scenarios where the number of sources exceeds the number of antennas.