Implementation Challenges in Probabilistic Positional Attention Mechanisms

View PDF

Abstract:This paper documents our investigation into probabilistic positional priors for transformer attention mechanisms and the technical challenges encountered during implementation. We propose a modification to standard attention that incorporates learnable positional decay and scale parameters, building on prior work in relative position encodings and learned attention biases. While our baseline implementation of the Qwen attention achieved a validation loss of 5.13 on the FineWeb dataset (compared to the reference Qwen baseline of 4.9266), we encountered persistent tensor shape mismatches when integrating our probabilistic modifications. We analyze these implementation challenges in detail and discuss lessons learned for future work in attention mechanism modifications.