Position-Aware Gompertz Gating for Transformer Feedforward Networks

View PDF

Abstract:We present Position-Aware Gompertz Gating (PAGG), an improved feedforward module for transformers that systematically addresses three limitations of standard gated linear units (GLUs). Our method combines asymmetric activation with position-aware scaling and achieves a 4.889 validation loss on FineWeb, improving upon SwiGLU (4.927) while maintaining similar computational cost.