Polynomial Activation Units: A Systematic Approach to Enhancing Transformer Feedforward Networks

View PDF

Abstract:This paper introduces Polynomial Activation Units (PAU), a novel approach for transformer feedforward networks that combines the benefits of polynomial expansions with gating mechanisms. Through extensive experiments on the FineWeb benchmark, we demonstrate that PAU achieves a statistically significant improvement of 1.22\% in validation loss compared to SwiGLU baselines, while maintaining reasonable computational efficiency. Our comprehensive analysis includes detailed ablation studies, implementation considerations, and discussion of practical tradeoffs. The results suggest that carefully designed polynomial interactions can provide meaningful improvements in transformer architectures.