PolyNorm: An Adaptive Polynomial Activation for Transformer Feedforward Networks

View PDF

Abstract:We present PolyNorm, a novel adaptive polynomial activation function for transformer feedforward networks that combines learned polynomial features with input-dependent mixing. While modern transformers predominantly use fixed activation functions like SwiGLU, we demonstrate that learned polynomial expansions can provide more expressive feature transformations while maintaining training stability. Through careful architectural design including layer normalization and adaptive clipping, PolyNorm achieves a validation loss of 4.886 on FineWeb, outperforming the SwiGLU baseline (4.927) with only 0.6\% additional parameters and 3\% computational overhead. Extensive ablation studies validate our design choices and reveal consistent layer-wise patterns in polynomial mixing. The success of PolyNorm suggests that adaptive polynomial activations are a promising direction for improving transformer architectures.