PolySiLU: A Minimal Polynomial Enhancement to SiLU Activation

View PDF

Abstract:We present PolySiLU, a modified activation function combining SiLU (Sigmoid-Weighted Linear Unit) with learnable quadratic and cubic terms through an adaptive mixing mechanism. While recent work has demonstrated the effectiveness of gated activations like SwiGLU and polynomial-enhanced variants like PolyGate, we explore whether minimal polynomial additions can provide complementary benefits without significant parameter overhead. Our experiments on a 134M parameter transformer show PolySiLU achieves comparable performance (validation loss 4.9299) to SwiGLU (4.9266), with more pronounced benefits during early training stages. The work contributes: (1) analysis of polynomial-SILU mixing dynamics, (2) empirical validation of stable training despite higher-order terms, and (3) open questions about optimal polynomial integration in modern architectures.