Rethinking Polynomial Activations in Transformers: \\ A Comprehensive Study of the Contextual Gated Polynomial Network

View PDF

Abstract:This paper presents a rigorous empirical investigation of polynomial activation functions in transformer feedforward networks through our proposed Contextual Gated Polynomial Network (CGPN). Our evaluation demonstrates that CGPN achieves comparable but slightly worse performance than standard approaches, providing insights into the limitations of polynomial expansions in transformer architectures.