Polynomial Activations in Transformer Networks

View PDF

Abstract:We evaluate quadratic polynomial activations as alternatives to gated mechanisms in transformers. Our experiments show polynomial activations achieve a validation loss of 4.891, improving upon the SwiGLU baseline (4.9266). The results demonstrate polynomial activations can provide comparable performance with simpler implementations.