xATLU: Expanded Gating Ranges for Transformer Feedforward Networks

View PDF

Abstract:We introduce xATLU (Expanded ArcTan Linear Unit), a novel activation function for transformer feedforward networks that generalizes traditional gating mechanisms through learnable range expansion. Comprehensive experiments demonstrate that xATLU achieves consistent improvements over SwiGLU, with a 0.038 reduction in validation loss on FineWeb.