Expanding Activation Ranges in Transformer Feedforward Networks

View PDF

Abstract:We present xSiLU, an improved activation function for transformer feedforward networks that learns an optimal gating range. Experimental results show consistent improvements over baseline methods.