Probabilistic Asymmetric Gating Units for Transformer Networks

View PDF

Abstract:We present Probabilistic Asymmetric Gating Units (PAGU), a novel activation function combining Gompertz asymmetry with probabilistic gating. Experimental results on language modeling show PAGU achieves competitive performance (validation loss 5.115) compared to SwiGLU (4.927), with faster early convergence.