Adaptive Range SiLU: An Improved Activation Function

View PDF

Abstract:We present Adaptive Range SiLU (AR-SiLU), a novel activation function that introduces learnable range expansion and temperature scaling for Transformer feedforward networks. Experiments show AR-SiLU achieves a 0.4\% reduction in validation perplexity compared to SwiGLU baselines (p < 0.05) while maintaining computational efficiency.