Adaptive Activation Mixing for Transformer Feedforward Networks

View PDF

Abstract:We analyze adaptive activation mixing in transformer feedforward networks, combining GELU and SiLU activations with a learned gating mechanism. Our approach achieves 5.108 validation loss versus 4.927 for SwiGLU baseline, demonstrating the feasibility of dynamic activation selection with minimal parameter overhead.