Adaptive Sigmoid-Exponential Gated Units: \\ A Cautionary Study of Dynamic Activation Functions in Transformers

View PDF

Abstract:We present a systematic investigation of the Adaptive Sigmoid-Exponential Gated Unit (ASEGU), a novel feedforward architecture combining learnable gating mechanisms with exponential non-linearities in transformer networks. While recent work has demonstrated the effectiveness of adaptive components in neural architectures, our comprehensive evaluation reveals that ASEGU underperforms the SwiGLU baseline (5.313 vs 4.9266 validation loss) despite careful numerical stabilization and parameter efficiency considerations. Through detailed ablation studies and gradient analysis, we identify key failure modes including initialization sensitivity and exponential pathway instability. Our findings suggest that the benefits of dynamic range adjustment may be context-dependent, and that simpler, more stable architectures remain preferable for standard transformer feedforward components. This work provides valuable empirical evidence for the architecture design community and establishes important caveats for future work on adaptive activation functions.