Dynamic GEGLU: An Adaptive Gating Mechanism for Feedforward Networks

View PDF

Abstract:We introduce Dynamic GEGLU, an improved gating mechanism for feedforward networks in Transformer architectures. While our approach shows only modest absolute improvements in validation perplexity (4.926 vs 4.9266 for SwiGLU), the consistent outperformance across training suggests potential benefits of adaptive gating. Our method extends GEGLU by incorporating input-dependent gating coefficients stabilized through layer normalization, with minimal computational overhead. We provide detailed ablation studies showing the importance of proper initialization and normalization, along with comprehensive analysis of training dynamics. The paper transparently discusses the limitations of our approach and suggests directions for future improvements.