Multi-Head Dynamic Gating for Feedforward Networks

View PDF

Abstract:We present Multi-Head Dynamic Gating (MHDG), a novel approach to Transformer feedforward networks that combines multiple parallel gating pathways with learned temperature scaling. Through extensive experiments on the FineWeb dataset, we demonstrate a statistically significant 0.005 improvement in validation perplexity (p < 0.05) compared to SwiGLU baselines, albeit with a 33% memory overhead. Our ablation studies reveal the importance of both parallel gating and learned temperature control, while comparisons with leaderboard approaches properly position our work within the current research landscape.