Revisiting Sparse Gating in Transformer Feedforward Networks: An Empirical Study

View PDF

Abstract:This paper presents a systematic investigation of sparse gating mechanisms in transformer feedforward networks (FFNs). While recent work has demonstrated the effectiveness of sparsity in attention layers, its application to FFNs remains understudied. We evaluate a novel sparse gated FFN architecture combining key-value memory structures with selective activation patterns and residual gating. Our comprehensive experiments on a 134M parameter language model reveal that while the approach reduces theoretical FLOPs by 85\%, it results in a 3.3\% increase in validation loss compared to the SwiGLU baseline. We analyze the failure modes through extensive ablations and provide insights for future sparse FFN designs.