Analysis of Dynamic Gating in Transformer Feedforward Networks

View PDF

Abstract:We present a comprehensive study of dynamic gating mechanisms in transformer feedforward networks. Our proposed architecture introduces a lightweight controller network to generate input-dependent gating coefficients. While initial results on smaller models showed promise, our analysis reveals significant challenges in scaling these approaches, with our final implementation achieving a validation loss of 4.935 compared to the SwiGLU baseline of 4.927.