Dynamic Range Gated Linear Units: An Empirical Study

View PDF

Abstract:We present an empirical study of Dynamic Range Gated Linear Units (DRGLU), a simplified feedforward architecture for transformers. Our experiments on FineWeb with 134M parameter models show DRGLU achieves 4.996 validation loss compared to SwiGLU's 4.9266, while requiring 25\% more memory. These negative results provide insights into the limitations of simple scaling approaches.