Dynamic Memory Gating: An Investigation into Pattern-Specialized Feedforward Networks

View PDF

Abstract:This paper investigates Dynamic Memory Gating (DMG), a novel approach to transformer feedforward networks that employs parallel pattern detection heads with learned gating mechanisms. While theoretically promising for specialized processing of different input patterns, our implementation achieved a validation loss of 5.057 on the FineWeb benchmark, underperforming the SWiGLU baseline (4.927) and current state-of-the-art methods. We analyze the architectural choices, training dynamics, and potential limitations that may have contributed to these results, providing insights for future improvements in adaptive feedforward designs.