IsoGMLP: A Systematic Exploration of Isotropy in Gated MLP Architectures

View PDF

Abstract:We present IsoGMLP, a novel gated MLP architecture that explicitly incorporates isotropy maintenance through a parallel pathway. Through extensive experiments on the FineWeb benchmark, we demonstrate that while IsoGMLP achieves comparable performance to the SwiGLU baseline (validation loss of 4.948 vs 4.9266), it offers improved training stability and more consistent convergence patterns. Our analysis reveals that the isotropy pathway contributes meaningfully to model behavior, particularly in maintaining gradient norms and preventing representation collapse. We provide detailed ablation studies, statistical analysis across multiple runs, and computational efficiency measurements, offering insights into when and how isotropy maintenance can benefit transformer architectures.