Simplifying Feedforward Networks: When Less is More

View PDF

Abstract:Recent advances in transformer architectures have introduced increasingly complex feedforward network designs. Through systematic ablation studies on the FineWeb benchmark using an 83M parameter Qwen architecture, we demonstrate that a simplified feedforward network using single-stage SwiGLU gating can achieve competitive performance while maintaining computational efficiency. Our approach achieves a validation loss of 4.896 (mean across 3 runs, std=0.012), improving upon the standard SwiGLU baseline (4.927$\pm$0.015) while reducing parameter count by 8\%. The results suggest that current trends toward architectural complexity in feedforward networks may not always yield proportional benefits, particularly in medium-scale models.