Efficient Three-Layer Feedforward Networks with Optimized Gating and Normalization

View PDF

Abstract:We present a systematic study of feedforward network architectures in transformers, focusing on optimization of gating mechanisms and normalization placement. Through careful ablation studies, we identify an efficient three-layer design that achieves a validation loss of 4.857 on the FineWeb benchmark (vs. 4.9266 SwiGLU baseline) while maintaining parameter efficiency. Our architecture employs dimension-reduced projections with strategic normalization placement, demonstrating consistent improvements across model scales.