Oscillatory-Gated Feedforward Networks: Analysis of a Hybrid Activation Approach

View PDF

Abstract:This paper presents a detailed examination of Oscillatory-Gated Feedforward Networks (OGFN), a hybrid architecture combining sinusoidal activations with gated linear units in transformer models. While achieving a modest improvement over SwiGLU (4.912 vs. 4.927 validation loss on FineWeb), our analysis reveals significant trade-offs in memory efficiency and implementation complexity. We provide comprehensive ablation studies, statistical significance testing, and comparisons with contemporary approaches to better understand the limitations and potential applications of this technique. The paper concludes with recommendations for future work in hybrid activation designs.