Cross-Token Gated Feedforward Networks: \\ A Comprehensive Analysis of Spatial Interactions in Transformer Layers

View PDF

Abstract:This paper presents a rigorous investigation of cross-token gating mechanisms in transformer feedforward networks. While recent work has demonstrated the effectiveness of sophisticated gating approaches, the potential benefits of explicit cross-token interactions remain underexplored. We introduce a novel architecture combining multi-scale processing with spatial gating, employing both GEGLU and SiLU activations in parallel pathways. Through extensive experimentation across model scales, we find that while our approach shows promise in small-scale ablations (0.31\% improvement over baseline), it underperforms in full-scale evaluation (1.3\% worse than SwiGLU baseline). We provide comprehensive analysis of this scaling discrepancy, including memory overhead measurements, training dynamics visualization, and failure mode analysis. Our results suggest that while cross-token interactions can provide modest benefits in constrained settings, they may not be computationally justified in standard transformer architectures.