Adaptive Threshold Gating in Transformer Feedforward Networks

View PDF

Abstract:This paper investigates Adaptive Threshold Gating (ATG) for transformer feedforward networks. Our experiments show ATG achieves a validation loss of 4.966, slightly underperforming the SwiGLU baseline of 4.9266. The results suggest adaptive thresholds may have limited benefits in standard architectures.