Dynamic Adaptive Gating with Parallel Pathways

View PDF

Abstract:We present Dynamic Adaptive Gating with Parallel Pathways (DAG-PP), a novel feedforward architecture for transformers that combines multiple activation functions through learned blending weights. Our approach achieves improved validation loss compared to standard baselines while maintaining computational efficiency.