OrthoAdam: Gradient Orthogonalization for Transformer Optimization

View PDF

Abstract:We present OrthoAdam, an optimizer that applies singular value decomposition (SVD) to gradients of attention layer parameters in transformers. While building on established adaptive optimization principles, our method demonstrates a 4.3\% improvement in validation loss (4.72 vs 4.93) compared to AdamW on a 134M parameter language model. We analyze the computational trade-offs and provide practical recommendations for implementation. The paper includes a comprehensive comparison with recent orthogonal optimization methods and discusses limitations regarding scalability and generalization.