Exploring Feedforward Architectures for Language Models

View PDF

Abstract:Our study evaluates feedforward layer modifications in transformers, focusing on the complexity-performance trade-off in smaller models. Results show modest improvements from architectural innovations are often outweighed by computational costs.