Sparse SiLU: Efficient Feedforward Networks through Learned Activation Sparsity

View PDF

Abstract:We introduce Sparse SiLU, a variant of gated feedforward networks that incorporates activation sparsity through thresholding. Building on prior work in sparse neural networks and gated activations, our method applies a fixed threshold to the SiLU activation function to induce sparsity in transformer feedforward layers. Through experiments on the FineWeb dataset with an 83M parameter Qwen model, we demonstrate that Sparse SiLU achieves comparable performance (4.943 validation loss) to standard approaches like SwiGLU (4.927), while potentially offering memory efficiency benefits. We provide a detailed analysis of the method's limitations and practical considerations for implementation.