SIMPA: Restricting What Experts See, Not What They Produce, in MoE-PEFT

Mixture-of-Experts (MoE) routing collapses in parameter-efficient fine-tuning: on LLaMA-3 8B with MixLoRA, a single expert absorbs 82.5% of tokens. Prior MoE-PEFT remedies intervene at the output - routing reformulations, output-similarity penalties, shared-expert capacity - leaving each routed expert’s input view identical. We propose \textbf{SIMPA} (Sparse-Input Magnitude Partitioning Adaptation), which restricts what experts \emph{see} rather than what they produce: a magnitude-thresholded input drives a sparse routed-LoRA branch, a parallel dense LoRA branch (adapting DeepSeekMoE’s shared-expert topology to PEFT) provides an information safeguard, and a column-wise gate fuses the two. At a matched 2.9% trainable-parameter budget, SIMPA leads compared baselines on reasoning (86.2% on seven LLaMA-3 8B tasks, +3.9 over MixDoRA), code and arithmetic (38.4 HumanEval Pass@1 and 59.4 GSM8K, matching full fine-tuning under identical data and compute), and NLU (88.7 GLUE / 77.7 SuperGLUE on T5-base); against direct MoE-PEFT counterparts HydraLoRA and MoLoRA at matched setup, SIMPA leads HumanEval by +14.2 and +13.6 and GSM8K by +2.1 and +2.8. Direct measurement returns an expert-input Jaccard of 0.184 vs. MixLoRA’s 1.000 - a 5.4 reduction - and the same block-aligned mask drives a Triton kernel measuring 1.46 end-to-end inference speedup on A100. Closed-form partitioning and retention bounds, the consistency check against measured activation statistics, and a tail-index-driven closed form for the optimal sparsity are presented in §3.4 and Appendix E. Code: \url{https://anonymous.4open.science/r/simpa}.
Link