UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models
Abstract
UniMoMo compresses trained recommendation mixture-of-experts models into smaller standard MoE checkpoints via functional similarity grouping and layer-adaptive protection, preserving accuracy while accelerating inference.
Sparse mixture-of-experts (MoE) layers expand recommendation capacity through conditional computation, yet a trained checkpoint still stores and routes over its full expert bank. We study a deployment problem: convert that checkpoint to a smaller standard MoE under an explicit expert budget, without adding a compression-specific online module. To address this, we introduce UniMoMo, a post-training compression framework formulated as a constrained graph coarsening problem. Rather than relying on parameter distance, UniMoMo groups experts based on their functional similarity, using an unlabeled calibration set to measure how similarly experts respond to shared recommendation states. To prevent performance degradation, we introduce a layer-adaptive protection mechanism that restricts the merging of high-traffic experts based on their routing exposure. Across Amazon Beauty, KuaiRec, and TenRec with 2, 4, and 6 MoE blocks, the final four-expert checkpoints obtain source-relative five-run mean NDCG@10 ratios of 99.92%--102.30% and measured A100 speedups of 1.28times--1.63times. An aggressive two-expert, top-1 operating point obtains ratios of 98.36%--104.24% and speedups of 1.47times--2.21times. These endpoint results evaluate the complete conversion-and-adaptation workflow and show that a trained recommendation MoE can be exported at multiple serving budgets.
Community
🚀 UniMoMo, a practical approach for compressing MoE-based recommendation models.
The main idea is simple but effective: instead of merging experts based on parameter similarity, UniMoMo looks at how experts actually behave on calibration data and how much routing traffic each expert receives.
A few things I really like:
🔹 Functional expert similarity — experts are compared using their outputs on shared hidden states, rather than just weight distance.
🔹 Traffic-aware merging — heavily used experts are protected during clustering, reducing the risk of hurting important routing paths.
🔹 Calibration-aware reconstruction — after merging SwiGLU experts, UniMoMo uses a least-squares correction to better recover the original intermediate activations.
🔹 No extra serving complexity — the final compressed model is still a standard top-(k) MoE, so there is no special inference-time module.
The results are also strong: compressing 8 experts → 4 experts can retain nearly all recommendation quality while achieving meaningful inference speedups, and the more aggressive 8 → 2 / top-1 setting pushes acceleration even further. ⚡
Overall, a nice example of making MoE compression more function-aware, routing-aware, and deployment-friendly. Worth checking out if you work on large-scale recommendation or MoE efficiency. 👀
Get this paper in your agent:
hf papers read 2608.08627 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper