arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.04105cs.CLcs.CR
收藏

MoE模型中的专家选择揭示(几乎)与文本相同的信息

Expert Selections In MoE Models Reveal (Almost) As Much As Text

Amir Nuriyev, Gabriel Kulp

AI总结:

研究通过文本重建攻击揭示MoE模型中专家选择泄露的信息量,采用MLP和Transformer解码器实现高准确率的token恢复,指出专家选择应被视为敏感信息。

摘要:

我们提出了一种基于文本重建的攻击方法,针对混合专家(MoE)语言模型,仅利用专家选择即可恢复token。在MoE模型中,每个token被路由到专家子网络的子集;我们证明这些路由决策泄露的信息量远超之前理解。先前使用逻辑回归的方法效果有限,我们展示三层MLP将准确率提升至63.1%,而基于Transformer的序列解码器在OpenWebText上恢复32个token的top-1准确率为91.2%(top-10为94.8%)。这些结果将MoE路由与嵌入反转文献联系起来。我们概述了实际泄露场景(如分布式推理和侧通道)并表明添加噪声可减少但不能消除重建。我们的发现表明,MoE部署中的专家选择应被视为与底层文本一样敏感。

英文摘要:

We present a text-reconstruction attack on mixture-of-experts (MoE) language models that recovers tokens from expert selections alone. In MoE models, each token is routed to a subset of expert subnetworks; we show these routing decisions leak substantially more information than previously understood. Prior work using logistic regression achieves limited reconstruction; we show that a 3-layer MLP improves this to 63.1% top-1 accuracy, and that a transformer-based sequence decoder recovers 91.2% of tokens top-1 (94.8% top-10) on 32-token sequences from OpenWebText after training on 100M tokens. These results connect MoE routing to the broader literature on embedding inversion. We outline practical leakage scenarios (e.g., distributed inference and side channels) and show that adding noise reduces but does not eliminate reconstruction. Our findings suggest that expert selections in MoE deployments should be treated as sensitive as the underlying text.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑