DuDi: 跨语言动词化的双信号蒸馏
DuDi: Dual-Signal Distillation with Cross-Lingual Verbalizer
AI总结：
提出DuDi框架，通过结合序列级和词元级信号以及跨语言动词化器，提升小语言模型在多语言（尤其是东南亚语言）上的性能。
摘要：
小语言模型（SLM）高效且可扩展，但其多语言能力在十亿以下规模时严重下降，尤其是对于东南亚（SEA）语言。我们引入了DuDi，一个双信号多语言蒸馏框架，它结合了在线序列级信号与离策略和在线词元级信号。DuDi进一步使用跨语言动词化器来优化教师反馈并提高多语言设置下教师-学生的可迁移性。在SEA-HELM上跨多个模型家族、规模和教师-学生设置的实验表明，DuDi始终优于竞争性的蒸馏基线。消融和分析证实，序列级优化、词元级监督和跨语言动词化为多语言SLM提供了互补且可迁移的学习信号。
英文摘要：
Small language models (SLMs) are efficient and scalable, but their multilingual capabilities degrade severely at sub-billion scales, especially for Southeast Asian (SEA) languages. We introduce DuDi, a dual-signal multilingual distillation framework that combines an online sequence-level signal with off-policy and on-policy token-level signals. DuDi further uses a cross-lingual verbalizer to refine teacher feedback and improve teacher-student transferability in multilingual settings. Experiments on SEA-HELM across multiple model families, scales, and teacher-student settings show that DuDi consistently outperforms competitive distillation baselines. Ablations and analyses confirm that sequence-level optimization, token-level supervision, and cross-lingual verbalization provide complementary and transferable learning signals for multilingual SLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
