通过一致性驱动的强化学习改进跨语言事实回忆
PolyFact: Comparing Consistency-Driven Post-training Methods for Cross-Lingual Factual Recall
AI总结：
提出PolyFact数据集，利用GRPO强化学习方法提升大语言模型的跨语言事实回忆一致性，优于监督微调，并揭示其通过减少语言专用表示实现跨语言共享的机制。
摘要：
主要用英语数据训练的大型语言模型（LLMs）编码了丰富的世界知识，但通常无法在其他语言中可靠地表达这些知识，这种现象称为跨语言事实不一致性。为了研究和解决这一问题，我们引入了PolyFact，一个大规模并行多语言事实问答数据集，包含12种类型多样的语言中的10万个基于Wikidata的事实。利用PolyFact，我们比较了轻量持续预训练（CPT）、监督微调（SFT）和通过组相对策略优化（GRPO）的强化学习在Qwen-2.5-7B和OLMo-2-1124-7B中改进跨语言事实回忆的效果。我们发现GRPO始终优于SFT，提高了跨语言一致性和对未见语言的泛化能力，而并行数据上的CPT带来的额外收益有限。机制分析进一步表明，GRPO通过减少MLP层和注意力头中的语言专门化来重组多语言路由，从而促进更共享的跨语言表示。我们发布了代码、模型和数据集。
英文摘要：
Large language models (LLMs) trained predominantly on English data encode substantial world knowledge, yet often fail to express it reliably in other languages, a phenomenon known as cross-lingual factual inconsistency. To study this, we introduce PolyFact, a fully parallel multilingual factual QA dataset of 60K Wikidata-grounded facts across 12 typologically diverse languages, and propose consistency-driven GRPO with cross-lingual reward pooling. We compare our method against supervised fine-tuning (SFT) and the consistency-enhancement baselines DCO and CM-Align on OLMo-2-1124-7B and Qwen-2.5-7B, and analyze whether light continual pretraining (CPT) on parallel data provides a useful foundation for post-training. No single method dominates: SFT maximises in-distribution accuracy but not consistency, DCO yields the strongest consistency gains but fails to transfer to free-form generation, and our GRPO variant achieves the strongest transfer to free-form recall and unseen languages on the multilingual base model. CPT mildly aids monolingual models but harms multilingual ones. Mechanistic analyses suggest that GRPO is associated with reduced language specialization, consistent with greater sharing of representations across languages. We release our code, models, and dataset publicly.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
