arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.19152cs.CLcs.AI
收藏

VEPO:用于低资源语言基础模型的变量熵策略优化

VEPO: Variable Entropy Policy Optimization for Low-Resource Language Foundation Models

Chonghan Liu, Yimin Du, Qi An, Xin He, Cunqi Zhai, Fei Tan, Weijia Lin, Xiaochun Gong, Yongchao Deng, Shousheng Jia, Xiangzheng Zhang

AI总结:

本文提出VEPO,通过强化学习可验证奖励整合确定性结构约束,提升低资源语言的分词效率和翻译质量。

摘要:

大型语言模型在低资源语言上常表现不佳,主要由于子词分词效率低下和训练数据不平衡。本文提出变量熵策略优化(VEPO),利用可验证奖励强化学习将确定性结构约束纳入策略对齐过程。该框架确保指定序列长度、鲁棒格式一致性和严格语言正确性,均在训练期间强制执行。核心是变量熵机制,使模型能动态校准字面忠实度与语义自然度之间的平衡。通过整合熵 tempered 优势估计与非对称截断,VEPO在保持稳健探索的同时缓解策略崩溃。在90个FLORES-200、COMET-22、chrF方向上的实证评估表明,VEPO在分词效率和翻译质量上均有显著提升,缩小了低资源语言的性能差距。

英文摘要:

Large language models frequently exhibit suboptimal performance on low resource languages, primarily due to inefficient subword segmentation and systemic training data imbalances. In this paper, we propose Variable Entropy Policy Optimization (VEPO), which leverages Reinforcement Learning with Verifiable Rewards to incorporate deterministic structural constraints into the policy alignment process. This framework ensures prescribed sequence length, robust format consistency, and rigorous linguistic well formedness, all enforced during training. Central to our approach is a variable entropy mechanism that enables the model to dynamically calibrate the equilibrium between literal fidelity and semantic naturalness by modulating the exploration exploitation manifold. By integrating entropy tempered advantage estimation with asymmetric clipping, VEPO sustains robust exploration while mitigating policy collapse. Empirical evaluations across 90 FLORES-200, COMET-22, chrF directions demonstrate that VEPO yields substantial improvements in both tokenization efficiency and translation quality, bridging the performance gap for underrepresented languages.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑