QUATRO：用于LLM微调的查询自适应信任区域策略优化
QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning
AI总结：
QUATRO通过原则性优化直接施加信任区域约束，实现稳定且可控的LLM微调，提升训练稳定性与熵控制。
摘要：
基于强化学习（RL）的GRPO风格LLM微调算法近年来备受关注。然而，依赖启发式信任区域近似，它们可能导致脆弱的优化行为，因为全局重要性比裁剪和分组归一化无法调节那些重要性比超出裁剪范围的样本。我们提出了查询自适应信任区域策略优化（QUATRO），通过原则性的优化直接施加信任区域约束。这产生了一个清晰且可解释的目标，使政策更新能够得到明确控制，并实现稳定、熵控制的优化，其中稳定项源自精确的信任区域公式的内在特性。在多样化的数学推理基准上经 empirical 验证，QUATRO 在增加的政策陈旧度和激进学习率下表现出稳定的训练，整个训练过程中保持良好的熵控制。
英文摘要：
GRPO-style reinforcement learning (RL)-based LLM fine-tuning algorithms have recently gained popularity. Relying on heuristic trust-region approximations, however, they can lead to brittle optimization behavior, as global importance-ratio clipping and group-wise normalization fail to regulate samples whose importance ratios fall outside the clipping range. We propose Query-Adaptive Trust-Region policy Optimization (QUATRO), which directly enforces trust-region constraints through a principled optimization. This yields a clear and interpretable objective that enables explicit control over policy updates and stable, entropy-controlled optimization, with a stabilizer terms arising intrinsically from the exact trust-region formulation. Empirically verified on diverse mathematical reasoning benchmarks, QUATRO shows stable training under increased policy staleness and aggressive learning rates, maintaining well-controlled entropy throughout training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
