知识级一致性强化学习：长形式事实性的双事实对齐
Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factuality
AI总结：
本文提出KLCF框架，通过双事实对齐机制提升长形式生成的事实性，有效缓解幻觉和保守倾向，提升精度与召回率。
摘要：
大型语言模型在长形式生成中仍难以解决幻觉问题，现有强化学习从人类反馈（RLHF）框架的偏好奖励常忽视模型自身知识边界。本文提出知识级一致性强化学习框架（KLCF），从分布对齐角度重新审视该问题。KLCF将长形式事实性形式化为策略模型表达知识分布与基模型参数化知识分布之间的双向分布匹配目标：在生成不得超出基知识支持集的约束下，最大化高概率事实的覆盖，从而联合优化精度和召回率。为此，我们设计了双事实对齐机制，通过从基模型采样构建事实检查表来近似召回项，并通过轻量级真实性奖励模型约束幻觉。两个组件共同优化，无需外部检索。实验结果表明，KLCF在多个长形式基准和模型规模上一致提升事实性指标，有效缓解幻觉和保守倾向，同时保持效率和可扩展性。
英文摘要：
Hallucination in large language models (LLMs) during long-form generation remains difficult to address under existing reinforcement learning from human feedback (RLHF) frameworks, as their preference rewards often overlook the model's own knowledge boundaries. In this paper, we propose the $\textbf{K}$nowledge-$\textbf{L}$evel $\textbf{C}$onsistency Reinforcement Learning $\textbf{F}$ramework ($\textbf{KLCF}$), which re-examines this problem from a distribution alignment perspective. KLCF formalizes long-form factuality as a bidirectional distribution matching objective between the policy model's expressed knowledge distribution and the base model's parametric knowledge distribution: under the constraint that generation must not exceed the support set of the base knowledge, the objective maximizes coverage of high-probability facts, thereby jointly optimizing precision and recall. To achieve this, we design a Dual-Fact Alignment mechanism that approximates the recall term using a factual checklist constructed by sampling from the base model, and constrains hallucinations with a lightweight truthfulness reward model. Both components are jointly optimized and require no external retrieval throughout training. Experimental results demonstrate that KLCF consistently improves factuality metrics across multiple long-form benchmarks and model scales, effectively alleviating hallucination and over-conservatism while maintaining efficiency and scalability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
