CRAFT: 通过强化学习进行多跳问答的校准推理与答案忠实轨迹
Does Faithfulness-Guided Alignment Hurt Accuracy? Unlocking Accurate and Faithful Post-Retrieval Reasoning
AI总结：
CRAFT通过强化学习框架提升多跳问答的推理准确性和答案忠实度，结合确定性奖励和判官奖励，增强推理轨迹的可审计性与语义一致性。
摘要：
检索增强的大语言模型在优化时能实现强答案准确率，但在噪声检索下易出现
英文摘要：
Retrieval-augmented generation (RAG) can achieve strong answer accuracy on multi-hop questions, but outcome-level rewards often leave reasoning traces weakly grounded and difficult to audit. Under noisy retrieval, models may exhibit right-answer-wrong-reason failures, where the final answer is correct but the supporting rationale exploits shortcuts or unsupported evidence. We therefore ask whether faithfulness-guided alignment hurts answer accuracy in post-retrieval reasoning. To study this question, we propose CRAFT (Calibrated Reasoning with Answer-Faithful Traces), a reinforcement learning framework for the response-generation stage of retrieval-augmented multi-hop question answering. CRAFT trains models to produce structured reasoning traces with configurable auditability, while combining deterministic rewards for format compliance, answer correctness, and citation validity with a judge-based reward for semantic faithfulness. Experiments across model scales and benchmarks show that CRAFT unlocks task-specific reasoning capacity from 1.5B upward, improving both answer accuracy and Faithfulness; at 0.5B, performance remains sharply template-dependent. At 7B, CRAFT improves Faithfulness over the Base model in all evaluated settings and remains competitive with strong closed-source models. Code is available at https://github.com/Ameame1/CRAFT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
