教会语言模型使用人类应试策略检查基于事实的声明真实性
Teaching Language Models to Check Grounded Claim Factuality with Human Test-Taking Strategies
AI总结：
将基于事实的声明真实性检查建模为真假阅读理解任务，通过提示语言模型使用明确的应试策略进行高效推理，并训练小语言模型以降低推理成本。
摘要：
基于事实的声明真实性检查对于大型语言模型（LLM）应用（如检索增强生成）非常重要，因为它帮助用户评估生成输出的正确性。现有的使用蕴含分类器的指标需要针对数据集调整阈值，而基于LLM的方法通常使用直接提示，这未能充分利用LLM的推理能力。我们通过将基于事实的声明真实性检查建模为真假阅读理解任务，并提示LLM使用明确的应试策略进行高效推理来解决这一问题。与无引导的开放式推理相比，我们的方法减少了超过80%的令牌使用量，并在两个真实性基准测试中取得了与更昂贵替代方案竞争的性能，在一个基准上达到了新的最先进水平。为了进一步降低推理成本，我们训练小语言模型（SLM）来替代检查流程中的LLM。通过监督微调（SFT）和自我修正机制，SLM学会了改进其真实性判断。实验结果表明，生成的SLM在性能上与强基线相当，结合了低推理成本和生成支持理由以支持可解释性。代码和数据集将在接收后发布。
英文摘要：
Grounded claim factuality checking is important for large language model (LLM) applications such as retrieval-augmented generation, as it helps users assess the correctness of generated outputs. Existing metrics using entailment classifiers require dataset-specific threshold tuning, while LLM-based approaches often use direct prompting, which underutilises the reasoning capabilities of LLMs. We address this by formulating grounded claim factuality checking as a true/false reading comprehension task and prompting LLMs with explicit test-taking strategies for efficient reasoning. Our method reduces token usage by over 80% compared to unguided open-ended reasoning, and achieves competitive performance to more expensive alternatives across two factuality benchmarks, setting a new state of the art on one. To further reduce inference cost, we train small language models (SLMs) to replace LLMs in the checking pipeline. Using supervised fine-tuning (SFT) and a self-revision mechanism, the SLMs learn to improve their factuality judgements. Experimental results show that the resulting SLMs perform on par with strong baselines, combining low inference costs with generating supporting rationales to support interpretability. Code and datasets will be released upon acceptance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
