arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.02397cs.CL
收藏

学习使用细粒度自然语言反馈进行精炼

Learning to Refine with Fine-Grained Natural Language Feedback

The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Manya Wadhwa, Xinyu Zhao, Junyi Jessy Li, Greg Durrett

AI总结:

提出DCR方法,将精炼分解为检测、批评、精炼三步,通过细粒度反馈提升文档摘要的事实一致性,优于现有端到端方法。

摘要:

近期研究探索了大型语言模型(LLMs)识别并纠正其生成响应中错误的能力。这些精炼方法经常评估何种规模的模型能够针对哪些问题进行精炼,但较少关注有效的精炼反馈应具备何种形态。在本工作中,我们提出将带反馈的精炼视为三种不同LLM能力的组合:(1)检测不良生成;(2)生成细粒度自然语言批评;(3)使用细粒度反馈进行精炼。第一步可通过高性能判别模型实现,第二和第三步可通过提示或微调的LLM实现。所提出的“检测、批评、精炼”(DCR)方法的一个关键特性是,第二步的批评模型能够提供关于错误的细粒度反馈,这得益于将判别任务卸载到第一步的独立模型中。我们展示了不同能力的模型在提升文档摘要事实一致性任务上,通过DCR进行精炼均能受益。总体而言,DCR持续优于现有的端到端精炼方法以及当前未经事实性批评微调的已训练模型。

英文摘要:

Recent work has explored the capability of large language models (LLMs) to identify and correct errors in LLM-generated responses. These refinement approaches frequently evaluate what sizes of models are able to do refinement for what problems, but less attention is paid to what effective feedback for refinement looks like. In this work, we propose looking at refinement with feedback as a composition of three distinct LLM competencies: (1) detection of bad generations; (2) fine-grained natural language critique generation; (3) refining with fine-grained feedback. The first step can be implemented with a high-performing discriminative model and steps 2 and 3 can be implemented either via prompted or fine-tuned LLMs. A key property of the proposed Detect, Critique, Refine ("DCR") method is that the step 2 critique model can give fine-grained feedback about errors, made possible by offloading the discrimination to a separate model in step 1. We show that models of different capabilities benefit from refining with DCR on the task of improving factual consistency of document grounded summaries. Overall, DCR consistently outperforms existing end-to-end refinement approaches and current trained models not fine-tuned for factuality critiquing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑