arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.00827cs.CLcs.AIcs.LG
收藏

基于外部代理指标反馈的语言模型自我精炼

Self-Refinement of Language Models from External Proxy Metrics Feedback

University of Pennsylvania(宾夕法尼亚大学);IBM Research AI(IBM研究院人工智能部门)

Keshav Ramji, Young-Suk Lee, Ramón Fernandez Astudillo, Md Arafat Sultan, Tahira Naseem, Asim Munawar, Radu Florian, Salim Roukos

AI总结:

本文提出 ProMiSe,利用原则特定的外部代理指标反馈引导 LLM 逐项迭代精炼回复,并在文档问答任务中验证其可提升 Flan-T5-XXL 与 Llama-2-13B-Chat 的表现。

摘要:

大型语言模型(LLM)在提供回复时,通常需要兼顾多个目标。例如,在基于文档的回复生成中,智能体回复既要与用户查询相关,又要以给定文档为依据。本文提出 Proxy Metric-based Self-Refinement(ProMiSe,基于代理指标的自我精炼),使 LLM 能够在外部指标反馈的引导下,沿关键质量维度精炼其初始回复,从而获得整体更优的最终回复。ProMiSe 利用针对特定原则的代理指标所给出的回复质量反馈,每次围绕一项原则迭代精炼回复。我们将 ProMiSe 应用于开源语言模型 Flan-T5-XXL 和 Llama-2-13B-Chat,并在基于文档的问答数据集 MultiDoc2Dial 和 QuAC 上评估其性能,结果表明自我精炼提升了回复质量。我们进一步表明,使用 ProMiSe 生成的合成对话数据对 Llama-2-13B-Chat 进行微调,相比零样本基线以及在人工标注数据上进行监督微调的模型,均取得了显著性能提升。

英文摘要:

It is often desirable for Large Language Models (LLMs) to capture multiple objectives when providing a response. In document-grounded response generation, for example, agent responses are expected to be relevant to a user's query while also being grounded in a given document. In this paper, we introduce Proxy Metric-based Self-Refinement (ProMiSe), which enables an LLM to refine its own initial response along key dimensions of quality guided by external metrics feedback, yielding an overall better final response. ProMiSe leverages feedback on response quality through principle-specific proxy metrics, and iteratively refines its response one principle at a time. We apply ProMiSe to open source language models Flan-T5-XXL and Llama-2-13B-Chat, to evaluate its performance on document-grounded question answering datasets, MultiDoc2Dial and QuAC, demonstrating that self-refinement improves response quality. We further show that fine-tuning Llama-2-13B-Chat on the synthetic dialogue data generated by ProMiSe yields significant performance improvements over the zero-shot baseline as well as a supervised fine-tuned model on human annotated data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑