探究知识 grounded 对话中权衡取舍的内容规划
Investigating Content Planning for Navigating Trade-offs in Knowledge-Grounded Dialogue
AI总结：
本研究提出PLEDGE框架，探究内容规划能否平衡知识grounded对话中的特异性和归因性，发现度量感知规划在自动评估中更优但人工判断较差，提示需校准自动度量。
摘要：
知识 grounded 对话生成是一项具有挑战性的任务，因为它需要满足两个基本但往往相互竞争的约束：以特定于对话伙伴所说内容的方式做出响应，同时又要归因于底层的源文档。在这项工作中，我们揭示了这两个目标（特异性和归因性）之间的权衡，并提出了一个问题：在响应生成之前进行显式的内容规划能否帮助模型解决这一挑战？为了回答这个问题，我们设计了一个名为 PLEDGE 的框架，它允许我们实验先前工作中探索的各种规划变量，支持与度量无关和度量感知的方法。虽然内容规划显示出潜力，但关于它是否真的能帮助驾驭这一权衡的结果好坏参半——度量感知的规划机制（在训练期间使用自动度量）在自动评估中表现更好，但在人工判断中不如度量无关的机制。我们讨论了这可能是由于对自动度量的过拟合，以及未来工作需要更好地将这些度量校准到人工判断。我们希望我们分析的观察结果将为未来旨在在此背景下应用内容规划的工作提供信息。
英文摘要：
Knowledge-grounded dialogue generation is a challenging task because it requires satisfying two fundamental yet often competing constraints: being responsive in a manner that is specific to what the conversation partner has said while also being attributable to an underlying source document. In this work, we bring this trade-off between these two objectives (specificity and attribution) to light and ask the question: Can explicit content planning before the response generation help the model to address this challenge? To answer this question, we design a framework called PLEDGE, which allows us to experiment with various plan variables explored in prior work, supporting both metric-agnostic and metric-aware approaches. While content planning shows promise, our results on whether it can actually help to navigate this trade-off are mixed -- planning mechanisms that are metric-aware (use automatic metrics during training) are better at automatic evaluations but underperform in human judgment compared to metric-agnostic mechanisms. We discuss how this may be caused by over-fitting to automatic metrics and the need for future work to better calibrate these metrics towards human judgment. We hope the observations from our analysis will inform future work that aims to apply content planning in this context.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
