arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.09558cs.CL
收藏

如果你换一种说法呢?:解释格式如何影响人类反馈效果与用户感知

What if you said that differently?: How Explanation Formats Affect Human Feedback Efficacy and User Perception

University of Pennsylvania(宾夕法尼亚大学)

Chaitanya Malaviya, Subin Lee, Dan Roth, Mark Yatskar

AI总结:

本文研究问答模型解释理由的呈现格式如何影响用户反馈效果与用户理解信任,通过两项用户研究发现理由格式显著影响反馈难易度和模型执行反馈的效果,且带上下文归因和深入推理的格式更能提升用户理解与信任。

摘要:

从自然语言处理模型的最终用户那里获取反馈有助于改进模型。然而,我们应该如何向用户呈现模型响应,以便它们最容易被用户反馈所纠正?此外,用户重视哪些属性来理解和信任响应?我们通过分析问答模型为支持其答案而生成的理由(或解释)的效果来回答这些问题。我们特别考虑分解式问答模型,这类模型首先根据上下文和问题提取一个中间理由,然后仅使用该理由来回答问题。理由概述了模型回答问题所采用的方法。我们的工作考虑了这些理由的各种格式,这些格式根据明确定义的感兴趣属性而变化。我们使用少样本提示从语言模型中为两个数据集采样理由,然后进行两项用户研究。首先,我们向用户展示错误答案以及各种格式的相应理由,并要求他们提供自然语言反馈来修改理由。然后,我们通过上下文学习来衡量这种反馈在修补这些理由方面的有效性。第二项研究评估了当模型答案正确时,不同的理由格式在多大程度上能够帮助用户理解和信任模型答案。我们发现,理由格式显著影响(1)用户为理由提供反馈的难易程度,以及(2)模型随后执行这种反馈的难易程度。此外,带有对上下文的归因和深入推理的格式显著增强了用户报告的对模型输出的理解和信任。

英文摘要:

Eliciting feedback from end users of NLP models can be beneficial for improving models. However, how should we present model responses to users so they are most amenable to be corrected from user feedback? Further, what properties do users value to understand and trust responses? We answer these questions by analyzing the effect of rationales (or explanations) generated by QA models to support their answers. We specifically consider decomposed QA models that first extract an intermediate rationale based on a context and a question and then use solely this rationale to answer the question. A rationale outlines the approach followed by the model to answer the question. Our work considers various formats of these rationales that vary according to well-defined properties of interest. We sample rationales from language models using few-shot prompting for two datasets, and then perform two user studies. First, we present users with incorrect answers and corresponding rationales in various formats and ask them to provide natural language feedback to revise the rationale. We then measure the effectiveness of this feedback in patching these rationales through in-context learning. The second study evaluates how well different rationale formats enable users to understand and trust model answers, when they are correct. We find that rationale formats significantly affect how easy it is (1) for users to give feedback for rationales, and (2) for models to subsequently execute this feedback. In addition, formats with attributions to the context and in-depth reasoning significantly enhance user-reported understanding and trust of model outputs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑