arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.12145cs.CL
收藏

Syn-QA2:利用合成问答数据集评估长尾问题中的错误假设

Syn-QA2: Evaluating False Assumptions in Long-tail Questions with Synthetic QA Datasets

Boston University(波士顿大学)

Ashwin Daswani, Rohan Sawant, Najoung Kim

AI总结:

本文提出Syn-QA2合成问答数据集,用于评估长尾问题中的错误假设,发现检测任务比生成式问答更难,且长尾问题更具挑战性。

摘要:

信息寻求问题中对错误假设(或错误前提)的敏感性对于稳健的问答(QA)系统至关重要。近期研究表明,自然发生的问题中的错误假设对当前模型构成挑战,其在生成式问答和简单检测任务上的表现均较低(Kim等人,2023)。然而,现有工作聚焦于自然发生的问题,导致对可能问题分布长尾上的模型行为分析存在空白。为此,我们引入了Syn-(QA)$^2$,一组两个合成生成的问答数据集:一个使用来自Wikidata的扰动关系生成,另一个通过扰动HotpotQA(Yang等人,2018)生成。我们对一系列大型语言模型的评估结果有三点:(1)问答中的错误假设具有挑战性,这与先前工作的发现一致;(2)二元检测任务即使与生成式问答本身的难度相比也更具挑战性,这可能归因于问题的语言结构;(3)与自然发生的问题相比,检测任务在长尾问题上更具挑战性,这凸显了我们合成数据集及生成方法的实用性。

英文摘要:

Sensitivity to false assumptions (or false premises) in information-seeking questions is critical for robust question-answering (QA) systems. Recent work has shown that false assumptions in naturally occurring questions pose challenges to current models, with low performance on both generative QA and simple detection tasks (Kim et al. 2023). However, the focus of existing work on naturally occurring questions leads to a gap in the analysis of model behavior on the long tail of the distribution of possible questions. To this end, we introduce Syn-(QA)$^2$, a set of two synthetically generated QA datasets: one generated using perturbed relations from Wikidata, and the other by perturbing HotpotQA (Yang et al. 2018). Our findings from evaluating a range of large language models are threefold: (1) false assumptions in QA are challenging, echoing the findings of prior work, (2) the binary detection task is challenging even compared to the difficulty of generative QA itself, possibly due to the linguistic structure of the problem, and (3) the detection task is more challenging with long-tail questions compared to naturally occurring questions, highlighting the utility of our synthetic datasets and generation method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑