arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.07765cs.CL
收藏

TANQ:一个表答回答问题的开放域数据集

TANQ: An open domain dataset of table answered questions

ETH Zurich(苏黎世联邦理工学院);Google DeepMind(谷歌DeepMind);Universidad Nacional de Córdoba(科尔多瓦国立大学)

Mubashara Akhtar, Chenxi Pang, Andreea Marzoca, Yasemin Altun, Julian Martin Eisenschlos

AI总结:

本文介绍了首个要求跨多源信息构建表格的开放域问答数据集TANQ,通过基准测试发现最佳模型Gemini Flash的F1分数落后人类12.3分,揭示了该任务的复杂性。

摘要:

语言模型,可能结合检索等工具使用,正成为回答问题的首选手段。在现实场景中理解和回答问题通常需要从不同来源检索信息,处理和聚合数据以提取洞见,并以结构化产物(如新表格、图表或信息图)的形式呈现复杂发现。本文介绍了TANQ,这是首个开放域问答数据集,其答案需要整合来自多个来源的信息来构建表格。我们发布了生成表格中每个单元格的完整来源归因,并在开放、预言机和闭卷设置下对最先进的语言模型进行了基准测试。我们表现最佳的基线模型Gemini Flash达到了60.7的总体F1分数,落后于人类表现12.3个百分点。我们分析了基线在不同数据集属性(如该任务所需的不同技能,包括多跳推理、数学运算和单位转换)下的表现。我们进一步讨论了模型生成答案中的常见失败,表明TANQ是一项充满诸多挑战的复杂任务。

英文摘要:

Language models, potentially augmented with tool usage such as retrieval are becoming the go-to means of answering questions. Understanding and answering questions in real-world settings often requires retrieving information from different sources, processing and aggregating data to extract insights, and presenting complex findings in form of structured artifacts such as novel tables, charts, or infographics. In this paper, we introduce TANQ, the first open domain question answering dataset where the answers require building tables from information across multiple sources. We release the full source attribution for every cell in the resulting table and benchmark state-of-the-art language models in open, oracle, and closed book setups. Our best-performing baseline, Gemini Flash reaches an overall F1 score of 60.7, lagging behind human performance by 12.3 points. We analyse baselines' performance across different dataset attributes such as different skills required for this task, including multi-hop reasoning, math operations, and unit conversions. We further discuss common failures in model-generated answers, suggesting that TANQ is a complex task with many challenges ahead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑