arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.14829cs.CL
收藏

这是一张糟糕的表格吗?细看从文本生成表格的评估方法

Is This a Bad Table? A Closer Look at the Evaluation of Table Generation from Text

Adobe Research(奥多比研究院)

Pritika Ramu, Aparna Garimella, Sambaran Bandyopadhyay

AI总结:

本文提出 TabEval 评估策略,通过将表格分解为原子陈述并用蕴含度量与真实陈述比较来捕捉表格语义,并在包含 1,250 个维基百科表格的数据集上验证其与人类判断的相关性优于现有指标。

摘要:

理解生成的表格是否具有良好质量,对于能否在自动创建或编辑文档时使用它至关重要。在本文中,我们强调现有的表格质量评估指标未能捕捉表格的整体语义,有时会不公平地惩罚好的表格并奖励差的表格。我们提出 TabEval,一种新颖的表格评估策略,它通过先将表格分解为一系列自然语言原子陈述,再使用基于蕴含关系的度量将这些陈述与真实陈述进行比较,从而捕捉表格语义。为了验证我们的方法,我们构建了一个数据集,其中包含 1,250 个多样化维基百科表格的文本描述,涵盖一系列主题和结构,与现有数据集的有限范围形成对比。我们使用无监督和有监督的文本到表格生成方法,将 TabEval 与现有指标进行比较,证明其在四个数据集上与人类对表格质量判断的相关性更强。

英文摘要:

Understanding whether a generated table is of good quality is important to be able to use it in creating or editing documents using automatic methods. In this work, we underline that existing measures for table quality evaluation fail to capture the overall semantics of the tables, and sometimes unfairly penalize good tables and reward bad ones. We propose TabEval, a novel table evaluation strategy that captures table semantics by first breaking down a table into a list of natural language atomic statements and then compares them with ground truth statements using entailment-based measures. To validate our approach, we curate a dataset comprising of text descriptions for 1,250 diverse Wikipedia tables, covering a range of topics and structures, in contrast to the limited scope of existing datasets. We compare TabEval with existing metrics using unsupervised and supervised text-to-table generation methods, demonstrating its stronger correlation with human judgments of table quality across four datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑