面向开放域数值事实验证的可信基准：基于声明分解增强
A Benchmark for Open-Domain Numerical Fact-Checking Enhanced by Claim Decomposition
AI总结：
针对现有数值事实验证基准证据相关性低和时间泄漏问题，提出QuanTemp++数据集，通过模拟人类核查员的声明分解过程收集证据，并评估不同分解范式的检索与验证性能。
摘要：
对数值声明进行事实验证至关重要，因为数字的存在会给人一种真实性的错觉，尽管其可能是虚假的，从而可能对社会造成灾难性影响。以往自动事实核查的研究并未主要关注自然语言中的数值声明。典型的人类事实核查员首先检索与声明不同数值方面相关的证据，然后对这些证据进行推理以预测声明的真实性。因此，人类事实核查员的搜索过程是一项关键技能，构成了验证过程的基础。模拟真实世界场景对于开发包含此类技能的自动化方法至关重要。然而，现有基准采用启发式声明分解方法，并辅以弱监督的网络搜索来收集验证声明的证据。这有时会导致证据相关性较低，且来源带有噪声，并存在时间泄漏问题，从而使声明验证的检索设置不够真实。为此，我们引入了QuanTemp++：一个包含自然数值声明、开放域语料库以及每条声明对应相关证据的数据集。证据通过一个近似模拟人类事实核查员方法的声明分解过程收集，并带有真实性标签，确保不存在时间泄漏。基于该数据集，我们还刻画了关键声明分解范式的检索性能。最后，我们观察了这些范式对验证流程结果的影响，并得出了相关见解。数据管道的代码及数据链接可在https://github.com/VenkteshV/QuanTemp_Plus获取。
英文摘要：
Fact-checking numerical claims is critical as the presence of numbers provide mirage of veracity despite being fake potentially causing catastrophic impacts on society. The prior works in automatic fact verification do not primarily focus on natural numerical claims. A typical human fact-checker first retrieves relevant evidence addressing the different numerical aspects of the claim and then reasons about them to predict the veracity of the claim. Hence, the search process of a human fact-checker is a crucial skill that forms the foundation of the verification process. Emulating a real-world setting is essential to aid in the development of automated methods that encompass such skills. However, existing benchmarks employ heuristic claim decomposition approaches augmented with weakly supervised web search to collect evidences for verifying claims. This sometimes results in less relevant evidences and noisy sources with temporal leakage rendering a less realistic retrieval setting for claim verification. Hence, we introduce QuanTemp++: a dataset consisting of natural numerical claims, an open domain corpus, with the corresponding relevant evidence for each claim. The evidences are collected through a claim decomposition process approximately emulating the approach of human fact-checker and veracity labels ensuring there is no temporal leakage. Given this dataset, we also characterize the retrieval performance of key claim decomposition paradigms. Finally, we observe their effect on the outcome of the verification pipeline and draw insights. The code for data pipeline along with link to data can be found at https://github.com/VenkteshV/QuanTemp_Plus
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
