FactIR：面向事实核查的真实世界零样本开放域检索基准
FactIR: A Real-World Zero-shot Open-Domain Retrieval Benchmark for Fact-Checking
AI总结：
本文提出源自Factiverse生产日志并经人工标注的真实世界开放域检索基准FactIR，在零样本设置下评估最先进检索模型，为事实核查检索系统开发提供实践见解。
摘要：
自动化事实核查领域越来越依赖于检索基于网络的证据，以判断现实场景中声明的真实性。这一过程中的一个重大挑战不仅是检索相关信息，还要识别能够同时支持和反驳复杂声明的证据。传统检索方法可能返回直接针对声明的文档或倾向于支持声明的文档，但往往难以处理需要间接推理的更复杂声明。虽然已有一些基准和方法针对事实核查的检索任务，但一直缺乏一个全面的真实世界开放域基准。在本文中，我们提出了一个真实世界检索基准 FactIR，该基准源自 Factiverse 的生产日志，并经过人工标注增强。我们在零样本设置下对 FactIR 上的最先进检索模型进行了严格评估，并为开发实用的事实核查检索系统提供了见解。代码和数据可在 https://github.com/factiverse/factIR 获取。
英文摘要：
The field of automated fact-checking increasingly depends on retrieving web-based evidence to determine the veracity of claims in real-world scenarios. A significant challenge in this process is not only retrieving relevant information, but also identifying evidence that can both support and refute complex claims. Traditional retrieval methods may return documents that directly address claims or lean toward supporting them, but often struggle with more complex claims requiring indirect reasoning. While some existing benchmarks and methods target retrieval for fact-checking, a comprehensive real-world open-domain benchmark has been lacking. In this paper, we present a real-world retrieval benchmark FactIR, derived from Factiverse production logs, enhanced with human annotations. We rigorously evaluate state-of-the-art retrieval models in a zero-shot setup on FactIR and offer insights for developing practical retrieval systems for fact-checking. Code and data are available at https://github.com/factiverse/factIR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
