矛盾下的覆盖度:一种感知否定的检索基准
Coverage Under Contradiction: A Benchmark for Negation-Aware Retrieval
AI总结:
本文构建了感知否定的检索基准CoverageBench-Negate,发现BM25区分矛盾证据的能力优于Qwen3等密集检索器,相关基准已在Hugging Face Datasets发布。
摘要:
覆盖度评估基准用于量化检索到的证据是否涵盖满足信息需求所需的不同信息片段。现有基准大多考虑预训练检索组件可能已掌握的事实主张,未探究当证据与这些先验知识矛盾时检索器能否找到语料库中的证据。本文研究检索器在其参数知识与语料库信息不一致时的表现,为此开发了一种在基于 nugget 的覆盖度基准中加入可控矛盾的方法:为每个 nugget 创建否定对应项,并编辑其支撑文档以支持该否定同时保留上下文。将该方法应用于衡量信息覆盖度的基准 CoverageBench,构建出涵盖6个数据集、约63000组原始-编辑证据对(每个 nugget-文档链接对应一对)的 CoverageBench-Negate。为每组对附加了丰富的构建元数据,还提供了所有对的跨族大语言模型(LLM)判断及样本的人工审计。基线实验显示,BM25区分原始与编辑的近孪生证据的能力远优于密集检索器(Qwen3和Jina),后者表现接近随机水平。本文将该基准(包含nuggets、文档、证据对)及构建材料(含用于创建的提示)发布在Hugging Face Datasets上。
英文摘要:
Coverage-evaluation benchmarks quantify whether retrieved evidence covers the different pieces of information needed to satisfy an information need. Existing benchmarks mostly consider factual claims that may already be known to pretrained retrieval components, leaving open whether retrievers find corpus evidence when it contradicts those priors. We study how retrievers behave when their parametric knowledge and the information in a corpus disagree. To do so, we develop a process to augment a nugget-based coverage benchmark with controlled contradictions. For each nugget, we create a negated counterpart and edit its supporting documents to support that negation while preserving the context. Applying this process to CoverageBench, a benchmark measuring information coverage, we create CoverageBench-Negate, spanning six datasets and roughly 63,000 original-edited evidence pairs (one per nugget-document link). We attach rich construction metadata to every pair and additionally provide out-of-family LLM judgments over all pairs and a human audit of a sample. In baseline experiments, BM25 separates original from edited near-twin evidence substantially better than dense retrievers (Qwen3 and Jina), which stay near chance. We release the benchmark on Hugging Face Datasets (nuggets, documents, pairs) together with the construction materials, including the prompts used to create it.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。