补全缺失注释：基于多智能体辩论的准确且可扩展的相关性评估框架用于信息检索基准测试
Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks
AI总结：
DREAM通过多智能体辩论框架提高IR基准测试的相关性评估准确性，减少人力参与，构建BRIDGE基准并揭示缺失相关片段，改进检索器比较。
摘要：
信息检索（IR）评估仍然具有挑战性，因为不完整的IR基准数据集包含未标记的相关片段。尽管LLM和LLM-人类混合策略减少了昂贵的人力努力，但它们仍然容易产生LLM过度自信和无效的AI-人类升级。为此，我们提出了DREAM，一个基于多轮辩论的相关性评估框架，基于对立的初始立场和迭代相互批评。通过我们的协议辩论，它在某些情况下产生更准确的标签，在不确定的情况下产生更可靠的AI-人类升级，实现了95.2%的标签准确率，仅需3.5%的人力参与。使用DREAM，我们构建了BRIDGE，一个经过改进的基准，通过揭示29,824个缺失的相关片段来减轻评估偏差，并通过公平的检索器比较来启用更公平的检索器比较。然后我们重新基准测试IR系统，并将评估扩展到RAG，表明未解决的漏洞不仅扭曲了检索器排名，还驱动了检索生成的不一致。相关性评估框架可在https://github.com/DISL-Lab/DREAM-ICLR-26上获得；BRIDGE数据集可在https://github.com/DISL-Lab/BRIDGE-Benchmark上获得。
英文摘要：
Information retrieval (IR) evaluation remains challenging due to incomplete IR benchmark datasets that contain unlabeled relevant chunks. While LLMs and LLM-human hybrid strategies reduce costly human effort, they remain prone to LLM overconfidence and ineffective AI-to-human escalation. To address this, we propose DREAM, a multi-round debate-based relevance assessment framework with LLM agents, built on opposing initial stances and iterative reciprocal critique. Through our agreement-based debate, it yields more accurate labeling for certain cases and more reliable AI-to-human escalation for uncertain ones, achieving 95.2% labeling accuracy with only 3.5% human involvement. Using DREAM, we build BRIDGE, a refined benchmark that mitigates evaluation bias and enables fairer retriever comparison by uncovering 29,824 missing relevant chunks. We then re-benchmark IR systems and extend evaluation to RAG, showing that unaddressed holes not only distort retriever rankings but also drive retrieval-generation misalignment. The relevance assessment framework is available at https: //github.com/DISL-Lab/DREAM-ICLR-26; and the BRIDGE dataset is available at https://github.com/DISL-Lab/BRIDGE-Benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
