SciNet: 评估关系感知科学文献检索中的AI代理
SciNet: Evaluating AI Agents in Relation-Aware Scientific Literature Retrieval
AI总结：
针对现有检索代理难以理解论文间复杂关系网络的问题，提出首个关系感知数据集SciNet，通过三类任务系统评估代理，发现准确率低于20%，但下游综述质量提升25.3%。
摘要：
AI代理在科学研究的文献检索中已被广泛采用，催生了如Deep Research等工具。然而，现有的检索代理主要依赖基于关键词或嵌入的方法。虽然它们在捕捉内容级相似性方面有效，但难以理解科学论文之间的复杂关系网络，例如识别相互印证或冲突的研究以及追踪技术谱系。这一根本性限制常常导致知识结构碎片化、研究情感误解以及集体科学进展建模失效。为解决这一限制，我们引入了SciNet，这是首个用于信息检索代理的科学网络关系感知数据集。该数据集基于包含7个学科、2.69亿篇论文的元数据库，并包含8940个精心设计的任务，系统性地捕捉了三个层次的关系理解：以自我为中心检索具有新颖知识结构的论文、成对识别学术关系以及路径式重建科学演化。对三类检索代理的广泛评估表明，它们在关系感知任务上的准确率通常低于20%，凸显了当前检索范式的根本缺陷。重要的是，在下游文献综述应用中，配备SciNet的代理在综述质量上实现了25.3%的提升，突出了关系感知检索对深化科学见解的关键价值。我们在https://github.com/tsinghua-fib-lab/SciNet公开发布SciNet以支持未来研究。
英文摘要：
AI agents have seen widespread adoption in information retrieval for scientific research, giving rise to tools such as Deep Research. However, existing retrieval agents mainly rely on keyword- or embedding-based methods. While effective at capturing content-level similarities, they struggle to understand complex relational networks among scientific papers, such as identifying corroborating or conflicting studies and tracing technological lineages. This fundamental limitation often results in fragmented knowledge structures, misinterpreted research sentiment, and ineffective modeling of collective scientific progress. To address this limitation, we introduce SciNet, the first Scientific Network relation-aware dataset for information retrieval agents. Built on a meta-database of 269 million papers across 7 disciplines and containing 8,940 carefully designed tasks, SciNet systematically captures three levels of relational understanding: ego-centric retrieval of papers with novel knowledge structures, pairwise identification of scholarly relationships, and path-wise reconstruction of scientific evolution. Extensive evaluation of three categories of retrieval agents shows that their accuracy on relation-aware tasks often falls below 20%, highlighting a fundamental shortcoming of current retrieval paradigms. Importantly, in a downstream literature review application, agents empowered with SciNet achieve a 25.3% improvement in review quality, highlighting the critical value of relation-aware retrieval for deepening scientific insights. We publicly release SciNet at https://github.com/tsinghua-fib-lab/SciNet to support future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
