arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.41396cs.IRcs.AIcs.CL
收藏

ProSearch:实验科学中多约束协议检索的基准测试

ProSearch: Benchmarking Multi-Constraint Protocol Retrieval in Experimental Science

Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心);Institute of Zoology, Chinese Academy of Sciences(中国科学院动物研究所);University of Chinese Academy of Sciences(中国科学院大学);Westlake University(西湖大学)

Wenliang Liang, Zhiyuan Ning, Haolin Chen, Yuanchun Zhou, Wenjuan Cui, Yi Du

AI总结:

本研究提出ProSearch基准测试,针对实验科学多约束协议检索,构建含1515个查询的测试集,实验显示现有检索器性能有限,为科学检索系统评估提供平台。

摘要:

在科学领域的人工智能应用场景中,实验设计与方法复现需要检索满足多个相互依赖约束的协议,但现有基准测试缺乏对该能力的系统性评估。为填补这一空白,我们提出ProSearch——一个用于实验科学多约束协议检索的基准测试。ProSearch具有三个核心特性:(1)真实性,查询基于真实实验记录与专家定义模板;(2)复杂性,涵盖60个实验领域及其依赖关系;(3)诊断性,每个查询包含3-5个显式约束,包括数值约束与否定约束。我们通过基于大语言模型(LLM)的提取、数据驱动的约束选择、受控查询生成以及基于证据的相关性验证构建ProSearch。该基准测试包含六个实验科学领域的1515个查询。我们使用19种代表性检索器在ProSearch上开展大量实验并进行补充分析。表现最佳的检索器平均nDCG@10仅为0.290,经重排序后提升至0.474。结构化诊断分析显示,显式约束匹配提供了强排序信号,而人工误差分析表明联合约束绑定是关键瓶颈。ProSearch为系统性评估科学检索系统提供了测试平台。

英文摘要:

In AI for Science scenarios, experimental design and method reproduction require retrieving protocols that satisfy multiple interdependent constraints, yet existing benchmarks lack systematic evaluation of this capability. To address this gap, we introduce ProSearch, a benchmark for multi-constraint protocol retrieval in experimental science. ProSearch has three key features: (1) realism, with queries grounded in real experimental records and expert-defined templates; (2) complexity, with 60 experimental fields and their dependencies; and (3) diagnosticity, with 3-5 explicit constraints per query, including numerical and negation constraints. We construct ProSearch through LLM-based extraction, data-driven constraint selection, controlled query generation, and evidence-based relevance validation. The resulting benchmark contains 1,515 queries across six experimental science domains. We conduct extensive experiments on ProSearch with 19 representative retrievers and complementary analyses. The best-performing retriever achieves an average nDCG@10 of only 0.290, which improves to 0.474 with reranking. Structured diagnostic analysis shows that explicit constraint matching provides a strong ranking signal, while manual error analysis identifies joint constraint binding as a key bottleneck. ProSearch provides a testbed for systematically evaluating scientific retrieval systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑