ProSearch: Benchmarking Multi-Constraint Protocol Retrieval in Experimental Science
Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心);Institute of Zoology, Chinese Academy of Sciences(中国科学院动物研究所);University of Chinese Academy of Sciences(中国科学院大学);Westlake University(西湖大学)
Wenliang Liang, Zhiyuan Ning, Haolin Chen, Yuanchun Zhou, Wenjuan Cui, Yi Du
In AI for Science scenarios, experimental design and method reproduction require retrieving protocols that satisfy multiple interdependent constraints, yet existing benchmarks lack systematic evaluation of this capability. To address this gap, we introduce ProSearch, a benchmark for multi-constraint protocol retrieval in experimental science. ProSearch has three key features: (1) realism, with queries grounded in real experimental records and expert-defined templates; (2) complexity, with 60 experimental fields and their dependencies; and (3) diagnosticity, with 3-5 explicit constraints per query, including numerical and negation constraints. We construct ProSearch through LLM-based extraction, data-driven constraint selection, controlled query generation, and evidence-based relevance validation. The resulting benchmark contains 1,515 queries across six experimental science domains. We conduct extensive experiments on ProSearch with 19 representative retrievers and complementary analyses. The best-performing retriever achieves an average nDCG@10 of only 0.290, which improves to 0.474 with reranking. Structured diagnostic analysis shows that explicit constraint matching provides a strong ranking signal, while manual error analysis identifies joint constraint binding as a key bottleneck. ProSearch provides a testbed for systematically evaluating scientific retrieval systems.
UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
University of Waterloo(滑铁卢大学);Georgia Institute of Technology(佐治亚理工学院);Hong Kong University of Science and Technology(香港科技大学);Google DeepMind(谷歌DeepMind)
ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition
Fudan University(复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Nanyang Technological University(南洋理工大学);University of New South Wales(新南威尔士大学);National University of Singapore(新加坡国立大学);Wuhan University(武汉大学)
SWE-Explore: Benchmarking How Coding Agents Explore Repositories
Shanghai Jiao Tong University(上海交通大学);Xinjiang University(新疆大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Independent Researcher(独立研究者);The Chinese University of Hong Kong(香港中文大学)
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Huawei Poisson Lab(华为泊松实验室);Waseda University(早稻田大学)
Benchmarking Retrieval-Augmented Generation for Chemistry
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校Siebel计算与数据科学学院);National Library of Medicine, National Institutes of Health(美国国立卫生研究院国家医学图书馆)
RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Model
School of Information, Renmin University of China(中国人民大学信息学院);Institute for Advanced Algorithms Research(高级算法研究院);School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院)
SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Tencent Yuanbao(腾讯元宝);Tsinghua University(清华大学);ARC Lab, Tencent(腾讯ARC实验室);The University of Hong Kong(香港大学)
Benchmarking and Enabling Efficient Chinese Medical Retrieval via Asymmetric Encoders
University of Science and Technology of China(中国科学技术大学);State Key Laboratory of Cognitive Intelligence(认知智能全国重点实验室);iFlytek Research(科大讯飞研究院);HeFei Institute of Technology(合肥工业大学)