RATIO：面向科学文献中类型化构思操作检索的基准
RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature
- The Hebrew University of Jerusalem（耶路撒冷希伯来大学）
- The Allen Institute for AI (AI2)（艾伦人工智能研究所）
机构由 AI 辅助整理，请以论文原文为准。
中文总结 AI 辅助
本研究提出名为RATIO的大规模基准，定义三类构思操作的检索任务，构建时结合话语标记远程监督与LLM及人工审核，实验显示操作特定微调可提升检索器性能，为科学灵感检索提供新框架。
中文摘要 AI 辅助
检索到的科学文献可为人与AI科学家提供灵感，灵感有不同形式：现有工作可直接建议如何解决问题，或呈现不同抽象层级的方向——放大到更通用视角或缩小到具体实现。我们提出RATIO（Retrieval Across Typed Ideation Operations），这是一个大规模基准，其相关性由三个名为构思动作（ideation moves）的操作定义：Address（针对所述问题检索潜在方法）、Broaden（检索更通用的表述）、Specify（检索具体实例）。RATIO通过通用流程从计算机科学领域数百万篇全文科学论文构建，该流程将仅用于分类的话语标记远程监督扩展到语料库规模检索，并结合大量大语言模型（LLM）与人工审核。实验表明，特定操作的微调可显著提升检索器性能，但仍有很大改进空间。RATIO为支持基于文献的构思的检索组件提供了可扩展的训练与评估框架，为科学灵感检索开辟了新的研究方向。
英文摘要
Retrieved scientific literature can serve as inspiration for both human and AI scientists. Inspiration can take different forms: prior work may directly suggest how to address a problem, or surface directions at different levels of abstraction - zooming out to a more general view or zooming in to a concrete realization. We introduce RATIO (Retrieval Across Typed Ideation Operations), a large-scale benchmark in which relevance is defined by three operations which we name ideation moves: Address retrieves potential approaches for stated problems, Broaden retrieves more general formulations, and Specify retrieves concrete instantiations. RATIO is constructed from millions of full-text scientific papers across CS literature via a general recipe that extends discourse-marker distant supervision - previously used only for classification - to corpus-scale retrieval, combined with extensive LLM and human vetting. Experiments show that operation-specific fine-tuning substantially boosts retrievers but leaves much room for further improvements. RATIO provides a scalable training and evaluation framework for retrieval components that support literature-grounded ideation, opening up new research avenues on scientific inspiration retrieval.
