arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.07649cs.IR
收藏

LitXBench: 一个用于从科学文献中提取实验的基准测试

LitXBench: A Benchmark for Extracting Experiments from Scientific Literature

Radical AI

Curtis Chong, Jorge Colindres

AI总结:

本文提出LitXBench基准测试,用于评估从科学文献中提取实验的方法,通过引入LitXAlloy密集基准,包含1426个测量数据,展示了前沿语言模型在提取实验方面的优越性能。

摘要:

将论文中的实验数据汇总使材料科学家能够构建更准确的性质预测模型并促进科学发现。最近,兴趣逐渐转向不仅提取单个材料属性,还提取整个实验测量。为了支持这一转变,我们介绍了LitXBench,一个用于评估从文献中提取实验方法的框架。我们还提出了LitXAlloy,一个包含1426个总测量数据来自19篇合金论文的密集基准。通过将基准的条目存储为Python对象,而不是基于文本的格式如CSV或JSON,我们提高了可审计性并启用了程序化数据验证。我们发现,前沿语言模型,如Gemini 3.1 Pro Preview,比现有的多轮提取管道高出多达0.37 F1。我们的结果表明,这种性能差距出现是因为提取管道将测量与组成而不是定义材料的加工步骤相关联。

英文摘要:

Aggregating experimental data from papers enables materials scientists to build better property prediction models and to facilitate scientific discovery. Recently, interest has grown in extracting not only single material properties but also entire experimental measurements. To support this shift, we introduce LitXBench, a framework for benchmarking methods that extract experiments from literature. We also present LitXAlloy, a dense benchmark comprising 1426 total measurements from 19 alloy papers. By storing the benchmark's entries as Python objects, rather than text-based formats such as CSV or JSON, we improve auditability and enable programmatic data validation. We find that frontier language models, such as Gemini 3.1 Pro Preview, outperform existing multi-turn extraction pipelines by up to 0.37 F1. Our results suggest that this performance gap arises because extraction pipelines associate measurements with compositions rather than the processing steps that define a material.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑