IsoSci: 用于评估LLM中推理与知识检索的同构跨域科学问题基准
IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs
AI总结:
提出IsoSci基准,通过逻辑结构相同但领域知识不同的科学问题对,分离推理能力与领域知识检索,发现91.3%的推理增益依赖于知识而非结构,挑战了思维链推理提升科学问题解决的假设。
摘要:
我们引入了ISOSCI,一个由同构跨域科学问题对组成的基准,用于在LLM评估中分离推理能力与领域知识检索。每个问题对共享相同的逻辑结构,但需要不同的领域特定知识,从而能够控制推理模式增益的归因。跨越四个模型家族的五个模型对,我们发现91.3%的推理模式增益是知识依赖的而非结构不变的(63/69个增益;Wilson 95% CI [82.3%, 96.0%]),这直接挑战了思维链推理能改善短视程序性科学问题解决的假设。在高能力模型上启用推理,在所有领域提供的准确率提升不到5个百分点,而一个推理专用模型(o3-mini)在GPQA Diamond上优于其标准对应模型(+19.2个百分点),但在ISOSCI上表现更差(-24.7个百分点),表明基准选择决定了关于推理效用的结论。我们在https://this.url发布ISOSCI。
英文摘要:
We introduce ISOSCI, a benchmark of isomorphic cross-domain science problem pairs that separates reasoning ability from domain knowledge retrieval in LLM evaluation. Each pair shares identical logical structure but requires different domain-specific knowledge, enabling controlled attribution of reasoning-mode gains. Across five model pairs spanning four model families, we find that 91.3% of reasoning-mode gains are knowledge-dependent rather than structure-invariant (63/69 gains; Wilson 95% CI [82.3%, 96.0%]), directly challenging the assumption that chain-of-thought reasoning improves short-horizon procedural scientific problem-solving. Reasoning toggles on highly capable models provide less than 5 percentage points accuracy gain across all domains, and a reasoning-specialized model (o3-mini) that outperforms its standard counterpart on GPQA Diamond (+19.2 percentage points) underperforms on ISOSCI (-24.7 percentage points), showing that benchmark choice determines conclusions about reasoning utility. We release ISOSCI at https://huggingface.co/datasets/isosci/isosci
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。