NewtonBench: 在LLM代理中评估可推广的科学定律发现
NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents
AI总结：
NewtonBench通过反事实定律变化生成科学定律发现任务，评估LLM代理在复杂系统中发现隐藏原理的能力，并揭示其在系统复杂性和观测噪声下的脆弱性。
摘要：
大型语言模型正逐渐成为科学定律发现的强大工具，这是人工智能驱动科学中的基础挑战。然而，现有此类任务的基准测试存在根本性的方法论三难困境，迫使在科学相关性、可扩展性和抗记忆性之间做出权衡。此外，它们将发现简化为静态函数拟合，未能捕捉到通过复杂模型系统的交互探索来揭示隐藏定律的真实科学过程。为解决这些关键缺口，我们引入了NewtonBench，一个包含324个科学定律发现任务的基准测试，涵盖12个物理领域。我们的设计通过使用反事实定律变化——系统性地改变经典定律——来生成大量问题，这些问题是可扩展的、科学上相关的，并且抗记忆的。此外，我们将评估从静态函数拟合提升到交互式模型发现，要求代理实验性地探测模拟的复杂系统以揭示隐藏原则。我们的广泛实验揭示了在前沿LLM中发现能力的明显但脆弱的特征：这种能力随着系统复杂性的增加而迅速下降，并且对观测噪声极其敏感。值得注意的是，我们发现工具辅助的悖论效应：提供代码解释器可能会阻碍更强大模型，通过诱导从探索转向利用，使它们在次优解决方案上满意。这些结果表明，在复杂、交互式环境中稳健、可推广的发现仍然是核心挑战。通过提供可扩展、稳健且科学上真实的测试环境，NewtonBench为衡量真正的进步并指导下一代能够真正进行科学发现的AI代理的发展提供了关键工具。
英文摘要：
Large language models are emerging as powerful tools for scientific law discovery, a foundational challenge in AI-driven science. However, existing benchmarks for this task suffer from a fundamental methodological trilemma, forcing a trade-off between scientific relevance, scalability, and resistance to memorization. Furthermore, they oversimplify discovery as static function fitting, failing to capture the authentic scientific process of uncovering embedded laws through the interactive exploration of complex model systems. To address these critical gaps, we introduce NewtonBench, a benchmark comprising 324 scientific law discovery tasks across 12 physics domains. Our design mitigates the evaluation trilemma by using counterfactual law shifts - systematic alterations of canonical laws - to generate a vast suite of problems that are scalable, scientifically relevant, and memorization-resistant. Moreover, we elevate the evaluation from static function fitting to interactive model discovery, requiring agents to experimentally probe simulated complex systems to uncover hidden principles. Our extensive experiment reveals a clear but fragile capability for discovery in frontier LLMs: this ability degrades precipitously with increasing system complexity and exhibits extreme sensitivity to observational noise. Notably, we uncover a paradoxical effect of tool assistance: providing a code interpreter can hinder more capable models by inducing a premature shift from exploration to exploitation, causing them to satisfice on suboptimal solutions. These results demonstrate that robust, generalizable discovery in complex, interactive environments remains the core challenge. By providing a scalable, robust, and scientifically authentic testbed, NewtonBench offers a crucial tool for measuring true progress and guiding the development of next-generation AI agents capable of genuine scientific discovery.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
