ForeSci: 评估LLM智能体在前瞻性AI研究判断中的能力
ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment
AI总结：
提出ForeSci基准，通过时间控制的500个任务评估LLM智能体基于历史证据做出前瞻性研究判断的能力，发现证据与决策脱节问题。
摘要：
AI研究通常需要在未来证据出现之前做出决策：攻击哪个瓶颈、追求哪个方向、项目应如何定位。我们引入了ForeSci，一个时间控制的基准，用于评估LLM智能体是否能够从历史证据中做出此类前瞻性研究判断。ForeSci包含500个任务，涵盖四个快速发展的AI领域和四个决策家族。每个任务配有一个截止对齐的离线知识库；截止日期后的论文在生成过程中被隐藏，仅用于验证。为避免随机未来事件预测，任务源自截止前的分类分支和证据信号，并选择早于任务截止日期的答案生成骨干。我们评估了原生LLM、混合RAG以及四种骨干上的三种研究智能体适配。结果表明，显式证据组织提高了可追溯性和事实支持，但收益强烈依赖于决策家族。诊断揭示了一个反复出现的证据-决策脱节：智能体可能引用相关证据，但预测错误的研究对象。ForeSci将前瞻性AI研究判断转化为一个受控基准，用于评估作为决策系统的研究智能体。
英文摘要：
AI research often requires decisions before future evidence exists: which bottleneck to attack, which direction to pursue, or where a project should be positioned. We introduce ForeSci, a temporally controlled benchmark for evaluating whether LLM agents can make such forward-looking research judgements from historical evidence. ForeSci contains 500 tasks across four fast-moving AI domains and four decision families. Each task is paired with a cutoff-aligned offline knowledge base; post-cutoff papers are hidden during generation and used only for validation. To avoid random future-event prediction, tasks are derived from pre-cutoff taxonomy branches and evidence signals, while generation-time retrieval and tools are restricted to the cutoff-aligned evidence package. We evaluate native LLMs, Hybrid RAG, and three research-agent adaptations across four backbones. Agent-based methods improve traceability over Hybrid RAG, while their gains in future-target alignment over native LLMs are modest and task dependent. Diagnostics reveal a recurring evidence-decision decoupling: agents may cite relevant evidence while forecasting the wrong research object. Task-specific checks between evidence and final decision reduce drift and improve future-target alignment. As research agents move from assisting known tasks to shaping open research directions, ForeSci offers a first controlled step toward measuring whether their judgement can be trusted to guide that future.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
