DeepInstructor：一种用于经验驱动型想法评估的智能体式AI导师

中文摘要 AI 辅助

随着自动化科学发现的推进，大型语言模型（LLMs）如今能够以前所未有的规模生成研究想法，这使得瓶颈从想法生成转向了想法评估。现有的评估器主要依赖参数化LLM知识或非结构化检索，产生的判断缺乏人类导师所使用的基于经验的推理。为解决这一问题，我们提出了DeepInstructor，一个将想法评估构建为对结构化学术经验进行推理的智能体框架。DeepInstructor从58,607条同行评审中构建了一个经验图（Experience Graph），并采用基于ReAct的智能体来检索针对特定维度的证据，以实现可追溯的评估。我们进一步引入了DeepInstruct，一个包含新颖性、重要性和可行性方面受控成对比较的数据集。实验表明，DeepInstructor显著优于现有基线，在与人类判断的一致性上，Hit@1和Hit@2分别提高了24.4%和29.7%。我们的研究结果表明，科学想法评估可以基于对结构化学术经验的显式推理。

英文摘要

As automated scientific discovery advances, Large Language Models (LLMs) can now generate research ideas at an unprecedented scale, shifting the bottleneck from idea generation to idea evaluation. Existing evaluators mainly rely on parametric LLM knowledge or unstructured retrieval, producing judgments that lack the experience-grounded reasoning used by human instructors. To address this, we propose DeepInstructor, an agentic framework that formulates idea evaluation as reasoning over structured scholarly experience. DeepInstructor constructs an Experience Graph from 58,607 peer reviews and employs a ReAct-based agent to retrieve dimension-specific evidence for traceable evaluation. We further introduce DeepInstruct, a dataset with controlled pairwise comparisons across novelty, significance, and feasibility. Experiments show that DeepInstructor substantially outperforms existing baselines, improving Hit@1 and Hit@2 alignment with human judgments by 24.4% and 29.7%, respectively. Our findings suggest that scientific idea evaluation can be grounded in explicit reasoning over structured scholarly experience

发表机构

• Tongji University（同济大学）
• Westlake University（西湖大学）
• Zhejiang University（浙江大学）
• Shanghai AI Lab（上海人工智能实验室）
• Fudan University（复旦大学）

机构由 AI 辅助整理，请以论文原文为准。