Towards Knowledgeable Deep Research: Framework and Benchmark
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所人工智能安全国家重点实验室);National University of Singapore(新加坡国立大学);University of Chinese Academy of Sciences(中国科学院大学)
Deep Research (DR) requires LLM agents to autonomously perform multi-step information seeking, processing, and reasoning to generate comprehensive reports. In contrast to existing studies that mainly focus on unstructured web content, a more challenging DR task should additionally utilize structured knowledge to provide a solid data foundation, facilitate quantitative computation, and lead to in-depth analyses. In this paper, we refer to this novel task as Knowledgeable Deep Research (KDR), which requires DR agents to generate reports with both structured and unstructured knowledge. Furthermore, we propose the Hybrid Knowledge Analysis framework (HKA), a multi-agent architecture that reasons over both kinds of knowledge and integrates the texts, figures, and tables into coherent multimodal reports. The key design is the Structured Knowledge Analyzer, which utilizes both coding and vision-language models to produce figures, tables, and corresponding insights. To support systematic evaluation, we construct KDR-Bench, which covers 9 domains, includes 41 expert-level questions, and incorporates a large number of structured knowledge resources (e.g., 1,252 tables). We further annotate the main conclusions and key points for each question and propose three categories of evaluation metrics including general-purpose, knowledge-centric, and vision-enhanced ones. Experimental results demonstrate that HKA consistently outperforms most existing DR agents on general-purpose and knowledge-centric metrics, and even surpasses the Gemini DR agent on vision-enhanced metrics, highlighting its effectiveness in deep, structure-aware knowledge analysis. Finally, we hope this work can serve as a new foundation for structured knowledge analysis in DR agents and facilitate future multimodal DR studies.
TCS-BENCH: Benchmarking State-of-the-Art Generative AI Theoretical Computer Science Research Ability
Google Research(谷歌研究院);CNRS(法国国家科学研究中心);IRIF(法国信息学研究所);Université Paris-Cité(巴黎城市大学);Princeton University(普林斯顿大学);Université Paris-Saclay(巴黎萨克雷大学);CEA(法国原子能和替代能源委员会);California Institute of Technology(加州理工学院);Google DeepMind(谷歌DeepMind)
PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research
School of Artificial Intelligence(人工智能学院);Zhiyuan College(智远学院);School of Physics and Astronomy(物理天文学院);Tsung-Dao Lee Institute(李政道研究所);State Key Laboratory of Dark Matter Physics(暗物质物理国家重点实验室);Shanghai Innovation Institute(上海创新研究院);Institute of Theoretical Physics(理论物理研究所);Chinese Academy of Sciences(中国科学院);Zhejiang Lab(浙江实验室);SciLand;DP Technology(DP技术)
MLGym: A New Framework and Benchmark for Advancing AI Research Agents
University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);University College London(伦敦大学学院);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);University of Oxford(牛津大学);PyTorch Core Libraries at Meta(元公司PyTorch核心库);FAIR at Meta(元公司FAIR研究院);GenAI at Meta(元公司GenAI部门)
IDRBench: Benchmarking the Interactive Capabilities of Deep Research Agents
School of Computing, National University of Singapore(新加坡国立大学计算机学院);School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)智能科学与工程学院);Zhejiang University(浙江大学);State Key Lab of CAD&CG, Zhejiang University(浙江大学计算机辅助设计与图形学国家重点实验室)
MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research
Stanford University(斯坦福大学);Tsinghua University(清华大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Princeton University(普林斯顿大学);KTH Royal Institute of Technology(瑞典皇家理工学院);Chan Zuckerberg Biohub Network(陈·扎克翰生物中心网络)
Paper2Agent: Reimagining Research Papers As Interactive and Reliable AI Agents
Department of Genetics, Stanford University(遗传学系,斯坦福大学);Department of Biomedical Data Science, Stanford University(生物医学数据科学系,斯坦福大学);Department of Electrical Engineering, Stanford University(电气工程系,斯坦福大学);Department of Biology, Stanford University(生物学系,斯坦福大学);Department of Computer Science, Stanford University(计算机科学系,斯坦福大学)
ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution
Institute of Trustworthy Embodied AI(可信具身AI研究院);Shanghai Innovation Institution(上海创新机构);Shanghai Key Laboratory of Multimodal Embodied AI(上海多模态具身AI重点实验室);College of Computer Science and Artificial Intelligence(计算机科学与人工智能学院);OpenMOSS;Wuhan University(武汉大学);Nanjing University(南京大学);Jilin University(吉林大学)
QMBench: A Research Level Benchmark for Quantum Materials Research
Department of Materials Science and Engineering, Stanford University(斯坦福大学材料科学与工程系);Department of Physics, Pennsylvania State University(宾夕法尼亚州立大学物理系);Department of Condensed Matter Physics, Weizmann Institute of Science(魏茨曼科学研究院凝聚态物理系);Path Integral Technology, Inc.(路径积分技术公司)
OpenHoldem: A Benchmark for Large-Scale Imperfect-Information Game Research
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
AI Idea Bench 2025: AI Research Idea Generation Benchmark
Wuhan University(武汉大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Innovation Institute(上海创新研究院);Harbin Institute of Technology(哈尔滨工业大学)
本文提出AI Idea Bench 2025,一个用于定量评估和比较LLMs在AI研究领域生成思路的框架,包含3,495篇论文的数据集和双维度评估方法,以促进科学发现自动化。