Large Language Models (LLMs) have transformed how people interact with artificial intelligence (AI) systems, achieving state-of-the-art results in various tasks, including scientific discovery and hypothesis generation. However, the lack of a comprehensive and systematic evaluation framework for generating research ideas using LLMs poses a significant obstacle to understanding and assessing their generative capabilities in scientific discovery. To address this gap, we propose IdeaBench, a benchmark system that includes a comprehensive dataset and an evaluation framework for standardizing the assessment of research idea generation using LLMs. Our dataset comprises titles and abstracts from a diverse range of influential papers, along with their referenced works. To emulate the human process of generating research ideas, we profile LLMs as domain-specific researchers and ground them in the same context considered by human researchers. This maximizes the utilization of the LLMs' parametric knowledge to dynamically generate new research ideas. We also introduce an evaluation framework for assessing the quality of generated research ideas. Our evaluation framework is a two-stage process: first, using GPT-4o to rank ideas based on user-specified quality indicators such as novelty and feasibility, enabling scalable personalization; and second, calculating relative ranking based "Insight Score" to quantify the chosen quality indicator. The proposed benchmark system will be a valuable asset for the community to measure and compare different LLMs, ultimately advancing the automation of the scientific discovery process.
Rationality Check! Benchmarking the Rationality of Large Language Models
Department of Electronic Engineering, BNRist, Tsinghua University（电子工程系、BNRist、清华大学）；Department of Sociology, University of Chicago（社会学系、芝加哥大学）；Santa Fe Institute（圣菲研究所）
AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions
TMLR Group, Hong Kong Baptist University（香港 Baptist 大学 TMLR 团体）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Sydney AI Centre, The University of Sydney（悉尼大学 AI 中心）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；International Center of Future Science, Jilin University（吉林大学未来科学国际中心）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China（教育部知识驱动人机智能工程研究中心）
AI Idea Bench 2025: AI Research Idea Generation Benchmark
Wuhan University（武汉大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Innovation Institute（上海创新研究院）；Harbin Institute of Technology（哈尔滨工业大学）
本文提出AI Idea Bench 2025，一个用于定量评估和比较LLMs在AI研究领域生成思路的框架，包含3,495篇论文的数据集和双维度评估方法，以促进科学发现自动化。
Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarking
The Hebrew University of Jerusalem（耶路撒冷希伯来大学）；Allen Institute for AI（艾伦人工智能研究院）；Bar-Ilan University（巴伊兰大学）；Tel-Aviv University（特拉维夫大学）；Stanford University（斯坦福大学）
NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens
Westlake University（西湖大学）；University of Waterloo（多伦多大学）；Hangzhou Normal University（杭州师范大学）；Shanghai AI Lab（上海人工智能实验室）；SJTU（上海交通大学）；NYU Shanghai（纽约大学上海分校）
