The rapid evolution of Large Language Models has catalyzed a surge in scientific idea production, yet this leap has not been accompanied by a matching advance in idea evaluation. The fundamental nature of scientific evaluation needs knowledgeable grounding, collective deliberation, and multi-criteria decision-making. However, existing idea evaluation methods often suffer from narrow knowledge horizons, flattened evaluation dimensions, and the inherent bias in LLM-as-a-Judge. To address these, we regard idea evaluation as a knowledge-grounded, multi-perspective reasoning problem and introduce InnoEval, a deep innovation evaluation framework designed to emulate human-level idea assessment. We apply a heterogeneous deep knowledge search engine that retrieves and grounds dynamic evidence from diverse online sources. We further achieve review consensus with an innovation review board containing reviewers with distinct academic backgrounds, enabling a multi-dimensional decoupled evaluation across multiple metrics. We construct comprehensive datasets derived from authoritative peer-reviewed submissions to benchmark InnoEval. Experiments demonstrate that InnoEval can consistently outperform baselines in point-wise, pair-wise, and group-wise evaluation tasks, exhibiting judgment patterns and consensus highly aligned with human experts.
ScholarEval: Research Idea Evaluation Grounded in Literature
Department of Computer Science and Engineering, The Ohio State University（俄亥俄州立大学计算机科学与工程系）；Division of Medicinal Chemistry and Pharmacognosy, The Ohio State University（俄亥俄州立大学药物化学与药理学系）；Department of Wildlife, Fisheries, and Conservation Biology, The University of Maine（缅因大学野生动物、渔业与保育生物学系）；McGovern Institute for Brain Research, Massachusetts Institute of Technology（麻省理工学院麦格尼恩脑研究 institute）；Center for Cognitive and Behavioral Brain Imaging, The Ohio State University（俄亥俄州立大学认知与行为脑成像中心）；Department of Chemistry, University of Wisconsin-Madison（威斯康星大学麦迪逊分校化学系）；Allen Institute for Artificial Intelligence（人工智能研究院）
AI Idea Bench 2025: AI Research Idea Generation Benchmark
Wuhan University（武汉大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Innovation Institute（上海创新研究院）；Harbin Institute of Technology（哈尔滨工业大学）
本文提出AI Idea Bench 2025，一个用于定量评估和比较LLMs在AI研究领域生成思路的框架，包含3,495篇论文的数据集和双维度评估方法，以促进科学发现自动化。
University of Washington（华盛顿大学）；Microsoft Research（微软研究院）
Although neural conversation models are effective in learning how to produce fluent responses, their primary challenge lies in knowing what to say to make the conversation contentful and non-vacuous. We present a new end-to-end approach to contentful neural conversation that jointly models response generation and on-demand machine reading. The key idea is to provide the conversation model with relevant long-form text on the fly as a source of external knowledge. The model performs QA-style reading comprehension on this text in response to each conversational turn, thereby allowing for more foc…
IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation
Carnegie Mellon University（卡内基梅隆大学）；Meta；Meta Reality Labs；FAIR at Meta（Meta FAIR）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Toronto（多伦多大学）；Princeton University（普林斯顿大学）；The University of Tokyo（东京大学）；RIKEN AIP（日本理化学研究所人工智能中心）
IdeaScientist 通过强化学习训练三个角色（发现空白、创新、报告撰写），并利用 Svalbard Idea Vault 语料库，在时间控制评估中显著超越开源基线，甚至超过闭源模型，实现有依据的科学构思。
Conversational search is an approach to information retrieval (IR), where users engage in a dialogue with an agent in order to satisfy their information needs. Previous conceptual work described properties and actions a good agent should exhibit. Unlike them, we present a novel conceptual model defined in terms of conversational goals, which enables us to reason about current research practices in conversational search. Based on the literature, we elicit how existing tasks and test collections from the fields of IR, natural language processing (NLP) and dialogue systems (DS) fit into this mode…
SciMON: Scientific Inspiration Machines Optimized for Novelty
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Allen Institute for Artificial Intelligence (AI2)（艾伦人工智能研究所）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）
Modern Artificial Intelligence (AI) systems excel at diverse tasks, from image classification to strategy games, even outperforming humans in many of these domains. After making astounding progress in language learning in the recent decade, AI systems, however, seem to approach the ceiling that does not reflect important aspects of human communicative capacities. Unlike human learners, communicative AI systems often fail to systematically generalize to new data, suffer from sample inefficiency, fail to capture common-sense semantic knowledge, and do not translate to real-world communicative si…
Rochester Institute of Technology（罗切斯特理工学院）；PAIGE；Cornell Tech（康奈尔理工学院）
Language grounded image understanding tasks have often been proposed as a method for evaluating progress in artificial intelligence. Ideally, these tasks should test a plethora of capabilities that integrate computer vision, reasoning, and natural language understanding. However, rather than behaving as visual Turing tests, recent studies have demonstrated state-of-the-art systems are achieving good performance through flaws in datasets and evaluation procedures. We review the current state of affairs and outline a path forward.
Personalized Auto-Research: Towards a True AI Co-Scientist
Vanderbilt University（范德堡大学）；Adobe Research（奥多比研究院）；Dolby Laboratories（杜比实验室）；University of Georgia（佐治亚大学）；Cisco AI Research（思科人工智能研究院）；Texas A&M University（德克萨斯农工大学）
