E3: 面向自动化研究评论的问题级回测
E3: Issue-Level Backtesting for Automated Research Critique
AI总结：
提出E3自动化评论助手，通过问题级回测协议评估其在识别研究论文技术问题上的表现，相比人类评审和LLM基线实现最高召回率。
摘要：
我们提出E3，一个自动化评论助手，通过识别研究论文中与决策相关的技术问题来增强评审者和工程团队。对于每个问题，E3报告其性质、位置、对贡献的影响以及解决该问题所需的分析或证据，涵盖无根据的主张、缺失的消融实验、弱基线、隐藏假设、有效性威胁和数据泄露风险。为了在没有污染混杂因素的情况下评估E3，我们采用问题级回测协议：语料库仅限于每个自动化来源训练截止日期之后发表的论文，并且对于每篇论文，一个仅观察匿名评审的元裁判将每个问题-来源对标记为“捕获”、“部分”或“遗漏”。应用于100篇ICLR 2026论文和4598个被评判的问题行，将E3与ICLR人类评审以及基于OpenAI的gpt-5.4和Anthropic的claude-opus-4-6构建的两个提示匹配的LLM基线进行比较，使用元裁判gpt-5.5，E3在每个聚合指标上达到最高召回率。包含部分的召回率达到90.2%，比GPT高15.5个百分点，比Claude高17.1个百分点，比人类评审高29.2个百分点，严格召回率保持顺序为65.8%。在人类评审提出的问题上，E3恢复了89.6%；在人类评审遗漏的问题上，它额外发现了1635行被纳入评判联合集，比次优来源多406行。语料库、基线提示、裁判提示模板和评估代码已发布。
英文摘要：
We present E3, an automated review assistant that augments reviewers and engineering teams by identifying decision-relevant technical concerns in research papers. For each concern, E3 reports its nature, its location, its bearing on the contribution, and the analysis or evidence that would resolve it, covering unsupported claims, missing ablations, weak baselines, hidden assumptions, threats to validity, and leakage risks. To evaluate E3 without contamination confounds we adopt an issue-level backtesting protocol: the corpus is restricted to papers postdating the training cutoff of every automated source, and for each paper a meta-judge that observes only anonymised reviews labels every issue-source pair as Caught, Partial, or Missed. Applied to 100 ICLR 2026 papers and 4598 judged issue rows, comparing E3 against the ICLR human reviews and two prompt-matched LLM baselines built on gpt-5.4 from OpenAI and claude-opus-4-6 from Anthropic, with meta-judge gpt-5.5, E3 attains the highest recall on every aggregate metric. Partial-inclusive recall reaches 90.2 percent, which is 15.5 points over GPT, 17.1 points over Claude, and 29.2 points over the human reviews, and strict recall preserves the ordering at 65.8 percent. On concerns raised by the human reviewers, E3 recovers 89.6 percent; on concerns the human reviewers missed it surfaces 1635 additional rows admitted into the judged union, 406 above the next-best source. Corpus, baseline prompts, judge prompt template, and evaluation code are released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
