LazyReview：一个用于揭示NLP同行评审中惰性思维的数据集
LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews
AI总结：
本文提出LazyReview数据集，标注NLP同行评审中的细粒度惰性思维，发现LLM零样本检测困难，但经指令微调后性能提升10-20点，并证明基于惰性思维反馈修订的评审更全面可操作。
摘要：
同行评审是科学出版中质量控制的基石。随着工作量的不断增加，无意中使用“快速”启发式方法（即所谓的惰性思维）已成为一个反复出现的问题，损害了评审质量。检测此类启发式方法的自动化手段有助于改进同行评审流程。然而，目前关于这一问题的NLP研究十分有限，也不存在可用于支持检测工具开发的真实世界数据集。本工作引入了LazyReview，这是一个同行评审句子的数据集，标注了细粒度的惰性思维类别。我们的分析表明，大型语言模型（LLMs）在零样本设置下难以检测这些实例。然而，基于指令的微调在我们的数据集上显著将性能提升了10-20个性能点，凸显了高质量训练数据的重要性。此外，一项对照实验表明，根据惰性思维反馈修订后的评审比没有此类反馈的评审更全面、更具可操作性。我们将发布我们的数据集以及增强后的指南，这些指南可用于培训社区中的初级审稿人。（代码见：https://github.com/UKPLab/acl2025-lazy-review）
英文摘要：
Peer review is a cornerstone of quality control in scientific publishing. With the increasing workload, the unintended use of `quick' heuristics, referred to as lazy thinking, has emerged as a recurring issue compromising review quality. Automated methods to detect such heuristics can help improve the peer-reviewing process. However, there is limited NLP research on this issue, and no real-world dataset exists to support the development of detection tools. This work introduces LazyReview, a dataset of peer-review sentences annotated with fine-grained lazy thinking categories. Our analysis reveals that Large Language Models (LLMs) struggle to detect these instances in a zero-shot setting. However, instruction-based fine-tuning on our dataset significantly boosts performance by 10-20 performance points, highlighting the importance of high-quality training data. Furthermore, a controlled experiment demonstrates that reviews revised with lazy thinking feedback are more comprehensive and actionable than those written without such feedback. We will release our dataset and the enhanced guidelines that can be used to train junior reviewers in the community. (Code available here: https://github.com/UKPLab/acl2025-lazy-review)
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
