大型语言模型能否评估项目申请质量？重新审视文纳斯和沃尔德同行评审数据
Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data
AI总结：
本文评估了大型语言模型在评分项目申请方面的准确性，发现其与专家评分的相关性较低，但可能在筛选或打破平局时有应用价值。
摘要：
目的：尽管同行评审对资助决策至关重要，但学者们往往不愿进行评审，导致提交与最终决定之间存在较长的延迟，以及非专业学者可能产生低质量评审的风险。至少有一个资助机构现在使用大型语言模型（LLMs）来减轻评审负担，但需要评估LLMs在评分项目申请方面的准确性。设计/方法/研究方法：本文比较了多种中等规模的开源LLMs的评分与同行评审评分，使用了一个经过深入研究的数据集，即瑞典医学委员会1994年的博士后奖学金申请。发现：尽管LLM评分之间相关性中等（平均Spearman相关性：0.34），但它们与专家评分的相关性较弱但正相关且大部分统计显著（平均Spearman相关性：0.22）。专家评分与LLM评分之间的最高等级相关性为0.33，基于申请标题和摘要，不包括主要内容，使用Gemma 3 27b模型。研究限制：小样本量、旧的资助申请和异质的评估标准都削弱了分析的稳健性。实际意义：尽管LLM在评分项目申请方面的定量能力弱于专家，但至少在这一特殊情况下，它们可能在筛选或打破平局时发挥作用。原创性/价值：这是首次评估LLM评分在资助申请中的价值。
英文摘要：
Purpose: Despite the importance of peer review for grant funding decisions, academics are often reluctant to conduct it. This can lead to long delays between submission and the final decision as well as the risk of substandard reviews from busy or non-specialist scholars. At least one funder now uses Large Language Models (LLMs) to reduce the reviewing burden but the accuracy of LLMs for scoring grant proposals needs to be assessed. Design/methodology/approach: This article compares scores from a range of medium sized open weights LLMs with peer review scores for a well-researched dataset, the Swedish Medical Council's post-doctoral fellowship applications from 1994. Findings: Whilst the LLM scores correlate moderately between each other (mean Spearman correlation: 0.34), they correlated weakly but positively and mostly statistically significantly with the average expert scores (mean Spearman correlation: 0.22). The highest rank correlation between expert scores and LLMs was 0.33 for Gemma 3 27b based on proposal titles and summaries without their main texts, which is about half (56%) of the correlation between reviewers. Research limitations: The small sample size, old funding call and heterogeneous evaluation criteria all undermine the robustness of the analysis. Practical implications: Despite the ability of LLMs to score grant proposals being quantitatively weaker than that of experts, at least in this special case, they may have role in application triage or tie-breaking. Originality/value: This is the first assessment of the value of LLM scores for funding proposals.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
