通过结构化扰动评估基于LLM的项目申请评审
Evaluating LLM-Based Grant Proposal Review via Structured Perturbations
AI总结:
本文通过结构化扰动评估基于LLM的项目评审,发现逐部分分析方法在检测和评分可靠性上表现最佳,但LLM反馈存在偏颇,需进一步优化。
摘要:
随着AI辅助的项目申请在研究生态系统中超越了人工评审能力,形成一种“马尔萨斯陷阱”,本文研究了基于LLM的项目评审在高风险评估中的能力和局限性。使用六个EPSRC申请,我们开发了一种基于扰动的框架,探测LLM在六个质量轴上的敏感性:资金、时间线、能力、一致性、清晰度和影响。我们比较了三种评审架构:单次评审、逐部分分析,以及模拟专家小组的“人物委员会”集成方法。逐部分方法在检测率和评分可靠性方面显著优于其他方法,而计算成本较高的委员会方法表现不如下落基。检测在扰动类型上变化显著,一致性问题容易被识别,但清晰度问题则被所有系统所遗漏。人类评估显示LLM反馈总体上是有效的,但偏向于合规检查而非全面评估。我们得出结论,当前LLM可能在EPSRC评审中提供补充价值,但存在高变异性和不一致的评审优先级。我们释放我们的代码和任何非受保护的数据。
英文摘要:
As AI-assisted grant proposals outpace manual review capacity in a kind of ``Malthusian trap'' for the research ecosystem, this paper investigates the capabilities and limitations of LLM-based grant reviewing for high-stakes evaluation. Using six EPSRC proposals, we develop a perturbation-based framework probing LLM sensitivity across six quality axes: funding, timeline, competency, alignment, clarity, and impact. We compare three review architectures: single-pass review, section-by-section analysis, and a 'Council of Personas' ensemble emulating expert panels. The section-level approach significantly outperforms alternatives in both detection rate and scoring reliability, while the computationally expensive council method performs no better than baseline. Detection varies substantially by perturbation type, with alignment issues readily identified but clarity flaws largely missed by all systems. Human evaluation shows LLM feedback is largely valid but skewed toward compliance checking over holistic assessment. We conclude that current LLMs may provide supplementary value within EPSRC review but exhibit high variability and misaligned review priorities. We release our code and any non-protected data.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。