回忆、风险与治理在研究资金自动筛选中的作用：来自国家资金计划的证据
Recall, Risk, and Governance in Automated Proposal Screening for Research Funding: Evidence from a National Funding Programme
AI总结：
本文研究了自动化筛选系统在研究资金申请中的表现，发现基于规则的方法在召回率和假阴性控制上优于基于大型语言模型的方法。
摘要：
研究资金机构正越来越多地探索自动化工具来支持早期提案筛选。近年来，大型语言模型（LLMs）的进展引发了对其用于文本评估的乐观态度，但其在高风险筛选决策中的机构适用性仍缺乏深入探讨。特别是，关于自动化筛选系统在评估机构错误成本时的表现，缺乏实证证据。 本研究比较了两种自动化方法用于提案筛选，以符合国家资金申请的优先事项：一种透明的基于规则的方法，使用术语频率-逆文档频率（TF-IDF）与领域特定关键词工程；另一种基于大型语言模型的语义分类方法。利用选拔委员会决策作为959个提案的真实数据，我们评估了性能，特别关注错误结构。结果表明，基于TF-IDF的方法在标准指标上优于基于LLM的系统，实现了显著更高的召回率（78.95% vs 45.82%）并产生更少的假阴性（68 vs 175）。基于LLM的系统排除了超过一半最终被委员会选择的提案。虽然假阳性可以通过后续同行评审进行纠正，但假阴性代表了无法恢复的专家评估排除。 通过突出错误不对称性和机构背景，本研究证明了自动化筛选系统的适用性不仅取决于模型的复杂性，还取决于其错误特征、透明度和可审计性是否与研究评估实践相一致。这些发现表明，评估设计和错误容忍度应指导研究资金中AI辅助筛选工具的更广泛使用。
英文摘要：
Research funding agencies are increasingly exploring automated tools to support early-stage proposal screening. Recent advances in large language models (LLMs) have generated optimism regarding their use for text-based evaluation, yet their institutional suitability for high-stakes screening decisions remains underexplored. In particular, there is limited empirical evidence on how automated screening systems perform when evaluated against institutional error costs. This study compares two automated approaches for proposal screening against the priorities of a national funding call: A transparent, rule-based method using term frequency-inverse document frequency (TF-IDF) with domain-specific keyword engineering, and a semantic classification approach based on a large language model. Using selection committee decisions as ground truth for 959 proposals, we evaluate performance with particular attention to error structure. The results show that the TF-IDF-based approach outperforms the LLM-based system across standard metrics, achieving substantially higher recall (78.95\% vs 45.82\%) and producing far fewer false negatives (68 vs 175). The LLM-based system excludes more than half of the proposals ultimately selected by the committee. While false positives can be corrected through subsequent peer review, false negatives represent an irrecoverable exclusion from expert evaluation. By foregrounding error asymmetry and institutional context, this study demonstrates that the suitability of automated screening systems depends not on model sophistication alone, but on how their error profiles, transparency, and auditability align with research evaluation practice. These findings suggest that evaluation design and error tolerance should guide the use of AI-assisted screening tools in research funding more broadly.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
