GIScholarBench: 在GIS研究中评估大语言模型的过度自信
GIScholarBench: Benchmarking LLM Overconfidence in GIS Research
AI总结：
针对大语言模型在学术研究中的过度自信问题，构建了包含10865篇论文的GIScholarBench基准，通过元数据检索、文献链接和研究方向生成三项任务评估模型表现，发现所有模型均存在任务不变的过度自信现象。
摘要：
大型语言模型（LLMs）越来越多地用于学术研究工作流程，但学术任务需要高事实精度，因此暴露了一个关键弱点：过度自信。这里，过度自信被行为定义为即使在底层知识不完整或不可验证时，也倾向于产生自信、果断且格式良好的输出，而不是陈述信心与准确性之间的校准差距。为了研究这一问题，我们引入了GIScholarBench，这是一个基于2020年至2025年间发表在25个核心GIScience期刊上的10865篇论文构建的基准。该基准涵盖三个认知复杂度递增的任务：元数据检索、文献链接和研究方向生成。我们通过原生网络界面在真实用户条件下评估了Claude Sonnet 4.5、Gemini 3和ChatGPT 5.3。结果显示所有任务均存在一致的过度自信。在元数据检索中，ChatGPT 5.3取得了最高准确率，但所有模型在预测错误时仍生成确定的标题和DOI。在文献链接中，Claude Sonnet 4.5恢复了最多的参考文献，但所有模型在排名靠前的检索和更长的引文列表之间显示出明显差距，表明参考文献被扩展到可靠检索能力之外。在研究方向生成中，AI生成的方向相比真实未来引用论文显示出更低的主题覆盖率、更高的新颖性缺失率和更低的语义多样性。这些发现表明，LLM的过度自信是任务不变的，但表现形式不同：检索中的事实过度生成、文献链接中不可靠的引文扩展，以及研究构思中输出完整性的过度自信。
英文摘要：
Large language models (LLMs) are increasingly used in academic research workflows, but scholarly tasks require high factual precision and therefore expose a key weakness: overconfidence. Here, overconfidence is defined behaviorally as the tendency to produce confident, assertive, and well-formatted outputs even when the underlying knowledge is incomplete or unverifiable, rather than as a calibration gap between stated confidence and accuracy. To examine this issue, we introduce GIScholarBench, a benchmark built from 10,865 papers published in 25 core GIScience journals between 2020 and 2025. The benchmark covers three tasks with increasing cognitive complexity: metadata retrieval, literature linking, and research direction generation. We evaluate Claude Sonnet 4.5, Gemini 3, and ChatGPT 5.3 through their native web interfaces under real-world user-facing conditions. Results show consistent overconfidence across all tasks. In metadata retrieval, ChatGPT 5.3 achieves the highest accuracy, but all models still generate definitive titles and DOIs when predictions are wrong. In literature linking, Claude Sonnet 4.5 recovers the most references, but all models show a clear gap between top-ranked retrieval and longer citation lists, suggesting that references are extended beyond reliable retrieval capacity. In research direction generation, AI-generated directions show lower topic coverage, higher novel miss rates, and lower semantic diversity than real future-citing papers. These findings suggest that LLM overconfidence is task-invariant but takes different forms: factual overgeneration in retrieval, unreliable citation expansion in literature linking, and overconfidence in output completeness during research ideation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
