元认知监控电池：一个跨领域用于LLM自我监控的基准测试
The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring
AI总结：
本文提出一个跨领域评估LLM元认知监控的基准测试，结合人类心理测量方法，通过524项任务测试学习、元认知校准等六个认知领域，揭示LLM在自信度和自我监控上的差异。
摘要：
我们介绍了一个跨领域的元认知监控-控制耦合行为测试，基于Nelson和Narens（1990）的元认知框架，并应用人类心理测量方法评估LLM。该电池包含524项任务，覆盖六个认知领域（学习、元认知校准、社会认知、注意力、执行功能、前瞻性调节），每个领域均基于已建立的实验范式。任务T1-T5在数据收集前已在OSF预注册；T6作为探索性扩展被添加。在每次强制选择响应后，双探针（改编自Koriat和Goldsmith，1996）要求模型保持或撤销其答案，并进行投注或拒绝。关键指标是撤回差：错误与正确项目之间的撤回率差异。应用于20个前沿LLM（共10,480次评估），该电池区分出三种与Nelson-Narens架构一致的配置：普遍自信、普遍撤回和选择性敏感。准确性排名和元认知敏感性排名大致相反。回顾性监控与前瞻性调节似乎分离（r = .17，95%置信区间宽，n=20；基于实例的证据是主要支持）。在元认知校准上的扩展取决于架构：单调递减（Qwen）、单调递增（GPT-5.4）或平坦（Gemma）。行为发现与独立的类型-2 SDT方法在结构上一致，提供初步的跨方法构念有效性。所有项目、数据和代码：https://github.com/synthiumjp/metacognitive-monitoring-battery.
英文摘要：
We introduce a cross-domain behavioural assay of monitoring-control coupling in LLMs, grounded in the Nelson and Narens (1990) metacognitive framework and applying human psychometric methodology to LLM evaluation. The battery comprises 524 items across six cognitive domains (learning, metacognitive calibration, social cognition, attention, executive function, prospective regulation), each grounded in an established experimental paradigm. Tasks T1-T5 were pre-registered on OSF prior to data collection; T6 was added as an exploratory extension. After every forced-choice response, dual probes adapted from Koriat and Goldsmith (1996) ask the model to KEEP or WITHDRAW its answer and to BET or decline. The critical metric is the withdraw delta: the difference in withdrawal rate between incorrect and correct items. Applied to 20 frontier LLMs (10,480 evaluations), the battery discriminates three profiles consistent with the Nelson-Narens architecture: blanket confidence, blanket withdrawal, and selective sensitivity. Accuracy rank and metacognitive sensitivity rank are largely inverted. Retrospective monitoring and prospective regulation appear dissociable (r = .17, 95% CI wide given n=20; exemplar-based evidence is the primary support). Scaling on metacognitive calibration is architecture-dependent: monotonically decreasing (Qwen), monotonically increasing (GPT-5.4), or flat (Gemma). Behavioural findings converge structurally with an independent Type-2 SDT approach, providing preliminary cross-method construct validity. All items, data, and code: https://github.com/synthiumjp/metacognitive-monitoring-battery.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
