3-9B开放权重指令微调LLM中的言语自信饱和：一项预注册的心理计量效度筛查
Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs: A Pre-Registered Psychometric Validity Screen
AI总结：
研究测试了七种指令微调的开放权重模型在最小数值提示下产生言语化自信是否满足最低效度标准，发现所有模型在数值自信上无效，且类别提示未能恢复效度。
摘要：
言语自信提取被广泛用于从LLM中获取不确定性估计。我们测试了七种指令微调的开放权重模型（3-9B参数，四家族）在最小数值提示下是否产生符合最低效度标准的言语化自信。在一项预注册研究（OSF: osf.io/azbvx）中，524个TriviaQA问题在数值（0-100）和类别（10类）提示下，于消费者硬件上对八种模型进行Q5_K_M量化，产生8,384次确定性试验。每个模型-格式单元应用了心理计量效度筛查。所有七种指令模型在数值自信上均被分类为无效（H2确认，7/7 vs. 预测>=4/7），平均天花板率91.7%（H1确认）。类别提示未能恢复效度，反而在六种模型中破坏任务表现，产生低于5%的准确性（H4未确认）。令牌级对数概率未在观察到的方差范围内有效预测言语化自信（H5确认，均值交叉验证R² < 0.01）。在推理蒸馏模型中，推理轨迹长度与自信呈强负部分相关（rho = -0.36，p < .001），与推理污染效应一致。这些结果不意味着内部不确定性表示不存在。它们表明，在此模型规模下，最小言语提示未能在输出接口保留内部信号。心理计量筛查应在任何下游使用这些信号之前进行。
英文摘要：
Verbal confidence elicitation is widely used to extract uncertainty estimates from LLMs. We tested whether seven instruction-tuned open-weight models (3-9B parameters, four families) produce verbalised confidence that meets minimal validity criteria for item-level Type-2 discrimination under minimal numeric elicitation with greedy decoding. In a pre-registered study (OSF: osf.io/azbvx), 524 TriviaQA items were administered under numeric (0-100) and categorical (10-class) elicitation to eight models at Q5_K_M quantisation on consumer hardware, yielding 8,384 deterministic trials. A psychometric validity screen was applied to each model-format cell. All seven instruct models were classified Invalid on numeric confidence (H2 confirmed, 7/7 vs. predicted >=4/7), with a mean ceiling rate of 91.7% (H1 confirmed). Categorical elicitation did not rescue validity. Instead, it disrupted task performance in six of seven models, producing accuracy below 5% (H4 not confirmed). Token-level logprobability did not usefully predict verbalised confidence under the observed variance regime (H5 confirmed, mean cross-validated R^2 < 0.01). Within the reasoning-distilled model, reasoning-trace length showed a strong negative partial correlation with confidence (rho = -0.36, p < .001), consistent with the Reasoning Contamination Effect. These results do not imply that internal uncertainty representations are absent. They show that minimal verbal elicitation fails to preserve internal signals at the output interface in this model-size regime. Psychometric screening should precede any downstream use of such signals.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
