arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2604.17707cs.CLcs.AI

在解释轮廓之前:大型语言模型元认知自我报告的有效性缩放

Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report

Independent Researcher(独立研究者)

Jon-Paul Cacioli

AI总结:

本文提出通过有效性缩放框架评估大型语言模型的元认知自我报告,识别出四款模型构造无效,两款模型存在异常,研究揭示了有效性指标与模型性能的关系。

摘要:

临床人格评估筛查在解释实质性量表前会验证有效性。LLM评估却不做此操作。本文将PAI和MMPI-3的有效性框架应用于20种前沿模型的元认知探测数据,共524个项目。六个有效性指标被定义:L(在错误上维持信心)、K(在错误上下注)、F(撤回共识支持的项目)、Fp(撤回正确答案)、RBS(反向监控)和TRIN(固定回应)。分层分类系统识别出四款模型为构造层面无效,两款为升高。有效轮廓模型产生项目敏感的信心(均值r=0.18,16项中14项显著)。无效轮廓模型不产生(均值r=-0.20,d=2.17,p=0.001)。链式思维训练产生两种相反的响应扭曲。两个潜在维度解释了94.6%的指标方差。配套论文提取了可移植的筛查协议(Cacioli, 2026e)并验证了其对选择性预测的有效性(Cacioli, 2026f)。所有数据和代码:https://github.com/synthiumjp/validity-scaling-llm

英文摘要:

Clinical personality assessment screens response validity before interpreting substantive scales. LLM evaluation does not. We apply the validity scaling framework from the PAI and MMPI-3 to metacognitive probe data from 20 frontier models across 524 items. Six validity indices are operationalised: L (maintaining confidence on errors), K (betting on errors), F (withdrawing consensus-endorsed items), Fp (withdrawing correct answers), RBS (inverted monitoring), and TRIN (fixed responding). A tiered classification system identifies four models as construct-level invalid and two as elevated. Valid-profile models produce item-sensitive confidence (mean r = .18, 14 of 16 significant). Invalid-profile models do not (mean r = -.20, d = 2.17, p = .001). Chain-of-thought training produces two opposite response distortions. Two latent dimensions account for 94.6% of index variance. Companion papers extract a portable screening protocol (Cacioli, 2026e) and validate it against selective prediction (Cacioli, 2026f). All data and code: https://github.com/synthiumjp/validity-scaling-llm

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑