ActTraitBench: 通过人类行为验证量化大型语言模型中的知识-决策差距
ActTraitBench: Quantifying the Knowledge-Decision Gap in Large Language Models via Human-Grounded Behavioral Validation
AI总结：
提出ActTraitBench框架，基于人类数据建立心理测量方面与行为范式的一一映射，并通过分位数映射校准LLM评分分布，揭示LLM在自我报告与行为决策之间的知识-决策差距，并引入CoCA干预来缓解该差距。
摘要：
虽然大型语言模型（LLM）在显式自我报告中能够令人信服地模拟人格，但它们在隐式行为决策中常常出现偏差，揭示了显著的知识-决策差距（$G_{\text{KD}}$）。现有的基准由于结构效度有限、多维度纠缠以及基于LLM评估中的分布偏差，难以衡量这种不对称性。为了解决这些问题，我们提出了ActTraitBench，一个基于人类数据的评估框架，用于衡量LLM中的人格一致性。基于经验人类数据，ActTraitBench建立了心理测量方面与行为范式之间的一一映射，并应用通过分位数映射的分布校准程序，使LLM评判者的分数分布与人类规范对齐。在14个主流LLM上的实验揭示了普遍的知识-决策不对称性，其中更大、能力更强的模型尽管自我报告高度一致，但往往表现出更强的行为分歧。为了缓解这一差距，我们进一步引入了认知对齐链（CoCA），一种即插即用的推理时干预措施，可改善具有推理能力的前沿模型的对齐，同时暴露出较小架构中明显的能力限制。
英文摘要：
While Large Language Models (LLMs) can convincingly simulate personas in explicit self-reports, they often deviate in implicit behavioral decisions, revealing a substantial Knowledge-Decision Gap ($G_{\mathrm{KD}}$). Existing benchmarks struggle to measure this discrepancy due to limited construct validity, multidimensional entanglement, and distributional biases in LLM-based evaluation. To address these issues, we propose ActTraitBench, a human-grounded evaluation framework for measuring personality consistency in LLMs. Grounded in empirical human data, ActTraitBench establishes one-to-one mappings between psychometric facets and behavioral paradigms and applies Distributional Calibration via Quantile Mapping to reduce distributional mismatch between LLM-judge scores and human responses. Experiments on 14 mainstream LLMs reveal substantial knowledge-decision gaps and show that assigned personas are reflected more consistently in self-reports than in behavioral decisions for most evaluated models. To mitigate this gap, we further introduce the Chain of Cognitive Alignment (CoCA), an inference-time intervention that reduces $G_{\mathrm{KD}}$ for 12 of the 13 models with paired results. Code and resources are available at https://github.com/Selina233/ActTraitBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
