对LLM心理测评中数据污染程度的量化
Quantifying Data Contamination in Psychometric Evaluations of LLMs
AI总结：
本文提出了一种量化LLM心理测评中数据污染的框架，发现主流心理量表清单存在显著污染问题。
摘要：
近期研究将心理量表问卷应用于大型语言模型（LLMs），以评估诸如价值观、个性、道德基础和黑暗特质等高层心理构念。尽管先前工作已提出对心理量表清单可能造成数据污染的担忧，这种污染可能威胁此类评估的可靠性，但此前尚无系统尝试量化这种污染的程度。为填补这一空白，我们提出一个框架，系统地测量LLM心理测评中的数据污染，评估三个方面：（1）项目记忆，（2）评估记忆，（3）目标分数匹配。将此框架应用于21个主要家族的模型和四个广泛使用的心理量表清单，我们提供了证据表明，如大五清单（BFI-44）和肖像价值观问卷（PVQ-40）等流行清单表现出强烈的污染，其中模型不仅记忆项目，还能调整其响应以达到特定的目标分数。
英文摘要：
Recent studies apply psychometric questionnaires to Large Language Models (LLMs) to assess high-level psychological constructs such as values, personality, moral foundations, and dark traits. Although prior work has raised concerns about possible data contamination from psychometric inventories, which may threaten the reliability of such evaluations, there has been no systematic attempt to quantify the extent of this contamination. To address this gap, we propose a framework to systematically measure data contamination in psychometric evaluations of LLMs, evaluating three aspects: (1) item memorization, (2) evaluation memorization, and (3) target score matching. Applying this framework to 21 models from major families and four widely used psychometric inventories, we provide evidence that popular inventories such as the Big Five Inventory (BFI-44) and Portrait Values Questionnaire (PVQ-40) exhibit strong contamination, where models not only memorize items but can also adjust their responses to achieve specific target scores.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
