arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.08663cs.CLcs.AIcs.CY
收藏

通过信息确定评分(IDS)框架用文本衍生项目增强评分尺度测量

Augmenting Rating-Scale Measures with Text-Derived Items Using the Information-Determined Scoring (IDS) Framework

Psychometrics Centre, Judge Business School, University of Cambridge(剑桥大学心理测量中心、剑桥大学商学院);Faculty of Law, University of Cambridge(剑桥大学法学院);School of Computer Science, University of Birmingham(伯明翰大学计算机科学学院);Faculty of Psychology, Beijing Normal University(北京师范大学心理学学院)

Joe Watson, Ivan O'Connor, Chia-Wen Chen, Luning Sun, Fang Luo, David Stillwell

AI总结:

本文提出IDS框架,利用大型语言模型生成文本衍生项目,提升心理测量的精度和效度,通过案例研究展示其在抑郁症测量中的有效性。

摘要:

心理评估通常依赖于评分尺度项目,要求受访者将复杂经验浓缩为预定义类别。尽管丰富的无结构文本常伴随这些量表,但很少贡献于测量目标特质,因为缺乏直接映射到潜在量表。我们引入信息确定评分(IDS)框架,其中大型语言模型(LLMs)使用简单提示对自由文本响应进行评分,生成候选项目,这些项目与基线量表共同校准,并基于其提供的关于目标特质的心理测量信息保留。这标志着传统自动化文本评分的理论突破,优先考虑信息增益而非专家评分或人工标注数据的忠实度。以抑郁为例,我们在上等中学学生(n=693)和匹配的合成数据集(n=3,000)中开发并测试了该方法。在保留测试集上,将19项评分尺度测量与LLM衍生项目结合,显著提高了测量精度和准确性,并在适应性测试中与外部自杀性测量具有更强的收敛效度。在适应性模拟中,LLM衍生项目提供的信息相当于在真实和合成数据中分别增加多达6.3和16.0项评分尺度项目。这使得早期高精度测量成为可能:在10个项目后,46.3%的受访者在最强的增强测试中达到SE<=.3,而在真实数据中基线为35.5%,在合成数据中为60.4%对34.7%。这些发现展示了IDS框架如何利用无结构文本增强现有心理测量,应用于临床健康等领域。

英文摘要:

Psychological assessments commonly rely on rating-scale items, which require respondents to condense complex experiences into predefined categories. Although rich, unstructured text is often captured alongside these scales, it rarely contributes to measuring the target trait because it lacks direct mapping to the latent scale. We introduce the Information-Determined Scoring (IDS) framework, where large language models (LLMs) score free-text responses with simple prompts to generate candidate items that are co-calibrated with a baseline scale and retained based on the psychometric information they provide about the target trait. This marks a conceptual departure from traditional automated text scoring by prioritising information gain over fidelity to expert rubrics or human-annotated data. Using depression as a case study, we developed and tested the method in upper-secondary students (n = 693) and a matched synthetic dataset (n = 3,000). Across held-out test sets, augmenting a 19-item rating-scale measure with LLM-derived items yielded significant improvements in measurement precision and accuracy, and stronger convergent validity with an external suicidality measure throughout the adaptive test. In adaptive simulations, LLM-derived items contributed information equivalent to adding up to 6.3 and 16.0 rating-scale items in real and synthetic data, respectively. This enabled earlier high-precision measurement: after 10 items, 46.3% of respondents reached SE <= .3 under the strongest augmented test compared with 35.5% at baseline in real data, and 60.4% versus 34.7% in synthetic data. These findings illustrate how the IDS framework leverages unstructured text to enhance existing psychological measures, with applications in clinical health and beyond.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑