arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.08413cs.CL
收藏

使用语言模型集成对自由文本数据进行标注

Labeling Free-text Data using Language Model Ensembles

University of Virginia, School of Data Science(弗吉尼亚大学数据科学学院);Northern Arizona University, College of Health and Human Services(北亚利桑那大学健康与人类服务学院);University of Louisville, Department of Psychology and Brain Sciences(路易斯维尔大学心理学与脑科学系);University of Virginia, Department of Psychology(弗吉尼亚大学心理学系)

Jiaxing Qiu, Dongliang Guo, Natalie Papini, Noelle Peace, Hannah F. Fitterman-Harris, Cheri A. Levinson, Tom Hartvigsen, Teague R. Henry

AI总结:

本研究提出一种组装本地开源LLMs的集成框架,在隐私约束下标注自由文本数据,通过相关性评分平衡模型异质性,在饮食失调数据上实现优于单个模型的最优精确度-敏感性权衡。

摘要:

在心理学研究中,自由文本回复被广泛收集,提供了定量测量可能无法捕捉的丰富定性见解。由多名训练有素的人类编码员对自由文本数据中感兴趣的研究主题进行标注,通常是劳动密集且耗时的。尽管大型语言模型(LLMs)在语言处理方面表现出色,但依赖闭源LLMs的LLM辅助标注技术,在未经明确同意用于外部使用的情况下,不能直接应用于自由文本数据。在本研究中,我们提出了一种组装本地可部署LLMs的框架,以在隐私约束下增强对自由文本数据中预定主题的标注。类似于多名人类评分者的标注,该框架利用了多样化开源LLMs的异质性。集成方法在LLMs之间的一致性和分歧之间寻求平衡,由利用主题描述与LLMs推理之间的嵌入距离的相关性评分方法指导。我们使用来自饮食失调相关论坛的公开可访问的Reddit数据,以及饮食失调患者的自由文本回复,两者均辅以人类标注,对集成方法进行了评估。我们发现:(1)在相同规模的LLMs中,标注性能存在异质性,一些显示出低敏感性但高精确度,而另一些则显示出高敏感性但低精确度。(2)与单个LLMs相比,LLMs集成在预测人类标注方面实现了最高的准确性和最佳的精确度-敏感性权衡。(3)各LLMs之间的相关性评分比二分标签显示出更大的一致性,表明相关性评分方法有效缓解了LLMs标注中的异质性。

英文摘要:

Free-text responses are commonly collected in psychological studies, providing rich qualitative insights that quantitative measures may not capture. Labeling curated topics of research interest in free-text data by multiple trained human coders is typically labor-intensive and time-consuming. Though large language models (LLMs) excel in language processing, LLM-assisted labeling techniques relying on closed-source LLMs cannot be directly applied to free-text data, without explicit consent for external use. In this study, we propose a framework of assembling locally-deployable LLMs to enhance the labeling of predetermined topics in free-text data under privacy constraints. Analogous to annotation by multiple human raters, this framework leverages the heterogeneity of diverse open-source LLMs. The ensemble approach seeks a balance between the agreement and disagreement across LLMs, guided by a relevancy scoring methodology that utilizes embedding distances between topic descriptions and LLMs' reasoning. We evaluated the ensemble approach using both publicly accessible Reddit data from eating disorder related forums, and free-text responses from eating disorder patients, both complemented by human annotations. We found that: (1) there is heterogeneity in the performance of labeling among same-sized LLMs, with some showing low sensitivity but high precision, while others exhibit high sensitivity but low precision. (2) Compared to individual LLMs, the ensemble of LLMs achieved the highest accuracy and optimal precision-sensitivity trade-off in predicting human annotations. (3) The relevancy scores across LLMs showed greater agreement than dichotomous labels, indicating that the relevancy scoring method effectively mitigates the heterogeneity in LLMs' labeling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑