arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.11501cs.CLcs.AIcs.CY
收藏

CoMPosT:LLM模拟中夸张化刻画的特征分析与评估

CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations

Stanford University(斯坦福大学)

Myra Cheng, Tiziano Piccardi, Diyi Yang

AI总结:

提出CoMPosT框架,通过上下文、模型、人物设定和主题四个维度,利用个性化和夸张化标准评估LLM模拟的夸张化刻画问题,发现GPT-4在模拟特定人群和主题时极易出现夸张化。

摘要:

近期的研究旨在通过使用LLM在社会科学实验和民意调查等场景中模拟特定人群的回应,来捕捉人类行为的细微差别。然而,目前尚无既定的方法来讨论或评估此类LLM模拟的质量。此外,人们越来越担心这些LLM模拟是对其旨在模拟的人物的扁平化夸张刻画,未能捕捉人类的多维性并使刻板印象固化。为了弥补这些空白,我们提出了CoMPosT,一个使用四个维度来刻画LLM模拟的框架:Context(上下文)、Model(模型)、Persona(人物设定)和Topic(主题)。我们使用该框架来衡量开放式LLM模拟对夸张化刻画的易感性,这通过两个标准来定义:个性化和夸张化。我们评估了现有LLM模拟研究中的场景的夸张化程度。我们发现,对于GPT-4,某些人群(政治和边缘群体)和主题(一般的、无争议的)的模拟极易出现夸张化刻画。

英文摘要:

Recent work has aimed to capture nuances of human behavior by using LLMs to simulate responses from particular demographics in settings like social science experiments and public opinion surveys. However, there are currently no established ways to discuss or evaluate the quality of such LLM simulations. Moreover, there is growing concern that these LLM simulations are flattened caricatures of the personas that they aim to simulate, failing to capture the multidimensionality of people and perpetuating stereotypes. To bridge these gaps, we present CoMPosT, a framework to characterize LLM simulations using four dimensions: Context, Model, Persona, and Topic. We use this framework to measure open-ended LLM simulations' susceptibility to caricature, defined via two criteria: individuation and exaggeration. We evaluate the level of caricature in scenarios from existing work on LLM simulations. We find that for GPT-4, simulations of certain demographics (political and marginalized groups) and topics (general, uncontroversial) are highly susceptible to caricature.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑