arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.04076cs.CL
收藏

大型语言模型是否表现出类似人类的响应偏差?一项关于调查设计的案例研究

Do LLMs exhibit human-like response biases? A case study in survey design

Carnegie Mellon University(卡内基梅隆大学)

Lindia Tjuatja, Valerie Chen, Sherry Tongshuang Wu, Ameet Talwalkar, Graham Neubig

AI总结:

本研究通过调查设计案例,评估LLMs是否展现人类响应偏差,发现多数模型(尤其RLHF后)未能反映人类行为,凸显其作为人类代理的局限性。

摘要:

随着大型语言模型(LLMs)能力的增强,人们越来越期待将LLMs用作人类在需要主观标签的真实世界任务(如调查和民意测验)中的代理。LLMs作为人类代理在主观任务中广泛应用的一个常被引用的障碍是其对提示措辞的敏感性——但有趣的是,人类也会以响应偏差的形式对指令变化表现出敏感性。我们研究了LLMs在多大程度上反映人类的响应偏差(如果有的话)。我们关注调查设计,其中由“提示”措辞变化引起的人类响应偏差已在社会心理学文献中得到广泛探讨。借鉴这些工作,我们设计了一个数据集和框架,以评估LLMs在调查问卷中是否表现出类似人类的响应偏差。我们对九个模型的全面评估表明,流行的开放和商业LLMs通常无法反映类似人类的行为,尤其是在经过RLHF的模型中。此外,即使某个模型在与人类相同的方向上表现出显著变化,我们也发现它们对不会引起人类显著变化的扰动很敏感。这些结果凸显了使用LLMs作为人类代理的陷阱,并强调了对模型行为进行更细粒度表征的必要性。我们的代码、数据集和收集的样本可在https://github.com/lindiatjuatja/BiasMonkey获取。

英文摘要:

As large language models (LLMs) become more capable, there is growing excitement about the possibility of using LLMs as proxies for humans in real-world tasks where subjective labels are desired, such as in surveys and opinion polling. One widely-cited barrier to the adoption of LLMs as proxies for humans in subjective tasks is their sensitivity to prompt wording - but interestingly, humans also display sensitivities to instruction changes in the form of response biases. We investigate the extent to which LLMs reflect human response biases, if at all. We look to survey design, where human response biases caused by changes in the wordings of "prompts" have been extensively explored in social psychology literature. Drawing from these works, we design a dataset and framework to evaluate whether LLMs exhibit human-like response biases in survey questionnaires. Our comprehensive evaluation of nine models shows that popular open and commercial LLMs generally fail to reflect human-like behavior, particularly in models that have undergone RLHF. Furthermore, even if a model shows a significant change in the same direction as humans, we find that they are sensitive to perturbations that do not elicit significant changes in humans. These results highlight the pitfalls of using LLMs as human proxies, and underscore the need for finer-grained characterizations of model behavior. Our code, dataset, and collected samples are available at https://github.com/lindiatjuatja/BiasMonkey

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑