大型语言模型中相似性判断的上下文效应研究
Investigating Context Effects in Similarity Judgements in Large Language Models
AI总结：
本研究通过复现著名人类实验，调查大型语言模型在相似性判断中是否表现出与人类类似的顺序效应偏差，并探讨其对基于LLM应用设计的启示。
摘要：
大型语言模型（LLMs）彻底改变了AI模型在理解和生成自然语言文本方面的能力。它们越来越多地被用于在现实场景中赋能和部署智能体，这些智能体基于对上下文的理解做出决策并采取行动。因此，研究人员、政策制定者与企业都在致力于确保这些智能体做出的决策符合人类价值观和用户期望。尽管如此，人类价值观和决策并不总是易于衡量，且容易受到不同认知偏差的影响。行为科学领域有大量文献研究人类判断中的偏差。在本工作中，我们报告了一项关于LLMs与受顺序偏差影响的人类判断对齐情况的持续调查。具体而言，我们聚焦于一项著名的人类研究，该研究展示了相似性判断中的顺序效应证据，并使用各种流行的LLMs对其进行复现。我们报告了LLMs表现出类似人类顺序效应偏差的不同设置，并讨论了这些发现对基于LLM的应用的设计与开发所带来的启示。
英文摘要：
Large Language Models (LLMs) have revolutionised the capability of AI models in comprehending and generating natural language text. They are increasingly being used to empower and deploy agents in real-world scenarios, which make decisions and take actions based on their understanding of the context. Therefore researchers, policy makers and enterprises alike are working towards ensuring that the decisions made by these agents align with human values and user expectations. That being said, human values and decisions are not always straightforward to measure and are subject to different cognitive biases. There is a vast section of literature in Behavioural Science which studies biases in human judgements. In this work we report an ongoing investigation on alignment of LLMs with human judgements affected by order bias. Specifically, we focus on a famous human study which showed evidence of order effects in similarity judgements, and replicate it with various popular LLMs. We report the different settings where LLMs exhibit human-like order effect bias and discuss the implications of these findings to inform the design and development of LLM based applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
