arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.17115cs.CLcs.CY
收藏

LLM距离可信AI有多远?一个评估人类行为模拟可信性的基准

How Far Are LLMs from Believable AI? A Benchmark for Evaluating the Believability of Human Behavior Simulation

The Hong Kong Polytechnic University(香港理工大学);National University of Singapore(新加坡国立大学);Shanghai Jiao Tong University(上海交通大学)

Yang Xiao, Yi Cheng, Jinlan Fu, Jiashuo Wang, Wenjie Li, Pengfei Liu

AI总结:

本文提出SimulateBench基准,从一致性与鲁棒性两方面、基于65个角色档案和8,400个问题评估10种LLM的人类行为模拟可信性,发现现有模型仍难贴合角色且易受扰动。

摘要:

近年来,AI在模拟人类行为方面展现出显著能力,尤其是使用大型语言模型(LLM)实现的行为模拟。然而,由于缺乏对LLM模拟行为的系统性评估,LLM在人类中的可信性仍然不明确,即尚不清楚LLM的哪些行为能令人信服地表现得像人类,哪些行为还需要进一步改进。在本研究中,我们设计了SimulateBench,用于评估LLM在模拟人类行为时的可信性。具体而言,我们基于两个关键维度评估LLM的可信性:1)一致性:LLM的行为在多大程度上能够与待模拟人类的给定信息保持一致;2)鲁棒性:LLM的模拟行为在面对扰动时保持稳健的能力。SimulateBench包含65个角色档案和共计8,400个问题,用以检验LLM的模拟行为。基于SimulateBench,我们评估了10种广泛使用的LLM在模拟角色时的表现。实验结果表明,当前LLM难以使其行为与所分配角色保持一致,并且在面对某些因素的扰动时较为脆弱。

英文摘要:

In recent years, AI has demonstrated remarkable capabilities in simulating human behaviors, particularly those implemented with large language models (LLMs). However, due to the lack of systematic evaluation of LLMs' simulated behaviors, the believability of LLMs among humans remains ambiguous, i.e., it is unclear which behaviors of LLMs are convincingly human-like and which need further improvements. In this work, we design SimulateBench to evaluate the believability of LLMs when simulating human behaviors. In specific, we evaluate the believability of LLMs based on two critical dimensions: 1) consistency: the extent to which LLMs can behave consistently with the given information of a human to simulate; and 2) robustness: the ability of LLMs' simulated behaviors to remain robust when faced with perturbations. SimulateBench includes 65 character profiles and a total of 8,400 questions to examine LLMs' simulated behaviors. Based on SimulateBench, we evaluate the performances of 10 widely used LLMs when simulating characters. The experimental results reveal that current LLMs struggle to align their behaviors with assigned characters and are vulnerable to perturbations in certain factors.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑