arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.09853cs.CLcs.AI
收藏

X-TURING:面向长期对话代理的增强型与高效图灵测试

X-TURING: Towards an Enhanced and Efficient Turing Test for Long-Term Dialogue Agents

School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学学院)

Weiqi Wu, Hongqiu Wu, Hai Zhao

AI总结:

本文提出X-Turing测试,通过突发对话模式和伪对话历史迭代生成机制减少人类参与,并引入X-Turn通过率指标评估LLMs在长期对话中的类人程度,发现GPT-4等模型随对话轮次增加难以保持一致性。

摘要:

图灵测试检验AI在自然语言对话中是否表现出类人行为。传统设定限制每位参与者每次仅发送一条消息,且需要人类持续参与。这无法反映自然的对话风格,并阻碍了对基于大型语言模型(LLMs)的对话代理在复杂且长期交互中的评估。本文提出X-Turing,通过突发对话模式增强原测试,允许使用连续消息进行更动态的交流。它通过迭代生成模拟代理与人类间长期交互的对话来构成测试过程的主要部分,进一步减少了人类工作量。借助伪对话历史,代理随后与真实人类进行较短对话,该对话与同主题的人类-人类对话配对,通过问卷进行评判。我们引入X-Turn通过率指标,以评估LLMs在不同持续时间下的类人程度。尽管GPT-4等LLMs初期表现良好,在3轮和10轮对话中分别达到51.9%和38.9%的通过率,但随着对话进行其表现下降,凸显了长期保持一致性的困难。

英文摘要:

The Turing test examines whether AIs exhibit human-like behaviour in natural language conversations. The traditional setting limits each participant to one message at a time and requires constant human participation. This fails to reflect a natural conversational style and hinders the evaluation of dialogue agents based on Large Language Models (LLMs) in complex and prolonged interactions. This paper proposes \textbf{\textsc{X-Turing}}, which enhances the original test with a \textit{burst dialogue} pattern, allowing more dynamic exchanges using consecutive messages. It further reduces human workload by iteratively generating dialogues that simulate the long-term interaction between the agent and a human to compose the majority of the test process. With the \textit{pseudo-dialogue} history, the agent then engages in a shorter dialogue with a real human, which is paired with a human-human conversation on the same topic to be judged using questionnaires. We introduce the \textit{X-Turn Pass-Rate} metric to assess the human likeness of LLMs across varying durations. While LLMs like GPT-4 initially perform well, achieving pass rates of 51.9\% and 38.9\% during 3 turns and 10 turns of dialogues respectively, their performance drops as the dialogue progresses, which underscores the difficulty in maintaining consistency in the long term.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑