arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.05337cs.CLcs.AI
收藏

迈向在动作层面客观基准化语言智能体的社会智能

Towards Objectively Benchmarking Social Intelligence for Language Agents at Action Level

Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系)

Chenxu Wang, Bin Dai, Huaping Liu, Baoyuan Wang

AI总结:

本文提出 STSS 沙盒模拟基准,在动作层面对语言智能体的社会智能进行客观评估,并通过 TDP 模块验证其区分模型与智能体架构的有效性。

摘要:

主流大语言模型已在众多领域展现出人类水平的性能,甚至使衍生出的智能体能够模拟人类与社会互动。尽管实践工作已经证实将语言智能体落地到沙盒模拟或具身模拟器中的可行性,但当前的社会智能基准要么停留在语言层面,要么使用主观指标。为追求更真实、客观的评估,我们提出 Social Tasks in Sandbox Simulation(STSS)基准,通过在多智能体模拟中审查目标达成情况,在动作层面客观评估语言智能体。此外,我们抽样对话场景以构建语言层面基准,从而提供经济审慎的初步评估,并与主流基准保持一致。为衡量智能体架构的重要性,我们实现了一个目标驱动规划(TDP)模块,作为现有智能体的附加组件。我们的评估结果表明,STSS 基准对最先进的语言智能体具有挑战性。此外,它能有效区分不同的语言智能体,表明其适合作为评估语言模型和智能体架构的基准。

英文摘要:

Prominent large language models have exhibited human-level performance in many domains, even enabling the derived agents to simulate human and social interactions. While practical works have substantiated the practicability of grounding language agents in sandbox simulation or embodied simulators, current social intelligence benchmarks either stay at the language level or use subjective metrics. In pursuit of a more realistic and objective evaluation, we introduce the Social Tasks in Sandbox Simulation (STSS) benchmark, which assesses language agents \textbf{objectively} at the \textbf{action level} by scrutinizing the goal achievements within the multi-agent simulation. Additionally, we sample conversation scenarios to build a language-level benchmark to provide an economically prudent preliminary evaluation and align with prevailing benchmarks. To gauge the significance of agent architecture, we implement a target-driven planning (TDP) module as an adjunct to the existing agent. Our evaluative findings highlight that the STSS benchmark is challenging for state-of-the-art language agents. Furthermore, it effectively discriminates between distinct language agents, suggesting its usefulness as a benchmark for evaluating both language models and agent architectures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑