ReFract：基于文本世界模型的语言模型智能体视角感知基准测试

中文摘要 AI 辅助

大型语言模型（LLM）智能体越来越多地部署在工业维护和设备故障排除等高风险场景中，这些场景中工人扮演着各种角色。因此，一个合格的智能体必须根据用户的角色来调整其行为：采取行动和提供信息时，要尊重该角色的知识和能力边界。与编码不同，编码中的错误通常可以恢复，而智能体在这些场景中的响应是在物理设备上执行的，因此可能导致不可逆转的设备损坏、生产损失或人员伤害。然而，现有基准大多忽视了智能体需要推断角色的意图，并且仅通过该角色可能合法使用的工具来行动，我们将这种能力称为“视角感知”。为此，我们引入了ReFract，一个包含150个专家验证条目的基准，其中智能体必须根据用户的角色对相同的查询做出不同的响应。ReFract的条目基于领域支持对话中的匿名查询，我们针对这些查询构建了文本世界模型，以模拟智能体的操作环境，并组装视角感知的动作轨迹。最先进的LLM最多能解决69%的任务，且超过50%的轨迹包含尝试采取违反视角的动作。ReFract揭示了视角感知是智能体评估中一个独立的、很大程度上未解决的维度，并促使智能体不仅校准如何行动，还要校准为谁行动。

英文摘要

Large Language Model (LLM) agents are increasingly deployed in high-stakes settings such as industrial maintenance and equipment fault troubleshooting, where workers occupy a variety of roles. A capable agent must therefore act in a way that is calibrated to user's role: taking actions and providing information that respect the role's knowledge and capability boundaries. Unlike coding, where mistakes are usually recoverable, agent responses in these settings are enacted on physical equipment, and can therefore cause irreversible equipment damage, production loss, or personnel harm. Existing benchmarks, however, largely overlook the need for agents to infer what a role intends and acting only through tools that role may legitimately use, a capability which we term Perspective Awareness. To this end, we introduce ReFract, a benchmark of 150 expert-validated entries in which an agent must act differently in response to the same query depending on user's role. Entries of ReFract are grounded in anonymized queries from domain support conversations, against which we construct Text World Models that simulate the agent's operating environments and assemble perspective-aware action trajectories. State-of-the-art LLMs solve at most 69% of the tasks with more than 50% of their trajectories contain attempts of taking perspective-violating actions. ReFract exposes perspective awareness as a distinct, largely unsolved axis of agent evaluation and motivates agents that calibrate not just how to act, but for whom.

发表机构

• Amazon（亚马逊）
• The Alan Turing Institute（艾伦·图灵研究所）
• Universidade Federal Fluminense（弗鲁米嫩塞联邦大学）

机构由 AI 辅助整理，请以论文原文为准。