SimuHome: 一个时间与环境感知的智能家庭LLM代理基准
SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents
AI总结：
SimuHome是一个基于Matter协议的智能家庭LLM代理基准，用于评估智能家庭代理在时间与环境感知方面的能力，特别是工作流调度的挑战。
摘要：
我们介绍了SimuHome，一个高保真的智能家庭模拟器和一个包含600个回合的LLM智能家庭代理基准。现有的智能家庭基准将家庭视为静态系统，既不模拟设备操作如何随时间影响环境变量，也不支持设备命令的工作流调度。SimuHome基于Matter协议，该协议是行业标准，定义了真实智能家庭设备如何通信和操作。代理通过SimuHome的API与设备交互，并观察其行为如何持续影响环境变量，如温度和湿度。我们的基准涵盖了状态查询、隐式用户意图推断、显式设备控制和工作流调度，每种都有可行和不可行的请求。对于工作流调度，模拟器加速时间，使调度的工作流可以立即评估。对18个代理的评估表明，工作流调度是最难的类别，失败在不同代理框架和微调中持续存在。这些发现表明，SimuHome的时间加速模拟可以作为环境，让代理在真正世界中执行之前预验证其行为。
英文摘要：
We introduce $\textbf{SimuHome}$, a high-fidelity smart home simulator and a benchmark of 600 episodes for LLM-based smart home agents. Existing smart home benchmarks treat the home as a static system, neither simulating how device operations affect environmental variables over time nor supporting workflow scheduling of device commands. SimuHome is grounded in the Matter protocol, the industry standard that defines how real smart home devices communicate and operate. Agents interact with devices through SimuHome's APIs and observe how their actions continuously affect environmental variables such as temperature and humidity. Our benchmark covers state inquiry, implicit user intent inference, explicit device control, and workflow scheduling, each with both feasible and infeasible requests. For workflow scheduling, the simulator accelerates time so that scheduled workflows can be evaluated immediately. An evaluation of 18 agents reveals that workflow scheduling is the hardest category, with failures persisting across alternative agent frameworks and fine-tuning. These findings suggest that SimuHome's time-accelerated simulation could serve as an environment for agents to pre-validate their actions before committing them to the real world.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
