PersonalHomeBench：评估智能家庭中的代理系统
PersonalHomeBench: Evaluating Agents in Personalized Smart Homes
AI总结：
本文提出PersonalHomeBench，用于评估代理在个性化智能家庭中的表现，通过迭代构建家庭状态生成任务，结合工具箱支持真实交互，揭示任务复杂度增加时代理能力下降的问题。
摘要：
Agentic AI系统正快速向实际应用发展，但其在复杂个性化环境中的准备程度仍不足。为填补这一空白，我们引入PersonalHomeBench，一个用于评估基础模型作为代理助手的基准。该基准通过迭代过程逐步构建丰富的家庭状态，然后用于生成个性化、上下文依赖的任务。为了支持真实的代理-环境交互，我们提供了PersonalHomeTools，一个综合工具箱，能够实现家庭信息检索、家电控制和情境理解。PersonalHomeBench在单模态和多模态观察下评估反应性和前瞻性代理能力。彻底的实验揭示了随着任务复杂性增加，性能系统性下降，特别是在因果推理和部分可观测性中出现明显失败，需要有效的基于工具的信息收集。这些结果使PersonalHomeBench成为分析个性化代理推理和规划鲁棒性和局限性的严格评估平台。
英文摘要：
Agentic AI systems are rapidly advancing toward real-world applications, yet their readiness in complex and personalized environments remains insufficiently characterized. To address this gap, we introduce PersonalHomeBench, a benchmark for evaluating foundation models as agentic assistants in personalized smart home environments. The benchmark is constructed through an iterative process that progressively builds rich household states, which are then used to generate personalized, context-dependent tasks. To support realistic agent-environment interaction, we provide PersonalHomeTools, a comprehensive toolbox enabling household information retrieval, appliance control, and situational understanding. PersonalHomeBench evaluates both reactive and proactive agentic abilities under unimodal and multimodal observations. Thorough experimentation reveals a systematic performance reduction as task complexity increases, with pronounced failures in counterfactual reasoning and under partial observability, where effective tool-based information gathering is required. These results position PersonalHomeBench as a rigorous evaluation platform for analyzing the robustness and limitations of personalized agentic reasoning and planning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
