发表机构
State University of New York at Buffalo（纽约州立大学布法罗分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究提出多智能体大语言模型框架MistyPilot，可解释自然语言指令并编排Misty社交机器人技能，经评估其在多项任务上准确率高、方差低，用户反馈积极，代码将公开。
AI 中文摘要
从小型社交机器人的自然语言指令编程，仅调用孤立API是不够的。交互任务需将反应式物理行为与有状态的社交行为结合，而现有接口常要求开发者手动将API组合成技能、配置其参数、绑定传感器事件到技能，并在运行时管理任务状态。我们提出MistyPilot，一个多智能体大语言模型框架，可解释高级自然语言指令并在Misty社交机器人上编排对应技能。任务路由器将每条指令分派给两个专用智能体之一：用于传感器触发的机器人控制和直接技能调用的物理交互智能体，以及用于面向对话的任务状态管理和依赖上下文的多模态响应生成的社交交互智能体。为提升效率，社交交互智能体在适用时复用先前生成的结果，否则调用完整生成。我们在五个组件级套件上评估MistyPilot，传感器绑定和技能调用在实体Misty机器人上执行，并开展了有12名参与者的初步用户研究。MistyPilot在路由、传感器-技能绑定、任务状态解析、结果复用及最多100个技能的技能扩展上达到高准确率，且与其他完全相同的单智能体基线相比方差更低，同时参与者对可用性和交互质量报告了积极评价。代码将通过项目页面公开提供。
英文摘要
Programming small social robots from natural-language instructions requires more than invoking isolated APIs. Interactive tasks combine reactive physical behaviors with stateful social behaviors, while existing interfaces often require developers to manually compose APIs into skills, configure their parameters, bind sensor events to skills, and manage task states at runtime. We present MistyPilot, a multi-agent LLM framework that interprets high-level natural-language instructions and orchestrates the corresponding skills on the Misty social robot. A Task Router dispatches each instruction to one of two specialized agents: a Physically Interactive Agent for sensor-triggered robot control and direct skill invocation, and a Social Interaction Agent for dialogue-oriented task-state management and context-dependent multimodal response generation. To improve efficiency, the Social Interaction Agent reuses previously generated results when applicable and invokes full generation otherwise. We evaluate MistyPilot on five component-level suites, with sensor bindings and skill invocations executed on the physical Misty robot, and a preliminary user study with 12 participants. MistyPilot attains high accuracy on routing, sensor-skill binding, task-state parsing, result reuse, and skill extension up to 100 skills, and lower variance than an otherwise identical single-agent baseline, while participants report positive perceptions of usability and interaction quality. The code will be made publicly available via the project page.
CommentsAccepted at the ECCV 2026 ACVR Workshop
