ProcAgent：一种用于边缘端人机协作的过程性任务指导的智能体框架
ProcAgent: An Agentic Framework for Procedural Task Guidance on Edge with Human-in-the-Loop
- University of Tennessee Knoxville（田纳西大学诺克斯维尔分校）
- Worcester Polytechnic Institute（伍斯特理工学院）
机构由 AI 辅助整理，请以论文原文为准。
中文总结 AI 辅助
研究针对家具组装等过程性任务，提出ProcAgent框架，利用提议与验证架构，结合多种技术，在边缘设备上实现实时自适应指导，经多维度评估及用户研究，证明能在不牺牲可用性的情况下于边缘硬件达成自适应过程性辅助。
中文摘要 AI 辅助
诸如家具组装和家庭维修等过程性任务对认知要求很高，因为用户在执行身体动作时必须解释指令、跟踪任务进度、推理空间状态并从错误中恢复。先前的多模态助手在过程性指导方面有前景，但大多依赖云推理和固定的始终开启的感知，不适用于对隐私敏感、对延迟要求高的家庭环境。我们提出ProcAgent，一种完全在设备上运行、基于视觉的智能体过程性助手，用于在单个NVIDIA Jetson AGX Orin上进行实时自适应指导。ProcAgent使用一种提议与验证架构，结合低延迟连续感知、符号任务图、按需视觉语言验证和基于大语言模型的交互智能体。该系统不断提出用户进展，仅在出现歧义或可能的偏差时调用昂贵的视觉推理，并支持带有人机循环确认的反应式问答和主动干预。我们从感知准确性、推理、任务级性能和用户体验四个维度评估ProcAgent。尽管完全在设备上运行，该系统保持响应式交互，纯文本查询约2秒解决，视觉基础查询约8秒解决。在一项有10名参与者完成组装任务的用户研究中，ProcAgent在可理解性、可操作性和隐私舒适度方面获得正面评价。这些结果表明，自适应过程性辅助可以在不牺牲可用性的情况下完全在边缘硬件上实现。
英文摘要
Procedural tasks such as furniture assembly and home repair impose substantial cognitive demands because users must interpret instructions, track task progress, reason about spatial state, and recover from errors while performing physical actions. Prior multimodal assistants have shown promise for procedural guidance, but most rely on cloud inference and fixed always-on perception, making them poorly suited to privacy-sensitive, latency-critical domestic settings. We present ProcAgent, a fully on-device, agentic, vision-based procedural assistant for real-time adaptive guidances on a single NVIDIA Jetson AGX Orin. ProcAgent uses a propose-and-verify architecture that combines low-latency continuous perception, a symbolic task graph, on-demand vision-language verification, and an LLM-based interaction agent. The system continuously proposes user progress, invokes expensive visual reasoning only when ambiguity or likely deviation arises, and supports both reactive question answering and proactive intervention with human-in-the- loop confirmation. We evaluate ProcAgent along four dimensions: perception accuracy, reasoning, task-level performance, and user experience. Despite running entirely on-device, the system maintains responsive interaction, resolving text-only queries in approximately 2 seconds and visually grounded queries in approximately 8 seconds. In a user study with 10 participants completing assembly tasks, ProcAgent receives positive ratings for comprehensibility, actionability, and privacy comfort. These results show that adaptive procedural assistance can be achieved entirely on edge hardware without sacrificing usability.
