DigitalCoach：人类与智能体计算机使用辅导中的沟通与基础差距
DigitalCoach: Communication and Grounding Gaps in Human and Agentic Computer Use Coaching
AI总结：
提出DigitalCoach数据集，包含72次人机辅导会话，评估模型在计算机使用教学中的表现，发现模型在解释、错误诊断和视觉基础方面存在不足。
摘要：
智能体越来越能够自动化软件任务，但它们能否教会人类自己使用软件？我们引入了DigitalCoach，这是一个多模态数据集，包含72次人类专家-新手计算机使用辅导会话，包括22,752轮对话，基于28.1小时的屏幕和输入事件记录，涵盖五个软件应用程序。我们使用DigitalCoach评估最先进的模型是否能教会人类使用计算机。自动评估显示，模型在辅导方式上与人类不同：模型提供更直接的指令，但更少的解释、错误诊断和知识检查问题。当我们固定辅导方法时，模型产生的语句与人类参考相似，但在视觉上下文中基础较差。交互评估证实，模型辅导员导致学习者被动地遵循指令，缺乏深度参与，并且在视觉基础方面存在不足。DigitalCoach为协作和主动的计算机使用辅导智能体奠定了基础。
英文摘要：
Agents are increasingly capable of automating software tasks, but can they teach humans how to use software themselves? We introduce DigitalCoach, a multimodal dataset of 72 human expert-novice computer use coaching sessions consisting of 22,752 dialogue turns grounded in 28.1 hours of screen and input event recordings across five software applications. We use DigitalCoach to evaluate whether state-of-the-art models can teach humans how to use computers. Automated evaluation shows that models differ from humans in how they coach: models provide more direct instructions, but fewer explanations, error diagnoses, and knowledge-check questions. When we fix the coaching method, models produce utterances similar to human references yet poorly grounded in visual context. Interactive evaluation confirms that model coaches cause learners to passively follow instructions without deeper engagement and fall short in visual grounding. DigitalCoach lays a foundation for collaborative and proactive computer use coaching agents. Data and code are available at https://project-digital-coach.vercel.app.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
