发表机构
Sungkyunkwan University; Nanyang Technological University（成均馆大学; 南洋理工大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出ProactiveCoach套件，通过层次化程序指导数据训练VLM，提升主动式AI助手在正确时间提供恰当指导的能力，相比固定粒度监督性能提升9.6%，自适应系统比基线高57.1%。
AI 中文摘要
主动式AI助手持续观察用户的活动，并决定是提供新的指导还是保持沉默。它们应该为任务提供恰当的指导，根据任务进度确定何时提供下一步指导，并根据用户的专业水平和需求调整指导的粒度。支持这些能力需要反映程序结构以及指导如何适应任务进度和用户需求的训练和评估数据。然而，现有数据集要么侧重于基于检测的主动理解，要么以固定粒度提供程序性指导。固定粒度的指导提供的关于细粒度进度和更广泛的程序上下文的信息有限，使得难以确定完成情况并调整指导粒度。为解决这些限制，我们引入了ProactiveCoach套件，包括用于训练的ProactiveCoach-Instruct、用于评估的ProactiveCoachBench，以及带有自适应指导系统的微调视觉语言模型（VLM）。ProactiveCoach-Instruct在阶段、步骤和动作级别提供层次化结构指导，以学习任务进度和程序上下文。ProactiveCoachBench评估模型是否在不同指导级别上在正确时间提供恰当指导，并在请求级别变化时进行调整。我们在ProactiveCoach-Instruct上微调预训练的VLM，并展示其在不同骨干网络上的有效性。与固定粒度监督相比，层次化监督在不同骨干网络上的整体性能提高了最多9.6个百分点。我们进一步通过将微调模型与轻量级指导路由器相结合，构建了自适应指导系统。无需额外微调，我们的系统在四种指导级别转换中比上下文自适应基线高出57.1个百分点。我们的项目页面可在此https URL访问。
英文摘要
Proactive AI assistants continuously observe a user's activity and decide whether to provide new guidance or remain silent. They should provide appropriate guidance for the task, determine when to provide the next guidance based on task progress, and adjust the guidance level to the user's expertise and needs. Supporting these capabilities requires training and evaluation data that reflect procedural structure and capture how guidance should adapt to task progress and user needs. However, existing datasets either focus on detection-based proactive understanding or provide procedural guidance at a fixed granularity. Fixed-granularity guidance provides limited information about fine-grained progress and broader procedural context, making it difficult to determine completion and adapt guidance granularity. To address these limitations, we introduce the ProactiveCoach suite, comprising ProactiveCoach-Instruct for training, ProactiveCoachBench for evaluation, and fine-tuned VLMs with an adaptive guidance system. ProactiveCoach-Instruct provides hierarchically structured guidance at the phase, step, and action levels for learning task progress and procedural context. ProactiveCoachBench evaluates whether models provide appropriate guidance at the right time across different guidance levels and adapt when the requested level changes. We fine-tune pretrained VLMs on ProactiveCoach-Instruct and demonstrate its effectiveness across backbones. Compared with fixed-granularity supervision, hierarchical supervision improves overall performance across backbones by up to 9.6%p. We further build an adaptive guidance system by combining our fine-tuned model with a lightweight guidance router. Without additional fine-tuning, our system outperforms the in-context adaptation baseline by 57.1%p across four guidance-level transitions. Our project page is available at https://jinsuby.github.io/ProactiveCoach/.
Comments30 pages
