基于单次演示的多模态线索实现任务辅助的语境化
Grounding Task Assistance with Multimodal Cues from a Single Demonstration
AI总结：
本文提出MICA框架，通过融合眼动注视与语音等多模态线索来增强单次演示中的任务辅助，实验表明多模态结合优于基于帧的检索，并揭示了任务类型对隐式与显式线索有效性的影响。
摘要：
一个人的演示通常作为他人学习同一任务的关键参考。然而，RGB视频作为表示这些演示的主要媒介，往往无法捕捉细粒度的情境线索，例如意图、安全关键的环境因素以及嵌入在人类行为中的微妙偏好。这种感官差距从根本上限制了视觉语言模型（Vision Language Models, VLMs）推理动作发生原因以及如何适应个体用户的能力。为了解决这一问题，我们提出了MICA（Multimodal Interactive Contextualized Assistance，多模态交互式语境化辅助）框架，该框架通过整合眼动注视和语音线索来改进用于任务辅助的对话智能体。MICA将演示分割为有意义的子任务，并提取关键帧和字幕，以捕捉细粒度的意图和用户特定线索，从而为视觉问答提供更丰富的语境基础。基于实时聊天辅助任务复现中产生的问题进行的评估表明，与基于帧的检索相比，多模态线索显著提升了回答质量。值得注意的是，仅使用注视线索即可达到语音性能的93%，而二者结合则取得了最高的准确率。任务类型决定了隐式（注视）线索与显式（语音）线索的有效性，这凸显了对可适应多模态模型的需求。这些结果揭示了基于帧的语境的局限性，并证明了多模态信号在现实世界AI任务辅助中的价值。
英文摘要：
A person's demonstration often serves as a key reference for others learning the same task. However, RGB video, the dominant medium for representing these demonstrations, often fails to capture fine-grained contextual cues such as intent, safety-critical environmental factors, and subtle preferences embedded in human behavior. This sensory gap fundamentally limits the ability of Vision Language Models (VLMs) to reason about why actions occur and how they should adapt to individual users. To address this, we introduce MICA (Multimodal Interactive Contextualized Assistance), a framework that improves conversational agents for task assistance by integrating eye gaze and speech cues. MICA segments demonstrations into meaningful sub-tasks and extracts keyframes and captions that capture fine-grained intent and user-specific cues, enabling richer contextual grounding for visual question answering. Evaluations on questions derived from real-time chat-assisted task replication show that multimodal cues significantly improve response quality over frame-based retrieval. Notably, gaze cues alone achieves 93% of speech performance, and their combination yields the highest accuracy. Task type determines the effectiveness of implicit (gaze) vs. explicit (speech) cues, underscoring the need for adaptable multimodal models. These results highlight the limitations of frame-based context and demonstrate the value of multimodal signals for real-world AI task assistance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
