YETI（YET to Intervene）：多模态AI智能体在增强现实任务中的主动干预
YETI (YET to Intervene) Proactive Interventions by Multimodal AI Agents in Augmented Reality Tasks
AI总结：
针对现有多模态AI智能体被动响应的局限，研究提出YETI智能体，基于SSIM场景理解信号与对齐信号判断主动干预时机，在AR任务辅助场景的HoloAssist基准上验证了效果。
摘要：
多模态AI智能体是具备交互协作能力、可辅助人类用户完成日常任务的AI模型。增强现实（AR）头戴设备能够为AI智能体提供第一人称视角的多模态（音频与视频）感知能力，从而独特地提升用户完成流程化日常任务的体验。这类AR能力可帮助AI智能体观察并聆听用户执行的动作，这些动作可与人类用户的多模态能力相关联。现有AI智能体，无论是大语言模型（LLMs）还是多模态视觉语言模型（VLMs），本质上都属于被动响应式，即模型在未读取或接收人类用户提示的情况下无法采取行动。而AI智能体的主动性可帮助人类用户检测并纠正智能体所观察任务中的错误，在用户正确完成任务时给予鼓励，或仅与用户开展对话——类似于人类指导或辅助用户的场景。我们提出的YET to Intervene（YETI）多模态智能体聚焦于识别可能需要智能体主动干预的场景这一研究问题。这使得智能体能够判断何时可介入与人类用户的对话，从而帮助用户纠正烹饪等任务中的错误，且该过程通过AR实现。我们的YETI智能体基于连续视频帧上结构相似性（SSIM）的可解释概念，学习场景理解信号。我们还定义了对齐信号，AI智能体可通过学习该信号，判断对应用户任务操作的视频帧是否与预期动作一致。这些信号被我们的AI智能体用于判断何时应主动干预。我们在HoloAssist多模态基准的主动干预实例上对比了结果，该基准用于模拟专家智能体指导用户完成流程化任务的场景。
英文摘要：
Multimodal AI Agents are AI models that have the capability of interactively and cooperatively assisting human users to solve day-to-day tasks. Augmented Reality (AR) head worn devices can uniquely improve the user experience of solving procedural day-to-day tasks by providing egocentric multimodal (audio and video) observational capabilities to AI Agents. Such AR capabilities can help AI Agents see and listen to actions that users take which can relate to multimodal capabilities of human users. Existing AI Agents, either Large Language Models (LLMs) or Multimodal Vision-Language Models (VLMs) are reactive in nature, which means that models cannot take an action without reading or listening to the human user's prompts. Proactivity of AI Agents on the other hand can help the human user detect and correct any mistakes in agent observed tasks, encourage users when they do tasks correctly or simply engage in conversation with the user - akin to a human teaching or assisting a user. Our proposed YET to Intervene (YETI) multimodal agent focuses on the research question of identifying circumstances that may require the agent to intervene proactively. This allows the agent to understand when it can intervene in a conversation with human users that can help the user correct mistakes on tasks, like cooking, using AR. Our YETI Agent learns scene understanding signals based on interpretable notions of Structural Similarity (SSIM) on consecutive video frames. We also define the alignment signal which the AI Agent can learn to identify if the video frames corresponding to the user's actions on the task are consistent with expected actions. These signals are used by our AI Agent to determine when it should proactively intervene. We compare our results on the instances of proactive intervention in the HoloAssist multimodal benchmark for an expert agent guiding a user to complete procedural tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
