EgoVoice：来自第一人称多模态流的主动语音助手
EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams
浏览论文内容
中文总结 AI 辅助
EgoVoice是用于训练评估主动第一人称语音助手的框架，通过处理HoloAssist数据微调全模态LLM并结合直接偏好优化，在干预时机、内容相关性和人类偏好上优于零样本基准模型。
中文摘要 AI 辅助
可穿戴增强现实（AR）助手正朝着与现实世界持续交互的方向发展，这类助手通过第一人称视频和音频感知用户的活动，并在用户未明确请求时提供及时的语音指导。尽管主动视频助手、语音对话系统和第一人称任务理解各自取得了快速进展，但现有系统并未解决从连续第一人称流中决定何时发言以及说什么的联合问题。我们提出EgoVoice，这是一个用于训练和评估主动第一人称语音助手的框架。我们从真实人类指导者的HoloAssist视频记录中，通过源分离和语音重合成构建干净的音频流，并将每个视频会话转换为模型必须在每个时刻决定是保持沉默还是提供语音指导的格式。我们用我们的数据对全模态大语言模型（LLM）进行微调，并通过直接偏好优化进一步改进其主动干预行为。在闭源和开源模型上进行的实验表明，现有系统很少产生时机恰当、有意义的主动干预，而EgoVoice在干预时机、内容相关性和人类偏好方面，相较于零样本基准模型取得了明显改进。
英文摘要
Wearable augmented reality (AR) assistants are moving toward continuous real-world interaction, where they perceive the user's activity through first-person video and audio and provide timely spoken guidance without being explicitly asked. While proactive video assistants, spoken dialog systems, and egocentric task understanding have each advanced rapidly, existing systems do not address the joint problem of deciding when to speak and what to say from continuous first-person streams. We introduce EgoVoice, a framework for training and evaluating proactive egocentric spoken assistants. From HoloAssist video recordings of real human instructors, we construct clean audio streams through source separation and speech resynthesis, and convert each video session into a format where the model must decide at each moment whether to remain silent or provide spoken guidance. We fine-tune an omni-modal LLM with our data, and further improve its proactive intervention behavior with direct preference optimization. Experiments across closed and open-source models show that existing systems rarely produce well-timed, meaningful proactive interventions, while EgoVoice yields clear improvements in intervention timing, content relevance, and human preference over the zero-shot backbone.
发表机构
- University of Seoul（首尔大学）
机构由 AI 辅助整理，请以论文原文为准。
