从流式第一人称视频生成主动助手对话
Proactive Assistant Dialogue Generation from Streaming Egocentric Videos
AI总结：
本文针对流式第一人称视频下的实时主动任务引导难题，提出包含大规模合成对话数据集、经人类研究验证的自动评估指标及端到端生成模型在内的综合框架，为构建实时主动式AI助手奠定基础。
摘要：
近年来对话式人工智能取得了显著进展，但开发用于感知任务引导的实时系统仍具挑战性。这些系统必须基于流式视觉输入提供交互式、主动的辅助，然而其开发受到数据收集和系统评估过程成本高昂且劳动密集的制约。为应对这些局限，我们提出了一个综合框架，包含三项关键贡献。首先，我们引入了一种新颖的数据整理流程，从标注的第一人称视频中合成对话，从而构建了\dataset，一个跨多个领域的大规模合成对话数据集。其次，我们开发了一套自动评估指标，并通过广泛的人类研究进行了验证。第三，我们提出了一个端到端模型，该模型处理流式视频输入以生成符合上下文的响应，并融合了处理数据不平衡和长时视频的新技术。这项工作为开发能够引导用户完成多样化任务的实时、主动式人工智能助手奠定了基础。项目页面：https://pro-assist.github.io/
英文摘要：
Recent advances in conversational AI have been substantial, but developing real-time systems for perceptual task guidance remains challenging. These systems must provide interactive, proactive assistance based on streaming visual inputs, yet their development is constrained by the costly and labor-intensive process of data collection and system evaluation. To address these limitations, we present a comprehensive framework with three key contributions. First, we introduce a novel data curation pipeline that synthesizes dialogues from annotated egocentric videos, resulting in \dataset, a large-scale synthetic dialogue dataset spanning multiple domains. Second, we develop a suite of automatic evaluation metrics, validated through extensive human studies. Third, we propose an end-to-end model that processes streaming video inputs to generate contextually appropriate responses, incorporating novel techniques for handling data imbalance and long-duration videos. This work lays the foundation for developing real-time, proactive AI assistants capable of guiding users through diverse tasks. Project page: https://pro-assist.github.io/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
