Ego-EXTRA：视频语言眼动数据集用于专家-学员协助
Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistance
AI总结：
Ego-EXTRA数据集通过专家-学员双向对话评估多模态大语言模型，揭示其在专家级帮助中的局限性。
摘要：
我们提出了Ego-EXTRA，一个用于专家-学员协助的视频语言眼动数据集。Ego-EXTRA包含50小时的无剧本眼动视频，记录受试者（学员）在专家指导下进行程序性活动。按照“奥兹巫师”数据收集范式，专家扮演可穿戴智能助手，仅从自身视角观察学员活动，回答学员提问或主动与建议互动。这种独特的数据收集协议使Ego-EXTRA能够捕捉高质量的对话，其中专家级反馈提供给学员。专家与学员之间的双向对话被记录、转录，并用于创建包含超过15000个高质量视觉问答数据集的新型基准，我们用其评估多模态大语言模型。结果显示，Ego-EXTRA具有挑战性，突显了当前模型在提供专家级帮助时的局限性。Ego-EXTRA数据集已公开供支持眼动视频语言助手基准使用：https://fpv-iplab.github.io/Ego-EXTRA/。
英文摘要：
We present Ego-EXTRA, a video-language Egocentric Dataset for EXpert-TRAinee assistance. Ego-EXTRA features 50 hours of unscripted egocentric videos of subjects performing procedural activities (the trainees) while guided by real-world experts who provide guidance and answer specific questions using natural language. Following a ``Wizard of OZ'' data collection paradigm, the expert enacts a wearable intelligent assistant, looking at the activities performed by the trainee exclusively from their egocentric point of view, answering questions when asked by the trainee, or proactively interacting with suggestions during the procedures. This unique data collection protocol enables Ego-EXTRA to capture a high-quality dialogue in which expert-level feedback is provided to the trainee. Two-way dialogues between experts and trainees are recorded, transcribed, and used to create a novel benchmark comprising more than 15k high-quality Visual Question Answer sets, which we use to evaluate Multimodal Large Language Models. The results show that Ego-EXTRA is challenging and highlight the limitations of current models when used to provide expert-level assistance to the user. The Ego-EXTRA dataset is publicly available to support the benchmark of egocentric video-language assistants: https://fpv-iplab.github.io/Ego-EXTRA/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
