EgoInstruct：一个面对面教学互动的第一视角视频数据集及多模态大语言模型基准评测
EgoInstruct: An Egocentric Video Dataset of Face-to-face Instructional Interactions with Multi-modal LLM Benchmarking
AI总结：
本文构建面对面教学第一视角视频数据集 EgoInstruct，标注流程步骤分割与对话状态分类任务，并评测联合处理图像、音频和文本的 MLLMs，证明其无需任务专用微调即可优于专用基线。
摘要：
分析指导者与学习者共同处于同一物理空间中的教学互动，是教育支持和技能迁移中的一个关键问题。然而，这类面对面教学场景尚未在计算机视觉中得到系统研究。我们指出两个主要原因：i）缺乏合适的数据集；ii）分析技术有限。为弥补这一空白，我们提出了一个新的面对面教学第一视角视频数据集，并为两项基础任务提供真值标注，作为全面理解教学互动的第一步：流程步骤分割和对话状态分类。利用该数据集，我们将多模态大语言模型（MLLMs）与传统的任务专用模型进行基准比较。由于面对面教学涉及多种模态（语音内容与韵律、凝视与身体动作以及视觉情境），有效理解需要以整合方式处理言语和非言语交流的方法。因此，我们评估了近期提出的、能够联合处理图像、音频和文本的 MLLMs。该评估量化了当前机器学习模型对面对面教学场景的理解程度。实验中，MLLMs 即使在没有任务专用微调的情况下也优于专用基线，表明其在整体理解教学互动方面具有前景。
英文摘要：
Analyzing instructional interactions between an instructor and a learner who are co-present in the same physical space is a critical problem for educational support and skill transfer. Yet such face-to-face instructional scenes have not been systematically studied in computer vision. We identify two key reasons: i) the lack of suitable datasets and ii) limited analytical techniques. To address this gap, we present a new egocentric video dataset of face-to-face instruction and provide ground-truth annotations for two fundamental tasks that serve as a first step toward a comprehensive understanding of instructional interactions: procedural step segmentation and conversation-state classification. Using this dataset, we benchmark multimodal large language models (MLLMs) against conventional task-specific models. Since face-to-face instruction involves multiple modalities (speech content and prosody, gaze and body motion, and visual context), effective understanding requires methods that handle verbal and nonverbal communication in an integrated manner. Accordingly, we evaluate recently introduced MLLMs that jointly process images, audio, and text. This evaluation quantifies the extent to which current machine learning models understand face-to-face instructional scenes. In experiments, MLLMs outperform specialized baselines even without task-specific fine-tuning, suggesting their promise for holistic understanding of instructional interactions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
