ExpertAF: 从视频中获取专家可执行反馈
ExpertAF: Expert Actionable Feedback from Video
AI总结:
针对现有视频技能评估仅提供分数或对比演示的局限,提出ExpertAF方法,利用Ego-Exo4D数据集构建弱监督训练集,通过多模态视频-语言模型生成含专家评论与视觉演示的可执行反馈,性能优于现有视觉-语言模型。
摘要:
反馈对于学习新技能或提升现有技能水平至关重要。然而,当前基于视频的技能评估方法仅提供分数或对比演示,将“需要做出哪些改变”的认知负担留给了用户。我们提出一种从人体进行体育活动(如篮球或足球)的视频中生成可执行反馈(AF)的新方法。该方法接收视频演示及其附带的3D人体姿态,生成(1)描述用户表现优点与改进空间的自由形式专家评论,以及(2)包含必要修正的视觉专家演示。我们展示了如何利用Ego-Exo4D[29]的技能活动视频和专家评论,结合强大的语言模型创建该任务的弱监督训练数据集,并设计了多模态视频-语言模型来推断教练反馈。我们的方法能够跨多模态输入组合进行推理,输出全谱可执行教练专家评论、专家视频检索和专家姿态生成,在既定指标和人类偏好研究中均优于强大的视觉-语言模型。
英文摘要:
Feedback is essential for learning a new skill or improving one's current skill-level. However, current methods for skill-assessment from video only provide scores or compare demonstrations, leaving the burden of knowing what to do differently on the user. We introduce a novel method to generate actionable feedback (AF) from video of a person doing a physical activity, such as basketball or soccer. Our method takes a video demonstration and its accompanying 3D body pose and generates (1) free-form expert commentary describing what the person is doing well and what they could improve, and (2) a visual expert demonstration that incorporates the required corrections. We show how to leverage Ego-Exo4D's [29] videos of skilled activity and expert commentary together with a strong language model to create a weakly-supervised training dataset for this task, and we devise a multimodal video-language model to infer coaching feedback. Our method is able to reason across multi-modal input combinations to output full spectrum, actionable coaching-expert commentary, expert video retrieval, and expert pose generation-outperforming strong vision-language models on both established metrics and human preference studies.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。