Sporthesia:使用自然语言增强体育视频
Sporthesia: Augmenting Sports Videos Using Natural Language
AI总结:
本文提出Sporthesia系统,通过自然语言解说检测可视化实体、映射并调度可视化以生成增强型体育视频,评估显示实体检测F1为0.9且获得体育分析师认可。
摘要:
增强型体育视频结合可视化与视频特效,在真实场景中呈现数据,能够以引人入胜的方式传达洞见,因此在全球体育爱好者中日益流行。然而,制作增强型体育视频仍然是一项具有挑战性的任务,需要大量时间和视频编辑技能。另一方面,体育洞见通常通过自然语言传达,例如解说、口头报告和文章,但往往缺乏视觉线索。因此,本工作旨在通过让分析师直接利用以自然语言表达的洞见,创建嵌入视频中的可视化,从而促进增强型体育视频的制作。为实现这一目标,我们提出了一种三步方法:1)检测文本中的可视化实体,2)将这些实体映射到可视化,3)调度这些可视化使其与视频同步播放;并分析了155个体育视频片段及配套解说,以完成这些步骤。基于分析结果,我们设计并实现了概念验证系统Sporthesia,该系统以持拍类运动视频和文本解说为输入,并输出增强视频。我们在两个示例场景中展示了Sporthesia的适用性,即使用文本创作增强型体育视频,以及基于语音解说增强历史体育视频。技术评估表明,Sporthesia在检测文本中的可视化实体方面达到了较高准确率(F1分数为0.9)。一项由八名体育分析师参与的专家评估表明,我们这种语言驱动的创作方法具有很高的实用性、有效性和满意度,并为未来改进和机会提供了洞见。
英文摘要:
Augmented sports videos, which combine visualizations and video effects to present data in actual scenes, can communicate insights engagingly and thus have been increasingly popular for sports enthusiasts around the world. Yet, creating augmented sports videos remains a challenging task, requiring considerable time and video editing skills. On the other hand, sports insights are often communicated using natural language, such as in commentaries, oral presentations, and articles, but usually lack visual cues. Thus, this work aims to facilitate the creation of augmented sports videos by enabling analysts to directly create visualizations embedded in videos using insights expressed in natural language. To achieve this goal, we propose a three-step approach - 1) detecting visualizable entities in the text, 2) mapping these entities into visualizations, and 3) scheduling these visualizations to play with the video - and analyzed 155 sports video clips and the accompanying commentaries for accomplishing these steps. Informed by our analysis, we have designed and implemented Sporthesia, a proof-of-concept system that takes racket-based sports videos and textual commentaries as the input and outputs augmented videos. We demonstrate Sporthesia's applicability in two exemplar scenarios, i.e., authoring augmented sports videos using text and augmenting historical sports videos based on auditory comments. A technical evaluation shows that Sporthesia achieves high accuracy (F1-score of 0.9) in detecting visualizable entities in the text. An expert evaluation with eight sports analysts suggests high utility, effectiveness, and satisfaction with our language-driven authoring method and provides insights for future improvement and opportunities.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。