arXiv 2507.201632025-07-27cs.CV

面向大语言模型身份感知篮球视频字幕生成的以球员为中心的多模态提示生成

Player-Centric Multimodal Prompt Generation for Large Language Model Based Identity-Aware Basketball Video Captioning

Beijing University of Technology（北京理工大学）；Shanghai Jiao Tong University（上海交通大学）；Chinese Academy of Sciences（中国科学院）；The Hong Kong Polytechnic University（香港理工大学）

Zeyu Xi, Haoying Sun, Yaofei Wu, Junchi Yan, Haoran Zhang, Lifang Wu, Liang Wang, Changwen Chen

备注：: Accepted by ICCV 2025 (Poster)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对体育视频字幕中球员身份易出错的问题，本文提出LLM-IAVC以球员为中心生成多模态提示，并构建NBA-Identity数据集，实验验证其先进性能。

摘要：

现有的体育视频字幕生成方法通常关注动作，却忽视球员身份，限制了其适用性。尽管一些方法整合额外信息以生成身份感知描述，但由于额外信息独立于视频内容，球员身份有时会出错。本文提出一种用于身份感知体育视频字幕生成的以球员为中心的多模态提示生成网络（LLM-IAVC），其重点是从视觉视角识别球员身份。具体而言，设计了身份相关信息提取模块（IRIEM）以提取与球员相关的多模态嵌入。IRIEM包括用于提取视觉特征和球员姓名的球员识别网络（PIN），以及用于将球员特征与视频内容关联以实现相互增强的双向语义交互模块（BSIM）。此外，设计了视觉上下文学习模块（VCLM）以捕获关键视频上下文信息。最后，通过整合上述模块的输出作为大语言模型（LLM）的多模态提示，促进生成带有球员身份的描述。为支持这项工作，我们构建了名为NBA-Identity的新基准，这是一个大型身份感知篮球视频字幕生成数据集，包含9,726个视频，覆盖9种主要事件类型。在NBA-Identity和VC-NBA-2022上的实验结果表明，我们提出的模型取得了先进性能。代码和数据集可在https://github.com/Zeyu1226-mt/LLM-IAVC公开获取。

英文摘要：

Existing sports video captioning methods often focus on the action yet overlook player identities, limiting their applicability. Although some methods integrate extra information to generate identity-aware descriptions, the player identities are sometimes incorrect because the extra information is independent of the video content. This paper proposes a player-centric multimodal prompt generation network for identity-aware sports video captioning (LLM-IAVC), which focuses on recognizing player identities from a visual perspective. Specifically, an identity-related information extraction module (IRIEM) is designed to extract player-related multimodal embeddings. IRIEM includes a player identification network (PIN) for extracting visual features and player names, and a bidirectional semantic interaction module (BSIM) to link player features with video content for mutual enhancement. Additionally, a visual context learning module (VCLM) is designed to capture the key video context information. Finally, by integrating the outputs of the above modules as the multimodal prompt for the large language model (LLM), it facilitates the generation of descriptions with player identities. To support this work, we construct a new benchmark called NBA-Identity, a large identity-aware basketball video captioning dataset with 9,726 videos covering 9 major event types. The experimental results on NBA-Identity and VC-NBA-2022 demonstrate that our proposed model achieves advanced performance. Code and dataset are publicly available at https://github.com/Zeyu1226-mt/LLM-IAVC.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。