OST：利用最优时空描述符精炼文本知识以实现通用视频识别
OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition
AI总结：
本文提出OST方法，通过LLM将动作类别扩展为时空描述符，并设计最优描述符求解器匹配视频帧与描述符，在零样本、少样本和全监督视频识别中取得领先性能，Kinetics-600零样本准确率达75.1%。
摘要：
由于在大规模视频数据上训练视觉-语言模型需要大量资源，大多数研究集中于将预训练的图像-语言模型适配到视频领域。主流方法提出通过额外的时序学习器来解决视觉差异，但忽视了网络规模描述性文本与简洁动作类别名称之间的显著差异，导致语义空间区分度不足并可能限制性能。在本工作中，我们优先精炼文本知识以促进可泛化的视频识别。为解决类别名称语义空间区分度不足的问题，我们提示大语言模型（LLM）将动作类别名称扩展为时空描述符，从而弥合文本差异并作为通用识别的知识库。此外，为了给不同视频实例分配最佳描述符，我们提出了最优描述符求解器，将视频识别问题形式化为在帧级表示与描述符之间求解最优匹配流。在零样本、少样本和全监督视频识别中的综合评估突显了我们方法的有效性。我们的最佳模型在Kinetics-600上取得了75.1%的最先进零样本准确率。
英文摘要：
Due to the resource-intensive nature of training vision-language models on expansive video data, a majority of studies have centered on adapting pre-trained image-language models to the video domain. Dominant pipelines propose to tackle the visual discrepancies with additional temporal learners while overlooking the substantial discrepancy for web-scaled descriptive narratives and concise action category names, leading to less distinct semantic space and potential performance limitations. In this work, we prioritize the refinement of text knowledge to facilitate generalizable video recognition. To address the limitations of the less distinct semantic space of category names, we prompt a large language model (LLM) to augment action class names into Spatio-Temporal Descriptors thus bridging the textual discrepancy and serving as a knowledge base for general recognition. Moreover, to assign the best descriptors with different video instances, we propose Optimal Descriptor Solver, forming the video recognition problem as solving the optimal matching flow across frame-level representations and descriptors. Comprehensive evaluations in zero-shot, few-shot, and fully supervised video recognition highlight the effectiveness of our approach. Our best model achieves a state-of-the-art zero-shot accuracy of 75.1% on Kinetics-600.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
