匹配、扩展与改进：利用语言知识进行零样本动作识别的无监督微调
MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge
AI总结：
针对视觉-语言模型动词表征不足、零样本动作识别需额外调优的问题，提出无监督微调方法MAXI，利用无标注视频与动作词典，通过匹配、扩展、字幕构建文本袋并结合多示例学习适配模型，性能提升最高达14%，迁移性优于全监督基线。
摘要：
大规模视觉-语言（Vision-Language, VL）模型在对齐视觉与文本模态的表征方面已取得巨大成功，推动了零样本识别、图像生成与编辑等诸多前沿任务的显著进展。然而，VL模型往往过度表征物体，对动词的关注度却低得多，因此需要在视频数据上进行额外调优，才能实现最佳的零样本动作识别性能。\n此前的研究依赖大规模全标注数据，而本文提出了一种无监督方法：我们利用一批无标注视频和一个未配对的动作词典，对VL模型进行适配以用于零样本和少样本动作识别。在此基础上，我们借助大语言模型（Large Language Models）和VL模型，通过匹配、文本扩展和字幕生成为每个无标注视频构建文本袋。我们在多示例学习（Multiple Instance Learning）框架下使用这些文本袋，将图文骨干模型适配到视频数据上。\n尽管是在无标注视频数据上微调，所得模型在众多未见过的零样本下游任务中展现出很强的迁移能力，相比基础VL模型性能提升最高达14%，在零样本和少样本视频识别迁移任务中甚至优于全监督基线。代码稍后将在https://github.com/wlin-at/MAXI发布。
英文摘要：
Large scale Vision-Language (VL) models have shown tremendous success in aligning representations between visual and text modalities. This enables remarkable progress in zero-shot recognition, image generation & editing, and many other exciting tasks. However, VL models tend to over-represent objects while paying much less attention to verbs, and require additional tuning on video data for best zero-shot action recognition performance. While previous work relied on large-scale, fully-annotated data, in this work we propose an unsupervised approach. We adapt a VL model for zero-shot and few-shot action recognition using a collection of unlabeled videos and an unpaired action dictionary. Based on that, we leverage Large Language Models and VL models to build a text bag for each unlabeled video via matching, text expansion and captioning. We use those bags in a Multiple Instance Learning setup to adapt an image-text backbone to video data. Although finetuned on unlabeled video data, our resulting models demonstrate high transferability to numerous unseen zero-shot downstream tasks, improving the base VL model performance by up to 14\%, and even comparing favorably to fully-supervised baselines in both zero-shot and few-shot video recognition transfer. The code will be released later at \url{https://github.com/wlin-at/MAXI}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
